Miscellaneous Knowledge

Around mid december, I noticed that traffic for my Left 4 Dead Servers page dropped sharply – from 100 unique visitors a day to 5, at most.

Traffic Graph for www.l4dservers.net

Most of my traffic comes from search engine referrals, so to have a drop off like this must have something to do it Google.

Cue Google’s Webmaster Tools, a handly place for you to view all statistics about your site. On their diagnostics section I was getting a lot of “Network Unreachable” errors on the robots.txt. Basically, Google couldn’t access my site to read the robots.txt – nothing wrong with the actual robots.txt at all, it was a web server problem. After having a look around, I came across some information on diagnosing the issue, no help from Google itself on this matter of course. The information listed here indicated that certain software was blocking google for making too many requests the to the server, in a DoS attack fashion. This site is hosted on reseller hosting, however it shares the machine with a lot of other websites. In which case, Google has been accessing many sites on this server generating a high number of requests. Therefore, the software on the machine automatically blocked Google falsely beleiving that it was attacking sites on the machine.

This blog was also hosted on this reseller account, and traffic figures showed a similar decline. I’ve recently bought a dedicated cPanel server to replace the reseller and moved the blog onto there – bingo! This blog now appears back in search engine listings. If we look at Google’s Webmaster Tools, we can see the amount of pages crawled by it.

Google Crawl Statistics for Miscellaneous Knowledge

Right after moving it off the reseller hosting and onto the server, loads of pages are instantly and successfully crawled. The blog is back in the Google index.

www.l4dservers.net is still pending move to the new hosting, but I expect to see similar results once I have completed the move, just waiting on the DNS to be updated for it.

So what can you do if you experience this issue? First, read this excellent article and make sure that your lack of Google listing isn’t your fault. If you are not at fault, contact your hosting provider and point them to the article, ask them if they have any firewalls that block multiple requests such as the Googlebot. I would also highly suggest signing up to Google’s Webmaster Tools to help diagnose any problems Google has accessing your site. While they where not particually helpful in this incident, it did help to get me started on the diagnosis. Not only this, but it allows you to submit sitemaps which vastly improve your ranking and help make sure that every page is in Google’s index. Sitemaps are checked very often and any new URLs are quickly crawled helping you get any content into Google quickly.

From the more technical side, this article might be of interest. It details how ConfigServer Firewall might be a culprit in this.

Update: I’m no longer recommending Google Checkout after they raised their fees to be equal to that of PayPal. Google’s only real advantage was their low transaction rates, now that it’s at parity it does not make sense to continue using them. See here for more information. I leave the article below for referrence on what was once a great service.

Customers of my game server business may have recently noticed that we’ve added a new payment method, Google Checkout. There have been many reasons driving this, both ones that benefit customers and me.

First of all, from the customer side, Google Checkout is much simpler to use and offers a lot more control over payment history. We’ve had a few customers who have had a bad experience with PayPal and refused to use it. However, unlike PayPal, Google Checkout can only accept a Credit/Debit card as a funding source, so you can’t pay using existing funds or a bank account. However, we found that most people used a Credit or Debit card anyway.

On our side however, it’s even better. While we have to hand-process each order (charging the card etc) the way is done is very good. Plus, we also get a lot more control over the order and like it is for the user side, the history is excellent. There is also the added benefit of having the Google Checkout badge on your AdSense results, which helps make your ad stand out.

One major factor that many might consider is fees. PayPal’s base rate is 0.25p + 4%. This can quickly mount up, with a £24 payment having over £1 in fees, and if you have micropayments it gets even worse. Google Checkout however has 0.15p + 1.5%, faw lower. It doesn’t stop there however, if you advertise using their AdWords service you get £10 worth of sales processed for free for every £1 in advertising you spend. So say you spend £10 on AdWords a month, that gives you £100 of sales processed for free (note: not £100 worth of fees). So the savings are even greater! (More lower prices I can pass on to users)

Perhaps the most interesting bit for us however is that payments are initated to our bank account daily. Unlike PayPal, where you have to get £50 first before you can withdraw. That’s upto £50 that you are not earning interest on and is completely under PayPal’s control.

I will however keep offering PayPal, while Google Checkout is far better when you compare it it has the disadvantage of being unfamiliar to users and not supporting Bank Account funding, so as not to alienate parts of my userbase i’m running both but recommending Google Checkout.

Update: Got an email the other day from Google saying that I didn’t qualify to have the Google Checkout badge on my AdWords adverts. Apparently, they couldn’t find the badge, so they must just quickly check through. I emailed them back and they checked again and found it (you have to fill out an order form). While the AdWords badge isn’t particually important, it helps you stand out from competitors.