Miscellaneous Knowledge

iptables is a common linux software firewall. However, for many new linux users who are unfamiliar with the command line (or google!) some simple tasks such as opening a port on the firewall in order to allow a web-facing service to be reachable by everyone can be daunting.

However, it’s actually quite simple. Just issue the following command;

iptables -I INPUT -p tcp –dport PORT -j ACCEPT

Where PORT is the port number you wish to open, e.g 80. Note that there is a double dash before dport.

The syntax of this command is quite simple. -I INPUT indicates you are altering the input table -p tcp indicates that the protocol is TCP, –dport PORT indicates the destination port and -j accept indicates the action to be taken on matching the rule – in this case to accept traffic. Note that the command requires superuser privileges to execute.

Don’t forget that this will be in vain if iptables isn’t actually running. Under CentOS, you can start the iptables service by issuing the command service iptables start. You’ll also need to save the iptables rules if you want them to persist across a reboot, under CentOS you can issue the command /sbin/service iptables save to achieve this.

Around mid december, I noticed that traffic for my Left 4 Dead Servers page dropped sharply – from 100 unique visitors a day to 5, at most.

Traffic Graph for www.l4dservers.net

Most of my traffic comes from search engine referrals, so to have a drop off like this must have something to do it Google.

Cue Google’s Webmaster Tools, a handly place for you to view all statistics about your site. On their diagnostics section I was getting a lot of “Network Unreachable” errors on the robots.txt. Basically, Google couldn’t access my site to read the robots.txt – nothing wrong with the actual robots.txt at all, it was a web server problem. After having a look around, I came across some information on diagnosing the issue, no help from Google itself on this matter of course. The information listed here indicated that certain software was blocking google for making too many requests the to the server, in a DoS attack fashion. This site is hosted on reseller hosting, however it shares the machine with a lot of other websites. In which case, Google has been accessing many sites on this server generating a high number of requests. Therefore, the software on the machine automatically blocked Google falsely beleiving that it was attacking sites on the machine.

This blog was also hosted on this reseller account, and traffic figures showed a similar decline. I’ve recently bought a dedicated cPanel server to replace the reseller and moved the blog onto there – bingo! This blog now appears back in search engine listings. If we look at Google’s Webmaster Tools, we can see the amount of pages crawled by it.

Google Crawl Statistics for Miscellaneous Knowledge

Right after moving it off the reseller hosting and onto the server, loads of pages are instantly and successfully crawled. The blog is back in the Google index.

www.l4dservers.net is still pending move to the new hosting, but I expect to see similar results once I have completed the move, just waiting on the DNS to be updated for it.

So what can you do if you experience this issue? First, read this excellent article and make sure that your lack of Google listing isn’t your fault. If you are not at fault, contact your hosting provider and point them to the article, ask them if they have any firewalls that block multiple requests such as the Googlebot. I would also highly suggest signing up to Google’s Webmaster Tools to help diagnose any problems Google has accessing your site. While they where not particually helpful in this incident, it did help to get me started on the diagnosis. Not only this, but it allows you to submit sitemaps which vastly improve your ranking and help make sure that every page is in Google’s index. Sitemaps are checked very often and any new URLs are quickly crawled helping you get any content into Google quickly.

From the more technical side, this article might be of interest. It details how ConfigServer Firewall might be a culprit in this.