December 19, 2005

The Work of Blogging

In the old days, running a blog used to be fun. You just set the blog up and writing about whatever took your fancy. But then came the comment spam. And then the referral spam (or was it the other way around? :p). Then the trackback spam, the pingback spam, the tagboard spam … you name it, there was sapm for it. When writing your blog became secondary and combating all this spam was the primary job, that was when blogging became a not-so-fun endeavour – at least for me.

I had not worked on my sites or my blog for such a long time that I had totally given up monitoring what was going on with my sites. Just a week or so ago, I noticed that the then SM sub-domain had about 1GB of traffic in about a week and I had not posted an entry there in months! So I began doing some digging and discovered that I’d done something really stupid back in the day and had not realized it :p I had set up a top 10 referrers list for my blog a long time ago and had not realized that referrer spam had been hitting it in ever increasing numbers till the referrer spam was doing around 300MB of transfers a day! That was when I finally noticed it.

But taking the top 10 referrer list down did nothing of course. These guys probably mark sites with referrer lists, set their bots up and then forget all about it themselves. So nobody was going to notice that I’d actually taken the top 10 referrer list down. I had to do something myself. A quick Google around was all that was needed. There were a lot of people who had to deal with the same issue and they were doing so by using mod_rewrite and .htaccess to do the job. So I sat down and went through the list of spam referrers to my site, came up with a list of rules and then put them up. The next day, the transfers from the SM subdomain went down to about 10MB from a whopping 300MB! I’ve had to keep an eye out on the traffic figures since then and tweak the referrer blocking list (there’s automated scripts out there which will parse your Apache log files and add new referrer filtering rules ….) but it’s been worth it.

Now I need to find a solution for the damn tagboard spam :p (The tagboard’s been disabled for a while now since all I get on it is spam …)

Be Sociable, Share!
Tags: Site
Posted by Fahim at 7:23 am   Comments (2)