If anybody is worried that I might have gone into hibernation again because there have been no posts at all this week, fear not :p The fact of the matter is that some relatives are in town and are staying with us for a couple of weeks. I took this week off from work so that we could take them shopping and to see places and so on and so, I’m only online long enough to make sure that things are running smoothly and to get my e-mail. In fact, I haven’t had any time to even work on any code since Monday – when they arrived 🙂 So, updates will be slow and any new builds would be even slower to arrive but I certainly haven’t gone into hibernation – hopefully, I should be back online on a more regular basis from next week on.

In the meantime, I know that some people are eagerly awaiting the next build of Blog – I could release it even today but you would hate me for it because there is one major bug in the current build – the fact that all HTML code and formatting gets dropped from all new entries in the WYSIWYG view. The author of the HTMLEdit component has promised a fix for this in his next build and I am awaiting that build so that I can release the next build of Blog.

I’ve been working on a new release of my WPBlacklist plugin for WordPress (I really should get a page up for the plugin soon so that I can have a place to point people to :p) and personally, I think the new one kicks butt :p I have a few more features to complete and some testing to do and if I can find some time this week, I’ll probably finish that and try to get a release out by the beginning of next week so that people can better combat all that annoying spam which is floating around.

Speaking of spam, last week and this week seems to have been especially bad since I have a feeling that somebody either released a new tool that allows people to submit spam via HTTP POSTs or a lot of spammers just found out that they could do it since I was getting spam on anything where you could do a POST. I was getting spam on the forums, I was getting spam on WordPress comments and of course, I was receiving spam on e-mail! After a lot of quick fixes and tightening of security measures, the spam flood seems to have been slowed to a trickle. I got an e-mail telling me that somebody who tried to access my old WP comments script (which is now just a spider-trap) got banned by my spider-trap script. So I know they are still trying to flood me but not getting anywhere. I also set up a trap for people who try to sign on to the forums as new members just to display their spam links – that trap has not been triggered yet but then again, I haven’t had any new spam sign-ups either – so I’m happy 🙂

First it was e-mail spam, then it was comment spam and now it’s user account spam? At least, that’s what it seems like to me from what I’ve been observing over at the forums. Recently, I noticed several people signing up with user names that looked slightly strange. At first, I gave it no thought but I’ve noticed more and more people signing up over the last few days but none of them seemed to be posting. Now, as you might know, you don’t need to sign up to read the forums, just to post. So there seemed to be no reason for these people to just sign up if they only wanted to browse the forums to find an answer to a question. So, I went and took a look at the profiles of the latest members of the forums and guess what I found? All of them had one thing in common, URLs that seemed to point to less than reputable sites :p

My first reaction was something like "Oh damn it! Now I have to find a way to stop these spammers as well?" Then I realized that in this case at least, I didn’t have to take so drastic a measure – at least not yet :p All I had to do was simply turn on the "approve" feature which lets the administrator approve a new user who has signed up. At least, I think that should do the trick unless of course, phpBB simply prohibits users waiting approval from posting and still displays them on the user list. I believe that the whole aim here for these spammers (if spammers they be) is to get their URL displayed on the member list. If phpBB simply allows anybody who signs up to be displayed on the member list whether they have been approved or not, then I’ll have to look at a different approach.

For the moment, I’ve gone through the member list on the forums via SQL and deleted all users who had never visited the site since they had signed up and who had no posts at all on the forums. However, this might not be enough if this is going to be the next spam frontier. I guess I’d better start thinking about new spam combatting methods just in case this is the start of a new spamming trend 🙁

I was getting all set to add more features to my WPBlacklist plugin yesterday when I realized that I was going about things all wrong :p I was adding more features to a plugin which I had to keep on maintaining and updating each time a new spam URL came in or the spammer found a new IP to flood me with spam from. Instead, I should have thought about why I was getting spammed so often and come up with a solution which prevented this. So I thought about it a bit …

I have two blogs this one (DC) and the one over at SM. Now DC, is maintained via Blog and uses a commenting system that I wrote while SM uses WordPress as the backend blogging system and uses the built-in comments from WordPress. Now, I’ve probably received like 3-5 spam comments here on DC whereas I’ve been spam bombed on a regular basis over on SM. So, why is it that I get more spam on SM than DC when DC has been existence longer than SM and probably has a higher presence on Google? I realized that the answer might lie in the fact that I used WordPress over on SM.

My logic was along these lines – if you ran a standard WordPress installation, you’d have a specific comments file (the file name itself is well known) and anybody who takes a look at the WP source, can figure out what the comment submission variables are. Then, all a spammer has to do is to write a script which calls the comments form’s action script (the one which processes the submitted comment) with the correct variables populated with the spam values. So, what if I changed the WordPress code so that the comments script was named differently? This would take care of part of the most basic robot scripts which simply hit every site with a WordPress blog and looked for the standard comments processing script but what if the robot actually went through your index page to figure out what the actual comments submission script was and then simply submitted comments to the script name it had discovered? That was easy enough to tackle too – I simply changed the comment variable names on my installation of WordPress.

Of course, the spam robot might be written so that it would parse the main page not only for the comments script name but also the comments variables. So, I added a third level of security by populating my main page with a couple of dummy forms which pointed to (among other things) the original WP comment submission script but I had replaced the code in the original script with innocuous stuff which did nothing at all :p

That probably would have been as far as I would have gone if I hadn’t started reading up about spam comment prevention online :p The above methods would not discourage a determined spammer but they would have been enough to deter most of the script kiddies out there who are probably simply using a script written by somebody else and are basically using their spam robots to target standard WP installations which are easy prey. But what if I was dealing with a more determined attacker who had a robot script which was really thorough? I found an answer to this in my online research in a blog entry about how to set spider traps :p This article basically explained how you could create a script which would identify bad robots which tried to access disallowed content on your site and then ban the IP they came from. I modified that method to mark the dummy spam submission scripts I’d entered into my main page as disallowed content so that any robot script which disregarded my warnings and tried to parse the dummy scripts would automatically get banned. Now, I’m waiting to see what happens – if I still continue to get spam or if the floods stop … or, I might become the next challenge for some idiot out there who becomes determined that he should get me since I went to such lengths to stop the spamming and explained it here so others could do the same :p

I heard back from Fabian, the developer of HTMLEdit, and he was kind enough to promise a fix for the problems that I’ve been facing with the component and since he’s been talking of a new build soon, I have decided to wait for his build before releasing the next beta of Blog 8.0 🙂 In the meantime, I’ve been using the last build I created for several days now and I find it to be fairly stable since I was able to fix one of the last problems which had been nagging me, yesterday. The problem? There would be extreme slow-downs when switching from the WYSIWG view to the HTML view and I would have to wait over 20 seconds before the right-click menu for a misspelt word came up. Both these problems seem to have disappeared since I replaced the spellchecking component used in the application with a fresh copy. Yes, that’s exactly what I said – I didn’t replace the component with a different component, but I replaced the copy of the component in the application with a fresh copy of the exact same component and it seems to work fine now – go figure :p I did read something on the forums about the HTMLEdit component not liking to work with a shared copy of the Addict spellcheck component but so far, after putting in the new copy of the component, I don’t seem to have problems with sharing. Guess I’ll wait and see.

In other news not related to Blog or Delphi coding, I seem to be getting heavily hit by comment spam over on SM. So far, I’ve been talking about comment spam related development over on SM since that’s the WordPress blog and so I thought I might as well talk about WordPress related development over there. But then, that breaks the whole "development here", "meandering thoughts there" rule :p But I’ve done it that way anyway since I like to try and post in both places several times a week and sometimes I’m too busy to post in both places on the same day 🙂 However, today I had something I really wanted to say over there on SM and then I came into work and discovered that I’d been hit by comment spam again and this time my WPBlacklist plugin had actually not caught the spam – darn it! Of course, a blacklist is only as good as the entries in it and in this instance, the spammer was not in my blacklist. So I immediately rectified the situation, deleted all the comment spam and then decided that I was going to spend some time working on adding all the extra bells and whistles that I’d been meaning to add to the WPBlacklist plugin. So, that’s what I’ll be working on for the next few days :p

The Miracles around us

Have you recently sat in the rain and looked up at the lightning playing across the sky? Or, sat on the grass on an early morning, feeling the freshness and the coolness of the grass under you and the warmth of the sun on your face while the birds sang and the fragrance of flowers wafted all around you? I don’t know about you, but I seem to be doing less and less of this kind of thing which actually makes us realize how good life is and all the little miracles that surround us … and this is not a good thing.

It started raining heavily in the morning today just as I got up. I washed, prayed and then sat on my couch for a little while, staring out the window at the rain falling down and the lightning streaking across the sky. I really wanted to go out and just sit in the rain and let it splash all over me like I used to do when I was a child. Why is it that as we get older, we get more and more conservative and so …. dull? :p Why is it that we start thinking about what other people will think/say, whether we’d get sick and so on, instead of just going out and doing something which is fun? I don’t know. But I do know that I didn’t go out and just romp around in the rain. But I did sit there and look at the rain and think about all these things 🙂

It was then that I had the thought that the rain falling outside, the sunshine on our face, all these things are little miracles of God. When something big happens, like a huge storm coming up (I was thinking of "The Day After Tomorrow" at this point but that’s a different discussion :p), we start thinking of God and sometimes even say that it’s an "act of God". But we always disregard the little miracles which are all around us, I guess because we’ve gotten so used to them and they have lost their miraculousness. This is probably why I love a child’s viewpoint of the world much better than I love an adult’s – because children have still not become so blasé about all the little wonders that are all around us and they still look out at the world with innocent eyes. But the truth of the matter is that there is beauty and wonder to be seen, enjoyed and marvelled at, all around us … if we’d just take the time from our daily grind to notice these things. So just take a minute off your day to smell the roses, to stare out your window at the greenery of the trees or how blue the sky is and realize that the world might not be as dreary as it sometimes might seem 🙂

Soul Brothers and other stuff

Do you think people can have soul brothers or sisters? Or at least spiritual brothers/sisters? Actually, that is a throwaway question prompted by the fact that I was going to write about somebody who I consider to be a kindred spirit – or at least of his blog :p I’ve known Tyran for quite a long time now and whether he’s writing about wireless or wagons, Firefox or Frodo, blogs or Barsoom, I know what he’s talking about … Or I can empathize with what he said. I’ve always made it a point to drop by Tyran’s blog first thing in the morning because whatever he writes about, it’s bound to be interesting, if not thought provoking. Which is why it surprised me today when I discovered that I had not been by Tyran’s blog in almost a year! Guess, that’s what marriage, shifting to a new job, going back to your old job again and all the other stuff in between does to you :p

Anyway, I decided to take a day off from coding and scripting and debugging and all the other good stuff that keeps me busy during the day, and catch up on my reading over at the Whinery. Of course, as usual, it was interesting and luckily for me, Tyran seems not to have been too prolific in his own blog output since I was able to get through a full year’s worth of entries in one day. Of course, if you’re interested, you can go over there and read his blog yourself :p

Then there is Nigel with his Red Ferret Journal. I’ve known Nige for almost as long as I’ve known Tyran and Nige also probably has the distinction of the Blog user with the most entries in his database :p I’ve been using Blog longer than anybody (I wrote it after all :p) but I’ve only got around 750 entries when you combine four separate blogs (some of which are no longer active) whereas Nigel has a whopping 4000+ entries in his single blog! Nige (like Tyran) has been a very good friend through the years though I’ve never met either of them in person. And this, probably is the greatest gift that I’ve recieved in the years I’ve been developing freeware applications – the great people I get to know and to make friends with. There are many others that I count as friends but I’ve lost touch with them through the years due to some reason or other but this is in remembering all those people who’ve enriched my life over the years 🙂

And that meandering entry is my excuse for not doing too much in the way of coding today :p Speaking of coding, I seem to have caught my first comment spam with the new and improved WPBlacklist 1.22 plugin 🙂 Unlike the old version, I even got an e-mail notification of the fact that a comment was held for moderation. Now, if I could just add the option to check external spam lists and a mechanism to search existing comments for specific keywords and IP addresses, I should be ready to release WPBlacklist 2.0 🙂 I’m kind of eager to get started on it but I think I should get Blog done first. But if you’re looking for information on Blog, this is not the place to be … DC is :p

Funky Dung suggested a few days ago that I should think about providing a syndication feed for DC so that people can figure out that I’d come back from my hibernation (I expect that he expected this hibernation thing to be a regular occurrence … and he might not be wrong :p) instead of having to check back on DC every so often. I told him that I had to first get the RSS template going and put that on backburner. Yesterday, I had enough time to look into implementing an RSS template and since I also needed to figure out if the new build of Blog works nice with RSS templates, I gave it a go 🙂

The result, after some messing around, reading about RSS standards and so on, was a fairly workable RSS template. I still have a few validation problems but most of the major ones are gone. The new format attribute for the date and time tags has helped tremendously and I’m hoping that others will find this little feature useful too 🙂 In fact, in testing out the RSS template, I found one bug in the format attribute, so I guess it’s a good thing that I did work on the RSS stuff :p I still get a validation error on non-UTF-8 characters in my entries but that might be due to the fact that the validator thinks my feed is encoded in UTF-8 for some reason … perhaps because that’s the default? :p As you can see, I still have a lot to learn about RSS and so will have to keep on plugging away at my RSS template till I get it working perfectly. Then, I’ll probably include that in the next build of Blog as a default template 🙂

In Blog related news, the latest builds of Blog seem to be working fairly well with WordPress. Nigel suggested a great feature – that of uploading imges via FTP to a remote blogging server 🙂 I had initially been looking at implementing image uploads via the XMLRPC interface but couldn’t do that for WordPress since their file upload feature wasn’t really complete the last time I looked :p However, when Nigel suggested using FTP, I realized that all I had to do was use what was already available in Blog – I didn’t have to wait on the XMLRPC stuff to be worked on! So I added a new option to remote server settings where you can specify the FTP server to upload images and added a couple of other settings to making things work and voila, it was done 🙂 Nigel has tested it out since then and tells me it all seems to work fine 🙂

I’ve also heard from Fabian the author of the HTMLEdit component that I use. He explained why all formatting (and image info) would be dropped when I saved a new entry – the HTMLEdit component decides whether the data it is working with is plain text or HTML by examining the data when it is first loaded. So, since there is nothing to examine when you start a new entry – it thinks that the data is plain text! While I do understand his explanation (in fact, I had deduced that it must be something like this and was going to do some experimentation as soon as I got the latest stuff in Blog working), I don’t think the behaviour is very intuitive – if it is a new record, the component should actually check the data it is saving rather than what it loads. I’ve asked Fabian whether it would be possible to make this change and am waiting on his response but in the meantime, I’m sure that I can at least do a kludgy workaround that will allow me to release the next build of Blog fairly soon 🙂

Blacklists and other bits

I talked about this particular bug in the WPBlacklist plugin that I wrote a few days ago as well. I mean the one where a comment which had been held for moderation by the internal WordPress moderation system could be approved by the WPBlacklist plugin because it doesn’t check the status of the comment before running the comment through the blacklisted item list. I got bitten by this bug just yesterday when a spam comment which was held for moderation because it had multiple URL’s in it, was approved by the blacklist plugin because those URLs were not in the blacklist. I immediately added the necessary changes to the code, tested it out and uploaded the new script to my server. Since this is a bug which will affect others as well, I decided to release WPBlacklist 1.22.

However, I didn’t stop at changing just the plugin code. I also attacked another problem which had been bugging me – the fact that I don’t get a "held for moderation" e-mail when the WPBlacklist plugin holds an e-mail for moderation. This was due to the way the internal WP code worked and so I decided to change the code a little bit so that it worked the way I wanted. I’ve included the changes in the readme.txt file in the WPBlacklist distribution but in case somebody is interested, here is a excerpt direct from the readme.txt file … Umm, I tried posting that bit but it really messes up the paragraphs for entries here on SM :p So I guess you’ll just have to read the readme.txt file to find out what you need to do …

The Warrior’ Path

I watched "The Last Samurai" yesterday. Or rather, I actually began watching it on Saturday but couldn’t finish it on Sunday since I had some other stuff to do and so finished watching it yesterday :p I loved the movie. I had not thought I would like it as much as I did when I started watching it but by the time the movie ended, I was loving it. There were moments in the movie which left me pondering about things and that always is a good thing for a movie. What struck me most about the film was the fact that we all have habits, customs or a way of life that we think is the "right" one. It is only when we come face-to-face with something totally alien do we realize that there are other ways than ours and while that might not be the suitable way for us, we must respect the ways of others if we are to ever live in peace on this Earth of ours.

One scene that struck me especially was towards the end when Algren (Tom Cruise) and Katsumoto (Ken Watanabe) charge the army of Omura (Masato Harada). While I by no means like violence and would rather find a solution through discussion, I was left with the feeling "this is as it should be, person to person, face to face" when I saw them charging the opposing army. And then … they get cut to pieces by the Gatling gun of Omura and fall in their tracks. I was thinking how low have we fallen, no longer will we even take the honourable way of fighting (if we must fight) by facing our enemy and taking the same risks as s/he does. Instead, we stay far away in safety and fire upon our enemy with weapons that kill without distinction – combatant or non-combatant … there is no difference. Worse yet, our commanders and leaders (the ones who actually order these wars/killings) sit even further away (perhaps even in a different country) safe from all the destruction that they order. I see no honour in such a way.

But then again, was is not about honour any longer, is it? No, wars are fought by the greedy, the opportunistic – people like Omura in "The Last Samurai". People who are only concerned with gaining power or wealth – not about something as transient as honour. Of course, on the other hand, one might say, "What honour is there in killing somebody else?" but that is not my point. My point is that if kill you must, then kill face-to-face, against an opponent who is armed the same as you and has as equal an opportunity of killing you as you them. Perhaps then we would have less wars … but then again, given humanity’s track record so far, that seems unlikely.

So what’s the stat, Matt? I’ve been working over the weekend on the weird XMLRPC problem when trying to send a post from Blog to WordPress. The problem, as I mentioned in the previous entry, had to do with various non-standard characters like the pound sign (£) or the Euro sign (€). The problem seemed to crop up because the XMLRPC code in PHP expects the character data sent from Blog to be UTF-8 encoded whereas I am not exactly certain how the data from Blog is actually encoded – but it certainly is not UTF-8 :p I had been hoping to find a solution which would allow me to do a fix from the Blog end without touching the PHP code in WordPress but unfortunately, after about a day of testing and trying various avenues of attack, I was basically no further ahead than I had been the day before.

So, I finally decided to settle on a solution which would involve changes both at the Blog end and the WordPress end. Basically, I would encode selected characters with their numeric HTML entity equivalent at the Blog end. So, £ would become £ at the Blog end. You might ask why I didn’t use the HTML character entity equivalent of £ for £ instead of £ and it would be a good question since that’s what I too wanted to do originally :p However, at the WordPress end things blow up if I used the character entity since apparently, the £ entity is not defined in the XML spec and so is not recognized by the PHP XML parser 🙁 I looked at a few suggested workarounds for this problem but none of them really worked for me and so, I decided to go with the numeric entity since that at least worked … after a fashion.

But this still wasn’t the whole solution because while the numeric entity was not outright rejected by the XML parser, I would always get a strange character in front of the actual character whenever it was parsed by the PHP XML parser. So, instead of the single character that I originally had, I would get a garbage character followed by the actual character. This turned out to be due to the whole non-UTF-8 encoding problem that I’d already talked about and so, I decided to add some extra code to the XML parsing routines in WordPress to handle this particular scenario. The new code looks for particular instances of two character sets which meet certain criteria and then encodes them into UTF-8 and so far, it has been working fairly well. Of course, as to whether this particular change would affect something else in WordPress is something that I can’t confirm or deny totally at the moment :p However, I’ve managed to transfer around 600 out of the 4,000+ entries that Nigel gave me and progress seems to be much smoother now. Stay tuned for further developments …