OK, I’m much further along the path to the next beta build of Blog though I realize that the next build is going to be very much a hodgepodge of not quite ready features 🙁 There are certain major problems which aren’t always apparent with this new build that I’m working with. For instance, there seems to be a bug with the spellchecking since I could right-click on a misspelt word and it would take almost a minute before a menu would pop up. Since I have no access to the source for the HTMLEdit component, it’s very difficult to debug this kind of problem. It used to be much easier when the author used to respond immediately when I submitted a problem but these days, I either get no response at all from him or something like “I’ll look into this tomorrow” and then silence. I can’t really blame him too much though since I haven’t paid for the component and I after all am using a beta version. So while I do thank him again for his kind sponsorship, I am beginning to worry about where the component itself is headed if he’s either too busy to provide support or is not interested in doing so.

But on to the good news :p I’ve been working with Nigel’s data in trying to get it all over to his WordPress installation since it was Nigel who woke me up and got me working on Blog again 🙂 He’s got over 4,000 entries if I’m not mistaken and so far, I’m still in the first hundred or so :p However, I have fixed quite a few bugs and added a few user-friendly features that I might not even have thought of if I hadn’t been working with this data just because going through the transfers over and over again made me see things that I might not have seen otherwise :p The transfers to WordPress are working fairly well now – in fact, I am using WordPress over at SM and have been posting quite regularly with the new build of Blog with no problems whatsoever. However, there is one aspect of transferring to WordPress which is still giving me problems – special characters like the British pound sign (£ … not to be confused with the hash sign which is also called the pound sign :p) or the Euro (€) sign. These seem to cause the XMLRPC server on the WordPress end to throw a hissy fit :p I have tried encoding the characters via HTML encoding but have met with mixed results – it sometimes works but has an extra character in front of the character I wanted and sometimes even the encoding throws a hissy fit. I have a feeling that this is connected to the fact that WordPress uses UTF-8 encoding for all XMLRPC stuff but haven’t had enough time to delve into things to figure out what exactly is happening and how I could try to fix it.

In fact, my aim today is to rip out some of the XML parsing routines from the WP code into a separate routine, dump the XML sent from Blog into that code and see if something blows up. Having to debug between two different languages is a pain in the posterior and this way, I’ll only have to deal with PHP till I figure out where exactly things are going wrong in the first place – whether it’s the PHP end of things in WP or the Delphi end of things in Blog. Once I figure that one out, I should be fairly on my way to getting all of Nigel’s data transferred over. But as to when the next beta will be released, all I can say is that I hope it’s soon but I’d hate to inflict people with a not-quite-working WYSIWYG view since that’s supposed to be one of the major features of Blog 8.0 :p

Blacklists and blind-spots

After yesterday’s post about the apparent problems with my WPBlacklist plugin, I decided to do some testing on my own to see if things were indeed not working. And I’ve got both good and bad news :p The good news is that the WPBlacklist plugin does work and there are no hidden loopholes – the bad news is that it doesn’t exactly work the way you think it would :p I realized during the course of my investigation that I had either not paid as much attention to what happens in the WordPress core as I should have when I wrote the WPBlacklist plugin or that I did know one critical piece of information which I had then proceeded to forget all about :p

The piece of information? Well, actually it’s two bits of information :p One was the fact that I had not added any code to notify the user when a comment was held for moderation by the WPBlacklist plugin – it simply holds the comment silently – the strong, silent type :p The second bit of information was that the WP core code actually sends out confirmation of a comment posting (or of it being held for moderation) based on its internal spam list before it initiates the hook which allows external plugins like WPBlacklist to take a shot at comment spam. So basically, if a comment is not caught as spam by the internal spam list in WP, then the user gets a notification saying a comment was posted on his/her site but WPBlacklist in the meantime might have actually caught the problem and put the comment on hold silently. This actually explains several reports I got from users who said that WPBlacklist wasn’t working even when certain words were in their blacklist – they were probably depending on the e-mail confirmation instead of checking the actual comments appearing under an entry – or, at least, that’s what I think now :p

Of course, finding the cause is just half the solution. Now I need to find a way around it. Unfortunately, to provide a solution that works for all, it looks as if I will have to modify some of the core WP code and submit it to the WP devs and hope that they accept the change – or something. I’ve already checked the CVS code from the end of August and it still has the same problem. So, unless they’ve fixed it within the last two weeks, I think I’ll need to submit my own fix to the WP devs. Of course, this also brought up the possibility that I might have my own bug hiding in the WPBlacklist plugin code – I don’t remember whether I checked to see if a comment was already held for moderation and if so, then simply exit the check routine. Because if I didn’t do that, then it is possible that something which got tagged as spam by the internal routines (perhaps because it had too many links in the comment ..) might get untagged by WPBlacklist – now that just wouldn’t do :p So now I’ve got lots of stuff to keep me occupied for the next few days 🙂

The blogger’s bane …

Comment spam is increasingly becoming a major problem 🙁 There used to be a time when I’d get maybe one or two spam entries a month but of late, there are days when I literally get flooded by spam entries. I had written/modified the WPBlacklist plugin soon after I shifted to WordPress just because of this problem. I had been using a blacklist plugin while using Movable Type and I knew that I’d probably get hit by comment spam if I wasn’t protected on WP too. The blacklist plugin seemed to work and I was content for a while but a couple of weeks back, the latest problems started.

I’d been offline for a couple of days and came back to discover that I’d been hit by around a hundred spam comments on various posts scattered over my blog. Fortunately, WP makes it fairly easy to remove comments en masse and an SQL command was even simpler in this particular instance because the spammer used the same URL for all of his comment spam. Once I removed the spam however, I went back and took a look at the spam itself and discovered a few things:

1. The spam was from many different URLs
2. The particular site s/he was spamming for was not in my blacklist
3. I had set the WP option which marks a comment as spam depending on the number of links in the comment to look for five links.

So, I went ahead and added the URL for the site to my blacklist and lowered the link threshold for spam to two. I thought I was safe but that wasn’t the case as I discovered a couple of days later.

I again got hit by spam but this time, I was online when the attack started. I discovered that the new spam comments had only two links and they were getting through WP’s spam protection. I went back and checked the wording on the threshold setting and it seemed to indicate that if I wanted two links to trigger spam blocking, that I should actually set the count to 1. Or so it seemed at that point though I’m not really sure about that since I’ve messed about with that setting a bit since then and it seems to work either way now. Or something. Anyway, I was able to stem the flood of comments to about 75 on this round and nothing got really published except for a couple in between me changing settings since all the comments got held for moderation that time.

I was kind of wondering about my blacklist plugin by this time though since I did have the URL in the blacklist and the blacklist didn’t seem to catch the spam – it was just the built-in moderation in WP based on the number of links in the comments which was putting the stuff in to a queue to be moderated. I decided to add a few more variants on the URL in case the one I had originally added didn’t work and wait for further developments.

A few days ago, I got attacked again. This time, all the comments went into moderation since I had the link count set correctly but when I went in in the middle of the attack and removed the link count check, I started getting comments which were published. So, my blacklist plugin for some strange reason wasn’t working properly! Or maybe the spammer was using a method which bypassed the blacklist checking? I don’t know … I’ll have to take a look at the blacklist plugin again and see what transpires – guess I’ll have to do that soon though since based on the regularity of the attacks, I probably should have one occurring in a day or two …

Done, done at last! Thank God almighty, done at last!

The last entry for this site, which was made somewhere in June, was never published at all due to the fact that Blog wouldn’t cooperate :p I finally got that entry to publish today after getting most of the major problems in Blog. Actually, I’m kind of happy to get Blog working again since now, I use one application, Blog, to post to both my WordPress site and my non-WordPress site :p That is kind of cool.

A lot has happened since I last posted here and I’d probably have to make a humongous entry to catch up with all the stuff that has happened :p I just don’t feel like making the aforementioned humongous entry due to the fact that I’m not feeling totally well today. So I guess I’ll do the Cliff’s Notes version of what has been happening so far instead :p

OK, I left my then employer and joined a new company somewhere in May. This new company had been a project of my then employer’s and I’d been involved in the project for about a year before they decided to go off and form a separate company and they wanted me involved. I did get involved but found that I’d taken on way too much than I’d bargained for. I found myself doing systems administration, HR duties, procurement and a heck of a lot more besides when I had actually been hired on to be their lead developer. Things got a bit too hairy for my tastes and so, I went back to my old employer (who now is my new employer … again :p).

I’ve been here at my new/old employer for a couple of months now and things have been fairly hectic since I’ve had to do a couple of proposals as well as a full promotional video for the Information and Communication Technology Agency (ICTA) of Sri Lanka. I really enjoyed the work on the ICTA video since it was something different from the usual run of the mill stuff I get to do. Unfortunately, till about a week ago, all this stuff meant that I was kept fairly busy since I also had taken up writing for another monthly magazine here in Sri Lanka (in addition to my existing column on a different monthly magazine :p) and what with trying to do all that and keep up with my comics, movies, TV shows and so on, not much time was left over for doing other things … such as coding.

Since I couldn’t post here till I got back to coding and got Blog working, that presented a bit of a problem as far as posting on SM was concerned. Of course, I really didn’t have much time to worry about SM and so, I’d think about getting back to the code, but really wouldn’t do it :p I’ve finally been able to get back to working on Blog – mostly because a friend needed Blog working more so than because I needed it working :p – and so, hopefully, I should get back to posting here too. We’ll see …

And then he was back :p Basically, I haven’t been posting for the last few months because Blog was broken and I was too lazy to go back to an older version. Let me first see if this posts fine and then maybe think about writing a longer entry …

Hallelujah! It works!! At least, the posting via FTP does … but there are other behind the scenes issues which are going to make things a bit difficult for a while. Basically, I got all the work necessary for translating the existing data into the new database formats which support storing journal entries irregardless of where the entries are finally published to at least a couple of months ago. Unfortunately, just after I got all that work done and added the code in for publishing to remote blogging servers, I ran into several problems with the WYSIWYG HTML editing components. The usually reliable author of HTMLEdit would not respond to my bug reports and I was totally baffled by the problem since I didn’t have access to the source.

So, I decided to find a replacement component and looked at several. The authors of WPTools and TRichView were extremely kind in providing me with a free license to each of their components. However, I found that in each instance, I would have to do a lot of recoding and would lose a few existing features that I found extremely useful in Blog using HTMLEdit. So, I have been playing a waiting game, trying different components when I ran across them and then discarding them and at the same time, hoping that the HTMLEdit author would release a new version which fixed the problems that I had. Of course, nothing seemed to happen at all for a couple of months … or a bit more.

Then I was shaken out of my apathy by Nigel yesterday :p Nige has been a long-time Blog user. His site, The Red Ferret Journal, has around three years of archives totalling probably thousands of entries – all in Blog. Nige has been thinking of shifting to WordPress so that he can allow multiple users to post to his site but still be able to use the Blog interface by using the new functionalities that I was supposed to add with the next release of 8.0 :p So Nige has been waiting patiently for quite a while for me to get my posterior in gear and get Blog out but of course, it wasn’t happening. He finally decided to write to me and see if he could get me to expedite the process since he urgently needed to shift to WP and I finally decided to give the code another look and see if perhpas I could decouple the WYSIWYG component from everything else so that I could at least come up with a Blog version which did what Nige wanted for the short-term while I figured out the WYSIWYG problem. However, in trying to sort out the code, I actually figured out how to get things to work – or so I thought at that time :p

Of course, things haven’t turned out to be that easy. I did figure out the problems with the HTMLEdit componet but I discovered in the process of making this entry why I’d had problems in the first place. The original problem which started all this stagnation was related to the HTMLEdit component dropping images when an entry was saved. I realized today that only images but even formatting is dropped by the HTMLEdit component on first save. However, I realized that if I went to the HTML view and added some HTML tags and then switched back to the WYSIWYG view and saved the entry, it saves perfectly fine :p Ah well, the joys of coding …

In the meantime, I’ve also discovered that the code which seemed to work fine with WordPress a couple of months ago is now giving me trouble. But then again, I just came across an article online which suggests that this might actually be XMLRPC and PHP related problems than Blog related problems. So I’m going back to the code to see if there is a way in which I can fix all these things so that I can finally get that long-delayed version of Blog out :p