This week has been good coding-wise but unfortunately, none of it was on Blog 🙁 Speaking of Blog, the modified HTMLEdit component which fixes new entries losing all HTML tags when saved, is here. In fact, it has been here for over a week now. Unfortunately, I’ve been either too busy or too caught up in other stuff (like downloading and reading a whole bunch of comics :p) to really do much about it. The problem is not just laziness – it’s also the fact that I’ll have to work on a new area in Blog before I can do the next release. After I did my last set of changes, I thought I was ready for the next beta release but then Nigel ran that particular build through the mill and discovered a few problems – such as the fact that this new build of Blog does not make allowances for the More … tag. Any entry with a More … tag will publish fine if you publish via Blog but not on a remote blogging server like WP. I’ll need to introduce another change that I’d planned but hadn’t implemented yet – a different tab for extended entries and perhaps even another tab for excerpts – before I can do the next beta release of Blog.

Of course, while thinking about all these changes to Blog, I’ve been coding two new apps. I don’t know if either of them will ever be released publicly and would probably remain in that huge collection of half-done/non-public apps that I’m building up :p The first app is actually based on somebody else’s code and wasn’t coded from scratch as are my usual aps. Since I’ve been downloading a lot of comics recently, I found myself running out of hard disk space and needed to burn some stuff on to DVD. However, the nature of downloads being such, I really didn’t want to do a permanent burn because I knew that I might find a better version of the comic later or not want to keep it after a while because I didn’t like the art or the story. So, I decided to give packet writing another go. Packet writing, for those of you who might be wondering, involves formatting your rewritable DVD (or CD) disk in such a manner that you can use the disk as a hard disk – you can drag and drop files onto it from Explorer and you can even delete individual files instead of having to do another session for each burn you do. I had tried packet writing a few years back but hadn’t been happy with it but decided to give it another shot now that I had a DVD burner.

The packet writing actually seemed to work fine (touch wood) but I wasn’t sure if it had actually gone as smoothly as it seemed to :p I needed a way to verify that the files that I’d written to disk had indeed been written correctly. Now this was where my first application came in 🙂 I needed something which would create a checksum for files on my hard disk and then verify the checksum against the same files written to DVD and let me know if the files were copied over fine. You would think that was simple enough, wouldn’t you? Unfortunately, most of the software I found online wouldn’t do exactly what I wanted or wasn’t free or seemed to work but would end up telling me that all the copied files were invalid. I finally found the source for a Delphi app that I was originally going to modify to do what I wanted but once I got it compiled, I realized that it already did almost all the stuff I wanted and did it right 🙂 So I just mucked around a bit with the interface since I wanted it to look a specific way and I was done.

The second application wasn’t that easy. Since I was downloading a lot of comics and most of them were in CBR or CBZ format, I needed something to read the comics with. While CBR and CBZ files are simply RAR or ZIP files and you can always rename them, extract them and then go through the individual JPG files, I just didn’t want to go through all that hassle. There is an existing application called CDisplay which lets you view CBR and CBZ files with ease and it’s free to boot but I don’t like CDisplay since it displays each page in fullscreen mode and I hate having any app in fullscreen mode :p So, I decided to write my own app which would work with CBR and CBZ files or even directories full of images and will work as a window. The app, which I named Tapestry, was easy enough to write and it seems to work fine but I’m not sure that I will release it publicly since it seems to be a bit slow in changing pages and that might be slightly irritating for any other user but me :p

While writing all of the above, I’ve noticed another problem with the new build of Blog – it seems to slow down the machine quite a bit and this is a fairly hefty, workhorse kind of machine too! I guess I’ll have to look into that before the next release as well …

Funky Dung reminded me over on the RookSoft Forums that I had promised to make available a new version of Cee which supports categorization of fortune cookies. Now while categorization does nothin in the Cee app itself, the reason for introducing categories was so that people could display more than one cookie in Blog – yes, there are people out there who like the fortune cookie feature so much that they want to use it more than once on the same blog :p I had suggested that I could add a new attribute to the blog cookie tag so that you could specify which category to use for each cookie instance and this is where the categorization feature in Cee came in. I had forgotten to implement this and since Funky Dung reminded me and since there still seems to be no new release of HTMLEdit available so that I could get the next release of Blog out the door, I decided to work on Cee instead. So, yesterday I started on the categorization feature and it is more or less done. There are a few UI matters to be resolved so that all categories would display on the UI correctly but once that gets done, I guess I can release Cee 2.1 🙂

However, I haven’t integrated the new change into Blog. So that would mean that I’d then have to go back to Blog and implement a category attribute for the cookie tag. Hopefully, I should be able to get that done in a day or less but as always, you can never be sure how things turn out till they are actually done …

If anybody is worried that I might have gone into hibernation again because there have been no posts at all this week, fear not :p The fact of the matter is that some relatives are in town and are staying with us for a couple of weeks. I took this week off from work so that we could take them shopping and to see places and so on and so, I’m only online long enough to make sure that things are running smoothly and to get my e-mail. In fact, I haven’t had any time to even work on any code since Monday – when they arrived 🙂 So, updates will be slow and any new builds would be even slower to arrive but I certainly haven’t gone into hibernation – hopefully, I should be back online on a more regular basis from next week on.

In the meantime, I know that some people are eagerly awaiting the next build of Blog – I could release it even today but you would hate me for it because there is one major bug in the current build – the fact that all HTML code and formatting gets dropped from all new entries in the WYSIWYG view. The author of the HTMLEdit component has promised a fix for this in his next build and I am awaiting that build so that I can release the next build of Blog.

I’ve been working on a new release of my WPBlacklist plugin for WordPress (I really should get a page up for the plugin soon so that I can have a place to point people to :p) and personally, I think the new one kicks butt :p I have a few more features to complete and some testing to do and if I can find some time this week, I’ll probably finish that and try to get a release out by the beginning of next week so that people can better combat all that annoying spam which is floating around.

Speaking of spam, last week and this week seems to have been especially bad since I have a feeling that somebody either released a new tool that allows people to submit spam via HTTP POSTs or a lot of spammers just found out that they could do it since I was getting spam on anything where you could do a POST. I was getting spam on the forums, I was getting spam on WordPress comments and of course, I was receiving spam on e-mail! After a lot of quick fixes and tightening of security measures, the spam flood seems to have been slowed to a trickle. I got an e-mail telling me that somebody who tried to access my old WP comments script (which is now just a spider-trap) got banned by my spider-trap script. So I know they are still trying to flood me but not getting anywhere. I also set up a trap for people who try to sign on to the forums as new members just to display their spam links – that trap has not been triggered yet but then again, I haven’t had any new spam sign-ups either – so I’m happy 🙂

First it was e-mail spam, then it was comment spam and now it’s user account spam? At least, that’s what it seems like to me from what I’ve been observing over at the forums. Recently, I noticed several people signing up with user names that looked slightly strange. At first, I gave it no thought but I’ve noticed more and more people signing up over the last few days but none of them seemed to be posting. Now, as you might know, you don’t need to sign up to read the forums, just to post. So there seemed to be no reason for these people to just sign up if they only wanted to browse the forums to find an answer to a question. So, I went and took a look at the profiles of the latest members of the forums and guess what I found? All of them had one thing in common, URLs that seemed to point to less than reputable sites :p

My first reaction was something like "Oh damn it! Now I have to find a way to stop these spammers as well?" Then I realized that in this case at least, I didn’t have to take so drastic a measure – at least not yet :p All I had to do was simply turn on the "approve" feature which lets the administrator approve a new user who has signed up. At least, I think that should do the trick unless of course, phpBB simply prohibits users waiting approval from posting and still displays them on the user list. I believe that the whole aim here for these spammers (if spammers they be) is to get their URL displayed on the member list. If phpBB simply allows anybody who signs up to be displayed on the member list whether they have been approved or not, then I’ll have to look at a different approach.

For the moment, I’ve gone through the member list on the forums via SQL and deleted all users who had never visited the site since they had signed up and who had no posts at all on the forums. However, this might not be enough if this is going to be the next spam frontier. I guess I’d better start thinking about new spam combatting methods just in case this is the start of a new spamming trend 🙁

I was getting all set to add more features to my WPBlacklist plugin yesterday when I realized that I was going about things all wrong :p I was adding more features to a plugin which I had to keep on maintaining and updating each time a new spam URL came in or the spammer found a new IP to flood me with spam from. Instead, I should have thought about why I was getting spammed so often and come up with a solution which prevented this. So I thought about it a bit …

I have two blogs this one (DC) and the one over at SM. Now DC, is maintained via Blog and uses a commenting system that I wrote while SM uses WordPress as the backend blogging system and uses the built-in comments from WordPress. Now, I’ve probably received like 3-5 spam comments here on DC whereas I’ve been spam bombed on a regular basis over on SM. So, why is it that I get more spam on SM than DC when DC has been existence longer than SM and probably has a higher presence on Google? I realized that the answer might lie in the fact that I used WordPress over on SM.

My logic was along these lines – if you ran a standard WordPress installation, you’d have a specific comments file (the file name itself is well known) and anybody who takes a look at the WP source, can figure out what the comment submission variables are. Then, all a spammer has to do is to write a script which calls the comments form’s action script (the one which processes the submitted comment) with the correct variables populated with the spam values. So, what if I changed the WordPress code so that the comments script was named differently? This would take care of part of the most basic robot scripts which simply hit every site with a WordPress blog and looked for the standard comments processing script but what if the robot actually went through your index page to figure out what the actual comments submission script was and then simply submitted comments to the script name it had discovered? That was easy enough to tackle too – I simply changed the comment variable names on my installation of WordPress.

Of course, the spam robot might be written so that it would parse the main page not only for the comments script name but also the comments variables. So, I added a third level of security by populating my main page with a couple of dummy forms which pointed to (among other things) the original WP comment submission script but I had replaced the code in the original script with innocuous stuff which did nothing at all :p

That probably would have been as far as I would have gone if I hadn’t started reading up about spam comment prevention online :p The above methods would not discourage a determined spammer but they would have been enough to deter most of the script kiddies out there who are probably simply using a script written by somebody else and are basically using their spam robots to target standard WP installations which are easy prey. But what if I was dealing with a more determined attacker who had a robot script which was really thorough? I found an answer to this in my online research in a blog entry about how to set spider traps :p This article basically explained how you could create a script which would identify bad robots which tried to access disallowed content on your site and then ban the IP they came from. I modified that method to mark the dummy spam submission scripts I’d entered into my main page as disallowed content so that any robot script which disregarded my warnings and tried to parse the dummy scripts would automatically get banned. Now, I’m waiting to see what happens – if I still continue to get spam or if the floods stop … or, I might become the next challenge for some idiot out there who becomes determined that he should get me since I went to such lengths to stop the spamming and explained it here so others could do the same :p

I heard back from Fabian, the developer of HTMLEdit, and he was kind enough to promise a fix for the problems that I’ve been facing with the component and since he’s been talking of a new build soon, I have decided to wait for his build before releasing the next beta of Blog 8.0 🙂 In the meantime, I’ve been using the last build I created for several days now and I find it to be fairly stable since I was able to fix one of the last problems which had been nagging me, yesterday. The problem? There would be extreme slow-downs when switching from the WYSIWG view to the HTML view and I would have to wait over 20 seconds before the right-click menu for a misspelt word came up. Both these problems seem to have disappeared since I replaced the spellchecking component used in the application with a fresh copy. Yes, that’s exactly what I said – I didn’t replace the component with a different component, but I replaced the copy of the component in the application with a fresh copy of the exact same component and it seems to work fine now – go figure :p I did read something on the forums about the HTMLEdit component not liking to work with a shared copy of the Addict spellcheck component but so far, after putting in the new copy of the component, I don’t seem to have problems with sharing. Guess I’ll wait and see.

In other news not related to Blog or Delphi coding, I seem to be getting heavily hit by comment spam over on SM. So far, I’ve been talking about comment spam related development over on SM since that’s the WordPress blog and so I thought I might as well talk about WordPress related development over there. But then, that breaks the whole "development here", "meandering thoughts there" rule :p But I’ve done it that way anyway since I like to try and post in both places several times a week and sometimes I’m too busy to post in both places on the same day 🙂 However, today I had something I really wanted to say over there on SM and then I came into work and discovered that I’d been hit by comment spam again and this time my WPBlacklist plugin had actually not caught the spam – darn it! Of course, a blacklist is only as good as the entries in it and in this instance, the spammer was not in my blacklist. So I immediately rectified the situation, deleted all the comment spam and then decided that I was going to spend some time working on adding all the extra bells and whistles that I’d been meaning to add to the WPBlacklist plugin. So, that’s what I’ll be working on for the next few days :p

Funky Dung suggested a few days ago that I should think about providing a syndication feed for DC so that people can figure out that I’d come back from my hibernation (I expect that he expected this hibernation thing to be a regular occurrence … and he might not be wrong :p) instead of having to check back on DC every so often. I told him that I had to first get the RSS template going and put that on backburner. Yesterday, I had enough time to look into implementing an RSS template and since I also needed to figure out if the new build of Blog works nice with RSS templates, I gave it a go 🙂

The result, after some messing around, reading about RSS standards and so on, was a fairly workable RSS template. I still have a few validation problems but most of the major ones are gone. The new format attribute for the date and time tags has helped tremendously and I’m hoping that others will find this little feature useful too 🙂 In fact, in testing out the RSS template, I found one bug in the format attribute, so I guess it’s a good thing that I did work on the RSS stuff :p I still get a validation error on non-UTF-8 characters in my entries but that might be due to the fact that the validator thinks my feed is encoded in UTF-8 for some reason … perhaps because that’s the default? :p As you can see, I still have a lot to learn about RSS and so will have to keep on plugging away at my RSS template till I get it working perfectly. Then, I’ll probably include that in the next build of Blog as a default template 🙂

In Blog related news, the latest builds of Blog seem to be working fairly well with WordPress. Nigel suggested a great feature – that of uploading imges via FTP to a remote blogging server 🙂 I had initially been looking at implementing image uploads via the XMLRPC interface but couldn’t do that for WordPress since their file upload feature wasn’t really complete the last time I looked :p However, when Nigel suggested using FTP, I realized that all I had to do was use what was already available in Blog – I didn’t have to wait on the XMLRPC stuff to be worked on! So I added a new option to remote server settings where you can specify the FTP server to upload images and added a couple of other settings to making things work and voila, it was done 🙂 Nigel has tested it out since then and tells me it all seems to work fine 🙂

I’ve also heard from Fabian the author of the HTMLEdit component that I use. He explained why all formatting (and image info) would be dropped when I saved a new entry – the HTMLEdit component decides whether the data it is working with is plain text or HTML by examining the data when it is first loaded. So, since there is nothing to examine when you start a new entry – it thinks that the data is plain text! While I do understand his explanation (in fact, I had deduced that it must be something like this and was going to do some experimentation as soon as I got the latest stuff in Blog working), I don’t think the behaviour is very intuitive – if it is a new record, the component should actually check the data it is saving rather than what it loads. I’ve asked Fabian whether it would be possible to make this change and am waiting on his response but in the meantime, I’m sure that I can at least do a kludgy workaround that will allow me to release the next build of Blog fairly soon 🙂

So what’s the stat, Matt? I’ve been working over the weekend on the weird XMLRPC problem when trying to send a post from Blog to WordPress. The problem, as I mentioned in the previous entry, had to do with various non-standard characters like the pound sign (£) or the Euro sign (€). The problem seemed to crop up because the XMLRPC code in PHP expects the character data sent from Blog to be UTF-8 encoded whereas I am not exactly certain how the data from Blog is actually encoded – but it certainly is not UTF-8 :p I had been hoping to find a solution which would allow me to do a fix from the Blog end without touching the PHP code in WordPress but unfortunately, after about a day of testing and trying various avenues of attack, I was basically no further ahead than I had been the day before.

So, I finally decided to settle on a solution which would involve changes both at the Blog end and the WordPress end. Basically, I would encode selected characters with their numeric HTML entity equivalent at the Blog end. So, £ would become £ at the Blog end. You might ask why I didn’t use the HTML character entity equivalent of £ for £ instead of £ and it would be a good question since that’s what I too wanted to do originally :p However, at the WordPress end things blow up if I used the character entity since apparently, the £ entity is not defined in the XML spec and so is not recognized by the PHP XML parser 🙁 I looked at a few suggested workarounds for this problem but none of them really worked for me and so, I decided to go with the numeric entity since that at least worked … after a fashion.

But this still wasn’t the whole solution because while the numeric entity was not outright rejected by the XML parser, I would always get a strange character in front of the actual character whenever it was parsed by the PHP XML parser. So, instead of the single character that I originally had, I would get a garbage character followed by the actual character. This turned out to be due to the whole non-UTF-8 encoding problem that I’d already talked about and so, I decided to add some extra code to the XML parsing routines in WordPress to handle this particular scenario. The new code looks for particular instances of two character sets which meet certain criteria and then encodes them into UTF-8 and so far, it has been working fairly well. Of course, as to whether this particular change would affect something else in WordPress is something that I can’t confirm or deny totally at the moment :p However, I’ve managed to transfer around 600 out of the 4,000+ entries that Nigel gave me and progress seems to be much smoother now. Stay tuned for further developments …

OK, I’m much further along the path to the next beta build of Blog though I realize that the next build is going to be very much a hodgepodge of not quite ready features 🙁 There are certain major problems which aren’t always apparent with this new build that I’m working with. For instance, there seems to be a bug with the spellchecking since I could right-click on a misspelt word and it would take almost a minute before a menu would pop up. Since I have no access to the source for the HTMLEdit component, it’s very difficult to debug this kind of problem. It used to be much easier when the author used to respond immediately when I submitted a problem but these days, I either get no response at all from him or something like “I’ll look into this tomorrow” and then silence. I can’t really blame him too much though since I haven’t paid for the component and I after all am using a beta version. So while I do thank him again for his kind sponsorship, I am beginning to worry about where the component itself is headed if he’s either too busy to provide support or is not interested in doing so.

But on to the good news :p I’ve been working with Nigel’s data in trying to get it all over to his WordPress installation since it was Nigel who woke me up and got me working on Blog again 🙂 He’s got over 4,000 entries if I’m not mistaken and so far, I’m still in the first hundred or so :p However, I have fixed quite a few bugs and added a few user-friendly features that I might not even have thought of if I hadn’t been working with this data just because going through the transfers over and over again made me see things that I might not have seen otherwise :p The transfers to WordPress are working fairly well now – in fact, I am using WordPress over at SM and have been posting quite regularly with the new build of Blog with no problems whatsoever. However, there is one aspect of transferring to WordPress which is still giving me problems – special characters like the British pound sign (£ … not to be confused with the hash sign which is also called the pound sign :p) or the Euro (€) sign. These seem to cause the XMLRPC server on the WordPress end to throw a hissy fit :p I have tried encoding the characters via HTML encoding but have met with mixed results – it sometimes works but has an extra character in front of the character I wanted and sometimes even the encoding throws a hissy fit. I have a feeling that this is connected to the fact that WordPress uses UTF-8 encoding for all XMLRPC stuff but haven’t had enough time to delve into things to figure out what exactly is happening and how I could try to fix it.

In fact, my aim today is to rip out some of the XML parsing routines from the WP code into a separate routine, dump the XML sent from Blog into that code and see if something blows up. Having to debug between two different languages is a pain in the posterior and this way, I’ll only have to deal with PHP till I figure out where exactly things are going wrong in the first place – whether it’s the PHP end of things in WP or the Delphi end of things in Blog. Once I figure that one out, I should be fairly on my way to getting all of Nigel’s data transferred over. But as to when the next beta will be released, all I can say is that I hope it’s soon but I’d hate to inflict people with a not-quite-working WYSIWYG view since that’s supposed to be one of the major features of Blog 8.0 :p

And then he was back :p Basically, I haven’t been posting for the last few months because Blog was broken and I was too lazy to go back to an older version. Let me first see if this posts fine and then maybe think about writing a longer entry …

Hallelujah! It works!! At least, the posting via FTP does … but there are other behind the scenes issues which are going to make things a bit difficult for a while. Basically, I got all the work necessary for translating the existing data into the new database formats which support storing journal entries irregardless of where the entries are finally published to at least a couple of months ago. Unfortunately, just after I got all that work done and added the code in for publishing to remote blogging servers, I ran into several problems with the WYSIWYG HTML editing components. The usually reliable author of HTMLEdit would not respond to my bug reports and I was totally baffled by the problem since I didn’t have access to the source.

So, I decided to find a replacement component and looked at several. The authors of WPTools and TRichView were extremely kind in providing me with a free license to each of their components. However, I found that in each instance, I would have to do a lot of recoding and would lose a few existing features that I found extremely useful in Blog using HTMLEdit. So, I have been playing a waiting game, trying different components when I ran across them and then discarding them and at the same time, hoping that the HTMLEdit author would release a new version which fixed the problems that I had. Of course, nothing seemed to happen at all for a couple of months … or a bit more.

Then I was shaken out of my apathy by Nigel yesterday :p Nige has been a long-time Blog user. His site, The Red Ferret Journal, has around three years of archives totalling probably thousands of entries – all in Blog. Nige has been thinking of shifting to WordPress so that he can allow multiple users to post to his site but still be able to use the Blog interface by using the new functionalities that I was supposed to add with the next release of 8.0 :p So Nige has been waiting patiently for quite a while for me to get my posterior in gear and get Blog out but of course, it wasn’t happening. He finally decided to write to me and see if he could get me to expedite the process since he urgently needed to shift to WP and I finally decided to give the code another look and see if perhpas I could decouple the WYSIWYG component from everything else so that I could at least come up with a Blog version which did what Nige wanted for the short-term while I figured out the WYSIWYG problem. However, in trying to sort out the code, I actually figured out how to get things to work – or so I thought at that time :p

Of course, things haven’t turned out to be that easy. I did figure out the problems with the HTMLEdit componet but I discovered in the process of making this entry why I’d had problems in the first place. The original problem which started all this stagnation was related to the HTMLEdit component dropping images when an entry was saved. I realized today that only images but even formatting is dropped by the HTMLEdit component on first save. However, I realized that if I went to the HTML view and added some HTML tags and then switched back to the WYSIWYG view and saved the entry, it saves perfectly fine :p Ah well, the joys of coding …

In the meantime, I’ve also discovered that the code which seemed to work fine with WordPress a couple of months ago is now giving me trouble. But then again, I just came across an article online which suggests that this might actually be XMLRPC and PHP related problems than Blog related problems. So I’m going back to the code to see if there is a way in which I can fix all these things so that I can finally get that long-delayed version of Blog out :p