stream by aspheric.lens, on Flickr

The word “lifestreaming” feels overly broad and I do a mental eye roll every time I say it, but I’ll use it for now for the sake of expediency.

So I’ve been using Flickr and Delicious forever and Twitter for a while and Google Reader and all of these services are great and let me share things in different ways. But what’s always bothered me is that these Shared Things were not only siloed off from each other, they were on sites I didn’t control. Yes, they all make it very easy to republish your stuff. I was doing that for a while with sidebar widgets for each of those services (I still have them on my Asides page, actually). But it still all felt very unconnected.

What I really wanted was to hook into the sharing functionality on those sites so that when I pushed the Share button in Google Reader, say, it would also send the item to my personal blog. That would allow me to aggregate my photos and bookmarks and tweets in one place and keep a permanent record of all of those things that is in my control.

I looked into several services (thinking I might let someone do the aggregating for me and then download the result), standalone software and plugins for WordPress. One of the better and easy to use solutions in the WordPress plugin category is Lifestream by David Cramer. It does almost everything I need except that I didn’t like the way the resulting lifestream ended up being something separate from the actual blog. I wanted my blog itself to become the lifestream.

In the standalone software category, PubWich looks very promising. But, again, I wanted something with seamless integration with WordPress.

So I ended up writing a plugin over the weekend. What it does:

• Periodically scans and downloads feeds from sources you define
• Makes a new blog post for every new item in each feed
• Blog posts can be created as new WordPress 3.0 Post Types or posted to specific categories
• Defines filters so that you can hook into each individual feed as it is being imported to tweak the content
• Saves a history of every feed

Then based on that, I created a WordPress theme that styles posts depending on the source. That’s how I display the post icons and the “View original” link with the domain of the source website, for example. Because every item becomes a blog post they inherit everything WordPress provides: they go into the feed, comments, permalinks, etc. That’s cool! (In a very nerdy way.)

So far, after one day of use, I’m very happy with it. I intend to release the plugin after I do some more work on it. Let me know if you’re interested in helping me test it.

Received my copy today of the Pro Web 2.0 Mashups book I did the technical review for. Always neat to see something you’ve worked on become real. Also, I hate writing bios like this.

Anyway, it’s an interesting book about remixing and mashing up data from multiple online services via APIs. But one of the coolest things about the book—are you ready for this?—the author, Raymond Yee, and publisher have agreed to release it under a Creative Commons Non-commercial Share-Alike license! In addition, the draft of every chapter is available online right now and the published version will be available shortly. It’s practically an open-source book.

I’ve already been paid for my work on the book but I still feel too close to it to be able to give an unbiased review. But who needs a review? Go check out the book yourself and if you like it please show your support for openness and buy a copy when it hits Amazon.

A little background…

BigHugeLabs.com allows visitors to use dozens of “toys” to make fun projects out of their digital photos. On any given day it is visited by around 40,000 people and approximately 40% of those are first-time visitors.

Why switch?

I really like Prototype and Script.aculo.us but my main problem with both of them is their bulk. They just do way more than I need. The entire P+S stack (I was using version 1.6.5) comes in at over 250k. It’s monstrous and those 16,400 new users every day had to download the whole thing. I don’t even want to think about what the site was like for dial-up or mobile users.

I also have a strong dislike for the Script.aculo.us documentation wiki. “Clear” and “organized” are two adjectives I can’t imagine anyone using to describe it.

But a bigger problem with Script.aculo.us than the documentation is that the visual effects sometimes stutter and skip. Even the Script.aculo.us home page staggers and stutters into view as the page loads. In one case recently I was trying to simply slide a small DIV about 100 pixels down a full page. No matter what I tried, I just couldn’t get the effect to run smoothly. With MooTools, it just worked. Granted, it could have been pilot error, but the bottom line is that MooTools gave me exactly the result I wanted.

A final swaying factor were some of the built-in effects. In particular, I think the Bounce easing transition, Accordion, and Fx.Elements are awesome.

Implementation

Switching out the libraries was straightforward. MooTools uses similar syntax and has a 1:1 replacement for everything I was already doing with P+S (except one). Swapping libraries was a simple matter of searching for P+S code and swapping it with equivalent MooTools code. The one exception is that MooTools doesn’t implement Element.hide() or .show() out of the box. Fortunately, it’s a simple workaround (by extending the Element class, for example).

The entire conversion took about 12 hours spread over four days. That includes some extra time I took to rewrite the few XML proxies I had as JSON and refactoring where it presented itself. I thought about abstracting all of the MooTools Javascript into my own classes so that if I ever decided to switch frameworks again all of the code that would need to change would be in one location. But I was already finished with the entire conversion before that thought got any traction. Anyway, it would have probably tripled the amount of work for a dubious payoff (I may never switch from MooTools).

Final thoughts…

All combined, the switch has allowed me to shave 190k off of BigHugeLabs.com’s Javascript, shrinking it down to a much more reasonable 56k. That’s a savings of almost 3 gigabytes of transfer per day. And the experience of first-time visitors should be improved as the site loads more quickly for them.

MooTools’ superior documentation and demos made it a breeze to pick up. It’s not perfect, but most everything works as advertised.

From a construction perspective, working with it just feels better to me than P+S. It’s a subtle point but MooTools just seems to be a better fit for my style.

I had the task recently of taking over a project from a group of developers (who shall remain nameless) who were unable to finish it. When I saw what they had written, I could see why. The project had clearly been assigned to inexperienced, junior-level developers who did not have the guidance of an experienced mentor. For the benefit of other rookies, I offer this advice, based on problems I found in that project:

• Your professional reputation among your peers will be based on the quality of the code you write.
• Don’t litter code with comments telling me what you changed. That’s what diff is for.
• Don’t comment out sections of code and tell me you deleted them. If you want to delete something, please delete it.
• Poor spelling and grammar are signs of a programmer who doesn’t care about details. Care.
• Never make a change to something just because it looks wrong—especially to mature code that has been in production for years. It looks wrong for a very good reason and you will break something if you change it.
• I really don’t care how you indent or use braces but please be consistent.
• Don’t leave comments explaining the obvious. I can read code.
• Please learn how to use an image editor. You don’t need to take a Photoshop class. Just learn how to slice, choose the right image format and compression, and how to get anti-aliasing right for graphics that will appear on a given background.
• If you are duplicating a snippet of code throughout your project, you are doing it wrong.
• Doing something the “easy” way today will cost you 10x what it would have cost to do it right in the first place. And I’m being optimistic. Be lazy for the long-term.
• Learn CSS and the correct way to name your classes and IDs. And I don’t mean learn just enough to get by. It’s as important as any other code in your project whether it’s Ruby or PHP or C.
• Learn SQL. Inefficient SQL queries will kill your project.
• More hardware may not be the best solution. Your app is slow because of inefficient code.
• Cross-browser compatibility is no longer a holy grail. It is a minimum requirement and it is not difficult.

Sincerely,

John

As if grungy graphics and loud music was going to turn 4,000 nerds into rock stars.

Lots of people had visited the smallish vendor area and got some kind of neon blue flashy light thing. Some of them were wearing it around their necks and some of them just had it in their transparent conference bags. It’s good marketing. It instantly got attention from everyone who saw it. But it’s kind of sad, isn’t it? That all these allegedly smart people can be lured to a vendor booth by a couple of blinky lights like some kind of prize bass. The sales droids at the booth should have been sporting fishing poles and hip waders and kept the flashy things in a tackle box.

Subject: New(?) anti-spam technique

I’ve seen an increase in comment spam the last couple of days. This particular species looks a lot like legitimate comments in that it isn’t riddled with links, the comment text is not advertising, and it isn’t anonymous. Of course, the email address is fake and the spam comes from dozens, probably hundreds, of different IP addresses. Flood control was the only thing keeping it in check. The blacklist worked for a little while but then the spam would change and I’d have to update the blacklist again. So.

The first thing I did was check my logs and noticed that the bot was posting directly to wp-comments-post.php with no referrer. So, this took care of the problem immediately (in wp-comments-post.php):

// Anti-spam
// Prevent posting directly to this page without having come from another page
// on this site.
$siteurl = strtolower(get_settings('siteurl'));
$referer = strtolower($_SERVER['HTTP_REFERER']);
if ( strpos($referer, $siteurl) === false ) {
  // I chose this action because I don't want to give a spammer any clues.
  die();
}

This works for now but since HTTP_REFERER is ridiculously easy to spoof it probably won’t work for long. I’m a little surprised that spam bots aren’t sophisticated enough to do this already.

So, that took care of the immediate problem but I wanted a more long term solution. Captchas came to mind immediately but I dislike captchas because they require legitimate commenters to jump through hoops. I remembered the discussion Kitty started last month about putting a unique hash in the page that must be posted back when the form is submitted. This is a good idea, and I implemented it, but it is also easy for a bot programmer to circumvent since all of the information you need to circumvent it is right there in the form. I made it a little more complicated by requiring the client to compute an md5 hash of the comment + post title and post that back as well (Javascript can do this nicely). But, while that makes a spammer jump through more hoops, once they’ve done that we’re back where we started again.

Unsatisfied, I tried “hashcash”. Hashcash is a clever technique that requires the client to expend a certain amount of work computing a “stamp”. It requires 2-3 seconds to compute a hashcash stamp, it can’t be faked, and it takes a trivial amount of time to verify. This delay is no problem for a reader submitting a single comment but is a huge drain on a spammer who wants to post hundreds of comments a second. And if you use date, post_id, server address, and remote address as the stamp seed then the spammer has to compute a new stamp for every post on every server every day. I implemented this as well, in Javascript, but unfortunately Javascript is too slow to compute stamps in a reasonable amount of time. I found that a 16-bit collision takes much too long to compute in Javascript but it only takes a fraction of a second in C which is probably more like what spammers would use (even 12-bit collisions are barely possible with Javascript). I chose Javascript because I wanted the requirements burden on the client to be as low as possible (Java, Flash, ActiveX are all unnacceptable requirements for posting comments). Here’s something like the algorithm I implemented:

The form onsubmit action would call something like this:

function hashcash() {
  stampseed1 = 'some combination of date + post id + server address + remote address';
  stampseed2 = 0;
  while(true) {
    hash = hex_md5(stampseed1 + stampseed2);
    if (hash.substr(0,4) == "0000") { // Look for a 16-bit collision
      document.getElementById('commentform').stamp.value = stampseed2;
      break;
    }
    stampseed2++;
  }
}


And then wp-comments-post.php would just need to substr(md5(some combination of date + post id + server address + remote address + $_POST['stamp']), 0, 4) == “0000″ to verify the stamp.

I still like this idea the best, even though I abandonded it. If Javascript could compute these faster then it would be a viable solution.

Then it occurred to me that one way to tell humans from bots is that humans are much slower. This is what I came up with:

• When a client hits the comment form, start a timer. Start a unique timer for each client (by IP address or some other method).
• When the client posts the form, check the timer. If the time elapsed is less than 3 seconds or more than 5 minutes (the “window”), moderate the comment.
• Delete the timer.

Here’s how I implemented it:

[1] At the top of wp-comments.php:
  $_SESSION['CommentTimer_'.$_SERVER['REMOTE_ADDR']] = time();

[2] In wp-comments-post.php after check_comment():
  if (!isset($_SESSION['CommentTimer_'.$_SERVER['REMOTE_ADDR']])
  || time()-$_SESSION['CommentTimer_'.$_SERVER['REMOTE_ADDR']] < 3
  || time()-$_SESSION['CommentTimer_'.$_SERVER['REMOTE_ADDR']] > 300) {
    $approved = 0;
    $comment = "** COMMENT TIMER SPAM **\n".$comment;
  }
  unset($_SESSION['CommentTimer_'.$_SERVER['REMOTE_ADDR']]);

[3] In wp-config.php:
  session_start();


This is simple, completely server-side, and transparent to readers. It has zero client-side requirements other than you take a couple of seconds to enter your comment. It busts spambots that spoof IPs on every request, that don’t use the form, and that don’t wait. I used $_SESSION for this quick implementation but there is no reason why you couldn’t store the timers in a database table. You’d just need to clean out old timers occassionally (perhaps automatically each time the admin went to comment moderation). Also, these could probably be wrapped in existing function calls so templates wouldn’t have to be modified (for example, embedding [2] in check_comment()).

So far it has stopped my spam completely. The main drawback that comes to mind is that it requires the client to have a stable IP address during a session, so it might be a problem for people with certain ISPs (AOL?). This can be gotten around by using cookie based identification but then it doesn’t work for people who won’t accept cookies. This method is partially defeatable too. Defeating this method requires a bot to:

Hit/Wait/Post cycle:

• Hit hundreds of comment forms on different sites.
• Hit them all again within the window to post the comment.
• Repeat.

I say partially because, while the above spamming technique will post the spam, it should drastically slow down the spammer and thus reduce the number of sites he can cover thus reducing the spread of spam. If this is widely implemented, you will be less likely to even encounter spam because fewer spammers will even be able to get to your site in their lists.

Assume:

• The spambot has 100,000 wp-comment-post.php entries in its index
• Each of those blogs implements a 10-second flood control
• The spammer can currently post to them all in 10 seconds

A naive spam bot would Hit/Wait/Post each post reducing its output from 100,000 posts/10 seconds to 1 post/3 seconds. Even if he has a network of zombie machines doing his bidding, each zombie doing naive Hit/Wait/Post would be reduced to 1 post/3 seconds. 100 zombies in aggregate would be able to do 100 posts/3 seconds.

A more sophisticated circumvention would be to hit them all once (10 seconds), then hit them all again (10 seconds), reducing the output to 100,000 spams every 20 seconds. But using this technique, if a spambot has millions of posts in his index, then by the time it got around to do the second hit it might have fallen outside of the window.

Another technique would be to start multiple threads on the same spamming machine that do Hit/Wait/Post. A machine that could start 100,000 threads in 10 seconds would be reduced to 100,000 posts/13 seconds. But I don’t think 100,000 threads on a single machine in 10 seconds is feasible (is it?). If he could start 100 threads then he would be reduced to 100 posts/3 seconds. If a zombie network using this technique split up the 100,000 posts among 100 machines and each machine could do 100,000 posts/10 seconds then they could do all 100,000 posts in just over 3 seconds (vs. 0.1 seconds without the comment timer).

Someone check my math? :-)

But is this the general case? This spammer appeared to be coming from hundreds of different IP addresses, but I wonder if he really had access to a zombie network or if he was just using IP spoofing from a single machine.

Update 12/30/2005: A new version has just been released that is compatible with WordPress 2.0. The older version is still available and works with versions prior to 2.0.