Twitter, scaling and solving the wrong problem

Apologies for typographical errors, spelling mistakes and other mistakes. I’m having one of those ‘I should be sleeping by now, but I am feeling inspired’ days.

I’ve been reading a lot of buzz lately about Twitter. Downtime fury, vast armwaving guesses at potential problems, lots of stuffy opinionated blathering by people such as Michael Arrington, the vacillating cockmunch who runs TechCrunch.

There have even been some attempts to supposedly solve the problem that people think Twitter has. Namely, that it doesn’t scale very well and that it can’t handle its user load anymore. However, I am of the opinion that the problem needs to be tackled from a completely different perspective.

It’s not so much that Twitter can’t scale. They’ve got millions and developers. A few smart people could solve the problem, no doubt. It’s that.. well, should it? Twitter is, essentially, a pseudo-realtime multiple-destination complex communications service. Handling all that will be a pain, even with so-called fixes like sharding.

But, I prefer to think of it from another perspective, namely the style after which the service, Twitter, was built. That being the Short Message Service (or SMS), a method of sending short text messages between cellphones.

The telecommunications industry seems able to handle massive flows of traffic, between a lot of different people, with relative ease. The reason for this, or at least, what I believe is the reason for this, is distribution on a scale that a single service would be hard-pressed to match.

So instead of thinking of Twitter as a big, huge service, like something akin to GMail, I thought of it as being exactly the same as SMS, except in a different medium. Now, SMS isn’t nearly as complex as what Twitter does, but that’s okay, it doesn’t need to be to enable you to solve the problem.

Ask people what the problem is with Twitter and they’ll say, “Rails sucks!”, “It wasn’t designed for this.” among other things. A quick googling would no doubt clue you in to the buzz. But, perhaps it doesn’t HAVE to be this big, efficient, scalable beast.

Perhaps, all that is needed it to take a leaf from the arguably biggest innovation this decade. Peer-to-Peer. A lot of twittering seems to be not just people posting, but replying, conversing (and presumably, directly messaging). If this is the case, then why not seperate things out on a user level.

A surefire way to ‘best’ Twitter, would be to make it so that people aren’t restricted to using Twitter’s service (or even a single provider, style, methodoly, application or language). Why not develop a protocol for use in a simple text transaction service (And that is orders of magnitude more simple than XMPP).

Wouldn’t it be nicer if you could run your own twitter-esque app, at twatter.myblog.com? That not only let you have all the expected features (Though, doing SMS would be.. interesting, but perhaps provides a way to make income off of this style of implementaiton?), being able to @reply, d irect message and use applications of choice for twitterizerating?

I’m going to try and throw up an implementation written in PHP tomorrow, along with a draft specification on message formatting and authentications/whitelisting.

Lets see if we can’t do this in proper 21st century style, eh?

Update: If you’re interested, here’s my twitter