From the archive: The software industry is quite crap too
[The second in my series of posts that have disappeared with Interesource, but which I would like to keep available. This one was a follow up to my post Don't ignore the crapness factor (which I've also republished here). It was originally published on 16th October 2006 and reflects the fact that at that time I was still stuck in Windows at work.]
A couple of days ago I blogged about how Amazon’s and other recommendation engines do a poor job. Recommendation was the cornerstone of Tim Malbon’s argument [link no longer available] that there will soon be so much data about our tastes publicly available that we will only ever recieve customised views of sites, and that this will ultimately result in less choice: stuff that the recommendation engine doesn’t think we’re interested in won’t even appear. In that post, I argued that just because we buy a book, it doesn’t necessarily mean that we want to buy anything else like it.
Today, I want to consider some of the factors in the software industry that will prevent recommendation engines from taking over our lives, even if the basic problems I outlined last time could be solved.
It occured to me recently that Web 2.0 services are the first new utility of the post-privatisation world. Most of the other services we rely on were installed, and are now regulated by, the government (I’m writing this from a UK prespective, so my terminology may not be 100% compatible). Trains, phones, roads, electicity, gas and the rest: they were all previously state monopolies and are now privatised with some form of regulator making sure that there is adequate competition and that consumers don’t get ripped off. Crucially, they also make sure that profits are ploughed back in so that infrastructure improves over time. Well, that’s the theory at least.
In the new web economy though, this isn’t the case: services like Technorati, Digg, Del.icio.us, Flickr and the rest are there with no thanks to governments. The so-called ‘blogosphere’ relies heavily on these and other services, but they are entirely unregulated. The W3C has proved itself to be so sluggish that cutting edge services cannot be expected to wait for them. Rather, someone proposes a new XML format or XML-RPC interface and it either survives or it doesn’t. RSS is a perfect example of this. Nobody is going to argue that RSS is a bullet-proof specification, nor is anyone going to suggest that it’s semantically particularly elegant. From the technology purists’ point of view, ATOM is far superior.
But here’s the thing: RSS is very very simple to create, whereas ATOM insists on using XML namespaces well and on mandatory elements and so forth. When I’m asked to describe what ATOM is, I’m left with ‘well, it’s like RSS only a bit more strict’. People don’t get enough extra for doing things ‘properly’, so they just stick with the technology that is widely adopted and good enough.
Even though RSS is imperfect, people have used it to do loads of things that would previously not have made any sense. I have an RSS feed of Subversion checkins to the projects I’m responsible for so that I can see what other people are doing right in my news aggregator. I probably use Omea Reader more than Outlook these days. There are entire businesses, like Feedburner, that are built on RSS.
For several years, RSS was the province of Blogging and XML geeks, was constantly changing, and competing extensions kept appearing and then being either swallowed or adopted very rapidly. If there is to be a world recommendation system, it will go through a much more painful process because it’s a lot more complicated than RSS. Not the least of the problems is that there isn’t a single way of identifying objects - even Amazon don’t know that two books published in two territories are the same text in different covers. You may love avocados but hate avocado bathroom suites. Good luck with the recommendation engines.
Now, imagine a world like Tim does. This is a world where everyone has agreed to work together, but that agreement has led to an undesirable restriction of choice and has been hijacked by Google, Yahoo, Microsoft and whoever else has bet on the right technology. But wait. Are Google, Yahoo and Microsoft ever going to cooperate on something like this? Can you ever see Microsoft ditching Passport Windows Live ID in favour of integrating with Yahoo’s ID system, or Google’s? No chance.
Will Hotmail Windows Live Mail, Yahoo Mail and Gmail ever be best buddies? I don’t think so.
Look at the reaction to Microsoft’s infamous Hailstorm product. I think there genuinely were people at Microsoft who thought people would enthusiastically embrace the idea of giving their entire online profile to the biggest software company in the world. When ‘don’t be evil’ Google announced something similar, even they were roundly slated.
The bottom line is that it is not in software companies’ interests to collaborate at the cutting edge. It takes years and years for them to work together on a technology (think Microsoft and Sun on Java, Microsoft and IBM on Web Services), by which time the cutting edge is elsewhere. It’s just that the battle front has moved on.
There’s simply no motivation for Google, Yahoo and Microsoft to create some über recommendation engine together. Still less for Amazon: it’s their most valuable data after all. As soon as the market for recommendation engines has settled down, the battle will move elsewhere and there will be incompatible fronobricator engines or whatever.
As far as I can see, there will always be competing pseudo-standards at the cutting edge. Collaboration on the scale Tim imagines is about as likely as the United Nations becoming the world government.
