A couple of days ago I blogged about how Amazon’s and other recommendation engines do a poor job. Recommendation was the cornerstone of Tim Malbon’s argument [link no longer available] that there will soon be so much data about our tastes publicly available that we will only ever recieve customised views of sites, and that this will ultimately result in less choice: stuff that the recommendation engine doesn’t think we’re interested in won’t even appear. In that post, I argued that just because we buy a book, it doesn’t necessarily mean that we want to buy anything else like it.

Today, I want to consider some of the factors in the software industry that will prevent recommendation engines from taking over our lives, even if the basic problems I outlined last time could be solved.

It occured to me recently that Web 2.0 services are the first new utility of the post-privatisation world. Most of the other services we rely on were installed, and are now regulated by, the government (I’m writing this from a UK prespective, so my terminology may not be 100% compatible). Trains, phones, roads, electicity, gas and the rest: they were all previously state monopolies and are now privatised with some form of regulator making sure that there is adequate competition and that consumers don’t get ripped off. Crucially, they also make sure that profits are ploughed back in so that infrastructure improves over time. Well, that’s the theory at least.

In the new web economy though, this isn’t the case: services like Technorati, Digg, Del.icio.us, Flickr and the rest are there with no thanks to governments. The so-called ‘blogosphere’ relies heavily on these and other services, but they are entirely unregulated. The W3C has proved itself to be so sluggish that cutting edge services cannot be expected to wait for them. Rather, someone proposes a new XML format or XML-RPC interface and it either survives or it doesn’t. RSS is a perfect example of this. Nobody is going to argue that RSS is a bullet-proof specification, nor is anyone going to suggest that it’s semantically particularly elegant. From the technology purists’ point of view, ATOM is far superior.

But here’s the thing: RSS is very very simple to create, whereas ATOM insists on using XML namespaces well and on mandatory elements and so forth. When I’m asked to describe what ATOM is, I’m left with ‘well, it’s like RSS only a bit more strict’. People don’t get enough extra for doing things ‘properly’, so they just stick with the technology that is widely adopted and good enough.

Even though RSS is imperfect, people have used it to do loads of things that would previously not have made any sense. I have an RSS feed of Subversion checkins to the projects I’m responsible for so that I can see what other people are doing right in my news aggregator. I probably use Omea Reader more than Outlook these days. There are entire businesses, like Feedburner, that are built on RSS.

For several years, RSS was the province of Blogging and XML geeks, was constantly changing, and competing extensions kept appearing and then being either swallowed or adopted very rapidly. If there is to be a world recommendation system, it will go through a much more painful process because it’s a lot more complicated than RSS. Not the least of the problems is that there isn’t a single way of identifying objects – even Amazon don’t know that two books published in two territories are the same text in different covers. You may love avocados but hate avocado bathroom suites. Good luck with the recommendation engines.

Now, imagine a world like Tim does. This is a world where everyone has agreed to work together, but that agreement has led to an undesirable restriction of choice and has been hijacked by Google, Yahoo, Microsoft and whoever else has bet on the right technology. But wait. Are Google, Yahoo and Microsoft ever going to cooperate on something like this? Can you ever see Microsoft ditching Passport Windows Live ID in favour of integrating with Yahoo’s ID system, or Google’s? No chance.

Will Hotmail Windows Live Mail, Yahoo Mail and Gmail ever be best buddies? I don’t think so.

Look at the reaction to Microsoft’s infamous Hailstorm product. I think there genuinely were people at Microsoft who thought people would enthusiastically embrace the idea of giving their entire online profile to the biggest software company in the world. When ‘don’t be evil’ Google announced something similar, even they were roundly slated.

The bottom line is that it is not in software companies’ interests to collaborate at the cutting edge. It takes years and years for them to work together on a technology (think Microsoft and Sun on Java, Microsoft and IBM on Web Services), by which time the cutting edge is elsewhere. It’s just that the battle front has moved on.

There’s simply no motivation for Google, Yahoo and Microsoft to create some über recommendation engine together. Still less for Amazon: it’s their most valuable data after all. As soon as the market for recommendation engines has settled down, the battle will move elsewhere and there will be incompatible fronobricator engines or whatever.

As far as I can see, there will always be competing pseudo-standards at the cutting edge. Collaboration on the scale Tim imagines is about as likely as the United Nations becoming the world government.

Norwich City fan, book afficionado, and all-round nice guy Shane Richmond has been blogging about the future of newspapers and how they might adapt to the ‘next web’. Tim Malbon responded in his blog [link no longer available] with an even more extreme vision of a world in which there’s so much data available about our preferences that we will end up with no real choices of our own, only those deemed to be of interest to us by Google or whoever. I think this latter vision is completely wrong.

Shane’s posts are well argued, as always, and are informed by his experience as the News Editor of the Telegraph.co.uk. Tim’s is compelling, but, I think, fundamentally flawed. Here’s the first part of why I think that: things are generally quite crap.

There are two oft-quoted exemplars of relevance on the Internet today: Google AdSense and Amazon. I have to say that I remain unimpressed with both of these, even though they represent the best there is at the moment. (I should say that my musical taste precludes me from using either last.fm or Pandora, so I can’t judge these.)

Let’s take Amazon. I buy broadly three kinds of books. First, books that I want to read myself for pleasure. These are generally so-called ‘literary’ novels. Then, I buy computer books for work. Finally, I buy presents for other people, things I would never buy for myself. Amazon doesn’t know why I’m buying these things, and unless I tell it, which I can’t really be bothered to do, it doesn’t know what I thought of them. Maybe there are people whoi obsessively keep their Amazon perferences up to date, but I have other things to do that are more exciting.

I recently bought and read all of the Booker Prize shortlisted books, most of them from Amazon. Once I’d bought them my recommendations were updated – to include the Booker shortlisted books that I hadn’t bought, plus one book from each author if they had a back catalogue (there was one first novel on the list this year). Clever, you’re thinking? Not so clever, if you ask me. This is the most basic possible recommendation. It’s a bit like buying the number one single and then being asked ‘have you thought about buying the rest of the top ten?’

In fact, the only time Amazon recommendations are useful to me is when I buy books so obscure that only a few other people have ever bought them there. Then, they very often uncover some completely unexpected ‘related’ books, related for no other reason that some other Amazon user has an eclectic taste that is similar to mine. This is the ‘long tail’, which is why Amazon is succesful, or at least why it is useful to me (other than the obvious pricing reasons).

The Guardian Review on saturday and the Observer Review on sunday are miles better at identifying books I may be interested in than Amazon are. Why? Because there’s an editorial policy (in the case of the Observer, a former editor at Faber, Robert McCrum) in place and it is broadly in line with my tastes. At Amazon there’s no editorial policy, just a load of people buying stuff and then some assumptions about their relatedness. Plus, Amazon are trying to sell me stuff, so I can’t trust their editorial judgement anyway. Newspapers, at the moment, can be trusted to slate stuff they think is rubbish.

One final Amazon example. I once bought Abbado’s most recent recording of Mahler’s 7th Symphony and Don Box’s Essential .NET, Volume 1. I can’t think of a single link between those two items, other than that I happened to buy them at the same time. But guess what showed up as a related item to the Mahler Symphony? Even someone who had little or no knowledge of classical music or computing can see that there’s no relationship between these things.

All these things are instances of the crapness I’m talking about.

I can’t see how this problem can ever be solved, unless computers have a way of knowing why I buy things and not just what I buy. Shane’s argument (as does Tim’s perhaps to an even greater extent) rests on someone solving this fundamental problem.

Next time I’ll look at human factors in the software business that may retard our march towards an automated society.