News publishers would like to eliminate what they see as “free riding” that the major Internet search engines do on their content.  The search engines index the content, make it available in search results, and monetize the traffic through ads and various other ways.  Users go to search engines for their news and don’t visit the news publishers’ sites (or those of the publishers’ syndication partners), thereby depriving publishers of traffic and revenue.

Back in 2007, many news publishers — particularly in Europe — got together to develop a standard called ACAP (Automated Content Access Protocol) that was supposed to solve this problem.  It would enable news publishers to specify what rights search engines should have to index their content and display it in search results.  The ACAP members tried to get the major search engines — such as Google, Yahoo, and MSN — to implement ACAP, on the grounds that doing so would encourage more news publishers to make their content available to search engines.

Fast forward to November 2009 and the depressing revelation that Microsoft is in negotiations with Rupert Murdoch’s News Corp. to pay News Corp. to block Google from indexing its web content.  This is perfectly feasible through the Robots Exclusion Protocol (REP) technology that ACAP purports to replace.

The problem with ACAP is, and has been, that while it benefits publishers, there is little in it for the search engines.  If everyone were to implement ACAP, the search engines would get additional content to index that amounts to a minute, minuscule increment to the oceans of content that they already index — and a somewhat less minute incremental amount of monetizable traffic, on the theory that name-brand news content is more popular than average.

In other words, the economic benefits of ACAP are not equitable; the standard is not a win-win for all participants.  So it’s little wonder that the elegantly-designed ACAP has been languishing; the list of ACAP participants is even no longer available on the ACAP website.

News Corp. was an early participant in ACAP through its subsidiary News Ltd. Australia.  But now it is circumventing a fair resolution of the free-riding question by taking advantage of Microsoft’s hunger to promote its new Bing search engine against Google.

News Corp. may rightly claim that it does benefit from having Google index its content for discovery through search, and therefore that it needs to be compensated for the loss of traffic it would incur from being excluded from the world’s most popular search engine.  But that, as we say in the technology biz, is not a scalable solution.

A proliferation of such deals will only lead to a world of confusion for users that gets even more confusing as the economics shift over time.  Throwing money at the free-riding problem brings it no closer to a solution.