might actually make sense from a business point of view even if it was not necessary for copyright reasons.
As Martin Varsavski points out here, Technorati is run under opt-in. You need to ping them to include your latest blog post in their database.
This seems to be much more effective than just running around the whole web every two weeks and find 99 percent of pages unchanged.
Of course, opt-out is illegal under current copyright. The search engines have no right to copy the whole web without a license. Most current search engines are based on just ignoring these clear requirements.
That means that opt-in is the only option.
However, it is interesting to note that Technorati is running a successful search engine based on opt-in and actually gaining efficiency while respecting copyright. It also means that those who think that current copyright somehow gives people the right to include pages in search databases and offer cache pages to everybody without a license can't say that requiring opt-in would make Internet search impossible altogether (though they probably will say so anyway).
Posted by Karl-Friedrich Lenz at December 31, 2005 10:58 PM