May 15, 2005

Shutting Out Google

with "robots.txt" was easy. All I needed to do was add

"User-agent: Googlebot
Disallow: / # I want to keep Google completely out of my site.

User-agent: Googlebot-Image
Disallow: / # I want to keep Google completely out of my site."

to a file named "robots.txt" in my web home directory. This was easily done in a couple of seconds.


Update August 8, 2005: For reasons described in this post, as of today, I have stopped blocking Google's robots over robots.txt. The reason for that is not that I want them to index my files, but that I don't want to contribute to their point of view that they are entitled to violate copyright as a default as long as people can easily shut them out with robots.txt. In my view respecting copyright should be the default, and the Google robot should be shut out of all sites that don't specify in robots.txt that they want to be indexed.

Let those who want Google in say so in their robots.txt files.

Posted by Karl-Friedrich Lenz at May 15, 2005 11:31 AM