Not Too Clear on the Concept

[jeff_duntemann]

I'm away from broadband most of the day here in Chicago-away from my laptop, for that matter-and thus I don't have time to write elaborate entries. But last night before hitting the sack I was reading up on the Free Companies, which were the wandering groups of mercenaries of the 14th century

Comments

[etfb]

Google the robots.txt protocol. It's possible to say "hide this from ordinary viewers but show it to search engines", because every browser and spider sends an identification string. To save yourself some clicks, you can even tell Firefox to identify itself as the Google spider, then you can see the world with the omniscient eyes of Googled.

Interestingly, you can also use the same protocol to tell Google not to cache your pages. The fact that they knew one trick and not the other indicates a remarkable lack of clue...

[beamjockey]

My essay on Dead Sea Googling looks at this from a slightly different angle.

Just last month I managed to obtain a significant bit of historical research by Dead Sea Googling.

What's more, I was pulling text out of a published U.S. Congressional hearing from 1973. It makes no sense to me that Google Books should treat this as a proprietary copyrighted document, but they do. I asked them to change, and they won't.