[LINK] deep web

[Eric Scheid]

On 24/2/09 8:59 AM, "Michael Still" <mikal at stillhq.com> wrote:

> Imagine you're a first year computer science student... Surely you can
> think of a way of avoiding infinite loops?

Infinite loops? Sure. That's hypertext defined.

Infinite branching & linking? Googlebot has followed a trillion links so
far, and doesn't look to be stopping any time soon.

Imagine a company contacts directory, organised by name, by initial, by
department name, by department code. How many links to departments would you
follow before giving up? What if you were a spambot crawling for email
addresses, blithely ignoring robots.txt, and every page has oodles of email
addresses. Each page is different and juicy. So what if the directory
web-server appears to be a little slow.

http://en.wikipedia.org/wiki/Teergrube

e.