[LINK] deep web

[Stilgherrian]

On 24/02/2009, at 9:15 AM, Eric Scheid wrote:
> would be handy if there was some established place they could say to  
> robots:
> "don't go here, it would be a waste of your time and would consume  
> resources
> at my end for no gain".


Like, um, robots.txt? Or does that some subtlety?

Stil



-- 
Stilgherrian http://stilgherrian.com/
Internet, IT and Media Consulting, Sydney, Australia
mobile +61 407 623 600
fax +61 2 9516 5630
Twitter: stilgherrian
Skype: stilgherrian
ABN 25 231 641 421