[LINK] deep web

[Stilgherrian]

On 24/02/2009, at 8:00 AM, Jan Whitaker wrote:
> This topic looks ripe for debate.

Not really.


> If a person or company has intentionally incorporated a database
> strategy to keep information from being accessible by search engines,
> are they now going to have to reconfigure their systems to avoid that?
>
> If someone has a 'locked' file cupboard in their public area, is it
> acceptable for someone with a lock pick to open it?

If they've "intentionally incorporated a database strategy" to "keep  
information from being accessible by search engines" then they're an  
idiot.

The analogy to a "locked file cupboard" is incorrect. The cupboard is  
not locked. The owner is just hoping that some users (the search  
engines) don't know how to grasp and turn the handle and can't ever  
learn that skill.

To stop information being indexed, and accessible only to human users,  
the they need an access control mechanism, like asking human users to  
register. They *could* use a robots.txt file, but that's only a polite  
request ("Please do not look at this information") rather than a lock.

Mind you, a human could register, then hand their login details to the  
robot. "On the Internet, no-one knows you're a human."

(There's also another question. If you put information on the public  
web, why *wouldn't* you want it indexed so people can find it? Either  
you want it public or you don't. Don't you?)

Stil



-- 
: Stilgherrian <stil at prussia.net>
: MACS (Prov)
: Prussia.Net: Managing Information for Small Business
: http://prussia.net
: personal website: http://stilgherrian.com
: voice: +61 2 9557 5545 fax: +61 2 9516 5630
: Twitter: stilgherrian
: Skype: stilgherrian
: PO Box 103, Enmore NSW 2042, Australia
: ABN 25 231 641 421