[LINK] deep web

[Birch, Jim]

> (There's also another question. If you put information on the public  
web, why *wouldn't* you want it indexed so people can find it? Either  
you want it public or you don't. Don't you?)

Interesting question. One of the objectives of trading in a "market" is
to avoid being part of the market wrt your existing customers (ie keep
them) but to be a highly visible part of the market for new customers
and other traders' customers.  Businesses spend a lot of time and
resources on this.  For a web site, relying on advertising or sales, you
want to drag punters from anywhere but stop them from drifting away.  If
you have a product database that Google summarises as the
meta-information "Cheaper Elsewhere" you may have a serious problem.

I can see a kind of arms race developing between database owners and
indexers.  Enforcing logins is a partial solution, but it turns away
casual visitors, eg, me often.  More likely, database owners will try
detecting trawling behaviours and limit their access to tasty titbits.
But for the likes of Google, the cloud can make asynchronous requests
from many locations to build it's image of a database so a detection and
countermeasure race seems possible.

On the other hand, "any publicity is good publicity" might win out.  For
example, Qantas doesn't block iwantthatflight.com.au, even though it's
virtually always offering the most expensive flight.

Jim


CONFIDENTIALITY NOTICE AND DISCLAIMER

The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission. If the transmission contains advice, the advice is based on instructions in relation to, and is provided to the addressee in connection with, the matter mentioned above. Responsibility is not accepted for reliance upon it by any other person or for any other purpose.