[LINK] Latency in indexing robots

[Craig Sanders]

On Wed, Dec 06, 2006 at 01:49:38PM +1100, Antony Barry wrote:
> to each page. I've now got to the stage where I _want_ my new pages  
> to be indexed but not the pmwiki documentation which comes with the  
> software.
> 
> I changed the section with my pages to
> 
> <meta name='robots' content='index,follow' />

also put a plain text file called "robots.txt" at the top of your
site's document tree (i.e. the "Document Root").

in it, put the following lines:

User-agent: *
Disallow: /path/to/pmwiki/documentation
Disallow: /path/to/anything/else/you/want/excluded

where the paths are relative to the Document Root.



well behaved robots (including google and most other search engine) will obey
the instructions in robots.txt



craig

-- 
craig sanders <cas at taz.net.au>           (part time cyborg)