[LINK] Latency in indexing robots

Craig Sanders cas at taz.net.au
Thu Dec 7 08:09:43 AEDT 2006

On Wed, Dec 06, 2006 at 01:49:38PM +1100, Antony Barry wrote:
> to each page. I've now got to the stage where I _want_ my new pages  
> to be indexed but not the pmwiki documentation which comes with the  
> software.
> I changed the section with my pages to
> <meta name='robots' content='index,follow' />

also put a plain text file called "robots.txt" at the top of your
site's document tree (i.e. the "Document Root").

in it, put the following lines:

User-agent: *
Disallow: /path/to/pmwiki/documentation
Disallow: /path/to/anything/else/you/want/excluded

where the paths are relative to the Document Root.

well behaved robots (including google and most other search engine) will obey
the instructions in robots.txt


craig sanders <cas at taz.net.au>           (part time cyborg)

More information about the Link mailing list