[LINK] Latency in indexing robots
cas at taz.net.au
Thu Dec 7 08:09:43 AEDT 2006
On Wed, Dec 06, 2006 at 01:49:38PM +1100, Antony Barry wrote:
> to each page. I've now got to the stage where I _want_ my new pages
> to be indexed but not the pmwiki documentation which comes with the
> I changed the section with my pages to
> <meta name='robots' content='index,follow' />
also put a plain text file called "robots.txt" at the top of your
site's document tree (i.e. the "Document Root").
in it, put the following lines:
where the paths are relative to the Document Root.
well behaved robots (including google and most other search engine) will obey
the instructions in robots.txt
craig sanders <cas at taz.net.au> (part time cyborg)
More information about the Link