-
<blockquote>
I am using the robots.htm file in the way I have believed was normal practice</p></blockquote>
I can't vouch for 'normal practice', but I for one have never seen this approach used before!</p>
Currently, in IE, I see an explanation page with a link to the 'non-JavaScript' version of the page at the bottom. But the non-JavaScript page turns out to be a redirect implemented in JavaScript, so I don't get any further. (Tip: JavaScript and meta-refresh are not good substitutes for real HTTP redirects.)</p>
How about linking directly to sitemap.htm from the explanation page instead? Or even better, including the sitemap in index1.htm as the noscript content?</p>
-
Thanks for these tips, Andrew and Jen. I have both a robots.txt file for exclusions and a robots.htm file for inclusions. The latter is part of the Joust environment that I use, but I can replace it with a site map better integrated into the user-visible part of the site.
How about this for a set of guidelines for people to shoot at? I realise it is a constantly evolving situation; this is what makes handling it such hard work.
For all sites:
Use a robots.txt file to list directories/files you don't want crawled.
Do not use a file of links to pages which does not appear in user-visible navigation; this will irritate spiders.
Include meta-keywords if you wish but they are now given less weight. The meta-description has most value but more for the users of a search engine.
For a frame-based site:
Use a sitemap.htm file to help spiders reach the pages you want. This file must be included in the user-visible site navigation.
Also, either put these same links to all your pages as a sitemap in the <noframes> section or add a link to the sitemap file.
For a non-frame site where the links are handled through JavaScript:
Use a sitemap.htm file to help spiders reach the pages you want. This file must be included in the user-visible site navigation.
www.bricksandbrass.co.uk
[This message was edited by Jen Worden on March 15, 2002 at 17:29.]
-
I'd say you about hit it on the head, Simon! [img]/infopop/emoticons/icon_smile.gif[/img]
cfn ... Jen
Jen Worden
Web Developer
www.meadoworks.com