It is correct for what it can do for a robot that reads the meta tags, however if you submit your url to an engine like Yahoo, AltaVista, or DMOZ then your site will be listed anyways. The "noindex" prevents that page from being indexed by a robot, while the "nofollow" prevents the indexing of links on the page, proper syntax must be observed to make the meta tags work because many sites are spidered even though they have the tags but the syntax is all wrong. Doing your site in PHP, JSP, or ASP (or any of the other scripting languages, Flash included) can also confuse engines, sometimes to the point that they will ignore your pages or site. Also not all engine rely on robots. But I saw a discussion surrounding the use of some head tags that don't utilize the Robots exclusion which prevents engines from seeing you.
Peter Kaulback In the hour of 10:15 AM 7/27/2002 -0500, [EMAIL PROTECTED] spoke this: >I have always heard from HTML design sites that the 'no >index-no follow' type tags WILL prevent SE's from seeing you. >This is how you can keep a page FROM being indexed by an >engine. This is incorrect? >-Clint > >God Bless Us All >Clint Hamilton, Owner >http://OrpheusComputing.com � > >----- Original Message ----- >From: "Peter Kaulback" <[EMAIL PROTECTED]> >To: <[EMAIL PROTECTED]> >Sent: Saturday, July 27, 2002 9:42 AM >Subject: Re: PCWorks: html robots meta-tag > > >Jeff, >This will not prevent spiders from finding your site, there >is some >specific <head> tags that can do this fine without the >robots.txt file. >Also if you have back links to your site then search engines >will find you >this way as well. If you go through your logs on your web >server you should >find a large percentage of accesses attempting to find a >robots.txt file, >many engines look for it and some ignore it. > >On a side note, if you are interested in spidering sites then >you can make >your own here: >http://www.searchtools.com/robots/robot-code.html > >Peter Kaulback > >In the hour of 09:27 PM 7/25/2002 -0400, Jeff Dougherty spoke >this: > > >Peter, > >Won't this method also prevent search engines from spidering >the site? > >Jeff > > > >Intrepid Video & Electronics Be careful of your >thoughts. > >501 Luther Rd They may become your >words > >Harrisburg, PA 17111 any moment. > > > >Original Message ----- > >From: "Peter Kaulback" <[EMAIL PROTECTED]> > > > >In the hour of 04:08 PM 7/24/2002 -0600, Roger Williams >spoke this: > > > > >Hi workers, > > >Do you know of any html tag or meta-tag which will stop >robots from > > >accessing a web page? I currently use the meta-tag "no >index, no follow," > > >but I know that few robots actually support that. The >point is to deny > > >harvesting or indexing of certain pages by robots on a web >site. Anybody > > >have any solutions? > > > > > >All the best, > > >Roger > > > >Roger, you can add a robots text file to your site which can >hamper robots > >BUT if you are linked to any other sites the robots will >find you. > >More info can be found here: > >http://www.robotstxt.org/wc/robots.html > >http://www.searchengineworld.com/robots/robots_tutorial.htm > > > >And there is also the robots text file generator here: > >http://www.webtoolcentral.com/webmaster/tools/robots_txt_fil >e_generator/ > > > >Peter Kaulback ============= PCWorks Mailing List ================= Don't see your post? Check our posting guidelines & make sure you've followed proper posting procedures, http://pcworkers.com/rules.htm Contact list owner <[EMAIL PROTECTED]> Unsubscribing and other changes: http://pcworkers.com =====================================================
