Hello Jay. Actually, Google Webmaster Tools are pretty unhappy about sites without (even empty) robots.txt files. I don't know whether this affects the ranking in the search engine, but definitely it affects the frequency of Google crawling your webpage. I have noticed the frequency to go dramatically down on one of my sites where the robots txt has not been available for a little over a week, and google presented me with a lot of messages that it stopped crawling due to a missing robots.txt.
I was not talking about how to increase a websites rank btw, only how to exclude pages from being displayed in the search results. Just to clarify. Regards Bernhard On Jan 31, 2008 5:00 PM, Jay Klehr <[EMAIL PROTECTED]> wrote: > > Search engines use robots.txt to decide if they're *allowed* to index all > of your site, it won't improve your ranking if you have one. I don't know > where this "if you have a robots.txt file, even with nothing in it, you'll > get ranked better" idea came from (you're not the first person I've seen say > something like this), but it's false. Don't waste your time with a > robots.txt file unless there are particular areas of your site that you > DON'T want search engines to index. > > Spend more of your time building a site-map, making your html search engine > friendly, and most importantly, make the site "useful". If your site has > value to its users, google's ranking will reflect that. > > Jay > > > > Bernhard Schussek wrote: > You can write the url of the page to the robots.txt in your /web/ > directory, which is scanned by search engines. Search the web for > robots.txt for more information about it. > > Additionally, you might like to get comfortable with the Google > Webmaster Tools. They allow viewing the crawler status, indexed and > erroneous pages etc. Just sign up for a Google account and unlock the > website for this account. See the Google Webmaster Tools documentation > for more information. > > Regards > Bernhard > > On Jan 31, 2008 6:55 AM, Birchandra Sanasam <[EMAIL PROTECTED]> wrote: > > > Dear All, > > I developed and announced my second project on this group. So I am facing a > big problem for google search engine. > When I search for "amrit cwt" on google I got "My Second Symfony Project - > symfony users" at first rank > but I want to get site home page first. What can I do for it? > > My client asked for take off that list i.e. "My Second Symfony Project - > symfony users" from google. > But I don't know How could I take off it? Please help me! > > > Thanks! > > I am looking forward your help! > > -- > Birchandra Sanasam > Web Developer & Analyst > # +91 9810 191478 > > > > > > > > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -~----------~----~----~----~------~----~------~--~---
