I am quite confident your problem has nothing to do with app engine. Your robots.txt looks fine to me. As you also mention there can be some legitimate reasons that Google can visit a page excluded by robots.txt /datasource/ can be an html page or any other resource for that matter as far a browser is concerned. My small piece of advice : sign in for G webmasters tools and submit a sitemap.xml, then use webmasters tools to see what is going on with your site's indexing and all seo related issues hm.. I have seen you have already submitted the site. By the way you have left DEBUG=True in your production server. hope all this helped ;-)
On Sep 16, 11:25 pm, "mvilr...@gmail.com" <mvilr...@gmail.com> wrote: > I asked this question already in the Google Webmasters group but I was > recommended to ask it here as they seem to think it might be AppEngine > related ... > > "I have read the FAQs and checked for similar issues: YES > > My site's URL (web address) is:http://bptrack.appspot.com > > Description (including timeline of any changes made): > I have the following in my robot.txt file: > > User-agent: * > Allow: / > Disallow: /datasource/ > > It has been that for over a month now yet when I look for my site on > google (site:http://bptrack.appspot.com), it only > findshttp://bptrack.appspot.com/datasource/, exactly the URL I do not want > it to index. It doesn't find any other URLs, including the main one > (http://bptrack.appspot.com). Can anybody please explain to me how > that is possible? Do I have a mistake in my robot.txt file > (eventhough I had it generated by Google Webmasters Tool)? > > Cheers, > Mark." > > Through the google Websmasters Group I figured out that the robot.txt > file might not actually prevent that URL from being crawled and/or > indexed, however,http://bptrack.appspot.com/datasource/is not even a > webpage, so it is "empty" AND inaccessible so why would that one be > indexed at all and none of the other pages which are accessible > ("home" page, "about" and "contacts")? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
