I asked this question already in the Google Webmasters group but I was recommended to ask it here as they seem to think it might be AppEngine related ...
"I have read the FAQs and checked for similar issues: YES My site's URL (web address) is: http://bptrack.appspot.com Description (including timeline of any changes made): I have the following in my robot.txt file: User-agent: * Allow: / Disallow: /datasource/ It has been that for over a month now yet when I look for my site on google (site:http://bptrack.appspot.com), it only finds http://bptrack.appspot.com/datasource/, exactly the URL I do not want it to index. It doesn't find any other URLs, including the main one (http://bptrack.appspot.com). Can anybody please explain to me how that is possible? Do I have a mistake in my robot.txt file (eventhough I had it generated by Google Webmasters Tool)? Cheers, Mark." Through the google Websmasters Group I figured out that the robot.txt file might not actually prevent that URL from being crawled and/or indexed, however, http://bptrack.appspot.com/datasource/ is not even a webpage, so it is "empty" AND inaccessible so why would that one be indexed at all and none of the other pages which are accessible ("home" page, "about" and "contacts")? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
