Re: [Full-disclosure] Google's robots.txt handling
Yes I think you misunderstood or more likely I poorly worded the post. White listing is better than black listing. Black listing something you don't want googlebot to index just makes it easier for someone to find something you don't want indexed. If that content is sensitive, it probably should not be publicly accessible in the first place. But people never put sensitive content on web server (weak attempt at humor, my apologies). I am beating the dead horse here, but robots.txt is not a security control. Most of the time robots.txt is great for recon sense and not Amy measure of defense. White listing just helps in not exposing too much information, a speed bump if anything security related. I think this falls under the 'defense in depth' heading. -Original Message- From: full-disclosure-boun...@lists.grok.org.uk [mailto:full-disclosure-boun...@lists.grok.org.uk] On Behalf Of Christoph Gruber Sent: Wednesday, December 12, 2012 3:19 AM To: full-disclosure@lists.grok.org.uk Subject: Re: [Full-disclosure] Google's robots.txt handling On 12.12.2012 at 00:23 "Lehman, Jim" wrote: > It is possible to use white listing for robots.txt. Allow what you want > google to index and deny everything else. That way google doesn't make you a > goole dork target and someone browsing to your robots.txt file doesn't glean > any sensitive files or folders. But this will not stop directory bruting to > discover your publicly exposed sensitive data, that probably should not be > exposed to the web in the first place. Maybe I misunderstood something, but do you really think that "sensitive" can be hidden in "secret" directories on publicly reachable web servers? -- Christoph Gruber By not reading this email you don't agree you're not in any way affiliated with any government, police, ANTI- Piracy Group, RIAA, MPAA, or any other related group, and that means that you CANNOT read this email. By reading you are not agreeing to these terms and you are violating code 431.322.12 of the Internet Privacy Act signed by Bill Clinton in 1995. (which doesn't exist) ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/ *** This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. *** ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Re: [Full-disclosure] Google's robots.txt handling
It is possible to use white listing for robots.txt. Allow what you want google to index and deny everything else. That way google doesn't make you a goole dork target and someone browsing to your robots.txt file doesn't glean any sensitive files or folders. But this will not stop directory bruting to discover your publicly exposed sensitive data, that probably should not be exposed to the web in the first place. I would rather have some one pound on my server to find something, I might have more time to respond, rather than having mr. bad googleing for the weakness in the web site and only making one request to get what they are after. http://www.sans.org/reading_room/whitepapers/awareness/robotstxt_33955 Its not a great paper, but it might have some value for those that have not looked into how this file works. -Original Message- From: full-disclosure-boun...@lists.grok.org.uk [mailto:full-disclosure-boun...@lists.grok.org.uk] On Behalf Of Hurgel Bumpf Sent: Monday, December 10, 2012 11:26 AM To: full-disclosure@lists.grok.org.uk Subject: [Full-disclosure] Google's robots.txt handling Hi list, i tried to contact google, but as they didn't answer my email, i do forward this to FD. This "security" feature is not cleary a google vulnerability, but exposes websites informations that are not really intended to be public. (Additionally i have to say that i advocate robots.txt files without sensitive content and working security mechanisms.) Here is an example: An admin has a public webservice running with folders containing sensitive informations. Enter these folders in his robots.txt and "protect" them from the indexing process of spiders. As he doesn't want the /admin/ gui to appear in the search results he also puts his /admin in the robots text and finaly makes a backup to the folder /backup. Nevertheless these folders arent browsable but they might contain f(a)iles with easy to guess namestructures, non-encrypted authentications (simple AUTH) , you name it... Without a robots.txt nobody would know about the existance of these folders, but as some folders might be linked somewhere, these folders might appear in search results when not defined in the robots.txt The admin finds himself in a catch-22 situation where he seems to prefer the robots.txt file. Long story short. Although google accepts and respects the directives of the robots.txt file, google INDEXES these files. This my concern. http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fadmin http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fbackup http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+Disallow%3A+%2Fpassword As these searches can be used less for targeted attacks, they more can be used to find victims. http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+wp-admin http://www.google.com/search?q=inurl:robots.txt+filetype%3Atxt+%2FDisallow%3A+typo3 This shouldn't be a discussion about bad practice but the google feature itself. Indexing a file which is used to prevent indexing.. isn't that just paradox and hypocrite? Thanks, Conan the bavarian ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/ *** This message (including any files transmitted with it) may contain confidential and/or proprietary information, is the property of Interactive Data Corporation and/or its subsidiaries, and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution, or use of this message or any attachments is prohibited and may be unlawful. *** ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
Re: [Full-disclosure] metasploit.com = 127.0.0.1
The incoming connection rate has exceeded 15Mbps of just SYN packets, so we decided to point www.metasploit.com and metasploit.com back to 127.0.0.1 for a little while. This is more to keep our ISP happy than any fear of bandwidth charges. We ran a packet capture of the incoming SYN traffic for about 8 hours; it takes up approximately 60Gb of disk space. In the meantime, if you want to access the Metasploit web site, please use: http://metasploit.org -Original Message- From: full-disclosure-boun...@lists.grok.org.uk [mailto:full-disclosure-boun...@lists.grok.org.uk] On Behalf Of Jeremy Brown Sent: Wednesday, February 11, 2009 8:34 AM To: full-disclosure@lists.grok.org.uk Subject: Re: [Full-disclosure] metasploit.com = 127.0.0.1 balliwicked2 On Wed, Feb 11, 2009 at 11:05 AM, sr. wrote: > Well, i can resolve the IP's just fine. just can't connect to port 80. > I'm the fw / network person at my job, and i don't remember adding a > rule for this :-P > > I can get there just fine now, seemed inaccessible to me for a short time. > > thx all... > > fabrizio > > On Wed, Feb 11, 2009 at 11:00 AM, Michael Holstein > wrote: >> >>> that's all fine and dandy. still can't reach port 80. >>> >> >> Have you tried using OpenDNS, etc. to see if it resolves? >> >> eg: host -t a www.metasploit.org *208.67.222.222 >> >> Perhaps your school/employeer/ISP has decided that Metasploit is off-limits. >> >> ~Mike.* >> > > ___ > Full-Disclosure - We believe in it. > Charter: http://lists.grok.org.uk/full-disclosure-charter.html > Hosted and sponsored by Secunia - http://secunia.com/ > ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/ DISCLAIMER: This message (including any files transmitted with it) may contain confidential and / or proprietary information, is the property of Interactive Data Corporation and / or its subsidiaries and is directed only to the addressee(s). If you are not the designated recipient or have reason to believe you received this message in error, please delete this message from your system and notify the sender immediately. An unintended recipient's disclosure, copying, distribution or use of this message, or any attachments, is prohibited and may be unlawful. ___ Full-Disclosure - We believe in it. Charter: http://lists.grok.org.uk/full-disclosure-charter.html Hosted and sponsored by Secunia - http://secunia.com/
