Chris, I throw rack-rewrite (https://github.com/jtrupiano/rack-rewrite) in most apps for various purposes, this being one of them. You can use an if: -> (rack_env) { rack_env["HTTP_USER_AGENT"].downcase == "genieo" } or something like it to handle your case.
James On Thu, Mar 13, 2014 at 11:38 AM, Chris McCann <[email protected]>wrote: > Yeah, the user agent I'm getting is what I posted in the original post, > uppercase "Genieo". I could add another disallow for the lower case to see > if it makes a difference. > > > On Thu, Mar 13, 2014 at 11:36 AM, Adam Grant <[email protected]>wrote: > >> The only other thing I can think of is case sensitivity: >> http://www.genieo.com/robots.txt >> >> They use lower case "g" in "genieo". Might just double check in your web >> server logs that the user agent isn't lowercased, even though their site >> says it's uppercased. >> >> Good luck! >> - Adam >> >> >> On Thu, Mar 13, 2014 at 11:28 AM, Chris McCann <[email protected]>wrote: >> >>> Thanks, Adam. According to the documentation Genieo provides it sounds >>> like their crapware is making additional requests on behalf of the user: >>> >>> Genieo is designed to automatically retrieve and filter information >>>> items from across the web, based on the user's specific individual >>>> interests, and display them on a personal Homepage. >>>> Genieo studies the user's preferred individual interests and sources at >>>> a high resolution, by analyzing their browsing routine. Our desktop >>>> configuration runs on the user's compuer to increase privacy. Genieo then >>>> continuously explores the internet for specific information items which are >>>> mostly related to these interests, and presents them on the personal >>>> Homepage. >>>> The Genieo homepage is styled as a newspaper front page with selected >>>> items, consisting of title, short snippet, media, link to article, and >>>> share button >>> >>> >>> My initial request to them for help indicated that their software DID >>> respect robots.txt, which is why I went this route. >>> >>> I am also looking at a Rack-based solution to simply redirect any >>> requests with that user agent. >>> >>> Chris >>> >>> >>> On Thu, Mar 13, 2014 at 11:24 AM, Adam Grant <[email protected]>wrote: >>> >>>> Question: Does the crapware modify your user's browser request, or is >>>> it making a second request unbeknownst to your user? >>>> >>>> Crapware doesn't usually respect robots.txt. You might need to put a >>>> middleware in your Rails app to filter out that user agent, or something at >>>> the Apache/Nginx level. Not ideal, but at least it won't trigger a routing >>>> error. >>>> >>>> Regards, >>>> - Adam >>>> >>>> >>>> On Thu, Mar 13, 2014 at 11:17 AM, Chris McCann >>>> <[email protected]>wrote: >>>> >>>>> I've had a rash of exception notifications come through from one of my >>>>> Rails apps lately. >>>>> >>>>> A ActionController::MethodNotAllowed occurred in application#index: >>>>>> Only put and delete requests are allowed. >>>>> >>>>> >>>>> The culprit appears to be this: >>>>> >>>>> HTTP_USER_AGENT : Mozilla/5.0 (compatible; Genieo/1.0 >>>>> http://www.genieo.com/webfilter.html) >>>>> >>>>> You can click the link to see what Genieo is (tl;dr: widely panned as >>>>> crapware). It looks to me like one of my users has this crapware on their >>>>> computer, had it running when they interacted with my app at a URL that >>>>> should take a PUT, and now Genieo keeps hitting that link with a GET, >>>>> triggering the error. >>>>> >>>>> Looking for ways to prevent this it seemed that putting a disallow for >>>>> Genieo in robots.txt would solve it. Here's my robots.txt file: >>>>> >>>>> # See http://www.robotstxt.org/wc/norobots.html for documentation on how >>>>> to use the robots.txt file >>>>> >>>>> # >>>>> >>>>> # To ban all spiders from the entire site uncomment the next two lines: >>>>> >>>>> User-Agent: Genieo >>>>> >>>>> Disallow: / >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> After redeploying I'm still getting the hits but don't understand why. >>>>> I've put a request into their support to figure this out, but I'm curious >>>>> if anyone else here has dealt with a problem like this in a similar >>>>> fashion. >>>>> >>>>> >>>>> Cheers, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Chris >>>>> >>>>> -- >>>>> -- >>>>> SD Ruby mailing list >>>>> [email protected] >>>>> http://groups.google.com/group/sdruby >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "SD Ruby" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> -- >>>> SD Ruby mailing list >>>> [email protected] >>>> http://groups.google.com/group/sdruby >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "SD Ruby" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/sdruby/v8YBg5uRDj0/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> http://groups.google.com/group/sdruby >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "SD Ruby" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "SD Ruby" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/sdruby/v8YBg5uRDj0/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to the Google Groups > "SD Ruby" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
