Yeah, the user agent I'm getting is what I posted in the original post, uppercase "Genieo". I could add another disallow for the lower case to see if it makes a difference.
On Thu, Mar 13, 2014 at 11:36 AM, Adam Grant <[email protected]> wrote: > The only other thing I can think of is case sensitivity: > http://www.genieo.com/robots.txt > > They use lower case "g" in "genieo". Might just double check in your web > server logs that the user agent isn't lowercased, even though their site > says it's uppercased. > > Good luck! > - Adam > > > On Thu, Mar 13, 2014 at 11:28 AM, Chris McCann <[email protected]>wrote: > >> Thanks, Adam. According to the documentation Genieo provides it sounds >> like their crapware is making additional requests on behalf of the user: >> >> Genieo is designed to automatically retrieve and filter information items >>> from across the web, based on the user's specific individual interests, and >>> display them on a personal Homepage. >>> Genieo studies the user's preferred individual interests and sources at >>> a high resolution, by analyzing their browsing routine. Our desktop >>> configuration runs on the user's compuer to increase privacy. Genieo then >>> continuously explores the internet for specific information items which are >>> mostly related to these interests, and presents them on the personal >>> Homepage. >>> The Genieo homepage is styled as a newspaper front page with selected >>> items, consisting of title, short snippet, media, link to article, and >>> share button >> >> >> My initial request to them for help indicated that their software DID >> respect robots.txt, which is why I went this route. >> >> I am also looking at a Rack-based solution to simply redirect any >> requests with that user agent. >> >> Chris >> >> >> On Thu, Mar 13, 2014 at 11:24 AM, Adam Grant <[email protected]>wrote: >> >>> Question: Does the crapware modify your user's browser request, or is >>> it making a second request unbeknownst to your user? >>> >>> Crapware doesn't usually respect robots.txt. You might need to put a >>> middleware in your Rails app to filter out that user agent, or something at >>> the Apache/Nginx level. Not ideal, but at least it won't trigger a routing >>> error. >>> >>> Regards, >>> - Adam >>> >>> >>> On Thu, Mar 13, 2014 at 11:17 AM, Chris McCann <[email protected]>wrote: >>> >>>> I've had a rash of exception notifications come through from one of my >>>> Rails apps lately. >>>> >>>> A ActionController::MethodNotAllowed occurred in application#index: >>>>> Only put and delete requests are allowed. >>>> >>>> >>>> The culprit appears to be this: >>>> >>>> HTTP_USER_AGENT : Mozilla/5.0 (compatible; Genieo/1.0 >>>> http://www.genieo.com/webfilter.html) >>>> >>>> You can click the link to see what Genieo is (tl;dr: widely panned as >>>> crapware). It looks to me like one of my users has this crapware on their >>>> computer, had it running when they interacted with my app at a URL that >>>> should take a PUT, and now Genieo keeps hitting that link with a GET, >>>> triggering the error. >>>> >>>> Looking for ways to prevent this it seemed that putting a disallow for >>>> Genieo in robots.txt would solve it. Here's my robots.txt file: >>>> >>>> # See http://www.robotstxt.org/wc/norobots.html for documentation on how >>>> to use the robots.txt file >>>> >>>> # >>>> >>>> # To ban all spiders from the entire site uncomment the next two lines: >>>> >>>> User-Agent: Genieo >>>> >>>> Disallow: / >>>> >>>> >>>> >>>> >>>> >>>> After redeploying I'm still getting the hits but don't understand why. >>>> I've put a request into their support to figure this out, but I'm curious >>>> if anyone else here has dealt with a problem like this in a similar >>>> fashion. >>>> >>>> >>>> Cheers, >>>> >>>> >>>> >>>> >>>> >>>> Chris >>>> >>>> -- >>>> -- >>>> SD Ruby mailing list >>>> [email protected] >>>> http://groups.google.com/group/sdruby >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "SD Ruby" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> http://groups.google.com/group/sdruby >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "SD Ruby" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/sdruby/v8YBg5uRDj0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> -- >> SD Ruby mailing list >> [email protected] >> http://groups.google.com/group/sdruby >> --- >> You received this message because you are subscribed to the Google Groups >> "SD Ruby" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/v8YBg5uRDj0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
