Re: [Rails] disallow devise paths in robots.txt ?

2018-10-08 Thread fugee ohu
On Monday, October 8, 2018 at 12:18:27 AM UTC-4, Hassan Schroeder wrote: > > On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu > wrote: > > I see bots crawling devise pages in my log: > > > Should I disallow those paths in robots.txt ? > > 1) why? > > 2) do your logs show these bots *reading* robot

Re: [Rails] disallow devise paths in robots.txt ?

2018-10-07 Thread Hassan Schroeder
On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu wrote: > I see bots crawling devise pages in my log: > Should I disallow those paths in robots.txt ? 1) why? 2) do your logs show these bots *reading* robots.txt? -- Hassan Schroeder hassan.schroe...@gmail.com twitter: @hassa

[Rails] disallow devise paths in robots.txt ?

2018-10-05 Thread fugee ohu
I see bots crawling devise pages in my log: 141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] "GET /users/sign_in HTTP/1.1" 200 4022 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" Should I disallow those paths in robots.txt ? -- You received this message because you are sub