On Monday, October 8, 2018 at 12:18:27 AM UTC-4, Hassan Schroeder wrote:
>
> On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu > wrote:
> > I see bots crawling devise pages in my log:
>
> > Should I disallow those paths in robots.txt ?
>
> 1) why?
>
> 2) do your logs show these bots *reading* robot
On Fri, Oct 5, 2018 at 11:04 PM, fugee ohu wrote:
> I see bots crawling devise pages in my log:
> Should I disallow those paths in robots.txt ?
1) why?
2) do your logs show these bots *reading* robots.txt?
--
Hassan Schroeder hassan.schroe...@gmail.com
twitter: @hassa
I see bots crawling devise pages in my log:
141.8.143.179 - - [06/Oct/2018:00:35:45 -0400] "GET /users/sign_in
HTTP/1.1" 200 4022 "-" "Mozilla/5.0 (compatible; YandexBot/3.0;
+http://yandex.com/bots)"
Should I disallow those paths in robots.txt ?
--
You received this message because you are sub
3 matches
Mail list logo