- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: Maxime
Subject: Re: robots.txt:
Where you have placed "Robots no" in your indexer.conf ?
It should be before Server/Realm command that it should affects.
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Read the full topic here
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: mico
Subject: Re: robots.txt:
Actually, I got some everywhere (Beginning, Middle, Before Server, End of file,
etc.)
:)
But here's what i just tested in indexer.conf:
Robots no
Server http://some.site.com/
I don't have any Realm
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: mico
Subject: robots.txt: 'Disallow /'
I'm trying to get some pages from a Server that has in robots.txt:
User-agent: *
Disallow: /
I did put in indexer.conf:
Robots no
But it didn't change anything I still get the same: "robots.txt
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: mico
Subject: Re: robots.txt:
and...
I dumped manually the table 'robots' in search DB.
I deleted the whole DB and created it again.
But nothing changes, robots.txt is still not ignored.
Thx for help,
cheers,
mico
- - - - -
