Dennis
Beats wrote:
hi... anybody can plz help me on this....... i m stuck with itthanx
Actually no. Nutch is made to obey robots.txt. All polite crawlers
will obey rules set down in robots.txt.
- Ignoring robots.txt Beats
- Re: Ignoring robots.txt Beats
- Re: Ignoring robots.txt Dennis Kubes
- Ignoring Robots.txt Super Man
- Re: Ignoring Robots.txt David M. Cole
- Re: Ignoring Robots.txt Super Man
- Re: Ignoring Robots.txt John Mendenhall
- RE: Ignoring Robots.txt Fuad Efendi
- Re: Ignoring Robots.txt Guillermo Garrido
- Strange search results alxsss
- Re: Ignoring Robots.txt Kirby Bohling
