- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Name: joe
Subject: crawl delay
I am having a problem with indexing this site.... when i index with ./indexer
-N8
are those threads trying to index the same urls?
what i mean is
{03} trying to index http://abc.com/page1.html
{07} also try to index http://abc.com/page1.html
{06} also try to index http://abc.com/page1.html and so on.......
indexer[30899]: {03} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {07} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {03} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {07} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {05} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {08} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {08} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {03} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {07} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 9 of
10 sec.
indexer[30899]: {03} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {07} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {07} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {06} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {04} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
indexer[30899]: {02} cancerres.aacrjournals.org/robots.txt: Crawl-delay: 10 of
10 sec.
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
Read the full topic here:
http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=01;post=
