[
https://issues.apache.org/jira/browse/LABS-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638988#action_12638988
]
Javier Puerto commented on LABS-190:
------------------------------------
>Thorsten Scherler - 12/Oct/08 01:12 PM
>I reviewed your patch but I do not get:
>Why should parseRobots be cofigurable?
If you want to make a droid that read the robots.txt or no to the parser. I
make configurable because the code before was fixed to true, i think that it
must be false, but with possibility to change. There's no much sense to let
parse this file.
> concurrency error in NoRobotClient
> ----------------------------------
>
> Key: LABS-190
> URL: https://issues.apache.org/jira/browse/LABS-190
> Project: Labs
> Issue Type: Bug
> Components: Droids
> Environment: Ubuntu 8.04, JDK 1.6
> Reporter: Javier Puerto
> Priority: Blocker
> Attachments: NoRobotClient.diff
>
>
> Testing with droids, when the number of workers rise the NoRobotClient thows
> an exception. I was searching for the error without sucess but it seems to be
> caused by the concurrency with the base url.
> This is the error:
> pool-1-thread-3: Starting org.apache.droids.crawler.CrawlingWorker
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.String.substring(String.java:1938)
> at java.lang.String.substring(String.java:1905)
> at
> org.apache.http.norobots.NoRobotClient.isUrlAllowed(NoRobotClient.java:202)
> at org.apache.droids.protocol.http.Http.isAllowed(Http.java:87)
> at
> org.apache.droids.crawler.CrawlingWorker.execute(CrawlingWorker.java:49)
> at
> org.apache.droids.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
> at
> org.apache.droids.impl.MultiThreadedTaskMaster$WorkerRunner.run(MultiThreadedTaskMaster.java:186)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> pool-1-thread-1: Worker "76" has finished.
> pool-1-thread-2: Url is allowed
> java.lang.StringIndexOutOfBoundsException: String index out of range: -2
> at java.lang.String.substring(String.java:1938)
> at java.lang.String.substring(String.java:1905)
> at
> org.apache.http.norobots.NoRobotClient.isUrlAllowed(NoRobotClient.java:202)
> at org.apache.droids.protocol.http.Http.isAllowed(Http.java:87)
> at
> org.apache.droids.crawler.CrawlingWorker.execute(CrawlingWorker.java:49)
> at
> org.apache.droids.crawler.CrawlingWorker.execute(CrawlingWorker.java:1)
> at
> org.apache.droids.impl.MultiThreadedTaskMaster$WorkerRunner.run(MultiThreadedTaskMaster.java:186)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
