On Tue, 2009-11-10 at 08:40 +0100, Chapuis Bertil wrote: > I had the same problem and solved it in my handler's implementation by only > using local variables and limited concurrent accesses. As I understand the > issue the same limitation could occurs with a custom CrawlingDroid > implementation since all workers are using the same Droid. A nice fix could > be to make the Droid and the GenericFactory abstractions clonable and > invoking the clone method in the Worker's constructor.
Could you provide a patch? When I talked with Javier we discussed this option as well and IMO it is an elegant solution. Thanks for your feedback Bertil. salu2 > > Best regards, > > Bertil Chapuis > > > On Mon, Nov 9, 2009 at 1:37 PM, Thorsten Scherler < > [email protected]> wrote: > > > On Fri, 2009-11-06 at 14:29 +0100, Javier Puerto wrote: > > > Hi, I'm working with Droids and made some URL crawlers to save a lot of > > web > > > pages in disk. In JUnit test, I run a little http server and crawl 20 > > pages, > > > the most times everything works ok but in rare cases I get an error. I > > found > > > the problem in the HandlerFactory implementation, in the example the call > > to > > > handlers is like this: > > > > > > protected void handle(ContentEntity entity, Link link) > > > throws DroidsException, IOException > > > { > > > droid.getHandlerFactory().handle(link.getURI(), entity); > > > } > > > > > > > > > If two or more workers is trying to handle at same time, the > > HandlerFactory > > > will handle the all with the same instance of the handler. The solution > > > could be saving memory or improving performance. > > > > > > The first solution could be implemented adding a "synchronized" to > > > HandlerFactory.handle like this. > > > > > > public synchronized boolean handle(URI uri, ContentEntity entity) > > > throws DroidsException, IOException { > > > for (Handler handler : getMap().values()) { > > > handler.handle(uri, entity); > > > } > > > return true; > > > } > > > > > > Only one handler to share with all workers but this solution is a > > > performance killer. The other approx should be the opposite, each worker > > had > > > his own instance of > > > the handlerfactory or handler. > > > > > > Solution that you think might be more appropriate? > > > > It depends on the usecase I guess. However I think the second option is > > the more common solution. > > > > salu2 > > > > > > > > Salu2. > > -- > > Thorsten Scherler <thorsten.at.apache.org> > > Open Source Java <consulting, training and solutions> > > > > Sociedad Andaluza para el Desarrollo de la Sociedad > > de la Información, S.A.U. (SADESI) > > > > > > > > > > -- Thorsten Scherler <thorsten.at.apache.org> Open Source Java <consulting, training and solutions> Sociedad Andaluza para el Desarrollo de la Sociedad de la Información, S.A.U. (SADESI)
