I wrote something on Lucene-dev: AFAIK Seda was developed for asynchronous I/O, which would mean a redesign of the central FetcherTask class. If every thread downloads 50 files at once, you only need a couple of them in parallel to saturate the network interfaces.
Clemens ----- Original Message ----- From: "Peter Donald" <[EMAIL PROTECTED]> To: "Avalon framework users" <[EMAIL PROTECTED]> Sent: Sunday, March 02, 2003 4:44 AM Subject: Re: [LARM] using SEDA, pros and cons > Any follow up on this? > > On Tue, 18 Feb 2003 14:53, David Worms wrote: > > This email was originaly send earlier today, but I got an delivery > > error so I'm sending it again to a new address (using old email > > address). It was also send to the lucene-dev mailing list > > > > In order to move our project to the next level, it has been discussed > > the use of an SEDA architecture. > > > > With the help of the Excalibur / Event library, I created a simplified > > version of Larm which use SEDA as its backbone architecture. The goal > > was not to make it pretty or even working, but to see how each > > application components could feet in an event stage architecture. > > > > The great thing I notice is how flexible the application become. it is > > extremely easy to map each stage with the others. One of the features > > of LARM is the ability to have different sources( db, web, filesystem, > > ... ), process them, and store them (lucene index, log, ...). This > > seems easily achieveable with SEDA. Also clustering the LARM could be > > done through a specific stage implementation. > > > > This was for the pros. However, it will be great to get some feedback > > because I am really not sure on how to deal with SEDA. here is some > > problems I am facing. > > > > - In order for the crawler to be efficient, I had to raise the number > > of threads, but from what I read in the past, only one or two threads > > should be used in a SEDA environment. > > > > - Also, it looks to consume a lot of memory, which could be due to the > > number of messages put into the queue. > > > > Clemens, and others, please have a look of it, and give me some > > feedback. > > > > http://67.116.155.180/~wdavidw/stage.zip > > > > David. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > -- > Cheers, > > Peter Donald > Doubt is not a pleasant condition, but certainty is absurd. > -- Voltaire > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
