Thanks Jan. For the JMSUpdateHandler option, how does one plugin a custom UpdateHandler? I want to make sure I'm not missing any important pieces of Solr processing pipeline.
Best, Rich On Tue, Feb 15, 2011 at 4:36 AM, Jan Høydahl <jan....@cominvent.com> wrote: > Solr is multi threaded, so you are free to send as many parallel update > requests needed to utilize your HW. Each request will get its own thread. > Simply configure StreamingUpdateSolrServer from your client. > > If there is some lengthy work to be done, it needs to be done in SOME > thread, and I guess you just have to choose where :) > > A JMSUpdateHandler sounds heavy weight, but does not need to be, and might > be the logically best place for such a feature imo. > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > > On 14. feb. 2011, at 17.42, Rich Cariens wrote: > > > Thanks Jan, > > > > I don't think I want to tie up a thread on two boxes waiting for an > > UpdateRequestProcessor to finish. I'd prefer to offload it all to the > target > > shards. And a special JMSUpdateHandler feels like overkill. I *think* I'm > > really just looking for a simple API that allows me to add a > > SolrInputDocument to the index in-process. > > > > Perhaps I just need to use the EmbeddedSolrServer in the Solrj packages? > I'm > > worried that this will break all the nice stuff one gets with the > standard > > SOLR webapp (stats, admin, etc). > > > > Best, > > Rich > > > > > > On Mon, Feb 14, 2011 at 11:18 AM, Jan Høydahl <jan....@cominvent.com> > wrote: > > > >> Hi, > >> > >> One option would be to keep the JMS listener as today but move the > >> downloading > >> and transforming part to a SolrUpdateRequestProcessor on each shard. The > >> benefit > >> is that you ship only a tiny little SolrInputDocument over the wire with > a > >> reference to the doc to download, and do the heavy lifting on Solr side. > >> > >> If each JMS topic/channel corresponds to a particular shard, you could > >> move the whole thing to Solr. If so, a new JMSUpdateHandler could > perhaps > >> be a way to go? > >> > >> -- > >> Jan Høydahl, search solution architect > >> Cominvent AS - www.cominvent.com > >> > >> On 14. feb. 2011, at 16.53, Rich Cariens wrote: > >> > >>> Hello, > >>> > >>> I've built a system that receives JMS events containing links to docs > >> that I > >>> must download and index. Right now the JMS receiving, downloading, and > >>> transformation into SolrInputDoc's happens in a separate JVM that then > >> uses > >>> Solrj javabin HTTP POSTs to distribute these docs across many index > >> shards. > >>> > >>> For various reasons I won't go into here, I'd like to relocate/deploy > >> most > >>> of my processing (JMS receiving, downloading, and transformation into > >>> SolrInputDoc's) into the SOLR webapps running on each distributed shard > >>> host. I might be wrong, but I don't think the request-driven idiom of > the > >>> DataImportHandler is not a good fit for me as I'm not kicking off full > or > >>> delta imports. If that's true, what's the correct way to hook my > >> components > >>> into SOLR's update facilities? Should I try to get a reference a > >> configured > >>> DirectUpdateHandler? > >>> > >>> I don't know if this information helps, but I'll put it out there > >> anyways: > >>> I'm using Spring 3 components to receive JMS events, wired up via > webapp > >>> context hooks. My plan would be to add all that to my SOLR shard > webapp. > >>> > >>> Best, > >>> Rich > >> > >> > >
