Re: Data Import from a Queue
Yes this is a good reason for using a queue. I have used Amazon SQS this way and it was simple to set up. Bill Bell Sent from mobile On Jul 20, 2011, at 2:59 AM, Stefan Matheis wrote: > Brandon, > > i don't know how they are using it in detail, but Part of Chef's Architecture > is this one: > > Chef Server -> RabbitMQ -> Chef Solr Indexer -> Solr > http://wiki.opscode.com/download/attachments/7274878/chef-server-arch.png > > Perhaps not exactly, what you're looking for - but may give you an idea? > > Regards > Stefan > > Am 19.07.2011 19:04, schrieb Brandon Fish: >> Let me provide some more details to the question: >> >> I was unable to find any example implementations where individual documents >> (single document per message) are read from a message queue (like ActiveMQ >> or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another >> method. Does anyone know of any available examples for this type of import? >> >> If no examples exist, what would be a recommended commit strategy for >> performance? My best guess for this would be to have a queue per core and >> commit once the queue is empty. >> >> Thanks. >> >> On Mon, Jul 18, 2011 at 6:52 PM, Erick >> Ericksonwrote: >> >>> This is a really cryptic problem statement. >>> >>> you might want to review: >>> >>> http://wiki.apache.org/solr/UsingMailingLists >>> >>> Best >>> Erick >>> >>> On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish >>> wrote: Does anyone know of any existing examples of importing data from a queue into Solr? Thank you. >>> >>
Re: Data Import from a Queue
Brandon, i don't know how they are using it in detail, but Part of Chef's Architecture is this one: Chef Server -> RabbitMQ -> Chef Solr Indexer -> Solr http://wiki.opscode.com/download/attachments/7274878/chef-server-arch.png Perhaps not exactly, what you're looking for - but may give you an idea? Regards Stefan Am 19.07.2011 19:04, schrieb Brandon Fish: Let me provide some more details to the question: I was unable to find any example implementations where individual documents (single document per message) are read from a message queue (like ActiveMQ or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another method. Does anyone know of any available examples for this type of import? If no examples exist, what would be a recommended commit strategy for performance? My best guess for this would be to have a queue per core and commit once the queue is empty. Thanks. On Mon, Jul 18, 2011 at 6:52 PM, Erick Ericksonwrote: This is a really cryptic problem statement. you might want to review: http://wiki.apache.org/solr/UsingMailingLists Best Erick On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish wrote: Does anyone know of any existing examples of importing data from a queue into Solr? Thank you.
Re: Data Import from a Queue
Let me provide some more details to the question: I was unable to find any example implementations where individual documents (single document per message) are read from a message queue (like ActiveMQ or RabbitMQ) and then added to Solr via SolrJ, a HTTP POST or another method. Does anyone know of any available examples for this type of import? If no examples exist, what would be a recommended commit strategy for performance? My best guess for this would be to have a queue per core and commit once the queue is empty. Thanks. On Mon, Jul 18, 2011 at 6:52 PM, Erick Erickson wrote: > This is a really cryptic problem statement. > > you might want to review: > > http://wiki.apache.org/solr/UsingMailingLists > > Best > Erick > > On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish > wrote: > > Does anyone know of any existing examples of importing data from a queue > > into Solr? > > > > Thank you. > > >
Re: Data Import from a Queue
This is a really cryptic problem statement. you might want to review: http://wiki.apache.org/solr/UsingMailingLists Best Erick On Fri, Jul 15, 2011 at 1:52 PM, Brandon Fish wrote: > Does anyone know of any existing examples of importing data from a queue > into Solr? > > Thank you. >
Data Import from a Queue
Does anyone know of any existing examples of importing data from a queue into Solr? Thank you.
