Re: Fastest way to import big amount of documents in SolrCloud
re: optimize after every import This is not recommended in 4.x unless and until you have evidence that it really does help, reviews are very mixed, and it's been renamed force merge in 4.x just so people don't think "Of course I want to do this, who wouldn't?". bq: Doing a commit instead of optimize is usually bringing the master and slave nodes down This isn't expected unless you're committing far too frequently. I'd dis-recommend doing any commits except, possibly, a single commit after all my clients had finished indexing. But even that isn't necessary. In batch modes in SolrCloud, reasonable setups are autocommit: 15 seconds WITH openSearcher="false" autosoftcommit: the interval it takes you to run all your indexing. Seems odd, but here's the backtround: http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Best, Erick On Thu, May 1, 2014 at 11:12 PM, Alexander Kanarsky wrote: > If you build your index in Hadoop, read this (it is about the Cloudera > Search but in my understanding also should work with Solr Hadoop contrib > since 4.7) > http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-User-Guide/csug_batch_index_to_solr_servers_using_golive.html > > > On Thu, May 1, 2014 at 1:47 PM, Costi Muraru wrote: > >> Hi guys, >> >> What would you say it's the fastest way to import data in SolrCloud? >> Our use case: each day do a single import of a big number of documents. >> >> Should we use SolrJ/DataImportHandler/other? Or perhaps is there a bulk >> import feature in SOLR? I came upon this promising link: >> http://wiki.apache.org/solr/UpdateCSV >> Any idea on how UpdateCSV is performance-wise compared with >> SolrJ/DataImportHandler? >> >> If SolrJ, should we split the data in chunks and start multiple clients at >> once? In this way we could perhaps take advantage of the multitude number >> of servers in the SolrCloud configuration? >> >> Either way, after the import is finished, should we do an optimize or a >> commit or none ( >> http://wiki.solarium-project.org/index.php/V1:Optimize_command)? >> >> Any tips and tricks to perform this process the right way are gladly >> appreciated. >> >> Thanks, >> Costi >>
Re: Fastest way to import big amount of documents in SolrCloud
If you build your index in Hadoop, read this (it is about the Cloudera Search but in my understanding also should work with Solr Hadoop contrib since 4.7) http://www.cloudera.com/content/cloudera-content/cloudera-docs/Search/latest/Cloudera-Search-User-Guide/csug_batch_index_to_solr_servers_using_golive.html On Thu, May 1, 2014 at 1:47 PM, Costi Muraru wrote: > Hi guys, > > What would you say it's the fastest way to import data in SolrCloud? > Our use case: each day do a single import of a big number of documents. > > Should we use SolrJ/DataImportHandler/other? Or perhaps is there a bulk > import feature in SOLR? I came upon this promising link: > http://wiki.apache.org/solr/UpdateCSV > Any idea on how UpdateCSV is performance-wise compared with > SolrJ/DataImportHandler? > > If SolrJ, should we split the data in chunks and start multiple clients at > once? In this way we could perhaps take advantage of the multitude number > of servers in the SolrCloud configuration? > > Either way, after the import is finished, should we do an optimize or a > commit or none ( > http://wiki.solarium-project.org/index.php/V1:Optimize_command)? > > Any tips and tricks to perform this process the right way are gladly > appreciated. > > Thanks, > Costi >
Re: Fastest way to import big amount of documents in SolrCloud
Thanks for the reply, Anshum. Please see my answers to your questions below. * Why do you want to do a full index everyday? Not sure I understand what you mean by full index. Every day we want to import additional documents to the existing ones. Of course, we want to remove older ones as well, so the total amount remains roughly the same. * How much of data are we talking about? The number of new documents is around 500k each day. * What's your SolrCloud setup like? We're currently using Solr 3.6 with 16 shards and planning to switch to SolrCloud, hence the inquiry. * Do you already have some benchmarks which you're not happy with? Not yet. Planning to do some tests quite soon. I was looking for some guidance before jumping in. "Also, it helps to set the commit intervals reasonable." What do you mean by *reasonable*? Also, do you recommend using autoCommit? We are currently doing an optimize after each import (in Solr 3), in order to speed up future queries. This is proving to take very long though (several hours). Doing a commit instead of optimize is usually bringing the master and slave nodes down. We reverted to calling optimize on every ingest. On Thu, May 1, 2014 at 11:57 PM, Anshum Gupta wrote: > Hi Costi, > > I'd recommend SolrJ, parallelize the inserts. Also, it helps to set the > commit intervals reasonable. > > Just to get a better perspective > * Why do you want to do a full index everyday? > * How much of data are we talking about? > * What's your SolrCloud setup like? > * Do you already have some benchmarks which you're not happy with? > > > > On Thu, May 1, 2014 at 1:47 PM, Costi Muraru > wrote: > > > Hi guys, > > > > What would you say it's the fastest way to import data in SolrCloud? > > Our use case: each day do a single import of a big number of documents. > > > > Should we use SolrJ/DataImportHandler/other? Or perhaps is there a bulk > > import feature in SOLR? I came upon this promising link: > > http://wiki.apache.org/solr/UpdateCSV > > Any idea on how UpdateCSV is performance-wise compared with > > SolrJ/DataImportHandler? > > > > If SolrJ, should we split the data in chunks and start multiple clients > at > > once? In this way we could perhaps take advantage of the multitude number > > of servers in the SolrCloud configuration? > > > > Either way, after the import is finished, should we do an optimize or a > > commit or none ( > > http://wiki.solarium-project.org/index.php/V1:Optimize_command)? > > > > Any tips and tricks to perform this process the right way are gladly > > appreciated. > > > > Thanks, > > Costi > > > > > > -- > > Anshum Gupta > http://www.anshumgupta.net >
Re: Fastest way to import big amount of documents in SolrCloud
Hi Costi, I'd recommend SolrJ, parallelize the inserts. Also, it helps to set the commit intervals reasonable. Just to get a better perspective * Why do you want to do a full index everyday? * How much of data are we talking about? * What's your SolrCloud setup like? * Do you already have some benchmarks which you're not happy with? On Thu, May 1, 2014 at 1:47 PM, Costi Muraru wrote: > Hi guys, > > What would you say it's the fastest way to import data in SolrCloud? > Our use case: each day do a single import of a big number of documents. > > Should we use SolrJ/DataImportHandler/other? Or perhaps is there a bulk > import feature in SOLR? I came upon this promising link: > http://wiki.apache.org/solr/UpdateCSV > Any idea on how UpdateCSV is performance-wise compared with > SolrJ/DataImportHandler? > > If SolrJ, should we split the data in chunks and start multiple clients at > once? In this way we could perhaps take advantage of the multitude number > of servers in the SolrCloud configuration? > > Either way, after the import is finished, should we do an optimize or a > commit or none ( > http://wiki.solarium-project.org/index.php/V1:Optimize_command)? > > Any tips and tricks to perform this process the right way are gladly > appreciated. > > Thanks, > Costi > -- Anshum Gupta http://www.anshumgupta.net
Fastest way to import big amount of documents in SolrCloud
Hi guys, What would you say it's the fastest way to import data in SolrCloud? Our use case: each day do a single import of a big number of documents. Should we use SolrJ/DataImportHandler/other? Or perhaps is there a bulk import feature in SOLR? I came upon this promising link: http://wiki.apache.org/solr/UpdateCSV Any idea on how UpdateCSV is performance-wise compared with SolrJ/DataImportHandler? If SolrJ, should we split the data in chunks and start multiple clients at once? In this way we could perhaps take advantage of the multitude number of servers in the SolrCloud configuration? Either way, after the import is finished, should we do an optimize or a commit or none ( http://wiki.solarium-project.org/index.php/V1:Optimize_command)? Any tips and tricks to perform this process the right way are gladly appreciated. Thanks, Costi