Re: ingest performance degrades sharply along with the documents having more fileds

2014-06-24 Thread Cindy Hsin
Looks like the memory usage increased a lot with 10k fields with these two parameter disabled. Based on the experiment we have done, looks like ES have abnormal memory usage and performance degradation when number of fields are large (ie. 10k). Where Solr memory usage and performance remains

Re: ingest performance degrades sharply along with the documents having more fileds

2014-06-23 Thread Cindy Hsin
Thanks! I have asked Maco to re-test ES with these two parameter disabled. One more question regard Lucene's capability with large amount of metadata fields. What is the largest meta data fileds Lucene supports per Index? What are different strategy to solve the large metadata fields issue? Do

Re: ingest performance degrades sharply along with the documents having more fileds

2014-06-17 Thread Cindy Hsin
The way we make Solr ingest faster (single document ingest) is by turn off the engine soft commit and hard commit and use a client to commit the changes every 10 seconds. Solr ingest speed remains at 800 docs per second where ES ingest speed drops in half when we increase the fields (ie. from

Re: ingest performance degrades sharply along with the documents having more fileds

2014-06-13 Thread Cindy Hsin
Hi, Mark: We are doing single document ingestion. We did a performance comparison between Solr and Elastic Search (ES). The performance for ES degrades dramatically when we increase the metadata fields where Solr performance remains the same. The performance is done in very small data set (ie.