New segment file created too often
Hi, I am using solr 3.5. I seeing solr keeps creating new segment files (1MB files) so often that it triggers segment merge about every one minute. I search the news archive, but could not find any info on this issue. I am indexing about 10 docs of less 2KB each every second. Commit is called after adding each document. Relevant config params are: mergeFactor10/mergeFactor ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs What might be triggering this frequent new segment files creation? Thanks! Huy -- Huy Le Spring Partners, Inc. http://springpadit.com
Re: New segment file created too often
Commit is called after adding each document you should add enough documents and then calling a commit. commit is a cost operation. if you want to get latest feeded documents, you could use NRT On Tue, Feb 14, 2012 at 12:47 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. I seeing solr keeps creating new segment files (1MB files) so often that it triggers segment merge about every one minute. I search the news archive, but could not find any info on this issue. I am indexing about 10 docs of less 2KB each every second. Commit is called after adding each document. Relevant config params are: mergeFactor10/mergeFactor ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs What might be triggering this frequent new segment files creation? Thanks! Huy -- Huy Le Spring Partners, Inc. http://springpadit.com
Re: New segment file created too often
Hi, I am using solr 3.5. As I understood it, NRT is a solr 4 feature, but solr 4 is not released yet. I understand commit after adding each document is expensive, but the application requires that documents be available after adding to the index. What I don't understand is why new segment files are created so often. Are the commit calls triggering new segment files being created? I don't see this behavior in another environment of the same version of solr. Huy On Mon, Feb 13, 2012 at 11:55 AM, Li Li fancye...@gmail.com wrote: Commit is called after adding each document you should add enough documents and then calling a commit. commit is a cost operation. if you want to get latest feeded documents, you could use NRT On Tue, Feb 14, 2012 at 12:47 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. I seeing solr keeps creating new segment files (1MB files) so often that it triggers segment merge about every one minute. I search the news archive, but could not find any info on this issue. I am indexing about 10 docs of less 2KB each every second. Commit is called after adding each document. Relevant config params are: mergeFactor10/mergeFactor ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs What might be triggering this frequent new segment files creation? Thanks! Huy -- Huy Le Spring Partners, Inc. http://springpadit.com -- Huy Le Spring Partners, Inc. http://springpadit.com
Re: New segment file created too often
as far as I know, there are three situation it will be flushed to a new segment: RAM buffer for posting data structure is used up; added doc numbers are exceeding threshold and there are many deletions in a segment but your configuration seems it is not likely to flush many small segments. ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs On Tue, Feb 14, 2012 at 1:10 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. As I understood it, NRT is a solr 4 feature, but solr 4 is not released yet. I understand commit after adding each document is expensive, but the application requires that documents be available after adding to the index. What I don't understand is why new segment files are created so often. Are the commit calls triggering new segment files being created? I don't see this behavior in another environment of the same version of solr. Huy On Mon, Feb 13, 2012 at 11:55 AM, Li Li fancye...@gmail.com wrote: Commit is called after adding each document you should add enough documents and then calling a commit. commit is a cost operation. if you want to get latest feeded documents, you could use NRT On Tue, Feb 14, 2012 at 12:47 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. I seeing solr keeps creating new segment files (1MB files) so often that it triggers segment merge about every one minute. I search the news archive, but could not find any info on this issue. I am indexing about 10 docs of less 2KB each every second. Commit is called after adding each document. Relevant config params are: mergeFactor10/mergeFactor ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs What might be triggering this frequent new segment files creation? Thanks! Huy -- Huy Le Spring Partners, Inc. http://springpadit.com -- Huy Le Spring Partners, Inc. http://springpadit.com
Re: New segment file created too often
can you post your config file? I found there are 2 places to config ramBufferSizeMB in latest svn of 3.6's example solrconfig.xml. trying to modify them both? indexDefaults useCompoundFilefalse/useCompoundFile mergeFactor10/mergeFactor !-- Sets the amount of RAM that may be used by Lucene indexing for buffering added documents and deletions before they are flushed to the Directory. -- ramBufferSizeMB32/ramBufferSizeMB !-- If both ramBufferSizeMB and maxBufferedDocs is set, then Lucene will flush based on whichever limit is hit first. -- !-- maxBufferedDocs1000/maxBufferedDocs -- maxFieldLength1/maxFieldLength writeLockTimeout1000/writeLockTimeout . !-- termIndexInterval256/termIndexInterval -- /indexDefaults !-- Main Index Values here override the values in the indexDefaults section for the main on disk index. -- mainIndex useCompoundFilefalse/useCompoundFile ramBufferSizeMB32/ramBufferSizeMB mergeFactor10/mergeFactor /mainIndex On Tue, Feb 14, 2012 at 1:10 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. As I understood it, NRT is a solr 4 feature, but solr 4 is not released yet. I understand commit after adding each document is expensive, but the application requires that documents be available after adding to the index. What I don't understand is why new segment files are created so often. Are the commit calls triggering new segment files being created? I don't see this behavior in another environment of the same version of solr. Huy On Mon, Feb 13, 2012 at 11:55 AM, Li Li fancye...@gmail.com wrote: Commit is called after adding each document you should add enough documents and then calling a commit. commit is a cost operation. if you want to get latest feeded documents, you could use NRT On Tue, Feb 14, 2012 at 12:47 AM, Huy Le hu...@springpartners.com wrote: Hi, I am using solr 3.5. I seeing solr keeps creating new segment files (1MB files) so often that it triggers segment merge about every one minute. I search the news archive, but could not find any info on this issue. I am indexing about 10 docs of less 2KB each every second. Commit is called after adding each document. Relevant config params are: mergeFactor10/mergeFactor ramBufferSizeMB1024/ramBufferSizeMB maxMergeDocs2147483647/maxMergeDocs What might be triggering this frequent new segment files creation? Thanks! Huy -- Huy Le Spring Partners, Inc. http://springpadit.com -- Huy Le Spring Partners, Inc. http://springpadit.com
