We consistently reproduce this problem on multiple systems configured with
6GB and 12GB of heap space. To quickly reproduce many cases for
troubleshooting we reduced the heap space to 64, 128 and 512MB. With 6 or
12GB configured it takes hours to see the error.
On Fri, Apr 18, 2014 at 5:54 PM, Walter Underwood <wun...@wunderwood.org>wrote:
> I see heap size commands for 128 Meg and 512 Meg. That will certainly run
> out of memory. Why do you think you have 6G of heap with these settings?
>
> –Xmx128m –Xms128m
> –Xmx512m –Xms512m
>
> wunder
>
> On Apr 18, 2014, at 5:15 PM, Candygram For Mongo <
> candygram.for.mo...@gmail.com> wrote:
>
> > I have lots of log files and other files to support this issue (sometimes
> > referenced in the text below) but I am not sure the best way to submit.
> I
> > don't want to overwhelm and I am not sure if this email will accept
> graphs
> > and charts. Please provide direction and I will send them.
> >
> >
> > *Issue Description*
> >
> >
> >
> > We are getting Out Of Memory errors when we try to execute a full import
> > using the Data Import Handler. This error originally occurred on a
> > production environment with a database containing 27 million records.
> Heap
> > memory was configured for 6GB and the server had 32GB of physical memory.
> > We have been able to replicate the error on a local system with 6 million
> > records. We set the memory heap size to 64MB to accelerate the error
> > replication. The indexing process has been failing in different
> scenarios.
> > We have 9 test cases documented. In some of the test cases we increased
> > the heap size to 128MB. In our first test case we set heap memory to
> 512MB
> > which also failed.
> >
> >
> >
> >
> >
> > *Environment Values Used*
> >
> >
> >
> > *SOLR/Lucene version: *4.2.1*
> >
> > *JVM version:
> >
> > Java(TM) SE Runtime Environment (build 1.7.0_07-b11)
> >
> > Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
> >
> > *Indexer startup command:
> >
> > set JVMARGS= -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m
> >
> > java " %JVMARGS% ^
> >
> > -Dcom.sun.management.jmxremote.port=1092 ^
> >
> > -Dcom.sun.management.jmxremote.ssl=false ^
> >
> > -Dcom.sun.management.jmxremote.authenticate=false ^
> >
> > -jar start.jar
> >
> > *SOLR indexing HTTP parameters request:
> >
> > webapp=/solr path=/dataimport
> > params={clean=false&command=full-import&wt=javabin&version=2}
> >
> >
> >
> > The information we use for the database retrieve using the Data Import
> > Handler is as follows:
> >
> >
> >
> > <dataSource
> >
> > name="org_only"
> >
> > type="JdbcDataSource"
> >
> > driver="oracle.jdbc.OracleDriver"
> >
> > url="jdbc:oracle:thin:@{server
> name}:1521:{database
> > name}"
> >
> > user="{username}"
> >
> > password="{password}"
> >
> > readOnly="false"
> >
> > />
> >
> >
> >
> >
> >
> > *The Query (simple, single table)*
> >
> >
> >
> > *select*
> >
> >
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(100)), 'null')*
> >
> > *as SOLR_ID,*
> >
> >
> >
> > *'STU.ACCT_ADDRESS_ALL'*
> >
> > *as SOLR_CATEGORY,*
> >
> >
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.R_ID as varchar2(255)), ' ') as
> > ADDRESSALLRID,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_TYPE as varchar2(255)), ' ') as
> > ADDRESSALLADDRTYPECD,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.LONGITUDE as varchar2(255)), ' ') as
> > ADDRESSALLLONGITUDE,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.LATITUDE as varchar2(255)), ' ') as
> > ADDRESSALLLATITUDE,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.ADDR_NAME as varchar2(255)), ' ') as
> > ADDRESSALLADDRNAME,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.CITY as varchar2(255)), ' ') as
> > ADDRESSALLCITY,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.STATE as varchar2(255)), ' ') as
> > ADDRESSALLSTATE,*
> >
> > *NVL(cast(STU.ACCT_ADDRESS_ALL.EMAIL_ADDR as varchar2(255)), ' ') as
> > ADDRESSALLEMAILADDR *
> >
> >
> >
> > *from STU.ACCT_ADDRESS_ALL*
> >
> >
> >
> > You can see this information in the database.xml file.
> >
> >
> >
> > Our main solrconfig.xml file contains the following differences compared
> to
> > a new downloaded solrconfig.xml
> >
> file<file:///D:/Solr%20Full%20Indexing%20issue/solrconfig%20(default%20content).xml>(the
> > original content).
> >
> >
> >
> > <config>
> >
> > <lib dir="../../../dist/"
> > regex="solr-dataimporthandler-.*\.jar" />
> >
> > <!—Our libraries containing customized filters-->
> >
> > <lib path="../../../../default/lib/common.jar" />
> >
> > <lib path="../../../../default/lib/webapp.jar" />
> >
> > <lib path="../../../../default/lib/commons-pool-1.4.jar" />
> >
> >
> >
> >
> <abortOnConfigurationError>${solr.abortOnConfigurationError:true}</abortOnConfigurationError>
> >
> >
> >
> > <directoryFactory name="DirectoryFactory"
> > class="org.apache.solr.core.StandardDirectoryFactory" />
> >
> >
> >
> > <requestHandler name="/dataimport"
> > class="org.apache.solr.handler.dataimport.DataImportHandler">
> >
> > <lst name="defaults">
> >
> > <str name="config">database.xml</str>
> >
> > </lst>
> >
> > </requestHandler>
> >
> > </config>
> >
> >
> >
> >
> >
> > *Custom Libraries*
> >
> >
> >
> > The common.jar contains a customized TokenFiltersFactory implementation
> > that we use for indexing. They do some special treatment to the fields
> > read from the database. How those classes are used is described in the
> > schema.xml file. The webapp.jar file contains other related classes.
> The
> > commons-pool-1.4.jar is an API from apache used for instances reuse.
> >
> >
> >
> > The logic used in the TokenFiltersFactory is contained in the following
> > files:
> >
> >
> >
> >
>
> ConcatFilterFactory.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/ConcatFilterFactory.java>
> >
> >
>
> ConcatFilter.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/ConcatFilter.java>
> >
> >
>
> MDFilterSchemaFactory.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilterSchemaFactory.java>
> >
> >
>
> MDFilter.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilter.java>
> >
> >
>
> MDFilterPoolObjectFactory.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/MDFilterPoolObjectFactory.java>
> >
> >
>
> NullValueFilterFactory.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/NullValueFilterFactory.java>
> >
> >
>
> NullValueFilter.java<file:///D:/Solr%20Full%20Indexing%20issue/source%20files/NullValueFilter.java>
> >
> >
> >
> > How we use them is described in the schema.xml file.
> >
> >
> >
> > We have been experimenting with the following configuration values:
> >
> >
> >
> > maxIndexingThreads
> >
> > ramBufferSizeMB
> >
> > maxBufferedDocs
> >
> > mergePolicy
> >
> > maxMergeAtOnce
> >
> > segmentsPerTier
> >
> > maxMergedSegmentMB
> >
> > autoCommit
> >
> > maxDocs
> >
> > maxTime
> >
> > autoSoftCommit
> >
> > maxTime
> >
> >
> >
> > Using numerous combinations of these values, the indexing fails.
> >
> >
> >
> >
> >
> > *IMPORTANT NOTE*
> >
> >
> >
> > When we disable all of the copyfield tags contained in the schema.xml
> file,
> > or all but relatively few, the indexing completes successfully (see Test
> > Case 1).
> >
> >
> >
> >
> >
> > *TEST CASES*
> >
> >
> >
> > All of the test cases have been analyzed with the Visual VM tool. All
> SOLR
> > configuration files and indexer log content are in the test case
> > directories included in a zip file. We have included the most relevant
> > screenshots. Test Case 2 is the only one that includes the thread dump.
> >
> >
> >
> >
> >
> > *Test Case 1 *
> >
> >
> >
> > JVM arguments = -XX:MaxPermSize=364m -Xss256K –Xmx512m –Xms512m
> >
> >
> >
> > Results:
> >
> > Indexing status: Completed
> >
> > Time taken: 1:8:32.519
> >
> > Error detail: NO ERROR.
> >
> > Index data directory size = 995 MB
> >
> >
> >
> >
> >
> > *Test Case 2*
> >
> >
> >
> > JVM arguments = -XX:MaxPermSize=364m -Xss256K –Xmx512m –Xms512m
> >
> >
> >
> > Results:
> >
> > Indexing status: failed
> >
> > Time taken: 1:30 hours
> >
> > Error detail: GC overhead limit exceeded (details see
> log
> > file <file:///D:/Solr%20Full%20Indexing%20issue/test%20case%201/log.txt>)
> >
> >
> > Index data directory size = 3.48 GB
> >
> > Memory Dump file :
> >
> heapdump-1397500800627.hprof<file:///D:/Solr%20Full%20Indexing%20issue/test%20case%201/heapdump-1397500800627.hprof>
> >
> >
> >
> > The indexer was running with the javamelody profiling tool configured but
> > the visual VM is where the crash was captured. I mention this because
> > there are some messages in the log referring to this tool. In the
> > following picture, you can see that the CPU processing increases before
> the
> > crash.
> >
> >
> >
> >
> >
> > [image: overview]
> >
> >
> >
> >
> >
> > In the following figure we can see the memory used per thread.
> >
> >
> >
> >
> >
> >
> >
> > [image: thread dump - summary]
> >
> >
> >
> >
> >
> > [image: threads summary]
> >
> >
> >
> >
> >
> > *Test Case 3*
> >
> >
> >
> > JVM arguments = -XX:MaxPermSize=364m -Xss256K –Xmx128m –Xms128m
> >
> >
> >
> > Results:
> >
> > Indexing status: failed (ROLLBACK see in log).
> >
> > Error detail: java.lang.OutOfMemoryError: Java heap
> space
> > (details see log
> > file<file:///D:/Solr%20Full%20Indexing%20issue/test%20case%202/log.txt>)
> >
> >
> > Time taken: 20 minutes
> >
> > Index data directory size = 647MB
> >
> > Memory Dump file :
> >
> heapdump-1397579134605.hprof<file:///D:/Solr%20Full%20Indexing%20issue/test%20case%202/heapdump-1397579134605.hprof>
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 4 (default solrconfig with 1 copyfield tag in the schema.xml)*
> >
> >
> >
> > Results:
> >
> > Indexing status: *Completed successful.*
> >
> > Error detail: No Errors.
> >
> > Time taken: 1:19:54
> >
> > Index data directory size = 929MB
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 5 (default solrconfig with 2 copyfield tags in the
> schema.xml)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Completed successful.
> >
> > Error detail: No Errors.
> >
> > Time taken: 1:8:52.291
> >
> > Index data directory size = 1.81 GB
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 6 (default solrconfig with 3 copyfield tags in the
> schema.xml)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Completed successful.
> >
> > Error detail: No Errors.
> >
> > Time taken: 1:0:52.624
> >
> > Index data directory size = 953 MB
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 7 (default solrconfig with 18 copyfield tags in the
> schema.xml)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Fail (Rollback).
> >
> > Error detail: java.lang.OutOfMemoryError: Java heap
> space.
> >
> > Time taken: 43 minutes approx.
> >
> > Index data directory size = 1.30 GB
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 8 (default solrconfig with 16 copyfield tags in the
> schema.xml)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Completed
> >
> > Error detail: No error.
> >
> > Time taken: 1:10:48.641
> >
> > Index data directory size = 2.11 GB
> >
> >
> >
> >
> >
> >
> >
> > *Test Case 9 (default solrconfig with 15 copyfield tags in the schema.xml
> > but including the percentmatchwordonly+soundex)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Fail (Rolback)
> >
> > Error detail: java.lang.OutOfMemoryError: Java heap
> space
> >
> > Time taken: 1:04 approx.
> >
> > Index data directory size = 1.90 GB
> >
> >
> >
> >
> >
> > *Test Case 9 (default solrconfig with 2 copyfield tags in the schema.xml
> > but including the percentmatchwordonly_abc +
> > percentmatchanyword.soundex_abc)*
> >
> >
> >
> > Results:
> >
> > Indexing status: Completed
> >
> > Error detail: none
> >
> > Time taken: 1:04 approx.
> >
> > Index data directory size = 0.99 GB
> >
> >
> >
> > The graphic is not included because it looks very similar to the others.
> >
> >
> >
> > *Conclusions*
> >
> >
> >
> > There are two threads consuming more memory. We saw these values in the
> > Per Thread Allocations in the Java Visual VM:
> >
> >
> >
> > commitScheduler Thread
> >
> > Lucene Merge Thread
> >
> >
> >
> > Note how the ‘Allocated Bytes/sec’ needed exceeds the configured heap
> > size. We understand that the memory shown in that column doesn’t mean
> that
> > all of that memory exists in the heap but it appears that Solr or Lucene
> > can handle more memory that what is assigned to the process.
> >
> >
> >
> > With 128MB of heap size configured, there were times when the
> > commitScheduler thread was allocating 400MB while at other times the
> Lucene
> > Merge was allocating around 1.5 GB of memory. So why the heap memory
> > problem? At the time the memory error was thrown, both thread requests
> > weren’t at their maximum memory consumption.
>
> --
> Walter Underwood
> wun...@wunderwood.org
>
>
>
>
