Consider giving the MR tasks more RAM, for example via
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar
org.apache.solr.hadoop.MapReduceIndexerTool -D
'mapred.child.java.opts=-Xmx2000m’ ...
Wolfgang.
On May 26, 2014, at 10:48 AM, Costi Muraru <costimur...@gmail.com> wrote:
> Hey Erick,
>
> The job reducers began to die with "Error: Java heap space", after 1h and
> 22 minutes being stucked at ~80%.
>
> I did a few more tests:
>
> Test 1.
> 80,000 documents
> Each document had *20* fields. The field names were* the same *for all the
> documents. Values were different.
> Job status: successful
> Execution time: 33 seconds.
>
> Test 2.
> 80,000 documents
> Each document had *20* fields. The field names were *different* for all the
> documents. Values were also different.
> Job status: successful
> Execution time: 643 seconds.
>
> Test 3.
> 80,000 documents
> Each document had *50* fields. The field names were *the same* for all the
> documents. Values were different.
> Job status: successful
> Execution time: 45.96 seconds.
>
> Test 4.
> 80,000 documents
> Each document had *50* fields. The field names were *different* for all the
> documents. Values were also different.
> Job status: failed
> Execution time: after 1h reducers failed.
> Unfortunately, this is my use case.
>
> My guess is that the reduce time (to perform the merges) depends if the
> field names are the same across the documents. If they are different the
> merge time increases very much. I don't have any knowledge behind the solr
> merge operation, but is it possible that it tries to group the fields with
> the same name across all the documents?
> In the first case, when the field names are the same across documents, the
> number of buckets is equal to the number of unique field names which is 20.
> In the second case, where all the field names are different (my use case),
> it creates a lot more buckets (80k documents * 50 different field names = 4
> million buckets) and the process gets slowed down significantly.
> Is this assumption correct / Is there any way to get around it?
>
> Thanks again for reaching out. Hope this is more clear now.
>
> This is how one of the 80k documents looks like (json format):
> {
> "id" : "442247098240414508034066540706561683636",
> "items" : {
> "IT49597_1180_i" : 76,
> "IT25363_1218_i" : 4,
> "IT12418_1291_i" : 95,
> "IT55979_1051_i" : 31,
> "IT9841_1224_i" : 36,
> "IT40463_1010_i" : 87,
> "IT37932_1346_i" : 11,
> "IT17653_1054_i" : 37,
> "IT59414_1025_i" : 96,
> "IT51080_1133_i" : 5,
> "IT7369_1395_i" : 90,
> "IT59974_1245_i" : 25,
> "IT25374_1345_i" : 75,
> "IT16825_1458_i" : 28,
> "IT56643_1050_i" : 76,
> "IT46274_1398_i" : 50,
> "IT47411_1275_i" : 11,
> "IT2791_1000_i" : 97,
> "IT7708_1053_i" : 96,
> "IT46622_1112_i" : 90,
> "IT47161_1382_i" : 64
> }
> }
>
> Costi
>
>
> On Mon, May 26, 2014 at 7:45 PM, Erick Erickson
> <erickerick...@gmail.com>wrote:
>
>> The MapReduceIndexerTool is really intended for very large data sets,
>> and by today's standards 80K doesn't qualify :).
>>
>> Basically, MRIT creates N sub-indexes, then merges them, which it
>> may to in a tiered fashion. That is, it may merge gen1 to gen2, then
>> merge gen2 to gen3 etc. Which is great when indexing a bazillion
>> documents into 20 shards, but all that copying around may take
>> more time than you really gain for 80K docs.
>>
>> Also be aware that MRIT does NOT update docs with the same ID, this
>> is due to the inherent limitation of the Lucene mergeIndex process.
>>
>> How long is "a long time"? attachments tend to get filtered out, so if you
>> want us to see the graph you might paste it somewhere and provide a link.
>>
>> Best,
>> Erick
>>
>> On Mon, May 26, 2014 at 8:51 AM, Costi Muraru <costimur...@gmail.com>
>> wrote:
>>> Hey guys,
>>>
>>> I'm using the MergeReduceIndexerTool to import data into a SolrCloud
>>> cluster made out of 3 decent machines.
>>> Looking in the JobTracker, I can see that the mapper jobs finish quite
>>> fast. The reduce jobs get to ~80% quite fast as well. It is here where
>>> they get stucked for a long period of time (picture + log attached).
>>> I'm only trying to insert ~80k documents with 10-50 different fields
>>> each. Why is this happening? Am I not setting something correctly? Is
>>> the fact that most of the documents have different field names, or too
>>> many for that matter?
>>> Any tips are gladly appreciated.
>>>
>>> Thanks,
>>> Costi
>>>
>>> From the reduce logs:
>>> 60208 [main] INFO org.apache.solr.update.UpdateHandler - start
>>>
>> commit{,optimize=false,openSearcher=true,waitSearcher=false,expungeDeletes=false,softCommit=false,prepareCommit=false}
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [IW][main]: commit: start
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [IW][main]: commit: enter lock
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [IW][main]: commit: now prepare
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [IW][main]: prepareCommit: flush
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [IW][main]: index before flush
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [DW][main]: main startFullFlush
>>> 60208 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [DW][main]: anyChanges? numDocsInRam=25603 deletes=true
>>> hasTickets:false pendingChangesInFullFlush: false
>>> 60209 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [DWFC][main]: addFlushableState DocumentsWriterPerThread
>>> [pendingDeletes=gen=0 25602 deleted terms (unique count=25602)
>>> bytesUsed=5171604, segment=_0, aborting=false, numDocsInRAM=25603,
>>> deleteQueue=DWDQ: [ generation: 0 ]]
>>> 61542 [main] INFO org.apache.solr.update.LoggingInfoStream -
>>> [DWPT][main]: flush postings as segment _0 numDocs=25603
>>> 61664 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 125115 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 199408 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 271088 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 336754 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 417810 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 479495 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 552357 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 621450 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>> 683173 [Thread-32] INFO org.apache.solr.hadoop.HeartBeater - Issuing
>>> heart beat for 1 threads
>>>
>>> This is the run command I'm using:
>>> hadoop jar
>> /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar
>>> org.apache.solr.hadoop.MapReduceIndexerTool \
>>> --log4j /home/cmuraru/solr/log4j.properties \
>>> --morphline-file morphline.conf \
>>> --output-dir hdfs://nameservice1:8020/tmp/outdir \
>>> --verbose --go-live --zk-host localhost:2181/solr \
>>> --collection collection1 \
>>> hdfs://nameservice1:8020/tmp/indir
>>
