Re: Delete Duplicates Error

kaveh minooie Wed, 25 Jan 2012 17:03:06 -0800

Hi I think I am havign a simillar problem. this is what i got in thehadoop.log file (nutch log file) after running this command :

bin/nutch crawl urls/ -solr http://solr3:8983/solr/core8 -dir mycrawldir-threads 2 -depth 2 -topN 20


and here is the result( from hadoop.log):

2012-01-25 16:42:37,174 INFO indexer.IndexingFilters - Addingorg.apache.nutch.indexer.anchor.AnchorIndexingFilter2012-01-25 16:42:40,151 INFO indexer.IndexingFilters - Addingorg.apache.nutch.indexer.basic.BasicIndexingFilter2012-01-25 16:42:40,151 INFO anchor.AnchorIndexingFilter - Anchordeduplication is: off2012-01-25 16:42:40,151 INFO indexer.IndexingFilters - Addingorg.apache.nutch.indexer.anchor.AnchorIndexingFilter2012-01-25 16:42:40,167 WARN solr.SolrMappingReader -java.net.MalformedURLException

2012-01-25 16:42:40,341 INFO  solr.SolrWriter - Indexing 21 documents

2012-01-25 16:42:44,137 INFO solr.SolrIndexer - SolrIndexer: finishedat 2012-01-25 16:42:44, elapsed: 00:00:342012-01-25 16:42:44,143 INFO solr.SolrDeleteDuplicates -SolrDeleteDuplicates: starting at 2012-01-25 16:42:442012-01-25 16:42:44,144 INFO solr.SolrDeleteDuplicates -SolrDeleteDuplicates: Solr url: http://solr3:8983/solr/core82012-01-25 16:42:44,295 WARN mapred.FileOutputCommitter - Output pathis null in cleanup

2012-01-25 16:42:44,296 WARN  mapred.LocalJobRunner - job_local_0015
java.lang.NullPointerException

atorg.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrRecord.readSolrDocument(SolrDeleteDuplicates.java:131)atorg.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:271)atorg.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)atorg.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)atorg.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)

        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)

atorg.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)


what is it talking about in this line:

2012-01-25 16:42:44,295 WARN mapred.FileOutputCommitter - Output pathis null in cleanup


what ouput path is it talking about?

(I am running this locally not on hadoop)

On 01/24/2012 05:13 AM, Denis Sinner wrote:

hadoop.log:

2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: content dest: 
content
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: site dest: site
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: title dest: 
teaser
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: boost dest: boost
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: tstamp dest: 
changed
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: tstamp dest: 
created
2012-01-24 14:09:37,370 INFO  solr.SolrWriter - Adding 2 documents
2012-01-24 14:09:38,095 INFO  solr.SolrIndexer - SolrIndexer: finished at 
2012-01-24 14:09:38, elapsed: 00:00:02
2012-01-24 14:09:38,097 INFO  solr.SolrDeleteDuplicates - SolrDeleteDuplicates: 
starting at 2012-01-24 14:09:38
2012-01-24 14:09:38,097 INFO  solr.SolrDeleteDuplicates - SolrDeleteDuplicates: 
Solr url: http://192.168.0.47:8080/solr/core_en/
2012-01-24 14:09:38,457 WARN  mapred.LocalJobRunner - job_local_0010
java.lang.NullPointerException
        at org.apache.hadoop.io.Text.encode(Text.java:388)
        at org.apache.hadoop.io.Text.set(Text.java:178)
        at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:284)
        at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:249)
        at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:192)
        at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:176)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)

Solr (running out of eclipse with jetty):

24.01.2012 14:09:37 org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
        
commit{dir=/Users/dkd-sinner/Documents/solr/SolrTypo3Plugin/solr/typo3cores/data/core_en/index,segFN=segments_p,version=1326882792610,generation=25,filenames=[_1.frq,
 _b.nrm, _b.tvx, _2.tii, _1.fnm, _2.tvx, _2.tvd, _1.tii, _2.tvf, _1.tvx, 
_1.tis, _2.prx, _b.prx, _2.fdt, _2.frq, _b.tis, _2.fdx, _2.fnm, _b.tii, _b.frq, 
_1.prx, _1.fdx, _2.tis, _1.tvf, _b.tvd, _1.fdt, segments_p, _b.fnm, _b.fdt, 
_b.tvf, _1.tvd, _b.fdx, _1.nrm, _2.nrm]
24.01.2012 14:09:37 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1326882792610
24.01.2012 14:09:37 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: 
{add=[045756f6efde46c27a8e1016756bf99cc8153d51/nutch_external/http://www.dkd.de/,
 
5648ab376b909bc402c4ecbf45c26b4546e69f04/nutch_external/http://www.typo3-solr.com/]}
 0 71
24.01.2012 14:09:37 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/update params={wt=javabin&version=2} 
status=0 QTime=71
24.01.2012 14:09:37 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start 
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
24.01.2012 14:09:38 org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
        
commit{dir=/Users/dkd-sinner/Documents/solr/SolrTypo3Plugin/solr/typo3cores/data/core_en/index,segFN=segments_p,version=1326882792610,generation=25,filenames=[_1.frq,
 _b.nrm, _b.tvx, _2.tii, _1.fnm, _2.tvx, _2.tvd, _1.tii, _2.tvf, _1.tvx, 
_1.tis, _2.prx, _b.prx, _2.fdt, _2.frq, _b.tis, _2.fdx, _2.fnm, _b.tii, _b.frq, 
_1.prx, _1.fdx, _2.tis, _1.tvf, _b.tvd, _1.fdt, segments_p, _b.fnm, _b.fdt, 
_b.tvf, _1.tvd, _b.fdx, _1.nrm, _2.nrm]
        
commit{dir=/Users/dkd-sinner/Documents/solr/SolrTypo3Plugin/solr/typo3cores/data/core_en/index,segFN=segments_q,version=1326882792614,generation=26,filenames=[_1.frq,
 _2.tii, _c.tii, _c.fdx, _c.tvx, _1.fnm, _2.tvx, _c.fdt, _2.tvd, _c.tis, 
_c.nrm, _1.tii, _2.tvf, _1.tvx, _1.tis, _2.prx, _c.prx, _2.fdt, _2.frq, _2.fdx, 
_2.fnm, _1.prx, _1.fdx, _2.tis, _1.tvf, _1.fdt, segments_q, _c.tvf, _c.tvd, 
_c.fnm, _1.tvd, _c.frq, _1.nrm, _2.nrm]
24.01.2012 14:09:38 org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO: newest commit = 1326882792614
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher<init>
INFO: Opening Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main
        
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main
        
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main
        
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main
        
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main
        
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=44,cumulative_hits=32,cumulative_hitratio=0.72,cumulative_inserts=22,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main
        
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=44,cumulative_hits=32,cumulative_hitratio=0.72,cumulative_inserts=22,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main
        
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1136,cumulative_hits=618,cumulative_hitratio=0.54,cumulative_inserts=518,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main
        
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1136,cumulative_hits=618,cumulative_hitratio=0.54,cumulative_inserts=518,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
24.01.2012 14:09:38 
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener 
buildSpellIndex
INFO: Building spell index for spell checker: default
24.01.2012 14:09:38 org.apache.solr.core.SolrCore registerSearcher
INFO: [core_en] Registered new searcher Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@3d78cd7b main
        
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=44,cumulative_hits=32,cumulative_hitratio=0.72,cumulative_inserts=22,cumulative_evictions=0}
        
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1136,cumulative_hits=618,cumulative_hitratio=0.54,cumulative_inserts=518,cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {commit=} 0 212
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/update 
params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2} 
status=0 QTime=212
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select 
params={fl=id&wt=javabin&q=*:*&rows=1&version=2} hits=52 status=0 QTime=2
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select 
params={fl=id&wt=javabin&q=*:*&rows=1&version=2} hits=52 status=0 QTime=1
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select 
params={fl=id,boost,tstamp,digest&start=0&q=*:*&wt=javabin&rows=52&version=2} 
hits=52 status=0 QTime=2


--
Kaveh Minooie

www.plutoz.com

Re: Delete Duplicates Error

Reply via email to