Re: Delete Duplicates Error

Thu, 26 Jan 2012 17:43:18 -0800

So first of all thanks to everybody for responding. now here is my situation. the comments before about solrindex-mapping was correct since I didn't have that file at all at the time. ( I was having problem with url and id filed and the fact that they were not multivalued but I found out that url is the default id and u just have to comment to corresponding lines out, so that is working now).
By the way I am using the nightly build and all the logs and my runs have been done using Revision: 1236417 

so when i run this command now:


bin/nutch crawl urls/ -solr http://solr3:8983/solr/core8 -dir mycrawldir 
-threads 1 -depth 2 -topN 10


i get this:







2012-01-26 17:20:32,487 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:20:32,487 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:20:35,455 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:20:35,455 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:20:35,484 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:20:35,485 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:20:35,485 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:20:35,501 INFO  solr.SolrMappingReader - source: cache 
dest: cache
2012-01-26 17:20:35,501 INFO  solr.SolrMappingReader - source: anchor 
dest: anchor
2012-01-26 17:20:35,501 INFO  solr.SolrMappingReader - source: type 
dest: type
2012-01-26 17:20:35,501 INFO  solr.SolrMappingReader - source: 
contentLength dest: contentLength
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
lastModified dest: lastModified
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: date 
dest: date
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: lang 
dest: lang
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
subcollection dest: subcollection
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
plutoz_ranking dest: plutoz_ranking
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
user_ranking dest: user_ranking
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
domain_ranking dest: domain_ranking
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
categories dest: categories
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: author 
dest: author
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: terms 
dest: terms
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
publishedDate dest: publishedDate
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: 
updatedDate dest: updatedDate
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: content 
dest: content
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: site 
dest: site
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: title 
dest: title
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: host 
dest: host
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: segment 
dest: segment
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: boost 
dest: boost
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: digest 
dest: digest
2012-01-26 17:20:35,502 INFO  solr.SolrMappingReader - source: tstamp 
dest: tstamp

2012-01-26 17:20:35,698 INFO  solr.SolrWriter - Indexing 11 documents

2012-01-26 17:20:39,426 INFO  solr.SolrIndexer - SolrIndexer: finished 
at 2012-01-26 17:20:39, elapsed: 00:00:34
2012-01-26 17:20:39,430 INFO  solr.SolrDeleteDuplicates - 
SolrDeleteDuplicates: starting at 2012-01-26 17:20:39
2012-01-26 17:20:39,430 INFO  solr.SolrDeleteDuplicates - 
SolrDeleteDuplicates: Solr url: http://solr3:8983/solr/core8
2012-01-26 17:20:39,584 WARN  mapred.FileOutputCommitter - Output path 
is null in cleanup

2012-01-26 17:20:39,585 WARN  mapred.LocalJobRunner - job_local_0015
java.lang.NullPointerException

	at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrRecord.readSolrDocument(SolrDeleteDuplicates.java:131)
	at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:271)
	at 
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat$1.next(SolrDeleteDuplicates.java:241)
	at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
	at 
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)

        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)









but when i run this command (right after the perviouse one crawldb and 
the segments are the same ones that were generated during the perviouse 
run)



bin/nutch solrindex http://solr3:8983/solr/core8 mycrawldir/crawldb 
mycrawldir/segments/*


it goes through with out any error:







2012-01-26 17:22:58,797 INFO  solr.SolrIndexer - SolrIndexer: starting 
at 2012-01-26 17:22:58
2012-01-26 17:22:59,052 INFO  indexer.IndexerMapReduce - 
IndexerMapReduce: crawldb: mycrawldir/crawldb
2012-01-26 17:22:59,052 INFO  indexer.IndexerMapReduce - 
IndexerMapReduces: adding segment: mycrawldir/segments/20120126171745
2012-01-26 17:22:59,340 INFO  indexer.IndexerMapReduce - 
IndexerMapReduces: adding segment: mycrawldir/segments/20120126171827
2012-01-26 17:22:59,539 WARN  util.NativeCodeLoader - Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
2012-01-26 17:23:00,470 INFO  plugin.PluginRepository - Plugins: looking 
in: /home/kaveh/build/nutch/runtime/local/plugins
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - Plugin 
Auto-activation mode: [true]

2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - Registered Plugins:

2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	the nutch core 
extension points (nutch-extensionpoints)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Basic URL 
Normalizer (urlnormalizer-basic)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Basic Indexing 
Filter (index-basic)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Html Parse 
Plug-in (parse-html)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Http / Https 
Protocol Plug-in (protocol-httpclient)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	HTTP Framework 
(lib-http)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Plutoz Indexing 
Filter (index-plutoz)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	More Indexing 
Filter (index-more)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Regex URL 
Filter (urlfilter-regex)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Pass-through 
URL Normalizer (urlnormalizer-pass)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Regex URL 
Normalizer (urlnormalizer-regex)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	CyberNeko HTML 
Parser (lib-nekohtml)
2012-01-26 17:23:00,723 INFO  plugin.PluginRepository - 	Anchor Indexing 
Filter (index-anchor)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Regex URL 
Filter Framework (lib-regex-filter)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - Registered 
Extension-Points:
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch URL 
Normalizer (org.apache.nutch.net.URLNormalizer)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch Protocol 
(org.apache.nutch.protocol.Protocol)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch Segment 
Merge Filter (org.apache.nutch.segment.SegmentMergeFilter)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch URL 
Filter (org.apache.nutch.net.URLFilter)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch Indexing 
Filter (org.apache.nutch.indexer.IndexingFilter)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	HTML Parse 
Filter (org.apache.nutch.parse.HtmlParseFilter)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch Content 
Parser (org.apache.nutch.parse.Parser)
2012-01-26 17:23:00,724 INFO  plugin.PluginRepository - 	Nutch Scoring 
(org.apache.nutch.scoring.ScoringFilter)
2012-01-26 17:23:00,728 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:00,732 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:01,903 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:01,906 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:01,906 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:03,227 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:03,227 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:03,620 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:03,620 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:03,620 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:06,256 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:06,256 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:06,402 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:06,403 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:06,403 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:09,296 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:09,296 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:09,352 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:09,352 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:09,352 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:12,243 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:12,244 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:12,289 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:12,289 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:12,289 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:15,257 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:15,257 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:15,348 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:15,348 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:15,348 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:18,483 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:18,483 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:18,584 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:18,584 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:18,584 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:21,255 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:21,255 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:21,288 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:21,288 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:21,288 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:24,260 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:24,260 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:24,293 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:24,294 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:24,294 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:27,215 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2012-01-26 17:23:27,215 INFO  indexer.IndexingFilters - Adding 
com.plutoz.indexer.PlutozIndexingFilter
2012-01-26 17:23:27,264 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.more.MoreIndexingFilter
2012-01-26 17:23:27,264 INFO  anchor.AnchorIndexingFilter - Anchor 
deduplication is: off
2012-01-26 17:23:27,264 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: cache 
dest: cache
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: anchor 
dest: anchor
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: type 
dest: type
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: 
contentLength dest: contentLength
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: 
lastModified dest: lastModified
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: date 
dest: date
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: lang 
dest: lang
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: 
subcollection dest: subcollection
2012-01-26 17:23:27,311 INFO  solr.SolrMappingReader - source: 
plutoz_ranking dest: plutoz_ranking
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: 
user_ranking dest: user_ranking
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: 
domain_ranking dest: domain_ranking
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: 
categories dest: categories
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: author 
dest: author
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: terms 
dest: terms
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: 
publishedDate dest: publishedDate
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: 
updatedDate dest: updatedDate
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: content 
dest: content
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: site 
dest: site
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: title 
dest: title
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: host 
dest: host
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: segment 
dest: segment
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: boost 
dest: boost
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: digest 
dest: digest
2012-01-26 17:23:27,312 INFO  solr.SolrMappingReader - source: tstamp 
dest: tstamp

2012-01-26 17:23:27,614 INFO  solr.SolrWriter - Indexing 11 documents

2012-01-26 17:23:30,997 INFO  solr.SolrIndexer - SolrIndexer: finished 
at 2012-01-26 17:23:30, elapsed: 00:00:32





so I don't know what causes that (btw there was nothing relevent to this 
problem in the solr log files) also in both cases the solr gets updated.



Thanks,




On 01/26/2012 04:32 AM, Markus Jelsma wrote:



On Thursday 26 January 2012 12:58:33 Lewis John Mcgibbney wrote:

Hi Kaveh,

I'm not sure if your problem is the same at all.
You're problem stems from the solr mapping configuration used by
AnchorIndexingFilter in the index-anchor plugin.


He? That plugin has nothing to do with Solr mapping or Solr at all.


If this works properly then you should see a list of all of the source -->
destination field mappings,


Mappings are not loaded for deduplication.


this unfortunately is not the case and needs to
be resolved before you can progress.

Maybe once this is sorted you can address the MR NPE


Did anything strange pop up in the Solr logs?

We should get rid of this dedup implementation, it's flawed and seems to break
up everywhere.



hth

On Thu, Jan 26, 2012 at 1:02 AM, kaveh minooie<[email protected]>  wrote:

Hi I think I am havign a simillar problem. this is what i got in the
hadoop.log file (nutch log file) after running this command :


bin/nutch crawl urls/ -solr http://solr3:8983/solr/core8 -dir mycrawldir
-threads 2 -depth 2 -topN 20

and here is the result( from hadoop.log):

2012-01-25 16:42:37,174 INFO  indexer.IndexingFilters - Adding
org.apache.nutch.indexer.**anchor.AnchorIndexingFilter
2012-01-25 16:42:40,151 INFO  indexer.IndexingFilters - Adding
org.apache.nutch.indexer.**basic.BasicIndexingFilter
2012-01-25 16:42:40,151 INFO  anchor.AnchorIndexingFilter - Anchor
deduplication is: off
2012-01-25 16:42:40,151 INFO  indexer.IndexingFilters - Adding
org.apache.nutch.indexer.**anchor.AnchorIndexingFilter
2012-01-25 16:42:40,167 WARN  solr.SolrMappingReader -
java.net.MalformedURLException
2012-01-25 16:42:40,341 INFO  solr.SolrWriter - Indexing 21 documents
2012-01-25 16:42:44,137 INFO  solr.SolrIndexer - SolrIndexer: finished at
2012-01-25 16:42:44, elapsed: 00:00:34
2012-01-25 16:42:44,143 INFO  solr.SolrDeleteDuplicates -
SolrDeleteDuplicates: starting at 2012-01-25 16:42:44
2012-01-25 16:42:44,144 INFO  solr.SolrDeleteDuplicates -
SolrDeleteDuplicates: Solr url: http://solr3:8983/solr/core8
2012-01-25 16:42:44,295 WARN  mapred.FileOutputCommitter - Output path is
null in cleanup
2012-01-25 16:42:44,296 WARN  mapred.LocalJobRunner - job_local_0015
java.lang.NullPointerException

        at org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**

SolrRecord.readSolrDocument(**SolrDeleteDuplicates.java:131)

        at org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**

SolrInputFormat$1.next(**SolrDeleteDuplicates.java:271)

        at org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**

SolrInputFormat$1.next(**SolrDeleteDuplicates.java:241)

        at org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**

moveToNext(MapTask.java:236)

        at org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**

next(MapTask.java:216)

        at org.apache.hadoop.mapred.**MapRunner.run(MapRunner.java:**48)
        at org.apache.hadoop.mapred.**MapTask.runOldMapper(MapTask.**

java:436)

        at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.**LocalJobRunner$Job.run(**

LocalJobRunner.java:212)

what is it talking about in this line:

2012-01-25 16:42:44,295 WARN  mapred.FileOutputCommitter - Output path is
null in cleanup

what ouput path is it talking about?

(I am running this locally not on hadoop)

On 01/24/2012 05:13 AM, Denis Sinner wrote:

hadoop.log:

2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: content
dest: content
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: site
dest: site
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: title
dest: teaser
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: boost
dest: boost
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: tstamp
dest: changed
2012-01-24 14:09:37,156 INFO  solr.SolrMappingReader - source: tstamp
dest: created
2012-01-24 14:09:37,370 INFO  solr.SolrWriter - Adding 2 documents
2012-01-24 14:09:38,095 INFO  solr.SolrIndexer - SolrIndexer: finished
at 2012-01-24 14:09:38, elapsed: 00:00:02
2012-01-24 14:09:38,097 INFO  solr.SolrDeleteDuplicates -
SolrDeleteDuplicates: starting at 2012-01-24 14:09:38
2012-01-24 14:09:38,097 INFO  solr.SolrDeleteDuplicates -
SolrDeleteDuplicates: Solr url:
http://192.168.0.47:8080/solr/**core_en/<http://192.168.0.47:8080/solr/
core_en/>  2012-01-24 14:09:38,457 WARN  mapred.LocalJobRunner -
job_local_0010 java.lang.NullPointerException

        at org.apache.hadoop.io.Text.**encode(Text.java:388)
        at org.apache.hadoop.io.Text.set(**Text.java:178)
        at org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**

SolrInputFormat$1.next(**SolrDeleteDuplicates.java:284)

        at org.apache.nutch.indexer.solr.**SolrDeleteDuplicates$**

SolrInputFormat$1.next(**SolrDeleteDuplicates.java:249)

        at org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**

moveToNext(MapTask.java:192)

        at org.apache.hadoop.mapred.**MapTask$TrackedRecordReader.**

next(MapTask.java:176)

        at org.apache.hadoop.mapred.**MapRunner.run(MapRunner.java:**48)
        at org.apache.hadoop.mapred.**MapTask.runOldMapper(MapTask.**

java:358)

        at org.apache.hadoop.mapred.**MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.**LocalJobRunner$Job.run(**

LocalJobRunner.java:177)

Solr (running out of eclipse with jetty):

24.01.2012 14:09:37 org.apache.solr.core.**SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1

        commit{dir=/Users/dkd-sinner/**Documents/solr/**

SolrTypo3Plugin/solr/**typo3cores/data/core_en/index,**
segFN=segments_p,version=**1326882792610,generation=25,**filenames=[_1.f
rq, _b.nrm, _b.tvx, _2.tii, _1.fnm, _2.tvx, _2.tvd, _1.tii, _2.tvf,
_1.tvx, _1.tis, _2.prx, _b.prx, _2.fdt, _2.frq, _b.tis, _2.fdx, _2.fnm,
_b.tii, _b.frq, _1.prx, _1.fdx, _2.tis, _1.tvf, _b.tvd, _1.fdt,
segments_p, _b.fnm, _b.fdt, _b.tvf, _1.tvd, _b.fdx, _1.nrm, _2.nrm]
24.01.2012 14:09:37 org.apache.solr.core.**SolrDeletionPolicy
updateCommits
INFO: newest commit = 1326882792610
24.01.2012 14:09:37
org.apache.solr.update.**processor.LogUpdateProcessor finish
INFO: {add=[**045756f6efde46c27a8e1016756bf9**9cc8153d51/nutch_external/
http**://www.dkd.de/<http://www.dkd.de/>,
5648ab376b909bc402c4ecbf45c26b
**4546e69f04/nutch_external/http**://www.typo3-solr.com/<http://www.typ
o3-solr.com/>]} 0 71
24.01.2012 14:09:37 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/update params={wt=javabin&version=2}
status=0 QTime=71
24.01.2012 14:09:37 org.apache.solr.update.**DirectUpdateHandler2 commit
INFO: start commit(optimize=false,**waitFlush=true,waitSearcher=**
true,expungeDeletes=false)
24.01.2012 14:09:38 org.apache.solr.core.**SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2

        commit{dir=/Users/dkd-sinner/**Documents/solr/**

SolrTypo3Plugin/solr/**typo3cores/data/core_en/index,**
segFN=segments_p,version=**1326882792610,generation=25,**filenames=[_1.f
rq, _b.nrm, _b.tvx, _2.tii, _1.fnm, _2.tvx, _2.tvd, _1.tii, _2.tvf,
_1.tvx, _1.tis, _2.prx, _b.prx, _2.fdt, _2.frq, _b.tis, _2.fdx, _2.fnm,
_b.tii, _b.frq, _1.prx, _1.fdx, _2.tis, _1.tvf, _b.tvd, _1.fdt,
segments_p, _b.fnm, _b.fdt, _b.tvf, _1.tvd, _b.fdx, _1.nrm, _2.nrm]

        commit{dir=/Users/dkd-sinner/**Documents/solr/**

SolrTypo3Plugin/solr/**typo3cores/data/core_en/index,**
segFN=segments_q,version=**1326882792614,generation=26,**filenames=[_1.f
rq, _2.tii, _c.tii, _c.fdx, _c.tvx, _1.fnm, _2.tvx, _c.fdt, _2.tvd,
_c.tis, _c.nrm, _1.tii, _2.tvf, _1.tvx, _1.tis, _2.prx, _c.prx, _2.fdt,
_2.frq, _2.fdx, _2.fnm, _1.prx, _1.fdx, _2.tis, _1.tvf, _1.fdt,
segments_q, _c.tvf, _c.tvd, _c.fnm, _1.tvd, _c.frq, _1.nrm, _2.nrm]
24.01.2012 14:09:38 org.apache.solr.core.**SolrDeletionPolicy
updateCommits
INFO: newest commit = 1326882792614
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher<init>
INFO: Opening Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.update.**DirectUpdateHandler2 commit
INFO: end_commit_flush
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main

        fieldValueCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=0,cumulative_hits=0,**cumulative_hitratio=0.00,**
cumulative_inserts=0,**cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main

        fieldValueCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=0,cumulative_hits=0,**cumulative_hitratio=0.00,**
cumulative_inserts=0,**cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main

        filterCache{lookups=0,hits=0,**hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=0,**
cumulative_hits=0,cumulative_**hitratio=0.00,cumulative_**
inserts=0,cumulative_**evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main

        filterCache{lookups=0,hits=0,**hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=0,**
cumulative_hits=0,cumulative_**hitratio=0.00,cumulative_**
inserts=0,cumulative_**evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main

        queryResultCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=44,cumulative_hits=32,**cumulative_hitratio=0.72,**
cumulative_inserts=22,**cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main

        queryResultCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=44,cumulative_hits=32,**cumulative_hitratio=0.72,**
cumulative_inserts=22,**cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming Searcher@2a44fec1 main from Searcher@3d78cd7b main

        documentCache{lookups=0,hits=**0,hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=1136,**
cumulative_hits=618,**cumulative_hitratio=0.54,**cumulative_inserts=518,
* *cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher warm
INFO: autowarming result for Searcher@2a44fec1 main

        documentCache{lookups=0,hits=**0,hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=1136,**
cumulative_hits=618,**cumulative_hitratio=0.54,**cumulative_inserts=518,
* *cumulative_evictions=0}
24.01.2012 14:09:38 org.apache.solr.core.**QuerySenderListener
newSearcher
INFO: QuerySenderListener sending requests to Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.core.**QuerySenderListener
newSearcher
INFO: QuerySenderListener done.
24.01.2012 14:09:38 org.apache.solr.handler.**
component.SpellCheckComponent$**SpellCheckerListener buildSpellIndex
INFO: Building spell index for spell checker: default
24.01.2012 14:09:38 org.apache.solr.core.SolrCore registerSearcher
INFO: [core_en] Registered new searcher Searcher@2a44fec1 main
24.01.2012 14:09:38 org.apache.solr.search.**SolrIndexSearcher close
INFO: Closing Searcher@3d78cd7b main

        fieldValueCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=0,cumulative_hits=0,**cumulative_hitratio=0.00,**
cumulative_inserts=0,**cumulative_evictions=0}

        filterCache{lookups=0,hits=0,**hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=0,**
cumulative_hits=0,cumulative_**hitratio=0.00,cumulative_**
inserts=0,cumulative_**evictions=0}

        queryResultCache{lookups=0,**hits=0,hitratio=0.00,inserts=**

0,evictions=0,size=0,**warmupTime=0,cumulative_**
lookups=44,cumulative_hits=32,**cumulative_hitratio=0.72,**
cumulative_inserts=22,**cumulative_evictions=0}

        documentCache{lookups=0,hits=**0,hitratio=0.00,inserts=0,**

evictions=0,size=0,warmupTime=**0,cumulative_lookups=1136,**
cumulative_hits=618,**cumulative_hitratio=0.54,**cumulative_inserts=518,
* *cumulative_evictions=0}
24.01.2012 14:09:38
org.apache.solr.update.**processor.LogUpdateProcessor finish
INFO: {commit=} 0 212
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/update params={waitSearcher=true&**
waitFlush=true&wt=javabin&**commit=true&version=2} status=0 QTime=212
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select
params={fl=id&wt=javabin&q=*:** *&rows=1&version=2} hits=52 status=0
QTime=2
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select
params={fl=id&wt=javabin&q=*:** *&rows=1&version=2} hits=52 status=0
QTime=1
24.01.2012 14:09:38 org.apache.solr.core.SolrCore execute
INFO: [core_en] webapp=/solr path=/select params={fl=id,boost,tstamp,**
digest&start=0&q=*:*&wt=**javabin&rows=52&version=2} hits=52 status=0
QTime=2


--
Kaveh Minooie

www.plutoz.com




--
Kaveh Minooie

www.plutoz.com






















					Reply via email to