I've brought up the logs particular to a recent crawl where the error
occurred, it is posted below. I don't see anything that really stands out.
On further consideration it seems as though on repeated crawls, it is
updating the webpage crawls and nothing is being duplicated. I think it is
choking when it looks at the other records in the index, because the nutch
crawls are being indexed against non-webpage entities.
Might it be that it is working fine, it's just delivering the error because
it can't dedupe things that did not originate from Nutch (and hence don't
have url fields associated with them?)
What's more, are there any problems indexing nutch records alongside
non-nutch records in a solr core? I am not really seeing any other than the
odd error message.
------------------------------------------------------------------------------------------------------
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
------------------------------------------------------------------------------------------------------
Jul 31, 2013 11:26:31 AM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/opt/solr/core0/data/index,segFN=segments_8r,version=1370615988315,generation=315,filenames=[_97.fdt,
_99.fdx, _98.tis, _99.nrm, _97.fnm, _98.tii, _99.prx, _97.fdx, _99.fdt,
_99.frq, _96.prx, _96.frq, _98.nrm, _96.fnm, _97.frq, segments_8r, _99.tis,
_96.tii, _99.tii, _96.tis, _99.fnm, _96.fdx, _98.fdt, _97.tis, _97.nrm,
_96.nrm, _98.fnm, _97.prx, _96.fdt, _98.prx, _97.tii, _98.fdx, _98.frq]
Jul 31, 2013 11:26:31 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1370615988315
Jul 31, 2013 11:26:46 AM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[http://www.oursite.com/, http://www.oursite.com/about/,
http://www.oursite.com/aboutus/,
http://www.oursite.com/aboutus/becomingascanner.htm,
http://www.oursite.com/aboutus/whatisscanning.htm,
http://www.oursite.com/aboutus/yourspace.htm,
http://www.oursite.com/advertise/, http://www.oursite.com/advocacy/, ...
(158 adds)]} 0 15127
Jul 31, 2013 11:26:46 AM org.apache.solr.core.SolrCore execute
INFO: [core0] webapp=/solr path=/update params={wt=javabin&version=2}
status=0 QTime=15127
Jul 31, 2013 11:26:46 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrDeletionPolicy onCommit
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=/opt/solr/core0/data/index,segFN=segments_8r,version=1370615988315,generation=315,filenames=[_97.fdt,
_99.fdx, _98.tis, _99.nrm, _97.fnm, _98.tii, _99.prx, _97.fdx, _99.fdt,
_99.frq, _96.prx, _96.frq, _98.nrm, _96.fnm, _97.frq, segments_8r, _99.tis,
_96.tii, _99.tii, _96.tis, _99.fnm, _96.fdx, _98.fdt, _97.tis, _97.nrm,
_96.nrm, _98.fnm, _97.prx, _96.fdt, _98.prx, _97.tii, _98.fdx, _98.frq]
commit{dir=/opt/solr/core0/data/index,segFN=segments_8s,version=1370615988320,generation=316,filenames=[_97.fdt,
_98.tis, _99.nrm, _97.fnm, _98.tii, _9b.tii, _9a.tii, _97.fdx, _9a.tis,
_96.prx, _96.frq, _96.fnm, _9b.tis, _97.frq, _9a.prx, _9a.fdx, segments_8s,
_9b.nrm, _96.tii, _96.tis, _96.fdx, _9b.fdx, _98.fdt, _9a.nrm, _9b.fdt,
_98.fnm, _98.prx, _96.fdt, _98.fdx, _98.frq, _9a.fdt, _99.fdx, _9b.fnm,
_99.prx, _99.fdt, _99.frq, _98.nrm, _99.tis, _9a.fnm, _98_1.del, _99.tii,
_99_1.del, _9a.frq, _99.fnm, _9b.prx, _97.nrm, _97.tis, _96.nrm, _9b.frq,
_97.prx, _97.tii]
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1370615988320
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher <init>
INFO: Opening Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
queryResultCache{lookups=68,hits=56,hitratio=0.82,inserts=13,evictions=0,size=13,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@1052a2e3 main from Searcher@5484ff20 main
documentCache{lookups=3040,hits=2992,hitratio=0.98,inserts=50,evictions=0,size=50,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1052a2e3 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
Jul 31, 2013 11:26:49 AM org.apache.solr.core.QuerySenderListener
newSearcher
INFO: QuerySenderListener sending requests to Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.core.QuerySenderListener
newSearcher
INFO: QuerySenderListener done.
Jul 31, 2013 11:26:49 AM org.apache.solr.core.SolrCore registerSearcher
INFO: [core0] Registered new searcher Searcher@1052a2e3 main
Jul 31, 2013 11:26:49 AM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@5484ff20 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1,evictions=0,size=1,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=1,cumulative_evictions=0}
queryResultCache{lookups=68,hits=56,hitratio=0.82,inserts=13,evictions=0,size=13,warmupTime=0,cumulative_lookups=68,cumulative_hits=56,cumulative_hitratio=0.82,cumulative_inserts=12,cumulative_evictions=0}
documentCache{lookups=3040,hits=2992,hitratio=0.98,inserts=50,evictions=0,size=50,warmupTime=0,cumulative_lookups=3040,cumulative_hits=2992,cumulative_hitratio=0.98,cumulative_inserts=48,cumulative_evictions=0}
Jul 31, 2013 11:26:49 AM org.apache.solr.update.processor.LogUpdateProcessor
finish
------------------------------------------------------------------------------------------------------
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
------------------------------------------------------------------------------------------------------
--
View this message in context:
http://lucene.472066.n3.nabble.com/Deleting-Duplicates-works-fine-on-one-solr-core-but-not-on-antother-Nutch-1-5-tp4080931p4081671.html
Sent from the Nutch - User mailing list archive at Nabble.com.
