Re: Discrepancy between Full import and Delta import query
On Tue, Feb 17, 2015 at 8:21 PM, Aniket Bhoi wrote:
> Hi Folks,
>
> I am running Solr 3.4 and using DIH for importing data from a SQL server
> backend.
>
> The query for Full import and Delta import is the same ie both pull the
> same data.
>
> Full and Delta import query:
>
> SELECT KB_ENTRY.ADDITIONAL_INFO ,KB_ENTRY.KNOWLEDGE_REF
> ID,SU_ENTITY_TYPE.REF ENTRY_TYPE_REF,KB_ENTRY.PROFILE_REF,
> KB_ENTRY.ITEM_REF, KB_ENTRY.TITLE, KB_ENTRY.ABSTRACT, KB_ENTRY.SOLUTION,
> KB_ENTRY.SOLUTION_HTML, KB_ENTRY.FREE_TEXT, KB_ENTRY.DATE_UPDATED,
> KB_ENTRY.STATUS_REF, KB_ENTRY.CALL_NUMBER, SU_ENTITY_TYPE.DISPLAY
> ENTRY_TYPE, KB_PROFILE.NAME PROFILE_TYPE, AR_PRIMARY_ASSET.ASSET_REF
> SERVICE_TYPE, AR_PERSON.FULL_NAME CONTRIBUTOR, IN_SYS_SOURCE.NAME SOURCE,
> KB_ENTRY_STATUS.NAME STATUS,(SELECT COUNT (CL_KB_REFER.CALL_NUMBER) FROM
> CL_KB_REFER WHERE CL_KB_REFER.ARTICLE_REF = KB_ENTRY.KNOWLEDGE_REF)
> LINK_RATE FROM KB_ENTRY, SU_ENTITY_TYPE, KB_PROFILE, AR_PRIMARY_ASSET,
> AR_PERSON, IN_SYS_SOURCE, KB_ENTRY_STATUS WHERE KB_ENTRY.PARTITION = 1 AND
> KB_ENTRY.STATUS = 'A' AND AR_PERSON.OFFICER_IND = 1 AND
> KB_ENTRY.CREATED_BY_REF = AR_PERSON.REF AND KB_ENTRY.SOURCE =
> IN_SYS_SOURCE.REF AND KB_ENTRY.STATUS_REF = KB_ENTRY_STATUS.REF AND
> KB_ENTRY_STATUS.STATUS = 'A' AND KB_ENTRY.PROFILE_REF = KB_PROFILE.REF AND
> KB_ENTRY.ITEM_REF = AR_PRIMARY_ASSET.ITEM_REF AND KB_ENTRY.ENTITY_TYPE =
> SU_ENTITY_TYPE.REF AND KB_ENTRY.KNOWLEDGE_REF='${dataimporter.delta.ID}'"
>
>
> Delta query:select KNOWLEDGE_REF as ID from KB_ENTRY where (DATE_UPDATED
> > '${dataimporter.last_index_time}' OR DATE_CREATED >
> '${dataimporter.last_index_time}')">
>
>
> The Problem here is that When I run the full Import ,everything works fine
> and all the feilds .data are displayed fine in the search
>
> However When I run Delta import,for some records the ENTRY_TYPE field is
> not returned from the database,
>
> Let me illustrate it with an example:
>
> Search result After running Full Import:
>
> Record Name:John Doe
> Entry ID:500
> Entry Type:Worker
>
> Search result after running Delta import:
>
> Record Name:John Doe
> Entry ID:500
> Entry Type:
>
>
> FYI:I have run the Full and Delta import queries(Though both are the same)
> on the SQL Server IDE and both return The Entry Type feild correctly.
>
> Not sure why the entry Type feild vanishes from Solr when Delta import is
> run.
>
> Any idea why this would happen.
>
> Thanks,
>
> Aniket
>
>
Hi folks,
Anyone with any luck or knowledge on this?
Regards
Aniket
Discrepancy between Full import and Delta import query
Hi Folks,
I am running Solr 3.4 and using DIH for importing data from a SQL server
backend.
The query for Full import and Delta import is the same ie both pull the
same data.
Full and Delta import query:
SELECT KB_ENTRY.ADDITIONAL_INFO ,KB_ENTRY.KNOWLEDGE_REF
ID,SU_ENTITY_TYPE.REF ENTRY_TYPE_REF,KB_ENTRY.PROFILE_REF,
KB_ENTRY.ITEM_REF, KB_ENTRY.TITLE, KB_ENTRY.ABSTRACT, KB_ENTRY.SOLUTION,
KB_ENTRY.SOLUTION_HTML, KB_ENTRY.FREE_TEXT, KB_ENTRY.DATE_UPDATED,
KB_ENTRY.STATUS_REF, KB_ENTRY.CALL_NUMBER, SU_ENTITY_TYPE.DISPLAY
ENTRY_TYPE, KB_PROFILE.NAME PROFILE_TYPE, AR_PRIMARY_ASSET.ASSET_REF
SERVICE_TYPE, AR_PERSON.FULL_NAME CONTRIBUTOR, IN_SYS_SOURCE.NAME SOURCE,
KB_ENTRY_STATUS.NAME STATUS,(SELECT COUNT (CL_KB_REFER.CALL_NUMBER) FROM
CL_KB_REFER WHERE CL_KB_REFER.ARTICLE_REF = KB_ENTRY.KNOWLEDGE_REF)
LINK_RATE FROM KB_ENTRY, SU_ENTITY_TYPE, KB_PROFILE, AR_PRIMARY_ASSET,
AR_PERSON, IN_SYS_SOURCE, KB_ENTRY_STATUS WHERE KB_ENTRY.PARTITION = 1 AND
KB_ENTRY.STATUS = 'A' AND AR_PERSON.OFFICER_IND = 1 AND
KB_ENTRY.CREATED_BY_REF = AR_PERSON.REF AND KB_ENTRY.SOURCE =
IN_SYS_SOURCE.REF AND KB_ENTRY.STATUS_REF = KB_ENTRY_STATUS.REF AND
KB_ENTRY_STATUS.STATUS = 'A' AND KB_ENTRY.PROFILE_REF = KB_PROFILE.REF AND
KB_ENTRY.ITEM_REF = AR_PRIMARY_ASSET.ITEM_REF AND KB_ENTRY.ENTITY_TYPE =
SU_ENTITY_TYPE.REF AND KB_ENTRY.KNOWLEDGE_REF='${dataimporter.delta.ID}'"
Delta query:select KNOWLEDGE_REF as ID from KB_ENTRY where (DATE_UPDATED
> '${dataimporter.last_index_time}' OR DATE_CREATED >
'${dataimporter.last_index_time}')">
The Problem here is that When I run the full Import ,everything works fine
and all the feilds .data are displayed fine in the search
However When I run Delta import,for some records the ENTRY_TYPE field is
not returned from the database,
Let me illustrate it with an example:
Search result After running Full Import:
Record Name:John Doe
Entry ID:500
Entry Type:Worker
Search result after running Delta import:
Record Name:John Doe
Entry ID:500
Entry Type:
FYI:I have run the Full and Delta import queries(Though both are the same)
on the SQL Server IDE and both return The Entry Type feild correctly.
Not sure why the entry Type feild vanishes from Solr when Delta import is
run.
Any idea why this would happen.
Thanks,
Aniket
Re: Solr Full Import frozen after indexing a fixed number of records
On Sun, Jul 27, 2014 at 12:28 PM, Gora Mohanty wrote: > On 27 July 2014 12:13, Aniket Bhoi wrote: > > > On Fri, Jul 25, 2014 at 8:32 PM, Aniket Bhoi > > wrote: > > > > > I have Apache Solr,hosted on my apache Tomcat Server with SQLServer > > > Backend. > > > > [...] > > > > After I run a full import,Indexing proceeds sucessfully,but seems to > > > freeze everytime after fetching fixed number of records.What I mean is > > > after it fetches 10730 records it just freezes and doesnt process any > > more. > > > > > > Excerpt from dataimport.xml: > > > > > > > > > 0:15:31.959 > > > 0 > > > *10730* > > > 3579 > > > 0 > > > 2014-07-25 10:44:39 > > > > > > This seems to happen everytime. > > > > > > I checked the tomcatlog.Following is the excerpt when Solr freezes: > > > > > > INFO: Generating record for Unique ID :null attachment Ref:null > > > parent ref :nullexecuted by thread:25 > > > > [...] > > Something is wrong with your DIH config file: You seem to be getting null > for a document unique ID. Please share the file with us. > > Regards, > Gora > Hi, The thing is that I have 3 Solr instances deployed across Dev ,QA and Production.They have the exact same configuration files but point to different databases.The DIH config is the same across all 2 instances.It is only in QA that this issue occurs though.Thoughts on this? Regards, Aniket
Re: Solr Full Import frozen after indexing a fixed number of records
On Fri, Jul 25, 2014 at 8:32 PM, Aniket Bhoi wrote:
> I have Apache Solr,hosted on my apache Tomcat Server with SQLServer
> Backend.
>
>
> Details:
>
> *Solr Version:*
> Solr Specification Version: 3.4.0.2012.01.23.14.08.01
> Solr Implementation Version: 3.4
> Lucene Specification Version: 3.4
> Lucene Implementation Version: 3.4
>
> *Tomcat version:*
> Apache Tomcat/6.0.18
>
> *OS details:*
> SUSE Linux Enterprise Server 11 (x86_64)
>
> After I run a full import,Indexing proceeds sucessfully,but seems to
> freeze everytime after fetching fixed number of records.What I mean is
> after it fetches 10730 records it just freezes and doesnt process any more.
>
> Excerpt from dataimport.xml:
>
>
> 0:15:31.959
> 0
> *10730*
> 3579
> 0
> 2014-07-25 10:44:39
>
> This seems to happen everytime.
>
> I checked the tomcatlog.Following is the excerpt when Solr freezes:
>
> INFO: Generating record for Unique ID :null attachment Ref:null
> parent ref :nullexecuted by thread:25
> Jul 25, 2014 10:53:31 AM
> org.apache.solr.update.processor.LogUpdateProcessor processAdd
> FINE: add AH_12345
> Jul 25, 2014 10:53:31 AM
> org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
> INFO: Generating record for Unique ID :null attachment Ref:null
> parent ref :nullexecuted by thread:26
> Jul 25, 2014 10:53:31 AM
> org.apache.solr.update.processor.LogUpdateProcessor processAdd
> FINE: add AH_23451
> Jul 25, 2014 10:53:34 AM org.apache.solr.core.SolrCore execute
> INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
> Jul 25, 2014 10:53:36 AM org.apache.solr.core.SolrCore execute
> INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
> Jul 25, 2014 10:53:38 AM org.apache.solr.core.SolrCore execute
> INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
>
> Help appreciated.
>
> Regards,
>
> Aniket
>
>
>
Hi,
Anyone??..Help needed folks,
Regards,
Aniket
Solr Full Import frozen after indexing a fixed number of records
I have Apache Solr,hosted on my apache Tomcat Server with SQLServer Backend.
Details:
*Solr Version:*
Solr Specification Version: 3.4.0.2012.01.23.14.08.01
Solr Implementation Version: 3.4
Lucene Specification Version: 3.4
Lucene Implementation Version: 3.4
*Tomcat version:*
Apache Tomcat/6.0.18
*OS details:*
SUSE Linux Enterprise Server 11 (x86_64)
After I run a full import,Indexing proceeds sucessfully,but seems to freeze
everytime after fetching fixed number of records.What I mean is after it
fetches 10730 records it just freezes and doesnt process any more.
Excerpt from dataimport.xml:
0:15:31.959
0
*10730*
3579
0
2014-07-25 10:44:39
This seems to happen everytime.
I checked the tomcatlog.Following is the excerpt when Solr freezes:
INFO: Generating record for Unique ID :null attachment Ref:null
parent ref :nullexecuted by thread:25
Jul 25, 2014 10:53:31 AM
org.apache.solr.update.processor.LogUpdateProcessor processAdd
FINE: add AH_12345
Jul 25, 2014 10:53:31 AM
org.apache.solr.handler.dataimport.DocBuilder$EntityRunner runAThread
INFO: Generating record for Unique ID :null attachment Ref:null
parent ref :nullexecuted by thread:26
Jul 25, 2014 10:53:31 AM
org.apache.solr.update.processor.LogUpdateProcessor processAdd
FINE: add AH_23451
Jul 25, 2014 10:53:34 AM org.apache.solr.core.SolrCore execute
INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
Jul 25, 2014 10:53:36 AM org.apache.solr.core.SolrCore execute
INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
Jul 25, 2014 10:53:38 AM org.apache.solr.core.SolrCore execute
INFO: [calls] webapp=/solr path=/dataimport params={} status=0 QTime=0
Help appreciated.
Regards,
Aniket
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
On Tue, Jun 3, 2014 at 6:28 PM, Shawn Heisey wrote: > On 6/3/2014 3:04 AM, Aniket Bhoi wrote: > > I changed the value of removeAbandoned to false,this time the indexing > > failed due to a different exception: > > > > > Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection > > reset > > This is really the same error, but now the dataimport handler actually > sees the connection reset, rather than a higher level problem. > > Something somewhere closed the connection. If you have increased > maxMergeCount to at least 6 and turned off removeAbandoned, then I have > no idea what is happening. Perhaps the SQL Server itself has a very low > inactivity timeout value. > > Thanks, > Shawn > > I changed the indexing query in dataconfig to index data for the last 6months.That seemed to worked.Why is then failing if I index the whole data load.Something to do with the same stale connection being used in the connection pool or are the threads not sufficient ??.
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
On Sun, Jun 1, 2014 at 10:55 PM, Shawn Heisey wrote: > On 5/31/2014 1:54 PM, Aniket Bhoi wrote: > > Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The result > >>> set is closed. > > I still think this is an indication of the source of the problem. > Something closed the connection to your SQL server before Solr was done > with it. That could have been JDBC itself, or it might have been > something else. I think I can safely say that it wasn't Solr. > > Your stacktrace indicates that Tomcat's database connection pooling is > active. Tomcat's connection pooling has a feature for dropping > abandoned connections. If you have enabled this feature, this could be > the problem. > > Abandoned connection dropping (when it is enabled) happens by default 60 > seconds after the connection is established. If you have database > connections that last longer than 60 seconds (which Solr's dataimport is > very likely to do), you need to increase removeAbandonedTimeout to > something larger than the longest time an import is likely to last -- or > disable removeAbandoned entirely. The latter is probably a better option. > > http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html > > Thanks, > Shawn > > I changed the value of removeAbandoned to false,this time the indexing failed due to a different exception: SEVERE: Exception while processing: srch_call document : null:org.apache.solr.handler.dataimport.DataImportHandlerException: Error reading data from database at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.getARow(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$600(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.next(Unknown Source) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(Unknown Source) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.runAThread(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.access$000(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner$1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) *Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset* at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:170) at com.microsoft.sqlserver.jdbc.SimpleInputStream.getBytes(SimpleInputStream.java:350) at com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:419) at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:2007) at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:175) at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:113) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:1982) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:1967) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getObject(SQLServerResultSet.java:2256) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.getObject(SQLServerResultSet.java:2265) at org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.getObject(DelegatingResultSet.java:295) ... 13 more
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
still awaiting a response from someone
On Tue, May 27, 2014 at 1:35 PM, Aniket Bhoi wrote:
>
>
>
> On Mon, May 26, 2014 at 4:14 PM, Aniket Bhoi
> wrote:
>
>> Another thing I have noted is that the exception always follows a commit
>> operation.Log excerpt below:
>>
>> INFO: SolrDeletionPolicy.onCommit: commits:num=2
>> commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qt,version=1347458723267,generation=3557,filenames=[_3z9.tii,
>> _3z3.fnm, _3z9.nrm, _3za.prx, _3z9.fdt, _3z9.fnm, _3z9.fdx, _3z3.frq,
>> _3za.nrm, segments_2qt, _3z3.fdx, _3z9.prx, _3z3.fdt, _3za.fdx, _3z9.frq,
>> _3z3.prx, _3za.fdt, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm, _3z9.tis,
>> _3za.tii, _3za.frq, _3z3.tis]
>>
>> commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qu,version=1347458723269,generation=3558,filenames=[_3zb.fdt,
>> _3z9.tii, _3z3.fnm, _3z9.nrm, _3zb.tii, _3zb.tis, _3zb.fdx, _3za.prx,
>> _3z9.fdt, _3z9.fnm, _3z9.fdx, _3zb.frq, _3z3.frq, _3za.nrm, segments_2qu,
>> _3z3.fdx, _3zb.prx, _3z9.prx, _3zb.fnm, _3z3.fdt, _3za.fdx, _3z9.frq,
>> _3z3.prx, _3za.fdt, _3zb.nrm, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm,
>> _3z9.tis, _3za.tii, _3za.frq, _3z3.tis]
>> May 24, 2014 5:49:05 AM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1347458723269
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher
>> INFO: Opening Searcher@423dbcca main
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>>
>> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for Searcher@423dbcca main
>>
>> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>>
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for Searcher@423dbcca main
>>
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>>
>> queryResultCache{lookups=1,hits=1,hitratio=1.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for Searcher@423dbcca main
>>
>> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>>
>> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=40,evictions=0,size=40,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for Searcher@423dbcca main
>>
>> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>> May 24, 2014 5:49:05 AM org.apache.solr.core.QuerySenderListener
>> newSearcher
>> INFO: QuerySenderListener sending requests to Searcher@423dbcca main
>> May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore execute
>> INFO: [calls] webapp=null path=null
>> params={start=0&event=newSearcher&q=*:*&rows=20} hits=40028 status=0
>> QTime=2
>> May 24, 2014 5:49:05 AM org.apache.solr.update.DirectUpdateHandler2 commit
>> INFO: end_commit_flush
>> May 24, 2014 5:
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
On Mon, May 26, 2014 at 4:14 PM, Aniket Bhoi wrote:
> Another thing I have noted is that the exception always follows a commit
> operation.Log excerpt below:
>
> INFO: SolrDeletionPolicy.onCommit: commits:num=2
> commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qt,version=1347458723267,generation=3557,filenames=[_3z9.tii,
> _3z3.fnm, _3z9.nrm, _3za.prx, _3z9.fdt, _3z9.fnm, _3z9.fdx, _3z3.frq,
> _3za.nrm, segments_2qt, _3z3.fdx, _3z9.prx, _3z3.fdt, _3za.fdx, _3z9.frq,
> _3z3.prx, _3za.fdt, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm, _3z9.tis,
> _3za.tii, _3za.frq, _3z3.tis]
>
> commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qu,version=1347458723269,generation=3558,filenames=[_3zb.fdt,
> _3z9.tii, _3z3.fnm, _3z9.nrm, _3zb.tii, _3zb.tis, _3zb.fdx, _3za.prx,
> _3z9.fdt, _3z9.fnm, _3z9.fdx, _3zb.frq, _3z3.frq, _3za.nrm, segments_2qu,
> _3z3.fdx, _3zb.prx, _3z9.prx, _3zb.fnm, _3z3.fdt, _3za.fdx, _3z9.frq,
> _3z3.prx, _3za.fdt, _3zb.nrm, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm,
> _3z9.tis, _3za.tii, _3za.frq, _3z3.tis]
> May 24, 2014 5:49:05 AM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1347458723269
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher
> INFO: Opening Searcher@423dbcca main
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming result for Searcher@423dbcca main
>
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming result for Searcher@423dbcca main
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>
> queryResultCache{lookups=1,hits=1,hitratio=1.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming result for Searcher@423dbcca main
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=40,evictions=0,size=40,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
> INFO: autowarming result for Searcher@423dbcca main
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> May 24, 2014 5:49:05 AM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener sending requests to Searcher@423dbcca main
> May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore execute
> INFO: [calls] webapp=null path=null
> params={start=0&event=newSearcher&q=*:*&rows=20} hits=40028 status=0
> QTime=2
> May 24, 2014 5:49:05 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: end_commit_flush
> May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore execute
> INFO: [calls] webapp=null path=null
> params={start=0&event=newSearcher&q=banking&rows=20} hits=636 status=0
> QTime=3
> May 24, 2014 5:49:05 AM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener done.
> May 24, 2014 5:49:05 AM
> org.apache.solr.handler
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
Another thing I have noted is that the exception always follows a commit
operation.Log excerpt below:
INFO: SolrDeletionPolicy.onCommit: commits:num=2
commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qt,version=1347458723267,generation=3557,filenames=[_3z9.tii,
_3z3.fnm, _3z9.nrm, _3za.prx, _3z9.fdt, _3z9.fnm, _3z9.fdx, _3z3.frq,
_3za.nrm, segments_2qt, _3z3.fdx, _3z9.prx, _3z3.fdt, _3za.fdx, _3z9.frq,
_3z3.prx, _3za.fdt, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm, _3z9.tis,
_3za.tii, _3za.frq, _3z3.tis]
commit{dir=/opt/solr/cores/calls/data/index,segFN=segments_2qu,version=1347458723269,generation=3558,filenames=[_3zb.fdt,
_3z9.tii, _3z3.fnm, _3z9.nrm, _3zb.tii, _3zb.tis, _3zb.fdx, _3za.prx,
_3z9.fdt, _3z9.fnm, _3z9.fdx, _3zb.frq, _3z3.frq, _3za.nrm, segments_2qu,
_3z3.fdx, _3zb.prx, _3z9.prx, _3zb.fnm, _3z3.fdt, _3za.fdx, _3z9.frq,
_3z3.prx, _3za.fdt, _3zb.nrm, _3z3.tii, _3za.tis, _3za.fnm, _3z3.nrm,
_3z9.tis, _3za.tii, _3za.frq, _3z3.tis]
May 24, 2014 5:49:05 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1347458723269
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher
INFO: Opening Searcher@423dbcca main
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@423dbcca main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@423dbcca main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
queryResultCache{lookups=1,hits=1,hitratio=1.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@423dbcca main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=2,cumulative_lookups=47,cumulative_hits=46,cumulative_hitratio=0.97,cumulative_inserts=1,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@423dbcca main from Searcher@19c19869 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=40,evictions=0,size=40,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@423dbcca main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
May 24, 2014 5:49:05 AM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@423dbcca main
May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore execute
INFO: [calls] webapp=null path=null
params={start=0&event=newSearcher&q=*:*&rows=20} hits=40028 status=0
QTime=2
May 24, 2014 5:49:05 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore execute
INFO: [calls] webapp=null path=null
params={start=0&event=newSearcher&q=banking&rows=20} hits=636 status=0
QTime=3
May 24, 2014 5:49:05 AM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
May 24, 2014 5:49:05 AM
org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
newSearcher
INFO: Index is not optimized therefore skipping building spell check index
for: default
May 24, 2014 5:49:05 AM org.apache.solr.core.SolrCore registerSearcher
INFO: [calls] Registered new searcher Searcher@423dbcca main
May 24, 2014 5:49:05 AM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@19c19869 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,ins
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
On Thu, May 22, 2014 at 9:31 PM, Shawn Heisey wrote: > On 5/22/2014 8:31 AM, Aniket Bhoi wrote: > > On Thu, May 22, 2014 at 7:13 PM, Shawn Heisey wrote: > > > >> On 5/22/2014 1:53 AM, Aniket Bhoi wrote: > >>> Details: > >>> > >>> *Solr Version:* > >>> Solr Specification Version: 3.4.0.2012.01.23.14.08.01 > >>> Solr Implementation Version: 3.4 > >>> Lucene Specification Version: 3.4 > >>> Lucene Implementation Version: 3.4 > >>> > >>> *Tomcat version:* > >>> Apache Tomcat/6.0.18 > >>> > >>> *OS details:* > >>> SUSE Linux Enterprise Server 11 (x86_64) > >>> VERSION = 11 > >>> PATCHLEVEL = 1 > >>> > >>> While running indexing on this server,It failed. > >>> > >>> Log excerpt: > >>> > >>> Caused by: > org.apache.solr.handler.dataimport.DataImportHandlerException: > >>> com.microsoft.sqlserver.jdbc.SQLServerException: The result set is > >> closed. > >>> > >>> Out intial hypothesis was that there is a problem with the connection > >>> thread,so we made changes to the context.xml and added > >>> validationQuery,testOnBorrow etc..to make sure the thread doesnt time > >> out. > >>> We also killed a lot of sleeping sessions from the server to the > >> database. > >>> All of the above still didnt work > >> I have reduced your log excerpt to what I think is the important part. > >> > >> Removing the multithreaded support as others have suggested is a good > >> idea, but what I think is really happening here is that Solr is engaging > >> in a multi-tier merge, so it stops indexing for a while ... and > >> meanwhile, JDBC times out and closes your database connection because of > >> inactivity. When the largest merge tier finishes, indexing tries to > >> resume, which it can't do because the database connection is closed. > >> > >> The solution is to allow more simultaneous merges to happen, which > >> allows indexing to continue while a multi-tier merge is underway. This > >> is my indexConfig section from solrconfig.xml: > >> > >> > >> > >> 35 > >> 35 > >> 105 > >> > >>class="org.apache.lucene.index.ConcurrentMergeScheduler"> > >> 1 > >> 6 > >> > >> 48 > >> false > >> > >> > >> The important part for your purposes is the mergeScheduler config, and > >> in particular, maxMergeCount. Increase that to 6. If you are using > >> standard spinning hard disks, do not increase maxThreadCount beyond 1. > >> If you are using SSD, you can safely increase that a small amount, but I > >> don't think I'd go above 2 or 3. > >> > >> Thanks, > >> Shawn > >> > >> > > I may be missing something ,or looking in the wrong place,But I cannot > find > > an indexConfig section or any other mentioned detail above in the > > solrconfig.xml file > > Solr will work without one, in which case it will simply use the > defaults. With older 3.x versions the mergeScheduler config will > actually need to go in an indexDefaults section. The mainIndex and > indexDefaults sections were deprecated in 3.6 and removed entirely in 4.x. > > https://issues.apache.org/jira/browse/SOLR-1052 > > If you don't have indexDefaults either, you may need to add the config > as a top level element under . If you do this, here's what you > should add: > > > > 1 > 6 > > > > I think we should probably change the default value that Solr uses for > maxMergeCount. This problem comes up fairly often. As long as > maxThreadCount is 1, I cannot think of a really good reason to limit > maxMergeCount at the level that we currently do. > > Thanks, > Shawn > > I changed the solrconfig.xml file and included the changes you suggested.However,this didnt work out,the Indexing still fails..I have also added this *:maxActive="100" minIdle="10" maxWait="1" initialSize="10" logAbandoned="true" validationQuery="select 1" testOnBorrow="true" testOnReturn="true" validationQueryTimeout="30" removeAbandoned="true" removeAbandonedTimeout="3600" * to the* context.xml* file.This didnt work etiher.The thing to note is after I changed the solrconfig.xml to add the merge config changes,the indexing failed after 6 hours.Earlier,It used to fail after 1 hour .
Re: Full Indexing fails on Solr-Probable connection issue.HELP!
On Thu, May 22, 2014 at 7:13 PM, Shawn Heisey wrote: > On 5/22/2014 1:53 AM, Aniket Bhoi wrote: > > Details: > > > > *Solr Version:* > > Solr Specification Version: 3.4.0.2012.01.23.14.08.01 > > Solr Implementation Version: 3.4 > > Lucene Specification Version: 3.4 > > Lucene Implementation Version: 3.4 > > > > *Tomcat version:* > > Apache Tomcat/6.0.18 > > > > *OS details:* > > SUSE Linux Enterprise Server 11 (x86_64) > > VERSION = 11 > > PATCHLEVEL = 1 > > > > While running indexing on this server,It failed. > > > > Log excerpt: > > > > Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: > > com.microsoft.sqlserver.jdbc.SQLServerException: The result set is > closed. > > > > > > Out intial hypothesis was that there is a problem with the connection > > thread,so we made changes to the context.xml and added > > validationQuery,testOnBorrow etc..to make sure the thread doesnt time > out. > > > > We also killed a lot of sleeping sessions from the server to the > database. > > > > All of the above still didnt work > > I have reduced your log excerpt to what I think is the important part. > > Removing the multithreaded support as others have suggested is a good > idea, but what I think is really happening here is that Solr is engaging > in a multi-tier merge, so it stops indexing for a while ... and > meanwhile, JDBC times out and closes your database connection because of > inactivity. When the largest merge tier finishes, indexing tries to > resume, which it can't do because the database connection is closed. > > The solution is to allow more simultaneous merges to happen, which > allows indexing to continue while a multi-tier merge is underway. This > is my indexConfig section from solrconfig.xml: > > > > 35 > 35 > 105 > > > 1 > 6 > > 48 > false > > > The important part for your purposes is the mergeScheduler config, and > in particular, maxMergeCount. Increase that to 6. If you are using > standard spinning hard disks, do not increase maxThreadCount beyond 1. > If you are using SSD, you can safely increase that a small amount, but I > don't think I'd go above 2 or 3. > > Thanks, > Shawn > > I may be missing something ,or looking in the wrong place,But I cannot find an indexConfig section or any other mentioned detail above in the solrconfig.xml file
Full Indexing fails on Solr-Probable connection issue.HELP!
I have Apache Solr,hosted on my apache Tomcat Server with SQLServer Backend. Details: *Solr Version:* Solr Specification Version: 3.4.0.2012.01.23.14.08.01 Solr Implementation Version: 3.4 Lucene Specification Version: 3.4 Lucene Implementation Version: 3.4 *Tomcat version:* Apache Tomcat/6.0.18 *OS details:* SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 1 While running indexing on this server,It failed. Log excerpt: May 19, 2014 9:23:28 AM org.apache.solr.common.SolrException log SEVERE: Full Import failed:java.lang.RuntimeException: Error in multi-threaded import at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder.execute(Unknown Source) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(Unknown Source) at org.apache.solr.handler.dataimport.DataImporter.runCmd(Unknown Source) at org.apache.solr.handler.dataimport.DataImporter$1.run(Unknown Source) Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: com.microsoft.sqlserver.jdbc.SQLServerException: The result set is closed. at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(Unknown Source) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(Unknown Source) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.runAThread(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.access$000(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner$1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The result set is closed. at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:170) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.checkClosed(SQLServerResultSet.java:346) at com.microsoft.sqlserver.jdbc.SQLServerResultSet.next(SQLServerResultSet.java:916) at org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:174) ... 12 more SEVERE: Can not close connection java.sql.SQLException: Already closed. at org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:84) at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:189) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.closeResources(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(Unknown Source) at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(Unknown Source) at org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(Unknown Source) at org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.ThreadedEntityProcessorWrapper.nextRow(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.runAThread(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner.access$000(Unknown Source) at org.apache.solr.handler.dataimport.DocBuilder$EntityRunner$1.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Out intial hypothesis was that there is a problem with the connection thread,so we made changes to the context.xml and added validationQuery,testOnBorrow etc..to make sure the thread doesnt time out. We also killed a lot of sleeping sessions from the server to the database. All of the above still didnt work Need help now,I have no clue what is going wrong now. Regards, Aniket Bhoi
