Hi,
I am facing multiple issues with solr and I am not sure what happens in each
case. I am quite naive in Solr and there are some scenarios I'd like to
discuss with you.
We have a huge volume of documents to be indexed. Somewhere about 5 million.
We have a full indexer script which essentially picks up all the documents
from database and updates into Solr and an incremental script which adds new
documents to Solr.. Relevant areas of my config file goes like
<unlockOnStartup>false</unlockOnStartup>
<deletionPolicy class="solr.SolrDeletionPolicy">
<!-- Keep only optimized commit points -->
<str name="keepOptimizedOnly">false</str>
<!-- The maximum number of commit points to be kept -->
<str name="maxCommitsToKeep">1</str>
</deletionPolicy>
<updateHandler class="solr.DirectUpdateHandler2">
<autoCommit>
<maxDocs>100000</maxDocs>
</autoCommit>
</updateHandler>
<requestHandler name="/replication" class="solr.ReplicationHandler">
<lst name="master">
<str name="enable">${enable.master:false}</str>
<str name="replicateAfter">startup</str>
<str name="replicateAfter">commit</str>
</lst>
<lst name="slave">
<str name="enable">${enable.slave:false}</str>
<str name="masterUrl">http://hostname:port/solr/core0/replication</str>
</lst>
</requestHandler>
Sometimes, while the full indexer script breaks while adding documents to
Solr. The script adds the documents and then commits the operation. So, when
the script breaks, we have a huge lot of data which has been updated but not
committed. Next, the incremental index script executes, and figures out all
the new entries, adds them to Solr. It works successfully and commits the
operation.
- Will the commit by incremental indexer script also commit the
previously uncommitted changes made by full indexer script before it broke?
Sometimes, while during execution, Solr's avg response time 9avg resp time
for last 10 requests, read from log file) goes as high as 9000ms (which I am
still unclear why, any ideas how to start hunting for the problem?), so the
watchdog process restarts Solr (because it causes a pile of requests queue
at application server, which causes app server to crash). On my local
environment, I performed the same experiment by adding docs to Solr, killing
the process and restarting it. I found that the uncommitted changes were
applied and searchable. However, the updates were uncommitted. Could you
explain me as to how is this happening, or is there a configuration that can
be adjusted for this? Also, what would the index state be if after the
restarting Solr, a commit is applied or a commit is not applied?
I'd be happy to provide any other information that might be needed.
*Pranav Prakash*
"temet nosce"
Twitter <http://twitter.com/pranavprakash> | Blog <http://blog.myblive.com> |
Google <http://www.google.com/profiles/pranny>
On Tue, Jun 28, 2011 at 20:55, Shalin Shekhar Mangar <shalinman...@gmail.com
> wrote:
> On Tue, Jun 28, 2011 at 4:18 PM, Pranav Prakash <pra...@gmail.com> wrote:
>
> >
> > I am not sure what is the index number value? It looks like an epoch
> time,
> > but in my case, this points to one month back. However, i can see
> documents
> > which were added last week, to be in the index.
> >
>
> The index version shown on the dashboard is the time at which the most
> recent index segment was created. I'm not sure why it has a value older
> than
> a month if a commit has happened after that time.
>
> >
> > Even after I did a commit, the index number did not change? Isn't it
> > supposed to change on every commit? If not, is there a way to look into
> the
> > last index time?
> >
>
> Yeah, it changes after every commit which added/deleted a document.
>
>
> > Also, this page
> > http://wiki.apache.org/solr/SolrReplication#Replication_Dashboard shows
> a
> > Replication Dashboard. How is this dashboard invoked? Is there any URL
> > which
> > needs to be called?
> >
> >
> If you have configured replication correctly, the admin dashboard should
> show a "Replication" link right next to the "Schema Browser" link. The path
> should be /admin/replication/index.jsp
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
