1- There is an improvement on the issue. I add 10 seconds time interval
into the delta of data-config.xml, which will cover records that already
indexed.
"revision_time > DATE_SUB('${dataimporter.last_index_time}', INTERVAL 10
SECOND);"
In this case 1369 new records inserted with 7 records per sec frequency.
Solr response shows 1369 new records successfully.
2-
If I update bashscript to sleep 10 seconds and autosoftcommit to 1 sec,
index files are updated every 10 seconds
If I updated autosoftcommit to 10 seconds and bashscript to sleep 10 sec,
index files are updated every 10 seconds
In index folder after each update, I see that segments/index files are
changing.
I restart the server before fell into the autocommit interval. Delta's are
still in the result list.
Here is my solrconfig.
<autoCommit>
<maxTime>300000</maxTime>
</autoCommit>
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
4- I comment out the autocommit part. Still index files are changing.
<!--
<autoCommit>
<maxTime>300000</maxTime>
</autoCommit>
-->
<autoSoftCommit>
<maxTime>1000</maxTime>
</autoSoftCommit>
I did not modify the request part in all of these cases.
wget -O /dev/null '
http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false'
2>/dev/null
#curl http://localhost:8080/solr-jak/update -H "Content-Type: text/xml"
--data-binary '<commit softCommit="true" waitFlush="false"
waitSearcher="false"/>' 2>/dev/null
Erick, as you mentioned, I believe that commit=false is not working
properly. If you need any information, I can serve it.
Thank you for all to your quick responses and advices.
Bests,
Jak
On Thu, Nov 17, 2011 at 1:34 PM, Erick Erickson <erickerick...@gmail.com>wrote:
> Hmmm. It is suspicious that your index files change every
> second. If you change our cron task to update every 10
> seconds, do the index files change every 10 seconds?
>
> Regarding your question about
> "After a server restart last query results reserved. (In NRT they would
> disappear, right?)"
> not necessarily. If your <autoCommit> interval is exceeded, the soft
> commits
> will be committed to disk so your Solr restart would pick them up after
> restart.
>
> But if somehow you're getting a hard commit to happen every second, you
> should
> also be seeing a lot of segment merging going on, are you?
>
> I think I'd stop the cron job and execute this manually for a while in
> order to see exactly
> where the problem is. I'd go ahead and comment out the <autoCommit> section
> as well. That should give you a much more reproducible test scenario.
>
> Say you do that, issue your delta-import and immediately kill your
> server. When it
> starts up if you then see the delta-data, we should understand why.
> Because it sure
> would seem like the commit=false isn't doing what you expect.
>
> Erick
>
> On Thu, Nov 17, 2011 at 12:41 PM, Jak Akdemir <jakde...@gmail.com> wrote:
> > Yonik,
> >
> > I updated my solrconfig time based only as follows.
> > <autoCommit>
> > <maxTime>300000</maxTime>
> > </autoCommit>
> >
> > <autoSoftCommit>
> > <maxTime>1000</maxTime>
> > </autoSoftCommit>
> >
> > And changed my soft commit script to the first case.
> > while [ 1 ]; do
> > echo "Soft commit applied!"
> > wget -O /dev/null '
> >
> http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false
> '
> > 2>/dev/null
> > sleep 1
> > done
> >
> > After full-import, I inserted 420 new records in a minute. (7 new
> records
> > per second) And softCommitted every second as we can see in
> solrconfig.xml.
> > It seems that after all solr can return only 326 of these new 420
> records.
> > Index files should not change every second, is it true? (After inserting
> > 420 records if I call delta-import with commit true, all these records
> can
> > be seen in solr results)
> >
> > Thanks,
> >
> > Jak
> >
> > On Thu, Nov 17, 2011 at 12:14 PM, Yonik Seeley
> > <yo...@lucidimagination.com>wrote:
> >
> >> On Thu, Nov 17, 2011 at 11:48 AM, Jak Akdemir <jakde...@gmail.com>
> wrote:
> >> > 2) I am sure about delta-queries configured well. Full-Import is
> >> completed
> >> > in 40 secs for 400000 docs. And delta's are in 1 sec for 15 new
> records.
> >> > Also I checked it. There is no problem in it.
> >>
> >> That's 10,000 docs/sec. If you configure a soft commit for every 15
> >> documents, that means solr is trying to do 666 commits/sec.
> >> Autocommit by number of docs rarely makes sense anymore - I'd suggest
> >> configuring both soft and hard commits based on time only.
> >>
> >> -Yonik
> >> http://www.lucidimagination.com
> >>
> >
>
