Grant,
I am messing with the script, and with your tip I expect I can
make it recurse over as many releases as needed.
I did run it again using the full file, this time using my Imac:-
643465 took 22min 14sec 2008-04-01
734796 73min 58sec 2009-01-15
758795 70min 55sec 2009-03-26
I then ran it again using only the first 1M records:-
643465 took 2m51.516s 2008-04-01
734796 7m29.326s 2009-01-15
758795 8m18.403s 2009-03-26
this time with commit=true.
643465 took 2m49.200s 2008-04-01
734796 8m27.414s 2009-01-15
758795 9m32.459s 2009-03-26
this time with commit=false&overwrite=false.
643465 took 2m46.149s 2008-04-01
734796 3m29.909s 2009-01-15
758795 3m26.248s 2009-03-26
Just read your latest post. I will apply the patches and retest
the above.
>Can you try adding &overwrite=false and running against the latest
>version? My current working theory is that Solr/Lucene has changed
>how deletes are handled such that work that was deferred before is now
>not deferred as often. In fact, you are not seeing this cost paid (or
>at least not noticing it) because you are not committing, but I
>believe you do see it when you are closing down Solr, which is why it
>takes so long to exit.
It can take ages! (>15min to get tomcat to quit). Also my script does
have the separate commit step, which does not take any time!
>I also think that Lucene adding fsync() into
>the equation may cause some slow down, but that is a penalty we are
>willing to pay as it gives us higher data integrity.
Data integrity is always good. However if performance seems
unreasonable, user/customers tend to take things into their
own hands and kill the process or machine. This tends to be
very bad for data integrity.
>So, depending on how you have your data, I think a workaround is to:
>Add a field that contains a single term identifying the data type for
>this particular CSV file, i.e. something like field: type, value:
>fergs-csv
>Then, before indexing, you can issue a Delete By Query: type:fergs-csv
>and then add your CSV file using overwrite=false. This amounts to a
>batch delete followed by a batch add, but without the add having to
>issue deletes for each add.
Ok.. but... for these test cases I am starting off with an empty
index. The script does a "rm -rf solr/data" before tomcat is launched.
So I do not understand how the above helps. UNLESS there are duplicate
gaz entries.
>In the meantime, I'm trying to see if I can pinpoint down a specific
>change and see if there is anything that might help it perform better.
>
>-Grant
>
--
===============================================================
Fergus McMenemie Email:fer...@twig.me.uk
Techmore Ltd Phone:(UK) 07721 376021
Unix/Mac/Intranets Analyst Programmer
===============================================================
