I guess my first question is what evidence you have that Solr is unable to index fast enough? It's quite possible that your database connection is the thing that's unable to process fast enough.
That's certainly a guess, but unless your documents are quite complex, 15 records/second isn't likely to cause Solr problems. You might try to run a small Java program that executes your database queries and see. The other question I'd ask is if you're absolutely sure that your delta-import query is correct? Is it possible that you're re-indexing *everything* every time? There's an interactive debugging console you can use that may help, try: http://localhost:8983/solr/admin/dataimport.jsp Best Erick On Thu, Nov 17, 2011 at 3:19 AM, Jak Akdemir <jakde...@gmail.com> wrote: > Hi, > > I was trying to configure a Solr instance with the near real-time search > and auto-complete capabilities. I stuck in the NRT feature. There are > 15 new records per second that inserted into the database (mysql) and I > indexed them with DIH. First, I tried to manage autoCommits from > solrconfig.xml with the configuration below. > > <autoCommit> > <maxDocs>10000</maxDocs> > <maxTime>100000</maxTime> > </autoCommit> > > <autoSoftCommit> > <maxDocs>15</maxDocs> > <maxTime>1000</maxTime> > </autoSoftCommit> > > And the bash script below responsible for getting delta's without > committing. > > while [ 1 ]; do > wget -O /dev/null ' > http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false' > 2>/dev/null > sleep 1 > done > > Then I run my query from browser > http://localhost:8080/solr-jak/select?q=movie_name_prefix_full:"dogville"&defType=lucene&q.op=OR<http://localhost:8080/solr-sprongo/select?q=movie_name_prefix_full:%221398%22&defType=lucene&q.op=OR> > > But I realized that, with this configuration index files are changing every > second and after a minute there are only 600 new records in Solr index > while 900 new records in the database. > After experienced that, I removed autoCommit and autoSoftCommit elements in > solrconfig.xml And updated my bashscript as follows. But still index files > are changing and solr can not syncronized with database. > > while [ 1 ]; do > echo "Soft commit applied!" > wget -O /dev/null ' > http://localhost:8080/solr-jak/dataimport?command=delta-import&commit=false' > 2>/dev/null > curl http://localhost:8080/solr-jak/update -H "Content-Type: text/xml" > --data-binary '<commit softCommit="true" waitFlush="false" > waitSearcher="false"/>' 2>/dev/null > sleep 3 > done > > Even I decreased the pressure on Solr as 1 new record per sec. and soft > commits within 6 sec. still there is a gap between index and db. Is there > anything that I missed? I took a look to "/get" too, but it is working only > for pk. If there is an example configuration list (like 1 sec for soft > commit and 10 min for hard commit) as a best practice it would be great. > > Finally, here is my configuration. > Ubuntu 11.04 > JDK 1.6.0_27 > Tomcat 7.0.21 > Solr 4.0 2011-10-24_08-53-02 > > All advices are appreciated, > > Best Regards, > > Jak >
