*Very* slow Commit after upgrading to solr 1.3

[Ben Shlomo, Yatir]

Hi!

 

I am running on widows 64 bit ...
I have upgraded to solr 1.3 in order to use the distributed search.

I haven't changed the solrConfig and the schema xml files during the
upgrade.

I am indexing ~ 350K documents (each one is about 0.5 KB in size)

The indexing takes a reasonable amount of time (350 seconds)

See tomcat log:

INFO: {add=[8x-wbTscWftuu1sVWpdnGw==, VOu1eSv0obBl1xkj2jGjIA==,
YkOm-nKPrTVVVyeCZM4-4A==, rvaq_TyYsqt3aBc0KKDVbQ==,
9NdzWXsErbF_5btyT1JUjw==, ...(398728 more)]} 0 349875

 

But when I commit it takes more than an hour ! (5000 seconds!, the
optimize after the commit took 14 seconds)

INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true)

 

p.s. its not a machine problem I moved to another machine and the same
thing happened


I noticed something very strange during the time I wait for the commit:

While the solr index is 210MB in size

In the windows task manager I noticed that the java process is making a
HUGE amounts of IO reads:

It reads more than 350 GB ! (- which takes a lot of time.)

The process is constantly taking 25% of the cpu resources.

All my autowarmCount in Solrconfig  file do not exceed 256...

 

Any more ideas to check?

Thanks.

 

 

 

Here is part of my solrConfig file:

- <file:///C:\dss1\SolrHome\conf\solrconfig.xml##>  < - <indexDefaults>

- <!--  Values here affect all index writers and act as a default unless
overridden. 

  --> 

  <useCompoundFile>false</useCompoundFile> 

  <mergeFactor>1000</mergeFactor> 

  <maxBufferedDocs>1000</maxBufferedDocs> 

  <maxMergeDocs>2147483647</maxMergeDocs> 

  <maxFieldLength>10000</maxFieldLength> 

  <writeLockTimeout>1000</writeLockTimeout> 

  <commitLockTimeout>10000</commitLockTimeout> 

  </indexDefaults>

- <mainIndex>

- <!--  options specific to the main on-disk lucene index 

  --> 

  <useCompoundFile>false</useCompoundFile> 

  <mergeFactor>1000</mergeFactor> 

  <maxBufferedDocs>1000</maxBufferedDocs> 

  <maxMergeDocs>2147483647</maxMergeDocs> 

  <maxFieldLength>10000</maxFieldLength> 

- <!--  If true, unlock any held write or commit locks on startup. 

         This defeats the locking mechanism that allows multiple

         processes to safely access a lucene index, and should be

         used with care. 

  --> 

  <unlockOnStartup>true</unlockOnStartup> 

  </mainIndex>

 

 

 

 

 

Yatir Ben-shlomo | eBay, Inc. | Classification Track, Shopping.com
(Israel) | w: +972-9-892-1373 |  email: [EMAIL PROTECTED] |