No VM. -----Original Message----- From: Bruno [mailto:brun...@gmail.com] Sent: Saturday, June 20, 2009 10:10 PM To: solr-user@lucene.apache.org Subject: Re: Slowness during submit the index
We were having performance issues using servers running on VM. Are you running QA or Prod in a VM? 2009/6/21, Stephen Weiss <swe...@stylesight.com>: > Isn't it possible that the production equipment is simply under much > higher load (given that, since it's in production, your various users > are all actually using it), vs the QA equipment, which is only in use > by the people doing QA? > > We've found the same thing at one point - we had a very small index (< > 40000 rows), so small it didn't seem worth the effort to do delta > updates. So we would just refresh the whole thing every time - or so > we planned. In the test environment it updated within a minute. In > production, it would take as long as 15 minutes. What we finally > realized was, because the DB was under much higher load in production > than in the test environment, especially considering the amount of > joins that needed to take place to pull out the data properly, various > writes from the users to the affected tables would slow down the data > selection process dramatically as the indexer would have to wait for > locks to clear. Now of course we do delta updates and everything's > fine (and blazingly fast in both environments). > > Try simulating higher load (involving a "normal" amount of writes to > the DB) against your QA equipment and then building the index. See if > the QA equipment still runs so quickly. > > -- > Steve > > On Jun 20, 2009, at 11:29 PM, Otis Gospodnetic wrote: > >> >> Hi Francis, >> >> I can't tell what the problem is from the information you've >> provided so far. My gut instinct is that this is due to some >> difference in QA vs. PROD environments that isn't Solr-specific. >> >> Otis >> -- >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> >> >> ----- Original Message ---- >>> From: Francis Yakin <fya...@liquid.com> >>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> >>> Sent: Saturday, June 20, 2009 2:18:07 AM >>> Subject: RE: Slowness during submit the index >>> >>> The amount of data in Prod is about 20% more than QA. >>> We tested the network speed is fine. The hardware in Prod is larger >>> and more >>> powerful than QA. >>> But QA is faster during reload. It takes QA only one hour than 6 >>> hours in Prod. >>> >>> That's why we don't understand what's the reason, the amount of >>> data is only 20% >>> more but it will not take 5 times slower because the data only 20% >>> more. >>> >>> So, we looked into the config file for solr, but it's not much >>> different, except >>> Prod has master/slave environment which QA only master. >>> >>> Thanks for the response. >>> >>> Francis >>> >>> >>> -----Original Message----- >>> From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] >>> Sent: Friday, June 19, 2009 8:58 PM >>> To: solr-user@lucene.apache.org >>> Subject: Re: Slowness during submit the index >>> >>> >>> Francis, >>> >>> So it could easily be that your QA and PROD DBs are really just >>> simply different >>> (different amount of data, different network speed, different >>> hardware...) >>> >>> Otis >>> -- >>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>> >>> >>> >>> ----- Original Message ---- >>>> From: Francis Yakin >>>> To: "solr-user@lucene.apache.org" >>>> Sent: Friday, June 19, 2009 10:39:48 PM >>>> Subject: RE: Slowness during submit the index >>>> >>>> * is the java version the same on both machines (QA vs. PROD) - YES >>>> * are the same java parameters being used on both machines - >>>> YES >>>> * is the connection to the DB the same on both machines - >>>> Not sure, >>> need >>>> to ask the network guy >>>> * are both the PROD and QA DB servers the same and are both DB >>>> instances the >>>> same - they are not from the same DB >>>> >>>> -----Original Message----- >>>> From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] >>>> Sent: Friday, June 19, 2009 6:23 PM >>>> To: solr-user@lucene.apache.org >>>> Subject: Re: Slowness during submit the index >>>> >>>> >>>> Francis, >>>> >>>> I'm not sure if I understood your email correctly, but I think you >>>> are saying >>>> you are indexing your DB content into a Solr index. If this is >>>> correct, here >>>> are things to look at: >>>> * is the java version the same on both machines (QA vs. PROD) >>>> * are the same java parameters being used on both machines >>>> * is the connection to the DB the same on both machines >>>> * are both the PROD and QA DB servers the same and are both DB >>>> instances the >>>> same >>>> ... >>>> >>>> >>>> Otis >>>> -- >>>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>>> >>>> >>>> >>>> ----- Original Message ---- >>>>> From: Francis Yakin >>>>> To: "solr-user@lucene.apache.org" >>>>> Sent: Friday, June 19, 2009 5:27:59 PM >>>>> Subject: Slowness during submit the index >>>>> >>>>> >>>>> We are experiencing slowness during reloading/resubmitting index >>>>> from >>> Database >>>>> to the master. >>>>> >>>>> We have two environments: >>>>> >>>>> QA and Prod. >>>>> >>>>> The slowness is happened only in Production but not in QA. >>>>> >>>>> It only takes one hours to reload 2.5Mil indexes compare 5-6 >>>>> hours to load >>> the >>>>> same size of index in Prod. >>>>> >>>>> I checked both the config files in QA and Prod, they are all >>>>> identical, >>>> except: >>>>> >>>>> >>>>> In QA: >>>>> false >>>>> In Prod: >>>>> true >>>>> >>>>> I believe that we use "http" protocol reload/submit the index >>>>> from Database >>> to >>>>> Solr Master. >>>>> I did test copying big files thru network from database to the >>>>> solr box, I >>>> don't >>>>> see any issue. >>>>> >>>>> We are running solr 1.2 >>>>> >>>>> Any inputs will be much appreciated. >> > > -- Enviado do meu celular Bruno Morelli Vargas Mail: brun...@gmail.com Msn: brun...@hotmail.com Icq: 165055101 Skype: morellibmv
