Re: How to Configure Solr For Test Purposes?
Hi; I've developed a Proxy application that takes request from clients and sends them to Solr then gets response and sends them to client as response. So, I am testing my application, my proxy for Solr. Thanks; Furkan KAMACI 2014-05-27 14:52 GMT+03:00 Tomás Fernández Löbbe : > > What do you suggest for my purpose? If a test case fails re-running it > for > > some times maybe a solution? What kind of configuration do you suggest > for > > my Solr configuration? > > > > > From the snippet of test that you showed, it looks like it's testing only > Solr functionality. So, first make sure this is a test that you really > need. Solr has it's own tests, and it you feel it could use more (for some > specific case or context), I'd open a Jira and try to get the test inside > Solr. > If my impression is wrong and your test is actually testing your code, then > I'd suggest you to use a specific soft commit call with waitSearcher = true > on your test instead of relying on the autocommit (and remove the > autocommit completely from your solrconfig). > > Tomás > > > > Thanks; > > Furkan KAMACI > > 26 May 2014 21:03 tarihinde "Shawn Heisey" yazdı: > > > > > On 5/26/2014 10:57 AM, Furkan KAMACI wrote: > > > > Hi; > > > > > > > > I run Solr within my Test Suite. I delete documents or atomically > > update > > > > them and check whether if it works or not. I know that I have to > setup > > a > > > > hard/soft commit timing for my test Solr. However even I have that > > > settings: > > > > > > > > > > > >1 > > > >true > > > > > > > > > > > > > > > > 1 > > > > > > > > > > I hope you know that this is BAD configuration. Doing automatic > commits > > > on an interval of 1 millisecond is asking for a whole host of problems. > > > In some cases, this could do a commit after every single document that > > > is indexed, which is NOT recommended at all. The openSearcher setting > > > of "true" on autoCommit makes it even worse. There's no reason to do > > > both autoSoftCommit and autoCommit with openSearcher=true. I don't > know > > > which one "wins" between autoCommit and autoSoftCommit if they both > have > > > the same config, but I would guess the hard commit does. > > > > > > > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* > my > > > > tests are failed. I get fail error even I increase wait time. > Example > > > of a > > > > sometimes failed code piece: > > > > > > > > for (int i = 0; i < dummyDocumentSize; i++) { > > > > deleteById("id" + i); > > > > dummyDocumentSize--; > > > > queryResponse = query(solrParams); > > > > assertTrue(queryResponse.getResults().size() == > > > dummyDocumentSize); > > > > } > > > > > > > > at debug mode if I wait for Solr to reflect changes I see that I do > not > > > get > > > > error. What do you think, what kind of configuration I should have > for > > > such > > > > kind of purposes? > > > > > > Chances are that commits are going to take longer than 1 millisecond. > > > If you're actively indexing, the system is going to be trying to stack > > > up lots of commits at the same time. The maxWarmingSearchers value > will > > > limit the number of new searchers that can be opened, but it will not > > > stop the commits themselves. When lots of commits are going on, each > > > one will take *even longer* to complete, which probably explains the > > > problem. > > > > > > Thanks, > > > Shawn > > > > > > > > >
Re: How to Configure Solr For Test Purposes?
> What do you suggest for my purpose? If a test case fails re-running it for > some times maybe a solution? What kind of configuration do you suggest for > my Solr configuration? > > >From the snippet of test that you showed, it looks like it's testing only Solr functionality. So, first make sure this is a test that you really need. Solr has it's own tests, and it you feel it could use more (for some specific case or context), I'd open a Jira and try to get the test inside Solr. If my impression is wrong and your test is actually testing your code, then I'd suggest you to use a specific soft commit call with waitSearcher = true on your test instead of relying on the autocommit (and remove the autocommit completely from your solrconfig). Tomás Thanks; > Furkan KAMACI > 26 May 2014 21:03 tarihinde "Shawn Heisey" yazdı: > > > On 5/26/2014 10:57 AM, Furkan KAMACI wrote: > > > Hi; > > > > > > I run Solr within my Test Suite. I delete documents or atomically > update > > > them and check whether if it works or not. I know that I have to setup > a > > > hard/soft commit timing for my test Solr. However even I have that > > settings: > > > > > > > > >1 > > >true > > > > > > > > > > > > 1 > > > > > > > I hope you know that this is BAD configuration. Doing automatic commits > > on an interval of 1 millisecond is asking for a whole host of problems. > > In some cases, this could do a commit after every single document that > > is indexed, which is NOT recommended at all. The openSearcher setting > > of "true" on autoCommit makes it even worse. There's no reason to do > > both autoSoftCommit and autoCommit with openSearcher=true. I don't know > > which one "wins" between autoCommit and autoSoftCommit if they both have > > the same config, but I would guess the hard commit does. > > > > > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my > > > tests are failed. I get fail error even I increase wait time. Example > > of a > > > sometimes failed code piece: > > > > > > for (int i = 0; i < dummyDocumentSize; i++) { > > > deleteById("id" + i); > > > dummyDocumentSize--; > > > queryResponse = query(solrParams); > > > assertTrue(queryResponse.getResults().size() == > > dummyDocumentSize); > > > } > > > > > > at debug mode if I wait for Solr to reflect changes I see that I do not > > get > > > error. What do you think, what kind of configuration I should have for > > such > > > kind of purposes? > > > > Chances are that commits are going to take longer than 1 millisecond. > > If you're actively indexing, the system is going to be trying to stack > > up lots of commits at the same time. The maxWarmingSearchers value will > > limit the number of new searchers that can be opened, but it will not > > stop the commits themselves. When lots of commits are going on, each > > one will take *even longer* to complete, which probably explains the > > problem. > > > > Thanks, > > Shawn > > > > >
Re: How to Configure Solr For Test Purposes?
Hi All; I have defined just that: 1 then turned of hard commit. I've run my tests time and time again and did not get any error. Who wants to write a unit test that interacts with Solr as like my situation can use it. Thanks; Furkan KAMACI 2014-05-26 23:37 GMT+03:00 Furkan KAMACI : > Hi Shawn; > > I know that it is a bad practise but I just commit up to 5 documents and > there will not be more than 5 documents at any time at any test method. It > is just for test purpose to see that my API works. I want to have > automatic tests. > > What do you suggest for my purpose? If a test case fails re-running it for > some times maybe a solution? What kind of configuration do you suggest for > my Solr configuration? > > Thanks; > Furkan KAMACI > 26 May 2014 21:03 tarihinde "Shawn Heisey" yazdı: > > On 5/26/2014 10:57 AM, Furkan KAMACI wrote: >> > Hi; >> > >> > I run Solr within my Test Suite. I delete documents or atomically update >> > them and check whether if it works or not. I know that I have to setup a >> > hard/soft commit timing for my test Solr. However even I have that >> settings: >> > >> > >> >1 >> >true >> > >> > >> > >> > 1 >> > >> >> I hope you know that this is BAD configuration. Doing automatic commits >> on an interval of 1 millisecond is asking for a whole host of problems. >> In some cases, this could do a commit after every single document that >> is indexed, which is NOT recommended at all. The openSearcher setting >> of "true" on autoCommit makes it even worse. There's no reason to do >> both autoSoftCommit and autoCommit with openSearcher=true. I don't know >> which one "wins" between autoCommit and autoSoftCommit if they both have >> the same config, but I would guess the hard commit does. >> >> > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my >> > tests are failed. I get fail error even I increase wait time. Example >> of a >> > sometimes failed code piece: >> > >> > for (int i = 0; i < dummyDocumentSize; i++) { >> > deleteById("id" + i); >> > dummyDocumentSize--; >> > queryResponse = query(solrParams); >> > assertTrue(queryResponse.getResults().size() == >> dummyDocumentSize); >> > } >> > >> > at debug mode if I wait for Solr to reflect changes I see that I do not >> get >> > error. What do you think, what kind of configuration I should have for >> such >> > kind of purposes? >> >> Chances are that commits are going to take longer than 1 millisecond. >> If you're actively indexing, the system is going to be trying to stack >> up lots of commits at the same time. The maxWarmingSearchers value will >> limit the number of new searchers that can be opened, but it will not >> stop the commits themselves. When lots of commits are going on, each >> one will take *even longer* to complete, which probably explains the >> problem. >> >> Thanks, >> Shawn >> >>
Re: How to Configure Solr For Test Purposes?
Hi Shawn; I know that it is a bad practise but I just commit up to 5 documents and there will not be more than 5 documents at any time at any test method. It is just for test purpose to see that my API works. I want to have automatic tests. What do you suggest for my purpose? If a test case fails re-running it for some times maybe a solution? What kind of configuration do you suggest for my Solr configuration? Thanks; Furkan KAMACI 26 May 2014 21:03 tarihinde "Shawn Heisey" yazdı: > On 5/26/2014 10:57 AM, Furkan KAMACI wrote: > > Hi; > > > > I run Solr within my Test Suite. I delete documents or atomically update > > them and check whether if it works or not. I know that I have to setup a > > hard/soft commit timing for my test Solr. However even I have that > settings: > > > > > >1 > >true > > > > > > > > 1 > > > > I hope you know that this is BAD configuration. Doing automatic commits > on an interval of 1 millisecond is asking for a whole host of problems. > In some cases, this could do a commit after every single document that > is indexed, which is NOT recommended at all. The openSearcher setting > of "true" on autoCommit makes it even worse. There's no reason to do > both autoSoftCommit and autoCommit with openSearcher=true. I don't know > which one "wins" between autoCommit and autoSoftCommit if they both have > the same config, but I would guess the hard commit does. > > > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my > > tests are failed. I get fail error even I increase wait time. Example > of a > > sometimes failed code piece: > > > > for (int i = 0; i < dummyDocumentSize; i++) { > > deleteById("id" + i); > > dummyDocumentSize--; > > queryResponse = query(solrParams); > > assertTrue(queryResponse.getResults().size() == > dummyDocumentSize); > > } > > > > at debug mode if I wait for Solr to reflect changes I see that I do not > get > > error. What do you think, what kind of configuration I should have for > such > > kind of purposes? > > Chances are that commits are going to take longer than 1 millisecond. > If you're actively indexing, the system is going to be trying to stack > up lots of commits at the same time. The maxWarmingSearchers value will > limit the number of new searchers that can be opened, but it will not > stop the commits themselves. When lots of commits are going on, each > one will take *even longer* to complete, which probably explains the > problem. > > Thanks, > Shawn > >
Re: How to Configure Solr For Test Purposes?
On 5/26/2014 10:57 AM, Furkan KAMACI wrote: > Hi; > > I run Solr within my Test Suite. I delete documents or atomically update > them and check whether if it works or not. I know that I have to setup a > hard/soft commit timing for my test Solr. However even I have that settings: > > >1 >true > > > > 1 > I hope you know that this is BAD configuration. Doing automatic commits on an interval of 1 millisecond is asking for a whole host of problems. In some cases, this could do a commit after every single document that is indexed, which is NOT recommended at all. The openSearcher setting of "true" on autoCommit makes it even worse. There's no reason to do both autoSoftCommit and autoCommit with openSearcher=true. I don't know which one "wins" between autoCommit and autoSoftCommit if they both have the same config, but I would guess the hard commit does. > and even I wait (Thread.sleep()) for a time to wait Solr *sometimes* my > tests are failed. I get fail error even I increase wait time. Example of a > sometimes failed code piece: > > for (int i = 0; i < dummyDocumentSize; i++) { > deleteById("id" + i); > dummyDocumentSize--; > queryResponse = query(solrParams); > assertTrue(queryResponse.getResults().size() == dummyDocumentSize); > } > > at debug mode if I wait for Solr to reflect changes I see that I do not get > error. What do you think, what kind of configuration I should have for such > kind of purposes? Chances are that commits are going to take longer than 1 millisecond. If you're actively indexing, the system is going to be trying to stack up lots of commits at the same time. The maxWarmingSearchers value will limit the number of new searchers that can be opened, but it will not stop the commits themselves. When lots of commits are going on, each one will take *even longer* to complete, which probably explains the problem. Thanks, Shawn