It would be worth looking into iostats of your disks. On Aug 22, 2016 10:11 AM, "Alessandro Benedetti" <abenede...@apache.org> wrote:
> I agree with the suggestions so far. > The cache auto-warming doesn't seem the problem as the index is not massive > and the auto-warm is for only 10 docs. > Are you using any warming query for the new searcher ? > > Are you using soft or hard commit ? > This can make the difference ( soft are much cheaper, not free but cheaper) > . > You said : > " Actually earlier it was taking less but suddenly it has increased " > > What happened ? > Anyway, there are a lot of questions to answer before we can help you... > > Cheers > > On Fri, Aug 12, 2016 at 4:58 AM, Esther-Melaine Quansah < > esther.quan...@lucidworks.com> wrote: > > > Midas, > > > > I’d like further clarification as well. Are you sending commits along > with > > each document that you’re POSTing to Solr? If so, you’re essentially > either > > opening a new searcher or flushing to disk with each POST which could > > explain latency between each request. > > > > Thanks, > > > > Esther > > > On Aug 11, 2016, at 12:19 PM, Erick Erickson <erickerick...@gmail.com> > > wrote: > > > > > > bq: we post json documents through the curl it takes the time (same > > time i > > > would like to say that we are not hard committing ). that curl takes > time > > > i.e. 1.3 sec. > > > > > > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said > > > commit, I was thinking of Solr's commit operation, which is totally > > distinct > > > from just adding a doc to the index. But I read the above statement > > > as you're saying it takes 1.3 seconds just to send a doc to Solr. > > > > > > Let's see the exact curl command you're using please? > > > > > > Best, > > > Erick > > > > > > > > > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic > > > <emir.arnauto...@sematext.com> wrote: > > >> Hi Midas, > > >> > > >> 1. How many indexing threads? > > >> 2. Do you batch documents and what is your batch size? > > >> 3. How frequently do you commit? > > >> > > >> I would recommend: > > >> 1. Move commits to Solr (set auto soft commit to max allowed time) > > >> 2. Use batches (bulks) > > >> 3. tune bulk size and number of threads to achieve max performance. > > >> > > >> Thanks, > > >> Emir > > >> > > >> > > >> > > >> On 11.08.2016 08:21, Midas A wrote: > > >>> > > >>> Emir, > > >>> > > >>> other queries: > > >>> > > >>> a) Solr cloud : NO > > >>> b) <filterCache class="solr.FastLRUCache" > > >>> size="5000" initialSize="5000" autowarmCount="10"/> > > >>> c) <queryResultCache class="solr.LRUCache" > > >>> size="1000" initialSize="1000" autowarmCount="10"/> > > >>> d) <documentCache class="solr.LRUCache" > > >>> size="1000" initialSize="1000" autowarmCount="10"/> > > >>> e) we are using multi threaded system. > > >>> > > >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <test.mi...@gmail.com> > > wrote: > > >>> > > >>>> Emir, > > >>>> > > >>>> we post json documents through the curl it takes the time (same > time i > > >>>> would like to say that we are not hard committing ). that curl takes > > time > > >>>> i.e. 1.3 sec. > > >>>> > > >>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic < > > >>>> emir.arnauto...@sematext.com> wrote: > > >>>> > > >>>>> Hi Midas, > > >>>>> > > >>>>> According to your autocommit configuration and your worry about > > commit > > >>>>> time I assume that you are doing explicit commits from client code > > and > > >>>>> that > > >>>>> 1.3s is client observed commit time. If that is the case, than it > > might > > >>>>> be > > >>>>> opening searcher that is taking time. > > >>>>> > > >>>>> How do you index data - single threaded or multithreaded? How > > frequently > > >>>>> do you commit from client? Can you let Solr do soft commits instead > > of > > >>>>> explicitly committing? Do you have warmup queries? Is this > SolrCloud? > > >>>>> What > > >>>>> is number of servers (what spec), shards, docs? > > >>>>> > > >>>>> In any case monitoring can give you more info about server/Solr > > behavior > > >>>>> and help you diagnose issues more easily/precisely. One such > > monitoring > > >>>>> tool is our SPM <http://sematext.com/spm>. > > >>>>> > > >>>>> Regards, > > >>>>> Emir > > >>>>> > > >>>>> -- > > >>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log > > Management > > >>>>> Solr & Elasticsearch Support * http://sematext.com/ > > >>>>> > > >>>>> On 10.08.2016 05:20, Midas A wrote: > > >>>>> > > >>>>>> Thanks for replying > > >>>>>> > > >>>>>> index size:9GB > > >>>>>> 2000 docs/sec. > > >>>>>> > > >>>>>> Actually earlier it was taking less but suddenly it has increased > . > > >>>>>> > > >>>>>> Currently we do not have any monitoring tool. > > >>>>>> > > >>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic < > > >>>>>> emir.arnauto...@sematext.com> wrote: > > >>>>>> > > >>>>>> Hi Midas, > > >>>>>>> > > >>>>>>> Can you give us more details on your index: size, number of new > > docs > > >>>>>>> between commits. Why do you think 1.3s for commit is to much and > > why > > >>>>>>> do > > >>>>>>> you > > >>>>>>> need it to take less? Did you do any system/Solr monitoring? > > >>>>>>> > > >>>>>>> Emir > > >>>>>>> > > >>>>>>> > > >>>>>>> On 09.08.2016 14:10, Midas A wrote: > > >>>>>>> > > >>>>>>> please reply it is urgent. > > >>>>>>>> > > >>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <test.mi...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>> Hi , > > >>>>>>>> > > >>>>>>>>> commit is taking more than 1300 ms . what should i check on > > server. > > >>>>>>>>> > > >>>>>>>>> below is my configuration . > > >>>>>>>>> > > >>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit. > maxTime:15000}</maxTime> > > < > > >>>>>>>>> openSearcher>false</openSearcher> </autoCommit> > <autoSoftCommit> > > >>>>>>>>> <maxTime> > > >>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime> </autoSoftCommit> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>> > > >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log > > Management > > >>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >> > > >> -- > > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > >> Solr & Elasticsearch Support * http://sematext.com/ > > >> > > > > > > > -- > -------------------------- > > Benedetti Alessandro > Visiting card : http://about.me/alessandro_benedetti > > "Tyger, tyger burning bright > In the forests of the night, > What immortal hand or eye > Could frame thy fearful symmetry?" > > William Blake - Songs of Experience -1794 England >
