Hi Dominique Any suggestions?
On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <[email protected]> wrote: > HI Dominique > > softcommit=false is coming in logs, > > > INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1 > x:solrcollection_shard2_replica1] > org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor; > [solrcollection_shard2_replica1] webapp=/solr path=/update > params={update.distrib=FROMLEADER&waitSearcher=true& > *openSearcher=true&commit=true&softCommit=false*&distrib.from= > http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} > 0 676 > INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1 > x:solrcollection_shard2_replica1] > org.apache.solr.update.DirectUpdateHandler2; start > commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} > > > On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <[email protected]> > wrote: > >> Hi, >> >> If your tlogs grow, It looks like hard commit are never executed. >> >> Are you seeing hard commits in logs (grep 'softCommit=false') ? >> >> What is it happening if you force a hard commit ? Are tlog truncated ? >> http://localhost:8983/solr/[collection_name]/update?commit=true >> >> Did you read this article ? >> >> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >> >> Regards >> >> Dominique >> >> >> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <[email protected]> a >> écrit : >> >>> Hi Dominique >>> >>> Heavy query means high query rate on solr. >>> >>> Honestly for 2-3 days we have stopped the queries on solr only we are >>> doing indexing !!! >>> As you see we have enabled Hard commit also to decrease tlog log size or >>> more and the same is written on solr docs. Still the same behaviour is >>> occurring on replicas. In some replicas, tlog size is of 300 GB. >>> >>> We may increase softcommit timeout but as per my understanding it only >>> helps in searching availability prior to commit to indexes. >>> >>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> By heavy query, do you mean you have a high query rate and/or you need >>>> index update be available fast after few seconds (NRT search) ? >>>> >>>> Do you see the hard commits in logs ? >>>> Can you try to increase autoSoftCommit to 30 seconds or more ? >>>> >>>> Regards >>>> >>>> Dominique >>>> >>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <[email protected]> a >>>> écrit : >>>> >>>> > Hi Guys >>>> > >>>> > We have faced an issue where tlog size is increasing unnecessarily. >>>> We are >>>> > using a "heavy indexing, heavy query" approach. We enabled hard >>>> commit >>>> > also, >>>> > >>>> > solr cloud: 6.6 >>>> > zk: 3.4.10 >>>> > shards: 2, replication factor= 2 >>>> > >>>> > >>>> > solrconfig, >>>> > >>>> > <autoCommit> >>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>> > <maxDocs>10000</maxDocs> >>>> > <openSearcher>false</openSearcher> >>>> > >>>> > >>>> > </autoCommit> >>>> > >>>> > <!-- softAutoCommit is like autoCommit except it causes a >>>> > 'soft' commit which only ensures that changes are visible >>>> > but does not ensure that data is synced to disk. This is >>>> > faster and more near-realtime friendly than a hard commit. >>>> > --> >>>> > >>>> > <autoSoftCommit> >>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>> > </autoSoftCommit> >>>> > >>>> > In every replica, tlog is increasing for more than 200GB which >>>> exhaust disk >>>> > space. Please suggest someting >>>> > >>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <[email protected]> >>>> wrote: >>>> > >>>> > > Hi Guys >>>> > > >>>> > > We have faced an issue where tlog size is increasing unnecessarily. >>>> We >>>> > are >>>> > > using a "heavy indexing, heavy query" approach. We enabled hard >>>> commit >>>> > > also, >>>> > > >>>> > > solr cloud: 6.6 >>>> > > zk: 3.4.10 >>>> > > shards: 2, replication factor= 2 >>>> > > >>>> > > >>>> > > solrconfig, >>>> > > >>>> > > <autoCommit> >>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>> > > <maxDocs>10000</maxDocs> >>>> > > <openSearcher>false</openSearcher> >>>> > > >>>> > > >>>> > > </autoCommit> >>>> > > >>>> > > <!-- softAutoCommit is like autoCommit except it causes a >>>> > > 'soft' commit which only ensures that changes are visible >>>> > > but does not ensure that data is synced to disk. This is >>>> > > faster and more near-realtime friendly than a hard commit. >>>> > > --> >>>> > > >>>> > > <autoSoftCommit> >>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>> > > </autoSoftCommit> >>>> > > >>>> > > >>>> > >>>> >>>
