Hi, Did you try to force a hard commit in order to see the inpact on tlog ? http://localhost:8983/solr/[collection_name]/update?commit=true
Did you read this article ? https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ Regards Dominique Le lun. 22 mars 2021 à 17:29, Ritvik Sharma <[email protected]> a écrit : > Hi Dominique > > Any suggestions? > > On Mon, 22 Mar 2021 at 10:23, Ritvik Sharma <[email protected]> wrote: > >> HI Dominique >> >> softcommit=false is coming in logs, >> >> >> INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1 >> x:solrcollection_shard2_replica1] >> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor; >> [solrcollection_shard2_replica1] webapp=/solr path=/update >> params={update.distrib=FROMLEADER&waitSearcher=true& >> *openSearcher=true&commit=true&softCommit=false*&distrib.from= >> http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} >> 0 676 >> INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1 >> x:solrcollection_shard2_replica1] >> org.apache.solr.update.DirectUpdateHandler2; start >> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} >> >> >> On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <[email protected]> >> wrote: >> >>> Hi, >>> >>> If your tlogs grow, It looks like hard commit are never executed. >>> >>> Are you seeing hard commits in logs (grep 'softCommit=false') ? >>> >>> What is it happening if you force a hard commit ? Are tlog truncated ? >>> http://localhost:8983/solr/[collection_name]/update?commit=true >>> >>> Did you read this article ? >>> >>> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ >>> >>> Regards >>> >>> Dominique >>> >>> >>> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <[email protected]> a >>> écrit : >>> >>>> Hi Dominique >>>> >>>> Heavy query means high query rate on solr. >>>> >>>> Honestly for 2-3 days we have stopped the queries on solr only we are >>>> doing indexing !!! >>>> As you see we have enabled Hard commit also to decrease tlog log size >>>> or more and the same is written on solr docs. Still the same behaviour is >>>> occurring on replicas. In some replicas, tlog size is of 300 GB. >>>> >>>> We may increase softcommit timeout but as per my understanding it only >>>> helps in searching availability prior to commit to indexes. >>>> >>>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> By heavy query, do you mean you have a high query rate and/or you need >>>>> index update be available fast after few seconds (NRT search) ? >>>>> >>>>> Do you see the hard commits in logs ? >>>>> Can you try to increase autoSoftCommit to 30 seconds or more ? >>>>> >>>>> Regards >>>>> >>>>> Dominique >>>>> >>>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <[email protected]> a >>>>> écrit : >>>>> >>>>> > Hi Guys >>>>> > >>>>> > We have faced an issue where tlog size is increasing unnecessarily. >>>>> We are >>>>> > using a "heavy indexing, heavy query" approach. We enabled hard >>>>> commit >>>>> > also, >>>>> > >>>>> > solr cloud: 6.6 >>>>> > zk: 3.4.10 >>>>> > shards: 2, replication factor= 2 >>>>> > >>>>> > >>>>> > solrconfig, >>>>> > >>>>> > <autoCommit> >>>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>>> > <maxDocs>10000</maxDocs> >>>>> > <openSearcher>false</openSearcher> >>>>> > >>>>> > >>>>> > </autoCommit> >>>>> > >>>>> > <!-- softAutoCommit is like autoCommit except it causes a >>>>> > 'soft' commit which only ensures that changes are visible >>>>> > but does not ensure that data is synced to disk. This is >>>>> > faster and more near-realtime friendly than a hard commit. >>>>> > --> >>>>> > >>>>> > <autoSoftCommit> >>>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>>> > </autoSoftCommit> >>>>> > >>>>> > In every replica, tlog is increasing for more than 200GB which >>>>> exhaust disk >>>>> > space. Please suggest someting >>>>> > >>>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <[email protected]> >>>>> wrote: >>>>> > >>>>> > > Hi Guys >>>>> > > >>>>> > > We have faced an issue where tlog size is increasing >>>>> unnecessarily. We >>>>> > are >>>>> > > using a "heavy indexing, heavy query" approach. We enabled hard >>>>> commit >>>>> > > also, >>>>> > > >>>>> > > solr cloud: 6.6 >>>>> > > zk: 3.4.10 >>>>> > > shards: 2, replication factor= 2 >>>>> > > >>>>> > > >>>>> > > solrconfig, >>>>> > > >>>>> > > <autoCommit> >>>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime> >>>>> > > <maxDocs>10000</maxDocs> >>>>> > > <openSearcher>false</openSearcher> >>>>> > > >>>>> > > >>>>> > > </autoCommit> >>>>> > > >>>>> > > <!-- softAutoCommit is like autoCommit except it causes a >>>>> > > 'soft' commit which only ensures that changes are visible >>>>> > > but does not ensure that data is synced to disk. This is >>>>> > > faster and more near-realtime friendly than a hard commit. >>>>> > > --> >>>>> > > >>>>> > > <autoSoftCommit> >>>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> --> >>>>> > > </autoSoftCommit> >>>>> > > >>>>> > > >>>>> > >>>>> >>>>
