HI Dominique
softcommit=false is coming in logs,
INFO - 2021-03-22 07:40:58.129; [c:solrcollection s:shard2 r:core_node1
x:solrcollection_shard2_replica1]
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;
[solrcollection_shard2_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&
*openSearcher=true&commit=true&softCommit=false*&distrib.from=
http://xxxx:8983/solr/solrcollection_shard2_replica1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 676
INFO - 2021-03-22 07:40:58.361; [c:solrcollection s:shard2 r:core_node1
x:solrcollection_shard2_replica1]
org.apache.solr.update.DirectUpdateHandler2; start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
On Sun, 21 Mar 2021 at 21:06, Dominique Bejean <[email protected]>
wrote:
> Hi,
>
> If your tlogs grow, It looks like hard commit are never executed.
>
> Are you seeing hard commits in logs (grep 'softCommit=false') ?
>
> What is it happening if you force a hard commit ? Are tlog truncated ?
> http://localhost:8983/solr/[collection_name]/update?commit=true
>
> Did you read this article ?
>
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Regards
>
> Dominique
>
>
> Le dim. 21 mars 2021 à 05:22, Ritvik Sharma <[email protected]> a
> écrit :
>
>> Hi Dominique
>>
>> Heavy query means high query rate on solr.
>>
>> Honestly for 2-3 days we have stopped the queries on solr only we are
>> doing indexing !!!
>> As you see we have enabled Hard commit also to decrease tlog log size or
>> more and the same is written on solr docs. Still the same behaviour is
>> occurring on replicas. In some replicas, tlog size is of 300 GB.
>>
>> We may increase softcommit timeout but as per my understanding it only
>> helps in searching availability prior to commit to indexes.
>>
>> On Sun, 21 Mar 2021 at 4:29 AM, Dominique Bejean <
>> [email protected]> wrote:
>>
>>> Hi,
>>>
>>> By heavy query, do you mean you have a high query rate and/or you need
>>> index update be available fast after few seconds (NRT search) ?
>>>
>>> Do you see the hard commits in logs ?
>>> Can you try to increase autoSoftCommit to 30 seconds or more ?
>>>
>>> Regards
>>>
>>> Dominique
>>>
>>> Le sam. 20 mars 2021 à 18:53, Ritvik Sharma <[email protected]> a
>>> écrit :
>>>
>>> > Hi Guys
>>> >
>>> > We have faced an issue where tlog size is increasing unnecessarily. We
>>> are
>>> > using a "heavy indexing, heavy query" approach. We enabled hard commit
>>> > also,
>>> >
>>> > solr cloud: 6.6
>>> > zk: 3.4.10
>>> > shards: 2, replication factor= 2
>>> >
>>> >
>>> > solrconfig,
>>> >
>>> > <autoCommit>
>>> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> > <maxDocs>10000</maxDocs>
>>> > <openSearcher>false</openSearcher>
>>> >
>>> >
>>> > </autoCommit>
>>> >
>>> > <!-- softAutoCommit is like autoCommit except it causes a
>>> > 'soft' commit which only ensures that changes are visible
>>> > but does not ensure that data is synced to disk. This is
>>> > faster and more near-realtime friendly than a hard commit.
>>> > -->
>>> >
>>> > <autoSoftCommit>
>>> > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>> > </autoSoftCommit>
>>> >
>>> > In every replica, tlog is increasing for more than 200GB which exhaust
>>> disk
>>> > space. Please suggest someting
>>> >
>>> > On Fri, 19 Mar 2021 at 16:36, Ritvik Sharma <[email protected]>
>>> wrote:
>>> >
>>> > > Hi Guys
>>> > >
>>> > > We have faced an issue where tlog size is increasing unnecessarily.
>>> We
>>> > are
>>> > > using a "heavy indexing, heavy query" approach. We enabled hard
>>> commit
>>> > > also,
>>> > >
>>> > > solr cloud: 6.6
>>> > > zk: 3.4.10
>>> > > shards: 2, replication factor= 2
>>> > >
>>> > >
>>> > > solrconfig,
>>> > >
>>> > > <autoCommit>
>>> > > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> > > <maxDocs>10000</maxDocs>
>>> > > <openSearcher>false</openSearcher>
>>> > >
>>> > >
>>> > > </autoCommit>
>>> > >
>>> > > <!-- softAutoCommit is like autoCommit except it causes a
>>> > > 'soft' commit which only ensures that changes are visible
>>> > > but does not ensure that data is synced to disk. This is
>>> > > faster and more near-realtime friendly than a hard commit.
>>> > > -->
>>> > >
>>> > > <autoSoftCommit>
>>> > > <maxTime>${solr.autoSoftCommit.maxTime:15000}</maxTime> -->
>>> > > </autoSoftCommit>
>>> > >
>>> > >
>>> >
>>>
>>
