Re: tlogs not deleting
Brian,
If you are still facing the issue after disabling buffer, kindly shut down
all the nodes at source and then start them again, stale tlogs will start
purging themselves.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2
On Wed, Jun 20, 2018 at 8:15 PM, Susheel Kumar
wrote:
> Not in my knowledge. Please double check or wait for some time but after
> DISABLEBUFFER on source, your logs should start rolling and its the exact
> same issue I have faced with 6.6 which you resolve by DISABLEBUFFER.
>
> On Tue, Jun 19, 2018 at 1:39 PM, Brian Yee wrote:
>
> > Does anyone have any additional possible causes for this issue? I checked
> > the buffer status using "/cdcr?action=STATUS" and it says buffer disabled
> > at both target and source.
> >
> > -Original Message-
> > From: Erick Erickson [mailto:erickerick...@gmail.com]
> > Sent: Tuesday, June 19, 2018 11:55 AM
> > To: solr-user
> > Subject: Re: tlogs not deleting
> >
> > bq. Do you recommend disabling the buffer on the source SolrCloud as
> well?
> >
> > Disable them all on both source and target IMO.
> >
> > On Tue, Jun 19, 2018 at 8:50 AM, Brian Yee wrote:
> > > Thank you Erick. I am running Solr 6.6. From the documentation:
> > > "Replicas do not need to buffer updates, and it is recommended to
> > disable buffer on the target SolrCloud."
> > >
> > > Do you recommend disabling the buffer on the source SolrCloud as well?
> > It looks like I already have the buffer disabled at target locations but
> > not the source location. Would it even make sense at the source location?
> > >
> > > This is what I have at the target locations:
> > >
> > >
> > > 100
> > >
> > >
> > > disabled
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Erick Erickson [mailto:erickerick...@gmail.com]
> > > Sent: Tuesday, June 19, 2018 11:00 AM
> > > To: solr-user
> > > Subject: Re: tlogs not deleting
> > >
> > > Take a look at the CDCR section of your reference guide, be sure you
> get
> > the version which you can download from here:
> > > https://archive.apache.org/dist/lucene/solr/ref-guide/
> > >
> > > There's the CDCR API call you can use for in-flight disabling, and
> > depending on the version of Solr you can set it in solrconfig.
> > >
> > > Basically, buffering was there in the original CDCR to allow a larger
> > maintenance window, you could enable buffering and all updates were saved
> > until you disabled it, during which period you could do whatever you
> needed
> > with your target cluster and not lose any updates.
> > >
> > > Later versions can do the full sync of the index and buffering is being
> > removed.
> > >
> > > Best,
> > > Erick
> > >
> > > On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
> > >> Thanks for the suggestion. Can you please elaborate a little bit about
> > what DISABLEBUFFER does? The documentation is not very detailed. Is this
> > something that needs to be done manually whenever this problem happens or
> > is it something that we can do to fix it so it won't happen again?
> > >>
> > >> -Original Message-
> > >> From: Susheel Kumar [mailto:susheel2...@gmail.com]
> > >> Sent: Monday, June 18, 2018 9:12 PM
> > >> To: solr-user@lucene.apache.org
> > >> Subject: Re: tlogs not deleting
> > >>
> > >> You may have to DISABLEBUFFER in source to get rid of tlogs.
> > >>
> > >> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
> > >>
> > >>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
> > >>> understand, after a hard commit, solr is supposed to delete old
> > >>> tlogs depending on the numRecordsToKeep and maxNumLogsToKeep values
> > >>> in the autocommit settings in solrconfig.xml. I am occasionally
> > >>> seeing solr fail to do this and the tlogs just build up over time
> > >>> and eventually we run out of disk space on the VM and this causes
> > problems for us.
> > >>> This does not happen all the time, only sometimes. I currently have
> > >>> a tlog directory that has 123G worth of tlogs. The last hard commit
> > >>> on this node was 10 minutes ago but these tlogs date back to 3 days
> > ago.
> > >>>
> > >>> We have sometimes found that restarting solr on the node will get it
> > >>> to clean up the old tlogs, but we really want to find the root cause
> > >>> and fix it if possible so we don't keep getting disk space alerts
> > >>> and have to adhoc restart nodes. Has anyone seen an issue like this
> > before?
> > >>>
> > >>> My update handler settings look like this:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> ${solr.ulog.dir:}
> > >>> ${solr.ulog.numVersionBuckets:
> > >>> 65536}
> > >>>
> > >>>
> > >>> 60
> > >>> 25
> > >>> false
> > >>>
> > >>>
> > >>> 12
> > >>>
> > >>>
> > >>>
> > >>> 100
> > >>>
> > >>>
> > >>>
> > >>>
> >
>
Re: tlogs not deleting
Not in my knowledge. Please double check or wait for some time but after
DISABLEBUFFER on source, your logs should start rolling and its the exact
same issue I have faced with 6.6 which you resolve by DISABLEBUFFER.
On Tue, Jun 19, 2018 at 1:39 PM, Brian Yee wrote:
> Does anyone have any additional possible causes for this issue? I checked
> the buffer status using "/cdcr?action=STATUS" and it says buffer disabled
> at both target and source.
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, June 19, 2018 11:55 AM
> To: solr-user
> Subject: Re: tlogs not deleting
>
> bq. Do you recommend disabling the buffer on the source SolrCloud as well?
>
> Disable them all on both source and target IMO.
>
> On Tue, Jun 19, 2018 at 8:50 AM, Brian Yee wrote:
> > Thank you Erick. I am running Solr 6.6. From the documentation:
> > "Replicas do not need to buffer updates, and it is recommended to
> disable buffer on the target SolrCloud."
> >
> > Do you recommend disabling the buffer on the source SolrCloud as well?
> It looks like I already have the buffer disabled at target locations but
> not the source location. Would it even make sense at the source location?
> >
> > This is what I have at the target locations:
> >
> >
> > 100
> >
> >
> > disabled
> >
> >
> >
> >
> > -Original Message-
> > From: Erick Erickson [mailto:erickerick...@gmail.com]
> > Sent: Tuesday, June 19, 2018 11:00 AM
> > To: solr-user
> > Subject: Re: tlogs not deleting
> >
> > Take a look at the CDCR section of your reference guide, be sure you get
> the version which you can download from here:
> > https://archive.apache.org/dist/lucene/solr/ref-guide/
> >
> > There's the CDCR API call you can use for in-flight disabling, and
> depending on the version of Solr you can set it in solrconfig.
> >
> > Basically, buffering was there in the original CDCR to allow a larger
> maintenance window, you could enable buffering and all updates were saved
> until you disabled it, during which period you could do whatever you needed
> with your target cluster and not lose any updates.
> >
> > Later versions can do the full sync of the index and buffering is being
> removed.
> >
> > Best,
> > Erick
> >
> > On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
> >> Thanks for the suggestion. Can you please elaborate a little bit about
> what DISABLEBUFFER does? The documentation is not very detailed. Is this
> something that needs to be done manually whenever this problem happens or
> is it something that we can do to fix it so it won't happen again?
> >>
> >> -Original Message-
> >> From: Susheel Kumar [mailto:susheel2...@gmail.com]
> >> Sent: Monday, June 18, 2018 9:12 PM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: tlogs not deleting
> >>
> >> You may have to DISABLEBUFFER in source to get rid of tlogs.
> >>
> >> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
> >>
> >>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
> >>> understand, after a hard commit, solr is supposed to delete old
> >>> tlogs depending on the numRecordsToKeep and maxNumLogsToKeep values
> >>> in the autocommit settings in solrconfig.xml. I am occasionally
> >>> seeing solr fail to do this and the tlogs just build up over time
> >>> and eventually we run out of disk space on the VM and this causes
> problems for us.
> >>> This does not happen all the time, only sometimes. I currently have
> >>> a tlog directory that has 123G worth of tlogs. The last hard commit
> >>> on this node was 10 minutes ago but these tlogs date back to 3 days
> ago.
> >>>
> >>> We have sometimes found that restarting solr on the node will get it
> >>> to clean up the old tlogs, but we really want to find the root cause
> >>> and fix it if possible so we don't keep getting disk space alerts
> >>> and have to adhoc restart nodes. Has anyone seen an issue like this
> before?
> >>>
> >>> My update handler settings look like this:
> >>>
> >>>
> >>>
> >>>
> >>> ${solr.ulog.dir:}
> >>> ${solr.ulog.numVersionBuckets:
> >>> 65536}
> >>>
> >>>
> >>> 60
> >>> 25
> >>> false
> >>>
> >>>
> >>> 12
> >>>
> >>>
> >>>
> >>> 100
> >>>
> >>>
> >>>
> >>>
>
RE: tlogs not deleting
Does anyone have any additional possible causes for this issue? I checked the
buffer status using "/cdcr?action=STATUS" and it says buffer disabled at both
target and source.
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Tuesday, June 19, 2018 11:55 AM
To: solr-user
Subject: Re: tlogs not deleting
bq. Do you recommend disabling the buffer on the source SolrCloud as well?
Disable them all on both source and target IMO.
On Tue, Jun 19, 2018 at 8:50 AM, Brian Yee wrote:
> Thank you Erick. I am running Solr 6.6. From the documentation:
> "Replicas do not need to buffer updates, and it is recommended to disable
> buffer on the target SolrCloud."
>
> Do you recommend disabling the buffer on the source SolrCloud as well? It
> looks like I already have the buffer disabled at target locations but not the
> source location. Would it even make sense at the source location?
>
> This is what I have at the target locations:
>
>
> 100
>
>
> disabled
>
>
>
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, June 19, 2018 11:00 AM
> To: solr-user
> Subject: Re: tlogs not deleting
>
> Take a look at the CDCR section of your reference guide, be sure you get the
> version which you can download from here:
> https://archive.apache.org/dist/lucene/solr/ref-guide/
>
> There's the CDCR API call you can use for in-flight disabling, and depending
> on the version of Solr you can set it in solrconfig.
>
> Basically, buffering was there in the original CDCR to allow a larger
> maintenance window, you could enable buffering and all updates were saved
> until you disabled it, during which period you could do whatever you needed
> with your target cluster and not lose any updates.
>
> Later versions can do the full sync of the index and buffering is being
> removed.
>
> Best,
> Erick
>
> On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
>> Thanks for the suggestion. Can you please elaborate a little bit about what
>> DISABLEBUFFER does? The documentation is not very detailed. Is this
>> something that needs to be done manually whenever this problem happens or is
>> it something that we can do to fix it so it won't happen again?
>>
>> -Original Message-
>> From: Susheel Kumar [mailto:susheel2...@gmail.com]
>> Sent: Monday, June 18, 2018 9:12 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: tlogs not deleting
>>
>> You may have to DISABLEBUFFER in source to get rid of tlogs.
>>
>> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
>>
>>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
>>> understand, after a hard commit, solr is supposed to delete old
>>> tlogs depending on the numRecordsToKeep and maxNumLogsToKeep values
>>> in the autocommit settings in solrconfig.xml. I am occasionally
>>> seeing solr fail to do this and the tlogs just build up over time
>>> and eventually we run out of disk space on the VM and this causes problems
>>> for us.
>>> This does not happen all the time, only sometimes. I currently have
>>> a tlog directory that has 123G worth of tlogs. The last hard commit
>>> on this node was 10 minutes ago but these tlogs date back to 3 days ago.
>>>
>>> We have sometimes found that restarting solr on the node will get it
>>> to clean up the old tlogs, but we really want to find the root cause
>>> and fix it if possible so we don't keep getting disk space alerts
>>> and have to adhoc restart nodes. Has anyone seen an issue like this before?
>>>
>>> My update handler settings look like this:
>>>
>>>
>>>
>>>
>>> ${solr.ulog.dir:}
>>> ${solr.ulog.numVersionBuckets:
>>> 65536}
>>>
>>>
>>> 60
>>> 25
>>> false
>>>
>>>
>>> 12
>>>
>>>
>>>
>>> 100
>>>
>>>
>>>
>>>
Re: tlogs not deleting
bq. Do you recommend disabling the buffer on the source SolrCloud as well?
Disable them all on both source and target IMO.
On Tue, Jun 19, 2018 at 8:50 AM, Brian Yee wrote:
> Thank you Erick. I am running Solr 6.6. From the documentation:
> "Replicas do not need to buffer updates, and it is recommended to disable
> buffer on the target SolrCloud."
>
> Do you recommend disabling the buffer on the source SolrCloud as well? It
> looks like I already have the buffer disabled at target locations but not the
> source location. Would it even make sense at the source location?
>
> This is what I have at the target locations:
>
>
> 100
>
>
> disabled
>
>
>
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Tuesday, June 19, 2018 11:00 AM
> To: solr-user
> Subject: Re: tlogs not deleting
>
> Take a look at the CDCR section of your reference guide, be sure you get the
> version which you can download from here:
> https://archive.apache.org/dist/lucene/solr/ref-guide/
>
> There's the CDCR API call you can use for in-flight disabling, and depending
> on the version of Solr you can set it in solrconfig.
>
> Basically, buffering was there in the original CDCR to allow a larger
> maintenance window, you could enable buffering and all updates were saved
> until you disabled it, during which period you could do whatever you needed
> with your target cluster and not lose any updates.
>
> Later versions can do the full sync of the index and buffering is being
> removed.
>
> Best,
> Erick
>
> On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
>> Thanks for the suggestion. Can you please elaborate a little bit about what
>> DISABLEBUFFER does? The documentation is not very detailed. Is this
>> something that needs to be done manually whenever this problem happens or is
>> it something that we can do to fix it so it won't happen again?
>>
>> -Original Message-
>> From: Susheel Kumar [mailto:susheel2...@gmail.com]
>> Sent: Monday, June 18, 2018 9:12 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: tlogs not deleting
>>
>> You may have to DISABLEBUFFER in source to get rid of tlogs.
>>
>> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
>>
>>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
>>> understand, after a hard commit, solr is supposed to delete old tlogs
>>> depending on the numRecordsToKeep and maxNumLogsToKeep values in the
>>> autocommit settings in solrconfig.xml. I am occasionally seeing solr
>>> fail to do this and the tlogs just build up over time and eventually
>>> we run out of disk space on the VM and this causes problems for us.
>>> This does not happen all the time, only sometimes. I currently have a
>>> tlog directory that has 123G worth of tlogs. The last hard commit on
>>> this node was 10 minutes ago but these tlogs date back to 3 days ago.
>>>
>>> We have sometimes found that restarting solr on the node will get it
>>> to clean up the old tlogs, but we really want to find the root cause
>>> and fix it if possible so we don't keep getting disk space alerts and
>>> have to adhoc restart nodes. Has anyone seen an issue like this before?
>>>
>>> My update handler settings look like this:
>>>
>>>
>>>
>>>
>>> ${solr.ulog.dir:}
>>> ${solr.ulog.numVersionBuckets:
>>> 65536}
>>>
>>>
>>> 60
>>> 25
>>> false
>>>
>>>
>>> 12
>>>
>>>
>>>
>>> 100
>>>
>>>
>>>
>>>
RE: tlogs not deleting
Thank you Erick. I am running Solr 6.6. From the documentation:
"Replicas do not need to buffer updates, and it is recommended to disable
buffer on the target SolrCloud."
Do you recommend disabling the buffer on the source SolrCloud as well? It looks
like I already have the buffer disabled at target locations but not the source
location. Would it even make sense at the source location?
This is what I have at the target locations:
100
disabled
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Tuesday, June 19, 2018 11:00 AM
To: solr-user
Subject: Re: tlogs not deleting
Take a look at the CDCR section of your reference guide, be sure you get the
version which you can download from here:
https://archive.apache.org/dist/lucene/solr/ref-guide/
There's the CDCR API call you can use for in-flight disabling, and depending on
the version of Solr you can set it in solrconfig.
Basically, buffering was there in the original CDCR to allow a larger
maintenance window, you could enable buffering and all updates were saved until
you disabled it, during which period you could do whatever you needed with your
target cluster and not lose any updates.
Later versions can do the full sync of the index and buffering is being removed.
Best,
Erick
On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
> Thanks for the suggestion. Can you please elaborate a little bit about what
> DISABLEBUFFER does? The documentation is not very detailed. Is this something
> that needs to be done manually whenever this problem happens or is it
> something that we can do to fix it so it won't happen again?
>
> -Original Message-
> From: Susheel Kumar [mailto:susheel2...@gmail.com]
> Sent: Monday, June 18, 2018 9:12 PM
> To: solr-user@lucene.apache.org
> Subject: Re: tlogs not deleting
>
> You may have to DISABLEBUFFER in source to get rid of tlogs.
>
> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
>
>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
>> understand, after a hard commit, solr is supposed to delete old tlogs
>> depending on the numRecordsToKeep and maxNumLogsToKeep values in the
>> autocommit settings in solrconfig.xml. I am occasionally seeing solr
>> fail to do this and the tlogs just build up over time and eventually
>> we run out of disk space on the VM and this causes problems for us.
>> This does not happen all the time, only sometimes. I currently have a
>> tlog directory that has 123G worth of tlogs. The last hard commit on
>> this node was 10 minutes ago but these tlogs date back to 3 days ago.
>>
>> We have sometimes found that restarting solr on the node will get it
>> to clean up the old tlogs, but we really want to find the root cause
>> and fix it if possible so we don't keep getting disk space alerts and
>> have to adhoc restart nodes. Has anyone seen an issue like this before?
>>
>> My update handler settings look like this:
>>
>>
>>
>>
>> ${solr.ulog.dir:}
>> ${solr.ulog.numVersionBuckets:
>> 65536}
>>
>>
>> 60
>> 25
>> false
>>
>>
>> 12
>>
>>
>>
>> 100
>>
>>
>>
>>
Re: tlogs not deleting
Take a look at the CDCR section of your reference guide, be sure you
get the version which you can download from here:
https://archive.apache.org/dist/lucene/solr/ref-guide/
There's the CDCR API call you can use for in-flight disabling, and
depending on the version of Solr you can set it in solrconfig.
Basically, buffering was there in the original CDCR to allow a larger
maintenance window, you could enable buffering and all updates were
saved until you disabled it, during which period you could do whatever
you needed with your target cluster and not lose any updates.
Later versions can do the full sync of the index and buffering is being removed.
Best,
Erick
On Tue, Jun 19, 2018 at 7:31 AM, Brian Yee wrote:
> Thanks for the suggestion. Can you please elaborate a little bit about what
> DISABLEBUFFER does? The documentation is not very detailed. Is this something
> that needs to be done manually whenever this problem happens or is it
> something that we can do to fix it so it won't happen again?
>
> -Original Message-
> From: Susheel Kumar [mailto:susheel2...@gmail.com]
> Sent: Monday, June 18, 2018 9:12 PM
> To: solr-user@lucene.apache.org
> Subject: Re: tlogs not deleting
>
> You may have to DISABLEBUFFER in source to get rid of tlogs.
>
> On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
>
>> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
>> understand, after a hard commit, solr is supposed to delete old tlogs
>> depending on the numRecordsToKeep and maxNumLogsToKeep values in the
>> autocommit settings in solrconfig.xml. I am occasionally seeing solr
>> fail to do this and the tlogs just build up over time and eventually
>> we run out of disk space on the VM and this causes problems for us.
>> This does not happen all the time, only sometimes. I currently have a
>> tlog directory that has 123G worth of tlogs. The last hard commit on
>> this node was 10 minutes ago but these tlogs date back to 3 days ago.
>>
>> We have sometimes found that restarting solr on the node will get it
>> to clean up the old tlogs, but we really want to find the root cause
>> and fix it if possible so we don't keep getting disk space alerts and
>> have to adhoc restart nodes. Has anyone seen an issue like this before?
>>
>> My update handler settings look like this:
>>
>>
>>
>>
>> ${solr.ulog.dir:}
>> ${solr.ulog.numVersionBuckets:
>> 65536}
>>
>>
>> 60
>> 25
>> false
>>
>>
>> 12
>>
>>
>>
>> 100
>>
>>
>>
>>
RE: tlogs not deleting
Thanks for the suggestion. Can you please elaborate a little bit about what
DISABLEBUFFER does? The documentation is not very detailed. Is this something
that needs to be done manually whenever this problem happens or is it something
that we can do to fix it so it won't happen again?
-Original Message-
From: Susheel Kumar [mailto:susheel2...@gmail.com]
Sent: Monday, June 18, 2018 9:12 PM
To: solr-user@lucene.apache.org
Subject: Re: tlogs not deleting
You may have to DISABLEBUFFER in source to get rid of tlogs.
On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
> understand, after a hard commit, solr is supposed to delete old tlogs
> depending on the numRecordsToKeep and maxNumLogsToKeep values in the
> autocommit settings in solrconfig.xml. I am occasionally seeing solr
> fail to do this and the tlogs just build up over time and eventually
> we run out of disk space on the VM and this causes problems for us.
> This does not happen all the time, only sometimes. I currently have a
> tlog directory that has 123G worth of tlogs. The last hard commit on
> this node was 10 minutes ago but these tlogs date back to 3 days ago.
>
> We have sometimes found that restarting solr on the node will get it
> to clean up the old tlogs, but we really want to find the root cause
> and fix it if possible so we don't keep getting disk space alerts and
> have to adhoc restart nodes. Has anyone seen an issue like this before?
>
> My update handler settings look like this:
>
>
>
>
> ${solr.ulog.dir:}
> ${solr.ulog.numVersionBuckets:
> 65536}
>
>
> 60
> 25
> false
>
>
> 12
>
>
>
> 100
>
>
>
>
Re: tlogs not deleting
You may have to DISABLEBUFFER in source to get rid of tlogs.
On Mon, Jun 18, 2018 at 6:13 PM, Brian Yee wrote:
> So I've read a bunch of stuff on hard/soft commits and tlogs. As I
> understand, after a hard commit, solr is supposed to delete old tlogs
> depending on the numRecordsToKeep and maxNumLogsToKeep values in the
> autocommit settings in solrconfig.xml. I am occasionally seeing solr fail
> to do this and the tlogs just build up over time and eventually we run out
> of disk space on the VM and this causes problems for us. This does not
> happen all the time, only sometimes. I currently have a tlog directory that
> has 123G worth of tlogs. The last hard commit on this node was 10 minutes
> ago but these tlogs date back to 3 days ago.
>
> We have sometimes found that restarting solr on the node will get it to
> clean up the old tlogs, but we really want to find the root cause and fix
> it if possible so we don't keep getting disk space alerts and have to adhoc
> restart nodes. Has anyone seen an issue like this before?
>
> My update handler settings look like this:
>
>
>
>
> ${solr.ulog.dir:}
> ${solr.ulog.numVersionBuckets:
> 65536}
>
>
> 60
> 25
> false
>
>
> 12
>
>
>
> 100
>
>
>
>
tlogs not deleting
So I've read a bunch of stuff on hard/soft commits and tlogs. As I understand,
after a hard commit, solr is supposed to delete old tlogs depending on the
numRecordsToKeep and maxNumLogsToKeep values in the autocommit settings in
solrconfig.xml. I am occasionally seeing solr fail to do this and the tlogs
just build up over time and eventually we run out of disk space on the VM and
this causes problems for us. This does not happen all the time, only sometimes.
I currently have a tlog directory that has 123G worth of tlogs. The last hard
commit on this node was 10 minutes ago but these tlogs date back to 3 days ago.
We have sometimes found that restarting solr on the node will get it to clean
up the old tlogs, but we really want to find the root cause and fix it if
possible so we don't keep getting disk space alerts and have to adhoc restart
nodes. Has anyone seen an issue like this before?
My update handler settings look like this:
${solr.ulog.dir:}
${solr.ulog.numVersionBuckets:65536}
60
25
false
12
100
Re: tlogs not deleting as usual in Solr 5.5.1?
On 6/16/2016 1:18 PM, Chris Morley wrote: > In Solr 4.10.3 tlogs ARE deleted after issuing update?commit=true. > (And deleted immediately.) That seems like a bug to me. A hard commit is supposed to close the current transaction log and prune old logs such that what's left will meet the "keep" requirements. It is not designed to delete ALL of the transaction logs. > In Solr 5.5.1 tlogs ARE NOT deleted after issuing update?commit=true. As I understand it, the default "keep requirements" are 10 logs or 100 documents. My guess would be that this means that if each transaction log only contains 1 document, Solr would keep 100 of them ... but if the newest transaction log (the one just closed) contains a million documents, it might only keep the newest one. The hard commit will trim the oldest logs to meet these requirements. This means that if you index a gigabyte or more of data without any commits, and then do a hard commit, Solr will keep at LEAST that latest and largest logfile around, because it's the only way it can guarantee a replay of at least 100 documents. The large log will stick around until additional logfiles are created that meet the requirement. Thanks, Shawn
Re: tlogs not deleting as usual in Solr 5.5.1?
Thanks Erick - that's what we have settled on doing until we are using SolrCloud, which will be later this year with any luck. We want to get up onto Solr 5.5.1 first (ASAP) and we tried disabling tlogs today and that seems to fit the bill. From: "Erick Erickson" Sent: Friday, June 17, 2016 2:36 PM To: "solr-user" , ch...@depahelix.com Subject: Re: tlogs not deleting as usual in Solr 5.5.1? If you are NOT using SolrCloud and don't care about Real Time Get, you can just disable the tlogs entirely. They're not doing you all that much good in that case... The tlogs are irrelevant when it comes to master/slave replication. FWIW, Erick On Fri, Jun 17, 2016 at 9:14 AM, Chris Morley wrote: > After some more searching, I found a thread online where Erick Erickson is > telling someone about how there are old tlogs left around in case there is > a need for a peer to sync even if SolrCloud is not enabled. That makes > sense, but we'll probably want to enable autoCommit and then trigger > replication on the slaves when we know everything is committed after a full > import. (We disable polling.) > > > > > > From: "Chris Morley" > Sent: Thursday, June 16, 2016 3:20 PM > To: "Solr Newsgroup" > Subject: tlogs not deleting as usual in Solr 5.5.1? > The repetition below is on purpose to show the contrast between solr > versions. > > In Solr 4.10.3, we have autocommits disabled. We do a dataimport of a few > hundred thousand records and have a tlog that grows to ~1.2G. > > In Solr 5.5.1, we have autocommits disabled. We do a dataimport of a few > hundred thousand records and have a tlog that grows to ~1.6G. (same exact > data, slightly larger tlog but who knows, that's fine) > > In Solr 4.10.3 tlogs ARE deleted after issuing update?commit=true. > (And deleted immediately.) > > In Solr 5.5.1 tlogs ARE NOT deleted after issuing update?commit=true. > > We want the tlog to delete like it did in Solr 4.10.3. Perhaps there is a > configuration setting or feature of Solr 5.5.1 that causes this? > > Would appreciate any tips on configuration or code we could change to > ensure the tlog will delete after a hard commit. > > >
Re: tlogs not deleting as usual in Solr 5.5.1?
If you are NOT using SolrCloud and don't care about Real Time Get, you can just disable the tlogs entirely. They're not doing you all that much good in that case... The tlogs are irrelevant when it comes to master/slave replication. FWIW, Erick On Fri, Jun 17, 2016 at 9:14 AM, Chris Morley wrote: > After some more searching, I found a thread online where Erick Erickson is > telling someone about how there are old tlogs left around in case there is > a need for a peer to sync even if SolrCloud is not enabled. That makes > sense, but we'll probably want to enable autoCommit and then trigger > replication on the slaves when we know everything is committed after a full > import. (We disable polling.) > > > > > > From: "Chris Morley" > Sent: Thursday, June 16, 2016 3:20 PM > To: "Solr Newsgroup" > Subject: tlogs not deleting as usual in Solr 5.5.1? > The repetition below is on purpose to show the contrast between solr > versions. > > In Solr 4.10.3, we have autocommits disabled. We do a dataimport of a few > hundred thousand records and have a tlog that grows to ~1.2G. > > In Solr 5.5.1, we have autocommits disabled. We do a dataimport of a few > hundred thousand records and have a tlog that grows to ~1.6G. (same exact > data, slightly larger tlog but who knows, that's fine) > > In Solr 4.10.3 tlogs ARE deleted after issuing update?commit=true. > (And deleted immediately.) > > In Solr 5.5.1 tlogs ARE NOT deleted after issuing update?commit=true. > > We want the tlog to delete like it did in Solr 4.10.3. Perhaps there is a > configuration setting or feature of Solr 5.5.1 that causes this? > > Would appreciate any tips on configuration or code we could change to > ensure the tlog will delete after a hard commit. > > >
re: tlogs not deleting as usual in Solr 5.5.1?
After some more searching, I found a thread online where Erick Erickson is telling someone about how there are old tlogs left around in case there is a need for a peer to sync even if SolrCloud is not enabled. That makes sense, but we'll probably want to enable autoCommit and then trigger replication on the slaves when we know everything is committed after a full import. (We disable polling.) From: "Chris Morley" Sent: Thursday, June 16, 2016 3:20 PM To: "Solr Newsgroup" Subject: tlogs not deleting as usual in Solr 5.5.1? The repetition below is on purpose to show the contrast between solr versions. In Solr 4.10.3, we have autocommits disabled. We do a dataimport of a few hundred thousand records and have a tlog that grows to ~1.2G. In Solr 5.5.1, we have autocommits disabled. We do a dataimport of a few hundred thousand records and have a tlog that grows to ~1.6G. (same exact data, slightly larger tlog but who knows, that's fine) In Solr 4.10.3 tlogs ARE deleted after issuing update?commit=true. (And deleted immediately.) In Solr 5.5.1 tlogs ARE NOT deleted after issuing update?commit=true. We want the tlog to delete like it did in Solr 4.10.3. Perhaps there is a configuration setting or feature of Solr 5.5.1 that causes this? Would appreciate any tips on configuration or code we could change to ensure the tlog will delete after a hard commit.
tlogs not deleting as usual in Solr 5.5.1?
The repetition below is on purpose to show the contrast between solr versions. In Solr 4.10.3, we have autocommits disabled. We do a dataimport of a few hundred thousand records and have a tlog that grows to ~1.2G. In Solr 5.5.1, we have autocommits disabled. We do a dataimport of a few hundred thousand records and have a tlog that grows to ~1.6G. (same exact data, slightly larger tlog but who knows, that's fine) In Solr 4.10.3 tlogs ARE deleted after issuing update?commit=true. (And deleted immediately.) In Solr 5.5.1 tlogs ARE NOT deleted after issuing update?commit=true. We want the tlog to delete like it did in Solr 4.10.3. Perhaps there is a configuration setting or feature of Solr 5.5.1 that causes this? Would appreciate any tips on configuration or code we could change to ensure the tlog will delete after a hard commit.
