Hi Erik,
the cause of having 2 tlog files but about 180 open file handles
was the point asking the question "when transaction logs are closing".
Second was the statement you also use "...every hard commit of any
flavor should close the current tlog".
This is somehow true, but I would expect that the meaning of
"close the current tlog" will physically close the tlog and release the
file handles. But actually the current tlog is only logically closed
with its END_MESSAGE "SOLR_TLOG_END" and _still_ physically open
because it is part of the current index e.g. for RTS.
I think this is a point which confuses most users about tlog.
Yes, optimize is part of our weekly bulk load process.
The question still stands, why are tlog files not closed physically
after full optimize down to 1 segment?
>From my point of view (with a huge static index built once a week)
the optimize is some kind of "sledgehammer" to force all data physically
to disk, wipe out all deleted docs and free any OS resources.
Regards
Bernd
Am 09.10.2017 um 19:46 schrieb Erick Erickson:
> bq: Actually there are only 2 tlog files but each java process has
> handles open to tlog.
>
> I'm a little confused by this. Each core in each Solr JVM should have
> a handle to it's _own_ tlog open. So if JVM1 has 10 cores (replicas),
> I'd expect 10 different tlogs to be open, one for each core. But I
> don't expect two processes to have a handle open to the _same_ tlog
> unless something's horribly wrong.
>
> And as Emir and Alessandro point out, every hard commit of any flavor
> should close the current tlog (for the replica/core) and open a new
> one. Soft commits have no effect.
>
> Oh, and please don't expungeDeletes or optimize if you can possibly
> help it, there'll be a blog coming soon on why this is A Bad Idea
> unless you have a pattern where you periodically (I'm thinking daily)
> update your index and optimize as part of your process.
>
> Best,
> Erick
>
> P.S. The reference guide is freely editable. Well, actually they're
> just files in asciidoc format. It'd be great if you wanted to edit
> them. I use Atom to edit them as it's free. If you do feel moved to do
> this, just raise a JIRA and add the diff as a patch. There's also an
> IntelliJ plugin that will do. I'm heavily editing the Near Real Time
> page...
>
> On Mon, Oct 9, 2017 at 5:35 AM, Bernd Fehling
> <bernd.fehl...@uni-bielefeld.de> wrote:
>> Thanks a lot Alessandro and Emir.
>>
>> Am 09.10.2017 um 13:40 schrieb alessandro.benedetti:
>>> In addition to what Emir mentioned, when Solr opens a new Transaction Log
>>> file it will delete the older ones up to some conditions :
>>> keep at least N number of records [1] and max K number of files[2].
>>> N is specified in the solrconfig.xml ( in the update handler section) and
>>> can be documents related or files related or both.
>>> So , potentially it could delete no one.
>>>
>>> This blog from Erick is quite explicative[3] .
>>> If you like to take a look to the code, this class should help[4]
>>>
>>>
>>>
>>> [1] <str name="numRecordsToKeep">${solr.ulog.numRecordsToKeep:100}</str>
>>> [2] <str name="maxNumLogsToKeep">${solr.ulog.maxNumLogsToKeep:10}</str>
>>> [3]
>>> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>> [4] org.apache.solr.update.UpdateLog
>>>
>>>
>>>
>>>
>>> -----
>>> ---------------
>>> Alessandro Benedetti
>>> Search Consultant, R&D Software Engineer, Director
>>> Sease Ltd. - www.sease.io
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>>
