[
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16191703#comment-16191703
]
Erick Erickson commented on LUCENE-7976:
----------------------------------------
I linked in SOLR-7733 for dealing with the admin UI optimize button (I favor
removing it entirely, make people put in some _effort_ to back themselves into
a corner).
re: read-only rather than optimize.....
It may be the cases I've seen where users think optimize gives a speed
improvement are really the result of squeezing out the deleted documents.
Question for the Lucene folks, what would you guess the performance differences
would be between.
a single 200G segment?
40 5G segments?
With no deleted documents? I see indexes on disk at that size in the wild.
If the perf in the two cases above is "close enough" then freezing rather than
optimize is an easier sell. The rest of this JIRA is about keeping the %
deleted documents small, which, if we do, would handle the perf issues people
get currently from forceMerge, assuming the above.
[~msoko...@gmail.com] The delete percentage isn't really the issue currently,
if TMP respects max segment size it can't merge two segments > 50% live docs.
If TMP were tweaked to merge _unlike_ size segments when some % deleted docs is
exceeded in the large one (i.e. merge a segment with 4.75G live docs with a
segment with 0.25G live docs) we could get there.
[~mikemccand]:
bq: Right, but that's a normal/acceptable index state, where up to 50% of your
docs are deleted
Gotta disagree with acceptable, normal I'll grant. We're way over indexes being
terabytes and on our way to petabytes. I have cases where they're running out
of physical room to add more disks. Saying that half your disk space being
occupied by deleted documents is a hard sell.
> Add a parameter to TieredMergePolicy to merge segments that have more than X
> percent deleted documents
> ------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Erick Erickson
>
> We're seeing situations "in the wild" where there are very large indexes (on
> disk) handled quite easily in a single Lucene index. This is particularly
> true as features like docValues move data into MMapDirectory space. The
> current TMP algorithm allows on the order of 50% deleted documents as per a
> dev list conversation with Mike McCandless (and his blog here:
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many
> TB) solutions like "you need to distribute your collection over more shards"
> become very costly. Additionally, the tempting "optimize" button exacerbates
> the issue since once you form, say, a 100G segment (by
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like
> <maxAllowedPctDeletedInBigSegments> (no, that's not serious name, suggestions
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with
> >> smaller segments to bring the resulting segment up to 5G. If no smaller
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize).
> >> It would be rewritten into a single segment removing all deleted docs no
> >> matter how big it is to start. The 100G example above would be rewritten
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the
> default would be the same behavior we see now. As it stands now, though,
> there's no way to recover from an optimize/forceMerge except to re-index from
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the
> wild" with 10s of shards replicated 3 or more times. And that doesn't even
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A
> new merge policy is certainly an alternative.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
