Are larger and older segments even certain to ever be merged in practice? I was assuming that if there is not a lot of new indexed content and not a lot of older documents being deleted, large older segment might never have to be merged.
On Tue 28 Nov 2023 at 20:53, Robert Muir <rcm...@gmail.com> wrote: > I don't think there's any problem with GDPR, and I don't think users > should be running unnecessary "optimize". GDRP just says data should > be erased without "undue" delay. waiting for a merge to nuke the > deleted docs isn't "undue", there is a good reason for it. > > On Tue, Nov 28, 2023 at 2:40 PM Patrick Zhai <zhai7...@gmail.com> wrote: > > > > Hi Folks, > > In LinkedIn we need to comply with GDPR for a large part of our data, > and an important part of it is that we need to be sure we have completely > deleted the data the user requested to delete within a certain period of > time. > > The way we have come up with so far is to: > > 1. Record the segment creation time somewhere (not decided yet, maybe > index commit userinfo, maybe some other place outside of lucene) > > 2. Create a new merge policy which delegate most operations to a normal > MP, like TieredMergePolicy, and then add extra single-segment (merge from 1 > segment to 1 segment, basically only do deletion) merges if it finds any > segment is about to violate the GDPR time frame. > > > > So here's my question: > > 1. Is there a better/existing way to do this? > > 2. I would like to directly contribute to Lucene about such a merge > policy since I think GDPR is more or less a common thing. Would like to > know whether people feel like it's necessary or not? > > 3. It's also nice if we can store the segment creation time to the index > directly by IndexWriter (maybe write to SegmentInfo?), I can try to do that > but would like to ask whether there's any objections? > > > > Best > > Patrick > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
