[ https://issues.apache.org/jira/browse/OAK-5192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002715#comment-16002715 ]
Michael Dürig commented on OAK-5192: ------------------------------------ The script for generating above diff: {code} import $ivy.`michid:script-oak:latest.integration` import michid.script.oak._ import michid.script.oak.nodestore.Projection import org.apache.jackrabbit.oak.api.{PropertyState, Type} import org.apache.jackrabbit.oak.segment.SegmentBlob import michid.script.oak.nodestore.Changes._ import michid.script.oak.nodestore.ItemStates implicit val fs = fileStoreAnalyser(builder = dummyBlobStoreBuilder) val changes = fs.changes(Projection("root/oak:index/lucene")). map{case(diffs, date) => (date.getTime, diffs)}. filter{ case(_, diffs) => !diffs.isEmpty } def blobId(p: PropertyState) = p.analyse. values.filter(_.tyqe == Type.BINARY). map(_.value.asInstanceOf[SegmentBlob].getBlobId). toList val diff = changes.map{ case(timestamp, diffs) => (timestamp, diffs.map { case PropertyAdded(path, after) => (PropertyAdded(path, after), blobId(after), ItemStates.propertySize(false)(after)) case PropertyRemoved(path, before) => (PropertyRemoved(path, before), blobId(before), ItemStates.propertySize(false)(before)) case x => x })} browse(diff, viewer="vim", colors = pprint.Colors.BlackWhite) {code} > Reduce Lucene related growth of repository size > ----------------------------------------------- > > Key: OAK-5192 > URL: https://issues.apache.org/jira/browse/OAK-5192 > Project: Jackrabbit Oak > Issue Type: Improvement > Components: lucene, segment-tar > Reporter: Michael Dürig > Assignee: Tommaso Teofili > Labels: perfomance, scalability > Fix For: 1.8, 1.7.3 > > Attachments: added-bytes-zoom.png, diff.txt.zip > > > I observed Lucene indexing contributing to up to 99% of repository growth. > While the size of the index itself is well inside reasonable bounds, the > overall turnover of data being written and removed again can be as much as > 99%. > In the case of the TarMK this negatively impacts overall system performance > due to fast growing number of tar files / segments, bad locality of > reference, cache misses/thrashing when looking up segments and vastly > prolonged garbage collection cycles. -- This message was sent by Atlassian JIRA (v6.3.15#6346)