[
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866436#comment-16866436
]
Adrien Grand commented on LUCENE-8862:
--------------------------------------
Could you pass the MemoryTrackerObject that you mentioned above at construction
time? I have something like that in mind:
{code:java}
class BitSetCollector extends SimpleCollector {
final MemoryTrackerObject tracker;
FixedBitSet bitSet = new FixedBitSet(0);
int length = 0;
int docBase = 0;
BitSetCollector(MemoryTrackerObject tracker) {
this.tracker = tracker;
tracker.addAndGet(bitSet.ramBytesUsed());
}
BitSet getBits() {
return bitSet;
}
void doSetNextReader(LeafReaderContext context) throws IOException {
length += context.reader().maxDoc();
FixedBitSet newBitSet = FixedBitSet.ensureCapacity(bitSet, length);
if (newBitSet != bitSet) {
tracker.addAndGet(newBitSet.ramBytesUsed() - bitSet.ramBytesUsed()); //
ideally we'd do that before actually allocating
bitSet = newBitSet;
}
}
void collect(int doc) {
bitSet.set(docBase + doc);
}
}
long perCollectorMemoryLimit = 5 * 1024L * 1024L; // 5MB
MemoryTrackerObject tracker = new MemoryTrackerObject() {
long ramBytesUsed = 0;
long addAndGet(long delta) {
long newRamBytesUsed = ramBytesUsed + delta;
if (newRamBytesUsed > limit) {
throw new Exception();
}
ramBytesUsed = newRamBytesUsed;
return ramBytesUsed;
}
};
BitSetCollector bitSetCollector = new BitSetCollector(tracker);
TotalHitCountCollector hitCountCollector = new TotalHitCountCollector();
searcher.search(new TermQuery(new Term("foo", "bar")),
MultiCollector.wrap(hitCountCollector, bitSetCollector));
int hitCount = hitCountCollector.getTotalHits();
BitSet hits = bitSetCollector.getBits();{code}
> Collector Level Dynamic Memory Accounting
> -----------------------------------------
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Atri Sharma
> Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which
> tracks dynamic memory used by Collectors. This shall allow users to get an
> accountability as to the memory usage of their Collectors and better plan
> their resource capacity. This shall also allow us to add Collector level
> limits for memory usage, thus allowing users a finer control over their
> resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
