leerho commented on issue #6743: IncrementalIndex generally overestimates theta sketch size URL: https://github.com/apache/incubator-druid/issues/6743#issuecomment-486759807 @himanshug @gianm > I looked into this a long time ago and one way was hacking it was to use "MemoryRegion" and "MemoryRequest" as in https://github.com/himanshug/druid/blob/growable_aggregator_final/extensions/datasketches/src/main/java/io/druid/query/aggregation/datasketches/theta/SketchResizableBufferAggregator.java#L120 (as you might guess this is based on pretty old version of DS library :) ) . Those mechanisms still exist in the current library, the MemoryRequest was just renamed [MemoryRequestServer](https://github.com/DataSketches/memory/blob/master/src/main/java/com/yahoo/memory/MemoryRequestServer.java). It is essentially the same as the Postgres concept and has been in the library long before we knew about Postgres. The underlying implementation, of course, would be up to Druid. The library also has a [DefaultMemoryRequestServer](https://github.com/DataSketches/memory/blob/master/src/main/java/com/yahoo/memory/DefaultMemoryRequestServer.java) which just allocates from the heap if the primary interface has not been implemented. Currently, only the DS-HLL, Theta, and Quantiles sketches are designed to take advantage of this. But if the demand is there, we could update other relevant sketches in the library to also use this mechanism. The above MemoryRequestServer interface is pretty simple, but if it doesn't exactly meet your needs, let us know. We would be happy to work with you as to what makes sense.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
