npawar opened a new issue #4801: Smarter resize buffer and algorithm resizing in IndexedTable URL: https://github.com/apache/incubator-pinot/issues/4801 The resizer has a very naive logic at the moment to decide the buffer capacity: From BaseTable.java: ``` // TODO: tune these numbers // Based on the capacity and maxCapacity, the resizer will smartly choose to evict/retain recors from the PQ if (capacity <= 100_000) { // Capacity is small, make a very large buffer. Make PQ of records to retain, during resize _maxCapacity = 1_000_000; } else { // Capacity is large, make buffer only slightly bigger. Make PQ of records to evict, during resize _maxCapacity = (int) (capacity * 1.2); } ``` This allocation is unfair for numbers which fall on the borderline. We need smarter formulas for computing this. The resize algorithm (in TableResizer) also needs a revisit and enhancement
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
