t0mpere opened a new issue, #9971:
URL: https://github.com/apache/pinot/issues/9971

   Hello everybody, My name is Tommaso and I'm a Data Platform Engineer at 
Bumble Inc. 
   I would like to contribute to Apache Pinot by implementing the HLL merge 
function as an Aggregation option for the Minion `MergeRollUpTask`
   
   ### What needs to be done?
   Currently the minion `MergeRollUpTask` only supports aggregation operations 
like: **Min, Max, Sum**
   I would like to add `HLLMerge` as an aggregation function, to aggregate 
serialised HLL estimators objects when performing rollup.
   
   ### Use cases
   At Bumble we rely on HLL for cardinality estimation on a large scale and it 
would be of great value if Apache Pinot could rollup segments that store 
serialised HLL counters in `BYTES` columns. This would allow us to have 
different time granularities in a single table allowing for longer retention 
and faster queries on big time windows that use HLL cardinality estimation.
   
   ### Initial Proposal
   
   The aggregation function will be a class implementing the 
`org.apache.pinot.core.segment.processing.aggregator.ValueAggregator` 
interface. This aggregation function will only support `BYTES` metric columns 
where `com.clearspring.analytics.stream.cardinality.HyperLogLog` objects are 
serialised. The operation will merge two existing HyperLogLog objects into a 
single resulting HyperLogLog object. 
   
   ### Limitations and Questions
   
   It could be useful to rollup column types like `STRING`, `INT`, `DOUBLE`, 
`LONG`, `MV` or `FLOAT` into a `BYTES` column with `HLL`. This kind of rollup 
is what we do already at Bumble with Apache Spark before ingesting data into 
Pinot. This would let users ingest **raw** data and then let the minions take 
care of the rollup with the option of having HLL distinct count as a metric.
   
   I'm afraid that this would not possible due to the fact that 
`MergeRollUpTask` can't change the type of a column to `BYTES` to store the HLL 
estimator. Has anyone ever thought about this?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to