lnbest0707-uber opened a new pull request, #13695: URL: https://github.com/apache/pinot/pull/13695
`ingestion` enhancement` `feature request` Resolving issues mentioned in https://github.com/apache/pinot/issues/13319. Pinot ingestion is currently using a strictly serial processing. It fetches a batch of messages from Kafka and then for messages in the batch, it would process one by one with the order of offsets to: 1. Decode 2. Transform 3. Index It provides benefits to reuse the objects created in between to achieve better memory efficiency but not able to utilize all system resources. There are multiple solutions with their pros and cons: 1. Async processing for each step as this patch introduced. - It could still preserve the same order of messages and retain same offset control logics. - It still could not fully utilize the system resources. - It brings overhead on memory and GC as not able to reuse objects as before. (There could be TODOs to make size of each batch configurable) 2. Batch (multiple executors) processing on decoding and transformation. - Hard to ensure the order of messages, it might only do at least once consumption instead of current (almost) exact once. 3. Full batch process on all 3 steps. - Current indexing logics and data structures do not really support parallel processing. Comparing the CPU usage and consumption speed on same server before and after (at 10:00) enabling "ASYNCHRONOUS":   The more CPU usage contributes a ~10% ingestion speed increase. Notes: The new mode is better performed on computation heavy ingestion but cannot really help on light weight use cases. In light computation use cases, the extra memory and GC overhead would compensate the async process gain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
