lnbest0707-uber opened a new pull request, #13695:
URL: https://github.com/apache/pinot/pull/13695

   `ingestion` enhancement` `feature request`
   Resolving issues mentioned in https://github.com/apache/pinot/issues/13319.
   Pinot ingestion is currently using a strictly serial processing. It fetches 
a batch of messages from Kafka and then for messages in the batch, it would 
process one by one with the order of offsets to:
   
   1. Decode
   2. Transform
   3. Index
   
   It provides benefits to reuse the objects created in between to achieve 
better memory efficiency but not able to utilize all system resources. There 
are multiple solutions with their pros and cons:
   
   1. Async processing for each step as this patch introduced.
       - It could still preserve the same order of messages and retain same 
offset control logics.
       - It still could not fully utilize the system resources.
       - It brings overhead on memory and GC as not able to reuse objects as 
before. (There could be TODOs to make size of each batch configurable)
   2. Batch (multiple executors) processing on decoding and transformation.
       - Hard to ensure the order of messages, it might only do at least once 
consumption instead of current (almost) exact once.
   3. Full batch process on all 3 steps.
       - Current indexing logics and data structures do not really support 
parallel processing.
   
   Comparing the CPU usage and consumption speed on same server before and 
after (at 10:00) enabling "ASYNCHRONOUS":
   
![image](https://github.com/user-attachments/assets/9d79aa35-0d6b-495c-818c-67848f85a84b)
   
![image](https://github.com/user-attachments/assets/d8bf1297-3356-4bfc-8028-6021d4b047fc)
   The more CPU usage contributes a ~10% ingestion speed increase.
   
   Notes:
   The new mode is better performed on computation heavy ingestion but cannot 
really help on light weight use cases. In light computation use cases, the 
extra memory and GC overhead would compensate the async process gain.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to