manikBS commented on issue #19119: URL: https://github.com/apache/pinot/issues/19119#issuecomment-5224391253
- Operational cost: The Flink approach requires maintaining separate Pinot and Flink clusters and kafka cluster for transferring data from flink to pinot. With the native approach, a single Pinot cluster can handle both stateful streaming operations (e.g. joins) and queries. - Latency: In the Flink approach, events typically go through Kafka -> Flink -> Kafka -> Pinot before becoming queryable. The native approach can directly update Pinot's realtime segments, eliminating the intermediate Kafka hop and reducing the latency between receiving an event from Kafka and making the resulting state queryable. - Architecture: Pinot currently handles ingestion and query serving through its servers. The proposed design introduces dedicated Stream Servers for ingestion and stateful processing, while query serving continues through the existing servers - Fault tolerance: Flink's checkpoint-based fault tolerance relies on checkpoin barriers and coordinated snapshots, which can be challenging when the upstream source is not fully under our control. In the proposed design state can be checkpointed at the realtime segment creation boundary as default checkpoint within Pinot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
