rkrishn7 commented on issue #24265: URL: https://github.com/apache/datafusion/issues/24265#issuecomment-5259092089
Thanks for this @alamb. [Massive](https://massive.com/) also has an internal stream processing engine built on DataFusion. [This talk](https://player.vimeo.com/video/1170973541?h=259424ec0c#t=3764s) from the Stockholm meetup earlier this year provides an overview. We ultimately opted out of DataFusion's physical layer and built our own for a few reasons: - Our implementation requires support for interleaving non-data payloads with data across operator edges. The core algorithm we use to provide exactly-once processing guarantees relies on these payloads to establish ordered cuts within the streaming graph. - Edges between operators can have different semantics, such as append-only and changelog streams. - Streaming engines also have specific requirements around watermark alignment, stateful node IDs, and stream keys. It would be great if DataFusion's physical execution/planning interfaces could accommodate some of these requirements, though a generic extension mechanism may be a better fit than baking stream-processing-specific semantics into DataFusion itself. Separately, we do a fair amount with aggregation and another pain point I've come across is the lack of a way to peek at aggregate state or evaluate only a subset of a group. Supporting this would make incremental emission from time-windowed aggregation operators more efficient. [This issue](https://github.com/apache/datafusion/issues/18773) describes a similar problem. I'm sure there is more that I'm not thinking of right now 😅 . Will update this thread as more comes to mind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
