I’m not sure. As far as I know APIs are already in place for statistics support
and only missing part is actual statistic provider. With this respect it batch
and streaming might be almost completely independent of one another, so it
shouldn’t be matter of “first that then that” but whatever wil
+1. Having table statistics is one of the main blockers for more advanced
optimization rules. I would love to contribute to this effort!
However I think @Alberts case is more on the data set side. Was there any
plan to integrate with data set table statistics first then extend to data
stream domai
Hi,
Currently the biggest limitation that prevents better query optimisation is
lack of table statistics (which are not trivial to provide in streaming), thus
Joins/Aggregation reordering doesn’t work. We have some ideas how to tackle
this issue and definitely at some point of time we will impr
Hi Albert,
Calcite provides a rule-based optimizer (as a framework), which means users can
customize it by adding rules. That’s exactly what Flink did. From the logical
plan to the physical plan, the translations are triggered by different sets of
rules, according to which the relational expres
Hi Albert,
First I guess the query optimizer you mentioned is about Flink table & sql
(for batch API there is another optimizer which is implemented by Flink).
Yes, now for table & sql, Flink use Apache Calcite's query optimizer to
translate into a Calcite plan
which is then optimized according t
Hello,
I am just wondering, does Flink use Apache Calcite's query optimizer to
generate an optimal logical plan, or does it have its own query optimizer?
>From what I observed so far, the Flink's query optimizer only groups
operator together without changing the order of aggregation operators
(e.g
