Thanks Xiao, a more up to date publication in a conference like VLDB will
certainly turn the the tide for many of us trying to defend Spark's
Optimizer.
On Wed, Jan 15, 2020 at 9:39 AM Xiao Li wrote:
> In the upcoming Spark 3.0, we introduced a new framework for Adaptive
> Query Execution in
In the upcoming Spark 3.0, we introduced a new framework for Adaptive Query
Execution in Catalyst. This can adjust the plans based on the runtime
statistics. This is missing in Calcite based on my understanding.
Catalyst is also very easy to enhance. We also use the dynamic programming
approach
Thanks all, and Matei.
TL;DR of the conclusion for my particular case:
Qualitatively, while Catalyst[1] tries to mitigate learning curve and
maintenance burden, it lacks the dynamic programming approach used by
Calcite[2] and risks falling into local minima.
Quantitatively, there is no
I’m pretty sure that Catalyst was built before Calcite, or at least in
parallel. Calcite 1.0 was only released in 2015. From a technical standpoint,
building Catalyst in Scala also made it more concise and easier to extend than
an optimizer written in Java (you can find various presentations
It's fairly common for adapters (Calcite's abstraction of a data
source) to push down predicates. However, the API certainly looks a
lot different than Catalyst's.
--
Michael Mior
mm...@apache.org
Le lun. 13 janv. 2020 à 09:45, Jason Nerothin
a écrit :
>
> The implementation they chose supports
The implementation they chose supports push down predicates, Datasets and
other features that are not available in Calcite:
https://databricks.com/glossary/catalyst-optimizer
On Mon, Jan 13, 2020 at 8:24 AM newroyker wrote:
> Was there a qualitative or quantitative benchmark done before a
Was there a qualitative or quantitative benchmark done before a design
decision was made not to use Calcite?
Are there limitations (for heuristic based, cost based, * aware optimizer)
in Calcite, and frameworks built on top of Calcite? In the context of big
data / TCPH benchmarks.
I was unable
