Practically, Apache Calcite doesn’t have public API. Real projects usually override not only public extension points, but also internals, such as SqlValidatorImpl or VolcanoPlanner.
Given the flexibility that Apache Calcite provides, this lack of public API surface is not necessarily a bad thing, since you may change almost everything without forking the project. On the other hand, almost any change to any part of Calcite code base may cause regressions in downstream projects. I am not sure there is an ideal way of versioning rels because even if you do so, some direct or indirect changes (metadata, rules, validator, etc) may cause changes to plans around the given RelNode in the downstream project. Maybe instead of providing versions manually, we may expose some sort of generated signatures for different components - rels, rules, metadata handlers, golden plans, etc. Then, we may provide the ability to verify expected and actual signatures, e.g., using some code generation: user invokes a command with the list of interesting classes, and gets the source code of the test that he includes into his project and CI. Now, when the user migrates to the new version, tests for changed entities will fail, and user will investigate the associated changes. The main difference s from the original proposal: 1. Wider scope, because validation of rels is often not sufficient. 2. Automation, because as implement or you cannot always predict the effect of your changes (e.g., metadata). WDYT? Вт, 14 дек. 2021 г. в 01:49, Konstantin Orlov <kostyaorlo...@gmail.com>: > > The case there is that "correlation variables" are added to the Logical* > nodes (e.g. LogicalFilter, LogicalProject). > > BTW, "correlation variables" were added to a LogicalFilter in CALCITE-816. > So what is wrong with CALCITE-3183? > > вт, 14 дек. 2021 г. в 01:24, Konstantin Orlov <kostyaorlo...@gmail.com>: > > > Vladimir, could you please clarify in what way the PR#2623 changes > > the semantics? > > > > The correlated project is already possible in the master. The > MongoProject > > already discards variablesSet (simply because it's currently not stored > > for > > project node) and either fails or returns invalid results. This behavior > > (alas, > > incorrect) will be preserved after this patch. > > > > пн, 13 дек. 2021 г. в 17:55, Vladimir Sitnikov < > > sitnikov.vladi...@gmail.com>: > > > >> Hi, > >> > >> It turns out Calcite's nodes can change semantics without notification > for > >> the end-users. > >> > >> Here are the release notes for Calcite 1.21, and it says **nothing** > like > >> "Ensure you handle Filter#variablesSet in case you implement Filter or > in > >> case you transform LogicalFilter in your rules" > >> https://calcite.apache.org/news/2019/09/11/release-1.21.0/ > >> > >> On top of that, the in-core adapters fail to handle that properly. For > >> example, MongoFilter discards Filter#variablesSet. > >> > >> Can we please stop changing the semantics of RelNodes or can we have a > >> better way to detect the changes in the client code? > >> > >> What if we add a "version" property to the corresponding RelNodes, and > we > >> increment it every time the semantic changes? > >> Then client APIs could be coded like "ok, I'm prepared to handle Project > >> v4, and Filter v5" (e.g. make "version" required when registering a > rule), > >> and there will be a runtime error in case Calcite generates Filter v6 in > >> runtime. > >> > >> --- > >> > >> Sample case: > >> CALCITE-3183 Trimming method for Filter rel uses wrong traitSet > >> CALCITE-4913 Correlated variables in a select list are not deduplicated > >> > >> The case there is that "correlation variables" are added to the Logical* > >> nodes (e.g. LogicalFilter, LogicalProject). > >> Unfortunately, that change is hard to notice: there's no compilation > >> failure, and there's no runtime error. > >> > >> The old client code just discards "correlation variables", and I guess > it > >> would result in wrong results or something like that. > >> Silent wrong results is a really sad outcome from the database. > >> > >> CALCITE-4913 / PR#2623 adds Project#variablesSet property as well, and I > >> guess it would in the same hidden semantic loss. > >> > >> Vladimir > >> > > > > > > -- > > Regards, > > Konstantin Orlov > > > > > -- > Regards, > Konstantin Orlov >
