I always thought of a table function as a lightweight relational operator. You could write your own UNION function, for instance. But the moment you want it to start participating in algebraic rewrites - if you want to push filters through it, for instance - then you had better make it into a relational operator (that is, a sub-class of RelNode). I think you’re at that point.
A table macro is powerful, but it has been expanded before the planner rules start to be applied, so can’t help. > On Sep 10, 2019, at 8:18 AM, Julian Feinauer <j.feina...@pragmaticminds.de> > wrote: > > Hey, > > when going through the Code I just had another Idea. > Currently a TableFunction is executed as EnumerableTableFunctionScan which > gets generated from a LogicalTableFunctionScan by the Rule > EnumerableTableFunctionScanRule. > What if you just remove that Rule and add a custom Rule of yours which > translates it of a TableScan of your taste? > > Julian > > Am 10.09.19, 08:13 schrieb "Julian Feinauer" <j.feina...@pragmaticminds.de>: > > Hi Gabriel, > > thats an interesting question for me too. > Do you need parameters for those "dynamic tables"? > If not you could do it similar to what Drill is doing and just implement a > Schema which always returns "true" if someone asks for a Table and then > returns a Table Implementation that you provide where you can hook in later > and add the functionality that you in fact need. This can then also be used > in optimization as you can then control your custom Table type. > Perhaps it helps to look at the DrillTable class in [1]. > > On a Side node I try to figure out what would be necessary to make > TableFunction wo also work with TranslatableTable. > Would you mind opening an issue in Jira for that? > > Julian > > [1] > https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillTable.java > > > Am 10.09.19, 03:25 schrieb "Gabriel Reid" <gabriel.r...@gmail.com>: > > Hi, > > I'm currently using a combination of TableFunctions and TableMacros to > expose various dynamic (relatively unstructured) data sources via > Calcite. > The underlying data sources are such that data can only be retrieved by > first specifying what you want (i.e. there is no catalog of all data > that > is available). > > I'm currently handling this by using a combination of TableFunctions > and > TableMacros. > > The issue that I'm running into comes when I want to implement custom > planner rules for the underlying functionality. As far I as I can see, > it's > not possible to register planner rules based on a TableFunctionImpl, > because a TableFunctionImpl only exposes a ScannableTable, so there's > no > chance to hook into RelOptNode.register. > > On the other hand, implementing a TableMacro does allow to return a > TranslatableTable, which then does allow intercepting the call to > RelOptNode.register to register rules. However, TableMacros require > that > all parameters are literals, and I'm providing timestamps, via > TIMESTAMPADD() and CURRENT_TIMESTAMP() calls, which then doesn't work > for > TableMacros (all parameters to a table macro need to be literals, > otherwise > query validation fails in Calcite). > > I'm wondering if I'm missing some built-in functionality which would > make > it possible to have a dynamic table function/macro that can also be > manipulated via custom planner rules. > > Options (which may or may not exist) that I can think of are: > * something that would/could visit all macro parameters ahead of time > and > resolve things like CURRENT_TIMESTAMP() to a literal, before further > query > validation occurs > * register rules somewhere outside of RelOptNode.register (e.g. when > the > schema is first created) > > Are there any currently-working options in Calcite that can help me do > what > I'm trying to do? And if there aren't and I would add such a thing to > Calcite, are there any suggestions as to what the most appropriate > approach > would be (either one of the two options I listed above, or something > else)? > > Thanks, > > Gabriel > > > >
