Hi JingsongLee, Sorry for not explain very well. I am gonna try a clarification of my idea. 1 - I want to use InMemoryExternalCatalog in a way to save some statistics which I create by listening to a stream. 2 - Then I will have my core application using Table API to execute some aggregation/join. 3 - Because the application on 2 uses Table API, I am able to influence its plan through Calcite configuration rules. So, I am gonna use the statistics from 1 to change the rules dynamic on 2.
Do you think it is clear? and it is a feasible application with the current capabilities of Table API? ps.: I am gonna look at the links that you mentioned. Thanks for that! Felipe *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Thu, Jun 27, 2019 at 7:23 AM JingsongLee <lzljs3620...@aliyun.com> wrote: > Hi Felipe: > > Yeah, you can use InMemoryExternalCatalog and CalciteConfig, > but I don't quite understand what you mean. > InMemoryExternalCatalog provides methods to create, drop, and > alter (sub-)catalogs or tables. And CalciteConfig is for defining a > custom Calcite configuration. They are two separate things. > > About InMemoryExternalCatalog, You can take a look at [1]. > Csv has been renamed to OldCsv [2], But recommendation > using the RFC-compliant `Csv` format in the dedicated > `flink-formats/flink-csv` module instead. > > > [1] > https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/ExternalCatalogTest.scala > [2] > https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/descriptors/OldCsv.scala > > Best, JingsongLee > > ------------------------------------------------------------------ > From:Felipe Gutierrez <felipe.o.gutier...@gmail.com> > Send Time:2019年6月26日(星期三) 20:58 > To:JingsongLee <lzljs3620...@aliyun.com> > Cc:user <user@flink.apache.org> > Subject:Re: Hello-world example of Flink Table API using a edited Calcite > rule > > Hi JingsongLee, > > it is still not very clear to me. I imagine that I can create an > InMemoryExternalCatalog and insert some tuples there (which will be in > memory). Then I can use Calcite to use the values of my > InMemoryExternalCatalog and change my plan. Is that correct? > > Do you have an example of how to create an InMemoryExternalCatalog using > Flink 1.8? Because I guess the Csv [1] class does not exist anymore. > > [1] > https://github.com/srapisarda/stypes-flink/blob/feature/sql/src/test/scala/uk/ac/bbk/dcs/stypes/flink/CalciteEmptyConsistencyTest.scala#L73 > > Kind Regards, > Felipe > > > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Wed, Jun 26, 2019 at 1:36 PM JingsongLee <lzljs3620...@aliyun.com> > wrote: > Hi Felipe: > I think your approach is absolutely right. You can try to do some plan > test just like [1]. > You can find more CalciteConfigBuilder API test in [2]. > > 1. > https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/batch/table/CorrelateTest.scala#L168 > 2. > https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/calcite/CalciteConfigBuilderTest.scala > > Best, JingsongLee > > ------------------------------------------------------------------ > From:Felipe Gutierrez <felipe.o.gutier...@gmail.com> > Send Time:2019年6月26日(星期三) 18:04 > To:user <user@flink.apache.org> > Subject:Hello-world example of Flink Table API using a edited Calcite rule > > Hi, > > does someone have a simple example using Table API and a Calcite rule > which change/optimize the query execution plan of a query in Flink? > > From the official documentation, I know that I have to create a > CalciteConfig object [1]. Then, I based my firsts tests on this > stackoverflow post [2] and I implemented this piece of code: > > // change the current calcite config plan > CalciteConfigBuilder ccb = new CalciteConfigBuilder(); > RuleSet ruleSets = RuleSets.ofList(FilterMergeRule.INSTANCE); > ccb.addLogicalOptRuleSet(ruleSets); > TableConfig tableConfig = new TableConfig(); > tableConfig.setCalciteConfig(ccb.build()); > > I suppose that with this I can change the query plan of the Flink Table > API. I am also not sure if I will need to use an external catalog like this > post assumes to use [3]. > In a nutshell, I would like to have a simple example where I can execute a > query using Flink Table API and change its query execution plan using a > Calcite rule. Does anyone have a Hello world of it? I plan to use it on > this example [4]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/common.html#query-optimization > [2] > https://stackoverflow.com/questions/54278508/how-can-i-create-an-external-catalog-table-in-apache-flink > [3] > https://stackoverflow.com/questions/54278508/how-can-i-create-an-external-catalog-table-in-apache-flink > [4] > https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/table/MqttSensorDataAverageTableAPI.java#L40 > > Kind Regards, > Felipe > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > >
