Got it, it's clear, TableStats is the important functions of ExternalCatalog. It is right way.
Best, JingsongLee ------------------------------------------------------------------ From:Felipe Gutierrez <felipe.o.gutier...@gmail.com> Send Time:2019年6月27日(星期四) 14:53 To:JingsongLee <lzljs3620...@aliyun.com> Cc:user <user@flink.apache.org> Subject:Re: Hello-world example of Flink Table API using a edited Calcite rule Hi JingsongLee, Sorry for not explain very well. I am gonna try a clarification of my idea. 1 - I want to use InMemoryExternalCatalog in a way to save some statistics which I create by listening to a stream. 2 - Then I will have my core application using Table API to execute some aggregation/join. 3 - Because the application on 2 uses Table API, I am able to influence its plan through Calcite configuration rules. So, I am gonna use the statistics from 1 to change the rules dynamic on 2. Do you think it is clear? and it is a feasible application with the current capabilities of Table API? ps.: I am gonna look at the links that you mentioned. Thanks for that! Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Thu, Jun 27, 2019 at 7:23 AM JingsongLee <lzljs3620...@aliyun.com> wrote: Hi Felipe: Yeah, you can use InMemoryExternalCatalog and CalciteConfig, but I don't quite understand what you mean. InMemoryExternalCatalog provides methods to create, drop, and alter (sub-)catalogs or tables. And CalciteConfig is for defining a custom Calcite configuration. They are two separate things. About InMemoryExternalCatalog, You can take a look at [1]. Csv has been renamed to OldCsv [2], But recommendation using the RFC-compliant `Csv` format in the dedicated `flink-formats/flink-csv` module instead. [1] https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/ExternalCatalogTest.scala [2] https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/descriptors/OldCsv.scala Best, JingsongLee ------------------------------------------------------------------ From:Felipe Gutierrez <felipe.o.gutier...@gmail.com> Send Time:2019年6月26日(星期三) 20:58 To:JingsongLee <lzljs3620...@aliyun.com> Cc:user <user@flink.apache.org> Subject:Re: Hello-world example of Flink Table API using a edited Calcite rule Hi JingsongLee, it is still not very clear to me. I imagine that I can create an InMemoryExternalCatalog and insert some tuples there (which will be in memory). Then I can use Calcite to use the values of my InMemoryExternalCatalog and change my plan. Is that correct? Do you have an example of how to create an InMemoryExternalCatalog using Flink 1.8? Because I guess the Csv [1] class does not exist anymore. [1] https://github.com/srapisarda/stypes-flink/blob/feature/sql/src/test/scala/uk/ac/bbk/dcs/stypes/flink/CalciteEmptyConsistencyTest.scala#L73 Kind Regards, Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Wed, Jun 26, 2019 at 1:36 PM JingsongLee <lzljs3620...@aliyun.com> wrote: Hi Felipe: I think your approach is absolutely right. You can try to do some plan test just like [1]. You can find more CalciteConfigBuilder API test in [2]. 1.https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/api/batch/table/CorrelateTest.scala#L168 2.https://github.com/apache/flink/blob/release-1.8/flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/calcite/CalciteConfigBuilderTest.scala Best, JingsongLee ------------------------------------------------------------------ From:Felipe Gutierrez <felipe.o.gutier...@gmail.com> Send Time:2019年6月26日(星期三) 18:04 To:user <user@flink.apache.org> Subject:Hello-world example of Flink Table API using a edited Calcite rule Hi, does someone have a simple example using Table API and a Calcite rule which change/optimize the query execution plan of a query in Flink? From the official documentation, I know that I have to create a CalciteConfig object [1]. Then, I based my firsts tests on this stackoverflow post [2] and I implemented this piece of code: // change the current calcite config plan CalciteConfigBuilder ccb = new CalciteConfigBuilder(); RuleSet ruleSets = RuleSets.ofList(FilterMergeRule.INSTANCE); ccb.addLogicalOptRuleSet(ruleSets); TableConfig tableConfig = new TableConfig(); tableConfig.setCalciteConfig(ccb.build()); I suppose that with this I can change the query plan of the Flink Table API. I am also not sure if I will need to use an external catalog like this post assumes to use [3]. In a nutshell, I would like to have a simple example where I can execute a query using Flink Table API and change its query execution plan using a Calcite rule. Does anyone have a Hello world of it? I plan to use it on this example [4]. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/table/common.html#query-optimization [2] https://stackoverflow.com/questions/54278508/how-can-i-create-an-external-catalog-table-in-apache-flink [3] https://stackoverflow.com/questions/54278508/how-can-i-create-an-external-catalog-table-in-apache-flink [4] https://github.com/felipegutierrez/explore-flink/blob/master/src/main/java/org/sense/flink/examples/stream/table/MqttSensorDataAverageTableAPI.java#L40 Kind Regards, Felipe -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com
