Hi Niklas, The planning time of a job does not depend on the data size. It would be the same whether you process 5MB or 5PB.
FLINK-10566 (as pointed to by Timo) fixed a problem for plans with many braching and joining nodes. Looking at your plan, there are some, but (IMO) not enough to be problematic. Your idea of providing hints is very good. You can do that for joins with size hints [1] or strategy hints [2]. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/dataset_transformations.html#join-with-dataset-size-hint [2] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/batch/dataset_transformations.html#join-algorithm-hints Am Mo., 7. Jan. 2019 um 17:36 Uhr schrieb Timo Walther <twal...@apache.org>: > Hi Niklas, > > it would be interesting to know which planner caused the long runtime. > Could you use a debugger to figure out more details? Is it really the > Flink Table API planner or the under DataSet planner one level deeper? > > There was an issue that was recently closed [1] about the DataSet > optimizer. Could this solve your problem? > > I will also loop in Fabian who might knows more. > > Regards, > Timo > > [1] https://issues.apache.org/jira/browse/FLINK-10566 > > Am 07.01.19 um 14:05 schrieb Niklas Teichmann: > > Hi everybody, > > > > I have a question concerning the planner for the Flink Table / Batch API. > > At the moment I try to use a library called Cypher for Apache Flink, a > > project that tries to implement > > the graph database query language Cypher on Apache Flink (CAPF, > > https://github.com/soerenreichardt/cypher-for-apache-flink). > > > > The problem is that the planner seemingly takes a very long time to > > plan and optimize the job created by CAPF. This example job in json > > format > > > > https://pastebin.com/J84grsjc > > > > takes on a 24 GB data set about 20 minutes to plan and about 5 minutes > > to run the job. That seems very long for a job of this size. > > > > Do you have any idea why this is the case? > > Is there a way to give the planner hints to reduce the planning time? > > > > Thanks in advance! > > Niklas > > >
