Re: Flink SQL hangs in StreamTableEnvironment.sqlUpdate, keeps executing and seems never stop, finally lead to java.lang.OutOfMemoryError: GC overhead limit exceeded

2019-04-08 Thread 徐涛 
Hi Fabian,
No, I did not add any optimization rules.
I have created two JIRAs about this issue, because when I modify the 
SQL a little, the error turns to StackOverflowError then:
https://issues.apache.org/jira/browse/FLINK-12097 

https://issues.apache.org/jira/browse/FLINK-12109 


The SQL is quite long(about 2000 lines), and some of them are written 
by the DSL my defined, but most of them are same to Flink SQL,  so I put it in 
the JIRA attachment.
Kindly check about it, thanks a lot.

Best,
Henry

> 在 2019年4月8日，下午6:25，Fabian Hueske  写道：
> 
> Hi Henry,
> 
> It seem that the optimizer is not handling this case well. 
> The search space might be too large (or rather the optimizer explores too 
> much of the search space).
> Can you share the query? Did you add any optimization rules?
> 
> Best, Fabian
> 
> Am Mi., 3. Apr. 2019 um 12:36 Uhr schrieb 徐涛  >:
> Hi Experts,
>   There is a Flink application(Version 1.7.2) which is written in Flink 
> SQL, and the SQL in the application is quite long, consists of about 10 
> tables, 1500 lines in total. When executing, I found it is hanged in 
> StreamTableEnvironment.sqlUpdate, keep executing some code about calcite and 
> the memory usage keeps grown up, after several minutes 
> java.lang.OutOfMemoryError: GC overhead limit exceeded is got.
> 
>   I get some thread dumps:
> at 
> org.apache.calcite.plan.volcano.RuleQueue.popMatch(RuleQueue.java:475)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:640)
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at 
> org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
> at 
> org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
> at 
> org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
> at 
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:817)
> at 
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:777)
>   
> 
> 
> at java.io.PrintWriter.write(PrintWriter.java:473)
> at 
> org.apache.calcite.rel.AbstractRelNode$1.explain_(AbstractRelNode.java:415)
> at 
> org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:156)
> at 
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:312)
> at 
> org.apache.calcite.rel.AbstractRelNode.computeDigest(AbstractRelNode.java:420)
> at 
> org.apache.calcite.rel.AbstractRelNode.recomputeDigest(AbstractRelNode.java:356)
> at 
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:350)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1484)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1755)
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
> at 
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at 
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at 
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> at 
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:646)
> at 
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at 
> org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
> at 
> org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
> at 
> org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
> at 
> org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
> at 
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.s

Re: Flink SQL hangs in StreamTableEnvironment.sqlUpdate, keeps executing and seems never stop, finally lead to java.lang.OutOfMemoryError: GC overhead limit exceeded

2019-04-08 Thread Fabian Hueske 
Hi Henry,

It seem that the optimizer is not handling this case well.
The search space might be too large (or rather the optimizer explores too
much of the search space).
Can you share the query? Did you add any optimization rules?

Best, Fabian

Am Mi., 3. Apr. 2019 um 12:36 Uhr schrieb 徐涛 :

> Hi Experts,
> There is a Flink application(Version 1.7.2) which is written in Flink SQL,
> and the SQL in the application is quite long, consists of about 10 tables,
> 1500 lines in total. When executing, I found it is hanged in
> StreamTableEnvironment.sqlUpdate, keep executing some code about calcite
> and the memory usage keeps grown up, after several minutes
> java.lang.OutOfMemoryError: GC overhead limit exceeded is got.
>
> I get some thread dumps:
> at
> org.apache.calcite.plan.volcano.RuleQueue.popMatch(RuleQueue.java:475)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:640)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at
> org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
> at
> org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
> at
> org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
> at
> org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
> at
> org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
> at
> org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
> at
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:817)
> at
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:777)
>
>
> at java.io.PrintWriter.write(PrintWriter.java:473)
> at
> org.apache.calcite.rel.AbstractRelNode$1.explain_(AbstractRelNode.java:415)
> at
> org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:156)
> at
> org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:312)
> at
> org.apache.calcite.rel.AbstractRelNode.computeDigest(AbstractRelNode.java:420)
> at
> org.apache.calcite.rel.AbstractRelNode.recomputeDigest(AbstractRelNode.java:356)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:350)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1484)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1755)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
> at
> org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
> at
> org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:646)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
> at
> org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
> at
> org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
> at
> org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
> at
> org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
> at
> org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
> at
> org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
> at
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:817)
> at
> org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:777)
> * Both point to some code about calcite.*
>
> And also I get the heap dump, found that there are *5703373 RexCall
> instances, and 5909525 String instances, 5909692 char[] instances ,**size
> is 6.8G*. I wonder why there are so many RexCall instances here, why it
> keeps on executing some calcite code and seems never stop.
> char[]
> 5,909,692 (16.4%) 6,873,430,938 (84.3%)
> java.lang.String
> 5,909,525 (16.4%) 165,466,700 (2%)
> org.apache.calcite.rex.RexLocalRef
> 5,901,313 (16.4%) 259,657,772 (3.2%)
>
> org.apache.flink.calcite.shaded.com.google.common.collect.RegularImmutableList
> 5,739,479 (15.9%) 229,579,160 (2.8%)
> java.lang.Object[]
> 5,732,702 (15.9%) 279,902,336 (3.4%)
> org.apache.calcite.rex

Flink SQL hangs in StreamTableEnvironment.sqlUpdate, keeps executing and seems never stop, finally lead to java.lang.OutOfMemoryError: GC overhead limit exceeded

2019-04-03 Thread 徐涛 
Hi Experts,
There is a Flink application(Version 1.7.2) which is written in Flink 
SQL, and the SQL in the application is quite long, consists of about 10 tables, 
1500 lines in total. When executing, I found it is hanged in 
StreamTableEnvironment.sqlUpdate, keep executing some code about calcite and 
the memory usage keeps grown up, after several minutes 
java.lang.OutOfMemoryError: GC overhead limit exceeded is got.

I get some thread dumps:
at 
org.apache.calcite.plan.volcano.RuleQueue.popMatch(RuleQueue.java:475)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:640)
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
at 
org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
at 
org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
at 
org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
at 
org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
at 
org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
at 
org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
at 
org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:817)
at 
org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:777)



at java.io.PrintWriter.write(PrintWriter.java:473)
at 
org.apache.calcite.rel.AbstractRelNode$1.explain_(AbstractRelNode.java:415)
at 
org.apache.calcite.rel.externalize.RelWriterImpl.done(RelWriterImpl.java:156)
at 
org.apache.calcite.rel.AbstractRelNode.explain(AbstractRelNode.java:312)
at 
org.apache.calcite.rel.AbstractRelNode.computeDigest(AbstractRelNode.java:420)
at 
org.apache.calcite.rel.AbstractRelNode.recomputeDigest(AbstractRelNode.java:356)
at 
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:350)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1484)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:859)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:879)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:1755)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo(VolcanoRuleCall.java:135)
at 
org.apache.calcite.plan.RelOptRuleCall.transformTo(RelOptRuleCall.java:234)
at 
org.apache.calcite.rel.convert.ConverterRule.onMatch(ConverterRule.java:141)
at 
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:212)
at 
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:646)
at 
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:339)
at 
org.apache.flink.table.api.TableEnvironment.runVolcanoPlanner(TableEnvironment.scala:373)
at 
org.apache.flink.table.api.TableEnvironment.optimizeLogicalPlan(TableEnvironment.scala:292)
at 
org.apache.flink.table.api.StreamTableEnvironment.optimize(StreamTableEnvironment.scala:812)
at 
org.apache.flink.table.api.StreamTableEnvironment.translate(StreamTableEnvironment.scala:860)
at 
org.apache.flink.table.api.StreamTableEnvironment.writeToSink(StreamTableEnvironment.scala:344)
at 
org.apache.flink.table.api.TableEnvironment.insertInto(TableEnvironment.scala:879)
at 
org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:817)
at 
org.apache.flink.table.api.TableEnvironment.sqlUpdate(TableEnvironment.scala:777)
Both point to some code about calcite.

And also I get the heap dump, found that there are 5703373 RexCall 
instances, and 5909525 String instances, 5909692 char[] instances ,size is 
6.8G. I wonder why there are so many RexCall instances here, why it keeps on 
executing some calcite code and seems never stop. 
char[]  
5,909,692 (16.4%)   6,873,430,938 (84.3%)
java.lang.String
5,909,525 (16.4%)   165,466,700 (2%)
org.apache.calcite.rex.RexLocalRef  
5,901,313 (16.4%)   259,657,772 (3.2%)

org.apache.flink.calcite.shaded.com.google.common.collect.RegularImmutableList  
5,739,479 (15.9%)   229,579,160 (2.8%)
java.lang.Object[]  
5,732,702 (15.9%)   279,902,336 (3.4%)
org.apache.calcite.rex.RexCall  
5,703,373 (15.8%)   273,761,904 (3.4%)

Look forward to your reply.
Thanks a lot.


Best
Henry