[ https://issues.apache.org/jira/browse/FLINK-34379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dalongliu resolved FLINK-34379. ------------------------------- Resolution: Fixed > table.optimizer.dynamic-filtering.enabled lead to OutOfMemoryError > ------------------------------------------------------------------ > > Key: FLINK-34379 > URL: https://issues.apache.org/jira/browse/FLINK-34379 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner > Affects Versions: 1.17.2, 1.18.1 > Environment: 1.17.1 > Reporter: zhu > Assignee: Jeyhun Karimov > Priority: Critical > Labels: pull-request-available > Fix For: 1.20.0 > > > When using batch computing, I union all about 50 tables and then join other > table. When compiling the execution plan, > there throws OutOfMemoryError: Java heap space, which was no problem in > 1.15.2. However, both 1.17.2 and 1.18.1 all throws same errors,This causes > jobmanager to restart. Currently,it has been found that this is caused by > table.optimizer.dynamic-filtering.enabled, which defaults is true,When I set > table.optimizer.dynamic-filtering.enabled to false, it can be compiled and > executed normally > code > TableEnvironment.create(EnvironmentSettings.newInstance() > .withConfiguration(configuration) > .inBatchMode().build()) > sql=select att,filename,'table0' as mo_name from table0 UNION All select > att,filename,'table1' as mo_name from table1 UNION All select > att,filename,'table2' as mo_name from table2 UNION All select > att,filename,'table3' as mo_name from table3 UNION All select > att,filename,'table4' as mo_name from table4 UNION All select > att,filename,'table5' as mo_name from table5 UNION All select > att,filename,'table6' as mo_name from table6 UNION All select > att,filename,'table7' as mo_name from table7 UNION All select > att,filename,'table8' as mo_name from table8 UNION All select > att,filename,'table9' as mo_name from table9 UNION All select > att,filename,'table10' as mo_name from table10 UNION All select > att,filename,'table11' as mo_name from table11 UNION All select > att,filename,'table12' as mo_name from table12 UNION All select > att,filename,'table13' as mo_name from table13 UNION All select > att,filename,'table14' as mo_name from table14 UNION All select > att,filename,'table15' as mo_name from table15 UNION All select > att,filename,'table16' as mo_name from table16 UNION All select > att,filename,'table17' as mo_name from table17 UNION All select > att,filename,'table18' as mo_name from table18 UNION All select > att,filename,'table19' as mo_name from table19 UNION All select > att,filename,'table20' as mo_name from table20 UNION All select > att,filename,'table21' as mo_name from table21 UNION All select > att,filename,'table22' as mo_name from table22 UNION All select > att,filename,'table23' as mo_name from table23 UNION All select > att,filename,'table24' as mo_name from table24 UNION All select > att,filename,'table25' as mo_name from table25 UNION All select > att,filename,'table26' as mo_name from table26 UNION All select > att,filename,'table27' as mo_name from table27 UNION All select > att,filename,'table28' as mo_name from table28 UNION All select > att,filename,'table29' as mo_name from table29 UNION All select > att,filename,'table30' as mo_name from table30 UNION All select > att,filename,'table31' as mo_name from table31 UNION All select > att,filename,'table32' as mo_name from table32 UNION All select > att,filename,'table33' as mo_name from table33 UNION All select > att,filename,'table34' as mo_name from table34 UNION All select > att,filename,'table35' as mo_name from table35 UNION All select > att,filename,'table36' as mo_name from table36 UNION All select > att,filename,'table37' as mo_name from table37 UNION All select > att,filename,'table38' as mo_name from table38 UNION All select > att,filename,'table39' as mo_name from table39 UNION All select > att,filename,'table40' as mo_name from table40 UNION All select > att,filename,'table41' as mo_name from table41 UNION All select > att,filename,'table42' as mo_name from table42 UNION All select > att,filename,'table43' as mo_name from table43 UNION All select > att,filename,'table44' as mo_name from table44 UNION All select > att,filename,'table45' as mo_name from table45 UNION All select > att,filename,'table46' as mo_name from table46 UNION All select > att,filename,'table47' as mo_name from table47 UNION All select > att,filename,'table48' as mo_name from table48 UNION All select > att,filename,'table49' as mo_name from table49 UNION All select > att,filename,'table50' as mo_name from table50 UNION All select > att,filename,'table51' as mo_name from table51 UNION All select > att,filename,'table52' as mo_name from table52 UNION All select > att,filename,'table53' as mo_name from table53; > Table allUnionTable = tEnv.sqlQuery(sql); > Table res = > allUnionTable.join( > allUnionTable > .groupBy(col("att")) > .select(col("att"), col("att").count().as(COUNT_NAME)) > .filter(col(COUNT_NAME).isGreater(1)) > .select(col(key).as("l_key")), > col(key).isEqual(col("l_key")) > ); > res.printExplain(ExplainDetail.JSON_EXECUTION_PLAN); > > > Exception trace > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:3181) > at java.util.ArrayList.grow(ArrayList.java:267) > at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:241) > at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:233) > at java.util.ArrayList.add(ArrayList.java:464) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.setTables(DynamicPartitionPruningUtils.java:240) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:163) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:175) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > at > org.apache.flink.table.planner.utils.DynamicPartitionPruningUtils$DppDimSideChecker.visitDimSide(DynamicPartitionPruningUtils.java:190) > > -- This message was sent by Atlassian Jira (v8.20.10#820010)