[
https://issues.apache.org/jira/browse/SPARK-25094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16577593#comment-16577593
]
Marco Gaido commented on SPARK-25094:
-------------------------------------
This is a duplicate of many. Unfortunately this problem has not yet been
solved, so in this case whole-stage code generation is disabled for the query.
There is an ongoing effort in order to enable to fix this issue in the future
though.
> proccesNext() failed to compile size is over 64kb
> -------------------------------------------------
>
> Key: SPARK-25094
> URL: https://issues.apache.org/jira/browse/SPARK-25094
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Izek Greenfield
> Priority: Major
>
> I have this tree:
> 2018-08-12T07:14:31,289 WARN []
> org.apache.spark.sql.execution.WholeStageCodegenExec - Whole-stage codegen
> disabled for plan (id=1):
> *(1) Project [, ... 10 more fields]
> +- *(1) Filter NOT exposure_calc_method#10141 IN
> (UNSETTLED_TRANSACTIONS,FREE_DELIVERIES)
> +- InMemoryTableScan [, ... 11 more fields], [NOT
> exposure_calc_method#10141 IN (UNSETTLED_TRANSACTIONS,FREE_DELIVERIES)]
> +- InMemoryRelation [, ... 80 more fields], StorageLevel(memory,
> deserialized, 1 replicas)
> +- *(5) SortMergeJoin [unique_id#8506], [unique_id#8722], Inner
> :- *(2) Sort [unique_id#8506 ASC NULLS FIRST], false, 0
> : +- Exchange(coordinator id: 1456511137)
> UnknownPartitioning(9), coordinator[target post-shuffle partition size:
> 67108864]
> : +- *(1) Project [, ... 6 more fields]
> : +- *(1) Filter (((((isnotnull(v#49) &&
> isnotnull(run_id#52)) && (asof_date#48 <=> 17531)) && (run_id#52 = DATA_REG))
> && (v#49 = DATA_REG)) && isnotnull(unique_id#39))
> : +- InMemoryTableScan [, ... 6 more fields], [,
> ... 6 more fields]
> : +- InMemoryRelation [, ... 6 more
> fields], StorageLevel(memory, deserialized, 1 replicas)
> : +- *(1) FileScan csv [,... 6 more
> fields] , ... 6 more fields
> +- *(4) Sort [unique_id#8722 ASC NULLS FIRST], false, 0
> +- Exchange(coordinator id: 1456511137)
> UnknownPartitioning(9), coordinator[target post-shuffle partition size:
> 67108864]
> +- *(3) Project [, ... 74 more fields]
> +- *(3) Filter (((isnotnull(v#51) && (asof_date#42
> <=> 17531)) && (v#51 = DATA_REG)) && isnotnull(unique_id#54))
> +- InMemoryTableScan [, ... 74 more fields], [,
> ... 4 more fields]
> +- InMemoryRelation [, ... 74 more
> fields], StorageLevel(memory, deserialized, 1 replicas)
> +- *(1) FileScan csv [,... 74 more
> fields] , ... 6 more fields
> Compiling "GeneratedClass": Code of method "processNext()V" of class
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1"
> grows beyond 64 KB
> and the generated code failed to compile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
