Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19440#discussion_r143094785
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
---
@@ -392,12 +392,16 @@ case class WholeStageCodegenExec(child: SparkPlan)
extends UnaryExecNode with Co
// Check if compiled code has a too large function
if (maxCodeSize > sqlContext.conf.hugeMethodLimit) {
- logWarning(s"Found too long generated codes and JIT optimization
might not work: " +
- s"the bytecode size was $maxCodeSize, this value went over the
limit " +
+ logInfo(s"Found too long generated codes and JIT optimization might
not work: " +
+ s"the bytecode size ($maxCodeSize) is above the limit " +
s"${sqlContext.conf.hugeMethodLimit}, and the whole-stage codegen
was disabled " +
s"for this plan. To avoid this, you can raise the limit " +
- s"${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}:\n$treeString")
- return child.execute()
+ s"`${SQLConf.WHOLESTAGE_HUGE_METHOD_LIMIT.key}`:\n$treeString")
+ child match {
+ // For batch file source scan, we should continue executing it
+ case f: FileSourceScanExec if f.supportsBatch => // do nothing
--- End diff --
yea, I totally agree that we need to refactor this in future. Anyway, it's
ok for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
