Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19083#discussion_r142058372
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -1092,13 +1082,30 @@ object CodeGenerator extends Logging {
logInfo(s"\n${CodeFormatter.format(code, maxLines)}")
throw new CompileException(msg, e.getLocation)
}
+
+ // Check if compiled code has a too large function
+ methodsToByteCodeSize.foreach { case (name, byteCodeSize) =>
+ if (byteCodeSize > SQLConf.get.hugeMethodLimit) {
+ val clazzName = evaluator.getClazz.getSimpleName
+ val methodName = name.replace("$", "")
+ val msg = s"failed to compile: the size of $clazzName.$methodName
was $byteCodeSize and " +
+ "this value went over the limit
`spark.sql.codegen.hugeMethodLimit`" +
+ s"(${SQLConf.get.hugeMethodLimit}). To avoid this error, you can
make this limit higher."
+ logError(msg)
+ val maxLines = SQLConf.get.loggingMaxLinesForCodegen
+ logInfo(s"\n${CodeFormatter.format(code, maxLines)}")
+ throw new CompileException(msg, null)
--- End diff --
We should turn back to non-codegen execution for too long function.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
