Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19480#discussion_r144684397
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
---
@@ -277,13 +292,25 @@ class CodegenContext {
funcName: String,
funcCode: String,
inlineToOuterClass: Boolean = false): String = {
+ val newFunction = addNewFunctionInternal(funcName, funcCode,
inlineToOuterClass)
+ newFunction match {
+ case NewFunction(functionName, None, None) => functionName
+ case NewFunction(functionName, Some(_), Some(subclassInstance)) =>
+ subclassInstance + "." + functionName
+ }
+ }
+
+ private[this] def addNewFunctionInternal(
+ funcName: String,
+ funcCode: String,
+ inlineToOuterClass: Boolean): NewFunction = {
// The number of named constants that can exist in the class is
limited by the Constant Pool
// limit, 65,536. We cannot know how many constants will be inserted
for a class, so we use a
- // threshold of 1600k bytes to determine when a function should be
inlined to a private, nested
+ // threshold of 1000k bytes to determine when a function should be
inlined to a private, nested
// sub-class.
val (className, classInstance) = if (inlineToOuterClass) {
outerClassName -> ""
- } else if (currClassSize > 1600000) {
+ } else if (currClassSize > 1000000) {
--- End diff --
Yeah, actually during several tries, I found that setting the value lower
can somehow reduce the chance to hit constant pool limit exception in nested
classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
