Github user bdrillard commented on a diff in the pull request: https://github.com/apache/spark/pull/19811#discussion_r153534679 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -177,11 +190,67 @@ class CodegenContext { * the list of default imports available. * Also, generic type arguments are accepted but ignored. * @param variableName Name of the field. - * @param initCode The statement(s) to put into the init() method to initialize this field. + * @param codeFunctions Function includes statement(s) to put into the init() method to + * initialize this field. An argument is the name of the mutable state variable * If left blank, the field will be default-initialized. + * @param inline whether the declaration and initialization code may be inlined rather than + * compacted. If true, the name is not changed + * @return the name of the mutable state variable, which is either the original name if the + * variable is inlined to the outer class, or an array access if the variable is to be + * stored in an array of variables of the same type and initialization. + * primitive type variables will be inlined into outer class when the total number of + * mutable variables is less than `CodeGenerator.OUTER_CLASS_VARIABLES_THRESHOLD` + * the max size of an array for compaction is given by + * `CodeGenerator.MUTABLESTATEARRAY_SIZE_LIMIT`. */ - def addMutableState(javaType: String, variableName: String, initCode: String = ""): Unit = { - mutableStates += ((javaType, variableName, initCode)) + def addMutableState( + javaType: String, + variableName: String, + codeFunctions: String => String = _ => "", + inline: Boolean = false): String = { + val varName = if (!inline) freshName(variableName) else variableName + val initCode = codeFunctions(varName) + + if (inline || + // want to put a primitive type variable at outerClass for performance + isPrimitiveType(javaType) && + (mutableStates.length < CodeGenerator.OUTER_CLASS_VARIABLES_THRESHOLD) || + // identify multi-dimensional array or no simply-assigned object + !isPrimitiveType(javaType) && + (javaType.contains("[][]") || + !initCode.matches("(^[\\w_]+\\d+\\s*=\\s*null;|" --- End diff -- @mgaido91, could you describe what you mean by "Isn't it enough that the init code used is always the same?" There are definitely some [complicated init codes](https://github.com/apache/spark/pull/19811/files/006b2fdad62fe64b623235314105c7b6f4849b5d#diff-39298b470865a4cbc67398a4ea11e767R122) used throughout the codebase where, I think as @kiszk was saying, the initcode makes use of a previously defined variable. Really it would be nice if we had a way of _knowing_ whether an initialization was simple (assigned to a default for primitives, or null or the 0-parameter constructor for objects). Maybe we could define an abstract `InitCode` holding a single `code` field and then extend that with `Simple` and `NonSimple` case classes, then we could pattern match on the additional type information rather than trying to regex match the code itself.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org