[ https://issues.apache.org/jira/browse/SPARK-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277221#comment-15277221 ]
Shixiong Zhu commented on SPARK-14983: -------------------------------------- In addition, the Scala type for `ArrayType[String]` is `Seq[String]`. You should use it instead of `Array[String]` as the UDF parameter type. > Getting CompileException when feed an UDF with an array and another paramter > ---------------------------------------------------------------------------- > > Key: SPARK-14983 > URL: https://issues.apache.org/jira/browse/SPARK-14983 > Project: Spark > Issue Type: Bug > Components: PySpark, Spark Core > Affects Versions: 1.5.0 > Environment: Spark 1.5.0 on EMR > Reporter: Sky Yin > > I'm trying to apply an UDF to an array column (generated from the {{array}} > funciton in SparkSQL) and another string column and got this error: > (some long source code before this...) > {code} > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:392) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:412) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:409) > at > org.spark-project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599) > at > org.spark-project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379) > ... 31 more > Caused by: org.codehaus.commons.compiler.CompileException: Line 1038, Column > 28: Redefinition of local variable "values" > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10174) > at > org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2554) > at org.codehaus.janino.UnitCompiler.access$4300(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:2434) > at > org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:2508) > at > org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2437) > at > org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2395) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2250) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:822) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:794) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:507) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:658) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:662) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:350) > at > org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1035) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:769) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:532) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:347) > at > org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1139) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:383) > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:315) > at > org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233) > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192) > at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84) > at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77) > at > org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:387) > ... 35 more > {code} > I tried to create another UDF with an array as the only parameter and that > worked well as the array got passed into the Python function as a list > without any problem. > Also I tried to define an UDF with an array and a string as paramters in > Scala and got the same {{org.codehaus.commons.compiler.CompileException: Line > 1038, Column 28: Redefinition of local variable "values"}} error. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org