[
https://issues.apache.org/jira/browse/SPARK-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15277221#comment-15277221
]
Shixiong Zhu commented on SPARK-14983:
--------------------------------------
In addition, the Scala type for `ArrayType[String]` is `Seq[String]`. You
should use it instead of `Array[String]` as the UDF parameter type.
> Getting CompileException when feed an UDF with an array and another paramter
> ----------------------------------------------------------------------------
>
> Key: SPARK-14983
> URL: https://issues.apache.org/jira/browse/SPARK-14983
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
> Affects Versions: 1.5.0
> Environment: Spark 1.5.0 on EMR
> Reporter: Sky Yin
>
> I'm trying to apply an UDF to an array column (generated from the {{array}}
> funciton in SparkSQL) and another string column and got this error:
> (some long source code before this...)
> {code}
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:392)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:412)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:409)
> at
> org.spark-project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
> org.spark-project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> ... 31 more
> Caused by: org.codehaus.commons.compiler.CompileException: Line 1038, Column
> 28: Redefinition of local variable "values"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10174)
> at
> org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2554)
> at org.codehaus.janino.UnitCompiler.access$4300(UnitCompiler.java:185)
> at
> org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:2434)
> at
> org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:2508)
> at
> org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2437)
> at
> org.codehaus.janino.UnitCompiler.buildLocalVariableMap(UnitCompiler.java:2395)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2250)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:822)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:794)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:507)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:658)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:662)
> at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:185)
> at
> org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:350)
> at
> org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1035)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354)
> at
> org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:769)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:532)
> at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393)
> at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185)
> at
> org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:347)
> at
> org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1139)
> at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354)
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:383)
> at
> org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:315)
> at
> org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:77)
> at
> org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:387)
> ... 35 more
> {code}
> I tried to create another UDF with an array as the only parameter and that
> worked well as the array got passed into the Python function as a list
> without any problem.
> Also I tried to define an UDF with an array and a string as paramters in
> Scala and got the same {{org.codehaus.commons.compiler.CompileException: Line
> 1038, Column 28: Redefinition of local variable "values"}} error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
