[jira] [Commented] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034223#comment-16034223 ] Nils Grabbert commented on SPARK-19104: --- [~marmbrus] Why are you moving this major bug to 2.3.0? As [~nathanwilliamgr...@gmail.com] has already mentioned, it is now almost impossible to work with case classes. > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0, 2.2.0 >Reporter: Nils Grabbert > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > {code} > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) > > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) > > at
[jira] [Commented] (SPARK-18891) Support for specific collection types
[ https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994566#comment-15994566 ] Nils Grabbert commented on SPARK-18891: --- This will not work on v2.2.0-rc1: {code} Seq((Map(1 -> (2,"abc")), List(3))).toDS {code} > Support for specific collection types > - > > Key: SPARK-18891 > URL: https://issues.apache.org/jira/browse/SPARK-18891 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.6.3, 2.1.0 >Reporter: Michael Armbrust >Priority: Critical > Fix For: 2.2.0 > > > Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which > force users to only define classes with the most generic type. > An [example > error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]: > {code} > case class SpecificCollection(aList: List[Int]) > Seq(SpecificCollection(1 :: Nil)).toDS().collect() > {code} > {code} > java.lang.RuntimeException: Error while decoding: > java.util.concurrent.ExecutionException: java.lang.Exception: failed to > compile: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 98, Column 120: No applicable constructor/method found > for actual parameters "scala.collection.Seq"; candidates are: > "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)" > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994509#comment-15994509 ] Nils Grabbert edited comment on SPARK-19104 at 5/3/17 9:05 AM: --- [~maropu] I think this issue is not related to SPARK-18891. The example is still not working on 2.2 branch. was (Author: nils.grabbert): [~maropu] I think this issues is not related to SPARK-18891. The example is still not working on 2.2 branch. > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0 >Reporter: Nils Grabbert > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) >
[jira] [Commented] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994509#comment-15994509 ] Nils Grabbert commented on SPARK-19104: --- [~maropu] I think this issues is not related to SPARK-18891. The example is still not working on 2.2 branch. > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0 >Reporter: Nils Grabbert > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) > > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) > > at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) > at
[jira] [Commented] (SPARK-18891) Support for specific collection types
[ https://issues.apache.org/jira/browse/SPARK-18891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992672#comment-15992672 ] Nils Grabbert commented on SPARK-18891: --- Example in SPARK-19104 still not working. > Support for specific collection types > - > > Key: SPARK-18891 > URL: https://issues.apache.org/jira/browse/SPARK-18891 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.6.3, 2.1.0 >Reporter: Michael Armbrust >Priority: Critical > Fix For: 2.2.0 > > > Encoders treat all collections the same (i.e. {{Seq}} vs {{List}}) which > force users to only define classes with the most generic type. > An [example > error|https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/1023043053387187/2398463439880241/2840265927289860/latest.html]: > {code} > case class SpecificCollection(aList: List[Int]) > Seq(SpecificCollection(1 :: Nil)).toDS().collect() > {code} > {code} > java.lang.RuntimeException: Error while decoding: > java.util.concurrent.ExecutionException: java.lang.Exception: failed to > compile: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 98, Column 120: No applicable constructor/method found > for actual parameters "scala.collection.Seq"; candidates are: > "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)" > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
[ https://issues.apache.org/jira/browse/SPARK-19104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nils Grabbert updated SPARK-19104: -- Component/s: Optimizer > CompileException with Map and Case Class in Spark 2.1.0 > > > Key: SPARK-19104 > URL: https://issues.apache.org/jira/browse/SPARK-19104 > Project: Spark > Issue Type: Bug > Components: Optimizer, SQL >Affects Versions: 2.1.0 >Reporter: Nils Grabbert > > The following code will run with Spark 2.0.2 but not with Spark 2.1.0: > {code} > case class InnerData(name: String, value: Int) > case class Data(id: Int, param: Map[String, InnerData]) > val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + > 100 > val ds = spark.createDataset(data) > {code} > Exception: > Caused by: org.codehaus.commons.compiler.CompileException: File > 'generated.java', Line 63, Column 46: Expression > "ExternalMapToCatalyst_value_isNull1" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) > > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) > at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) > > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) > at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) > at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) > at > org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) > at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) > at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) > > at > org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) > > at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) > > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) > at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) > > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) > > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) > > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) > > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) > > at > org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) > > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) > at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) > at > org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) > > at > org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) > > at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) > at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196) > at
[jira] [Created] (SPARK-19104) CompileException with Map and Case Class in Spark 2.1.0
Nils Grabbert created SPARK-19104: - Summary: CompileException with Map and Case Class in Spark 2.1.0 Key: SPARK-19104 URL: https://issues.apache.org/jira/browse/SPARK-19104 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.1.0 Reporter: Nils Grabbert The following code will run with Spark 2.0.2 but not with Spark 2.1.0: {code} case class InnerData(name: String, value: Int) case class Data(id: Int, param: Map[String, InnerData]) val data = Seq.tabulate(10)(i => Data(1, Map("key" -> InnerData("name", i + 100 val ds = spark.createDataset(data) {code} Exception: Caused by: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 63, Column 46: Expression "ExternalMapToCatalyst_value_isNull1" is not an rvalue at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004) at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639) at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001) at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984) at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633) at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563) at org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956) at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189) at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143) at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139) at org.codehaus.janino.Java$Assignment.accept(Java.java:3847) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112) at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377) at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370) at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894) at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377) at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420) at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369) at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369) at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345) at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396) at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311) at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229) at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196) at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91) at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:935) ... 77 more -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For