[jira] [Assigned] (SPARK-25582) Error in Spark logs when using the org.apache.spark:spark-sql_2.11:2.2.0 Java library
[ https://issues.apache.org/jira/browse/SPARK-25582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25582: Assignee: (was: Apache Spark) > Error in Spark logs when using the org.apache.spark:spark-sql_2.11:2.2.0 Java > library > - > > Key: SPARK-25582 > URL: https://issues.apache.org/jira/browse/SPARK-25582 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.2.0 >Reporter: Thomas Brugiere >Priority: Major > Attachments: fileA.csv, fileB.csv, fileC.csv > > > I have noticed an error that appears in the Spark logs when using the Spark > SQL library in a Java 8 project. > When I run the code below with the attached files as input, I can see the > ERROR below in the application logs. > I am using the *org.apache.spark:spark-sql_2.11:2.2.0* library in my Java > project > Note that the same logic implemented with the Python API (pyspark) doesn't > produce any Exception like this. > *Code* > {code:java} > SparkConf conf = new SparkConf().setAppName("SparkBug").setMaster("local"); > SparkSession sparkSession = SparkSession.builder().config(conf).getOrCreate(); > Dataset df_a = sparkSession.read().option("header", > true).csv("local/fileA.csv").dropDuplicates(); > Dataset df_b = sparkSession.read().option("header", > true).csv("local/fileB.csv").dropDuplicates(); > Dataset df_c = sparkSession.read().option("header", > true).csv("local/fileC.csv").dropDuplicates(); > String[] key_join_1 = new String[]{"colA", "colB", "colC", "colD", "colE", > "colF"}; > String[] key_join_2 = new String[]{"colA", "colB", "colC", "colD", "colE"}; > Dataset df_inventory_1 = df_a.join(df_b, arrayToSeq(key_join_1), "left"); > Dataset df_inventory_2 = df_inventory_1.join(df_c, > arrayToSeq(key_join_2), "left"); > df_inventory_2.show(); > {code} > *Error message* > {code:java} > 18/10/01 09:58:07 ERROR CodeGenerator: failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 202, Column 18: Expression "agg_isNull_28" is not an rvalue > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 202, Column 18: Expression "agg_isNull_28" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:7170) > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5332) > at org.codehaus.janino.UnitCompiler.access$9400(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$13$1.visitAmbiguousName(UnitCompiler.java:5287) > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4053) > at org.codehaus.janino.UnitCompiler$13.visitLvalue(UnitCompiler.java:5284) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3977) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:5280) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2391) > at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1474) > at > org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1466) > at org.codehaus.janino.Java$IfStatement.accept(Java.java:2926) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1466) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1546) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3075) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1336) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1309) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:799) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:958) > at org.codehaus.janino.UnitCompiler.access$700(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:393) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:385) > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1286) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:385) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1285) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:825) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:411) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:390) >
[jira] [Assigned] (SPARK-25582) Error in Spark logs when using the org.apache.spark:spark-sql_2.11:2.2.0 Java library
[ https://issues.apache.org/jira/browse/SPARK-25582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25582: Assignee: Apache Spark > Error in Spark logs when using the org.apache.spark:spark-sql_2.11:2.2.0 Java > library > - > > Key: SPARK-25582 > URL: https://issues.apache.org/jira/browse/SPARK-25582 > Project: Spark > Issue Type: Bug > Components: Java API >Affects Versions: 2.2.0 >Reporter: Thomas Brugiere >Assignee: Apache Spark >Priority: Major > Attachments: fileA.csv, fileB.csv, fileC.csv > > > I have noticed an error that appears in the Spark logs when using the Spark > SQL library in a Java 8 project. > When I run the code below with the attached files as input, I can see the > ERROR below in the application logs. > I am using the *org.apache.spark:spark-sql_2.11:2.2.0* library in my Java > project > Note that the same logic implemented with the Python API (pyspark) doesn't > produce any Exception like this. > *Code* > {code:java} > SparkConf conf = new SparkConf().setAppName("SparkBug").setMaster("local"); > SparkSession sparkSession = SparkSession.builder().config(conf).getOrCreate(); > Dataset df_a = sparkSession.read().option("header", > true).csv("local/fileA.csv").dropDuplicates(); > Dataset df_b = sparkSession.read().option("header", > true).csv("local/fileB.csv").dropDuplicates(); > Dataset df_c = sparkSession.read().option("header", > true).csv("local/fileC.csv").dropDuplicates(); > String[] key_join_1 = new String[]{"colA", "colB", "colC", "colD", "colE", > "colF"}; > String[] key_join_2 = new String[]{"colA", "colB", "colC", "colD", "colE"}; > Dataset df_inventory_1 = df_a.join(df_b, arrayToSeq(key_join_1), "left"); > Dataset df_inventory_2 = df_inventory_1.join(df_c, > arrayToSeq(key_join_2), "left"); > df_inventory_2.show(); > {code} > *Error message* > {code:java} > 18/10/01 09:58:07 ERROR CodeGenerator: failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 202, Column 18: Expression "agg_isNull_28" is not an rvalue > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 202, Column 18: Expression "agg_isNull_28" is not an rvalue > at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11821) > at > org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:7170) > at > org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5332) > at org.codehaus.janino.UnitCompiler.access$9400(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$13$1.visitAmbiguousName(UnitCompiler.java:5287) > at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4053) > at org.codehaus.janino.UnitCompiler$13.visitLvalue(UnitCompiler.java:5284) > at org.codehaus.janino.Java$Lvalue.accept(Java.java:3977) > at > org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:5280) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2391) > at org.codehaus.janino.UnitCompiler.access$1900(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1474) > at > org.codehaus.janino.UnitCompiler$6.visitIfStatement(UnitCompiler.java:1466) > at org.codehaus.janino.Java$IfStatement.accept(Java.java:2926) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1466) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1546) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3075) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1336) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1309) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:799) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:958) > at org.codehaus.janino.UnitCompiler.access$700(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:393) > at > org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:385) > at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1286) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:385) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1285) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:825) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:411) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212) > at >