[jira] [Comment Edited] (SPARK-22296) CodeGenerator - failed to compile when constructor has scala.collection.mutable.Seq vs. scala.collection.Seq
[ https://issues.apache.org/jira/browse/SPARK-22296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213015#comment-16213015 ] Randy Tidd edited comment on SPARK-22296 at 10/20/17 6:34 PM: -- Thank you I was just installing 2.2.0 and related components to try this but you beat me to it. Glad to hear it's fixed. Note that the example provided above by Liang-Chi Hsieh does not exhibit the problem in 2.1.0 or 2.1.1. was (Author: tiddman): Thank you I was just installing 2.2.0 and related components to try this but you beat me to it. Glad to hear it's fixed. > CodeGenerator - failed to compile when constructor has > scala.collection.mutable.Seq vs. scala.collection.Seq > > > Key: SPARK-22296 > URL: https://issues.apache.org/jira/browse/SPARK-22296 > Project: Spark > Issue Type: Bug > Components: Optimizer >Affects Versions: 2.1.0 >Reporter: Randy Tidd > > This is with Scala 2.11. > We have a case class that has a constructor with 85 args, the last two of > which are: > var chargesInst : > scala.collection.mutable.Seq[ChargeInstitutional] = > scala.collection.mutable.Seq.empty[ChargeInstitutional], > var chargesProf : > scala.collection.mutable.Seq[ChargeProfessional] = > scala.collection.mutable.Seq.empty[ChargeProfessional] > A mutable Seq in a the constructor of a case class is probably poor form but > Scala allows it. When we run this job we get this error: > build 17-Oct-2017 05:30:502017-10-17 09:30:50 [Executor task launch > worker-1] ERROR o.a.s.s.c.e.codegen.CodeGenerator - failed to compile: > org.codehaus.commons.compiler.CompileException: File 'generated.java', Line > 8217, Column 70: No applicable constructor/method found for actual parameters > "java.lang.String, java.lang.String, long, java.lang.String, long, long, > long, java.lang.String, long, long, double, scala.Option, scala.Option, > java.lang.String, java.lang.String, long, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, long, long, long, long, long, > scala.Option, scala.Option, scala.Option, scala.Option, scala.Option, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, long, java.lang.String, int, double, > double, java.lang.String, java.lang.String, java.lang.String, long, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, long, > long, long, long, java.lang.String, com.xyz.xyz.xyz.domain.Patient, > com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq, > java.lang.String, long, java.lang.String, int, int, boolean, boolean, > scala.collection.Seq, boolean, scala.collection.Seq, boolean, > scala.collection.Seq, scala.collection.Seq"; candidates are: > "com.xyz.xyz.xyz.domain.Account(java.lang.String, java.lang.String, long, > java.lang.String, long, long, long, java.lang.String, long, long, double, > scala.Option, scala.Option, java.lang.String, java.lang.String, long, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, int, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, long, > long, long, long, long, scala.Option, scala.Option, scala.Option, > scala.Option, scala.Option, java.lang.String, java.lang.String, > java.lang.String, java.lang.String, java.lang.String, java.lang.String, long, > java.lang.String, int, double, double, java.lang.String, java.lang.String, > java.lang.String, long, java.lang.String, java.lang.String, java.lang.String, > java.lang.String, long, long, long, long, java.lang.String, > com.xyz.xyz.xyz.domain.Patient, com.xyz.xyz.xyz.domain.Physician, > scala.collection.Seq, scala.collection.Seq, java.lang.String, long, > java.lang.String, int, int, boolean, boolean, scala.collection.Seq, boolean, > scala.collection.Seq, boolean, scala.collection.mutable.Seq, > scala.collection.mutable.Seq)" > The relevant lines are: > build 17-Oct-2017 05:30:50/* 093 */ private scala.collection.Seq > argValue84; > build 17-Oct-2017 05:30:50/* 094 */ private scala.collection.Seq > argValue85; > and > build 17-Oct-2017 05:30:54/* 8217 */ final > com.xyz.xyz.xyz.domain.Account value1 = false ? null : new > com.xyz.xyz.xyz.domain.Account(argValue2, argValue3
[jira] [Comment Edited] (SPARK-22296) CodeGenerator - failed to compile when constructor has scala.collection.mutable.Seq vs. scala.collection.Seq
[
https://issues.apache.org/jira/browse/SPARK-22296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212983#comment-16212983
]
Randy Tidd edited comment on SPARK-22296 at 10/20/17 6:17 PM:
--
The example above does not exhibit the problem. Here is a concise example that
does.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
17/10/20 14:09:04 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
17/10/20 14:09:08 WARN ObjectStore: Failed to get database global_temp,
returning NoSuchObjectException
Spark context Web UI available at http://10.10.43.134:4040
Spark context available as 'sc' (master = local[*], app id =
local-1508522945163).
Spark session available as 'spark'.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
scala> case class Foo1(x: Int, s: String, seq:
scala.collection.mutable.Seq[Int] = scala.collection.mutable.Seq.empty[Int])
defined class Foo1
scala> val ds1 = Seq(Foo1(1, "a"), Foo1(2, "b")).toDS
ds1: org.apache.spark.sql.Dataset[Foo1] = [x: int, s: string ... 1 more field]
scala>
scala> case class Foo2(x: Int, s: String)
defined class Foo2
scala> val ds2 = Seq(Foo2(1, "aa"), Foo2(3, "cc")).toDS
ds2: org.apache.spark.sql.Dataset[Foo2] = [x: int, s: string]
scala>
scala> ds1.joinWith(ds2, ds1.col("x") === ds2.col("x")).collect()
17/10/20 14:09:14 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
263, Column 69: No applicable constructor/method found for actual parameters
"int, java.lang.String, scala.collection.Seq"; candidates are:
"$line14.$read$$iw$$iw$Foo1(int, java.lang.String,
scala.collection.mutable.Seq)"
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
/* 006 */
/* 007 */ private Object[] references;
/* 008 */ private InternalRow mutableRow;
/* 009 */ private $line14.$read$$iw$$iw$Foo1 argValue;
/* 010 */ private $line16.$read$$iw$$iw$Foo2 argValue1;
/* 011 */ private int argValue2;
/* 012 */ private java.lang.String argValue3;
/* 013 */ private scala.collection.Seq argValue4;
/* 014 */ private boolean resultIsNull;
/* 015 */ private java.lang.Object[] argValue5;
/* 016 */ private boolean MapObjects_loopIsNull1;
/* 017 */ private int MapObjects_loopValue0;
/* 018 */ private int argValue6;
/* 019 */ private java.lang.String argValue7;
/* 020 */ private boolean isNull31;
/* 021 */ private boolean value31;
/* 022 */ private boolean isNull32;
/* 023 */ private $line16.$read$$iw$$iw$Foo2 value32;
/* 024 */ private boolean isNull33;
/* 025 */ private $line16.$read$$iw$$iw$Foo2 value33;
/* 026 */
/* 027 */ public SpecificSafeProjection(Object[] references) {
/* 028 */ this.references = references;
/* 029 */ mutableRow = (InternalRow) references[references.length - 1];
/* 030 */
/* 031 */
/* 032 */
/* 033 */
/* 034 */
/* 035 */
/* 036 */
/* 037 */
/* 038 */
/* 039 */
/* 040 */
/* 041 */ isNull31 = false;
/* 042 */ value31 = false;
/* 043 */ isNull32 = false;
/* 044 */ value32 = null;
/* 045 */ isNull33 = false;
/* 046 */ value33 = null;
/* 047 */
/* 048 */ }
/* 049 */
/* 050 */ public void initialize(int partitionIndex) {
/* 051 */
/* 052 */ }
/* 053 */
/* 054 */
/* 055 */ private void evalIfTrueExpr(InternalRow i) {
/* 056 */ final $line16.$read$$iw$$iw$Foo2 value22 = null;
/* 057 */ isNull32 = true;
/* 058 */ value32 = value22;
/* 059 */ }
/* 060 */
/* 061 */
/* 062 */ private void apply_1(InternalRow i) {
/* 063 */
/* 064 */
/* 065 */ resultIsNull = false;
/* 066 */ if (!resultIsNull) {
/* 067 */
/* 068 */ InternalRow value16 = i.getStruct(0, 3);
/* 069 */ boolean isNull15 = false;
/* 070 */ ArrayData value15 = null;
/* 071 */
/* 072 */
/* 073 */ if (value16.isNullAt(2)) {
/* 074 */ isNull15 = true;
/* 075 */ } else {
/* 076 */ value15 = value16.getArray(2);
/* 077 */ }
/* 078 */ ArrayData value14 = null;
/* 079 */
/* 080 */ if (!isNull15) {
/* 081 */
/* 082 */ Integer[] convertedArray = null;
/* 083 */ int dataLength = value15.numElements();
/* 084 */ convertedArray = new Integer[dataLength];
/* 085 */
/*
