- user@incubator Hi Keith,
I did reproduce this using local-cluster[2,2,1024], and the errors look almost the same. Just wondering, despite the errors did your program output any result for the join? On my machine, I could see the correct output. Zongheng On Tue, Jul 15, 2014 at 1:46 PM, Michael Armbrust <mich...@databricks.com> wrote: > Thanks for the extra info. At a quick glance the query plan looks fine to > me. The class IntegerType does build a type tag.... I wonder if you are > seeing the Scala issue manifest in some new way. We will attempt to > reproduce locally. > > > On Tue, Jul 15, 2014 at 1:41 PM, gorenuru <goren...@gmail.com> wrote: >> >> Just my "few cents" on this. >> >> I having the same problems with v 1.0.1 but this bug is sporadic and looks >> like is relayed to object initialization. >> >> Even more, i'm not using any SQL or something. I just have utility class >> like this: >> >> object DataTypeDescriptor { >> type DataType = String >> >> val BOOLEAN = "BOOLEAN" >> val STRING = "STRING" >> val TIMESTAMP = "TIMESTAMP" >> val LONG = "LONG" >> val INT = "INT" >> val SHORT = "SHORT" >> val BYTE = "BYTE" >> val DECIMAL = "DECIMAL" >> val DOUBLE = "DOUBLE" >> val FLOAT = "FLOAT" >> >> def $$(name: String, format: Option[String] = None) = >> DataTypeDescriptor(name, format) >> >> private lazy val nativeTypes: Map[String, NativeType] = Map( >> BOOLEAN -> BooleanType, STRING -> StringType, TIMESTAMP -> >> TimestampType, LONG -> LongType, INT -> IntegerType, >> SHORT -> ShortType, BYTE -> ByteType, DECIMAL -> DecimalType, DOUBLE >> -> >> DoubleType, FLOAT -> FloatType >> ) >> >> lazy val defaultValues: Map[String, Any] = Map( >> BOOLEAN -> false, STRING -> "", TIMESTAMP -> null, LONG -> 0L, INT -> >> 0, >> SHORT -> 0.toShort, BYTE -> 0.toByte, >> DECIMAL -> BigDecimal(0d), DOUBLE -> 0d, FLOAT -> 0f >> ) >> >> def apply(dataType: String): DataTypeDescriptor = { >> DataTypeDescriptor(dataType.toUpperCase, None) >> } >> >> def apply(dataType: SparkDataType): DataTypeDescriptor = { >> nativeTypes >> .find { case (_, descriptor) => descriptor == dataType } >> .map { case (name, descriptor) => DataTypeDescriptor(name, None) } >> .get >> } >> >> ..... >> >> and some test that check each of this methods. >> >> The problem is that this test fails randomly with this error. >> >> P.S.: I did not have this problem in Spark 1.0.0 >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Error-while-running-Spark-SQL-join-when-using-Spark-1-0-1-tp9776p9817.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > >