Sounds like: https://issues.apache.org/jira/browse/SPARK-15441, for which a
fix is in progress.
Please do keep reporting issues though, these are great!
Michael
On Fri, May 27, 2016 at 1:01 PM, Tim Gautier wrote:
> Is it truly impossible to left join a Dataset[T] on the right if T has any
> no
When I run it in 1.6.1 I get this:
java.lang.RuntimeException: Error while decoding:
java.lang.RuntimeException: Null value appeared in non-nullable field:
- field (class: "scala.Int", name: "id")
- root class: "$iwC.$iwC.Test"
If the schema is inferred from a Scala tuple/case class, or a Java bea
Interesting, I did that on 1.6.1, Scala 2.10
On Fri, May 27, 2016 at 2:41 PM Ted Yu wrote:
> Which release did you use ?
>
> I tried your example in master branch:
>
> scala> val test2 = Seq(Test(2), Test(3), Test(4)).toDS
> test2: org.apache.spark.sql.Dataset[Test] = [id: int]
>
> scala> test1
Which release did you use ?
I tried your example in master branch:
scala> val test2 = Seq(Test(2), Test(3), Test(4)).toDS
test2: org.apache.spark.sql.Dataset[Test] = [id: int]
scala> test1.as("t1").joinWith(test2.as("t2"), $"t1.id" === $"t2.id",
"left_outer").show
+---+--+
| _1|_2|
+---
Is it truly impossible to left join a Dataset[T] on the right if T has any
non-option fields? It seems Spark tries to create Ts with null values in
all fields when left joining, which results in null pointer exceptions. In
fact, I haven't found any other way to get around this issue without making
