Re: getting NullPointerException while doing left outer join

For anyone interested, the problem ended up being that in some rare cases, the value from the pair RDD on the right side of the left outer join was Java's null. The Spark optionToOptional method attempted to apply Some() to null, which caused the NPE to be thrown. The lesson is to filter out any

Re: getting NullPointerException while doing left outer join

Hi Ted, I am working on replicating the problem on a smaller scale. I saw that Spark 2.0 is moving to Java 8 Optional instead of Guava Optional, but in the meantime I'm stuck with 1.6.1. -Adam On Fri, May 6, 2016 at 9:40 AM, Ted Yu wrote: > Is it possible to write a

Re: getting NullPointerException while doing left outer join

Is it possible to write a short test which exhibits this problem ? For Spark 2.0, this part of code has changed: [SPARK-4819] Remove Guava's "Optional" from public API FYI On Fri, May 6, 2016 at 6:57 AM, Adam Westerman wrote: > Hi, > > I’m attempting to do a left outer

getting NullPointerException while doing left outer join

Hi, I’m attempting to do a left outer join in Spark, and I’m getting an NPE that appears to be due to some Spark Java API bug. (I’m running Spark 1.6.0 in local mode on a Mac). For a little background, the left outer join returns all keys from the left side of the join regardless of whether or