Re: Problems while moving from 0.8.0 to 0.8.1

The code would not call null.equals(b); only a.equals(null) is allowable (we check for k.eq(null) at the top). a.equals(null) returning false is part of the Object.equals() contract, so it is not invalid to us

Re: Problems while moving from 0.8.0 to 0.8.1

On Tue, Jan 28, 2014 at 1:25 AM, Aaron Davidson wrote: > Looks like your MyCustomKeyType.equals() method doesn't correctly handle a > null argument. In general, the contract of equals is to return false if > called with a null argument, which this code currently relies on. > > Ok. But what if it

Re: Problems while moving from 0.8.0 to 0.8.1

Looks like your MyCustomKeyType.equals() method doesn't correctly handle a null argument. In general, the contract of equals is to return false if called with a null argument, which this code currently relies on. This could still be patched for the sake of backwards compatibility with similarly in

Re: Problems while moving from 0.8.0 to 0.8.1

ERROR executor.Executor: Exception in task ID 20 java.lang.NullPointerException at com.xyz.spark.common.collection.MyCustomKeyType.equals(MyCustomKeyType.java:200) at org.apache.spark.util.AppendOnlyMap.changeValue(AppendOnlyMap.scala:122) at org.apache.spark.Aggregator.comb

Re: Problems while moving from 0.8.0 to 0.8.1

Do you mind pasting the whole stack trace for the NPE? On Mon, Jan 27, 2014 at 6:44 AM, Archit Thakur wrote: > Hi, > > Implementation of aggregation logic has been changed with 0.8.1 > (Aggregator.scala) > > It is now using AppendOnlyMap as compared to java.util.HashMap in 0.8.0 > release. > >

Problems while moving from 0.8.0 to 0.8.1

Hi, Implementation of aggregation logic has been changed with 0.8.1 (Aggregator.scala) It is now using AppendOnlyMap as compared to java.util.HashMap in 0.8.0 release. Aggregator.scala def combineValuesByKey(iter: Iterator[_ <: Product2[K, V]]) : Iterator[(K, C)] = { val combiners = new Appe