SO it was indeed my merge function. I created new result object for every
merge and its working now.
Thanks
On Wed, Jun 22, 2016 at 3:46 PM, Nirav Patel <npa...@xactlycorp.com> wrote:
> PS. In my reduceByKey operation I have two mutable object. What I do is
> merge mutable2 into mutable1 and return mutable1. I read that it works for
> aggregateByKey so thought it will work for reduceByKey as well. I might be
> wrong here. Can someone verify if this will work or be un predictable?
>
> On Wed, Jun 22, 2016 at 11:52 AM, Nirav Patel <npa...@xactlycorp.com>
> wrote:
>
>> Hi,
>>
>> I do not see any indication of errors or executor getting killed in spark
>> UI - jobs, stages, event timelines. No task failures. I also don't see any
>> errors in executor logs.
>>
>> Thanks
>>
>> On Wed, Jun 22, 2016 at 2:32 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>>
>>> For the run which returned incorrect result, did you observe any error
>>> (on workers) ?
>>>
>>> Cheers
>>>
>>> On Tue, Jun 21, 2016 at 10:42 PM, Nirav Patel <npa...@xactlycorp.com>
>>> wrote:
>>>
>>>> I have an RDD[String, MyObj] which is a result of Join + Map operation.
>>>> It has no partitioner info. I run reduceByKey without passing any
>>>> Partitioner or partition counts. I observed that output aggregation result
>>>> for given key is incorrect sometime. like 1 out of 5 times. It looks like
>>>> reduce operation is joining values from two different keys. There is no
>>>> configuration change between multiple runs. I am scratching my head over
>>>> this. I verified results by printing out RDD before and after reduce
>>>> operation; collecting subset at driver.
>>>>
>>>> Besides shuffle and storage memory fraction I use following options:
>>>>
>>>> sparkConf.set("spark.driver.userClassPathFirst","true")
>>>> sparkConf.set("spark.unsafe.offHeap","true")
>>>> sparkConf.set("spark.reducer.maxSizeInFlight","128m")
>>>> sparkConf.set("spark.serializer",
>>>> "org.apache.spark.serializer.KryoSerializer")
>>>>
>>>>
>>>>
>>>> [image: What's New with Xactly]
>>>> <http://www.xactlycorp.com/email-click/>
>>>>
>>>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
>>>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
>>>> <https://twitter.com/Xactly> [image: Facebook]
>>>> <https://www.facebook.com/XactlyCorp> [image: YouTube]
>>>> <http://www.youtube.com/xactlycorporation>
>>>
>>>
>>>
>>
>
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
