to see if there could be with any improvement, but
>> we assume your code is already optimized, as in your email. If the above
>> steps still cannot help your OOM, then maybe your data for one partition
>> just cannot fit with 12G heap, based on the logic you try to do in your
me your code is already optimized, as in your email. If the above steps
> still cannot help your OOM, then maybe your data for one partition just
> cannot fit with 12G heap, based on the logic you try to do in your code.
>
> Yong
>
> ------------------
> From: deepu
30
Subject: Expert advise needed. (POC is at crossroads)
To: user@spark.apache.org
I am at crossroads now and expert advise help me decide what the next course of
the project going to be.
Background : At out company we process tons of data to help build
experimentation platform. We fire more th
I am at crossroads now and expert advise help me decide what the next
course of the project going to be.
Background : At out company we process tons of data to help build
experimentation platform. We fire more than 300s of M/R jobs, Peta bytes of
data, takes 24 hours and does lots of joins. Its si
