Re: Maximum limit for akka.frame.size be greater than 500 MB ?

Currently, I am using 1.6.1 version. I continue to use it as my current code is heavily reliant on RDD's and not dataframes. Also, because 1.6.1 is stabler than newer versions. The input data is user behavior data of 20 fields and 1 billion records (~ 1.5 TB) . I am trying to group by user id

Re: Maximum limit for akka.frame.size be greater than 500 MB ?

Which Spark version are you using? What are you trying to do exactly and what is the input data? As far as I know, akka has been dropped in recent Spark versions. > On 30 Jan 2017, at 00:44, aravasai wrote: > > I have a spark job running on 2 terabytes of data which

Re: Error Saving Dataframe to Hive with Spark 2.0.0

Okey, you are saying that 2.0.0 don't have that patch fixed ? @dev cc-- I don't like everytime changing the service versions ! Thanks. On Mon, Jan 30, 2017 at 1:10 AM, Jacek Laskowski wrote: > Hi, > > I think you have to upgrade to 2.1.0. There were few changes wrt the ERROR >

Maximum limit for akka.frame.size be greater than 500 MB ?

I have a spark job running on 2 terabytes of data which creates more than 30,000 partitions. As a result, the spark job fails with the error "Map output statuses were 170415722 bytes which exceeds spark.akka.frameSize 52428800 bytes" (For 1 TB data) However, when I increase the akka.frame.size to