Hi ,
I am facing the following when running on EMR
Caused by: java.lang.IllegalStateException: There is no space for new record
at org.apache.spark.util.collection.unsafe.sort.
UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226)
at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(
UnsafeKVExternalSorter.java:132)
at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.
destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
I am using pyspark 2.2 , what spark configuration should be
changed/modified to get this resolved
Regards,
Snehasish
Regards,
Snehasish
On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA <info.snehas...@gmail.com>
wrote:
> Hi ,
>
> I am facing the following when running on EMR
>
> Caused by: java.lang.IllegalStateException: There is no space for new
> record
> at org.apache.spark.util.collection.unsafe.sort.
> UnsafeInMemorySorter.insertRecord(UnsafeInMemorySorter.java:226)
> at org.apache.spark.sql.execution.UnsafeKVExternalSorter.<init>(
> UnsafeKVExternalSorter.java:132)
> at org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.
> destructAndCreateExternalSorter(UnsafeFixedWidthAggregationMap.java:250)
>
> I am using spark 2.2 , what spark configuration should be changed/modified
> to get this resolved
>
>
> Regards,
> Snehasish
>
