Hi, Will it be possible to overcome this with some spark configuration tweak , since EMR has spark version available only till 2.2.1
Regards, Snehasish On Tue, Feb 13, 2018 at 2:00 PM, Marco Gaido <marcogaid...@gmail.com> wrote: > You can check all the versions where the fix is available on the > JIRA SPARK-23376. Anyway it will be available in the upcoming 2.3.0 release. > > Thanks. > > On 13 Feb 2018 9:09 a.m., "SNEHASISH DUTTA" <info.snehas...@gmail.com> > wrote: > >> Hi, >> >> In which version of Spark will this fix be available ? >> The deployment is on EMR >> >> Regards, >> Snehasish >> >> On Fri, Feb 9, 2018 at 8:51 PM, Wenchen Fan <cloud0...@gmail.com> wrote: >> >>> It should be fixed by https://github.com/apache/spark/pull/20561 soon. >>> >>> On Fri, Feb 9, 2018 at 6:16 PM, Wenchen Fan <cloud0...@gmail.com> wrote: >>> >>>> This has been reported before: http://apache-spark-de >>>> velopers-list.1001551.n3.nabble.com/java-lang-IllegalStateEx >>>> ception-There-is-no-space-for-new-record-tc20108.html >>>> >>>> I think we may have a real bug here, but we need a reproduce. Can you >>>> provide one? thanks! >>>> >>>> On Fri, Feb 9, 2018 at 5:59 PM, SNEHASISH DUTTA < >>>> info.snehas...@gmail.com> wrote: >>>> >>>>> Hi , >>>>> >>>>> I am facing the following when running on EMR >>>>> >>>>> Caused by: java.lang.IllegalStateException: There is no space for new >>>>> record >>>>> at org.apache.spark.util.collecti >>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem >>>>> orySorter.java:226) >>>>> at org.apache.spark.sql.execution >>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132) >>>>> at org.apache.spark.sql.execution >>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor >>>>> ter(UnsafeFixedWidthAggregationMap.java:250) >>>>> >>>>> I am using pyspark 2.2 , what spark configuration should be >>>>> changed/modified to get this resolved >>>>> >>>>> >>>>> Regards, >>>>> Snehasish >>>>> >>>>> >>>>> Regards, >>>>> Snehasish >>>>> >>>>> On Fri, Feb 9, 2018 at 1:26 PM, SNEHASISH DUTTA < >>>>> info.snehas...@gmail.com> wrote: >>>>> >>>>>> Hi , >>>>>> >>>>>> I am facing the following when running on EMR >>>>>> >>>>>> Caused by: java.lang.IllegalStateException: There is no space for >>>>>> new record >>>>>> at org.apache.spark.util.collecti >>>>>> on.unsafe.sort.UnsafeInMemorySorter.insertRecord(UnsafeInMem >>>>>> orySorter.java:226) >>>>>> at org.apache.spark.sql.execution >>>>>> .UnsafeKVExternalSorter.<init>(UnsafeKVExternalSorter.java:132) >>>>>> at org.apache.spark.sql.execution >>>>>> .UnsafeFixedWidthAggregationMap.destructAndCreateExternalSor >>>>>> ter(UnsafeFixedWidthAggregationMap.java:250) >>>>>> >>>>>> I am using spark 2.2 , what spark configuration should be >>>>>> changed/modified to get this resolved >>>>>> >>>>>> >>>>>> Regards, >>>>>> Snehasish >>>>>> >>>>> >>>>> >>>> >>> >>
