[
https://issues.apache.org/jira/browse/SPARK-9611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yin Huai updated SPARK-9611:
----------------------------
Description:
There are two corner cases related to the destructAndCreateExternalSorter
(class UnsafeKVExternalSorter) returned by UnsafeFixedWidthAggregationMap.
1. The constructor of UnsafeKVExternalSorter tries to first create a
UnsafeInMemorySorter based on the BytesToBytesMap of
UnsafeFixedWidthAggregationMap. However, when there is no entry in the map,
UnsafeInMemorySorter will throw an AssertionError because we are using the size
of map (0 at here) as the initialSize of UnsafeInMemorySorter, which is not
allowed.
2. Once we fixes the first problem, when we use UnsafeKVExternalSorter's
KVSorterIterator loads data back, you can find there is one extra records,
which is an empty record.
was:
There are a few corner cases related to the destructAndCreateExternalSorter
(class UnsafeKVExternalSorter) returned by UnsafeFixedWidthAggregationMap.
1. The constructor of UnsafeKVExternalSorter tries to first create a
UnsafeInMemorySorter based on the BytesToBytesMap of
UnsafeFixedWidthAggregationMap. However, when there is no entry in the map,
UnsafeInMemorySorter will throw an AssertionError because we are using the size
of map (0 at here) as the initialSize of UnsafeInMemorySorter, which is not
allowed.
2. Once we fixes the first problem, when we use UnsafeKVExternalSorter's
KVSorterIterator loads data back, you can find there is one extra records,
which is an empty record.
3. The next method of the KVSorterIterator in UnsafeKVExternalSorter does not
handle empty records well. If the length of a record is 0, the calculated
length of value will be -4, which causes a NegativeArraySizeException.
> UnsafeFixedWidthAggregationMap.destructAndCreateExternalSorter will add an
> empty entry to if the map is empty.
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-9611
> URL: https://issues.apache.org/jira/browse/SPARK-9611
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Yin Huai
> Assignee: Yin Huai
> Priority: Blocker
> Fix For: 1.5.0
>
>
> There are two corner cases related to the destructAndCreateExternalSorter
> (class UnsafeKVExternalSorter) returned by UnsafeFixedWidthAggregationMap.
> 1. The constructor of UnsafeKVExternalSorter tries to first create a
> UnsafeInMemorySorter based on the BytesToBytesMap of
> UnsafeFixedWidthAggregationMap. However, when there is no entry in the map,
> UnsafeInMemorySorter will throw an AssertionError because we are using the
> size of map (0 at here) as the initialSize of UnsafeInMemorySorter, which is
> not allowed.
> 2. Once we fixes the first problem, when we use UnsafeKVExternalSorter's
> KVSorterIterator loads data back, you can find there is one extra records,
> which is an empty record.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
