GitHub user guojc opened a pull request:
https://github.com/apache/incubator-spark/pull/612
Fix ExternalMap on case of key's hashCode equal to Int.Maxvalue
Currently StreamBuffer use hash value to compare between buffers. And it
want emmit
buffer with lowest HashValue from PriorityQue first and empty buffer last.
And it use Int.MaxValue
for empty buffer. This can cause problem with non-empty buffer with
HashValue equals to Int.MaxValue
and cause exception. This problem will happen with high probability when
key size approach Int.MaxValue.
This fix change the compareTo method to comparing empty case first then
hashValue and resolve the issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/guojc/incubator-spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-spark/pull/612.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #612
----
commit e5982960184d1c40d56f5ac4dd0fc9b7979c42bf
Author: Jiacheng Guo <[email protected]>
Date: 2014-02-18T10:28:34Z
Currently StreamBuffer use hash value to compare between buffers. And it
want emmit
buffer with lowest HashValue from PriorityQue first and empty buffer last.
And it use Int.MaxValue
for empty buffer. This can cause problem with non-empty buffer with
HashValue equals to Int.MaxValue
and cause exception. This problem will happen with high probability when
key size approach Int.MaxValue.
This fix change the compareTo method to comparing empty case first then
hashValue and resolve the issue.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
[email protected] or file a JIRA ticket with INFRA.
---
