[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Status: Open (was: Patch Available)
The issue was addressed in HIVE-15580. Cancel the patch here and close this for
now.
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.8.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: (was: HIVE-15527.7.patch)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.7.patch
Re-trigger tests.
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: (was: HIVE-15527.7.patch)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.7.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.7.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch,
> HIVE-15527.7.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.0.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.0.patch,
> HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.3.patch,
> HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.6.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.0.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.1.patch,
> HIVE-15527.2.patch, HIVE-15527.3.patch, HIVE-15527.4.patch,
> HIVE-15527.5.patch, HIVE-15527.6.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: (was: HIVE-15527.0.patch)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.1.patch,
> HIVE-15527.2.patch, HIVE-15527.3.patch, HIVE-15527.4.patch,
> HIVE-15527.5.patch, HIVE-15527.6.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.0.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.0.patch, HIVE-15527.1.patch,
> HIVE-15527.2.patch, HIVE-15527.3.patch, HIVE-15527.4.patch,
> HIVE-15527.5.patch, HIVE-15527.6.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.6.patch
Reuse cache between keys.
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.5.patch,
> HIVE-15527.6.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: (was: HIVE-15527.5.patch)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.5.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.5.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.5.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.4.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: (was: HIVE-15527.4.patch)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-15527:
Attachment: HIVE-15527.4.patch
Attaching a WIP patch for testing. Still need to add unit tests & more.
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Chao Sun
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.4.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.3.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch,
> HIVE-15527.3.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.2.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.1.patch, HIVE-15527.2.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.1.patch
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.1.patch, HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the same input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Description:
In SortByShuffler.java, an ArrayList is used to back the iterator for values
that have the same key in shuffled result produced by spark transformation
sortByKey. It's possible that memory can be exhausted because of a large key
group.
{code}
@Override
public Tuple2 next() {
// TODO: implement this by accumulating rows with the same key
into a list.
// Note that this list needs to improved to prevent excessive
memory usage, but this
// can be done in later phase.
while (it.hasNext()) {
Tuple2 pair = it.next();
if (curKey != null && !curKey.equals(pair._1())) {
HiveKey key = curKey;
List values = curValues;
curKey = pair._1();
curValues = new ArrayList();
curValues.add(pair._2());
return new Tuple2(key,
values);
}
curKey = pair._1();
curValues.add(pair._2());
}
if (curKey == null) {
throw new NoSuchElementException();
}
// if we get here, this should be the last element we have
HiveKey key = curKey;
curKey = null;
return new Tuple2(key,
curValues);
}
{code}
Since the output from sortByKey is already sorted on key, it's possible to
backup the value iterable using the same input iterator.
was:
In SortByShuffler.java, an ArrayList is used to back the iterator for values
that have the same key in shuffled result produced by spark transformation
sortByKey. It's possible that memory can be exhausted because of a large key
group.
{code}
@Override
public Tuple2 next() {
// TODO: implement this by accumulating rows with the same key
into a list.
// Note that this list needs to improved to prevent excessive
memory usage, but this
// can be done in later phase.
while (it.hasNext()) {
Tuple2 pair = it.next();
if (curKey != null && !curKey.equals(pair._1())) {
HiveKey key = curKey;
List values = curValues;
curKey = pair._1();
curValues = new ArrayList();
curValues.add(pair._2());
return new Tuple2(key,
values);
}
curKey = pair._1();
curValues.add(pair._2());
}
if (curKey == null) {
throw new NoSuchElementException();
}
// if we get here, this should be the last element we have
HiveKey key = curKey;
curKey = null;
return new Tuple2(key,
curValues);
}
{code}
Since the output from sortByKey is already sorted on key, it's possible to
backup the value iterable using the input iterator.
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
>
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Status: Patch Available (was: Open)
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15527) Memory usage is unbound in SortByShuffler for Spark
[
https://issues.apache.org/jira/browse/HIVE-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-15527:
---
Attachment: HIVE-15527.patch
CC: [~lirui], [~csun]
> Memory usage is unbound in SortByShuffler for Spark
> ---
>
> Key: HIVE-15527
> URL: https://issues.apache.org/jira/browse/HIVE-15527
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Affects Versions: 1.1.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-15527.patch
>
>
> In SortByShuffler.java, an ArrayList is used to back the iterator for values
> that have the same key in shuffled result produced by spark transformation
> sortByKey. It's possible that memory can be exhausted because of a large key
> group.
> {code}
> @Override
> public Tuple2 next() {
> // TODO: implement this by accumulating rows with the same key
> into a list.
> // Note that this list needs to improved to prevent excessive
> memory usage, but this
> // can be done in later phase.
> while (it.hasNext()) {
> Tuple2 pair = it.next();
> if (curKey != null && !curKey.equals(pair._1())) {
> HiveKey key = curKey;
> List values = curValues;
> curKey = pair._1();
> curValues = new ArrayList();
> curValues.add(pair._2());
> return new Tuple2(key,
> values);
> }
> curKey = pair._1();
> curValues.add(pair._2());
> }
> if (curKey == null) {
> throw new NoSuchElementException();
> }
> // if we get here, this should be the last element we have
> HiveKey key = curKey;
> curKey = null;
> return new Tuple2(key,
> curValues);
> }
> {code}
> Since the output from sortByKey is already sorted on key, it's possible to
> backup the value iterable using the input iterator.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
