[GitHub] spark pull request #22969: [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMe...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/22969#discussion_r232112326
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
---
@@ -787,7 +789,7 @@ case class HashAggregateExec(
|$unsafeRowKeys, ${hashEval.value});
| if ($unsafeRowBuffer == null) {
|// failed to allocate the first page
- |throw new OutOfMemoryError("No enough memory for
aggregation");
--- End diff --
opened a JIRA for banning this by a new lint rule:
https://issues.apache.org/jira/browse/SPARK-25986
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22969: [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMe...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22969 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22969: [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMe...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22969#discussion_r231783323
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
---
@@ -787,7 +789,7 @@ case class HashAggregateExec(
|$unsafeRowKeys, ${hashEval.value});
| if ($unsafeRowBuffer == null) {
|// failed to allocate the first page
- |throw new OutOfMemoryError("No enough memory for
aggregation");
+ |throw new $oomeClassName("No enough memory for aggregation");
--- End diff --
Yes, I think so based on my investigation. I grep-ed with
"OutOfMemoryError" and checked the suspicious places.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22969: [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMe...
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/22969#discussion_r231779387
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
---
@@ -787,7 +789,7 @@ case class HashAggregateExec(
|$unsafeRowKeys, ${hashEval.value});
| if ($unsafeRowBuffer == null) {
|// failed to allocate the first page
- |throw new OutOfMemoryError("No enough memory for
aggregation");
+ |throw new $oomeClassName("No enough memory for aggregation");
--- End diff --
Hi, @ueshin . Is this the final place? If not, can we have a separate JIRA
issue for this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22969: [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMe...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/22969 [SPARK-22827][SQL][FOLLOW-UP] Throw `SparkOutOfMemoryError` in `HashAggregateExec`, too. ## What changes were proposed in this pull request? This is a follow-up pr of #20014 which introduced `SparkOutOfMemoryError` to avoid killing the entire executor when an `OutOfMemoryError` is thrown. We should throw `SparkOutOfMemoryError` in `HashAggregateExec`, too. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-22827/oome Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22969.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22969 commit f07ab0938563fe63dd20fa756543b14478a27c2f Author: Takuya UESHIN Date: 2018-11-08T04:59:35Z Throw `SparkOutOfMemoryError` in `HashAggregateExec`, too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
