[
https://issues.apache.org/jira/browse/SPARK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948399#comment-16948399
]
angerszhu commented on SPARK-29424:
-----------------------------------
[~srowen]
Since resource limit is established, these bad behavior will cause program run
very slow but don't know why.
Make it abort early is better for user to recognize where the problem is.
Especially for Spark Thrift Server.
> Prevent Spark to committing stage of too much Task
> --------------------------------------------------
>
> Key: SPARK-29424
> URL: https://issues.apache.org/jira/browse/SPARK-29424
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0, 3.0.0
> Reporter: angerszhu
> Priority: Major
>
> Our user always submit bad SQL in query platform, Such as :
> # write wrong join condition but submit that sql
> # write wrong where condition
> # etc..
> This case will make Spark scheduler to submit a lot of task. It will cause
> spark run very slow and impact other user(spark thrift server) even run out
> of memory because of too many object generated by a big num of tasks.
> So I add a constraint when submit tasks and abort stage early when TaskSet
> size num is bigger then set limit . I wonder if the community will accept
> this way.
> cc [~srowen] [~dongjoon] [~yumwang]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
