[
https://issues.apache.org/jira/browse/SPARK-26877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-26877.
------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23786
[https://github.com/apache/spark/pull/23786]
> Support user-level app staging directory in yarn mode when
> spark.yarn.stagingDir specified
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-26877
> URL: https://issues.apache.org/jira/browse/SPARK-26877
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: liupengcheng
> Assignee: liupengcheng
> Priority: Minor
> Fix For: 3.0.0
>
>
> Currently, when running applications on yarn mode, the app staging directory
> of is controlled by `spark.yarn.stagingDir` config if specified, and this
> directory cannot separate different users, somtimes, it's inconvenient for
> file and quota management for users.
> For example, user may have different quota for their own app staging dir. and
> they may also not want to use the home directory as the staging dir, because
> it might used mainly for private user files or data. The quota may also be
> different between private data and the app staging dir.
> So I propose to add user sub directories under this app staging dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
