liupengcheng created SPARK-26877:
------------------------------------
Summary: Support user staging directory in yarn mode
Key: SPARK-26877
URL: https://issues.apache.org/jira/browse/SPARK-26877
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.4.0
Reporter: liupengcheng
Currently, when running applications on yarn mode, the app staging directory ofÂ
is controlled by `spark.yarn.stagingDir` config if specified, and this
directory cannot separate different users, somtimes, it's inconvenient for file
and quota management for users.
For example, user may have different quota for their own app staging dir. and
they may also not want to use the home directory as the staging dir, because it
might used mainly for private user files or data. The quota may also be
different between private data and the app staging dir.
So I propose to add user sub directories under this app staging dir.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
