Lantao Jin created SPARK-31591:
----------------------------------
Summary: namePrefix could be null in Utils.createDirectory
Key: SPARK-31591
URL: https://issues.apache.org/jira/browse/SPARK-31591
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Lantao Jin
In our production, we find that many shuffle files could be located in
/hadoop/2/yarn/local/usercache/b_carmel/appcache/application_1586487864336_4602/*null*-107d4e9c-d3c7-419e-9743-a21dc4eaeb3f/3a
The Util.createDirectory() uses a default parameter "spark"
{code}
def createDirectory(root: String, namePrefix: String = "spark"): File = {
{code}
But in some cases, the actual namePrefix is null. If the method is called with
null, then the default value would not be applied.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
