ulysses you created SPARK-29833:
-----------------------------------
Summary: Add FileNotFoundException for spark.yarn.jars
Key: SPARK-29833
URL: https://issues.apache.org/jira/browse/SPARK-29833
Project: Spark
Issue Type: Improvement
Components: YARN
Affects Versions: 2.4.4
Reporter: ulysses you
When set `spark.yarn.jars=/xxx/xxx` which is just a no schema path, spark will
throw a NullPointerException.
The reason is hdfs will return null if pathFs.globStatus(path) is not exist,
and spark just use `pathFs.globStatus(path).filter(_.isFile())` without check
it.
Related Globber code is here
{noformat}
/*
* When the input pattern "looks" like just a simple filename, and we
* can't find it, we return null rather than an empty array.
* This is a special case which the shell relies on.
*
* To be more precise: if there were no results, AND there were no
* groupings (aka brackets), and no wildcards in the input (aka stars),
* we return null.
*/
if ((!sawWildcard) && results.isEmpty() &&
(flattenedPatterns.size() <= 1)) {
return null;
}
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
