[ 
https://issues.apache.org/jira/browse/SPARK-29833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ulysses you updated SPARK-29833:
--------------------------------
    Summary: Add FileNotFoundException check  for spark.yarn.jars  (was: Add 
FileNotFoundException for spark.yarn.jars)

> Add FileNotFoundException check  for spark.yarn.jars
> ----------------------------------------------------
>
>                 Key: SPARK-29833
>                 URL: https://issues.apache.org/jira/browse/SPARK-29833
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 2.4.4
>            Reporter: ulysses you
>            Priority: Minor
>
> When set `spark.yarn.jars=/xxx/xxx` which is just a no schema path, spark 
> will throw a NullPointerException.
> The reason is hdfs will return null if pathFs.globStatus(path) is not exist, 
> and spark just use `pathFs.globStatus(path).filter(_.isFile())` without check 
> it.
> Related Globber code is here
> {noformat}
>     /*
>      * When the input pattern "looks" like just a simple filename, and we
>      * can't find it, we return null rather than an empty array.
>      * This is a special case which the shell relies on.
>      *
>      * To be more precise: if there were no results, AND there were no
>      * groupings (aka brackets), and no wildcards in the input (aka stars),
>      * we return null.
>      */
>     if ((!sawWildcard) && results.isEmpty() &&
>         (flattenedPatterns.size() <= 1)) {
>       return null;
>     }
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to