Github user mridulm commented on the issue: https://github.com/apache/spark/pull/19487 > If it does use it, it'll handle an invalid entry in setupJob/setupTask by throwing an exception there. This should currently happen and `hasValidPath` does not prevent it. That is, if committer is unable to handle specified output directory, it can throw exception in `committer.setupJob`; based on what is specified in the config passed in `TaskAttemptContext`. Note that `hasValidPath` and `path` handle the explicit case of absolute path based committer's where `HadoopMapReduceCommitProtocol` moves the result to the final destination (and removes them in case of failure) : see use of `commitJob#taskCommits`. `commitJob` does invoke `committer.commitJob` - so committer specific commit will happen. This is not relevant for non-path based committer's. What I would like clarification on is, what is to be done when `path` is invalid. My understanding was, this is up to the committer implementation to handle - since it could be a valid use : and if invalid, it would throw an exception in `setupJob` or `commitJob`. If this is incorrect assumption, then I will change it back to explicitly support `null` or `""` for `path` - instead of also unparseable path's
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org