Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/18979#discussion_r133918269
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
---
@@ -57,7 +60,14 @@ class BasicWriteTaskStatsTracker(hadoopConf:
Configuration)
private def getFileSize(filePath: String): Long = {
val path = new Path(filePath)
val fs = path.getFileSystem(hadoopConf)
- fs.getFileStatus(path).getLen()
+ try {
+ fs.getFileStatus(path).getLen()
+ } catch {
+ case e: FileNotFoundException =>
+ // may arise against eventually consistent object stores
+ logInfo(s"File $path is not yet visible", e)
+ 0
--- End diff --
The problem is: what can be picked up if the file isn't yet reported as
present by the endpoint? Adding a bool to say "results are unreliable" could be
used as a warning.
One thing to consider long term is: if hadoop FS output streams added a
simple <String, Long> map of statistics, could they be picked up by committers
& then aggregated in job reports. Hadoop filesystems have statistics (simple
ones in Hadoop <= 2.7, an arbitrary map of String -> Long in 2.8 (with
[standard key names across
filesystems](https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageStatistics.java)),
and certainly today S3ABlockOutputStream [collects stats on individual
streams](https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java#L756).
If that was made visible and collected, you could get a lot more detail on
what is going on. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
