This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 2fc9c0b [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker 2fc9c0b is described below commit 2fc9c0bfb5d43a6ee1dbbf941b84e4c3dd74d8ef Author: tooptoop4 <33283496+toopto...@users.noreply.github.com> AuthorDate: Thu Aug 19 10:31:10 2021 +0900 [SPARK-36147][SQL] Warn if less files visible after stats write in BasicWriteStatsTracker ### What changes were proposed in this pull request? This log should at least be WARN not INFO (in org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala ) "Expected $numSubmittedFiles files, but only saw $numFiles." ### Why are the changes needed? INFO logs don't indicate possible issue but WARN logs should ### Does this PR introduce any user-facing change? Yes, Log level changed. ### How was this patch tested? manual, trivial change Closes #33332 from tooptoop4/warn. Authored-by: tooptoop4 <33283496+toopto...@users.noreply.github.com> Signed-off-by: Hyukjin Kwon <gurwls...@apache.org> --- .../apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala index f3815ab..a8fae66 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala @@ -169,7 +169,7 @@ class BasicWriteTaskStatsTracker( } if (numSubmittedFiles != numFiles) { - logInfo(s"Expected $numSubmittedFiles files, but only saw $numFiles. " + + logWarning(s"Expected $numSubmittedFiles files, but only saw $numFiles. " + "This could be due to the output format not writing empty files, " + "or files being not immediately visible in the filesystem.") } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org