This is an automated email from the ASF dual-hosted git repository.
altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b5e4175 [BEAM-7877] Change the log level when deleting unknown
temoprary files in FileBasedSink
new dbcd265 Merge pull request #9227 from ihji/BEAM-7877
b5e4175 is described below
commit b5e4175107d33f1475c18bee2a5d4f010a509bf7
Author: Heejong Lee <[email protected]>
AuthorDate: Thu Aug 1 16:32:45 2019 -0700
[BEAM-7877] Change the log level when deleting unknown temoprary files in
FileBasedSink
Currently the log level is info. A new proposed log level is warning
since deleting unknown temporary files is a bad sign and sometimes leads
to data loss.
---
.../core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 2fb6aa2..21a2cf4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -791,7 +791,10 @@ public abstract class FileBasedSink<UserT, DestinationT,
OutputT>
FileSystems.match(Collections.singletonList(tempDir.toString() + "*")));
for (Metadata matchResult : singleMatch.metadata()) {
if (allMatches.add(matchResult.resourceId())) {
- LOG.info("Will also remove unknown temporary file {}",
matchResult.resourceId());
+ LOG.warn(
+ "Will also remove unknown temporary file {}. This might
indicate that other process/job is using "
+ + "the same temporary folder and result in data
consistency issues.",
+ matchResult.resourceId());
}
}
} catch (Exception e) {
