Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/741#discussion_r137913406
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
---
@@ -64,19 +64,34 @@ public SourceHandler(List<RotationAction>
rotationActions
this.rotationPolicy = rotationPolicy;
this.syncPolicy = syncPolicy;
this.fileNameFormat = fileNameFormat;
+ this.cleanupCallback = cleanupCallback;
initialize();
}
protected void handle(JSONObject message, String sensor,
WriterConfiguration config, SyncPolicyCreator syncPolicyCreator) throws
IOException {
byte[] bytes = (message.toJSONString() + "\n").getBytes();
synchronized (this.writeLock) {
- out.write(bytes);
+ try {
+ out.write(bytes);
+ } catch (IOException writeException) {
+ LOG.warn("IOException while writing output", writeException);
+ // If the stream is closed, attempt to rotate the file and try
again, hoping it's transient
+ if (writeException.getMessage().contains("Stream Closed")) {
--- End diff --
Using string compare on the message seems kind of thin. And there may be
other exceptions that come up that we want to handle.
Part of what I hoped would come from the discuss thread was a discussion
about what was recoverable and what was not recoverable, and a better way to
handle both cases.
---
