[ https://issues.apache.org/jira/browse/HADOOP-17657?focusedWorklogId=592601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-592601 ]
ASF GitHub Bot logged work on HADOOP-17657: ------------------------------------------- Author: ASF GitHub Bot Created on: 04/May/21 08:21 Start Date: 04/May/21 08:21 Worklog Time Spent: 10m Work Description: jojochuang merged pull request #2949: URL: https://github.com/apache/hadoop/pull/2949 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 592601) Time Spent: 3h 50m (was: 3h 40m) > SequeneFile.Writer should implement StreamCapabilities > ------------------------------------------------------ > > Key: HADOOP-17657 > URL: https://issues.apache.org/jira/browse/HADOOP-17657 > Project: Hadoop Common > Issue Type: Bug > Reporter: Kishen Das > Assignee: Kishen Das > Priority: Major > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on > S3 from Tez, which internally calls > org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS > DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not > implemented and throws java.lang.UnsupportedOperationException. > bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" > level="ERROR" thread="HistoryEventHandlingThread"] Thread > Thread[HistoryEventHandlingThread, 5,main] threw an > Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not > Syncable^M at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M > at org.apache.hadoop.fs.FS > DataOutputStream.hflush(FSDataOutputStream.java:136)^M at > org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at > org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr > iter.java:64)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M > at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han > dleEvent(ProtoHistoryLoggingService.java:198)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M > at java.lang.Thread.run(Thread.java:748)^M > In order to fix this issue we should implement StreamCapabilities in > SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not > supported. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org