[ https://issues.apache.org/jira/browse/HADOOP-17657?focusedWorklogId=590931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-590931 ]
ASF GitHub Bot logged work on HADOOP-17657: ------------------------------------------- Author: ASF GitHub Bot Created on: 29/Apr/21 10:58 Start Date: 29/Apr/21 10:58 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2949: URL: https://github.com/apache/hadoop/pull/2949#issuecomment-829134626 I do like the new test, now there's just the little detail that it's not quite working yet ``` [INFO] Running org.apache.hadoop.io.TestSequenceFile [ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.818 s <<< FAILURE! - in org.apache.hadoop.io.TestSequenceFile [ERROR] testSequenceFileWriter(org.apache.hadoop.io.TestSequenceFile) Time elapsed: 0.614 s <<< ERROR! java.io.IOException: wrong key class: org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.NullWritable at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.append(SequenceFile.java:1508) at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1425) at org.apache.hadoop.io.TestSequenceFile.testSequenceFileWriter(TestSequenceFile.java:745) ``` + minor checkstyles ``` ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java:737: Path p = new Path(GenericTestUtils.getTempPath("testSequenceFileWriter.seq"));: Line is longer than 80 characters (found 82). [LineLength] ./hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSequenceFile.java:745: writer.append(key,value);:24: ',' is not followed by whitespace. [WhitespaceAfter] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 590931) Time Spent: 3h (was: 2h 50m) > SequeneFile.Writer should implement StreamCapabilities > ------------------------------------------------------ > > Key: HADOOP-17657 > URL: https://issues.apache.org/jira/browse/HADOOP-17657 > Project: Hadoop Common > Issue Type: Bug > Reporter: Kishen Das > Assignee: Kishen Das > Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on > S3 from Tez, which internally calls > org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS > DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not > implemented and throws java.lang.UnsupportedOperationException. > bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" > level="ERROR" thread="HistoryEventHandlingThread"] Thread > Thread[HistoryEventHandlingThread, 5,main] threw an > Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not > Syncable^M at > org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M > at org.apache.hadoop.fs.FS > DataOutputStream.hflush(FSDataOutputStream.java:136)^M at > org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at > org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr > iter.java:64)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M > at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han > dleEvent(ProtoHistoryLoggingService.java:198)^M at > org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M > at java.lang.Thread.run(Thread.java:748)^M > In order to fix this issue we should implement StreamCapabilities in > SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not > supported. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org