Kyoungha Min created BEAM-9743: ---------------------------------- Summary: TFRecordCodec not attempt to fully read header/footer Key: BEAM-9743 URL: https://issues.apache.org/jira/browse/BEAM-9743 Project: Beam Issue Type: Bug Components: sdk-java-core Reporter: Kyoungha Min Assignee: Kyoungha Min
Seems like it only happens with Zstd compression (or any other picky input stream that refuse to read fully). Zstd seems very picky at giving out data. The parts with the issue are [https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L672] [https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L699] And not so problem within the beam application, but still not following the WritableByteChannel API, [https://github.com/apache/beam/blob/c7911043510a266078a3dc8faef7a1dbe1f598c5/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L720-L727] ReadableByteChannel/WritableByteChannel Javadoc specifies that they are not required to read/write fully, and can refuse to read/write time to time. -- This message was sent by Atlassian Jira (v8.3.4#803005)