[
https://issues.apache.org/jira/browse/FLINK-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262298#comment-15262298
]
ASF GitHub Bot commented on FLINK-3717:
---------------------------------------
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1895#discussion_r61444801
--- Diff:
flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java
---
@@ -284,18 +322,24 @@ public T nextRecord(T record) throws IOException {
protected abstract T deserialize(T reuse, DataInputView dataInput)
throws IOException;
/**
- * Writes a block info at the end of the blocks.<br>
- * Current implementation uses only int and not long.
+ * Reads the content of a block of data. The block contains its {@link
BlockInfo}
+ * at the end, and this method takes this into account when reading the
data.
*/
- protected class BlockBasedInput extends FilterInputStream {
+ private class BlockBasedInput extends FilterInputStream {
private final int maxPayloadSize;
private int blockPos;
- public BlockBasedInput(FSDataInputStream in, int blockSize) {
+ BlockBasedInput(FSDataInputStream in, long length) {
super(in);
this.blockPos = (int)
BinaryInputFormat.this.blockInfo.getFirstRecordStart();
- this.maxPayloadSize = blockSize -
BinaryInputFormat.this.blockInfo.getInfoSize();
+ this.maxPayloadSize = (int) (length -
BinaryInputFormat.this.blockInfo.getInfoSize());
+ }
+
+ BlockBasedInput(FSDataInputStream in, int startPos, long
length) {
--- End diff --
I think we can get away with only having one constructor and calling `new
BlockBasedInput(stream, blockInfo.getFirstRecordStart(), length)` in `open()`
when not restoring from a snapshot.
> Add functionality to be a able to restore from specific point in a
> FileInputFormat
> ----------------------------------------------------------------------------------
>
> Key: FLINK-3717
> URL: https://issues.apache.org/jira/browse/FLINK-3717
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Kostas Kloudas
> Assignee: Kostas Kloudas
>
> This is the first step in order to make the File Sources fault-tolerant. We
> have to be able to get the start from a specific point in a file despite any
> caching performed during reading. This will guarantee that the task that will
> take over the execution of the failed one will be able to start from the
> correct point in the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
