[ https://issues.apache.org/jira/browse/BEAM-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111671#comment-16111671 ]
Eugene Kirpichov commented on BEAM-2716: ---------------------------------------- I think this affects only Avro, other file-based formats either don't support dynamic splits, or don't know block lengths upfront. > AvroReader should refuse dynamic splits while in the last block > --------------------------------------------------------------- > > Key: BEAM-2716 > URL: https://issues.apache.org/jira/browse/BEAM-2716 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Eugene Kirpichov > Assignee: Eugene Kirpichov > Priority: Minor > > AvroReader is able to detect when it's in the last block: > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java#L728 > It could also use this information to avoid wastefully producing dynamic > splits starting in the range of the current block. > One way to do this would be to have OffsetRangeTracker have a "claim range" > operation: claim range of [a, b) is, in terms of correctness, equivalent to > claiming "a" (it checks whether "a" is within the range), but sets the last > claimed position to "b" rather than "a", thus protecting more positions from > being split away. -- This message was sent by Atlassian JIRA (v6.4.14#64029)