Thanks for the report! I moved the "Affects Version = 2.10.0" to "Fix Version = 2.10.0" to see if we can get this fixed for the ongoing release. I'll keep further commentary on the bug so it is in one place.
On Fri, Jan 11, 2019 at 8:00 AM Niel Markwick <[email protected]> wrote: > > I have found and narrowed down a regression in 2.9.0 (and 2.10.0/head) > where: > > - If you use DirectRunner (or TestPipeline which uses DirectRunner) > - AND you use FileIO.writeDynamic() > - AND you have a side input to the ContextFul.FN > - AND you do not limit to a single shard > - Then the pipeline will fail. > > > java.lang.IllegalStateException: All PCollectionViews that are consumed > must be written by some WriteView PTransform: Missing [<unnamed> > [RunnerPCollectionView]] > > This is due to the DirectRunner using TransformOverrides re-writing FileIO > sinks to use runner-determined-sharding > ( see DirectRunner.java line 226 > <https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java#L226> > ) > > No idea why this occurs or why it started failing in 2.9.0... > > Raised https://issues.apache.org/jira/browse/BEAM-6407 > > > > <https://cloud.google.com> > * • **Niel Markwick* > * • *Cloud Solutions Architect > * • *Google Belgium > * • *[email protected] > > > > Google Belgium NV/SA, Steenweg op Etterbeek 180, 1040 Brussel, Belgie. RPR: > 0878.065.378 > > If you received this communication by mistake, please don't forward it to > anyone else (it may contain confidential or privileged information), please > erase all copies of it, including all attachments, and please let the > sender know it went to the wrong person. Thanks >
