This closes #3938: [BEAM-2674] Add custom rehydration; reinstate proto roundtrip for Java DirectRunner
DirectRunner: Replace use of RawPTransform with NotSerializable.forUrn translators Clearer getOrDefault style in RehydratedComponents Add NotSerializable.forUrn to key by URN for non-serializable overrides Support side inputs in CombineTranslation Fix typo in UnboundedSource deserialization error message Reinstate proto round trip in Java DirectRunner Better error message for RehydratedComponents.getCoder Add custom rehydration for WriteFiles Add custom rehydration for ParDo Add custom rehydration for Combine Add custom rehydration for TestStream Add TransformPayloadTranslator.rehydrate to optionally specialize RawPTransform Add RawPTransform.migrate(SdkComponents) for re-serialization Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/de7cc05c Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/de7cc05c Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/de7cc05c Branch: refs/heads/master Commit: de7cc05cc67d1aa6331cddc17c2e02ed0efbe37d Parents: 0f7736d 505021e Author: Kenneth Knowles <k...@google.com> Authored: Tue Oct 17 13:48:57 2017 -0700 Committer: Kenneth Knowles <k...@google.com> Committed: Tue Oct 17 13:48:57 2017 -0700 ---------------------------------------------------------------------- .../core/construction/CombineTranslation.java | 192 +++++++++++-- .../CreatePCollectionViewTranslation.java | 7 +- .../core/construction/FlattenTranslator.java | 8 +- .../construction/GroupByKeyTranslation.java | 13 +- .../construction/PTransformTranslation.java | 261 +++++++++++++---- .../core/construction/ParDoTranslation.java | 282 ++++++++++++++----- .../core/construction/PipelineTranslation.java | 110 +------- .../core/construction/ReadTranslation.java | 43 ++- .../core/construction/RehydratedComponents.java | 5 +- .../core/construction/SplittableParDo.java | 8 + .../construction/TestStreamTranslation.java | 165 +++++++++-- .../TransformPayloadTranslatorRegistrar.java | 2 + .../construction/WindowIntoTranslation.java | 15 +- .../construction/WriteFilesTranslation.java | 172 +++++++++-- .../construction/CombineTranslationTest.java | 16 +- .../construction/TestStreamTranslationTest.java | 4 +- .../construction/WriteFilesTranslationTest.java | 3 +- .../core/SplittableParDoViaKeyedWorkItems.java | 7 + .../beam/runners/direct/DirectGroupByKey.java | 16 +- .../beam/runners/direct/DirectOptions.java | 8 - .../beam/runners/direct/DirectRunner.java | 15 +- .../beam/runners/direct/MultiStepCombine.java | 18 +- .../direct/ParDoMultiOverrideFactory.java | 9 +- .../direct/TestStreamEvaluatorFactory.java | 9 +- .../direct/TransformEvaluatorRegistry.java | 32 +-- .../runners/direct/ViewOverrideFactory.java | 8 +- runners/flink/pom.xml | 5 - .../FlinkStreamingTransformTranslators.java | 60 +--- 28 files changed, 1017 insertions(+), 476 deletions(-) ----------------------------------------------------------------------