This closes #3388: Sync gearpump-runner branch with master Remove unused codes Fix kryo exception Fix PCollectionView translation upgrade to gearpump 0.8.4-SNAPSHOT Fix side input handling in DoFnFunction Respect WindowFn#getOutputTime in gearpump-runner Activate Gearpump local-validates-runner-tests in precommit Update against master changes Update gearpump-runner against master changes Update gearpump-runner against master changes Update gearpump-runner against master changes. [BEAM-972] Add more unit test to Gearpump runner [BEAM-972] Add unit tests to Gearpump runner [BEAM-79] Fix gearpump-runner merge conflicts and test failure enable ParDoTest [BEAM-79] Add SideInput support for GearpumpRunner [BEAM-79] Support merging windows in GearpumpRunner [BEAM-79] Fix PostCommit test confs for Gearpump runner note thread is interrupted on InterruptedException Remove cache for Gearpump on travis reduce timeout to wait for result fix ParDo.BoundMulti translation return encoded key for GroupByKey translation support OutputTimeFn update to latest gearpump dsl function interface fix group by window activate ROS on Gearpump by default update ROS configurations [BEAM-1180] Implement GearpumpPipelineResult [BEAM-79] Upgrade to beam-0.5.0-incubating-SNAPSHOT [BEAM-79] Update to latest Gearpump API Fix NoOpAggregatorFactory Remove print to stdout Skip window assignment when windows don't change Add Window.Bound translator Upgrade Gearpump version [BEAM-79] fix gearpump runner build failure [BEAM-79] update GearpumpPipelineResult [BEAM-79] Port Gearpump runner from OldDoFn to new DoFn upgrade gearpump-runner to 0.4.0-incubating-SNAPSHOT remove "pipeline" in runner name post-merge fix [BEAM-79] fix integration-test failure fix import order !fixup Minor javadoc clean-up Added even more javadoc to TextIO#withHeader and TextIO#withFooter (2). Added even more javadoc to TextIO#withHeader and TextIO#withFooter. Added javadoc to TextIO#withHeader and TextIO#withFooter. Reverted header and footer to be of type String. Revised according to comments following a code review. Add header/footer support to TextIO.Write [BEAM-242] Enable and fix checkstyle in Flink runner examples Remove timeout in JAXBCoderTest Be more accepting in UnboundedReadDeduplicatorTest BigQuery: limit max job polling time to 1 minute [BEAM-242] Enable checkstyle and fix checkstyle errors in Flink runner [BEAM-456] Add MongoDbIO FluentBackoff: a replacement for a variety of custom backoff implementations Remove the DataflowRunner instructions from examples Put classes in runners-core package into runners.core namespace Delegate populateDipslayData to wrapped combineFn's Fixed Combine display data Cloud Datastore naming clean-up DatastoreIO SplitQueryFn integration test Add Latest CombineFn and PTransforms Remove empty unused method in TestStreamEvaluatorFactory Test that multiple instances of TestStream are supported Correct some accidental renames Fix condition in FlinkStreamingPipelineTranslator Address comments of Flink Side-Input PR [BEAM-569] Define maxNumRecords default value to Long.MAX_VALUE in JmsIO Add LeaderBoardTest take advantage of setup/teardown for KafkaWriter Returned KafkaIO getWatermark log line in debug mode [BEAM-572] Remove Spark Reference in WordCount Update Dataflow Container Version [BEAM-313] Provide a context for SparkRunner DataflowRunner: get PBegin from PInput [BEAM-592] Fix SparkRunner Dependency Problem in WordCount Fix javadoc in Kinesis Organize imports in Kinesis kinesis: a connector for Amazon Kinesis [BEAM-589] Fixing IO.Read transformation Query latest timestamp travis.yml: disable updating snapshots Added support for reporting aggregator values to Spark sinks [BEAM-294] Rename dataflow references to beam Modified BigtableIO to use DoFn setup/tearDown methods instead of startBundle/finishBundle checkstyle: prohibit API client repackaged Guava Make WriteTest more resilient to Randomness Update DoFn javadocs to remove references to OldDoFn and Dataflow [BEAM-545] Promote JobName to PipelineOptions Move the samples data to gs://apache-beam-samples/ Cleanup some javadoc that referring Dataflow BigQueryIO.Write: raise size limit to 11 TiB Optimize imports Update checkstyle.xml to put all imports in one group Fix Exception Unwrapping in TestFlinkRunner Make ParDoLifecycleTest Serializable to Fix Test with TupleTag Use AllPanes as the PaneExtractor in IterableAssert ...
Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/99f4f8b1 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/99f4f8b1 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/99f4f8b1 Branch: refs/heads/master Commit: 99f4f8b1b8a4da2ebfec8371a1c1d9a978f001a6 Parents: 8860cce b21fa04 Author: Kenneth Knowles <k...@google.com> Authored: Thu Jun 22 20:03:57 2017 -0700 Committer: Kenneth Knowles <k...@google.com> Committed: Thu Jun 22 20:03:57 2017 -0700 ---------------------------------------------------------------------- ...tCommit_Java_ValidatesRunner_Gearpump.groovy | 2 +- examples/java/pom.xml | 12 + pom.xml | 6 + runners/gearpump/README.md | 61 ++++ runners/gearpump/pom.xml | 281 +++++++++++++++++++ .../gearpump/GearpumpPipelineOptions.java | 67 +++++ .../gearpump/GearpumpPipelineResult.java | 109 +++++++ .../beam/runners/gearpump/GearpumpRunner.java | 119 ++++++++ .../gearpump/GearpumpRunnerRegistrar.java | 64 +++++ .../runners/gearpump/TestGearpumpRunner.java | 63 +++++ .../beam/runners/gearpump/package-info.java | 22 ++ ...CreateGearpumpPCollectionViewTranslator.java | 45 +++ .../CreateStreamingGearpumpView.java | 157 +++++++++++ .../FlattenPCollectionsTranslator.java | 83 ++++++ .../translators/GearpumpPipelineTranslator.java | 143 ++++++++++ .../translators/GroupByKeyTranslator.java | 258 +++++++++++++++++ .../translators/ParDoMultiOutputTranslator.java | 103 +++++++ .../translators/ReadBoundedTranslator.java | 46 +++ .../translators/ReadUnboundedTranslator.java | 48 ++++ .../translators/TransformTranslator.java | 30 ++ .../translators/TranslationContext.java | 105 +++++++ .../translators/WindowAssignTranslator.java | 103 +++++++ .../translators/functions/DoFnFunction.java | 193 +++++++++++++ .../translators/functions/package-info.java | 22 ++ .../translators/io/BoundedSourceWrapper.java | 45 +++ .../gearpump/translators/io/GearpumpSource.java | 121 ++++++++ .../translators/io/UnboundedSourceWrapper.java | 46 +++ .../gearpump/translators/io/ValuesSource.java | 173 ++++++++++++ .../gearpump/translators/io/package-info.java | 22 ++ .../gearpump/translators/package-info.java | 22 ++ .../translators/utils/DoFnRunnerFactory.java | 81 ++++++ .../translators/utils/NoOpStepContext.java | 41 +++ .../translators/utils/TranslatorUtils.java | 199 +++++++++++++ .../translators/utils/package-info.java | 22 ++ .../gearpump/GearpumpRunnerRegistrarTest.java | 55 ++++ .../runners/gearpump/PipelineOptionsTest.java | 73 +++++ ...teGearpumpPCollectionViewTranslatorTest.java | 56 ++++ .../FlattenPCollectionsTranslatorTest.java | 149 ++++++++++ .../translators/GroupByKeyTranslatorTest.java | 152 ++++++++++ .../translators/ReadBoundedTranslatorTest.java | 70 +++++ .../ReadUnboundedTranslatorTest.java | 70 +++++ .../translators/WindowAssignTranslatorTest.java | 110 ++++++++ .../translators/io/GearpumpSourceTest.java | 99 +++++++ .../gearpump/translators/io/ValueSoureTest.java | 79 ++++++ .../translators/utils/TranslatorUtilsTest.java | 73 +++++ runners/pom.xml | 9 + sdks/java/pom.xml | 2 +- 47 files changed, 3909 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/beam/blob/99f4f8b1/pom.xml ----------------------------------------------------------------------
