[jira] [Updated] (BEAM-79) Gearpump runner

[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Akidau updated BEAM-79: - Description: Intel is submitting Gearpump (http://www.gearpump.io) to ASF

[jira] [Updated] (BEAM-79) Gearpump runner

[ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Akidau updated BEAM-79: - Component/s: runner-ideas > Gearpump runner > --- > > Key: BEAM-79 >

[jira] [Created] (BEAM-79) Gearpump runner

Tyler Akidau created BEAM-79: Summary: Gearpump runner Key: BEAM-79 URL: https://issues.apache.org/jira/browse/BEAM-79 Project: Beam Issue Type: New Feature Reporter: Tyler Akidau

[jira] [Created] (BEAM-77) Reorganize Directory structure

Frances Perry created BEAM-77: - Summary: Reorganize Directory structure Key: BEAM-77 URL: https://issues.apache.org/jira/browse/BEAM-77 Project: Beam Issue Type: Task Components:

Build failed in Jenkins: beam » Google Cloud Dataflow Java SDK - Parent #3

See Changes: [davorbonaci] Apply Maven shading to the Guava dependency library. [davorbonaci] Update test_wordcount.sh following replacement of bundling with shading [alex] Ignore

[26/50] [abbrv] incubator-beam git commit: ProtoCoder: a Coder for Protocol Buffers Messages

ProtoCoder: a Coder for Protocol Buffers Messages Release Notes Adds ProtoCoder, which is a Coder for Protocol Buffers messages. ProtoCoder has the following advantages over Proto2Coder: * ProtoCoder is designed to support Protocol Buffers syntax versions 2 and 3, not just proto2. *

[40/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

Finish Flattenning InProcess package Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115601346 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[38/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManagerTest.java -- diff --git

[24/50] [abbrv] incubator-beam git commit: Proto2Coder: recompute the extension registry when mutated

Proto2Coder: recompute the extension registry when mutated Proto2Coder's deprecated methods mutate the existing coder's extension host set, but it may already have been memoized. Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115407399

[37/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManagerTest.java -- diff --git

[30/50] [abbrv] incubator-beam git commit: Use a static variable for CoderCalled in WriteTest

Use a static variable for CoderCalled in WriteTest The test is broken when Create serializes the created values at apply-time rather than at execution-time, as the modification happens after the Create is applied within the Write PTransform. Moving to a static variable allows the state of the

[33/50] [abbrv] incubator-beam git commit: Handle multiple requests in InProcess Read Primitives

Handle multiple requests in InProcess Read Primitives Each source that is invoked by a read should produce its elements in serial. Using a queue of available sources enforces only one worker having access to a source at a time. Add EmptyTransformEvaluator, to be returned in the case that there

[GitHub] incubator-beam pull request: [BEAM-7] Initial Dataflow code drop

Github user asfgit closed the pull request at: https://github.com/apache/incubator-beam/pull/1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[14/50] [abbrv] incubator-beam git commit: Migrate TextIO.Write to a custom sink

Migrate TextIO.Write to a custom sink Note for user requested sharding limits to be supported, each pipeline runner must support applying those sharding limits. Google Cloud Dataflow supports sharding limits. Release Notes [] - Created by MOE: https://github.com/google/moe

[50/50] [abbrv] incubator-beam git commit: This closes #1

This closes #1 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/211e76ab Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/211e76ab Diff:

[09/50] [abbrv] incubator-beam git commit: Revert "Migrate AvroIO.Write to a custom sink"

Revert "Migrate AvroIO.Write to a custom sink" Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115355272 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[15/50] [abbrv] incubator-beam git commit: Make TestPipeline slightly less DataflowPipelineRunner-centric

Make TestPipeline slightly less DataflowPipelineRunner-centric Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115302769 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[03/50] [abbrv] incubator-beam git commit: Migrate AvroIO.Write to a custom sink

Migrate AvroIO.Write to a custom sink Note for user requested sharding limits to be supported, each pipeline runner must support applying those sharding limits. Google Cloud Dataflow supports sharding limits. Release Notes [] - Created by MOE: https://github.com/google/moe

[29/50] [abbrv] incubator-beam git commit: Set worker harness container image to INVALID until next release

Set worker harness container image to INVALID until next release Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115473121 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[41/50] [abbrv] incubator-beam git commit: Rollback reverts "Move Google Cloud Dataflow native sinks to worker module"

Rollback reverts "Move Google Cloud Dataflow native sinks to worker module" This is for Apache Beam. Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115507028 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[43/50] [abbrv] incubator-beam git commit: Fix worker dependencies and turn on strict checking

Fix worker dependencies and turn on strict checking Here is a pruned and edited tree diff of the dependency:tree output. It is formatted to emphasize the packages that were SDK deps that are now direct deps. Many of these are actually provided by lots of transitive dependencies. [INFO]

[27/50] [abbrv] incubator-beam git commit: Add used-but-undeclared findbugs JSR305 dependencies

Add used-but-undeclared findbugs JSR305 dependencies This artifact provides annotations such as @Nullable, @ThreadSafe, and @NotThreadSafe, which we use throughout the SDK. Relevant pieces of diff from `mvn dependency:tree -Dverbose`: [INFO]

[05/50] [abbrv] incubator-beam git commit: Expose base output file name on FileBasedSink

Expose base output file name on FileBasedSink Release Notes Add the ability to get the base output filename to FileBasedSinks. [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115265994 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo

[31/50] [abbrv] incubator-beam git commit: Honor user requested shard limits for AvroIO.Write on DirectPipelineRunner

Honor user requested shard limits for AvroIO.Write on DirectPipelineRunner During the migration to custom sink within AvroIO, shard controls were removed for DirectPipelineRunner. This change adds them back. Release Notes [] - Created by MOE: https://github.com/google/moe

[42/50] [abbrv] incubator-beam git commit: Ensure a TypedPValue has a Coder on finishSpecifying

Ensure a TypedPValue has a Coder on finishSpecifying Coders cannot be set on a PValue that is is marked as finished specifying, and a coder is required for every TypedPValue in a pipeline. Ensure that a coder is always available when a TypedPValue has been finished by invoking getCoder() (which

[22/50] [abbrv] incubator-beam git commit: Resubmit "Migrate AvroIO.Write to a custom sink"

Resubmit "Migrate AvroIO.Write to a custom sink" Note for user requested sharding limits to be supported, each pipeline runner must support applying those sharding limits. Google Cloud Dataflow supports sharding limits. Release Notes [] - Created by MOE:

[17/50] [abbrv] incubator-beam git commit: Reverts "Move Google Cloud Dataflow native sinks to worker module"

Reverts "Move Google Cloud Dataflow native sinks to worker module" Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115349469 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[35/50] [abbrv] incubator-beam git commit: Rollback revert "Migrate TextIO.Write to a custom sink"

Rollback revert "Migrate TextIO.Write to a custom sink" Note for user requested sharding limits to be supported, each pipeline runner must support applying those sharding limits. DirectPipelineRunner and Google Cloud Dataflow supports sharding limits. Release Notes [] -

[11/50] [abbrv] incubator-beam git commit: Move some worker-and-example-only dependencies out of sdk

Move some worker-and-example-only dependencies out of sdk Following the separation of the worker module, running $ mvn dependency:analyze -pl sdk yields [WARNING] Used undeclared dependencies found: [WARNING]com.google.code.findbugs:jsr305:jar:3.0.1:compile [WARNING]

[01/50] [abbrv] incubator-beam git commit: Allow TransformEvaluatorFactory#forApplication to throw

Repository: incubator-beam Updated Branches: refs/heads/master 11e842717 -> 211e76abf Allow TransformEvaluatorFactory#forApplication to throw Certain Transforms (like sources) can throw an exception while being constructed. This allows the transforms to throw an exception while being

[04/50] [abbrv] incubator-beam git commit: Add new test proto messages that use map fields

Add new test proto messages that use map fields Directly, and transitively. Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115280894 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit:

[28/50] [abbrv] incubator-beam git commit: Handle PCollectionList.empty() in FlattenEvaluatorFactory

Handle PCollectionList.empty() in FlattenEvaluatorFactory PCollectionList.empty() is a valid argument to a Flatten#pCollections PTransform. It should succeed and produce no output. Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115455733

[08/50] [abbrv] incubator-beam git commit: Add KeyedWorkItemCoder

Add KeyedWorkItemCoder Release Notes [] - Created by MOE: https://github.com/google/moe MOE_MIGRATED_REVID=115354168 Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/635541a7 Tree:

[GitHub] incubator-beam pull request: Initial Dataflow code drop

GitHub user francesperry opened a pull request: https://github.com/apache/incubator-beam/pull/1 Initial Dataflow code drop Initial contribution of the [Google Cloud Dataflow Java SDK](https://github.com/GoogleCloudPlatform/DataflowJavaSDK) to Apache Beam. Caveat: There is