[jira] [Resolved] (BEAM-20) Create build jobs on Jenkins

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-20?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-20.
--
Resolution: Fixed

> Create build jobs on Jenkins
> 
>
> Key: BEAM-20
> URL: https://issues.apache.org/jira/browse/BEAM-20
> Project: Beam
>  Issue Type: Task
>  Components: project-management, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> As soon as we will have the sources on git-wip-us.apache.org, I will create a 
> job on Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-20) Create build jobs on Jenkins

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225725#comment-15225725
 ] 

Jean-Baptiste Onofré commented on BEAM-20:
--

The Jenkins jobs have been created. I proposed on the mailing list to update 
job for nightly build.

> Create build jobs on Jenkins
> 
>
> Key: BEAM-20
> URL: https://issues.apache.org/jira/browse/BEAM-20
> Project: Beam
>  Issue Type: Task
>  Components: project-management, testing
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> As soon as we will have the sources on git-wip-us.apache.org, I will create a 
> job on Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-132) Release process for Beam artifacts to Maven

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-132.
---
Resolution: Fixed

Now that the pom.xml files have been updated, we use the Apache parent POM and 
so, the artifacts (SNAPSHOT and release) are published on Maven (SNAPSHOT repo 
for SNAPSHOTs, and Central for release).

> Release process for Beam artifacts to Maven
> ---
>
> Key: BEAM-132
> URL: https://issues.apache.org/jira/browse/BEAM-132
> Project: Beam
>  Issue Type: New Feature
>  Components: project-management
>Reporter: Kostas Kloudas
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_MavenVerify #74

[Apache Jenkins Server]

See 

--
[...truncated 2524 lines...]
[INFO] --- jacoco-maven-plugin:0.7.5.201505241946:report (report) @ 
java8examples-all ---
[INFO] Analyzed bundle 'Apache Beam :: Examples :: Java 8 All' with 27 classes
[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ java8examples-all ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
java8examples-all ---
[INFO] 
[INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @ java8examples-all 
---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
java8examples-all ---
[INFO] No dependency problems found
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.12:check (default) @ java8examples-all ---
[INFO] Starting audit...
Audit done.

[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
java8examples-all ---
[INFO] Installing 

 to 

[INFO] Installing 
 to 

[INFO] Installing 

 to 

[INFO] Installing 

 to 

[INFO] Installing 

 to 

[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Examples 0.1.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ examples-parent ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
examples-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
examples-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
examples-parent ---
[INFO] Installing 
 to 

[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Parent 0.1.0-incubating-SNAPSHOT
[INFO] 
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
no dependency information available
[WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failure to find org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced
[INFO] 
[INFO] --- coveralls-maven-plugin:4.1.0:report (default-cli) @ parent ---
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5/httpclient-4.5.pom
[INFO] Downloaded: 

[1/2] incubator-beam git commit: Add coveralls plugin to the Maven build process

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ccbf2b802 -> 7892244c9


Add coveralls plugin to the Maven build process

No binding to the lifecycle.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/27d5825a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/27d5825a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/27d5825a

Branch: refs/heads/master
Commit: 27d5825afb2248ea6a30c93b2c112f6f24f700b3
Parents: ccbf2b8
Author: Davor Bonaci 
Authored: Mon Apr 4 18:07:14 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Apr 4 18:07:14 2016 -0700

--
 pom.xml | 7 +++
 1 file changed, 7 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/27d5825a/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3cae1a4..a198724 100644
--- a/pom.xml
+++ b/pom.xml
@@ -240,6 +240,13 @@
   
 
 
+
+
+  org.eluder.coveralls
+  coveralls-maven-plugin
+  4.1.0
+
+
 
   org.apache.maven.plugins
   maven-surefire-plugin

[jira] [Updated] (BEAM-170) Session windows should not be identified by their bounds

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-170:
-
Description: 
Today, if two session windows for the same key have the same bounds, they are 
considered the same window. This is an accident. It is not intended that any 
session windows are considered equal except via the operation of merging them 
into the same session.

A risk associated with this behavior is that two windows that happen to 
coincide will share per-window-and-key state rather than evolving separately 
and having their separate state reconciled by state merging logic. These code 
paths are not required to be coherent, and in practice they are not.

In particular, if the trigger for a session window ever finishes, then 
subsequent data in a window with the same bounds will be dropped, whereas if it 
had differed by a millisecond it would have created a new session, ignoring the 
previously closed session.

  was:
Today, if two session windows have the same bounds, they are considered the 
same window. This is an accident. It is not intended that any session windows 
are considered equal except via the operation of merging them into the same 
session.

A risk associated with this behavior is that two windows that happen to 
coincide will share per-window-and-key state rather than evolving separately 
and having their separate state reconciled by state merging logic. These code 
paths are not required to be coherent, and in practice they are not.

In particular, if the trigger for a session window ever finishes, then 
subsequent data in a window with the same bounds will be dropped, whereas if it 
had differed by a millisecond it would have created a new session, ignoring the 
previously closed session.


> Session windows should not be identified by their bounds
> 
>
> Key: BEAM-170
> URL: https://issues.apache.org/jira/browse/BEAM-170
> Project: Beam
>  Issue Type: Bug
>Reporter: Kenneth Knowles
>
> Today, if two session windows for the same key have the same bounds, they are 
> considered the same window. This is an accident. It is not intended that any 
> session windows are considered equal except via the operation of merging them 
> into the same session.
> A risk associated with this behavior is that two windows that happen to 
> coincide will share per-window-and-key state rather than evolving separately 
> and having their separate state reconciled by state merging logic. These code 
> paths are not required to be coherent, and in practice they are not.
> In particular, if the trigger for a session window ever finishes, then 
> subsequent data in a window with the same bounds will be dropped, whereas if 
> it had differed by a millisecond it would have created a new session, 
> ignoring the previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-53] Pub/sub client with gRPC im...

[mshields822]

GitHub user mshields822 opened a pull request:

https://github.com/apache/incubator-beam/pull/120

[BEAM-53] Pub/sub client with gRPC implementation

Support pub/sub via gRPC.
Will later use for pure Java pub/sub source/sink.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mshields822/incubator-beam pubsub-grpc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #120






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request: Add coveralls plugin to the Maven bui...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam/pull/119

Add coveralls plugin to the Maven build process

No binding to the lifecycle.

R: @dhalperi, @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam coveralls

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #119


commit 27d5825afb2248ea6a30c93b2c112f6f24f700b3
Author: Davor Bonaci 
Date:   2016-04-05T01:07:14Z

Add coveralls plugin to the Maven build process

No binding to the lifecycle.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_RunnableOnService_GoogleCloudDataflow » Apache Beam :: SDKs :: Java :: Core #24

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_RunnableOnService_GoogleCloudDataflow #23

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_RunnableOnService_GoogleCloudDataflow » Apache Beam :: SDKs :: Java :: Core #23

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-171) JAXBCoder ignores Context and inappropriately consumes input

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-171:


 Summary: JAXBCoder ignores Context and inappropriately consumes 
input
 Key: BEAM-171
 URL: https://issues.apache.org/jira/browse/BEAM-171
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Davor Bonaci


JAXBCoder ignores being part of a nested stream with elements following it, and 
consumes the entire input. This is broken, and should be fixed by prepending 
the length of the encoded representation and consuming that amount.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225227#comment-15225227
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/86


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-22] Add BundleFactory, Immutabi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/86


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #86

[bchambers]

This closes #86


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ccbf2b80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ccbf2b80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ccbf2b80

Branch: refs/heads/master
Commit: ccbf2b8025473d30121d1a5b17ba7118abefcb50
Parents: ac314ee 334ab99
Author: bchambers 
Authored: Mon Apr 4 15:44:35 2016 -0700
Committer: bchambers 
Committed: Mon Apr 4 15:44:35 2016 -0700

--
 .../sdk/runners/inprocess/BundleFactory.java|  50 +
 .../ExecutorServiceParallelExecutor.java|   5 +-
 .../ImmutabilityCheckingBundleFactory.java  | 131 +++
 .../inprocess/InProcessBundleFactory.java   | 157 +
 .../inprocess/InProcessEvaluationContext.java   |  18 +-
 .../inprocess/InProcessPipelineRunner.java  |   5 +
 .../BoundedReadEvaluatorFactoryTest.java|  21 +-
 .../inprocess/FlattenEvaluatorFactoryTest.java  |  11 +-
 .../GroupByKeyEvaluatorFactoryTest.java |  10 +-
 .../ImmutabilityCheckingBundleFactoryTest.java  | 220 +++
 .../inprocess/InMemoryWatermarkManagerTest.java |  49 +++--
 .../inprocess/InProcessBundleFactoryTest.java   | 197 +
 .../InProcessEvaluationContextTest.java |  11 +-
 .../ParDoMultiEvaluatorFactoryTest.java |  95 
 .../ParDoSingleEvaluatorFactoryTest.java| 129 ++-
 .../inprocess/TransformExecutorTest.java|  10 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |  10 +-
 .../inprocess/ViewEvaluatorFactoryTest.java |   5 +-
 18 files changed, 980 insertions(+), 154 deletions(-)
--

[2/3] incubator-beam git commit: Add BundleFactory, ImmutabilityCheckingBundleFactory

[bchambers]

Add BundleFactory, ImmutabilityCheckingBundleFactory

This allows checks to be made on the contents of bundles.
ImmutabilityCheckingBundleFactory produces bundles that ensure that
elements output to a bundle are not modified after being output.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/334ab99a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/334ab99a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/334ab99a

Branch: refs/heads/master
Commit: 334ab99ab39b7f0632848b789e2c0af1782b11c0
Parents: ac314ee
Author: Thomas Groh 
Authored: Thu Mar 17 17:39:45 2016 -0700
Committer: bchambers 
Committed: Mon Apr 4 15:44:26 2016 -0700

--
 .../sdk/runners/inprocess/BundleFactory.java|  50 +
 .../ExecutorServiceParallelExecutor.java|   5 +-
 .../ImmutabilityCheckingBundleFactory.java  | 131 +++
 .../inprocess/InProcessBundleFactory.java   | 157 +
 .../inprocess/InProcessEvaluationContext.java   |  18 +-
 .../inprocess/InProcessPipelineRunner.java  |   5 +
 .../BoundedReadEvaluatorFactoryTest.java|  21 +-
 .../inprocess/FlattenEvaluatorFactoryTest.java  |  11 +-
 .../GroupByKeyEvaluatorFactoryTest.java |  10 +-
 .../ImmutabilityCheckingBundleFactoryTest.java  | 220 +++
 .../inprocess/InMemoryWatermarkManagerTest.java |  49 +++--
 .../inprocess/InProcessBundleFactoryTest.java   | 197 +
 .../InProcessEvaluationContextTest.java |  11 +-
 .../ParDoMultiEvaluatorFactoryTest.java |  95 
 .../ParDoSingleEvaluatorFactoryTest.java| 129 ++-
 .../inprocess/TransformExecutorTest.java|  10 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |  10 +-
 .../inprocess/ViewEvaluatorFactoryTest.java |   5 +-
 18 files changed, 980 insertions(+), 154 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/334ab99a/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BundleFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BundleFactory.java
 
b/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BundleFactory.java
new file mode 100644
index 000..cb8a369
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BundleFactory.java
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package com.google.cloud.dataflow.sdk.runners.inprocess;
+
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.GroupByKeyEvaluatorFactory.InProcessGroupByKeyOnly;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.UncommittedBundle;
+import com.google.cloud.dataflow.sdk.transforms.PTransform;
+import com.google.cloud.dataflow.sdk.values.PCollection;
+
+/**
+ * A factory that creates {@link UncommittedBundle UncommittedBundles}.
+ */
+public interface BundleFactory {
+  /**
+   * Create an {@link UncommittedBundle} from an empty input. Elements added 
to the bundle belong to
+   * the {@code output} {@link PCollection}.
+   */
+  public  UncommittedBundle createRootBundle(PCollection output);
+
+  /**
+   * Create an {@link UncommittedBundle} from the specified input. Elements 
added to the bundle
+   * belong to the {@code output} {@link PCollection}.
+   */
+  public  UncommittedBundle createBundle(CommittedBundle input, 
PCollection output);
+
+  /**
+   * Create an {@link UncommittedBundle} with the specified keys at the 
specified step. For use by
+   * {@link InProcessGroupByKeyOnly} {@link PTransform PTransforms}. Elements 
added to the bundle
+   * belong to the {@code output} {@link PCollection}.
+   */
+  public  UncommittedBundle createKeyedBundle(
+  CommittedBundle 

[jira] [Commented] (BEAM-162) assert fail using session windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225226#comment-15225226
 ] 

ASF GitHub Bot commented on BEAM-162:
-

GitHub user mshields822 opened a pull request:

https://github.com/apache/incubator-beam/pull/118

[BEAM-162] Sensible interaction of merging and closed triggers

Same bug, better approach.
 - We always check for closed triggers on the post-merged window state.
 - We are careful not to leak NEW windows.
 - More unit tests.
 - Along the way, discovered we were clearing trigger finished bits.
   Since TriggerRunner does not currently have its own unit tests,
   added new unit test in ReduceFnRunnerTest to hit that specific case.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mshields822/incubator-beam beam-162

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit 55c34a553264491bd42db4197793eff436858b73
Author: Mark Shields 
Date:   2016-04-04T20:31:12Z

Clear finished bits from merged-away windows, not merge result window.

commit 69eb778b354cd63fb23a48789c93ed48c2a01961
Author: Mark Shields 
Date:   2016-03-31T18:36:01Z

Check for closed windows post-merging rather than pre-merging.
Make sure we garbage collect NEW windows which end up being for closed 
windows.
Add unit tests to confirm.




> assert fail using session windows
> -
>
> Key: BEAM-162
> URL: https://issues.apache.org/jira/browse/BEAM-162
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Mark Shields
>
> java.lang.IllegalStateException: Window 
> [2016-03-31T05:35:31.158Z..2016-03-31T06:05:31.158Z) should have been added
> at 
> com.google.cloud.dataflow.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:199)
> at 
> com.google.cloud.dataflow.sdk.util.ReduceFnRunner.processElement(ReduceFnRunner.java:440)
> at 
> com.google.cloud.dataflow.sdk.util.ReduceFnRunner.processElements(ReduceFnRunner.java:282)
> at 
> com.google.cloud.dataflow.sdk.util.GroupAlsoByWindowViaWindowSetDoFn.processElement(GroupAlsoByWindowViaWindowSetDoFn.java:83)
> at 
> com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
> at 
> com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:138)
> at 
> com.google.cloud.dataflow.sdk.util.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:67)
> at 
> com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:191)
> at 
> com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
> at 
> com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
> at 
> com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:53)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: Add BundleFactory, ImmutabilityCheckingBundleFactory

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ac314eefd -> ccbf2b802


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/334ab99a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformExecutorTest.java
--
diff --git 
a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformExecutorTest.java
 
b/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformExecutorTest.java
index a710753..b029dd3 100644
--- 
a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformExecutorTest.java
+++ 
b/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformExecutorTest.java
@@ -75,6 +75,7 @@ public class TransformExecutorTest {
 
   private RegisteringCompletionCallback completionCallback;
   private TransformExecutorService transformEvaluationState;
+  private BundleFactory bundleFactory;
   @Mock private InProcessEvaluationContext evaluationContext;
   @Mock private TransformEvaluatorRegistry registry;
   private Map scheduled;
@@ -83,6 +84,8 @@ public class TransformExecutorTest {
   public void setup() {
 MockitoAnnotations.initMocks(this);
 
+bundleFactory = InProcessBundleFactory.create();
+
 scheduled = new HashMap<>();
 transformEvaluationState =
 
TransformExecutorServices.parallel(MoreExecutors.newDirectExecutorService(), 
scheduled);
@@ -157,7 +160,7 @@ public class TransformExecutorTest {
 WindowedValue spam = WindowedValue.valueInGlobalWindow("spam");
 WindowedValue third = WindowedValue.valueInGlobalWindow("third");
 CommittedBundle inputBundle =
-
InProcessBundle.unkeyed(created).add(foo).add(spam).add(third).commit(Instant.now());
+
bundleFactory.createRootBundle(created).add(foo).add(spam).add(third).commit(Instant.now());
 when(
 registry.forApplication(
 downstream.getProducingTransformInternal(), inputBundle, 
evaluationContext))
@@ -203,7 +206,7 @@ public class TransformExecutorTest {
 
 WindowedValue foo = WindowedValue.valueInGlobalWindow("foo");
 CommittedBundle inputBundle =
-InProcessBundle.unkeyed(created).add(foo).commit(Instant.now());
+bundleFactory.createRootBundle(created).add(foo).commit(Instant.now());
 when(
 registry.forApplication(
 downstream.getProducingTransformInternal(), inputBundle, 
evaluationContext))
@@ -241,7 +244,8 @@ public class TransformExecutorTest {
   }
 };
 
-CommittedBundle inputBundle = 
InProcessBundle.unkeyed(created).commit(Instant.now());
+CommittedBundle inputBundle =
+bundleFactory.createRootBundle(created).commit(Instant.now());
 when(
 registry.forApplication(
 downstream.getProducingTransformInternal(), inputBundle, 
evaluationContext))

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/334ab99a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/UnboundedReadEvaluatorFactoryTest.java
--
diff --git 
a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/UnboundedReadEvaluatorFactoryTest.java
 
b/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/UnboundedReadEvaluatorFactoryTest.java
index ce4776d..dfcafaa 100644
--- 
a/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/UnboundedReadEvaluatorFactoryTest.java
+++ 
b/sdks/java/core/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/UnboundedReadEvaluatorFactoryTest.java
@@ -72,6 +72,8 @@ public class UnboundedReadEvaluatorFactoryTest {
   private InProcessEvaluationContext context;
   private UncommittedBundle output;
 
+  private BundleFactory bundleFactory = InProcessBundleFactory.create();
+
   @Before
   public void setup() {
 UnboundedSource source =
@@ -81,7 +83,7 @@ public class UnboundedReadEvaluatorFactoryTest {
 
 factory = new UnboundedReadEvaluatorFactory();
 context = mock(InProcessEvaluationContext.class);
-output = InProcessBundle.unkeyed(longs);
+output = bundleFactory.createRootBundle(longs);
 when(context.createRootBundle(longs)).thenReturn(output);
   }
 
@@ -118,7 +120,7 @@ public class UnboundedReadEvaluatorFactoryTest {
 tgw(1L), tgw(2L), tgw(4L), tgw(8L), tgw(9L), tgw(7L), tgw(6L), 
tgw(5L), tgw(3L),
 tgw(0L)));
 
-UncommittedBundle secondOutput = InProcessBundle.unkeyed(longs);
+UncommittedBundle secondOutput = 
bundleFactory.createRootBundle(longs);
 when(context.createRootBundle(longs)).thenReturn(secondOutput);
 TransformEvaluator secondEvaluator =
 factory.forApplication(longs.getProducingTransformInternal(), null, 
context);
@@ -141,6 +143,7 @@ 

[GitHub] incubator-beam pull request: [BEAM-162] Sensible interaction of me...

[mshields822]

GitHub user mshields822 opened a pull request:

https://github.com/apache/incubator-beam/pull/118

[BEAM-162] Sensible interaction of merging and closed triggers

Same bug, better approach.
 - We always check for closed triggers on the post-merged window state.
 - We are careful not to leak NEW windows.
 - More unit tests.
 - Along the way, discovered we were clearing trigger finished bits.
   Since TriggerRunner does not currently have its own unit tests,
   added new unit test in ReduceFnRunnerTest to hit that specific case.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mshields822/incubator-beam beam-162

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit 55c34a553264491bd42db4197793eff436858b73
Author: Mark Shields 
Date:   2016-04-04T20:31:12Z

Clear finished bits from merged-away windows, not merge result window.

commit 69eb778b354cd63fb23a48789c93ed48c2a01961
Author: Mark Shields 
Date:   2016-03-31T18:36:01Z

Check for closed windows post-merging rather than pre-merging.
Make sure we garbage collect NEW windows which end up being for closed 
windows.
Add unit tests to confirm.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_RunnableOnService_GoogleCloudDataflow #22

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-167] Fix custom source gzip input to read concatenated gzip files

[lcwik] [BEAM-167] Add comment for the test explaining why concatenation of gzip

[dhalperi] IntervalBoundedExponentialBackOff: add deprecated copy of old API

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/incubator-beam.git # 
 > timeout=10
Fetching upstream changes from https://github.com/apache/incubator-beam.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://github.com/apache/incubator-beam.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision ac314eefd15b1e86a29799547d08c0f832377887 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ac314eefd15b1e86a29799547d08c0f832377887
 > git rev-list fd049b52e196f1d04500470363cccbb652233610 # timeout=10
Parsing POMs
Established TCP socket on 39814
maven32-agent.jar already up to date
maven32-interceptor.jar already up to date
maven3-interceptor-commons.jar already up to date
[beam_RunnableOnService_GoogleCloudDataflow] $ 
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_66/bin/java -Xmx2g 
-Xms256m -XX:MaxPermSize=512m -cp 
/home/jenkins/jenkins-slave/maven32-agent.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Maven_MavenInstallation/maven-3.3.3/boot/plexus-classworlds-2.5.2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Maven_MavenInstallation/maven-3.3.3/conf/logging
 jenkins.maven3.agent.Maven32Main 
/home/jenkins/jenkins-slave/tools/hudson.tasks.Maven_MavenInstallation/maven-3.3.3
 /home/jenkins/jenkins-slave/slave.jar 
/home/jenkins/jenkins-slave/maven32-interceptor.jar 
/home/jenkins/jenkins-slave/maven3-interceptor-commons.jar 39814
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
support was removed in 8.0
<===[JENKINS REMOTING CAPACITY]===>   channel started
Executing Maven:  -B -f 

 
-Dmaven.repo.local=
 -B -e clean verify -pl .,sdks/java/core -P DataflowPipelineTests 
-DdataflowOptions=[ "--project=apache-beam-testing", 
"--stagingLocation=gs://staging-for-runnable-on-service-jenkins-tests/" ]
[INFO] Error stacktraces are turned on.
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Beam :: Parent
[INFO] Apache Beam :: SDKs :: Java :: Core
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Parent 0.1.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ parent ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ parent 
---
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: Core 0.1.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ java-sdk-all ---
[INFO] Deleting 

[INFO] 
[INFO] --- jacoco-maven-plugin:0.7.5.201505241946:prepare-agent (default) @ 
java-sdk-all ---
[INFO] argLine set to 
-javaagent:
[INFO] 
[INFO] --- avro-maven-plugin:1.7.7:schema (schemas) @ java-sdk-all ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ java-sdk-all 
---
[INFO] 
[INFO] --- 

Build failed in Jenkins: beam_RunnableOnService_GoogleCloudDataflow » Apache Beam :: SDKs :: Java :: Core #22

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-167] Fix custom source gzip input to read concatenated gzip files

[lcwik] [BEAM-167] Add comment for the test explaining why concatenation of gzip

[dhalperi] IntervalBoundedExponentialBackOff: add deprecated copy of old API

--
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: SDKs :: Java :: Core 0.1.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ java-sdk-all ---
[INFO] Deleting 

[INFO] 
[INFO] --- jacoco-maven-plugin:0.7.5.201505241946:prepare-agent (default) @ 
java-sdk-all ---
[INFO] argLine set to 
-javaagent:/home/jenkins/jenkins-slave/workspace/beam_RunnableOnService_GoogleCloudDataflow/.repository/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar=destfile=
[INFO] 
[INFO] --- avro-maven-plugin:1.7.7:schema (schemas) @ java-sdk-all ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ java-sdk-all 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
java-sdk-all ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ java-sdk-all 
---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 461 source files to 

[WARNING] 
:
 Some input files use or override a deprecated API.
[WARNING] 
:
 Recompile with -Xlint:deprecation for details.
[WARNING] 
:
 Some input files use unchecked or unsafe operations.
[WARNING] 
:
 Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] >>> maven-source-plugin:2.4:jar (attach-sources) > generate-sources @ 
java-sdk-all >>>
[INFO] 
[INFO] --- jacoco-maven-plugin:0.7.5.201505241946:prepare-agent (default) @ 
java-sdk-all ---
[INFO] argLine set to 
-javaagent:/home/jenkins/jenkins-slave/workspace/beam_RunnableOnService_GoogleCloudDataflow/.repository/org/jacoco/org.jacoco.agent/0.7.5.201505241946/org.jacoco.agent-0.7.5.201505241946-runtime.jar=destfile=
[INFO] 
[INFO] --- avro-maven-plugin:1.7.7:schema (schemas) @ java-sdk-all ---
[WARNING] Failed to getClass for org.apache.maven.plugin.source.SourceJarMojo
[INFO] 
[INFO] <<< maven-source-plugin:2.4:jar (attach-sources) < generate-sources @ 
java-sdk-all <<<
[INFO] 
[INFO] --- maven-source-plugin:2.4:jar (attach-sources) @ java-sdk-all ---
[INFO] Building jar: 

[INFO] 
[INFO] --- build-helper-maven-plugin:1.9.1:add-test-source (add-test-source) @ 
java-sdk-all ---
[INFO] Test Source directory: 

 added.
[INFO] 
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources) @ 
java-sdk-all ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
java-sdk-all ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 266 source files to 

[1/2] incubator-beam git commit: IntervalBoundedExponentialBackOff: add deprecated copy of old API

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master d75e9df50 -> ac314eefd


IntervalBoundedExponentialBackOff: add deprecated copy of old API

This fixes a bug submitting jobs to Google Cloud Dataflow service.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c4862c0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c4862c0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c4862c0e

Branch: refs/heads/master
Commit: c4862c0ea08b664c55a885b64039f4a4be16022a
Parents: d75e9df
Author: Dan Halperin 
Authored: Mon Apr 4 14:40:00 2016 -0700
Committer: Dan Halperin 
Committed: Mon Apr 4 15:24:43 2016 -0700

--
 .../dataflow/sdk/util/IntervalBoundedExponentialBackOff.java   | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c4862c0e/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/IntervalBoundedExponentialBackOff.java
--
diff --git 
a/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/IntervalBoundedExponentialBackOff.java
 
b/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/IntervalBoundedExponentialBackOff.java
index 51ff9f2..206c83b 100644
--- 
a/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/IntervalBoundedExponentialBackOff.java
+++ 
b/sdks/java/core/src/main/java/com/google/cloud/dataflow/sdk/util/IntervalBoundedExponentialBackOff.java
@@ -54,6 +54,12 @@ public class IntervalBoundedExponentialBackOff implements 
BackOff {
   private final long initialIntervalMillis;
   private int currentAttempt;
 
+  // BEAM-168: https://issues.apache.org/jira/browse/BEAM-168
+  @Deprecated
+  public IntervalBoundedExponentialBackOff(int maximumIntervalMillis, long 
initialIntervalMillis) {
+this((long) maximumIntervalMillis, initialIntervalMillis);
+  }
+
   public IntervalBoundedExponentialBackOff(long maximumIntervalMillis, long 
initialIntervalMillis) {
 Preconditions.checkArgument(
 maximumIntervalMillis > 0, "Maximum interval must be greater than 
zero.");

[2/2] incubator-beam git commit: Closes #117

[dhalperi]

Closes #117


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ac314eef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ac314eef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ac314eef

Branch: refs/heads/master
Commit: ac314eefd15b1e86a29799547d08c0f832377887
Parents: d75e9df c4862c0
Author: Dan Halperin 
Authored: Mon Apr 4 15:24:44 2016 -0700
Committer: Dan Halperin 
Committed: Mon Apr 4 15:24:44 2016 -0700

--
 .../dataflow/sdk/util/IntervalBoundedExponentialBackOff.java   | 6 ++
 1 file changed, 6 insertions(+)
--

[jira] [Created] (BEAM-170) Session windows should not be identified by their bounds

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-170:


 Summary: Session windows should not be identified by their bounds
 Key: BEAM-170
 URL: https://issues.apache.org/jira/browse/BEAM-170
 Project: Beam
  Issue Type: Bug
Reporter: Kenneth Knowles


Today, if two session windows have the same bounds, they are considered the 
same window. This is an accident. It is not intended that any session windows 
are considered equal except via the operation of merging them into the same 
session.

A risk associated with this behavior is that two windows that happen to 
coincide will share per-window-and-key state rather than evolving separately 
and having their separate state reconciled by state merging logic. These code 
paths are not required to be coherent, and in practice they are not.

In particular, if the trigger for a session window ever finishes, then 
subsequent data in a window with the same bounds will be dropped, whereas if it 
had differed by a millisecond it would have created a new session, ignoring the 
previously closed session.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-169) Need serialized form and serialVersionUID for user-facing superclasses

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-169:


 Summary: Need serialized form and serialVersionUID for user-facing 
superclasses
 Key: BEAM-169
 URL: https://issues.apache.org/jira/browse/BEAM-169
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles


When a class does not have an explicit {{serialVersionUID}}, it should be 
considered an unstable value based on the exact version of the code. This is 
fine for transmission most of the time, but never acceptable for persistence 
where backwards compatibility matters.

There are two use cases that require explicit serialized form and 
{{serialVersionUID}} even just for transmission. They are required for 
user-facing superclasses such as DoFn, WindowFn, etc, to support the following:

# Encoding a pipeline with a JDK and decoding with a JDK that computes defaults 
differently.
# Encoding a pipeline against a version of the Beam SDK and decoding with a 
different version.

The first situation should be rare since there is a deterministic spec, but we 
have unfortunately seen it.

The second situation is very reasonable; a runner might want to run with 
additional security fixes in the SDK, etc. Given a correct semantic version for 
the SDK, the pipeline author and runner author may reasonably expect it to work.

So we should add explicit serialization to superclasses that are necessarily 
encoded as part of a user's pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225168#comment-15225168
 ] 

ASF GitHub Bot commented on BEAM-22:


Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/112


> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-167) TextIO can't read concatenated gzip files

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225167#comment-15225167
 ] 

Luke Cwik commented on BEAM-167:


Merged PR, also backported to DataflowJavaSDK.

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Luke Cwik
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-22] Use an explicit coder in Xm...

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/112


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam-site git commit: This closes #8

[davor]

This closes #8


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ece35158
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ece35158
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ece35158

Branch: refs/heads/asf-site
Commit: ece35158936380ce6dc18ab2af8016561c97cdb3
Parents: 29f8287 cbb7427
Author: Davor Bonaci 
Authored: Mon Apr 4 14:54:13 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Apr 4 14:54:13 2016 -0700

--
 _includes/header.html   |   5 +-
 _pages/presentation_materials.md|  28 
 _posts/2016-04-03-presentation-materials.md |  24 
 .../2016/04/03/presentation-materials.html  | 140 +++
 content/blog/index.html |  21 ++-
 content/feed.xml|  39 +-
 content/index.html  |   7 +-
 content/presentation-materials/index.html   | 138 ++
 8 files changed, 397 insertions(+), 5 deletions(-)
--

[1/2] incubator-beam-site git commit: Addition of public materials page/post

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 29f8287fd -> ece351589


Addition of public materials page/post

Updates based on PR feedback

Updates based on PR feedback, 2

Updates based on PR feedback, 3

Updates based on PR feedback, 4

Updates based on PR feedback

Updates based on PR feedback


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/cbb74270
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/cbb74270
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/cbb74270

Branch: refs/heads/asf-site
Commit: cbb742704155b53a30b57fec161ad0e9df8b6c87
Parents: 29f8287
Author: James Malone 
Authored: Fri Apr 1 16:23:02 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Apr 4 14:52:32 2016 -0700

--
 _includes/header.html   |   5 +-
 _pages/presentation_materials.md|  28 
 _posts/2016-04-03-presentation-materials.md |  24 
 .../2016/04/03/presentation-materials.html  | 140 +++
 content/blog/index.html |  21 ++-
 content/feed.xml|  39 +-
 content/index.html  |   7 +-
 content/presentation-materials/index.html   | 138 ++
 8 files changed, 397 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/cbb74270/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 5121f74..6311f39 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -11,7 +11,10 @@
   Documentation 
   
 Getting 
Started
-   Capability 
Matrix
+Presentation Materials
+
+Technical Documentation
+Capability 
Matrix
 https://goo.gl/ps8twC;>Technical Docs
 https://goo.gl/nk5OM0;>Technical Vision
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/cbb74270/_pages/presentation_materials.md
--
diff --git a/_pages/presentation_materials.md b/_pages/presentation_materials.md
new file mode 100644
index 000..4ccb384
--- /dev/null
+++ b/_pages/presentation_materials.md
@@ -0,0 +1,28 @@
+---
+layout: page
+title: "Apache Beam Presentation Materials"
+permalink: /presentation-materials/
+---
+
+Are you interested in learning more about Apache Beam or giving a talk about 
Apache Beam? Excellent! The Apache Beam community has created this collection 
of materials to help you get started. The community periodically reviews these 
materials to ensure they are accurate and up-to-date.
+
+
+  
+https://drive.google.com/embeddedfolderview?id=0B-IhJZh9Ab52a3JLVXFWMDltcHM#list;
 width="700" height="300" frameborder="0">
+  
+
+
+## Using these materials
+These materials can be used by anyone. If you do use these materials, please 
remember to give credit to the authors cited in each presentation. These 
materials include speaker notes and you can view them by clicking **`View`** 
and selecting **`Show speaker notes`**. Some presentations have `[Long]` or 
`[Short]` in their name, such as the *Apache Beam Model*, which means there are 
two different presentations (a long and short version) depending on your needs 
and time constraints.
+
+If you have any questions about the these materials, please ask on the 
[`u...@beam.incubator.apache.org`](mailto:u...@beam.incubator.apache.org) 
mailing list.
+
+The Apache Beam community is looking for feedback on these materials to 
improve them over time. If you deliver a presentation based on these materials, 
we'd love to hear from you on the 
[`u...@beam.incubator.apache.org`](mailto:u...@beam.incubator.apache.org) 
mailing list. Please share your thoughts, questions you were asked, and (if 
available and appropriate) a link to the recording of your talk so the rest of 
the Apache Beam community can benefit from it.
+
+## Contributing new content
+The Apache Beam community is always looking for new core content and 
contributions!
+
+If you are interested in contributing new content, we recommend you first 
search the [Apache Beam JIRA](https://issues.apache.org/jira/browse/BEAM) to 
see if your idea currently has an open JIRA item. If it does not, send an email 
to the 
[`u...@beam.incubator.apache.org`](mailto:u...@beam.incubator.apache.org) 
mailing list or open a new issue in the [Apache Beam 
JIRA](https://issues.apache.org/jira/browse/BEAM) using the 

[jira] [Commented] (BEAM-167) TextIO can't read concatenated gzip files

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225122#comment-15225122
 ] 

Daniel Halperin commented on BEAM-167:
--

Thanks Eugene!

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Luke Cwik
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-168) IntervalBoundedExponentialBackOff change broke Beam-on-Dataflow

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-168:
-
Summary: IntervalBoundedExponentialBackOff change broke Beam-on-Dataflow  
(was: IntervalBoundedBackOff change broke Beam-on-Dataflow)

> IntervalBoundedExponentialBackOff change broke Beam-on-Dataflow
> ---
>
> Key: BEAM-168
> URL: https://issues.apache.org/jira/browse/BEAM-168
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> Changing the `int` to a `long` breaks ABI compatibility, which Dataflow 
> service uses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-167] Fix decompression of a con...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/114


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-167) TextIO can't read concatenated gzip files

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225007#comment-15225007
 ] 

Luke Cwik commented on BEAM-167:


Produced PR: https://github.com/apache/incubator-beam/pull/114
Assigned to [~kenn]

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Luke Cwik
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-161) Create FAQ

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15225003#comment-15225003
 ] 

ASF GitHub Bot commented on BEAM-161:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/114

[BEAM-161] Fix decompression of a concatenation of gzip streams in 
CompressedSource

I pulled in the patch from kirpic...@google.com and gave him authorship of 
the commit.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam161

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 88602c9ca09538f22557292e19e1dd38f8731744
Author: kirpichov 
Date:   2016-04-04T20:31:23Z

[BEAM-167] Fix custom source gzip input to read concatenated gzip files

This applies patch from kirpic...@google.com from 
https://gist.github.com/jkff/d8d984a33a41ec607328cee8e418c174

commit 7cc6acefc0d764852e88fe1ae50660e301b600d0
Author: Luke Cwik 
Date:   2016-04-04T20:35:25Z

[BEAM-161] Add comment for the test explaining why concatenation of gzip 
files is valid




> Create FAQ
> --
>
> Key: BEAM-161
> URL: https://issues.apache.org/jira/browse/BEAM-161
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: James Malone
>Assignee: James Malone
>
> Create FAQ for Beam website



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-167) TextIO can't read concatenated gzip files

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-167:
--

Assignee: Luke Cwik

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Luke Cwik
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-167) TextIO can't read concatenated gzip files

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-167:
--
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-167) TextIO can't read concatenated gzip files

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-167:
--
Component/s: sdk-java-core

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-2) Create website

[James Malone (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Malone resolved BEAM-2.
-
Resolution: Fixed

Website has been created. Resolving issue.

> Create website
> --
>
> Key: BEAM-2
> URL: https://issues.apache.org/jira/browse/BEAM-2
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-167) TextIO can't read concatenated gzip files

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224834#comment-15224834
 ] 

Eugene Kirpichov commented on BEAM-167:
---

Here's a test and a patch 
https://gist.github.com/jkff/d8d984a33a41ec607328cee8e418c174
(I haven't yet gone through the contribution guide steps. Will do as soon as I 
get to it; meanwhile anybody who has - feel free to use this directly).

> TextIO can't read concatenated gzip files
> -
>
> Key: BEAM-167
> URL: https://issues.apache.org/jira/browse/BEAM-167
> Project: Beam
>  Issue Type: Bug
>Reporter: Eugene Kirpichov
>
> $ cat < header.csv
> a,b,c
> END
> $ cat < body.csv
> 1,2,3
> 4,5,6
> 7,8,9
> END
> $ gzip -c header.csv > file.gz
> $ gzip -c body.csv >> file.gz
> The file is well-formed:
> $ gzip -dc file.gz
> a,b,c
> 1,2,3
> 4,5,6
> 7,8,9
> However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
> reproducible even when the file is on local disk and with the 
> DirectPipelineRunner.
> The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
> default reads only the first gzip stream in the file, but has an option to 
> read all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream 
> which reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-167) TextIO can't read concatenated gzip files

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-167:
-

 Summary: TextIO can't read concatenated gzip files
 Key: BEAM-167
 URL: https://issues.apache.org/jira/browse/BEAM-167
 Project: Beam
  Issue Type: Bug
Reporter: Eugene Kirpichov


$ cat < header.csv
a,b,c
END
$ cat < body.csv
1,2,3
4,5,6
7,8,9
END
$ gzip -c header.csv > file.gz
$ gzip -c body.csv >> file.gz

The file is well-formed:
$ gzip -dc file.gz
a,b,c
1,2,3
4,5,6
7,8,9

However, TextIO.Read.from("/path/to/file.gz") will read only "a,b,c" - 
reproducible even when the file is on local disk and with the 
DirectPipelineRunner.

The bug is in CompressedSource. It uses GzipCompressorInputStream, which by 
default reads only the first gzip stream in the file, but has an option to read 
all of them. Previously (in Dataflow SDK 1.4.0) we used GZIPInputStream which 
reads all streams.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam-site git commit: Capability matrix updates: - Add sliding window support for batch Spark runner. - Center capability matrix (expand/collapse looks cleaner). - Default to summary v

[takidau]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 8615ea0c0 -> 29f8287fd


Capability matrix updates:
- Add sliding window support for batch Spark runner.
- Center capability matrix (expand/collapse looks cleaner).
- Default to summary view (easier to grok).
- Add "Last updated" timestamp.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5ddac1f3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5ddac1f3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5ddac1f3

Branch: refs/heads/asf-site
Commit: 5ddac1f37c59438fe7d6e0b8833a7c11e788eaa4
Parents: 8615ea0
Author: Tyler Akidau 
Authored: Fri Apr 1 20:08:40 2016 -0700
Committer: Tyler Akidau 
Committed: Fri Apr 1 20:08:40 2016 -0700

--
 _data/capability-matrix.yml  |  4 ++--
 _pages/capability-matrix.md  |  9 +++--
 content/capability-matrix/index.html | 23 +++
 content/feed.xml |  4 ++--
 4 files changed, 30 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5ddac1f3/_data/capability-matrix.yml
--
diff --git a/_data/capability-matrix.yml b/_data/capability-matrix.yml
index 785854a..f34527d 100644
--- a/_data/capability-matrix.yml
+++ b/_data/capability-matrix.yml
@@ -249,8 +249,8 @@ categories:
 l2: supported
 l3: ''
   - class: spark
-l1: 'No'
-l2: ''
+l1: 'Partially'
+l2: currently only supported in batch
 l3: ''
 
   - name: Session windows

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5ddac1f3/_pages/capability-matrix.md
--
diff --git a/_pages/capability-matrix.md b/_pages/capability-matrix.md
index f1b4b05..359164d 100644
--- a/_pages/capability-matrix.md
+++ b/_pages/capability-matrix.md
@@ -4,7 +4,9 @@ title: "Apache Beam Capability Matrix"
 permalink: /capability-matrix/
 ---
 
+
 # Apache Beam Capability Matrix
+Last updated: {{ site.time | date: 
'%Y-%m-%d %H:%M %Z' }}
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the [Dataflow 
Model](http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf)), and implemented to 
varying degrees in each Beam runner. To help clarify the capabilities of 
individual runners, we've created the capability matrix below.
 
@@ -22,12 +24,14 @@ Note that in the future, we intend to add additional tables 
beyond the current s
 {% include capability-matrix-common.md %}
 {% assign cap-data=site.data.capability-matrix %}
 
+
+
 
 {% assign cap-style='cap-summary' %}
 {% assign cap-view='summary' %}
 {% assign cap-other-view='full' %}
 {% assign cap-toggle-details=1 %}
-{% assign cap-display='none' %}
+{% assign cap-display='block' %}
 
 {% include capability-matrix.md %}
 
@@ -36,6 +40,7 @@ Note that in the future, we intend to add additional tables 
beyond the current s
 {% assign cap-view='full' %}
 {% assign cap-other-view='summary' %}
 {% assign cap-toggle-details=0 %}
-{% assign cap-display='block' %}
+{% assign cap-display='none' %}
 
 {% include capability-matrix.md %}
+

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5ddac1f3/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index e8e1960..b7b86a2 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -79,6 +79,7 @@
 
   
 Apache Beam Capability 
Matrix
+Last updated: 2016-04-01 20:08 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf;>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 
@@ -103,9 +104,16 @@
   }
 
 
+
+
 
 
-
+
+
+
+
+
+
 
   
   
@@ -439,7 +447,7 @@
 
 
 
-
+~
 
 
   
@@ -866,9 +874,15 @@
 
 
 
+
 
 
-
+
+
+
+
+
+
 
   
   
@@ -1202,7 +1216,7 @@
 
 
 
-No
+Partially: currently 
only 

[jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224697#comment-15224697
 ] 

ASF GitHub Bot commented on BEAM-22:


GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/113

[BEAM-22] Give root transforms step names

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Fix a bug where steps would only be given step names if they were a
non-root node.

Use the ConsumerTrackingPipelineVisitor in the
InProcessEvaluationContext test to handle runner-expanded transforms

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam step_names_everywhere

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #113






> DirectPipelineRunner: support for unbounded collections
> ---
>
> Key: BEAM-22
> URL: https://issues.apache.org/jira/browse/BEAM-22
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Reporter: Davor Bonaci
>Assignee: Thomas Groh
>
> DirectPipelineRunner currently runs over bounded PCollections only, and 
> implements only a portion of the Beam Model.
> We should improve it to faithfully implement the full Beam Model, such as add 
> ability to run over unbounded PCollections, and better resemble execution 
> model in a distributed system.
> This further enables features such as a testing source which may simulate 
> late data and test triggers in the pipeline. Finally, we may want to expose 
> an option to select between "debug" (single threaded), "chaos monkey" (test 
> as many model requirements as possible), and "performance" (multi-threaded).
> more testing (chaos monkey) 
> Once this is done, we should update this StackOverflow question:
> http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request: [BEAM-22] Give root transforms step n...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/113

[BEAM-22] Give root transforms step names

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Fix a bug where steps would only be given step names if they were a
non-root node.

Use the ConsumerTrackingPipelineVisitor in the
InProcessEvaluationContext test to handle runner-expanded transforms

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam step_names_everywhere

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #113






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_RunnableOnService_GoogleCloudDataflow » Apache Beam :: SDKs :: Java :: Core #20

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_RunnableOnService_GoogleCloudDataflow #19

[Apache Jenkins Server]

See