[GitHub] beam pull request #2259: Fix shading opt out in io/google-cloud-platform

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2259

Fix shading opt out in io/google-cloud-platform

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam fix-google-io-shading-opt-out

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2259.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2259


commit dd1b001728375aeecfc577de7ea13761aa37322c
Author: Aviem Zur 
Date:   2017-03-16T08:14:36Z

Fix shading opt out in io/google-cloud-platform




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2239: Improve tox installation for python sdk

[iemejia]

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/2239


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1733) Python SDK build process leaks tox depenency into user space

[JIRA]

Ismaël Mejía created BEAM-1733:
--

 Summary: Python SDK build process leaks tox depenency into user 
space
 Key: BEAM-1733
 URL: https://issues.apache.org/jira/browse/BEAM-1733
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Ismaël Mejía
Assignee: Ahmet Altay
Priority: Minor


The current build process of the python SDK should be fully contained in a 
virtualenv, at this moment the tox dependency is not correctly contained and 
end up installed in the user account.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1733) Python SDK build process leaks tox dependency into user space

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-1733:
---
Summary: Python SDK build process leaks tox dependency into user space  
(was: Python SDK build process leaks tox depenency into user space)

> Python SDK build process leaks tox dependency into user space
> -
>
> Key: BEAM-1733
> URL: https://issues.apache.org/jira/browse/BEAM-1733
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ismaël Mejía
>Assignee: Ahmet Altay
>Priority: Minor
>
> The current build process of the python SDK should be fully contained in a 
> virtualenv, at this moment the tox dependency is not correctly contained and 
> end up installed in the user account.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1733) Python SDK build process leaks tox depenency into user space

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927709#comment-15927709
 ] 

Ismaël Mejía commented on BEAM-1733:


I note here for ref the PR that started this discussion.
https://github.com/apache/beam/pull/2239


> Python SDK build process leaks tox depenency into user space
> 
>
> Key: BEAM-1733
> URL: https://issues.apache.org/jira/browse/BEAM-1733
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ismaël Mejía
>Assignee: Ahmet Altay
>Priority: Minor
>
> The current build process of the python SDK should be fully contained in a 
> virtualenv, at this moment the tox dependency is not correctly contained and 
> end up installed in the user account.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #2924

[Apache Jenkins Server]

See 


--
[...truncated 744.79 KB...]
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-03-16T12:10:16.286 [INFO] 

2017-03-16T12:10:16.286 [INFO] Reactor Summary:
2017-03-16T12:10:16.286 [INFO] 
2017-03-16T12:10:16.286 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 19.390 s]
2017-03-16T12:10:16.286 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SUCCESS [  9.130 s]
2017-03-16T12:10:16.286 [INFO] Apache Beam :: SDKs 
 SUCCESS [  5.278 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Common 
.. SUCCESS [  1.880 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Common :: Fn API 
 SUCCESS [ 14.820 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Common :: Runner API 
 SUCCESS [ 10.914 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java 
 SUCCESS [  1.795 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Core 
 SUCCESS [02:31 min]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners 
. SUCCESS [  1.646 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners :: Core Java Construction 
... SUCCESS [ 12.313 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners :: Core Java 
 SUCCESS [ 42.196 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners :: Direct Java 
.. SUCCESS [02:22 min]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO 
.. SUCCESS [  1.946 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: 
Elasticsearch . SUCCESS [ 30.922 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners :: Google Cloud Dataflow 
 SUCCESS [ 26.396 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: Google 
Cloud Platform FAILURE [ 46.227 s]
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop 
Common . SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: HBase 
. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: HDFS 
.. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: JDBC 
.. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: JMS 
... SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: Kafka 
. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: Kinesis 
... SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: MongoDB 
... SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: IO :: MQTT 
.. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes 
 SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes 
:: Starter SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes 
:: Examples SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Maven Archetypes 
:: Examples - Java 8 SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Extensions 
.. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Extensions :: 
Jackson SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Extensions :: 
Join library SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Extensions :: 
Sorter  SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Harness 
. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Java :: Java 8 Tests 
 SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: SDKs :: Python 
.. SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Runners :: Flink 
 SKIPPED
2017-03-16T12:10:16.287 [INFO] Apache Beam :: Ru

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1257

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1718) Returning Duration.millis(Long.MAX_VALUE) in DoFn.getAllowedTimestampSkew() causes Overflow/Underflow

[Tobias Feldhaus (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927915#comment-15927915
 ] 

Tobias Feldhaus commented on BEAM-1718:
---

This is the triggering method in a class extending DoFn:
{code}
@Override
public Duration getAllowedTimestampSkew() {
return Duration.millis(Long.MAX_VALUE);
}
{code}

In  the 
[Javadoc|https://github.com/apache/beam/blob/f360f47f9ca4f4054e9fb583c2a0f5dda9ee19ea/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L326]
 it states:
"For infinite skew, return Duration.millis(Long.MAX_VALUE)"

This is the stacktrace:

{code}
java.lang.ArithmeticException: The calculation caused an overflow: 
-9223372036854775 + -9223372036854775807
at org.joda.time.field.FieldUtils.safeAdd(FieldUtils.java:88)
at org.joda.time.chrono.BaseChronology.add(BaseChronology.java:328)
at org.joda.time.Instant.withDurationAdded(Instant.java:174)
at org.joda.time.Instant.withDurationAdded(Instant.java:192)
at org.joda.time.Instant.minus(Instant.java:246)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.checkTimestamp(SimpleDoFnRunner.java:587)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.outputWithTimestamp(SimpleDoFnRunner.java:558)
at 
ch.localsearch.dataintel.logfiles.FrontendPipeline$ReadObjects.processElement(FrontendPipeline.java:138)
at 
ch.localsearch.dataintel.logfiles.FrontendPipeline$ReadObjects$auxiliary$pWS7oKLF.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:199)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:161)
at 
org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
at 
org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
at 
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
at 
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
at 
org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

I think this is a basic error and/or the Javadoc is misleading.

> Returning Duration.millis(Long.MAX_VALUE) in DoFn.getAllowedTimestampSkew() 
> causes Overflow/Underflow
> -
>
> Key: BEAM-1718
> URL: https://issues.apache.org/jira/browse/BEAM-1718
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Tobias Feldhaus
>Assignee: Thomas Groh
>
> Overriding getAllowedTimestampSkew() in DoFn and returning 
> Duration.millis(Long.MAX_VALUE) (as suggested in the JavaDoc for allowing 
> infinite skew) causes an Overflow/Underflow



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Fix shading opt out in io/google-cloud-platform

[amitsela]

Repository: beam
Updated Branches:
  refs/heads/master db005fffb -> b75888e32


Fix shading opt out in io/google-cloud-platform


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dd1b0017
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dd1b0017
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dd1b0017

Branch: refs/heads/master
Commit: dd1b001728375aeecfc577de7ea13761aa37322c
Parents: db005ff
Author: Aviem Zur 
Authored: Thu Mar 16 10:14:36 2017 +0200
Committer: Aviem Zur 
Committed: Thu Mar 16 10:14:36 2017 +0200

--
 sdks/java/io/google-cloud-platform/pom.xml | 31 +
 1 file changed, 11 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dd1b0017/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 44a1731..393db18 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -47,6 +47,17 @@
 
   
 
+  
+org.apache.maven.plugins
+maven-shade-plugin
+
+  
+bundle-and-repackage
+none
+  
+
+  
+
   
   
 org.apache.maven.plugins
@@ -90,26 +101,6 @@
   
 
   
-  
-org.apache.maven.plugins
-maven-shade-plugin
-
-  
-disable-shading
-package
-
-  shade
-
-
-  
-
-  *
-
-  
-
-  
-
-  
 
   
 

[2/2] beam git commit: This closes #2259

[amitsela]

This closes #2259


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b75888e3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b75888e3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b75888e3

Branch: refs/heads/master
Commit: b75888e327acd7d09a63d5a8a7d1331c896b6b75
Parents: db005ff dd1b001
Author: Amit Sela 
Authored: Thu Mar 16 14:26:15 2017 +0200
Committer: Amit Sela 
Committed: Thu Mar 16 14:26:15 2017 +0200

--
 sdks/java/io/google-cloud-platform/pom.xml | 31 +
 1 file changed, 11 insertions(+), 20 deletions(-)
--

[GitHub] beam pull request #2259: Fix shading opt out in io/google-cloud-platform

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2259


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #1258

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #2925

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2231: [BEAM-923] Add localhost option for DatastoreIO (fi...

[iemejia]

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/2231


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-923) Add support for local execution to DatastoreIO using the google cloud emulator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928019#comment-15928019
 ] 

ASF GitHub Bot commented on BEAM-923:
-

Github user iemejia closed the pull request at:

https://github.com/apache/beam/pull/2231


> Add support for local execution to DatastoreIO using the google cloud emulator
> --
>
> Key: BEAM-923
> URL: https://issues.apache.org/jira/browse/BEAM-923
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>Priority: Minor
>
> Adding locahost option enables to connect to a Datastore emulator that is 
> running locally



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1734) Potentially unbox null pointer in AutoValue_BigQueryIO_Write

[Ted Yu (JIRA)]

Ted Yu created BEAM-1734:


 Summary: Potentially unbox null pointer in 
AutoValue_BigQueryIO_Write
 Key: BEAM-1734
 URL: https://issues.apache.org/jira/browse/BEAM-1734
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Ted Yu
Assignee: Daniel Halperin
Priority: Minor


{code}
  if (validate == null) {
missing += " validate";
  }
...
  return new AutoValue_BigQueryIO_Write(
  this.jsonTableRef,
  this.tableRefFunction,
  this.jsonSchema,
  this.createDisposition,
  this.writeDisposition,
  this.tableDescription,
  this.validate,
  this.bigQueryServices);
{code}
If validate is null, AutoValue_BigQueryIO_Write would try to unbox it since the 
parameter is declared as boolean.

There is similar problem in AutoValue_BigQueryIO_Read, around line 166



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1724: [BEAM-991] Flush Datastore writes at max byte batch...

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/beam/pull/1724


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-991) DatastoreIO Write should flush early for large batches

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928508#comment-15928508
 ] 

ASF GitHub Bot commented on BEAM-991:
-

Github user vikkyrk closed the pull request at:

https://github.com/apache/beam/pull/1724


> DatastoreIO Write should flush early for large batches
> --
>
> Key: BEAM-991
> URL: https://issues.apache.org/jira/browse/BEAM-991
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> If entities are large (avg size > 20KB) then the a single batched write (500 
> entities) would exceed the Datastore size limit of a single request (10MB) 
> from https://cloud.google.com/datastore/docs/concepts/limits.
> First reported in: 
> http://stackoverflow.com/questions/40156400/why-does-dataflow-erratically-fail-in-datastore-access



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928540#comment-15928540
 ] 

Kenneth Knowles commented on BEAM-1676:
---

I mean that you can just exclude the surface tests from the versions test 
matrix. It is not useful to run against multiple JDKs, etc. But I don't think 
you should do this, but just fix/workaround the root cause instead.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Spark #1259

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928600#comment-15928600
 ] 

Davor Bonaci commented on BEAM-1676:


I think I wouldn't like to go down that path -- the matrix job is replacing the 
current jobs, and every cell should just run everything. Otherwise, we'll get a 
mess.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928607#comment-15928607
 ] 

Jason Kuster commented on BEAM-1676:


To bring the discussion we had offline yesterday to here, the real problem is 
the space in the directory name. We decided to fix that root cause instead of 
omitting suites which are broken by the space. Our consensus was that was the 
best way forward. Should we change this issue to reflect that? 

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2926

[Apache Jenkins Server]

See 

[1/2] beam git commit: [BEAM-547] [BEAM-1730] Cleanup versions.py

[altay]

Repository: beam
Updated Branches:
  refs/heads/master b75888e32 -> 444edae75


[BEAM-547] [BEAM-1730] Cleanup versions.py


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f11a44c6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f11a44c6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f11a44c6

Branch: refs/heads/master
Commit: f11a44c62b5b4701d11886eaee5cbe0e2dad88dd
Parents: b75888e
Author: Sourabh Bajaj 
Authored: Wed Mar 15 16:14:44 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Mar 16 11:39:46 2017 -0700

--
 sdks/python/MANIFEST.in |  3 ---
 sdks/python/apache_beam/__init__.py |  3 +++
 sdks/python/apache_beam/version.py  | 36 
 3 files changed, 3 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f11a44c6/sdks/python/MANIFEST.in
--
diff --git a/sdks/python/MANIFEST.in b/sdks/python/MANIFEST.in
index 57f684e..ff572ba 100644
--- a/sdks/python/MANIFEST.in
+++ b/sdks/python/MANIFEST.in
@@ -15,7 +15,4 @@
 # limitations under the License.
 #
 
-# This file is used from Python to sync versions
-include pom.xml
-
 include README.md

http://git-wip-us.apache.org/repos/asf/beam/blob/f11a44c6/sdks/python/apache_beam/__init__.py
--
diff --git a/sdks/python/apache_beam/__init__.py 
b/sdks/python/apache_beam/__init__.py
index 77c89ad..5a63fff 100644
--- a/sdks/python/apache_beam/__init__.py
+++ b/sdks/python/apache_beam/__init__.py
@@ -77,6 +77,9 @@ import apache_beam.internal.pickler
 from apache_beam import coders
 from apache_beam import io
 from apache_beam import typehints
+from apache_beam import version
 from apache_beam.pipeline import Pipeline
 from apache_beam.transforms import *
 # pylint: enable=wrong-import-position
+
+__version__ = version.__version__

http://git-wip-us.apache.org/repos/asf/beam/blob/f11a44c6/sdks/python/apache_beam/version.py
--
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index e5d61a1..22d1683 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,40 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-import re
-
-
 __version__ = '0.7.0.dev'
-
-
-# The following utilities are legacy code from the Maven integration;
-# see BEAM-378 for further details.
-
-
-# Reads the actual version from pom.xml file,
-def get_version_from_pom():
-  with open('pom.xml', 'r') as f:
-pom = f.read()
-regex = (r'.*\s*'
- r'[a-z\.]+\s*'
- r'[a-z\-]+\s*'
- r'([0-9a-zA-Z\.\-]+).*')
-pattern = re.compile(str(regex))
-search = pattern.search(pom)
-version = search.group(1)
-version = version.replace("-SNAPSHOT", ".dev")
-return version
-
-
-# Synchronizes apache_beam.__version__ field for later usage
-def sync_version(version):
-  init_path = 'apache_beam/__init__.py'
-  regex = r'^__version__\s*=\s*".*"'
-  with open(init_path, "r") as f:
-lines = f.readlines()
-  with open(init_path, "w") as f:
-for line in lines:
-  if re.search(regex, line):
-f.write(re.sub(regex, '__version__ = "%s"' % version, line))
-  else:
-f.write(line)

[2/2] beam git commit: This closes #2255

[altay]

This closes #2255


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/444edae7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/444edae7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/444edae7

Branch: refs/heads/master
Commit: 444edae75800cc07315613fdf5a25d5dc50a64c7
Parents: b75888e f11a44c
Author: Ahmet Altay 
Authored: Thu Mar 16 11:40:01 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Mar 16 11:40:01 2017 -0700

--
 sdks/python/MANIFEST.in |  3 ---
 sdks/python/apache_beam/__init__.py |  3 +++
 sdks/python/apache_beam/version.py  | 36 
 3 files changed, 3 insertions(+), 39 deletions(-)
--

[GitHub] beam pull request #2255: [BEAM-547] [BEAM-1730] Cleanup versions.py and root...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2255


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-547) Align Python SDK version with Maven

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928616#comment-15928616
 ] 

ASF GitHub Bot commented on BEAM-547:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2255


> Align Python SDK version with Maven
> ---
>
> Key: BEAM-547
> URL: https://issues.apache.org/jira/browse/BEAM-547
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Affects Versions: 0.3.0-incubating
>Reporter: Sergio Fernández
>Assignee: Frances Perry
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In BEAM-378 we've integrated the Python SDK in the main Maven build. 
> Initially I wanted to also align versions, but after discussing it with 
> [~silv...@google.com] we kept that aside for the moment. 
> Closing [PR #537|https://github.com/apache/incubator-beam/pull/537] [~altay] 
> brings the issue back. So it may make sense to revisit that idea.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2220

[altay]

This closes #2220


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a0b8a854
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a0b8a854
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a0b8a854

Branch: refs/heads/master
Commit: a0b8a854df6129e2dbbbd4bad683e53d975f3146
Parents: 444edae 2352c29
Author: Ahmet Altay 
Authored: Thu Mar 16 11:45:03 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Mar 16 11:45:03 2017 -0700

--
 sdks/python/apache_beam/transforms/ptransform_test.py |  2 ++
 sdks/python/apache_beam/transforms/sideinputs_test.py |  2 ++
 sdks/python/run_postcommit.sh | 10 ++
 3 files changed, 10 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-1687] Parallel ValidatesRunner Tests

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 444edae75 -> a0b8a854d


[BEAM-1687] Parallel ValidatesRunner Tests


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2352c291
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2352c291
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2352c291

Branch: refs/heads/master
Commit: 2352c291288957b78dce5bfa3cfbef128fcd56b4
Parents: 444edae
Author: Mark Liu 
Authored: Fri Mar 10 11:06:27 2017 -0800
Committer: Ahmet Altay 
Committed: Thu Mar 16 11:44:49 2017 -0700

--
 sdks/python/apache_beam/transforms/ptransform_test.py |  2 ++
 sdks/python/apache_beam/transforms/sideinputs_test.py |  2 ++
 sdks/python/run_postcommit.sh | 10 ++
 3 files changed, 10 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2352c291/sdks/python/apache_beam/transforms/ptransform_test.py
--
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py 
b/sdks/python/apache_beam/transforms/ptransform_test.py
index f7f157f..37ff2a8 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -45,6 +45,8 @@ from apache_beam.utils.pipeline_options import TypeOptions
 
 
 class PTransformTest(unittest.TestCase):
+  # Enable nose tests running in parallel
+  _multiprocess_can_split_ = True
 
   def assertStartswith(self, msg, prefix):
 self.assertTrue(msg.startswith(prefix),

http://git-wip-us.apache.org/repos/asf/beam/blob/2352c291/sdks/python/apache_beam/transforms/sideinputs_test.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py 
b/sdks/python/apache_beam/transforms/sideinputs_test.py
index 7df870f..9278f4b 100644
--- a/sdks/python/apache_beam/transforms/sideinputs_test.py
+++ b/sdks/python/apache_beam/transforms/sideinputs_test.py
@@ -29,6 +29,8 @@ from apache_beam.transforms.util import assert_that, equal_to
 
 
 class SideInputsTest(unittest.TestCase):
+  # Enable nose tests running in parallel
+  _multiprocess_can_split_ = True
 
   def create_pipeline(self):
 return TestPipeline()

http://git-wip-us.apache.org/repos/asf/beam/blob/2352c291/sdks/python/run_postcommit.sh
--
diff --git a/sdks/python/run_postcommit.sh b/sdks/python/run_postcommit.sh
index 1c0bbb2..4d17942 100755
--- a/sdks/python/run_postcommit.sh
+++ b/sdks/python/run_postcommit.sh
@@ -61,7 +61,6 @@ GCS_LOCATION=gs://temp-storage-for-end-to-end-tests
 
 # Job name needs to be unique
 JOBNAME_E2E_WC=py-wordcount-`date +%s`
-JOBNAME_VR_TEST=py-validatesrunner-`date +%s`
 
 PROJECT=apache-beam-testing
 
@@ -77,13 +76,15 @@ echo "mock" >> postcommit_requirements.txt
 # Run ValidatesRunner tests on Google Cloud Dataflow service
 echo ">>> RUNNING DATAFLOW RUNNER VALIDATESRUNNER TESTS"
 python setup.py nosetests \
-  -a ValidatesRunner --test-pipeline-options=" \
+  -a ValidatesRunner \
+  --processes=4 \
+  --process-timeout=600 \
+  --test-pipeline-options=" \
 --runner=TestDataflowRunner \
 --project=$PROJECT \
 --staging_location=$GCS_LOCATION/staging-validatesrunner-test \
 --temp_location=$GCS_LOCATION/temp-validatesrunner-test \
 --sdk_location=$SDK_LOCATION \
---job_name=$JOBNAME_VR_TEST \
 --requirements_file=postcommit_requirements.txt \
 --num_workers=1"
 
@@ -91,7 +92,8 @@ python setup.py nosetests \
 # and validate job that finishes successfully.
 echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
 python setup.py nosetests \
-  -a IT --test-pipeline-options=" \
+  -a IT \
+  --test-pipeline-options=" \
 --runner=TestDataflowRunner \
 --project=$PROJECT \
 --staging_location=$GCS_LOCATION/staging-wordcount \

[jira] [Commented] (BEAM-1687) Reduce Total Time of Running Python ValidatesRunner Tests in Postcommit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928621#comment-15928621
 ] 

ASF GitHub Bot commented on BEAM-1687:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2220


> Reduce Total Time of Running Python ValidatesRunner Tests in Postcommit
> ---
>
> Key: BEAM-1687
> URL: https://issues.apache.org/jira/browse/BEAM-1687
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Takes 1h+ for 14 ValidatesRunner tests that running in Python postcommit, 
> which can be paralleled by taking advantage of Nose and Jenkins multi-core 
> machine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2220: [BEAM-1687] Parallel Python ValidatesRunner Tests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2220


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1727) Add setForNowAlign(period, offset) to Timer

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928624#comment-15928624
 ] 

Kenneth Knowles commented on BEAM-1727:
---

It seems like a nice idea. For event time timers it could exceed the allowed 
time.

> Add setForNowAlign(period, offset) to Timer
> ---
>
> Key: BEAM-1727
> URL: https://issues.apache.org/jira/browse/BEAM-1727
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Jingsong Lee
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1735) Retry 403:rateLimitExceeded in GCS

[Rafael Fernandez (JIRA)]

Rafael Fernandez created BEAM-1735:
--

 Summary: Retry 403:rateLimitExceeded in GCS
 Key: BEAM-1735
 URL: https://issues.apache.org/jira/browse/BEAM-1735
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Rafael Fernandez
Assignee: Daniel Halperin


The GCS documentation [1] states that rateLimitExceeded, a 403 error, should be 
retried exponentially. We currently do not retry it.

[1] https://cloud.google.com/storage/docs/json_api/v1/status-codes 




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Spark #1260

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2927

[Apache Jenkins Server]

See 

[GitHub] beam-site pull request #180: Changes to web site related to recently release...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/180

Changes to web site related to recently released Python SDK

R: @davorbonaci 
cc: @sb2nov 

I will work on release guide changes on a separate PR.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/180.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #180


commit 5111853bc8fc6890633a377972d28cf9a72569d1
Author: Ahmet Altay 
Date:   2017-03-16T19:21:08Z

Changes to web site related to recently released Python SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2928

[Apache Jenkins Server]

See 

[3/3] beam-site git commit: This closes #180

[davor]

This closes #180


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/82564f9d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/82564f9d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/82564f9d

Branch: refs/heads/asf-site
Commit: 82564f9defb3faa803a016a484340a87182aa624
Parents: 4b64a03 0dd4a1e
Author: Davor Bonaci 
Authored: Thu Mar 16 12:34:39 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 12:34:39 2017 -0700

--
 content/contribute/testing/index.html|  5 ++---
 content/documentation/index.html |  2 +-
 content/get-started/beam-overview/index.html |  2 +-
 content/get-started/downloads/index.html | 12 ++--
 content/get-started/quickstart-py/index.html | 20 ++--
 src/contribute/testing.md|  5 ++---
 src/documentation/index.md   |  2 +-
 src/get-started/beam-overview.md |  2 +-
 src/get-started/downloads.md | 10 --
 src/get-started/quickstart-py.md | 13 ++---
 10 files changed, 30 insertions(+), 43 deletions(-)
--

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/0dd4a1e8
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/0dd4a1e8
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/0dd4a1e8

Branch: refs/heads/asf-site
Commit: 0dd4a1e8dc747150297d669ccbcdce8dfeb4e309
Parents: 5111853
Author: Davor Bonaci 
Authored: Thu Mar 16 12:34:38 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 12:34:38 2017 -0700

--
 content/contribute/testing/index.html|  5 ++---
 content/documentation/index.html |  2 +-
 content/get-started/beam-overview/index.html |  2 +-
 content/get-started/downloads/index.html | 12 ++--
 content/get-started/quickstart-py/index.html | 20 ++--
 5 files changed, 16 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd4a1e8/content/contribute/testing/index.html
--
diff --git a/content/contribute/testing/index.html 
b/content/contribute/testing/index.html
index 6fb76df..fb004c9 100644
--- a/content/contribute/testing/index.html
+++ b/content/contribute/testing/index.html
@@ -413,9 +413,8 @@ details on those testing types.
 
 Python SDK
 
-The Python SDK is currently under development on a feature branch. We have 
initial
-postcommit tests by a Jenkins build; precommit testing and a full testing
-matrix will be coming soon.
+The Python SDK has postcommit tests by a Jenkins build; precommit testing 
and a
+full testing matrix will be coming soon.
 
 Testing Scenarios
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd4a1e8/content/documentation/index.html
--
diff --git a/content/documentation/index.html b/content/documentation/index.html
index e998965..af5afb0 100644
--- a/content/documentation/index.html
+++ b/content/documentation/index.html
@@ -177,7 +177,7 @@
 
 
   Java SDK
-  [Under Development] Python SDK
+  Python SDK
 
 
 Runners

http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd4a1e8/content/get-started/beam-overview/index.html
--
diff --git a/content/get-started/beam-overview/index.html 
b/content/get-started/beam-overview/index.html
index f45903b..7a2b1f1 100644
--- a/content/get-started/beam-overview/index.html
+++ b/content/get-started/beam-overview/index.html
@@ -173,7 +173,7 @@
 
 
   Python
-  Coming Soon
+  Active Development
 
 
   Other

http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd4a1e8/content/get-started/downloads/index.html
--
diff --git a/content/get-started/downloads/index.html 
b/content/get-started/downloads/index.html
index 4053000..3438ee7 100644
--- a/content/get-started/downloads/index.html
+++ b/content/get-started/downloads/index.html
@@ -152,8 +152,9 @@
   
 Apache Beam Downloads
 
-The easiest way to use Apache Beam is via one of the released versions in 
the
-https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22";>Maven
 Central Repository.
+The easiest way to use Apache Beam is via one of the released versions in a 
central repository.
+Java SDK is available on https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22";>Maven
 Central Repository,
+and Python SDK is available on https://pypi.python.org/pypi/apache-beam";>PyPI.
 
 For example, if you are developing using Maven and want to use the SDK for
 Java with the DirectRunner, add the 
following dependencies to your
@@ -173,6 +174,13 @@ Java with the DirectRunner, add the follo
 
 
 
+Similarly in Python, if you are using PyPI and want to use the SDK for 
Python with
+DirectRunner, add the following 
requirement to your setup.py file:
+
+apache-beam==0.6.0
+
+
+
 Additionally, you may want to depend on additional SDK modules, such as IO
 connectors or other extensions, and additional runners to execute your pipeline
 at scale.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/0dd4a1e8/content/get-started/quickstart-py/index.html
--
diff --git a/content/get-started/quickstart-py/index.html 
b/content/get-started/quickstart-py/index.html
index b143d54..153eac6 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -220,24 +220,8 @@ environment’s directories.
 
 Download and install
 
-
-  
-Clone the Apache Beam repo from GitHub:
-  git clone 
https://github.com/apache/beam.git
-  
-  
-Navigate to the python directory:
-  cd beam/sdks/python/
-  
-  
-Create the Apache Beam Python SDK installation packa

[1/3] beam-site git commit: Changes to web site related to recently released Python SDK

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 4b64a0306 -> 82564f9de


Changes to web site related to recently released Python SDK


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/5111853b
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/5111853b
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/5111853b

Branch: refs/heads/asf-site
Commit: 5111853bc8fc6890633a377972d28cf9a72569d1
Parents: 4b64a03
Author: Ahmet Altay 
Authored: Thu Mar 16 12:21:08 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Mar 16 12:21:08 2017 -0700

--
 src/contribute/testing.md|  5 ++---
 src/documentation/index.md   |  2 +-
 src/get-started/beam-overview.md |  2 +-
 src/get-started/downloads.md | 10 --
 src/get-started/quickstart-py.md | 13 ++---
 5 files changed, 14 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/5111853b/src/contribute/testing.md
--
diff --git a/src/contribute/testing.md b/src/contribute/testing.md
index c3439a3..ee40a88 100644
--- a/src/contribute/testing.md
+++ b/src/contribute/testing.md
@@ -243,9 +243,8 @@ details on those testing types.
 
 ### Python SDK
 
-The Python SDK is currently under development on a feature branch. We have 
initial
-postcommit tests by a Jenkins build; precommit testing and a full testing
-matrix will be coming soon.
+The Python SDK has postcommit tests by a Jenkins build; precommit testing and a
+full testing matrix will be coming soon.
 
 ## Testing Scenarios
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5111853b/src/documentation/index.md
--
diff --git a/src/documentation/index.md b/src/documentation/index.md
index 630b2c4..03fb43b 100644
--- a/src/documentation/index.md
+++ b/src/documentation/index.md
@@ -29,7 +29,7 @@ Learn about the Beam Programming Model and the concepts 
common to all Beam SDKs
 Find status and reference information on all of the available Beam SDKs.
 
 * [Java SDK]({{ site.baseurl }}/documentation/sdks/java/) 
-* _[Under Development]_ [Python SDK]({{ site.baseurl 
}}/documentation/sdks/python/)
+* [Python SDK]({{ site.baseurl }}/documentation/sdks/python/)
 
 ## Runners
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5111853b/src/get-started/beam-overview.md
--
diff --git a/src/get-started/beam-overview.md b/src/get-started/beam-overview.md
index 6796c2e..152a08d 100644
--- a/src/get-started/beam-overview.md
+++ b/src/get-started/beam-overview.md
@@ -30,7 +30,7 @@ Beam currently supports the following language-specific SDKs:
 
 
   Python
-  Coming Soon
+  Active Development
 
 
   Other

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5111853b/src/get-started/downloads.md
--
diff --git a/src/get-started/downloads.md b/src/get-started/downloads.md
index 00a8e92..6a788ce 100644
--- a/src/get-started/downloads.md
+++ b/src/get-started/downloads.md
@@ -10,8 +10,9 @@ redirect_from:
 
 # Apache Beam Downloads
 
-The easiest way to use Apache Beam is via one of the released versions in the
-[Maven Central 
Repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22).
+The easiest way to use Apache Beam is via one of the released versions in a 
central repository.
+Java SDK is available on [Maven Central 
Repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22),
+and Python SDK is available on 
[PyPI](https://pypi.python.org/pypi/apache-beam).
 
 For example, if you are developing using Maven and want to use the SDK for
 Java with the `DirectRunner`, add the following dependencies to your
@@ -29,6 +30,11 @@ Java with the `DirectRunner`, add the following dependencies 
to your
   runtime
 
 
+Similarly in Python, if you are using PyPI and want to use the SDK for Python 
with
+`DirectRunner`, add the following requirement to your `setup.py` file:
+
+apache-beam=={{ site.release_latest }}
+
 Additionally, you may want to depend on additional SDK modules, such as IO
 connectors or other extensions, and additional runners to execute your pipeline
 at scale.

http://git-wip-us.apache.org/repos/asf/beam-site/blob/5111853b/src/get-started/quickstart-py.md
--
diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index 31a4676..c04bf09 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -62,17 +62,8 @@ For instructions using other shells, see the [virtualenv 
documentatio

[GitHub] beam-site pull request #180: Changes to web site related to recently release...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/180


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #181: Typo fix

[meunierd]

GitHub user meunierd opened a pull request:

https://github.com/apache/beam-site/pull/181

Typo fix

`s/paraemters/parameters/g`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/meunierd/beam-site fix-typo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #181


commit 6ec94f71b63d98dd7364686fe7a14fcba3d99592
Author: Devon Meunier 
Date:   2017-03-16T19:51:04Z

Typo fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam-site git commit: Typo fix

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 82564f9de -> 2ccd76628


Typo fix


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/6ec94f71
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/6ec94f71
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/6ec94f71

Branch: refs/heads/asf-site
Commit: 6ec94f71b63d98dd7364686fe7a14fcba3d99592
Parents: 82564f9
Author: Devon Meunier 
Authored: Thu Mar 16 15:51:04 2017 -0400
Committer: Devon Meunier 
Committed: Thu Mar 16 15:51:04 2017 -0400

--
 content/documentation/programming-guide/index.html | 2 +-
 src/documentation/programming-guide.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/6ec94f71/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index cce75e5..707c525 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -511,7 +511,7 @@
   Note: When you create your DoFn, be mindful of the General Requirements for Writing User Code for 
Beam Transforms and ensure that your code follows them.
 
 
-A DoFn 
processes one element at a time from the input PCollection. When you create a subclass of 
DoFn, you’ll need to provide type 
paraemters that match the types of the input and output elements. If your DoFn processes incoming String elements and produces Integer elements for the output collection 
(like our previous example, ComputeWordLengthFn), your class declaration 
would look like this:
+A DoFn 
processes one element at a time from the input PCollection. When you create a subclass of 
DoFn, you’ll need to provide type 
parameters that match the types of the input and output elements. If your DoFn processes incoming String elements and produces Integer elements for the output collection 
(like our previous example, ComputeWordLengthFn), your class declaration 
would look like this:
 
 static class ComputeWordLengthFn extends DoFn { ... }
 

http://git-wip-us.apache.org/repos/asf/beam-site/blob/6ec94f71/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 81a03fb..65a3062 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -328,7 +328,7 @@ The `DoFn` object that you pass to `ParDo` contains the 
processing logic that ge
 > **Note:** When you create your `DoFn`, be mindful of the [General 
 > Requirements for Writing User Code for Beam 
 > Transforms](#transforms-usercodereqs) and ensure that your code follows them.
 
 {:.language-java}
-A `DoFn` processes one element at a time from the input `PCollection`. When 
you create a subclass of `DoFn`, you'll need to provide type paraemters that 
match the types of the input and output elements. If your `DoFn` processes 
incoming `String` elements and produces `Integer` elements for the output 
collection (like our previous example, `ComputeWordLengthFn`), your class 
declaration would look like this:
+A `DoFn` processes one element at a time from the input `PCollection`. When 
you create a subclass of `DoFn`, you'll need to provide type parameters that 
match the types of the input and output elements. If your `DoFn` processes 
incoming `String` elements and produces `Integer` elements for the output 
collection (like our previous example, `ComputeWordLengthFn`), your class 
declaration would look like this:
 
 ```java
 static class ComputeWordLengthFn extends DoFn { ... }

[GitHub] beam-site pull request #181: Typo fix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/181


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam-site git commit: This closes #181

[davor]

This closes #181


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/2ccd7662
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/2ccd7662
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/2ccd7662

Branch: refs/heads/asf-site
Commit: 2ccd76628d5533acab599cd496a43ad9976737c7
Parents: 82564f9 6ec94f7
Author: Davor Bonaci 
Authored: Thu Mar 16 13:09:06 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 13:09:06 2017 -0700

--
 content/documentation/programming-guide/index.html | 2 +-
 src/documentation/programming-guide.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Created] (BEAM-1736) @DefaultCoder annotation should work with Create

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1736:
-

 Summary: @DefaultCoder annotation should work with Create
 Key: BEAM-1736
 URL: https://issues.apache.org/jira/browse/BEAM-1736
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


>From u...@beam.apache.org:

"My class is declared as follows:

@DefaultCoder(SerializableCoder.class) public class Data implements Serializable {

and it fails like this:

Caused by: org.apache.beam.sdk.coders.CannotProvideCoderException: Cannot 
provide coder based on value with class my.project.Data: No CoderFactory has 
been registered for the class."

It seems like this should work, though Create's value-based coder inference has 
more limitations and safety issues than usual coder inference. To investigate.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1439) Beam Example(s) exploring public document datasets

[khalid bin huda (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928932#comment-15928932
 ] 

khalid bin huda commented on BEAM-1439:
---

Hi, I'm Khalid Bin Huda, I am a Final year undergraduate from the Department of 
Computer Science (University of Karachi). I have programming experience with C 
,Java ,R and love to work on Project related to Data-mining  or Machine 
Learning.  I would like do this project for GSoC 2017. I would like to 
contribute in this Project.

> Beam Example(s) exploring public document datasets
> --
>
> Key: BEAM-1439
> URL: https://issues.apache.org/jira/browse/BEAM-1439
> Project: Beam
>  Issue Type: Wish
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Minor
>  Labels: gsoc2017, java, mentor, python
>
> In Beam, we have examples illustrating counting the occurrences of words and 
> performing a basic TF-IDF analysis on the works of Shakespeare (or whatever 
> you point it at). It would be even cooler to do these analyses, and more, on 
> a much larger data set that is really the subject of current investigations.
> In chatting with professors at the University of Washington, I've learned 
> that scholars of many fields would really like to explore new and highly 
> customized ways of processing the growing body of publicly-available 
> scholarly documents, such as PubMed Central. Queries like "show me documents 
> where chemical compounds X and Y were both used in the 'method' section"
> So I propose a Google Summer of Code project wherein a student writes some 
> large-scale Beam pipelines to perform analyses such as term frequency, bigram 
> frequency, etc.
> Skills required:
>  - Java or Python
>  - (nice to have) Working through the Beam getting started materials



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2260: Use underscore instead of hypen in Datastore kind n...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/2260

Use underscore instead of hypen in Datastore kind name.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam ds_test_Fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2260.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2260


commit 9bf2ea92ce9277fff9c978b2adf0c3a529ef8cac
Author: Vikas Kedigehalli 
Date:   2017-03-16T21:45:44Z

Use underscore instead of hypen in Datastore kind name.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2261: Remove duplicate build-helper-maven-plugin

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2261

Remove duplicate build-helper-maven-plugin

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam 
duplicate-helper-plugin-definition

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2261.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2261


commit eb964e6fe973a2b2dc627f33598e0f9867c889e6
Author: Aviem Zur 
Date:   2017-03-16T21:46:23Z

Remove duplicate build-helper-maven-plugin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929001#comment-15929001
 ] 

Mark Liu commented on BEAM-1676:


I built project in a directory that contains space on my Linux machine and 
found there are several sub-projects (including sdks-java-maven-archetype and 
sdks-python) are failed to build due to the directory-with-space issue, and 
none of them is easy fix for me. IMHO, I suggest to have a general JIRA to keep 
track all bugs from this issue. This one (BEAM-1676) can be converted to a 
sub-task. 

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed When Directory Contains Space

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Summary: SdkCoreApiSurfaceTest Failed When Directory Contains Space  (was: 
SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins)

> SdkCoreApiSurfaceTest Failed When Directory Contains Space
> --
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed When Directory Contains Space

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Description: 
Test failed if build directory contains space. For example: "~/dir with 
space/beam/..."
The failure happened on Jenkins and can be reproduced from local.

error is:
{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link from Jenkins:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/

One of the Jenkins job uses "JDK 1.8 (latest)" which is also part of project 
directory.

  was:
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in good 
health. And the maven command in version test is the same as 
beam_PostCommit_Java_MavenInstall.
Any ideas?


> SdkCoreApiSurfaceTest Fail

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed When Directory Contains Space

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Description: 
Test failed if build directory contains space. For example: "~/dir with 
space/beam/..."
The failure happened on Jenkins and can be reproduced from local.

GcpApiSurfaceTest may have same problem.

error is:
{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link from Jenkins:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/

One of the Jenkins job uses "JDK 1.8 (latest)" which is also part of project 
directory.

  was:
Test failed if build directory contains space. For example: "~/dir with 
space/beam/..."
The failure happened on Jenkins and can be reproduced from local.

error is:
{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link from Jenkins:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/

One of the Jenkins job uses "JDK 1.8 (latest)" which is also part of project 
directory.


> SdkCoreApiSurfaceTest Failed When Directory Contains Space
> --
>
> 

[jira] [Assigned] (BEAM-1676) SdkCoreApiSurfaceTest Failed When Directory Contains Space

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu reassigned BEAM-1676:
--

Assignee: Stas Levin  (was: Mark Liu)

> SdkCoreApiSurfaceTest Failed When Directory Contains Space
> --
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Stas Levin
>
> Test failed if build directory contains space. For example: "~/dir with 
> space/beam/..."
> The failure happened on Jenkins and can be reproduced from local.
> GcpApiSurfaceTest may have same problem.
> error is:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link from Jenkins:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> One of the Jenkins job uses "JDK 1.8 (latest)" which is also part of project 
> directory.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2929

[Apache Jenkins Server]

See 

[1/2] beam git commit: Use underscore in Datastore V1TestOptions kind name

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a0b8a854d -> 31473a7e7


Use underscore in Datastore V1TestOptions kind name


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/776006d1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/776006d1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/776006d1

Branch: refs/heads/master
Commit: 776006d1562a8eb05b6aa14d5d93347d586cb668
Parents: a0b8a85
Author: Vikas Kedigehalli 
Authored: Thu Mar 16 14:45:44 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 15:31:20 2017 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/776006d1/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java
index a77d669..efe1ef1 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java
@@ -28,7 +28,7 @@ import org.apache.beam.sdk.testing.TestPipelineOptions;
  */
 public interface V1TestOptions extends TestPipelineOptions {
   @Description("Datastore Entity kind")
-  @Default.String("beam-test")
+  @Default.String("beam_test")
   String getKind();
   void setKind(String value);
 

[2/2] beam git commit: This closes #2260

[tgroh]

This closes #2260


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/31473a7e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/31473a7e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/31473a7e

Branch: refs/heads/master
Commit: 31473a7e75f15eb784b929f194d19d959f8bdeb1
Parents: a0b8a85 776006d
Author: Thomas Groh 
Authored: Thu Mar 16 15:31:41 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 15:31:41 2017 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/datastore/V1TestOptions.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #2260: Use underscore instead of hypen in Datastore kind n...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2260


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove duplicate build-helper-maven-plugin

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 31473a7e7 -> a4142a3ca


Remove duplicate build-helper-maven-plugin


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/784a6cd5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/784a6cd5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/784a6cd5

Branch: refs/heads/master
Commit: 784a6cd55fc9307d448f26538691feec227267b1
Parents: 31473a7
Author: Aviem Zur 
Authored: Thu Mar 16 23:46:23 2017 +0200
Committer: Davor Bonaci 
Committed: Thu Mar 16 15:40:43 2017 -0700

--
 pom.xml | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/784a6cd5/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 3df7317..cec 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1169,13 +1169,7 @@
   versions-maven-plugin
   2.3
 
-   
-
-  org.codehaus.mojo
-  build-helper-maven-plugin
-  1.12
-
-   
+
 
   org.codehaus.mojo
   exec-maven-plugin

[2/2] beam git commit: This closes #2261

[davor]

This closes #2261


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a4142a3c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a4142a3c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a4142a3c

Branch: refs/heads/master
Commit: a4142a3ca53b11d4d1554ee1e63492a9b937a579
Parents: 31473a7 784a6cd
Author: Davor Bonaci 
Authored: Thu Mar 16 15:40:47 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 15:40:47 2017 -0700

--
 pom.xml | 8 +---
 1 file changed, 1 insertion(+), 7 deletions(-)
--

[GitHub] beam pull request #2261: Remove duplicate build-helper-maven-plugin

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2261


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Populate Display Data in Window.Assign

[tgroh]

Populate Display Data in Window.Assign

This populates the display data by delegating back to the enclosing
Window.Bound transform.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b5641d6f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b5641d6f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b5641d6f

Branch: refs/heads/master
Commit: b5641d6f72ec34a4ed001bef2a6bac0ebc4abcdc
Parents: a4142a3
Author: Thomas Groh 
Authored: Wed Mar 15 16:52:48 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 15:45:14 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 14 +++--
 .../sdk/transforms/windowing/WindowTest.java| 54 
 2 files changed, 65 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b5641d6f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
index 94870ff..373bbbd 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Window.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.transforms.windowing;
 
+import com.google.common.annotations.VisibleForTesting;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
@@ -479,7 +480,7 @@ public class Window {
 .setWindowingStrategyInternal(outputStrategy);
   } else {
 // This is the AssignWindows primitive
-return input.apply(new Assign(outputStrategy));
+return input.apply(new Assign<>(this, outputStrategy));
   }
 }
 
@@ -532,11 +533,11 @@ public class Window {
 }
   }
 
-
   /**
* A Primitive {@link PTransform} that assigns windows to elements based on 
a {@link WindowFn}.
*/
   public static class Assign extends PTransform, 
PCollection> {
+private final Bound original;
 private final WindowingStrategy updatedStrategy;
 
 /**
@@ -544,7 +545,9 @@ public class Window {
  * WindowingStrategy}. Windows should be assigned using the {@link 
WindowFn} returned by
  * {@link #getWindowFn()}.
  */
-private Assign(WindowingStrategy updatedStrategy) {
+@VisibleForTesting
+Assign(Bound original, WindowingStrategy updatedStrategy) {
+  this.original = original;
   this.updatedStrategy = updatedStrategy;
 }
 
@@ -554,6 +557,11 @@ public class Window {
   input.getPipeline(), updatedStrategy, input.isBounded());
 }
 
+@Override
+public void populateDisplayData(DisplayData.Builder builder) {
+  original.populateDisplayData(builder);
+}
+
 public WindowFn getWindowFn() {
   return updatedStrategy.getWindowFn();
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/b5641d6f/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
index 1101ebc..70635e9 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/WindowTest.java
@@ -30,6 +30,7 @@ import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 import static org.mockito.Mockito.when;
 
+import com.google.common.collect.Iterables;
 import java.io.Serializable;
 import java.util.Collection;
 import java.util.Collections;
@@ -50,6 +51,8 @@ import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.transforms.display.DisplayDataEvaluator;
+import org.apache.beam.sdk.transforms.windowing.Window.Bound;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.WindowingStrategy.AccumulationMode;
 import org.apache.beam.sdk.values.KV;
@@ -446,6 +449,57 @@ public class WindowTest implements Serializable {
   }
 
   @Test
+  @Category(RunnableOnService.class)
+  public void testPrimitiveDisplayData() {
+FixedWindows windowFn = FixedWindows.of(Duration.standardHours(5));
+AfterWatermark.FromEndOfWindow triggerBuilder = 
AfterWatermark.pastEndOfWindow();
+Duratio

[1/2] beam git commit: This closes #2258

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a4142a3ca -> 960f3e660


This closes #2258


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/960f3e66
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/960f3e66
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/960f3e66

Branch: refs/heads/master
Commit: 960f3e66030e33af9bdcec36b5f57ab9169fda39
Parents: a4142a3 b5641d6
Author: Thomas Groh 
Authored: Thu Mar 16 15:45:14 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 15:45:14 2017 -0700

--
 .../beam/sdk/transforms/windowing/Window.java   | 14 +++--
 .../sdk/transforms/windowing/WindowTest.java| 54 
 2 files changed, 65 insertions(+), 3 deletions(-)
--

[GitHub] beam pull request #2258: [BEAM-1732] Populate Display Data in Window.Assign

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2258


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1732) Window.Assign does not properly populate DisplayData of the enclosing Window transform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929072#comment-15929072
 ] 

ASF GitHub Bot commented on BEAM-1732:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2258


> Window.Assign does not properly populate DisplayData of the enclosing Window 
> transform
> --
>
> Key: BEAM-1732
> URL: https://issues.apache.org/jira/browse/BEAM-1732
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This should populate the same display data as the enclosing Window.Bound



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #182: Blog post for 0.6.0 release with python sdk

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/182

Blog post for 0.6.0 release with python sdk

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/182.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #182


commit a7b14e4cb7412c6bfe8297c41c1c2030615e2f12
Author: Ahmet Altay 
Date:   2017-03-16T22:43:27Z

Blog post for 0.6.0 release with python sdk




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1737) Interpreting a Single-output ParDo as a Multi-output ParDo with a single output causes serialization failures

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1737:
-

 Summary: Interpreting a Single-output ParDo as a Multi-output 
ParDo with a single output causes serialization failures
 Key: BEAM-1737
 URL: https://issues.apache.org/jira/browse/BEAM-1737
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Thomas Groh
Priority: Minor


This is the cause of having a separate path and implementation for 
single-output ParDos, even though both go through the same translator.

Partial stacktrace:

Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.946 sec <<< 
FAILURE! - in 
org.apache.beam.runners.spark.translation.streaming.CreateStreamTest
 [8233/41535]
testLateDataAccumulating(org.apache.beam.runners.spark.translation.streaming.CreateStreamTest)
  Time elapsed: 3.593 sec  <<< ERROR!
java.lang.RuntimeException: 
java.io.NotSerializableException: DStream checkpointing has been enabled but 
the DStreams with their functions are not serializable
org.apache.beam.runners.spark.translation.EvaluationContext
Serialization stack:
- object not serializable (class: 
org.apache.beam.runners.spark.translation.EvaluationContext, value: 
org.apache.beam.runners.spark.translation.EvaluationContext@a8c55d7)
- field (class: 
org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
 name: val$context, type: class 
org.apache.beam.runners.spark.translation.EvaluationContext)
- object (class 
org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1,
 
org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator$8$1@44f50940)
- field (class: 
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1, 
name: transformFunc$3, type: interface 
org.apache.spark.api.java.function.Function)
- object (class 
org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$transformToPair$1, 
)
- field (class: 
org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
 name: cleanedF$2, type: interface scala.Function1)
- object (class 
org.apache.spark.streaming.dstream.DStream$$anonfun$transform$1$$anonfun$apply$21,
 )
- field (class: 
org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
name: cleanedF$3, type: interface scala.Function2)
- object (class 
org.apache.spark.streaming.dstream.DStream$$anonfun$transform$2$$anonfun$5, 
)
- writeObject data (class: 
org.apache.spark.streaming.dstream.DStreamCheckpointData)
- object (class 
org.apache.spark.streaming.dstream.DStreamCheckpointData, [
0 checkpoint files 

])

at 
org.apache.beam.runners.spark.SparkPipelineResult.runtimeExceptionFrom(SparkPipelineResult.java:60)
at 
org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:77)
at 
org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
at 
org.apache.beam.runners.spark.TestSparkRunner.awaitWatermarksOrTimeout(TestSparkRunner.java:195)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:127)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:82)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1732) Window.Assign does not properly populate DisplayData of the enclosing Window transform

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1732.
---
   Resolution: Fixed
Fix Version/s: First stable release

> Window.Assign does not properly populate DisplayData of the enclosing Window 
> transform
> --
>
> Key: BEAM-1732
> URL: https://issues.apache.org/jira/browse/BEAM-1732
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> This should populate the same display data as the enclosing Window.Bound



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1645) Display data not populated on Window.Assign

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929093#comment-15929093
 ] 

Thomas Groh commented on BEAM-1645:
---

https://github.com/apache/beam/pull/2258 fixes this for window.assign in the 
present day. With the current implementation we can't add display data for a 
Window that only updates the windowing strategy, as it's implemented in terms 
of a flatten and we can't propagate the display data for the Window.Into to the 
Flatten

> Display data not populated on Window.Assign
> ---
>
> Key: BEAM-1645
> URL: https://issues.apache.org/jira/browse/BEAM-1645
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Ben Chambers
> Fix For: First stable release
>
>
> In 
> https://github.com/apache/beam/commit/eaf9b9b36dec1cc421335b27f225663ce42d0cca
>  the display data was put only on the composite, where no runner actually 
> locates it today.
> As a mitigation we can populate it on the {{Window.Assign}} transform, though 
> the DataflowRunner should likely override this to surface it at the top level.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/3] beam-site git commit: Blog post for 0.6.0 release with python sdk

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 2ccd76628 -> 3917f6e3d


Blog post for 0.6.0 release with python sdk


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/be9e207d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/be9e207d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/be9e207d

Branch: refs/heads/asf-site
Commit: be9e207ddd881cf6beda26b811b20ba878def648
Parents: 2ccd766
Author: Ahmet Altay 
Authored: Thu Mar 16 15:43:27 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 16:20:42 2017 -0700

--
 src/_data/authors.yml   | 18 +++---
 src/_posts/2017-03-16-python-sdk-release.md | 72 
 2 files changed, 83 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/be9e207d/src/_data/authors.yml
--
diff --git a/src/_data/authors.yml b/src/_data/authors.yml
index e4aa332..f66a69a 100644
--- a/src/_data/authors.yml
+++ b/src/_data/authors.yml
@@ -2,6 +2,9 @@ aljoscha:
 name: Aljoscha Krettek
 email: aljos...@apache.org
 twitter: aljoscha
+altay:
+name: Ahmet Altay
+email: al...@apache.org
 davor:
 name: Davor Bonaci
 email: da...@apache.org
@@ -18,6 +21,13 @@ jamesmalone:
 name: James Malone
 email: jamesmal...@apache.org
 twitter: chimerasaurus
+jesseanderson:
+name: Jesse Anderson
+twitter: jessetanderson
+klk:
+name: Kenneth Knowles
+email: k...@apache.org
+twitter: KennKnowles
 robertwb:
 name: Robert Bradshaw
 email: rober...@apache.org
@@ -29,14 +39,8 @@ takidau:
 tgroh:
 name: Thomas Groh
 email: tg...@google.com
-jesseanderson:
-name: Jesse Anderson
-twitter: jessetanderson
 thw:
 name: Thomas Weise
 email: t...@apache.org
 twitter: thweise
-klk:
-name: Kenneth Knowles
-email: k...@apache.org
-twitter: KennKnowles
+

http://git-wip-us.apache.org/repos/asf/beam-site/blob/be9e207d/src/_posts/2017-03-16-python-sdk-release.md
--
diff --git a/src/_posts/2017-03-16-python-sdk-release.md 
b/src/_posts/2017-03-16-python-sdk-release.md
new file mode 100644
index 000..72f5209
--- /dev/null
+++ b/src/_posts/2017-03-16-python-sdk-release.md
@@ -0,0 +1,72 @@
+---
+layout: post
+title:  "Python SDK released in Apache Beam 0.6.0"
+date:   2017-03-16 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+  - altay
+---
+
+Apache Beam’s latest release, version [0.6.0]({{ site.baseurl 
}}/get-started/downloads/), introduces a new SDK -- this time, for the Python 
programming language. The Python SDK joins the Java SDK as the second 
implementation of the Beam programming model.
+
+
+
+The Python SDK incorporates all of the main concepts of the Beam model, 
including ParDo, GroupByKey, Windowing, and others. It features extensible IO 
APIs for writing bounded sources and sinks, and provides built-in 
implementation for reading and writing Text, Avro, and TensorFlow record files, 
as well as connectors to Google BigQuery and Google Cloud Datastore.
+
+There are two runners capable of executing pipelines written with the Python 
SDK today: [Direct Runner]({{ site.baseurl }}/documentation/runners/direct/) 
and [Dataflow Runner]({{ site.baseurl }}/documentation/runners/dataflow/), both 
of which are currently limited to batch execution only. Upcoming features will 
shortly bring the benefits of the Python SDK to additional runners.
+
+ Try the Apache Beam Python SDK
+
+If you would like to try out the Python SDK, a good place to start is the 
[Quickstart]({{ site.baseurl }}/get-started/quickstart-py/). After that, you 
can take a look at additional 
[examples](https://github.com/apache/beam/tree/v0.6.0/sdks/python/apache_beam/examples),
 and deep dive into the [API reference]({{ site.baseurl 
}}/documentation/sdks/pydoc/).
+
+Let’s take a look at a quick example together. First, install the 
`apache-beam` package from PyPI and start your Python interpreter.
+
+```
+$ pip install apache-beam
+$ python
+```
+
+We will harness the power of Apache Beam to estimate Pi in honor of the 
recently passed Pi Day.
+
+```
+import random
+import apache_beam as beam
+
+def run_trials(count):
+  """Throw darts into unit square and count how many fall into unit circle."""
+  inside = 0
+  for _ in xrange(count):
+x, y = random.uniform(0, 1), random.uniform(0, 1)
+inside += 1 if x*x + y*y <= 1.0 else 0
+  return count, inside
+
+def combine_results(results):
+  """Given all the trial results, estimate pi."""
+  total, inside = sum(r[0] for r in results), sum(r[1] for r in results)
+  return total, inside, 4 * float(inside) / total if tota

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/4acb6411
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/4acb6411
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/4acb6411

Branch: refs/heads/asf-site
Commit: 4acb6411a230a543930e2672f1181ea64ad49094
Parents: be9e207
Author: Davor Bonaci 
Authored: Thu Mar 16 16:21:09 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 16:21:09 2017 -0700

--
 content/blog/2017/03/16/python-sdk-release.html | 255 +++
 content/blog/index.html |  16 ++
 content/feed.xml| 166 ++--
 content/index.html  |   4 +-
 4 files changed, 347 insertions(+), 94 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/4acb6411/content/blog/2017/03/16/python-sdk-release.html
--
diff --git a/content/blog/2017/03/16/python-sdk-release.html 
b/content/blog/2017/03/16/python-sdk-release.html
new file mode 100644
index 000..cb1320c
--- /dev/null
+++ b/content/blog/2017/03/16/python-sdk-release.html
@@ -0,0 +1,255 @@
+
+
+
+  
+  
+  
+  
+
+  Python SDK released in Apache Beam 0.6.0
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  https://beam.apache.org/blog/2017/03/16/python-sdk-release.html"; 
data-proofer-ignore>
+  https://beam.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+ Get Started 
+ 
+ Beam 
Overview
+Quickstart - Java
+Quickstart - Python
+ 
+ Example Walkthroughs
+ WordCount
+ Mobile Gaming
+  
+  Resources
+  Downloads
+  Support
+ 
+   
+
+ Documentation 
+ 
+ Using the 
Documentation
+ 
+ Beam Concepts
+ Programming Guide
+ Additional 
Resources
+ 
+  Pipeline Fundamentals
+  Design Your 
Pipeline
+  Create Your 
Pipeline
+  Test 
Your Pipeline
+  
+ SDKs
+ Java 
SDK
+ Java SDK API Reference 
+
+Python SDK
+Python 
SDK API Reference 
+
+ 
+ Runners
+ Capability Matrix
+ Direct 
Runner
+ Apache 
Apex Runner
+ Apache 
Flink Runner
+ Apache 
Spark Runner
+ Cloud 
Dataflow Runner
+ 
+   
+
+ Contribute 
+ 
+ Get Started 
Contributing
+
+Guides
+ Contribution Guide
+Testing Guide
+Release Guide
+PTransform Style 
Guide
+
+Technical References
+Design Principles
+ Ongoing 
Projects
+Source Repository
+
+ Promotion
+Presentation 
Materials
+Logos and Design
+
+Maturity Model
+Team
+ 
+   
+
+Blog
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png"; alt="Apache 
Logo" style="height:24px;">Apache Software Foundation
+  
+http://www.apache.org/";>ASF Homepage
+http://www.apache.org/licenses/";>License
+http://www.apache.org/security/";>Security
+http://www.apache.org/foundation/thanks.html";>Thanks
+http://www.apache.org/foundation/sponsorship.html";>Sponsorship
+https://www.apache.org/foundation/policies/conduct";>Code of 
Conduct
+  
+
+  
+
+  
+
+
+
+
+
+
+
+
+  
+
+
+http://schema.org/BlogPosting";

[GitHub] beam-site pull request #182: Blog post for 0.6.0 release with python sdk

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/182


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #182

[davor]

This closes #182


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/3917f6e3
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/3917f6e3
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/3917f6e3

Branch: refs/heads/asf-site
Commit: 3917f6e3de88d2a20449fdb2dbaaf08e14d9a7f0
Parents: 2ccd766 4acb641
Author: Davor Bonaci 
Authored: Thu Mar 16 16:21:09 2017 -0700
Committer: Davor Bonaci 
Committed: Thu Mar 16 16:21:09 2017 -0700

--
 content/blog/2017/03/16/python-sdk-release.html | 255 +++
 content/blog/index.html |  16 ++
 content/feed.xml| 166 ++--
 content/index.html  |   4 +-
 src/_data/authors.yml   |  18 +-
 src/_posts/2017-03-16-python-sdk-release.md |  72 ++
 6 files changed, 430 insertions(+), 101 deletions(-)
--

[1/2] beam git commit: This closes #2246

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 960f3e660 -> 25b52c5ac


This closes #2246


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/25b52c5a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/25b52c5a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/25b52c5a

Branch: refs/heads/master
Commit: 25b52c5acc3dddf8ee979b1b7a8c201b68c3f268
Parents: 960f3e6 58cc359
Author: Thomas Groh 
Authored: Thu Mar 16 16:39:47 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 16:39:47 2017 -0700

--
 .../apache/beam/sdk/util/ReifyTimestamps.java   |  76 +
 .../org/apache/beam/sdk/util/Reshuffle.java |  19 ++--
 .../beam/sdk/util/ReifyTimestampsTest.java  | 109 +++
 .../org/apache/beam/sdk/util/ReshuffleTest.java |  70 +++-
 4 files changed, 265 insertions(+), 9 deletions(-)
--

[GitHub] beam pull request #2246: [BEAM-1721] Do not shift Timestamps forwards in Res...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2246


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Do not Shift Timestamps in Reshuffle

[tgroh]

Do not Shift Timestamps in Reshuffle

Explicitly reify input timestamps and restore them after the output of
Reshuffle.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/58cc3597
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/58cc3597
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/58cc3597

Branch: refs/heads/master
Commit: 58cc35970665af99a9ba95d3f28e0974149d8f72
Parents: 960f3e6
Author: Thomas Groh 
Authored: Tue Mar 14 14:05:44 2017 -0700
Committer: Thomas Groh 
Committed: Thu Mar 16 16:39:47 2017 -0700

--
 .../apache/beam/sdk/util/ReifyTimestamps.java   |  76 +
 .../org/apache/beam/sdk/util/Reshuffle.java |  19 ++--
 .../beam/sdk/util/ReifyTimestampsTest.java  | 109 +++
 .../org/apache/beam/sdk/util/ReshuffleTest.java |  70 +++-
 4 files changed, 265 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/58cc3597/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestamps.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestamps.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestamps.java
new file mode 100644
index 000..3b291af
--- /dev/null
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReifyTimestamps.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.util;
+
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.TimestampedValue;
+
+/**
+ * {@link PTransform PTransforms} for reifying the timestamp of values and 
reemitting the original
+ * value with the original timestamp.
+ */
+public class ReifyTimestamps {
+  private ReifyTimestamps() {}
+
+  /**
+   * Create a {@link PTransform} that will output all input {@link KV KVs} 
with the timestamp inside
+   * the value.
+   */
+  public static 
+  PTransform>, PCollection>>>
+  inValues() {
+return ParDo.of(new ReifyValueTimestampDoFn());
+  }
+
+  /**
+   * Create a {@link PTransform} that consumes {@link KV KVs} with a {@link 
TimestampedValue} as the
+   * value, and outputs the {@link KV} of the input key and value at the 
timestamp specified by the
+   * {@link TimestampedValue}.
+   */
+  public static 
+  PTransform>>, 
PCollection>>
+  extractFromValues() {
+return ParDo.of(new ExtractTimestampedValueDoFn());
+  }
+
+  private static class ReifyValueTimestampDoFn
+  extends DoFn, KV>> {
+@ProcessElement
+public void processElement(ProcessContext context) {
+  context.output(
+  KV.of(
+  context.element().getKey(),
+  TimestampedValue.of(context.element().getValue(), 
context.timestamp(;
+}
+  }
+
+  private static class ExtractTimestampedValueDoFn
+  extends DoFn>, KV> {
+@ProcessElement
+public void processElement(ProcessContext context) {
+  KV> kv = context.element();
+  context.outputWithTimestamp(
+  KV.of(kv.getKey(), kv.getValue().getValue()), 
kv.getValue().getTimestamp());
+}
+  }
+}

http://git-wip-us.apache.org/repos/asf/beam/blob/58cc3597/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java
index 4d86c74..e80bc17 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Reshuffle.java
@@ -22,9 +22,11 @@ import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.P

[jira] [Commented] (BEAM-1721) Reshuffle can shift elements in time

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929164#comment-15929164
 ] 

ASF GitHub Bot commented on BEAM-1721:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2246


> Reshuffle can shift elements in time
> 
>
> Key: BEAM-1721
> URL: https://issues.apache.org/jira/browse/BEAM-1721
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The reshuffle transform is meant to have no visible effects on the data that 
> it processes. However, due to the use of a {{GroupByKey}}, the timestamp of 
> the output elements is determined by the {{OutputTimeFn}} of the input 
> {{WindowingStrategy}}
> Elements should not be shifted in time when being processed in {{Reshuffle}}. 
> Currently this would require reifying all timestamps before applying the 
> GroupByKey and reapplying them after. As an intermediate solution, elements 
> should never be shifted forwards in time, as doing so permits the watermark 
> to advance improperly (if the elements already contain their timestamps, for 
> example), and prevents the timestamps from being reassigned within a {{DoFn}} 
> or via the {{WithTimestamps}} transform.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1738) DataflowRunner should override Reshuffle transform

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-1738:
--

 Summary: DataflowRunner should override Reshuffle transform
 Key: BEAM-1738
 URL: https://issues.apache.org/jira/browse/BEAM-1738
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Ben Chambers
Assignee: Thomas Groh


Verify that the code works, and then remove the reification of windows for the 
Dataflow Runner since it handles Reshufle specially and doesn't need the 
explicit reification.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #183: update release notes post 0.6.0 release

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam-site/pull/183

update release notes post 0.6.0 release

R: @sb2nov @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam-site asf-site

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/183.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #183


commit abe87e9ddb362e7c24926a9b7f164d226869480b
Author: Ahmet Altay 
Date:   2017-03-16T23:59:20Z

update release notes post 0.6.0 release




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1582) ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929222#comment-15929222
 ] 

Eugene Kirpichov commented on BEAM-1582:


Still happening and failing precommits quite often.

> ResumeFromCheckpointStreamingTest flakes with what appears as a second firing.
> --
>
> Key: BEAM-1582
> URL: https://issues.apache.org/jira/browse/BEAM-1582
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: First stable release
>
>
> See: 
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/org.apache.beam$beam-runners-spark/2788/testReport/junit/org.apache.beam.runners.spark.translation.streaming/ResumeFromCheckpointStreamingTest/testWithResume/
> After some digging in it appears that a second firing occurs (though only one 
> is expected) but it doesn't come from a stale state (state is empty before it 
> fires).
> Might be a retry happening for some reason, which is OK in terms of 
> fault-tolerance guarantees (at-least-once), but not so much in terms of flaky 
> tests. 
> I'm looking into this hoping to fix this ASAP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1040) Hadoop InputFormat - IO Transform for reads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929425#comment-15929425
 ] 

ASF GitHub Bot commented on BEAM-1040:
--

Github user diptikul closed the pull request at:

https://github.com/apache/beam/pull/2115


> Hadoop InputFormat - IO Transform for reads
> ---
>
> Key: BEAM-1040
> URL: https://issues.apache.org/jira/browse/BEAM-1040
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Dipti Kulkarni
>
> We should build a IO read transform that will read data from services 
> supporting the Hadoop InputFormat  interface [1]
> This will make it easy to connect to a variety of databases while still 
> providing some aspects of scalability since the InputFormat interface 
> provides for parallel reading. 
> [1] 
> https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/mapred/InputFormat.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2115: BEAM-1040 Add HadoopInputFormatIO

[diptikul]

Github user diptikul closed the pull request at:

https://github.com/apache/beam/pull/2115


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2935

[Apache Jenkins Server]

See