[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839334#comment-15839334
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1828


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1828: [BEAM-79] Fix PostCommit test confs for Gearpump ru...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1828


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-79] Fix PostCommit test confs for Gearpump runner

[kenn]

Repository: beam
Updated Branches:
  refs/heads/gearpump-runner 1ed16f11a -> 4001aeb19


[BEAM-79] Fix PostCommit test confs for Gearpump runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4fd216b6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4fd216b6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4fd216b6

Branch: refs/heads/gearpump-runner
Commit: 4fd216b6b9f7427a7df118a60202752ebeb1836d
Parents: 1ed16f1
Author: manuzhang 
Authored: Tue Jan 24 13:16:07 2017 +0800
Committer: manuzhang 
Committed: Tue Jan 24 13:16:07 2017 +0800

--
 .../job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4fd216b6/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy
--
diff --git 
a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy 
b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy
index f07bbc6..847ade3 100644
--- a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy
+++ b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy
@@ -37,5 +37,5 @@ mavenJob('beam_PostCommit_Java_RunnableOnService_Gearpump') {
   common_job_properties.setPostCommit(delegate, '0 5 31 2 *')
 
   // Maven goals for this job.
-  goals('-B -e clean verify -am -pl runners/gearpump -DforkCount=0 
-DrunnableOnServicePipelineOptions=\'[ "--runner=TestGearpumpRunner", 
"--streaming=false" ]\'')
+  goals('-B -e clean verify -am -pl runners/gearpump  
-Plocal-runnable-on-service-tests -Prunnable-on-service-tests')
 }

[2/2] beam git commit: This closes #1828: Fix PostCommit test confs for Gearpump runner

[kenn]

This closes #1828: Fix PostCommit test confs for Gearpump runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4001aeb1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4001aeb1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4001aeb1

Branch: refs/heads/gearpump-runner
Commit: 4001aeb191062c5a7f83ef919f781ec413be8cb2
Parents: 1ed16f1 4fd216b
Author: Kenneth Knowles 
Authored: Wed Jan 25 22:40:04 2017 -0800
Committer: Kenneth Knowles 
Committed: Wed Jan 25 22:40:04 2017 -0800

--
 .../job_beam_PostCommit_Java_RunnableOnService_Gearpump.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-980) Ability to configure the DAG created by Apex Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839324#comment-15839324
 ] 

ASF GitHub Bot commented on BEAM-980:
-

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/1850

[BEAM-980] Support configuration of Apex DAG through properties file.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-980

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1850.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1850


commit 02bbf4357fe1e10c8646d4101a721721d400e997
Author: Thomas Weise 
Date:   2017-01-26T06:22:36Z

BEAM-980 Support configuration of Apex DAG through properties file.




> Ability to configure the DAG created by Apex Runner
> ---
>
> Key: BEAM-980
> URL: https://issues.apache.org/jira/browse/BEAM-980
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> The Beam pipeline is translated to an Apex DAG of operators that have names 
> that are derived from the transforms. In case of composite transforms those 
> look like path names. Apex lets the user configure things like memory, 
> vcores, parallelism through properties/attributes that reference the operator 
> names. The configuration approach needs to be documented and supplemented 
> with an example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1850: [BEAM-980] Support configuration of Apex DAG throug...

[tweise]

GitHub user tweise opened a pull request:

https://github.com/apache/beam/pull/1850

[BEAM-980] Support configuration of Apex DAG through properties file.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tweise/beam BEAM-980

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1850.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1850


commit 02bbf4357fe1e10c8646d4101a721721d400e997
Author: Thomas Weise 
Date:   2017-01-26T06:22:36Z

BEAM-980 Support configuration of Apex DAG through properties file.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-814) Improve performance when staging files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839315#comment-15839315
 ] 

ASF GitHub Bot commented on BEAM-814:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1849

[BEAM-814] Dataflow: parallelize staging of files

Resubmitting #1184 with a fix to preserve the classpath ordering (and a new 
test!)

R: @tgroh or @lukecwik (if avail)

The first commit is exactly the same as already-reviewed #1184; only need 
to review the second commit (or the relevant part of the union).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam dataflow-runner-speedup-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1849.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1849


commit 23e2b913946acb2690fbac2d751a5672d80121aa
Author: Dan Halperin 
Date:   2017-01-26T05:04:20Z

Recommit "DataflowRunner: parallelize staging of files"

Revert "This closes #1847"

This reverts commit 1c6e667414788fe99f583fac39d458a4984ae162, reversing
changes made to 6413299a20be57de849684479134479fa1acee2d.

commit b0b91c842e09aa7fdb5c1dc216574daa43b437ea
Author: Dan Halperin 
Date:   2017-01-26T06:15:59Z

PackageUtil: preserve classpath ordering when uploading

Also add a test




> Improve performance when staging files
> --
>
> Key: BEAM-814
> URL: https://issues.apache.org/jira/browse/BEAM-814
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.5.0
>
>
> Right now, {{PackageUtil}} hashes/compresses/sizes files, checks if they have 
> already been staged, and uploads them if not, completely serially.
> There is some low-hanging fruit to parallelize this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1849: [BEAM-814] Dataflow: parallelize staging of files

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1849

[BEAM-814] Dataflow: parallelize staging of files

Resubmitting #1184 with a fix to preserve the classpath ordering (and a new 
test!)

R: @tgroh or @lukecwik (if avail)

The first commit is exactly the same as already-reviewed #1184; only need 
to review the second commit (or the relevant part of the union).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam dataflow-runner-speedup-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1849.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1849


commit 23e2b913946acb2690fbac2d751a5672d80121aa
Author: Dan Halperin 
Date:   2017-01-26T05:04:20Z

Recommit "DataflowRunner: parallelize staging of files"

Revert "This closes #1847"

This reverts commit 1c6e667414788fe99f583fac39d458a4984ae162, reversing
changes made to 6413299a20be57de849684479134479fa1acee2d.

commit b0b91c842e09aa7fdb5c1dc216574daa43b437ea
Author: Dan Halperin 
Date:   2017-01-26T06:15:59Z

PackageUtil: preserve classpath ordering when uploading

Also add a test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839076#comment-15839076
 ] 

Pei He commented on BEAM-1315:
--

My main concern is about the directory permission. (Users only have write 
access under /home/output/)

I think temp files collision is rare, given it is unlikely for users to write 
with the same output prefix in two sinks. (if so, the output files will 
collide). There is also some randomness from the DATE when we name the temp 
directory.

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1322) Update WordCount snippet in snippets.py

[Hadar Hod (JIRA)]

Hadar Hod created BEAM-1322:
---

 Summary: Update WordCount snippet in snippets.py
 Key: BEAM-1322
 URL: https://issues.apache.org/jira/browse/BEAM-1322
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Hadar Hod
Assignee: Ahmet Altay


Code snippet (in snippets.py) named "examples_wordcount_minimal_options" needs 
to be updated to match Java and to remove mentions of Google.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam-site pull request #130: Python docs

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/beam-site/pull/130

Python docs

- Add Quickstart for Python
- Add Python to WordCount doc

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site python-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #130


commit 7b8364f82ecc7e403dbda6db58a9b4f6c4ad3a60
Author: Hadar Hod 
Date:   2017-01-25T03:37:16Z

WIP: Add Python to WordCount documentation

commit 55060b72d66bd8617dde8d1faf6574aeefdd6eef
Author: Hadar Hod 
Date:   2017-01-25T09:06:19Z

Added Python to WordCount doc.

commit 912f3b92096af81a8262462d3660cf5c02097d0a
Author: Hadar Hod 
Date:   2017-01-25T09:46:37Z

Added Python Quickstart

commit 44f972337a918ec7a92b983a49caf3587b23cd1b
Author: Hadar Hod 
Date:   2017-01-26T00:36:09Z

Updated WordCount. Renamed Quickstart for Java.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1321) Re-thinking BigQueryIO export/load/copy jobs retry logic

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-1321:
-
Priority: Minor  (was: Major)

> Re-thinking BigQueryIO export/load/copy jobs retry logic
> 
>
> Key: BEAM-1321
> URL: https://issues.apache.org/jira/browse/BEAM-1321
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Pei He
>Priority: Minor
>
> Currently, BigQueryIO retries failed jobs, and runners retries failed bundles.
> Needs to re-thinking about which components should own the retry logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1321) Re-thinking BigQueryIO export/load/copy jobs retry logic

[Pei He (JIRA)]

Pei He created BEAM-1321:


 Summary: Re-thinking BigQueryIO export/load/copy jobs retry logic
 Key: BEAM-1321
 URL: https://issues.apache.org/jira/browse/BEAM-1321
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-extensions
Reporter: Pei He


Currently, BigQueryIO retries failed jobs, and runners retries failed bundles.

Needs to re-thinking about which components should own the retry logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1319) PipelineOptions subclasses defined in the main session could be duplicated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839025#comment-15839025
 ] 

ASF GitHub Bot commented on BEAM-1319:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1848

[BEAM-1319] Add conflict resolution to the PipelineOptions internal 
argparse.

In some instances where a PipelineOptions subclass was defined in the
main session and save_main_session option is enabled, that subclass may
appear multiple times in the PipelineOptions.__subclassess__() list.
This is causing problems with the argparse because options are not
unique any more.

This changes filter the subclasses by name, and pick the last unique
instance of each subclass.

As an alternative option, we could use the conflict_handler='resolve' of
the argparse. However, the drawback of this approach would be that allowing
accidental overriding of the same option name in two different 
PipelineOptions
subclasses.

Also moves wordcount example to use PipelineOptions. This both serves as an
example of using PipelineOptions, also it is already integrated with
tests and will continue to test this case.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam wordo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1848.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1848


commit b488555304f1110125598dd624077eb94ed0d0d7
Author: Ahmet Altay 
Date:   2017-01-26T01:48:13Z

Add conflict resolution to the PipelineOptions internal argparse.

In some instances where a PipelineOptions subclass was defined in the
main session and save_main_session option is enabled, that subclass may
appear multiple times in the PipelineOptions.__subclassess__() list.
This is causing problems with the argparse because options are not
unique any more.

This changes filter the subclasses by name, and pick the last unique
instance of each subclass.

As an alternative option, we could use the conflict_handler='resolve' of
the argparse. However, the drawback of this approach would be that allowing
accidental overriding of the same option name in two different 
PipelineOptions
subclasses.

Also moves wordcount to use PipelineOptions. This both serves as an
example of using PipelineOptions, also it is already integrated with
tests and will continue to test this case.




> PipelineOptions subclasses defined in the main session could be duplicated
> --
>
> Key: BEAM-1319
> URL: https://issues.apache.org/jira/browse/BEAM-1319
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>
> Duplication is caused as a result of the save_main_session option.
> This also breaks argparse because same options will be defined multiple times.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1848: [BEAM-1319] Add conflict resolution to the Pipeline...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1848

[BEAM-1319] Add conflict resolution to the PipelineOptions internal 
argparse.

In some instances where a PipelineOptions subclass was defined in the
main session and save_main_session option is enabled, that subclass may
appear multiple times in the PipelineOptions.__subclassess__() list.
This is causing problems with the argparse because options are not
unique any more.

This changes filter the subclasses by name, and pick the last unique
instance of each subclass.

As an alternative option, we could use the conflict_handler='resolve' of
the argparse. However, the drawback of this approach would be that allowing
accidental overriding of the same option name in two different 
PipelineOptions
subclasses.

Also moves wordcount example to use PipelineOptions. This both serves as an
example of using PipelineOptions, also it is already integrated with
tests and will continue to test this case.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam wordo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1848.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1848


commit b488555304f1110125598dd624077eb94ed0d0d7
Author: Ahmet Altay 
Date:   2017-01-26T01:48:13Z

Add conflict resolution to the PipelineOptions internal argparse.

In some instances where a PipelineOptions subclass was defined in the
main session and save_main_session option is enabled, that subclass may
appear multiple times in the PipelineOptions.__subclassess__() list.
This is causing problems with the argparse because options are not
unique any more.

This changes filter the subclasses by name, and pick the last unique
instance of each subclass.

As an alternative option, we could use the conflict_handler='resolve' of
the argparse. However, the drawback of this approach would be that allowing
accidental overriding of the same option name in two different 
PipelineOptions
subclasses.

Also moves wordcount to use PipelineOptions. This both serves as an
example of using PipelineOptions, also it is already integrated with
tests and will continue to test this case.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Revert "This closes #1184"

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 6413299a2 -> 1c6e66741


Revert "This closes #1184"

This reverts commit c525783704e0cc47845df8cdec1715e1f1c74008, reversing
changes made to 979c9376f820577bad43c18cc1a7ee86fab9d942.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fee029f7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fee029f7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fee029f7

Branch: refs/heads/master
Commit: fee029f7f9963c9de821ff5792d7f45fabe6cb5d
Parents: 6413299
Author: Dan Halperin 
Authored: Wed Jan 25 15:54:26 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 15:54:26 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |   5 -
 .../beam/runners/dataflow/util/GcsStager.java   |  18 +-
 .../beam/runners/dataflow/util/PackageUtil.java | 349 +++
 .../runners/dataflow/util/PackageUtilTest.java  |  42 +--
 .../org/apache/beam/sdk/options/GcsOptions.java |   4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  12 -
 6 files changed, 149 insertions(+), 281 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/fee029f7/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 9858b3d..eea5502 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -203,11 +203,6 @@
 
 
 
-  com.google.apis
-  google-api-services-storage
-
-
-
   com.google.auth
   google-auth-library-credentials
 

http://git-wip-us.apache.org/repos/asf/beam/blob/fee029f7/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
index 53822e3..6ca4c3f 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
@@ -17,19 +17,13 @@
  */
 package org.apache.beam.runners.dataflow.util;
 
-import static com.google.common.base.MoreObjects.firstNonNull;
-import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.api.services.dataflow.model.DataflowPackage;
-import com.google.api.services.storage.Storage;
 import java.util.List;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.GcsUtil.GcsUtilFactory;
-import org.apache.beam.sdk.util.Transport;
 
 /**
  * Utility class for staging files to GCS.
@@ -41,7 +35,6 @@ public class GcsStager implements Stager {
 this.options = options;
   }
 
-  @SuppressWarnings("unused")  // used via reflection
   public static GcsStager fromOptions(PipelineOptions options) {
 return new GcsStager(options.as(DataflowPipelineOptions.class));
   }
@@ -55,16 +48,7 @@ public class GcsStager implements Stager {
 if (windmillBinary != null) {
   filesToStage.add("windmill_main=" + windmillBinary);
 }
-int uploadSizeBytes = firstNonNull(options.getGcsUploadBufferSizeBytes(), 
1024 * 1024);
-checkArgument(uploadSizeBytes > 0, "gcsUploadBufferSizeBytes must be > 0");
-uploadSizeBytes = Math.min(uploadSizeBytes, 1024 * 1024);
-Storage.Builder storageBuilder = Transport.newStorageClient(options);
-GcsUtil util = GcsUtilFactory.create(
-storageBuilder.build(),
-storageBuilder.getHttpRequestInitializer(),
-options.getExecutorService(),
-uploadSizeBytes);
 return PackageUtil.stageClasspathElements(
-options.getFilesToStage(), options.getStagingLocation(), util);
+options.getFilesToStage(), options.getStagingLocation());
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/fee029f7/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
index fa

[2/2] beam git commit: This closes #1847

[dhalperi]

This closes #1847


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1c6e6674
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1c6e6674
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1c6e6674

Branch: refs/heads/master
Commit: 1c6e667414788fe99f583fac39d458a4984ae162
Parents: 6413299 fee029f
Author: Dan Halperin 
Authored: Wed Jan 25 17:47:08 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 17:47:08 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |   5 -
 .../beam/runners/dataflow/util/GcsStager.java   |  18 +-
 .../beam/runners/dataflow/util/PackageUtil.java | 349 +++
 .../runners/dataflow/util/PackageUtilTest.java  |  42 +--
 .../org/apache/beam/sdk/options/GcsOptions.java |   4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  12 -
 6 files changed, 149 insertions(+), 281 deletions(-)
--

[GitHub] beam pull request #1847: Revert "This closes #1184"

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1847


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2106

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1320) Add sphinx or pydocs documentation for python-sdk

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838981#comment-15838981
 ] 

Ahmet Altay commented on BEAM-1320:
---

The result of this should be integrated into the release process.

> Add sphinx or pydocs documentation for python-sdk
> -
>
> Key: BEAM-1320
> URL: https://issues.apache.org/jira/browse/BEAM-1320
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1320) Add sphinx or pydocs documentation for python-sdk

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1320:
-

 Summary: Add sphinx or pydocs documentation for python-sdk
 Key: BEAM-1320
 URL: https://issues.apache.org/jira/browse/BEAM-1320
 Project: Beam
  Issue Type: Task
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1319) PipelineOptions subclasses defined in the main session could be duplicated

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1319:
-

 Summary: PipelineOptions subclasses defined in the main session 
could be duplicated
 Key: BEAM-1319
 URL: https://issues.apache.org/jira/browse/BEAM-1319
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay


Duplication is caused as a result of the save_main_session option.

This also breaks argparse because same options will be defined multiple times.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1318) PipelineOptions should warn if there are unused options

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1318:
-

 Summary: PipelineOptions should warn if there are unused options
 Key: BEAM-1318
 URL: https://issues.apache.org/jira/browse/BEAM-1318
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Ahmet Altay
Priority: Minor


Since PipelineOptions uses argparse, it is possible that some options are 
actually consumed by the program. In that case a better usage pattern would be 
to pass only unconsumed options to PipelineOptions but we cannot enforce this.

This cannot be an error because of the above reason, but we can show a warning.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1313) beam_PostCommit_Python_Verify failing at #1087

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1313.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> beam_PostCommit_Python_Verify failing at #1087
> --
>
> Key: BEAM-1313
> URL: https://issues.apache.org/jira/browse/BEAM-1313
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: Not applicable
>
>
> test_with_requirements_file is failing because of the leftover files from 
> other (or previous) tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1317) Strategy around snippet versioning

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1317:
-

 Summary: Strategy around snippet versioning
 Key: BEAM-1317
 URL: https://issues.apache.org/jira/browse/BEAM-1317
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Daniel Halperin
Assignee: Davor Bonaci


Now that the website has the ability to import snippets from code, presumably 
some of these are pulled in from {{master}}. Do we need a postcommit to 
regenerate the website automatically in case master changes? Should we have a 
policy to never link to or get snippets from master?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1090

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838936#comment-15838936
 ] 

Daniel Halperin commented on BEAM-1315:
---

Sounds like I was wrong about the above issues. Downgraded to "major".

The remaining issues:

* TempDirectories will be shared if files have the same prefix, but different 
suffixes. This means cleanup of the first to succeed will cause job failure as 
the second will have missing files.
* /home/output/ is a valid output path -- it should write to 
/home/output/-0-of-1.txt, e.g. But with the temp dir changes, that 
configuration would fail if I don't have access to create and write to 
/home/tmp-output-$DATE/.. but it used to work.

These are both considerably less bad than I thought.

Blocker for 0.5.0 -- only to decide if this needs to be fixed for that release.

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1315:
--
Priority: Major  (was: Blocker)

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1313) beam_PostCommit_Python_Verify failing at #1087

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838884#comment-15838884
 ] 

ASF GitHub Bot commented on BEAM-1313:
--

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1844


> beam_PostCommit_Python_Verify failing at #1087
> --
>
> Key: BEAM-1313
> URL: https://issues.apache.org/jira/browse/BEAM-1313
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> test_with_requirements_file is failing because of the leftover files from 
> other (or previous) tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1844: [BEAM-1313] Use a temp directory for requirements c...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1844


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1218) De-Googlify Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838883#comment-15838883
 ] 

ASF GitHub Bot commented on BEAM-1218:
--

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1837


> De-Googlify Python SDK
> --
>
> Key: BEAM-1218
> URL: https://issues.apache.org/jira/browse/BEAM-1218
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Mark Liu
>Assignee: Ahmet Altay
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1837: [BEAM-1218 ] Remove dataflow_test.py

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1837


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Closes #1837

[robertwb]

Closes #1837


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c6420df9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c6420df9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c6420df9

Branch: refs/heads/python-sdk
Commit: c6420df9791eb6083fba1f74bd88e06ce8f6a61f
Parents: 4e1028b 2aa7d47
Author: Robert Bradshaw 
Authored: Wed Jan 25 16:18:10 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 16:18:10 2017 -0800

--
 sdks/python/apache_beam/dataflow_test.py| 418 ---
 .../apache_beam/transforms/ptransform_test.py   |  67 +++
 .../apache_beam/transforms/sideinputs_test.py   | 208 -
 3 files changed, 274 insertions(+), 419 deletions(-)
--

[1/2] beam git commit: Revert "Revert "Remove dataflow_test.py""

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 4e1028b3d -> c6420df97


Revert "Revert "Remove dataflow_test.py""

This reverts commit 96fcc7d31c2540f867c3a73903c2aa99183a6b8b.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2aa7d47e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2aa7d47e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2aa7d47e

Branch: refs/heads/python-sdk
Commit: 2aa7d47e1491e0601b7b4d1476a8f182b2a14dc3
Parents: 4e1028b
Author: Robert Bradshaw 
Authored: Tue Jan 24 16:33:55 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 16:18:09 2017 -0800

--
 sdks/python/apache_beam/dataflow_test.py| 418 ---
 .../apache_beam/transforms/ptransform_test.py   |  67 +++
 .../apache_beam/transforms/sideinputs_test.py   | 208 -
 3 files changed, 274 insertions(+), 419 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2aa7d47e/sdks/python/apache_beam/dataflow_test.py
--
diff --git a/sdks/python/apache_beam/dataflow_test.py 
b/sdks/python/apache_beam/dataflow_test.py
deleted file mode 100644
index f410230..000
--- a/sdks/python/apache_beam/dataflow_test.py
+++ /dev/null
@@ -1,418 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-"""Integration tests for the dataflow package."""
-
-from __future__ import absolute_import
-
-import logging
-import re
-import unittest
-
-import apache_beam as beam
-from apache_beam.pvalue import AsDict
-from apache_beam.pvalue import AsIter as AllOf
-from apache_beam.pvalue import AsList
-from apache_beam.pvalue import AsSingleton
-from apache_beam.pvalue import EmptySideInput
-from apache_beam.pvalue import SideOutputValue
-from apache_beam.test_pipeline import TestPipeline
-from apache_beam.transforms import Create
-from apache_beam.transforms import DoFn
-from apache_beam.transforms import FlatMap
-from apache_beam.transforms import GroupByKey
-from apache_beam.transforms import Map
-from apache_beam.transforms import ParDo
-from apache_beam.transforms import WindowInto
-from apache_beam.transforms.util import assert_that
-from apache_beam.transforms.util import equal_to
-from apache_beam.transforms.window import IntervalWindow
-from apache_beam.transforms.window import WindowFn
-from nose.plugins.attrib import attr
-
-
-class DataflowTest(unittest.TestCase):
-  """Dataflow integration tests."""
-
-  SAMPLE_DATA = ['aa bb cc aa bb aa \n'] * 10
-  SAMPLE_RESULT = [('cc', 10), ('bb', 20), ('aa', 30)]
-
-  @beam.ptransform_fn
-  def Count(pcoll):  # pylint: disable=invalid-name, no-self-argument
-"""A Count transform: v, ... => (v, n), ..."""
-return (pcoll
-| 'AddCount' >> Map(lambda x: (x, 1))
-| 'GroupCounts' >> GroupByKey()
-| 'AddCounts' >> Map(lambda (x, ones): (x, sum(ones
-
-  @attr('ValidatesRunner')
-  def test_word_count(self):
-pipeline = TestPipeline()
-lines = pipeline | 'SomeWords' >> Create(DataflowTest.SAMPLE_DATA)
-result = (
-(lines | 'GetWords' >> FlatMap(lambda x: re.findall(r'\w+', x)))
-.apply('CountWords', DataflowTest.Count))
-assert_that(result, equal_to(DataflowTest.SAMPLE_RESULT))
-pipeline.run()
-
-  @attr('ValidatesRunner')
-  def test_map(self):
-pipeline = TestPipeline()
-lines = pipeline | 'input' >> Create(['a', 'b', 'c'])
-result = (lines
-  | 'upper' >> Map(str.upper)
-  | 'prefix' >> Map(lambda x, prefix: prefix + x, 'foo-'))
-assert_that(result, equal_to(['foo-A', 'foo-B', 'foo-C']))
-pipeline.run()
-
-  @attr('ValidatesRunner')
-  def test_par_do_with_side_input_as_arg(self):
-pipeline = TestPipeline()
-words_list = ['aa', 'bb', 'cc']
-words = pipeline | 'SomeWords' >> Create(words_list)
-prefix = pipeline | 'SomeString' >> Create(['xyz'])  # side in
-suffix = 'zyx'
-result = words | FlatMap(
-'DecorateWords',
-lambda x, pfx, sfx: ['%s-%s-%s' % (pfx, x, sfx)],
-   

[1/2] beam git commit: Closes #1844

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 592422059 -> 4e1028b3d


Closes #1844


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4e1028b3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4e1028b3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4e1028b3

Branch: refs/heads/python-sdk
Commit: 4e1028b3dfeaf02e51eb9f3b5d1a5e78c1cfcbb9
Parents: 5924220 5787e81
Author: Robert Bradshaw 
Authored: Wed Jan 25 16:16:52 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 16:16:52 2017 -0800

--
 .../python/apache_beam/utils/dependency_test.py | 47 +++-
 1 file changed, 27 insertions(+), 20 deletions(-)
--

[2/2] beam git commit: Use a temp directory for requirements cache in test_with_requirements_file

[robertwb]

Use a temp directory for requirements cache in
test_with_requirements_file

The test fails if there are leftover files in the default folder for
requirements cache either from earlier tests, or from the previous
workspaces.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5787e817
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5787e817
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5787e817

Branch: refs/heads/python-sdk
Commit: 5787e817a7eda4859963d535df21f2fa00edf8af
Parents: 5924220
Author: Ahmet Altay 
Authored: Wed Jan 25 09:57:18 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 16:16:52 2017 -0800

--
 .../python/apache_beam/utils/dependency_test.py | 47 +++-
 1 file changed, 27 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5787e817/sdks/python/apache_beam/utils/dependency_test.py
--
diff --git a/sdks/python/apache_beam/utils/dependency_test.py 
b/sdks/python/apache_beam/utils/dependency_test.py
index a484d60..75a89e2 100644
--- a/sdks/python/apache_beam/utils/dependency_test.py
+++ b/sdks/python/apache_beam/utils/dependency_test.py
@@ -106,27 +106,34 @@ class SetupTest(unittest.TestCase):
 dependency.stage_job_resources(options))
 
   def test_with_requirements_file(self):
-staging_dir = tempfile.mkdtemp()
-source_dir = tempfile.mkdtemp()
+try:
+  staging_dir = tempfile.mkdtemp()
+  requirements_cache_dir = tempfile.mkdtemp()
+  source_dir = tempfile.mkdtemp()
 
-options = PipelineOptions()
-options.view_as(GoogleCloudOptions).staging_location = staging_dir
-self.update_options(options)
-options.view_as(SetupOptions).requirements_file = os.path.join(
-source_dir, dependency.REQUIREMENTS_FILE)
-self.create_temp_file(
-os.path.join(source_dir, dependency.REQUIREMENTS_FILE), 'nothing')
-self.assertEqual(
-sorted([dependency.REQUIREMENTS_FILE,
-'abc.txt', 'def.txt']),
-sorted(dependency.stage_job_resources(
-options,
-populate_requirements_cache=self.populate_requirements_cache)))
-self.assertTrue(
-os.path.isfile(
-os.path.join(staging_dir, dependency.REQUIREMENTS_FILE)))
-self.assertTrue(os.path.isfile(os.path.join(staging_dir, 'abc.txt')))
-self.assertTrue(os.path.isfile(os.path.join(staging_dir, 'def.txt')))
+  options = PipelineOptions()
+  options.view_as(GoogleCloudOptions).staging_location = staging_dir
+  self.update_options(options)
+  options.view_as(SetupOptions).requirements_cache = requirements_cache_dir
+  options.view_as(SetupOptions).requirements_file = os.path.join(
+  source_dir, dependency.REQUIREMENTS_FILE)
+  self.create_temp_file(
+  os.path.join(source_dir, dependency.REQUIREMENTS_FILE), 'nothing')
+  self.assertEqual(
+  sorted([dependency.REQUIREMENTS_FILE,
+  'abc.txt', 'def.txt']),
+  sorted(dependency.stage_job_resources(
+  options,
+  populate_requirements_cache=self.populate_requirements_cache)))
+  self.assertTrue(
+  os.path.isfile(
+  os.path.join(staging_dir, dependency.REQUIREMENTS_FILE)))
+  self.assertTrue(os.path.isfile(os.path.join(staging_dir, 'abc.txt')))
+  self.assertTrue(os.path.isfile(os.path.join(staging_dir, 'def.txt')))
+finally:
+  shutil.rmtree(staging_dir)
+  shutil.rmtree(requirements_cache_dir)
+  shutil.rmtree(source_dir)
 
   def test_requirements_file_not_present(self):
 staging_dir = tempfile.mkdtemp()

[jira] [Comment Edited] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838874#comment-15838874
 ] 

Daniel Halperin edited comment on BEAM-1315 at 1/26/17 12:15 AM:
-

This means that in fact temp file cleanup will cause data loss, or at least 
failed jobs, if there are multiple sinks writing to :output1/ and :output2/. 
This is a release blocker.

Is the issue present in 0.4.0?


was (Author: dhalp...@google.com):
This means that in fact temp file cleanup will cause data loss if there are 
multiple sinks writing to :output1/ and :output2/. This is a release blocker.

Is the issue present in 0.4.0?

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Blocker
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838874#comment-15838874
 ] 

Daniel Halperin commented on BEAM-1315:
---

This means that in fact temp file cleanup will cause data loss if there are 
multiple sinks writing to :output1/ and :output2/. This is a release blocker.

Is the issue present in 0.4.0?

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Blocker
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1315:
--
Priority: Blocker  (was: Critical)

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Blocker
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1315:
--
Priority: Critical  (was: Minor)

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Critical
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1315:
--
Fix Version/s: 0.5.0

> FileBasedSink could write tmp files above the output prefix.
> 
>
> Key: BEAM-1315
> URL: https://issues.apache.org/jira/browse/BEAM-1315
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Pei He
>Priority: Critical
> Fix For: 0.5.0
>
>
> FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
> new FileBasedWriteOperation.TemporaryDirectoryBuilder();
> tempDirBuilder.apply("/home/output/"),
> apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"
> And, it was not under the output prefix "/home/output/".
> Examples of Path resolve:
> Paths.get("/home/output").resolve("temp-beam");
> Paths.get("/home/output/").resolve("temp-beam");
> /home/output/temp-beam
> /home/output/temp-beam
> Paths.get("/home/output").resolve("..").resolve("temp-beam");
> Paths.get("/home/output/").resolve("..").resolve("temp-beam");
> /home/output/../temp-beam
> /home/output/../temp-beam
> Paths.get("/home/output").resolveSibling("temp-beam");
> Paths.get("/home/output/").resolveSibling("temp-beam");
> /home/temp-beam
> /home/temp-beam



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1316) DoFn#startBundle and #finishBundle should not be able to output

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838870#comment-15838870
 ] 

Daniel Halperin commented on BEAM-1316:
---

I think one many need to output in finish bundle using the current "buffer, and 
flush half-full if this is the end of the bundle" pattern.

> DoFn#startBundle and #finishBundle should not be able to output
> ---
>
> Key: BEAM-1316
> URL: https://issues.apache.org/jira/browse/BEAM-1316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>
> While within startBundle and finishBundle, the window in which elements are 
> output is not generally defined. Elements must always be output from within a 
> windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
> operate appropriately.
> startBundle and finishBundle are suitable for operational duties, similarly 
> to {{setup}} and {{teardown}}, but within the scope of some collection of 
> input elements. This includes actions such as clearing field state within a 
> DoFn and ensuring all live RPCs complete successfully before committing 
> inputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1847: Revert "This closes #1184"

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1847

Revert "This closes #1184"

This reverts commit c525783704e0cc47845df8cdec1715e1f1c74008, reversing
changes made to 979c9376f820577bad43c18cc1a7ee86fab9d942.

I believe this is causing DataflowRunner flakes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam rollback-dataflow-runner-speedup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1847.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1847


commit fee029f7f9963c9de821ff5792d7f45fabe6cb5d
Author: Dan Halperin 
Date:   2017-01-25T23:54:26Z

Revert "This closes #1184"

This reverts commit c525783704e0cc47845df8cdec1715e1f1c74008, reversing
changes made to 979c9376f820577bad43c18cc1a7ee86fab9d942.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2104

[Dan Halperin]

This is 99.9% https://github.com/apache/beam/pull/1184. I will prepare and
test a rollback.

On Wed, Jan 25, 2017 at 3:27 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  OnService_Dataflow/changes>
>
>

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2104

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1316) DoFn#startBundle and #finishBundle should not be able to output

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838761#comment-15838761
 ] 

Thomas Groh commented on BEAM-1316:
---

Forbidding output from startBundle and finishBundle brings the contexts 
received by them in line with setup and teardown

> DoFn#startBundle and #finishBundle should not be able to output
> ---
>
> Key: BEAM-1316
> URL: https://issues.apache.org/jira/browse/BEAM-1316
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>
> While within startBundle and finishBundle, the window in which elements are 
> output is not generally defined. Elements must always be output from within a 
> windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
> operate appropriately.
> startBundle and finishBundle are suitable for operational duties, similarly 
> to {{setup}} and {{teardown}}, but within the scope of some collection of 
> input elements. This includes actions such as clearing field state within a 
> DoFn and ensuring all live RPCs complete successfully before committing 
> inputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1089

[Apache Jenkins Server]

See 

--
[...truncated 915 lines...]
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_invalid_ellipsis_type_param 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_params_must_be_type_or_constraint 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_raw_tuple (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_be_tuple 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_have_same_arity 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_duplicates_ignored 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param_cant_be_object_instance 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_nested_unions_flattened 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_nested_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_composite_type_in_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_not_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_repr (apache_beam.typehints.typehints_test.UnionHintTestCase) 
... ok
test_deprecated_with_since_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... 

[jira] [Created] (BEAM-1316) DoFn#startBundle and #finishBundle should not be able to output

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1316:
-

 Summary: DoFn#startBundle and #finishBundle should not be able to 
output
 Key: BEAM-1316
 URL: https://issues.apache.org/jira/browse/BEAM-1316
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh


While within startBundle and finishBundle, the window in which elements are 
output is not generally defined. Elements must always be output from within a 
windowed context, or the {{WindowFn}} used by the {{PCollection}} may not 
operate appropriately.

startBundle and finishBundle are suitable for operational duties, similarly to 
{{setup}} and {{teardown}}, but within the scope of some collection of input 
elements. This includes actions such as clearing field state within a DoFn and 
ensuring all live RPCs complete successfully before committing inputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] beam git commit: [BEAM-708] use AutoValue to reduce boilerplate in BoundedReadFromUnboundedSource

[lcwik]

Repository: beam
Updated Branches:
  refs/heads/master 95beda69b -> 6413299a2


[BEAM-708] use AutoValue to reduce boilerplate in BoundedReadFromUnboundedSource


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eeec9f12
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eeec9f12
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eeec9f12

Branch: refs/heads/master
Commit: eeec9f128f5d5fd4db6de4fd90d4967ff87587e4
Parents: c53249d
Author: Kai Jiang 
Authored: Wed Jan 18 07:00:44 2017 -0800
Committer: Kai Jiang 
Committed: Wed Jan 18 23:37:14 2017 -0800

--
 .../sdk/io/BoundedReadFromUnboundedSource.java  | 71 +---
 1 file changed, 46 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/eeec9f12/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
index 8b63bfd..7e25a01 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
@@ -18,12 +18,14 @@
 package org.apache.beam.sdk.io;
 
 import com.google.api.client.util.BackOff;
+import com.google.auto.value.AutoValue;
 import com.google.common.util.concurrent.Uninterruptibles;
 import java.io.IOException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.concurrent.TimeUnit;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.coders.Coder;
@@ -82,7 +84,12 @@ public class BoundedReadFromUnboundedSource extends 
PTransform(source, 
maxNumRecords, maxReadTime);
+this.adaptedSource =
+new 
AutoValue_BoundedReadFromUnboundedSource_UnboundedToBoundedSourceAdapter
+.Builder()
+.setSource(source)
+.setMaxNumRecords(maxNumRecords)
+.setMaxReadTime(maxReadTime).build();
   }
 
   /**
@@ -133,17 +140,27 @@ public class BoundedReadFromUnboundedSource extends 
PTransform
+  /**
+   * An Adapter wraps the underlying {@link UnboundedSource} with the 
specified bounds on
+   * number of records and read time into {@link BoundedSource}.
+   */
+  @AutoValue
+  public abstract static class UnboundedToBoundedSourceAdapter
   extends BoundedSource> {
-private final UnboundedSource source;
-private final long maxNumRecords;
-private final Duration maxReadTime;
-
-private UnboundedToBoundedSourceAdapter(
-UnboundedSource source, long maxNumRecords, Duration 
maxReadTime) {
-  this.source = source;
-  this.maxNumRecords = maxNumRecords;
-  this.maxReadTime = maxReadTime;
+@Nullable abstract UnboundedSource getSource();
+@Nullable abstract long getMaxNumRecords();
+@Nullable abstract Duration getMaxReadTime();
+
+public abstract String toString();
+
+abstract Builder toBuilder();
+
+@AutoValue.Builder
+abstract static class Builder {
+  abstract Builder setSource(UnboundedSource source);
+  abstract Builder setMaxNumRecords(long maxNumRecords);
+  abstract Builder setMaxReadTime(Duration maxReadTime);
+  abstract UnboundedToBoundedSourceAdapter build();
 }
 
 /**
@@ -174,14 +191,17 @@ public class BoundedReadFromUnboundedSource extends 
PTransform>> 
splitIntoBundles(
 long desiredBundleSizeBytes, PipelineOptions options) throws Exception 
{
   List> result = new ArrayList<>();
-  int numInitialSplits = numInitialSplits(maxNumRecords);
+  int numInitialSplits = numInitialSplits(getMaxNumRecords());
   List> splits =
-  source.generateInitialSplits(numInitialSplits, options);
+  getSource().generateInitialSplits(numInitialSplits, options);
   int numSplits = splits.size();
-  long[] numRecords = splitNumRecords(maxNumRecords, numSplits);
+  long[] numRecords = splitNumRecords(getMaxNumRecords(), numSplits);
   for (int i = 0; i < numSplits; i++) {
-result.add(
-new UnboundedToBoundedSourceAdapter(splits.get(i), 
numRecords[i], maxReadTime));
+result.add(toBuilder()
+.setSource(splits.get(i))
+.setMaxNumRecords(numRecords[i])
+.setMaxReadTime(getMaxReadTime())
+.build());
   }
   return result;
 }
@@ -194,34 +214,34 @@ public class BoundedReadFromUnboundedSource extends 
PTransform> getDefaultOutpu

[jira] [Resolved] (BEAM-708) Migrate BoundedReadFromUnboundedSource to use AutoValue to reduce boilerplate

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-708.

   Resolution: Fixed
 Assignee: Kai Jiang  (was: Davor Bonaci)
Fix Version/s: 0.5.0

> Migrate BoundedReadFromUnboundedSource to use AutoValue to reduce boilerplate
> -
>
> Key: BEAM-708
> URL: https://issues.apache.org/jira/browse/BEAM-708
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Kai Jiang
>Priority: Minor
>  Labels: io, simple, starter
> Fix For: 0.5.0
>
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1315) FileBasedSink could write tmp files above the output prefix.

[Pei He (JIRA)]

Pei He created BEAM-1315:


 Summary: FileBasedSink could write tmp files above the output 
prefix.
 Key: BEAM-1315
 URL: https://issues.apache.org/jira/browse/BEAM-1315
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Pei He
Priority: Minor


FileBasedWriteOperation.TemporaryDirectoryBuilder tempDirBuilder =
new FileBasedWriteOperation.TemporaryDirectoryBuilder();
tempDirBuilder.apply("/home/output/"),

apply() returns "/home/temp-beam-output-2017-01-25_22-14-49"

And, it was not under the output prefix "/home/output/".

Examples of Path resolve:
Paths.get("/home/output").resolve("temp-beam");
Paths.get("/home/output/").resolve("temp-beam");
/home/output/temp-beam
/home/output/temp-beam

Paths.get("/home/output").resolve("..").resolve("temp-beam");
Paths.get("/home/output/").resolve("..").resolve("temp-beam");
/home/output/../temp-beam
/home/output/../temp-beam

Paths.get("/home/output").resolveSibling("temp-beam");
Paths.get("/home/output/").resolveSibling("temp-beam");
/home/temp-beam
/home/temp-beam




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/4] beam git commit: fixup! Hide visibility of internal implementation class

[lcwik]

fixup! Hide visibility of internal implementation class


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a67ff91e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a67ff91e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a67ff91e

Branch: refs/heads/master
Commit: a67ff91e546cb77ad050e6b7573a884f190840cb
Parents: 968c311
Author: Luke Cwik 
Authored: Wed Jan 25 14:13:55 2017 -0800
Committer: Luke Cwik 
Committed: Wed Jan 25 14:13:55 2017 -0800

--
 .../org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a67ff91e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
index f52b822..04e1755 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
@@ -145,7 +145,7 @@ public class BoundedReadFromUnboundedSource extends 
PTransform
+  abstract static class UnboundedToBoundedSourceAdapter
   extends BoundedSource> {
 @Nullable abstract UnboundedSource getSource();
 @Nullable abstract long getMaxNumRecords();

[jira] [Commented] (BEAM-708) Migrate BoundedReadFromUnboundedSource to use AutoValue to reduce boilerplate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838712#comment-15838712
 ] 

ASF GitHub Bot commented on BEAM-708:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1794


> Migrate BoundedReadFromUnboundedSource to use AutoValue to reduce boilerplate
> -
>
> Key: BEAM-708
> URL: https://issues.apache.org/jira/browse/BEAM-708
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Davor Bonaci
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1794: [BEAM-708] Using AutoValue in BoundedReadFromUnboun...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1794


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/4] beam git commit: [BEAM-708] Using AutoValue in BoundedReadFromUnboundedSource

[lcwik]

[BEAM-708] Using AutoValue in BoundedReadFromUnboundedSource

This closes #1794


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6413299a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6413299a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6413299a

Branch: refs/heads/master
Commit: 6413299a20be57de849684479134479fa1acee2d
Parents: 95beda6 a67ff91
Author: Luke Cwik 
Authored: Wed Jan 25 14:22:56 2017 -0800
Committer: Luke Cwik 
Committed: Wed Jan 25 14:22:56 2017 -0800

--
 .../sdk/io/BoundedReadFromUnboundedSource.java  | 69 +---
 1 file changed, 44 insertions(+), 25 deletions(-)
--

[2/4] beam git commit: address comments

[lcwik]

address comments


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/968c3112
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/968c3112
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/968c3112

Branch: refs/heads/master
Commit: 968c31122395d120117ed725aad83d5e3a47e3b1
Parents: eeec9f1
Author: Kai Jiang 
Authored: Wed Jan 25 04:49:35 2017 -0800
Committer: Kai Jiang 
Committed: Wed Jan 25 05:09:33 2017 -0800

--
 .../org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java | 6 ++
 1 file changed, 2 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/968c3112/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
index 7e25a01..f52b822 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSource.java
@@ -141,8 +141,8 @@ public class BoundedReadFromUnboundedSource extends 
PTransform
@@ -151,8 +151,6 @@ public class BoundedReadFromUnboundedSource extends 
PTransform toBuilder();
 
 @AutoValue.Builder

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2103

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-325) Add Slack details to website

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838616#comment-15838616
 ] 

Daniel Halperin commented on BEAM-325:
--

JB, would you mind adding Slack details to website?

> Add Slack details to website
> 
>
> Key: BEAM-325
> URL: https://issues.apache.org/jira/browse/BEAM-325
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: James Malone
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Need to add details on the public Slack channel to the Beam website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-325) Add Slack details to website

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-325:
-
Assignee: Jean-Baptiste Onofré  (was: James Malone)

> Add Slack details to website
> 
>
> Key: BEAM-325
> URL: https://issues.apache.org/jira/browse/BEAM-325
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: James Malone
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Need to add details on the public Slack channel to the Beam website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-325) Add Slack details to website

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838587#comment-15838587
 ] 

Kenneth Knowles commented on BEAM-325:
--

There is an ongoing thread to open up the channel. For now it is invite only 
but we invite anyone who is interested. We should probably document things 
better, in the meantime. Sent you an invite. 

> Add Slack details to website
> 
>
> Key: BEAM-325
> URL: https://issues.apache.org/jira/browse/BEAM-325
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: James Malone
>Assignee: James Malone
>Priority: Minor
>
> Need to add details on the public Slack channel to the Beam website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2102

[Apache Jenkins Server]

See 

[GitHub] beam pull request #1811: Cleanup tests in pipeline_test.

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/beam/pull/1811


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Cleanup tests in pipeline_test.

[robertwb]

Cleanup tests in pipeline_test.

Notably, the runner_name parameter has been obsolete since the removal
of DiskCachedRunnerPipelineTest and is an inferior version of what
TestPipeline provides.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/61d8d3f0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/61d8d3f0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/61d8d3f0

Branch: refs/heads/python-sdk
Commit: 61d8d3f0690142f6dc87b1484d3ebd148a706837
Parents: 9540cf1
Author: Robert Bradshaw 
Authored: Sat Jan 21 21:07:39 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 12:38:03 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py | 57 ++-
 1 file changed, 21 insertions(+), 36 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/61d8d3f0/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index 93b68d1..833293f 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -38,8 +38,8 @@ from apache_beam.transforms import Read
 from apache_beam.transforms import WindowInto
 from apache_beam.transforms.util import assert_that
 from apache_beam.transforms.util import equal_to
-from apache_beam.transforms.window import IntervalWindow
-from apache_beam.transforms.window import WindowFn
+from apache_beam.transforms.window import SlidingWindows
+from apache_beam.transforms.window import TimestampedValue
 from apache_beam.utils.timestamp import MIN_TIMESTAMP
 
 
@@ -70,9 +70,6 @@ class FakeSource(NativeSource):
 
 class PipelineTest(unittest.TestCase):
 
-  def setUp(self):
-self.runner_name = 'DirectRunner'
-
   @staticmethod
   def custom_callable(pcoll):
 return pcoll | '+1' >> FlatMap(lambda x: [x + 1])
@@ -103,7 +100,7 @@ class PipelineTest(unittest.TestCase):
   self.leave_composite.append(transform_node)
 
   def test_create(self):
-pipeline = TestPipeline(runner=self.runner_name)
+pipeline = TestPipeline()
 pcoll = pipeline | 'label1' >> Create([1, 2, 3])
 assert_that(pcoll, equal_to([1, 2, 3]))
 
@@ -114,19 +111,19 @@ class PipelineTest(unittest.TestCase):
 pipeline.run()
 
   def test_create_singleton_pcollection(self):
-pipeline = TestPipeline(runner=self.runner_name)
+pipeline = TestPipeline()
 pcoll = pipeline | 'label' >> Create([[1, 2, 3]])
 assert_that(pcoll, equal_to([[1, 2, 3]]))
 pipeline.run()
 
   def test_read(self):
-pipeline = TestPipeline(runner=self.runner_name)
+pipeline = TestPipeline()
 pcoll = pipeline | 'read' >> Read(FakeSource([1, 2, 3]))
 assert_that(pcoll, equal_to([1, 2, 3]))
 pipeline.run()
 
   def test_visit_entire_graph(self):
-pipeline = Pipeline(self.runner_name)
+pipeline = Pipeline()
 pcoll1 = pipeline | 'pcoll' >> Create([1, 2, 3])
 pcoll2 = pcoll1 | 'do1' >> FlatMap(lambda x: [x + 1])
 pcoll3 = pcoll2 | 'do2' >> FlatMap(lambda x: [x + 1])
@@ -145,14 +142,14 @@ class PipelineTest(unittest.TestCase):
 self.assertEqual(visitor.leave_composite[0].transform, transform)
 
   def test_apply_custom_transform(self):
-pipeline = TestPipeline(runner=self.runner_name)
+pipeline = TestPipeline()
 pcoll = pipeline | 'pcoll' >> Create([1, 2, 3])
 result = pcoll | PipelineTest.CustomTransform()
 assert_that(result, equal_to([2, 3, 4]))
 pipeline.run()
 
   def test_reuse_custom_transform_instance(self):
-pipeline = Pipeline(self.runner_name)
+pipeline = Pipeline()
 pcoll1 = pipeline | 'pcoll1' >> Create([1, 2, 3])
 pcoll2 = pipeline | 'pcoll2' >> Create([4, 5, 6])
 transform = PipelineTest.CustomTransform()
@@ -167,7 +164,7 @@ class PipelineTest(unittest.TestCase):
 'pvalue | "label" >> transform')
 
   def test_reuse_cloned_custom_transform_instance(self):
-pipeline = TestPipeline(runner=self.runner_name)
+pipeline = TestPipeline()
 pcoll1 = pipeline | 'pc1' >> Create([1, 2, 3])
 pcoll2 = pipeline | 'pc2' >> Create([4, 5, 6])
 transform = PipelineTest.CustomTransform()
@@ -240,7 +237,7 @@ class PipelineTest(unittest.TestCase):
 def raise_exception(exn):
   raise exn
 with self.assertRaises(ValueError):
-  with Pipeline(self.runner_name) as p:
+  with Pipeline() as p:
 # pylint: disable=expression-not-assigned
 p | Create([ValueError]) | Map(raise_exception)
 
@@ -251,15 +248,12 @@ class PipelineTest(unittest.TestCase):
 
 class NewDoFnTest(unittest.TestCase):
 
-  def setUp(self):
-self.runner_name = 'DirectRunner'
-
   def test_element(self):
 class TestDoFn(NewDoFn):
   def process(self, element):
 yield el

[1/2] beam git commit: Closes #1811

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/python-sdk 9540cf176 -> 592422059


Closes #1811


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59242205
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59242205
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59242205

Branch: refs/heads/python-sdk
Commit: 592422059e21bf72fc7b4842d6fd6df000a7d2a7
Parents: 9540cf1 61d8d3f
Author: Robert Bradshaw 
Authored: Wed Jan 25 12:38:03 2017 -0800
Committer: Robert Bradshaw 
Committed: Wed Jan 25 12:38:03 2017 -0800

--
 sdks/python/apache_beam/pipeline_test.py | 57 ++-
 1 file changed, 21 insertions(+), 36 deletions(-)
--

[jira] [Commented] (BEAM-246) Streamline default Maven profile for user efficiency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838493#comment-15838493
 ] 

ASF GitHub Bot commented on BEAM-246:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1846


> Streamline default Maven profile for user efficiency
> 
>
> Key: BEAM-246
> URL: https://issues.apache.org/jira/browse/BEAM-246
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.4.0
>
>
> We should optimize the default maven profile for interactive human use. Right 
> now it does a lot of things that waste time & CPU and do not speed 
> development, like building the source jar and javadoc.
> We certainly still want to build all of these things in release tests, 
> probably in post-commit tests, and case-by-case in pre-commit tests. But for 
> these we can set up a profile to activate them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1846: [BEAM-246] re-enable Checkstyle by default

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1846


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-246] re-enable Checkstyle by default

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master c52578370 -> 95beda69b


[BEAM-246] re-enable Checkstyle by default

This adds 50%+ overhead to a clean build (with testing disabled), but
per dev@ discussion is a huge usability win for contributors and
committers alike.

https://lists.apache.org/thread.html/CAA8k_FKafuon8GEA3CXwR2MZh2kAXEFZQK=bgx5tk2fzjeb...@mail.gmail.com


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f05c5d32
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f05c5d32
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f05c5d32

Branch: refs/heads/master
Commit: f05c5d32cb5dbee6de4247a803d7b7c7fbe52173
Parents: c525783
Author: Dan Halperin 
Authored: Tue Jan 24 13:52:06 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 12:13:33 2017 -0800

--
 examples/pom.xml | 14 +-
 runners/pom.xml  | 14 +-
 sdks/pom.xml | 13 +++--
 3 files changed, 25 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f05c5d32/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index adfbaa9..4294c2d 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -51,11 +51,6 @@
   
 
   
-org.apache.maven.plugins
-maven-checkstyle-plugin
-  
-
-  
 org.codehaus.mojo
 findbugs-maven-plugin
   
@@ -64,4 +59,13 @@
 
   
 
+  
+
+  
+org.apache.maven.plugins
+maven-checkstyle-plugin
+  
+
+  
+
 

http://git-wip-us.apache.org/repos/asf/beam/blob/f05c5d32/runners/pom.xml
--
diff --git a/runners/pom.xml b/runners/pom.xml
index fb84164..ceaedfe 100644
--- a/runners/pom.xml
+++ b/runners/pom.xml
@@ -47,11 +47,6 @@
   
 
   
-org.apache.maven.plugins
-maven-checkstyle-plugin
-  
-
-  
 org.codehaus.mojo
 findbugs-maven-plugin
   
@@ -99,4 +94,13 @@
   
 
   
+
+  
+
+  
+org.apache.maven.plugins
+maven-checkstyle-plugin
+  
+
+  
 

http://git-wip-us.apache.org/repos/asf/beam/blob/f05c5d32/sdks/pom.xml
--
diff --git a/sdks/pom.xml b/sdks/pom.xml
index bfdfcd9..2682728 100644
--- a/sdks/pom.xml
+++ b/sdks/pom.xml
@@ -42,11 +42,6 @@
   
 
   
-org.apache.maven.plugins
-maven-checkstyle-plugin
-  
-
-  
 org.codehaus.mojo
 findbugs-maven-plugin
   
@@ -58,7 +53,6 @@
   
 
   
-
 
 
   org.apache.maven.plugins
@@ -74,6 +68,13 @@
 
   
 
+
+
+  
+org.apache.maven.plugins
+maven-checkstyle-plugin
+  
+
   
 
 

[2/2] beam git commit: This closes #1846

[dhalperi]

This closes #1846


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/95beda69
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/95beda69
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/95beda69

Branch: refs/heads/master
Commit: 95beda69bff7dfe519422fd19916c7a851dadf55
Parents: c525783 f05c5d3
Author: Dan Halperin 
Authored: Wed Jan 25 12:13:37 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 12:13:37 2017 -0800

--
 examples/pom.xml | 14 +-
 runners/pom.xml  | 14 +-
 sdks/pom.xml | 13 +++--
 3 files changed, 25 insertions(+), 16 deletions(-)
--

[jira] [Created] (BEAM-1314) DoFn per-key lifecycle (Setup/Teardown)

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1314:
--

 Summary: DoFn per-key lifecycle (Setup/Teardown)
 Key: BEAM-1314
 URL: https://issues.apache.org/jira/browse/BEAM-1314
 Project: Beam
  Issue Type: Improvement
  Components: beam-model
Reporter: Eugene Kirpichov
Assignee: Frances Perry


DoFn's that use state and timers are implicitly per-key. Setup/Teardown methods 
are usually used to establish expensive resources - long-standing connections 
and such.

For per-key DoFn's, we'd often want to use these per-key, so it'd be good to 
have ability in the model to ask that there be 1 instance of the DoFn per key, 
reused between e.g. different timer or trigger firings for this key, but not 
used for other keys.

E.g. this would be particularly useful for Splittable DoFn - for its ability to 
reuse expensive resources between checkpoints.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-814) Improve performance when staging files

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-814.
--
   Resolution: Fixed
Fix Version/s: 0.5.0

> Improve performance when staging files
> --
>
> Key: BEAM-814
> URL: https://issues.apache.org/jira/browse/BEAM-814
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.5.0
>
>
> Right now, {{PackageUtil}} hashes/compresses/sizes files, checks if they have 
> already been staged, and uploads them if not, completely serially.
> There is some low-hanging fruit to parallelize this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-814) Improve performance when staging files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838379#comment-15838379
 ] 

ASF GitHub Bot commented on BEAM-814:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1184


> Improve performance when staging files
> --
>
> Key: BEAM-814
> URL: https://issues.apache.org/jira/browse/BEAM-814
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Right now, {{PackageUtil}} hashes/compresses/sizes files, checks if they have 
> already been staged, and uploads them if not, completely serially.
> There is some low-hanging fruit to parallelize this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1184: [BEAM-814] Dataflow: parallelize staging of files

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1184


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: PackageUtil: parallelize staging of files

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 979c9376f -> c52578370


PackageUtil: parallelize staging of files

Proceeds in stages:
1. In parallel, hash and size all files.
2. Sort files by descending size.
3. In parallel, upload files.

Also a little cleanup for Dataflow 2.0:
* proper visibility
* removing some deprecated code
* refactoring into smaller methods.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3ecf7e70
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3ecf7e70
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3ecf7e70

Branch: refs/heads/master
Commit: 3ecf7e70bcc4775d804f096de647d13c407a8d52
Parents: 979c937
Author: Dan Halperin 
Authored: Mon Oct 24 17:27:23 2016 -0700
Committer: Dan Halperin 
Committed: Wed Jan 25 11:03:03 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |   5 +
 .../beam/runners/dataflow/util/GcsStager.java   |  18 +-
 .../beam/runners/dataflow/util/PackageUtil.java | 349 ---
 .../runners/dataflow/util/PackageUtilTest.java  |  42 ++-
 .../org/apache/beam/sdk/options/GcsOptions.java |   4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  12 +
 6 files changed, 281 insertions(+), 149 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3ecf7e70/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index eea5502..9858b3d 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -203,6 +203,11 @@
 
 
 
+  com.google.apis
+  google-api-services-storage
+
+
+
   com.google.auth
   google-auth-library-credentials
 

http://git-wip-us.apache.org/repos/asf/beam/blob/3ecf7e70/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
index 6ca4c3f..53822e3 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java
@@ -17,13 +17,19 @@
  */
 package org.apache.beam.runners.dataflow.util;
 
+import static com.google.common.base.MoreObjects.firstNonNull;
+import static com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.api.services.dataflow.model.DataflowPackage;
+import com.google.api.services.storage.Storage;
 import java.util.List;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.util.GcsUtil;
+import org.apache.beam.sdk.util.GcsUtil.GcsUtilFactory;
+import org.apache.beam.sdk.util.Transport;
 
 /**
  * Utility class for staging files to GCS.
@@ -35,6 +41,7 @@ public class GcsStager implements Stager {
 this.options = options;
   }
 
+  @SuppressWarnings("unused")  // used via reflection
   public static GcsStager fromOptions(PipelineOptions options) {
 return new GcsStager(options.as(DataflowPipelineOptions.class));
   }
@@ -48,7 +55,16 @@ public class GcsStager implements Stager {
 if (windmillBinary != null) {
   filesToStage.add("windmill_main=" + windmillBinary);
 }
+int uploadSizeBytes = firstNonNull(options.getGcsUploadBufferSizeBytes(), 
1024 * 1024);
+checkArgument(uploadSizeBytes > 0, "gcsUploadBufferSizeBytes must be > 0");
+uploadSizeBytes = Math.min(uploadSizeBytes, 1024 * 1024);
+Storage.Builder storageBuilder = Transport.newStorageClient(options);
+GcsUtil util = GcsUtilFactory.create(
+storageBuilder.build(),
+storageBuilder.getHttpRequestInitializer(),
+options.getExecutorService(),
+uploadSizeBytes);
 return PackageUtil.stageClasspathElements(
-options.getFilesToStage(), options.getStagingLocation());
+options.getFilesToStage(), options.getStagingLocation(), util);
   }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/3ecf7e70/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow

[2/2] beam git commit: This closes #1184

[dhalperi]

This closes #1184


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c5257837
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c5257837
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c5257837

Branch: refs/heads/master
Commit: c525783704e0cc47845df8cdec1715e1f1c74008
Parents: 979c937 3ecf7e7
Author: Dan Halperin 
Authored: Wed Jan 25 11:03:05 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 11:03:05 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml  |   5 +
 .../beam/runners/dataflow/util/GcsStager.java   |  18 +-
 .../beam/runners/dataflow/util/PackageUtil.java | 349 ---
 .../runners/dataflow/util/PackageUtilTest.java  |  42 ++-
 .../org/apache/beam/sdk/options/GcsOptions.java |   4 +-
 .../java/org/apache/beam/sdk/util/GcsUtil.java  |  12 +
 6 files changed, 281 insertions(+), 149 deletions(-)
--

[jira] [Commented] (BEAM-246) Streamline default Maven profile for user efficiency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838347#comment-15838347
 ] 

ASF GitHub Bot commented on BEAM-246:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1846

[BEAM-246] re-enable Checkstyle by default

This adds 50%+ overhead to a clean build (with testing disabled), but
per dev@ discussion is a huge usability win for contributors and
committers alike.


https://lists.apache.org/thread.html/CAA8k_FKafuon8GEA3CXwR2MZh2kAXEFZQK=bgx5tk2fzjeb...@mail.gmail.com

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam checkstyle-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1846.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1846


commit 0d0eb9bf3fa5b3facf403d486787c579c2e82228
Author: Dan Halperin 
Date:   2017-01-24T21:52:06Z

[BEAM-246] re-enable Checkstyle by default

This adds 50%+ overhead to a clean build (with testing disabled), but
per dev@ discussion is a huge usability win for contributors and
committers alike.


https://lists.apache.org/thread.html/CAA8k_FKafuon8GEA3CXwR2MZh2kAXEFZQK=bgx5tk2fzjeb...@mail.gmail.com




> Streamline default Maven profile for user efficiency
> 
>
> Key: BEAM-246
> URL: https://issues.apache.org/jira/browse/BEAM-246
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.4.0
>
>
> We should optimize the default maven profile for interactive human use. Right 
> now it does a lot of things that waste time & CPU and do not speed 
> development, like building the source jar and javadoc.
> We certainly still want to build all of these things in release tests, 
> probably in post-commit tests, and case-by-case in pre-commit tests. But for 
> these we can set up a profile to activate them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1846: [BEAM-246] re-enable Checkstyle by default

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1846

[BEAM-246] re-enable Checkstyle by default

This adds 50%+ overhead to a clean build (with testing disabled), but
per dev@ discussion is a huge usability win for contributors and
committers alike.


https://lists.apache.org/thread.html/CAA8k_FKafuon8GEA3CXwR2MZh2kAXEFZQK=bgx5tk2fzjeb...@mail.gmail.com

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam checkstyle-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1846.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1846


commit 0d0eb9bf3fa5b3facf403d486787c579c2e82228
Author: Dan Halperin 
Date:   2017-01-24T21:52:06Z

[BEAM-246] re-enable Checkstyle by default

This adds 50%+ overhead to a clean build (with testing disabled), but
per dev@ discussion is a huge usability win for contributors and
committers alike.


https://lists.apache.org/thread.html/CAA8k_FKafuon8GEA3CXwR2MZh2kAXEFZQK=bgx5tk2fzjeb...@mail.gmail.com




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1312) Move DataflowRunner properties to pom.xml

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1312.
---
   Resolution: Fixed
Fix Version/s: (was: 0.5.0)
   Not applicable

> Move DataflowRunner properties to pom.xml
> -
>
> Key: BEAM-1312
> URL: https://issues.apache.org/jira/browse/BEAM-1312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> By moving the properties into the pom.xml, it's easier to maintain them.
> This is a forward-port of 
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1312) Move DataflowRunner properties to pom.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838335#comment-15838335
 ] 

ASF GitHub Bot commented on BEAM-1312:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1843


> Move DataflowRunner properties to pom.xml
> -
>
> Key: BEAM-1312
> URL: https://issues.apache.org/jira/browse/BEAM-1312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: 0.5.0
>
>
> By moving the properties into the pom.xml, it's easier to maintain them.
> This is a forward-port of 
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1843: [BEAM-1312] DataflowRunner: move source for propert...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1843


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: DataflowRunner: move source for properties into pom

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master bf9d45429 -> 979c9376f


DataflowRunner: move source for properties into pom

Also drop unused properties that are now in the root
pom.xml.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e95335f0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e95335f0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e95335f0

Branch: refs/heads/master
Commit: e95335f0f6a94ebe257d8d5ce82bb82205dca95c
Parents: bf9d454
Author: Dan Halperin 
Authored: Wed Jan 25 07:50:57 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 10:40:13 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 4 ++--
 .../org/apache/beam/runners/dataflow/dataflow.properties   | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e95335f0/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 1d05193..eea5502 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,8 +33,8 @@
   jar
 
   
-${maven.build.timestamp}
--MM-dd 
HH:mm
+
beam-master-20170120
+6
   
 
   

http://git-wip-us.apache.org/repos/asf/beam/blob/e95335f0/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
index 9976ed9..47e316c 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
+++ 
b/runners/google-cloud-dataflow-java/src/main/resources/org/apache/beam/runners/dataflow/dataflow.properties
@@ -16,8 +16,8 @@
 #
 # Dataflow runtime properties
 
-environment.major.version=6
+environment.major.version=${dataflow.environment_major_version}
 
-worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20170120
+worker.image.batch=dataflow.gcr.io/v1beta3/beam-java-batch:${dataflow.container_version}
 
-worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20170120
+worker.image.streaming=dataflow.gcr.io/v1beta3/beam-java-streaming:${dataflow.container_version}

[2/2] beam git commit: This closes #1843

[dhalperi]

This closes #1843


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/979c9376
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/979c9376
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/979c9376

Branch: refs/heads/master
Commit: 979c9376f820577bad43c18cc1a7ee86fab9d942
Parents: bf9d454 e95335f
Author: Dan Halperin 
Authored: Wed Jan 25 10:40:16 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 10:40:16 2017 -0800

--
 runners/google-cloud-dataflow-java/pom.xml | 4 ++--
 .../org/apache/beam/runners/dataflow/dataflow.properties   | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
--

[GitHub] beam pull request #1845: Add snippet for reading from compressed text source...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1845

Add snippet for reading from compressed text sources

R: @chamikaramj PTAL

Does this need a beam issue as it is just improving documentation ? 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-add-snippet-for-compressed-text-source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1845.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1845


commit ac9f1eae59248252c9d011219afb26c2b6987e32
Author: Sourabh Bajaj 
Date:   2017-01-25T18:19:40Z

Add snippet for reading from compressed text sources




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1313) beam_PostCommit_Python_Verify failing at #1087

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838253#comment-15838253
 ] 

ASF GitHub Bot commented on BEAM-1313:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1844

[BEAM-1313] Use a temp directory for requirements cache in 
test_with_requirements_file

The test fails in jenkins when there are leftover files in the default 
folder
for the requirements cache either from earlier tests, or from the previous
workspaces.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam reqs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1844.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1844






> beam_PostCommit_Python_Verify failing at #1087
> --
>
> Key: BEAM-1313
> URL: https://issues.apache.org/jira/browse/BEAM-1313
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>
> test_with_requirements_file is failing because of the leftover files from 
> other (or previous) tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1844: [BEAM-1313] Use a temp directory for requirements c...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1844

[BEAM-1313] Use a temp directory for requirements cache in 
test_with_requirements_file

The test fails in jenkins when there are leftover files in the default 
folder
for the requirements cache either from earlier tests, or from the previous
workspaces.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam reqs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1844.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1844






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1313) beam_PostCommit_Python_Verify failing at #1087

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1313:
-

 Summary: beam_PostCommit_Python_Verify failing at #1087
 Key: BEAM-1313
 URL: https://issues.apache.org/jira/browse/BEAM-1313
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay


test_with_requirements_file is failing because of the leftover files from other 
(or previous) tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1311) Flink Translations are not properly 1-to-1 with the PValues, PInputs, or POutputs

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1311.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Flink Translations are not properly 1-to-1 with the PValues, PInputs, or 
> POutputs
> -
>
> Key: BEAM-1311
> URL: https://issues.apache.org/jira/browse/BEAM-1311
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Critical
> Fix For: Not applicable
>
>
> Flatten is broken in streaming, and ParDo.BoundMulti is broken in batch. This 
> has broken all Flink Pipelines in those contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1839: [BEAM-1311] Check that a Multi-Output map contains ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1839


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1311) Flink Translations are not properly 1-to-1 with the PValues, PInputs, or POutputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838133#comment-15838133
 ] 

ASF GitHub Bot commented on BEAM-1311:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1839


> Flink Translations are not properly 1-to-1 with the PValues, PInputs, or 
> POutputs
> -
>
> Key: BEAM-1311
> URL: https://issues.apache.org/jira/browse/BEAM-1311
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Critical
>
> Flatten is broken in streaming, and ParDo.BoundMulti is broken in batch. This 
> has broken all Flink Pipelines in those contexts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] beam git commit: This closes #1839

[dhalperi]

This closes #1839


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bf9d4542
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bf9d4542
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bf9d4542

Branch: refs/heads/master
Commit: bf9d454290bba7fac8829b2edeb416b7d9606062
Parents: 7402d76 a361b65
Author: Dan Halperin 
Authored: Wed Jan 25 09:03:51 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 09:03:51 2017 -0800

--
 .../runners/flink/translation/FlinkBatchTransformTranslators.java  | 2 +-
 .../flink/translation/FlinkStreamingTranslationContext.java| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[1/2] beam git commit: Fix Flink RunnableOnService tests

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 7402d7600 -> bf9d45429


Fix Flink RunnableOnService tests

* Check that a Multi-Output map contains the Tag, not the TaggedValue

* Return Inputs from getInputs

  Don't return outputs.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a361b65d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a361b65d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a361b65d

Branch: refs/heads/master
Commit: a361b65d6aa56d70769403d884abf48d1e1141a4
Parents: 7402d76
Author: Thomas Groh 
Authored: Tue Jan 24 17:41:07 2017 -0800
Committer: Dan Halperin 
Committed: Wed Jan 25 09:03:23 2017 -0800

--
 .../runners/flink/translation/FlinkBatchTransformTranslators.java  | 2 +-
 .../flink/translation/FlinkStreamingTranslationContext.java| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a361b65d/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 654b464..f7f1878 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -580,7 +580,7 @@ class FlinkBatchTransformTranslators {
   outputMap.put(transform.getMainOutputTag(), 0);
   int count = 1;
   for (TaggedPValue taggedValue : outputs) {
-if (!outputMap.containsKey(taggedValue)) {
+if (!outputMap.containsKey(taggedValue.getTag())) {
   outputMap.put(taggedValue.getTag(), count++);
 }
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/a361b65d/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTranslationContext.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTranslationContext.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTranslationContext.java
index 6db252e..7932f68 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTranslationContext.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTranslationContext.java
@@ -107,7 +107,7 @@ public class FlinkStreamingTranslationContext {
   }
 
   public  List getInputs(PTransform 
transform) {
-return currentTransform.getOutputs();
+return currentTransform.getInputs();
   }
 
   @SuppressWarnings("unchecked")

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Flink #1441

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1312) Move DataflowRunner properties to pom.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837954#comment-15837954
 ] 

ASF GitHub Bot commented on BEAM-1312:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1843

[BEAM-1312] DataflowRunner: move source for properties into pom

Also drop unused properties that are now in the root
pom.xml.

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam dataflow-runner-pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1843


commit a6621e595ab45428713c4a1674f69517f74b7a10
Author: Dan Halperin 
Date:   2017-01-25T15:50:57Z

DataflowRunner: move source for properties into pom

Also drop unused properties that are now in the root
pom.xml.




> Move DataflowRunner properties to pom.xml
> -
>
> Key: BEAM-1312
> URL: https://issues.apache.org/jira/browse/BEAM-1312
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: 0.5.0
>
>
> By moving the properties into the pom.xml, it's easier to maintain them.
> This is a forward-port of 
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1843: [BEAM-1312] DataflowRunner: move source for propert...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1843

[BEAM-1312] DataflowRunner: move source for properties into pom

Also drop unused properties that are now in the root
pom.xml.

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam dataflow-runner-pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1843.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1843


commit a6621e595ab45428713c4a1674f69517f74b7a10
Author: Dan Halperin 
Date:   2017-01-25T15:50:57Z

DataflowRunner: move source for properties into pom

Also drop unused properties that are now in the root
pom.xml.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1312) Move DataflowRunner properties to pom.xml

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1312:
-

 Summary: Move DataflowRunner properties to pom.xml
 Key: BEAM-1312
 URL: https://issues.apache.org/jira/browse/BEAM-1312
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Daniel Halperin
Assignee: Daniel Halperin
 Fix For: 0.5.0


By moving the properties into the pom.xml, it's easier to maintain them.

This is a forward-port of 
https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/526



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-980) Ability to configure the DAG created by Apex Runner

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837916#comment-15837916
 ] 

Thomas Weise commented on BEAM-980:
---

Apex runner does not currently expose the ability for the user to tweak the 
settings that influence physical plan and execution layer (these are called 
"attributes" in Apex). Ultimately the pipeline author knows what individual 
operators do and should be able to specify/package a configuration file that 
the Apex runner will pick up and pass on to the Apex engine to apply to its 
physical plan.


> Ability to configure the DAG created by Apex Runner
> ---
>
> Key: BEAM-980
> URL: https://issues.apache.org/jira/browse/BEAM-980
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> The Beam pipeline is translated to an Apex DAG of operators that have names 
> that are derived from the transforms. In case of composite transforms those 
> look like path names. Apex lets the user configure things like memory, 
> vcores, parallelism through properties/attributes that reference the operator 
> names. The configuration approach needs to be documented and supplemented 
> with an example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-980) Ability to configure the DAG created by Apex Runner

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-980:
--
Assignee: Thomas Weise  (was: Sandeep Deshmukh)

> Ability to configure the DAG created by Apex Runner
> ---
>
> Key: BEAM-980
> URL: https://issues.apache.org/jira/browse/BEAM-980
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>
> The Beam pipeline is translated to an Apex DAG of operators that have names 
> that are derived from the transforms. In case of composite transforms those 
> look like path names. Apex lets the user configure things like memory, 
> vcores, parallelism through properties/attributes that reference the operator 
> names. The configuration approach needs to be documented and supplemented 
> with an example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-980) Ability to configure the DAG created by Apex Runner

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-980:
--
Summary: Ability to configure the DAG created by Apex Runner  (was: 
Document how to configure the DAG created by Apex Runner)

> Ability to configure the DAG created by Apex Runner
> ---
>
> Key: BEAM-980
> URL: https://issues.apache.org/jira/browse/BEAM-980
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Sandeep Deshmukh
>
> The Beam pipeline is translated to an Apex DAG of operators that have names 
> that are derived from the transforms. In case of composite transforms those 
> look like path names. Apex lets the user configure things like memory, 
> vcores, parallelism through properties/attributes that reference the operator 
> names. The configuration approach needs to be documented and supplemented 
> with an example.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-325) Add Slack details to website

[Jonathan Bergknoff (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837898#comment-15837898
 ] 

Jonathan Bergknoff commented on BEAM-325:
-

Is there a public Slack channel? There is a link to 
https://apachebeam.slack.com/ on the project site, but it appears that I cannot 
log in without an @google.com or @apache.org email address.

> Add Slack details to website
> 
>
> Key: BEAM-325
> URL: https://issues.apache.org/jira/browse/BEAM-325
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: James Malone
>Assignee: James Malone
>Priority: Minor
>
> Need to add details on the public Slack channel to the Beam website.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #1087

[Apache Jenkins Server]

See 

--
[...truncated 915 lines...]
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_invalid_ellipsis_type_param 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_params_must_be_type_or_constraint 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_raw_tuple (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_invalid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_be_tuple 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_must_have_same_arity 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_composite_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_type_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_type_check_valid_simple_types 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_duplicates_ignored 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_must_be_valid_type_param_cant_be_object_instance 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_getitem_nested_unions_flattened 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_nested_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_compatibility 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_composite_type_in_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_not_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_enforcement_part_of_union 
(apache_beam.typehints.typehints_test.UnionHintTestCase) ... ok
test_union_hint_repr (apache_beam.typehints.typehints_test.UnionHintTestCase) 
... ok
test_deprecated_with_since_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... ok
test_deprecated_without_current 
(apache_beam.utils.annotations_test.AnnotationTests) ... 

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Flink #1439

[Apache Jenkins Server]

See