[GitHub] incubator-beam pull request #653: Fix min and max timestamp on 32-bit machin...

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/653

Fix min and max timestamp on 32-bit machines



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam fix-maxint

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #653


commit d755b63831092f10461a708451c1983e9b8bb2a2
Author: Charles Chen 
Date:   2016-07-14T07:39:01Z

Fix min and max timestamp on 32-bit machines




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #654: [BEAM-339] Archetype project version shoul...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/654

[BEAM-339] Archetype project version shouldn't be coupled to Beam version

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Follow-up from [a previous archetype 
PR](https://github.com/apache/incubator-beam/pull/444/files/ba55042275bd9b525ee8716e4e1007b7924a647f#r66819150),
 the maven artifact version for the generated project should not be tied to the 
version of Beam. The generated module is a new user project, so the version 
should represent "initial version", i.e. 0.1.

This PR drops the -SNAPSHOT suffix from the version and fixes the version 
to 0.1 in our tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam archetype-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/654.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #654


commit 5b317b852b1fac9f6320e1a074238e6e7da77b79
Author: Scott Wegner 
Date:   2016-07-13T22:59:16Z

Archetype generated projects shouldn't have SNAPSHOT version




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-339) Archetype project version shouldn't be coupled to Beam version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15377149#comment-15377149
 ] 

ASF GitHub Bot commented on BEAM-339:
-

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/654

[BEAM-339] Archetype project version shouldn't be coupled to Beam version

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Follow-up from [a previous archetype 
PR](https://github.com/apache/incubator-beam/pull/444/files/ba55042275bd9b525ee8716e4e1007b7924a647f#r66819150),
 the maven artifact version for the generated project should not be tied to the 
version of Beam. The generated module is a new user project, so the version 
should represent "initial version", i.e. 0.1.

This PR drops the -SNAPSHOT suffix from the version and fixes the version 
to 0.1 in our tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam archetype-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/654.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #654


commit 5b317b852b1fac9f6320e1a074238e6e7da77b79
Author: Scott Wegner 
Date:   2016-07-13T22:59:16Z

Archetype generated projects shouldn't have SNAPSHOT version




> Archetype project version shouldn't be coupled to Beam version
> --
>
> Key: BEAM-339
> URL: https://issues.apache.org/jira/browse/BEAM-339
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> tl;dr: The maven-archetype project has a version reference of 0.1-SNAPSHOT. 
> This is for the user project and shouldn't be tied to Beam versions.
>  
> In the maven-archetype projects, we have a test which injects property values 
> and verify that the generated project matches the expected. One of the 
> injected properties is "version", which is currently set to "0.1-snapshot" to 
> match the Beam project versions. The version property represents the version 
> of the user project being created and thus shouldn't be tied to the Beam 
> versioning. We should change it such that the intended usage is more clear 
> and to test that the version isn't being set from the Beam version.
> See: 
> https://github.com/apache/incubator-beam/pull/444/files/ba55042275bd9b525ee8716e4e1007b7924a647f#r66819150



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #655: Update Dataflow container image

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/655

Update Dataflow container image

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/655.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #655


commit 8030f0d53ce0b783734e7046c1521cdad7044fa2
Author: Daniel Halperin 
Date:   2016-07-14T15:48:37Z

Update Dataflow container image




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #372: [BEAM-242] Enable and fix checkstyle on Fl...

[jbonofre]

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/372


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15377203#comment-15377203
 ] 

ASF GitHub Bot commented on BEAM-242:
-

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/372


> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-286) Reorganize flink runner directories

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15377200#comment-15377200
 ] 

ASF GitHub Bot commented on BEAM-286:
-

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/348


> Reorganize flink runner directories
> ---
>
> Key: BEAM-286
> URL: https://issues.apache.org/jira/browse/BEAM-286
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.2.0-incubating
>
>
> The flink runner Maven module uses two sub-modules: runner and examples. It's 
> the only one which use such layout (compare to spark, dataflow or 
> inprocess/direct runners).
> I will propose a PR to align flink runner module with the other, keeping the 
> examples in a sub-directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #348: [BEAM-286] Reorganize flink runner module ...

[jbonofre]

Github user jbonofre closed the pull request at:

https://github.com/apache/incubator-beam/pull/348


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add type hints to bigshuffle to avoid pickle overhead.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 67a769a9a -> e8c39c798


Add type hints to bigshuffle to avoid pickle overhead.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f858ea93
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f858ea93
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f858ea93

Branch: refs/heads/python-sdk
Commit: f858ea9335b38c67778f47de63e1d1d16dc79fee
Parents: 67a769a
Author: Robert Bradshaw 
Authored: Tue Jul 12 13:05:46 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:17:09 2016 -0700

--
 sdks/python/apache_beam/examples/cookbook/bigshuffle.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f858ea93/sdks/python/apache_beam/examples/cookbook/bigshuffle.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/bigshuffle.py 
b/sdks/python/apache_beam/examples/cookbook/bigshuffle.py
index 692bd52..0b5da02 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigshuffle.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigshuffle.py
@@ -48,11 +48,14 @@ def run(argv=None):
   p = beam.Pipeline(argv=pipeline_args)
 
   # Read the text file[pattern] into a PCollection.
-  lines = p | beam.io.Read('read', beam.io.TextFileSource(known_args.input))
+  lines = p | beam.io.Read(
+  'read', beam.io.TextFileSource(known_args.input,
+ coder=beam.coders.BytesCoder()))
 
   # Count the occurrences of each word.
   output = (lines
-| beam.Map('split', lambda x: (x[:10], x[10:99]))
+| beam.Map('split', lambda x: (x[:10], x[10:99])
+  ).with_output_types(beam.typehints.KV[str, str])
 | beam.GroupByKey('group')
 | beam.FlatMap(
 'format',

[2/2] incubator-beam git commit: Fix typo in Dataflow runner monitoring message

[robertwb]

Fix typo in Dataflow runner monitoring message


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ffbaccaa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ffbaccaa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ffbaccaa

Branch: refs/heads/python-sdk
Commit: ffbaccaa823149ba76e95e1996f9c7a2ba606d3b
Parents: e8c39c7
Author: Charles Chen 
Authored: Tue Jul 12 22:00:22 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:18:05 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ffbaccaa/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index 99c7090..363b8e1 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -466,7 +466,7 @@ class DataflowApplicationClient(object):
 # The response is a Job proto with the id for the new job.
 logging.info('Created job with id: [%s]', response.id)
 logging.info(
-'To accesss the Dataflow monitoring console, please navigate to '
+'To access the Dataflow monitoring console, please navigate to '
 'https://console.developers.google.com/project/%s/dataflow/job/%s',
 self.google_cloud_options.project, response.id)
 

[2/2] incubator-beam git commit: Closes #652

[robertwb]

Closes #652


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b9d81fc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b9d81fc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b9d81fc

Branch: refs/heads/python-sdk
Commit: 2b9d81fcc87ebd3bfa484c034fe46f79b63beef2
Parents: 52c0d89 cd0178b
Author: Robert Bradshaw 
Authored: Thu Jul 14 10:19:06 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:19:06 2016 -0700

--
 sdks/python/setup.py | 1 +
 1 file changed, 1 insertion(+)
--

[2/2] incubator-beam git commit: Closes #645

[robertwb]

Closes #645


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7c0c27ac
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7c0c27ac
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7c0c27ac

Branch: refs/heads/python-sdk
Commit: 7c0c27ac00fd0ab6b8c2b982540e644485efc60e
Parents: 2b9d81f 245facd
Author: Robert Bradshaw 
Authored: Thu Jul 14 10:20:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:20:59 2016 -0700

--
 sdks/python/apache_beam/internal/pickler.py | 18 --
 sdks/python/tox.ini |  1 +
 2 files changed, 1 insertion(+), 18 deletions(-)
--

[1/2] incubator-beam git commit: Temporarily reverting pickler changes (@4e2d8ab).

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 2b9d81fcc -> 7c0c27ac0


Temporarily reverting pickler changes (@4e2d8ab).

It is causing infinite recursion when some tests are directly
invoked outside the testing framework.

Added one such test to tox tests for testing failing case
(directly by main function).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/245facdb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/245facdb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/245facdb

Branch: refs/heads/python-sdk
Commit: 245facdbc166298cdb96521165a7514653fa8e57
Parents: 2b9d81f
Author: Ahmet Altay 
Authored: Wed Jul 13 10:18:03 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:20:29 2016 -0700

--
 sdks/python/apache_beam/internal/pickler.py | 18 --
 sdks/python/tox.ini |  1 +
 2 files changed, 1 insertion(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/245facdb/sdks/python/apache_beam/internal/pickler.py
--
diff --git a/sdks/python/apache_beam/internal/pickler.py 
b/sdks/python/apache_beam/internal/pickler.py
index f427aa5..898e04b 100644
--- a/sdks/python/apache_beam/internal/pickler.py
+++ b/sdks/python/apache_beam/internal/pickler.py
@@ -159,24 +159,6 @@ if 'save_module' in dir(dill.dill):
   return old_save_module_dict(pickler, obj)
   dill.dill.save_module_dict = new_save_module_dict
 
-
-  old_save_function = dill.dill.save_function
-
-  @dill.dill.register(types.FunctionType)
-  def new_save_function(pickler, obj):
-globs = obj.__globals__ if dill.dill.PY3 else obj.func_globals
-if (dill.dill.is_dill(pickler) and globs == pickler._main.__dict__
-and not pickler._recurse):
-  try:
-pickler._recurse = True
-return old_save_function(pickler, obj)
-  finally:
-pickler._recurse = False
-else:
-  return old_save_function(pickler, obj)
-  dill.dill.save_function = new_save_function
-
-
   def _nest_dill_logging():
 """Prefix all dill logging with its depth in the callstack.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/245facdb/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 29674ed..5a2572e 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -31,6 +31,7 @@ deps=
   pep8
   pylint
 commands =
+  python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
   {toxinidir}/run_pylint.sh
 passenv = TRAVIS*

[GitHub] incubator-beam pull request #656: Removed BatchWrite from DataflowRunner

[ianzhou1]

GitHub user ianzhou1 opened a pull request:

https://github.com/apache/incubator-beam/pull/656

Removed BatchWrite from DataflowRunner



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ianzhou1/incubator-beam RemoveBatchWrite

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/656.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #656


commit 83c601c1db8b4747645607f45050507ceda78398
Author: Ian Zhou 
Date:   2016-07-14T17:20:29Z

Removed BatchWrite from DataflowRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #641

[robertwb]

Closes #641


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e8c39c79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e8c39c79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e8c39c79

Branch: refs/heads/python-sdk
Commit: e8c39c798e00444d5b1f6e53c3743fc41ac57b99
Parents: 67a769a f858ea9
Author: Robert Bradshaw 
Authored: Thu Jul 14 10:17:10 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:17:10 2016 -0700

--
 sdks/python/apache_beam/examples/cookbook/bigshuffle.py | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: Closes #644

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk e8c39c798 -> 52c0d893e


Closes #644


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/52c0d893
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/52c0d893
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/52c0d893

Branch: refs/heads/python-sdk
Commit: 52c0d893e73d8dae05acc5f5a6f16840ea79404f
Parents: e8c39c7 ffbacca
Author: Robert Bradshaw 
Authored: Thu Jul 14 10:18:05 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:18:05 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Added cy_combiners.py to the list of Cythonized Python SDK files.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 52c0d893e -> 2b9d81fcc


Added cy_combiners.py to the list of Cythonized Python SDK files.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cd0178b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cd0178b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cd0178b1

Branch: refs/heads/python-sdk
Commit: cd0178b1a2918a15005bdd485b0a554b1e3e6378
Parents: 52c0d89
Author: Marian Dvorsky 
Authored: Wed Jul 13 17:16:58 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 10:19:05 2016 -0700

--
 sdks/python/setup.py | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cd0178b1/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 789d976..2287b8e 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -75,6 +75,7 @@ setuptools.setup(
 '**/*.pyx',
 'apache_beam/coders/coder_impl.py',
 'apache_beam/runners/common.py',
+'apache_beam/transforms/cy_combiners.py',
 'apache_beam/utils/counters.py',
 ]),
 install_requires=REQUIRED_PACKAGES,

[GitHub] incubator-beam pull request #641: Add type hints to bigshuffle to avoid pick...

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/641


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #625: Made checksum_output optional in bigshuffl...

[mdvorsky]

Github user mdvorsky closed the pull request at:

https://github.com/apache/incubator-beam/pull/625


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #645: Temporarily reverting pickler changes

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/645


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #658: DoOutputsTuple cleanup

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/658

DoOutputsTuple cleanup

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This avoids the same (logical) PCollection from being re-created
with a different producer.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner-pvalues

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/658.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #658


commit e2eb3b520ce742bbf4aef63046ed2dfdb96c32e5
Author: Robert Bradshaw 
Date:   2016-07-14T18:03:41Z

DoOutputsTuple cleanup

This avoids the same (logical) PCollection from being re-created
with a different producer.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #657: Accept runners by fully qualified name.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/657

Accept runners by fully qualified name.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/657.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #657


commit 683262903817e012efbc649d8fe73532876b7c5b
Author: Robert Bradshaw 
Date:   2016-07-14T17:53:09Z

Accept runners by fully qualified name.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-339] Archetype project version shouldn't be coupled to Beam version

[lcwik]

[BEAM-339] Archetype project version shouldn't be coupled to Beam version

This closes #654


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a30f523
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a30f523
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a30f523

Branch: refs/heads/master
Commit: 2a30f523683529b0a41e4cb6b3c150f024283322
Parents: f0119b2 5b317b8
Author: Luke Cwik 
Authored: Thu Jul 14 11:34:06 2016 -0700
Committer: Luke Cwik 
Committed: Thu Jul 14 11:34:06 2016 -0700

--
 .../src/test/resources/projects/basic/archetype.properties | 2 +-
 .../starter/src/test/resources/projects/basic/archetype.properties | 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--

[1/2] incubator-beam git commit: Archetype generated projects shouldn't have SNAPSHOT version

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f0119b23a -> 2a30f5236


Archetype generated projects shouldn't have SNAPSHOT version


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5b317b85
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5b317b85
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5b317b85

Branch: refs/heads/master
Commit: 5b317b852b1fac9f6320e1a074238e6e7da77b79
Parents: f0119b2
Author: Scott Wegner 
Authored: Wed Jul 13 15:59:16 2016 -0700
Committer: Scott Wegner 
Committed: Wed Jul 13 15:59:16 2016 -0700

--
 .../src/test/resources/projects/basic/archetype.properties | 2 +-
 .../starter/src/test/resources/projects/basic/archetype.properties | 2 +-
 .../starter/src/test/resources/projects/basic/reference/pom.xml| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5b317b85/sdks/java/maven-archetypes/examples/src/test/resources/projects/basic/archetype.properties
--
diff --git 
a/sdks/java/maven-archetypes/examples/src/test/resources/projects/basic/archetype.properties
 
b/sdks/java/maven-archetypes/examples/src/test/resources/projects/basic/archetype.properties
index 35eb1b3..9483a44 100644
--- 
a/sdks/java/maven-archetypes/examples/src/test/resources/projects/basic/archetype.properties
+++ 
b/sdks/java/maven-archetypes/examples/src/test/resources/projects/basic/archetype.properties
@@ -15,7 +15,7 @@
 #limitations under the License.
 #
 package=it.pkg
-version=0.1-SNAPSHOT
+version=0.1
 groupId=archetype.it
 artifactId=basic
 targetPlatform=1.7

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5b317b85/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/archetype.properties
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/archetype.properties
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/archetype.properties
index 35eb1b3..9483a44 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/archetype.properties
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/archetype.properties
@@ -15,7 +15,7 @@
 #limitations under the License.
 #
 package=it.pkg
-version=0.1-SNAPSHOT
+version=0.1
 groupId=archetype.it
 artifactId=basic
 targetPlatform=1.7

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5b317b85/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
--
diff --git 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
index 6ae83d5..a778fb6 100644
--- 
a/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
+++ 
b/sdks/java/maven-archetypes/starter/src/test/resources/projects/basic/reference/pom.xml
@@ -22,7 +22,7 @@
 
   archetype.it
   basic
-  0.1-SNAPSHOT
+  0.1
 
   

[GitHub] incubator-beam pull request #654: [BEAM-339] Archetype project version shoul...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/654


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-339) Archetype project version shouldn't be coupled to Beam version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378091#comment-15378091
 ] 

ASF GitHub Bot commented on BEAM-339:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/654


> Archetype project version shouldn't be coupled to Beam version
> --
>
> Key: BEAM-339
> URL: https://issues.apache.org/jira/browse/BEAM-339
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>
> tl;dr: The maven-archetype project has a version reference of 0.1-SNAPSHOT. 
> This is for the user project and shouldn't be tied to Beam versions.
>  
> In the maven-archetype projects, we have a test which injects property values 
> and verify that the generated project matches the expected. One of the 
> injected properties is "version", which is currently set to "0.1-snapshot" to 
> match the Beam project versions. The version property represents the version 
> of the user project being created and thus shouldn't be tied to the Beam 
> versioning. We should change it such that the intended usage is more clear 
> and to test that the version isn't being set from the Beam version.
> See: 
> https://github.com/apache/incubator-beam/pull/444/files/ba55042275bd9b525ee8716e4e1007b7924a647f#r66819150



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #658

[robertwb]

Closes #658


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/762a2930
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/762a2930
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/762a2930

Branch: refs/heads/python-sdk
Commit: 762a2930a093c19580d9f631b923c126aceccc02
Parents: 7c0c27a e2eb3b5
Author: Robert Bradshaw 
Authored: Thu Jul 14 11:43:40 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 11:43:40 2016 -0700

--
 sdks/python/apache_beam/pvalue.py | 28 
 1 file changed, 16 insertions(+), 12 deletions(-)
--

[1/2] incubator-beam git commit: DoOutputsTuple cleanup

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 7c0c27ac0 -> 762a2930a


DoOutputsTuple cleanup

This avoids the same (logical) PCollection from being re-created
with a different producer.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e2eb3b52
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e2eb3b52
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e2eb3b52

Branch: refs/heads/python-sdk
Commit: e2eb3b520ce742bbf4aef63046ed2dfdb96c32e5
Parents: 7c0c27a
Author: Robert Bradshaw 
Authored: Thu Jul 14 11:03:41 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 11:03:41 2016 -0700

--
 sdks/python/apache_beam/pvalue.py | 28 
 1 file changed, 16 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e2eb3b52/sdks/python/apache_beam/pvalue.py
--
diff --git a/sdks/python/apache_beam/pvalue.py 
b/sdks/python/apache_beam/pvalue.py
index 8552a45..6fc3041 100644
--- a/sdks/python/apache_beam/pvalue.py
+++ b/sdks/python/apache_beam/pvalue.py
@@ -58,15 +58,15 @@ class PValue(object):
 self.producer = None
 
   def __str__(self):
-return '<%s>' % self._str_internal()
+return self._str_internal()
 
   def __repr__(self):
 return '<%s at %s>' % (self._str_internal(), hex(id(self)))
 
   def _str_internal(self):
-return '%s transform=%s' % (
-self.__class__.__name__,
-self.producer.transform if self.producer else 'n/a')
+return "%s[%s.%s]" % (self.__class__.__name__,
+  self.producer.full_label if self.producer else None,
+  self.tag)
 
   def apply(self, *args, **kwargs):
 """Applies a transform or callable to a PValue.
@@ -191,16 +191,20 @@ class DoOutputsTuple(object):
 # Check if we accessed this tag before.
 if tag in self._pcolls:
   return self._pcolls[tag]
+
 if tag is not None:
   self._transform.side_output_tags.add(tag)
-pcoll = PCollection(self._pipeline, tag=tag)
-# Transfer the producer from the DoOutputsTuple to the resulting
-# PCollection.
-pcoll.producer = self.producer
-# Add this as an output to both the inner ParDo and the outer _MultiParDo
-# PTransforms.
-self.producer.parts[0].add_output(pcoll, tag)
-self.producer.add_output(pcoll, tag)
+  pcoll = PCollection(self._pipeline, tag=tag)
+  # Transfer the producer from the DoOutputsTuple to the resulting
+  # PCollection.
+  pcoll.producer = self.producer
+  # Add this as an output to both the inner ParDo and the outer _MultiParDo
+  # PTransforms.
+  self.producer.parts[0].add_output(pcoll, tag)
+  self.producer.add_output(pcoll, tag)
+else:
+  # Main output is output of inner ParDo.
+  pcoll = self.producer.parts[0].outputs[0]
 self._pcolls[tag] = pcoll
 return pcoll
 

[GitHub] incubator-beam pull request #658: DoOutputsTuple cleanup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/658


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-339) Archetype project version shouldn't be coupled to Beam version

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-339.

   Resolution: Fixed
Fix Version/s: 0.2.0-incubating

> Archetype project version shouldn't be coupled to Beam version
> --
>
> Key: BEAM-339
> URL: https://issues.apache.org/jira/browse/BEAM-339
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> tl;dr: The maven-archetype project has a version reference of 0.1-SNAPSHOT. 
> This is for the user project and shouldn't be tied to Beam versions.
>  
> In the maven-archetype projects, we have a test which injects property values 
> and verify that the generated project matches the expected. One of the 
> injected properties is "version", which is currently set to "0.1-snapshot" to 
> match the Beam project versions. The version property represents the version 
> of the user project being created and thus shouldn't be tied to the Beam 
> versioning. We should change it such that the intended usage is more clear 
> and to test that the version isn't being set from the Beam version.
> See: 
> https://github.com/apache/incubator-beam/pull/444/files/ba55042275bd9b525ee8716e4e1007b7924a647f#r66819150



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #659: Fix DoFnTester side inputs

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/659

Fix DoFnTester side inputs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The side inputs were being stored as iterables, but being returned as
the raw type.

Store the side input values directly instead.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam do_fn_tester_side_inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/659.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #659


commit 91078cbe86dac5d7f145fa95f16b73223c3abfd5
Author: Thomas Groh 
Date:   2016-07-14T17:33:22Z

Fix DoFnTester side inputs

The side inputs were being stored as iterables, but being returned as
the raw type.

Store the side input values directly instead.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-450) Modules are shaded to the same path

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-450:


 Summary: Modules are shaded to the same path
 Key: BEAM-450
 URL: https://issues.apache.org/jira/browse/BEAM-450
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.1.0-incubating, 0.2.0-incubating
Reporter: Daniel Halperin


Right now multiple modules are using the same repackaged path. We should be 
using per-artifact paths so that they don't conflict.

One proposal was simply to adopt 
{{${project.groupId}.${project.artifactId}.repackaged}} as the shading 
location. If it works.

This is a good starter issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-450) Modules are shaded to the same path

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-450:
-
Labels: newbie starter  (was: )

> Modules are shaded to the same path
> ---
>
> Key: BEAM-450
> URL: https://issues.apache.org/jira/browse/BEAM-450
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>  Labels: newbie, starter
>
> Right now multiple modules are using the same repackaged path. We should be 
> using per-artifact paths so that they don't conflict.
> One proposal was simply to adopt 
> {{${project.groupId}.${project.artifactId}.repackaged}} as the shading 
> location. If it works.
> This is a good starter issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-451) When producing bundled jars, service files are not concatenated

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-451:
-
Labels: newbie starter  (was: )

> When producing bundled jars, service files are not concatenated
> ---
>
> Key: BEAM-451
> URL: https://issues.apache.org/jira/browse/BEAM-451
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>  Labels: newbie, starter
>
> This means that AutoService configs from different modules are lost, aka, the 
> PipelineOptions and runners from different modules may not survive bundling.
> Looks like this: 
> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#Concatenating_Service_Entries_with_the_ServicesResourceTransformer
>  may be a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-451) When producing bundled jars, service files are not concatenated

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-451:


 Summary: When producing bundled jars, service files are not 
concatenated
 Key: BEAM-451
 URL: https://issues.apache.org/jira/browse/BEAM-451
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.1.0-incubating, 0.2.0-incubating
Reporter: Daniel Halperin


This means that AutoService configs from different modules are lost, aka, the 
PipelineOptions and runners from different modules may not survive bundling.

Looks like this: 
https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#Concatenating_Service_Entries_with_the_ServicesResourceTransformer
 may be a solution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-435) DirectRunner GBK -- task per key?

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378247#comment-15378247
 ] 

Thomas Groh commented on BEAM-435:
--

Yes. Bundle processing is keyed after a GroupByKey to ensure that per-key state 
is accessed only by the appropriate key.

> DirectRunner GBK -- task per key?
> -
>
> Key: BEAM-435
> URL: https://issues.apache.org/jira/browse/BEAM-435
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
>
> See [BEAM-434] -- is the direct runner producing a bundle per-key in the GBK 
> output?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-452) Implement DoFn per-instance setup and teardown methods

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-452:


 Summary: Implement DoFn per-instance setup and teardown methods
 Key: BEAM-452
 URL: https://issues.apache.org/jira/browse/BEAM-452
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow, runner-direct, runner-flink, 
runner-spark, sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


https://docs.google.com/document/d/1LLQqggSePURt3XavKBGV7SZJYQ4NW8yCu63lBchzMRk/edit

BEAM-38 permits DoFns to be reused across bundles. DoFn instances may need to 
do per-instance setup and teardown, and to avoid redoing the work per-bundle, 
the system should provide hooks to call before a DoFn is first used and after 
it will no longer be used.

DoFn#setup is called before any other calls to DoFn methods. DoFn#teardown is 
called after any method throws an exception, or when the runner will no longer 
use a DoFn instance (e.g. when it evicts it from a cache).

Runners must call these methods appropriately in all cases (including if a DoFn 
is used exactly once, for a single bundle, and discarded).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-435) DirectRunner GBK -- task per key?

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378247#comment-15378247
 ] 

Thomas Groh edited comment on BEAM-435 at 7/14/16 7:59 PM:
---

Yes. Bundle processing is keyed after a GroupByKey to ensure that per-key state 
is accessed only by the appropriate key. Writing a limited number of output 
files requires bundles to be merged, either via the runner doing so or adding 
an intervening transform to do so.


was (Author: tgroh):
Yes. Bundle processing is keyed after a GroupByKey to ensure that per-key state 
is accessed only by the appropriate key.

> DirectRunner GBK -- task per key?
> -
>
> Key: BEAM-435
> URL: https://issues.apache.org/jira/browse/BEAM-435
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
>
> See [BEAM-434] -- is the direct runner producing a bundle per-key in the GBK 
> output?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: Closes #657

[robertwb]

Closes #657


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b695068
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3b695068
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3b695068

Branch: refs/heads/python-sdk
Commit: 3b69506897db7f21cc09eaa5ac8c08e4ee15ad2c
Parents: 762a293 c055e84
Author: Robert Bradshaw 
Authored: Thu Jul 14 14:16:09 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 14:16:09 2016 -0700

--
 .../apache_beam/runners/dataflow_runner.py  |  4 ++
 sdks/python/apache_beam/runners/runner.py   | 39 ++--
 sdks/python/apache_beam/runners/runner_test.py  |  2 +-
 3 files changed, 24 insertions(+), 21 deletions(-)
--

[1/3] incubator-beam git commit: Accept runners by fully qualified name.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 762a2930a -> 3b6950689


Accept runners by fully qualified name.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/84fe8954
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/84fe8954
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/84fe8954

Branch: refs/heads/python-sdk
Commit: 84fe8954669ef9a30448d140b5a578c14b863819
Parents: 762a293
Author: Robert Bradshaw 
Authored: Thu Jul 14 10:53:09 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 14:15:59 2016 -0700

--
 sdks/python/apache_beam/runners/runner.py  | 28 ++---
 sdks/python/apache_beam/runners/runner_test.py |  2 +-
 2 files changed, 14 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/84fe8954/sdks/python/apache_beam/runners/runner.py
--
diff --git a/sdks/python/apache_beam/runners/runner.py 
b/sdks/python/apache_beam/runners/runner.py
index 55b63f3..98f9758 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -41,26 +41,24 @@ def create_runner(runner_name):
 RuntimeError: if an invalid runner name is used.
   """
   # pylint: disable=wrong-import-order, wrong-import-position
-  if runner_name == 'DirectPipelineRunner':
-import apache_beam.runners.direct_runner
-return apache_beam.runners.direct_runner.DirectPipelineRunner()
-  if runner_name == 'DiskCachedPipelineRunner':
-import apache_beam.runners.direct_runner
-return apache_beam.runners.direct_runner.DiskCachedPipelineRunner(
-)
-  if runner_name == 'EagerPipelineRunner':
-import apache_beam.runners.direct_runner
-return apache_beam.runners.direct_runner.EagerPipelineRunner()
-  elif runner_name in ('DataflowPipelineRunner',
-   'BlockingDataflowPipelineRunner'):
+  if runner_name in ('DirectPipelineRunner', 'DiskCachedPipelineRunner',
+ 'EagerPipelineRunner'):
+runner_name = 'apache_beam.runners.direct_runner.' + runner_name
+
+  if runner_name in ('DataflowPipelineRunner',
+ 'BlockingDataflowPipelineRunner'):
 import apache_beam.runners.dataflow_runner
 return apache_beam.runners.dataflow_runner.DataflowPipelineRunner(
 blocking=runner_name == 'BlockingDataflowPipelineRunner')
+  elif '.' in runner_name:
+module, runner = runner_name.rsplit('.', 1)
+return getattr(__import__(module, {}, {}, [runner], -1), runner)()
   else:
-raise RuntimeError(
+raise ValueError(
 'Unexpected pipeline runner: %s. Valid values are '
-'DirectPipelineRunner, DataflowPipelineRunner, EagerPipelineRunner, or 
'
-'BlockingDataflowPipelineRunner.' % runner_name)
+'DirectPipelineRunner, DataflowPipelineRunner, EagerPipelineRunner, '
+'BlockingDataflowPipelineRunner or the fully qualified name of '
+'a PipelineRunner subclass.' % runner_name)
 
 
 class PipelineRunner(object):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/84fe8954/sdks/python/apache_beam/runners/runner_test.py
--
diff --git a/sdks/python/apache_beam/runners/runner_test.py 
b/sdks/python/apache_beam/runners/runner_test.py
index 20a7259..d2e70d7 100644
--- a/sdks/python/apache_beam/runners/runner_test.py
+++ b/sdks/python/apache_beam/runners/runner_test.py
@@ -44,7 +44,7 @@ class RunnerTest(unittest.TestCase):
 self.assertTrue(
 isinstance(create_runner('BlockingDataflowPipelineRunner'),
DataflowPipelineRunner))
-self.assertRaises(RuntimeError, create_runner, 'xyz')
+self.assertRaises(ValueError, create_runner, 'xyz')
 
   def test_remote_runner_translation(self):
 remote_runner = DataflowPipelineRunner()

[GitHub] incubator-beam pull request #657: Accept runners by fully qualified name.

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/657


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Cleanup known runners code.

[robertwb]

Cleanup known runners code.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c055e845
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c055e845
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c055e845

Branch: refs/heads/python-sdk
Commit: c055e845b11e8a89e26f968c11f7cb8f3943f0d5
Parents: 84fe895
Author: Robert Bradshaw 
Authored: Thu Jul 14 11:56:20 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 14:16:08 2016 -0700

--
 .../apache_beam/runners/dataflow_runner.py  |  4 +++
 sdks/python/apache_beam/runners/runner.py   | 27 ++--
 2 files changed, 18 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c055e845/sdks/python/apache_beam/runners/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow_runner.py
index 5a3f6a5..45bfb6e 100644
--- a/sdks/python/apache_beam/runners/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow_runner.py
@@ -44,6 +44,10 @@ from apache_beam.utils.options import StandardOptions
 from apache_beam.internal.clients import dataflow as dataflow_api
 
 
+def BlockingDataflowPipelineRunner(*args, **kwargs):
+  return DataflowPipelineRunner(*args, blocking=True, **kwargs)
+
+
 class DataflowPipelineRunner(PipelineRunner):
   """A runner that creates job graphs and submits them for remote execution.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c055e845/sdks/python/apache_beam/runners/runner.py
--
diff --git a/sdks/python/apache_beam/runners/runner.py 
b/sdks/python/apache_beam/runners/runner.py
index 98f9758..3a77766 100644
--- a/sdks/python/apache_beam/runners/runner.py
+++ b/sdks/python/apache_beam/runners/runner.py
@@ -26,6 +26,12 @@ import shutil
 import tempfile
 
 
+_KNOWN_DIRECT_RUNNERS = ('DirectPipelineRunner', 'DiskCachedPipelineRunner',
+ 'EagerPipelineRunner')
+_KNOWN_DATAFLOW_RUNNERS = ('DataflowPipelineRunner',
+   'BlockingDataflowPipelineRunner')
+
+
 def create_runner(runner_name):
   """Creates a runner instance from a runner class name.
 
@@ -40,25 +46,20 @@ def create_runner(runner_name):
   Raises:
 RuntimeError: if an invalid runner name is used.
   """
-  # pylint: disable=wrong-import-order, wrong-import-position
-  if runner_name in ('DirectPipelineRunner', 'DiskCachedPipelineRunner',
- 'EagerPipelineRunner'):
+  if runner_name in _KNOWN_DIRECT_RUNNERS:
 runner_name = 'apache_beam.runners.direct_runner.' + runner_name
+  elif runner_name in _KNOWN_DATAFLOW_RUNNERS:
+runner_name = 'apache_beam.runners.dataflow_runner.' + runner_name
 
-  if runner_name in ('DataflowPipelineRunner',
- 'BlockingDataflowPipelineRunner'):
-import apache_beam.runners.dataflow_runner
-return apache_beam.runners.dataflow_runner.DataflowPipelineRunner(
-blocking=runner_name == 'BlockingDataflowPipelineRunner')
-  elif '.' in runner_name:
+  if '.' in runner_name:
 module, runner = runner_name.rsplit('.', 1)
 return getattr(__import__(module, {}, {}, [runner], -1), runner)()
   else:
 raise ValueError(
-'Unexpected pipeline runner: %s. Valid values are '
-'DirectPipelineRunner, DataflowPipelineRunner, EagerPipelineRunner, '
-'BlockingDataflowPipelineRunner or the fully qualified name of '
-'a PipelineRunner subclass.' % runner_name)
+'Unexpected pipeline runner: %s. Valid values are %s '
+'or the fully qualified name of a PipelineRunner subclass.' % (
+runner_name,
+', '.join(_KNOWN_DIRECT_RUNNERS + _KNOWN_DATAFLOW_RUNNERS)))
 
 
 class PipelineRunner(object):

[2/2] incubator-beam git commit: Fix min and max timestamp on 32-bit machines

[robertwb]

Fix min and max timestamp on 32-bit machines


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/afc68bc3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/afc68bc3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/afc68bc3

Branch: refs/heads/python-sdk
Commit: afc68bc3aee9cafbe865628a9683bffdd73853ec
Parents: 3b69506
Author: Charles Chen 
Authored: Thu Jul 14 00:39:01 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 14:18:43 2016 -0700

--
 sdks/python/apache_beam/coders/coders_test_common.py   | 4 ++--
 .../apache_beam/examples/complete/top_wikipedia_sessions.py| 6 +++---
 sdks/python/apache_beam/transforms/timeutil.py | 5 ++---
 3 files changed, 7 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/afc68bc3/sdks/python/apache_beam/coders/coders_test_common.py
--
diff --git a/sdks/python/apache_beam/coders/coders_test_common.py 
b/sdks/python/apache_beam/coders/coders_test_common.py
index 07436cb..0266fdc 100644
--- a/sdks/python/apache_beam/coders/coders_test_common.py
+++ b/sdks/python/apache_beam/coders/coders_test_common.py
@@ -19,7 +19,6 @@
 
 import logging
 import math
-import sys
 import unittest
 
 import dill
@@ -121,9 +120,10 @@ class CodersTest(unittest.TestCase):
 # Multi-byte encoding starts at 128
 self.check_coder(coders.VarIntCoder(), *range(120, 140))
 # Large values
+MAX_64_BIT_INT = 0x7fff
 self.check_coder(coders.VarIntCoder(),
  *[int(math.pow(-1, k) * math.exp(k))
-   for k in range(0, int(math.log(sys.maxint)))])
+   for k in range(0, int(math.log(MAX_64_BIT_INT)))])
 
   def test_float_coder(self):
 self.check_coder(coders.FloatCoder(),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/afc68bc3/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py 
b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
index 55b7857..7337910 100644
--- a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
+++ b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
@@ -42,7 +42,6 @@ from __future__ import absolute_import
 import argparse
 import json
 import logging
-import sys
 
 import apache_beam as beam
 from apache_beam import combiners
@@ -50,6 +49,7 @@ from apache_beam import window
 
 ONE_HOUR_IN_SECONDS = 3600
 THIRTY_DAYS_IN_SECONDS = 30 * 24 * ONE_HOUR_IN_SECONDS
+MAX_TIMESTAMP = 0x7fff
 
 
 class ExtractUserAndTimestampDoFn(beam.DoFn):
@@ -128,8 +128,8 @@ class ComputeTopSessions(beam.PTransform):
 return (pcoll
 | beam.ParDo('ExtractUserAndTimestamp',
  ExtractUserAndTimestampDoFn())
-| beam.Filter(
-lambda x: abs(hash(x)) <= sys.maxint * self.sampling_threshold)
+| beam.Filter(lambda x: (abs(hash(x)) <=
+ MAX_TIMESTAMP * self.sampling_threshold))
 | ComputeSessions()
 | beam.ParDo('SessionsToStrings', SessionsToStringsDoFn())
 | TopPerMonth()

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/afc68bc3/sdks/python/apache_beam/transforms/timeutil.py
--
diff --git a/sdks/python/apache_beam/transforms/timeutil.py 
b/sdks/python/apache_beam/transforms/timeutil.py
index f72a9e4..4092b60 100644
--- a/sdks/python/apache_beam/transforms/timeutil.py
+++ b/sdks/python/apache_beam/transforms/timeutil.py
@@ -23,7 +23,6 @@ from abc import ABCMeta
 from abc import abstractmethod
 
 import datetime
-import sys
 
 
 class Timestamp(object):
@@ -115,8 +114,8 @@ class Timestamp(object):
 return Duration(micros=self.micros % other.micros)
 
 
-MIN_TIMESTAMP = Timestamp(micros=-sys.maxint - 1)
-MAX_TIMESTAMP = Timestamp(micros=sys.maxint)
+MIN_TIMESTAMP = Timestamp(micros=-0x7fff - 1)
+MAX_TIMESTAMP = Timestamp(micros=0x7fff)
 
 
 class Duration(object):

[1/2] incubator-beam git commit: Closes #653

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 3b6950689 -> c8cef2cba


Closes #653


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c8cef2cb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c8cef2cb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c8cef2cb

Branch: refs/heads/python-sdk
Commit: c8cef2cbac10b2a3be79b7087c96e254ce7aedd3
Parents: 3b69506 afc68bc
Author: Robert Bradshaw 
Authored: Thu Jul 14 14:18:43 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 14:18:43 2016 -0700

--
 sdks/python/apache_beam/coders/coders_test_common.py   | 4 ++--
 .../apache_beam/examples/complete/top_wikipedia_sessions.py| 6 +++---
 sdks/python/apache_beam/transforms/timeutil.py | 5 ++---
 3 files changed, 7 insertions(+), 8 deletions(-)
--

[2/2] incubator-beam git commit: This closes #532

[kenn]

This closes #532


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/84573e0b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/84573e0b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/84573e0b

Branch: refs/heads/master
Commit: 84573e0b41735ab9fba25b3883420c1ea8e03e3d
Parents: 2a30f52 ae0293d
Author: Kenneth Knowles 
Authored: Thu Jul 14 14:31:03 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 14:31:03 2016 -0700

--
 .../org/apache/beam/sdk/testing/PAssert.java| 222 ++---
 .../apache/beam/sdk/testing/PaneExtractors.java | 140 
 .../apache/beam/sdk/testing/PAssertTest.java|   4 +-
 .../beam/sdk/testing/PaneExtractorsTest.java| 323 +++
 4 files changed, 649 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/84573e0b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--

[1/2] incubator-beam git commit: Add Pane matchers to PAssert

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2a30f5236 -> 84573e0b4


Add Pane matchers to PAssert

This allows PCollections that have upstream triggers that fire multiple
times to assert the contents of panes with specific timings, rather than
restricting the trigger to fire only once.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ae0293d0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ae0293d0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ae0293d0

Branch: refs/heads/master
Commit: ae0293d0c84e1e49031f62e1026ab7a9b362bbf7
Parents: ff72e27
Author: Thomas Groh 
Authored: Mon Jun 20 14:38:11 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 14:30:49 2016 -0700

--
 .../org/apache/beam/sdk/testing/PAssert.java| 222 ++---
 .../apache/beam/sdk/testing/PaneExtractors.java | 140 
 .../apache/beam/sdk/testing/PAssertTest.java|   4 +-
 .../beam/sdk/testing/PaneExtractorsTest.java| 323 +++
 4 files changed, 649 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ae0293d0/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index 883b2b3..b8bd431 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -17,7 +17,6 @@
  */
 package org.apache.beam.sdk.testing;
 
-import static com.google.common.base.Preconditions.checkState;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.not;
@@ -38,9 +37,11 @@ import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.DoFn.RequiresWindowAccess;
 import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
+import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.transforms.Values;
 import org.apache.beam.sdk.transforms.View;
@@ -49,10 +50,13 @@ import 
org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.Never;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
 import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.GatherAllPanes;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
@@ -63,16 +67,15 @@ import org.apache.beam.sdk.values.PDone;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 
+import org.joda.time.Duration;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
-import java.util.List;
 import java.util.Map;
 import java.util.NoSuchElementException;
 
@@ -127,12 +130,51 @@ public class PAssert {
  * Creates a new {@link IterableAssert} like this one, but with the 
assertion restricted to only
  * run on the provided window.
  *
+ * The assertion will concatenate all panes present in the provided 
window if the
+ * {@link Trigger} produces multiple panes. If the windowing strategy 
accumulates fired panes
+ * and triggers fire multple times, consider using instead {@link 
#inFinalPane(BoundedWindow)}
+ * or {@link #inOnTimePane(BoundedWindow)}.
+ *
  * @return a new {@link IterableAssert} like this one but with the 
assertion only applied to the
  * specified window.
  */
 IterableAssert inWindow(BoundedWindow window);
 
 /**
+ * Creates a new {@link IterableAssert} like this one, but with the 
assertion restricted to only
+ * run on the provided window, running the checker only on the final pane 
for each key.
+ *
+ * If the input {@lin

[GitHub] incubator-beam pull request #532: [BEAM-155] Add Pane matchers to PAssert

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/532


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-155) Support asserting the contents of windows and panes in PAssert

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378477#comment-15378477
 ] 

ASF GitHub Bot commented on BEAM-155:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/532


> Support asserting the contents of windows and panes in PAssert
> --
>
> Key: BEAM-155
> URL: https://issues.apache.org/jira/browse/BEAM-155
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> This consists of reifying the output windows and panes, and running asserts 
> per-window about the contents of panes.
> This includes aggregated matching and final pane matching, e.g.
> PAssert.that(output).byOnTimePane().hasOutputElements(foo, bar);
> // For discarding mode - could have emitted (say) [spam, eggs], [spam], [], 
> [sausage], []
> PAssert.that(output).byFinalPane().hasOutputElements(spam, eggs, sausage, 
> spam);
> // For accumulating mode without late data
> PAssert.that(output).finalPane().containsInAnyOrder(spam, eggs, sausage, 
> spam);
> // For accumulating mode with late data
> PAssert.that(output).finalPane().containsInAnyOrder(foo, 
> bar).mayAlsoContain(baz, rab);
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-155) Support asserting the contents of windows and panes in PAssert

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-155.
--
   Resolution: Fixed
Fix Version/s: 0.2.0-incubating

https://github.com/apache/incubator-beam/pull/532

> Support asserting the contents of windows and panes in PAssert
> --
>
> Key: BEAM-155
> URL: https://issues.apache.org/jira/browse/BEAM-155
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.2.0-incubating
>
>
> This consists of reifying the output windows and panes, and running asserts 
> per-window about the contents of panes.
> This includes aggregated matching and final pane matching, e.g.
> PAssert.that(output).byOnTimePane().hasOutputElements(foo, bar);
> // For discarding mode - could have emitted (say) [spam, eggs], [spam], [], 
> [sausage], []
> PAssert.that(output).byFinalPane().hasOutputElements(spam, eggs, sausage, 
> spam);
> // For accumulating mode without late data
> PAssert.that(output).finalPane().containsInAnyOrder(spam, eggs, sausage, 
> spam);
> // For accumulating mode with late data
> PAssert.that(output).finalPane().containsInAnyOrder(foo, 
> bar).mayAlsoContain(baz, rab);
> See also: 
> https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #660: Bump required Maven version to 3.3

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/660

Bump required Maven version to 3.3

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam maven-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/660.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #660


commit 4a70283eeaa8f3fb8010d396f3dcfbe832049ecd
Author: Scott Wegner 
Date:   2016-07-14T22:50:12Z

Bump required Maven version to 3.3




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #648

[kenn]

This closes #648


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/779fb1a6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/779fb1a6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/779fb1a6

Branch: refs/heads/master
Commit: 779fb1a6cead6ba67434ad752a79b970d2e02c25
Parents: 84573e0 f0941d5
Author: Kenneth Knowles 
Authored: Thu Jul 14 16:25:33 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 16:25:33 2016 -0700

--
 examples/java/pom.xml  | 11 +++
 pom.xml|  1 +
 sdks/java/io/google-cloud-platform/pom.xml |  5 +++--
 3 files changed, 15 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #648: [BEAM-445] Switch to netty-tcnative uber-j...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-445) Beam-examples-java build failed through local "mvn install"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378595#comment-15378595
 ] 

ASF GitHub Bot commented on BEAM-445:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/648


> Beam-examples-java build failed through local "mvn install"
> ---
>
> Key: BEAM-445
> URL: https://issues.apache.org/jira/browse/BEAM-445
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
> Environment: linux
>Reporter: Mark Liu
>Assignee: Daniel Halperin
>Priority: Critical
>
> Build project under beam/examples/java with command "mvn clean install 
> -DskipTests" failed with following error:
> [ERROR] Failed to execute goal on project beam-examples-java: Could not 
> resolve dependencies for project 
> org.apache.beam:beam-examples-java:jar:0.2.0-incubating-SNAPSHOT: Could not 
> transfer artifact 
> io.netty:netty-tcnative-boringssl-static:jar:${os.detected.classifier}:1.1.33.Fork13
>  from/to central (http://repo.maven.apache.org/maven2): Illegal character in 
> path at index 138: 
> http://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork13/netty-tcnative-boringssl-static-1.1.33.Fork13-${os.detected.classifier}.jar
> Reason: can't resolve ${os.detected.classifier} in 
> beam/sdks/java/io/google-cloud-platform/pom file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Add os-maven-plugin to examples module for GCP IO

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 84573e0b4 -> 779fb1a6c


Add os-maven-plugin to examples module for GCP IO


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f0941d5e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f0941d5e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f0941d5e

Branch: refs/heads/master
Commit: f0941d5e05e40d8fb6e347f1af3790f75a14b0a7
Parents: 84573e0
Author: Scott Wegner 
Authored: Wed Jul 13 14:59:01 2016 -0700
Committer: Scott Wegner 
Committed: Thu Jul 14 15:09:26 2016 -0700

--
 examples/java/pom.xml  | 11 +++
 pom.xml|  1 +
 sdks/java/io/google-cloud-platform/pom.xml |  5 +++--
 3 files changed, 15 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f0941d5e/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index ca16f51..ba45c48 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -195,6 +195,17 @@
 
   
 
+
+
+  
+  
+kr.motd.maven
+os-maven-plugin
+${os-maven-plugin.version}
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f0941d5e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 7089c2c..d38768e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -126,6 +126,7 @@
 2.4
 4.11
 1.9.5
+1.4.0.Final
 3.0.0-beta-1
 v1-rev10-1.22.0
 1.7.14

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f0941d5e/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 93165ff..b86f3b9 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -88,11 +88,12 @@
 
 
 
-  
+  
   
 kr.motd.maven
 os-maven-plugin
-1.4.0.Final
+${os-maven-plugin.version}
   
 
   

[1/2] incubator-beam git commit: [BEAM-37] DoFnReflector: Add invoker interface and generate code

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 779fb1a6c -> 79c26d9c1


[BEAM-37] DoFnReflector: Add invoker interface and generate code

The method to call for a DoFnWithContext requires reflection since the
shape of the parameters may change. Doing so in each processElement call
puts this refelection in the hot path.

This PR introduces a DoFnInvoker interface which is bound to a specific
DoFnWithContext and delegates the three important methods (startBundle,
processElement, finishBundle).

It uses byte-buddy to generate a simple trampoline implementation of
the DoFnInvoker class for each type of DoFnWithContext.

This leads to 2-3x better performance in micro-benchmarks of method
dispatching.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b47919c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b47919c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b47919c

Branch: refs/heads/master
Commit: 2b47919c18d6485c3bb3df8452bd67c940f00d65
Parents: 2a30f52
Author: Ben Chambers 
Authored: Wed Jun 22 06:47:23 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 16:41:33 2016 -0700

--
 pom.xml |   6 +
 sdks/java/core/pom.xml  |   5 +
 .../beam/sdk/transforms/DoFnReflector.java  | 717 ++-
 .../beam/sdk/transforms/DoFnReflectorTest.java  | 197 -
 .../transforms/DoFnReflectorBenchmark.java  |   7 +-
 5 files changed, 724 insertions(+), 208 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b47919c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 7089c2c..1513735 100644
--- a/pom.xml
+++ b/pom.xml
@@ -590,6 +590,12 @@
 ${slf4j.version}
   
 
+  
+net.bytebuddy
+byte-buddy
+1.4.3
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b47919c/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index bda77cb..8b6fff7 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -434,6 +434,11 @@
 
 
 
+  net.bytebuddy
+  byte-buddy
+
+
+
   org.apache.avro
   avro
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b47919c/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnReflector.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnReflector.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnReflector.java
index e711d04..116b64d 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnReflector.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnReflector.java
@@ -34,15 +34,48 @@ import org.apache.beam.sdk.values.TypeDescriptor;
 
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Function;
+import com.google.common.base.Preconditions;
 import com.google.common.collect.FluentIterable;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.reflect.TypeParameter;
 import com.google.common.reflect.TypeToken;
 
+import net.bytebuddy.ByteBuddy;
+import net.bytebuddy.description.field.FieldDescription;
+import net.bytebuddy.description.method.MethodDescription;
+import net.bytebuddy.description.method.ParameterList;
+import net.bytebuddy.description.modifier.FieldManifestation;
+import net.bytebuddy.description.modifier.Visibility;
+import net.bytebuddy.description.type.TypeDescription;
+import net.bytebuddy.dynamic.DynamicType;
+import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
+import net.bytebuddy.dynamic.scaffold.InstrumentedType;
+import net.bytebuddy.dynamic.scaffold.subclass.ConstructorStrategy;
+import net.bytebuddy.implementation.Implementation;
+import net.bytebuddy.implementation.MethodCall.MethodLocator;
+import net.bytebuddy.implementation.StubMethod;
+import net.bytebuddy.implementation.bind.MethodDelegationBinder.MethodInvoker;
+import 
net.bytebuddy.implementation.bind.annotation.TargetMethodAnnotationDrivenBinder.TerminationHandler;
+import net.bytebuddy.implementation.bytecode.ByteCodeAppender;
+import net.bytebuddy.implementation.bytecode.Duplication;
+import net.bytebuddy.implementation.bytecode.StackManipulation;
+import net.bytebuddy.implementation.bytecode.Throw;
+import net.bytebuddy.implementation.bytecode.assign.Assigner;
+import net.bytebuddy.implementation.bytecode.member.FieldAccess;
+import net.bytebuddy.impleme

[GitHub] incubator-beam pull request #521: [BEAM-37] DoFnReflector: Add invoker inter...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/521


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #521

[kenn]

This closes #521


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/79c26d9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/79c26d9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/79c26d9c

Branch: refs/heads/master
Commit: 79c26d9c1577aac50a3e8f475d75629c87ba2c87
Parents: 779fb1a 2b47919
Author: Kenneth Knowles 
Authored: Thu Jul 14 16:42:11 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 16:42:11 2016 -0700

--
 pom.xml |   6 +
 sdks/java/core/pom.xml  |   5 +
 .../beam/sdk/transforms/DoFnReflector.java  | 717 ++-
 .../beam/sdk/transforms/DoFnReflectorTest.java  | 197 -
 .../transforms/DoFnReflectorBenchmark.java  |   7 +-
 5 files changed, 724 insertions(+), 208 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/79c26d9c/pom.xml
--

[jira] [Commented] (BEAM-37) Run DoFnWithContext without conversion to vanilla DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378636#comment-15378636
 ] 

ASF GitHub Bot commented on BEAM-37:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/521


> Run DoFnWithContext without conversion to vanilla DoFn
> --
>
> Key: BEAM-37
> URL: https://issues.apache.org/jira/browse/BEAM-37
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Ben Chambers
>
> DoFnWithContext is an enhanced DoFn where annotations and parameter lists are 
> inspected to determine whether it accesses windowing information, etc.
> Today, each feature of DoFnWithContext requires implementation on DoFn, which 
> precludes the easy addition of features that we don't have designs for in 
> DoFn.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/4] incubator-beam git commit: Fix DoFnTester side inputs

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 79c26d9c1 -> cb0356932


Fix DoFnTester side inputs

The side inputs were being stored as iterables, but being returned as
the raw type.

Store the side input values directly instead.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1c1af625
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1c1af625
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1c1af625

Branch: refs/heads/master
Commit: 1c1af62586db36212ebf76eb8307d1993666afa5
Parents: f0119b2
Author: Thomas Groh 
Authored: Thu Jul 14 10:33:22 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:13:10 2016 -0700

--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 70 --
 .../beam/sdk/transforms/DoFnTesterTest.java | 74 
 2 files changed, 91 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1c1af625/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 8cfb550..a638feb 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -103,50 +103,35 @@ public class DoFnTester {
* Registers the tuple of values of the side input {@link PCollectionView}s 
to
* pass to the {@link DoFn} under test.
*
-   * If needed, first creates a fresh instance of the {@link DoFn}
-   * under test.
+   * Resets the state of this {@link DoFnTester}.
*
* If this isn't called, {@code DoFnTester} assumes the
* {@link DoFn} takes no side inputs.
*/
-  public void setSideInputs(Map, 
Iterable>> sideInputs) {
+  public void setSideInputs(Map, Map> 
sideInputs) {
 this.sideInputs = sideInputs;
 resetState();
   }
 
   /**
-   * Registers the values of a side input {@link PCollectionView} to
-   * pass to the {@link DoFn} under test.
+   * Registers the values of a side input {@link PCollectionView} to pass to 
the {@link DoFn} under
+   * test.
*
-   * If needed, first creates a fresh instance of the {@code DoFn}
-   * under test.
+   * The provided value is the final value of the side input in the 
specified window, not
+   * the value of the input PCollection in that window.
*
-   * If this isn't called, {@code DoFnTester} assumes the
-   * {@code DoFn} takes no side inputs.
+   * If this isn't called, {@code DoFnTester} will return the default value 
for any side input
+   * that is used.
*/
-  public void setSideInput(PCollectionView sideInput, 
Iterable> value) {
-sideInputs.put(sideInput, value);
-  }
-
-  /**
-   * Registers the values for a side input {@link PCollectionView} to
-   * pass to the {@link DoFn} under test. All values are placed
-   * in the global window.
-   */
-  public void setSideInputInGlobalWindow(
-  PCollectionView sideInput,
-  Iterable value) {
-sideInputs.put(
-sideInput,
-Iterables.transform(value, new Function>() {
-  @Override
-  public WindowedValue apply(Object input) {
-return WindowedValue.valueInGlobalWindow(input);
-  }
-}));
+  public  void setSideInput(PCollectionView sideInput, BoundedWindow 
window, T value) {
+Map windowValues = (Map) 
sideInputs.get(sideInput);
+if (windowValues == null) {
+  windowValues = new HashMap<>();
+  sideInputs.put(sideInput, windowValues);
+}
+windowValues.put(window, value);
   }
 
-
   /**
* Registers the list of {@code TupleTag}s that can be used by the
* {@code DoFn} under test to output to side output
@@ -523,14 +508,14 @@ public class DoFnTester {
 private final TestContext context;
 private final TupleTag mainOutputTag;
 private final WindowedValue element;
-private final Map, ?> sideInputs;
+private final Map, Map> sideInputs;
 
 private TestProcessContext(
 DoFn fn,
 TestContext context,
 WindowedValue element,
 TupleTag mainOutputTag,
-Map, ?> sideInputs) {
+Map, Map> sideInputs) {
   fn.super();
   this.context = context;
   this.element = element;
@@ -545,9 +530,17 @@ public class DoFnTester {
 
 @Override
 public  T sideInput(PCollectionView view) {
-  @SuppressWarnings("unchecked")
-  T sideInput = (T) sideInputs.get(view);
-  return sideInput;
+  Map viewValues = sideInputs.get(view);
+  if (viewValues != null) {
+BoundedWindow sideInput

[2/4] incubator-beam git commit: Remove DoFnTester#setSideOutputTags

[kenn]

Remove DoFnTester#setSideOutputTags

Side Outputs are appended to the map of outputs on-demand.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b8cd5739
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b8cd5739
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b8cd5739

Branch: refs/heads/master
Commit: b8cd5739492b16c3481281074586f891f4554999
Parents: 1c1af62
Author: Thomas Groh 
Authored: Thu Jul 14 15:34:02 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:13:11 2016 -0700

--
 .../beam/runners/dataflow/DataflowRunnerTest.java   | 10 --
 .../org/apache/beam/sdk/transforms/DoFnTester.java  | 16 
 2 files changed, 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b8cd5739/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
index f3cbb38..fe288ad 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -18,7 +18,6 @@
 package org.apache.beam.runners.dataflow;
 
 import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
-
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
@@ -84,7 +83,6 @@ import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.PInput;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.apache.beam.sdk.values.TupleTag;
-import org.apache.beam.sdk.values.TupleTagList;
 
 import com.google.api.services.dataflow.Dataflow;
 import com.google.api.services.dataflow.model.DataflowPackage;
@@ -1060,8 +1058,6 @@ public class DataflowRunnerTest {
 keyCoder,
 ismCoder,
 false /* unique keys */));
-doFnTester.setSideOutputTags(TupleTagList.of(
-ImmutableList.>of(outputForSizeTag, 
outputForEntrySetTag)));
 
 IntervalWindow windowA = new IntervalWindow(new Instant(0), new 
Instant(10));
 IntervalWindow windowB = new IntervalWindow(new Instant(10), new 
Instant(20));
@@ -1162,8 +1158,6 @@ public class DataflowRunnerTest {
 keyCoder,
 ismCoder,
 true /* unique keys */));
-doFnTester.setSideOutputTags(TupleTagList.of(
-ImmutableList.>of(outputForSizeTag, 
outputForEntrySetTag)));
 
 IntervalWindow windowA = new IntervalWindow(new Instant(0), new 
Instant(10));
 
@@ -1203,8 +1197,6 @@ public class DataflowRunnerTest {
IsmRecord>> doFnTester = DoFnTester.of(
 new BatchViewAsMultimap.ToIsmMetadataRecordForSizeDoFn(
 windowCoder));
-doFnTester.setSideOutputTags(TupleTagList.of(
-ImmutableList.>of(outputForSizeTag, 
outputForEntrySetTag)));
 
 IntervalWindow windowA = new IntervalWindow(new Instant(0), new 
Instant(10));
 IntervalWindow windowB = new IntervalWindow(new Instant(10), new 
Instant(20));
@@ -1256,8 +1248,6 @@ public class DataflowRunnerTest {
IsmRecord>> doFnTester = DoFnTester.of(
 new BatchViewAsMultimap.ToIsmMetadataRecordForKeyDoFn(
 keyCoder, windowCoder));
-doFnTester.setSideOutputTags(TupleTagList.of(
-ImmutableList.>of(outputForSizeTag, 
outputForEntrySetTag)));
 
 IntervalWindow windowA = new IntervalWindow(new Instant(0), new 
Instant(10));
 IntervalWindow windowB = new IntervalWindow(new Instant(10), new 
Instant(20));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b8cd5739/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index a638feb..f8479de 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -24,7 +24,6 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.PTu

[3/4] incubator-beam git commit: Increase visibility Restrictions in DoFnTester

[kenn]

Increase visibility Restrictions in DoFnTester

No in-package representation refers to the fields of DoFnTester
directly.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/60443524
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/60443524
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/60443524

Branch: refs/heads/master
Commit: 604435249750729b223d90f22300571ff9f4bcfc
Parents: b8cd573
Author: Thomas Groh 
Authored: Thu Jul 14 15:34:36 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:13:11 2016 -0700

--
 .../org/apache/beam/sdk/transforms/DoFnTester.java  | 16 
 1 file changed, 8 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/60443524/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index f8479de..c38f0ab 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -639,10 +639,10 @@ public class DoFnTester {
 FINISHED
   }
 
-  final PipelineOptions options = PipelineOptionsFactory.create();
+  private final PipelineOptions options = PipelineOptionsFactory.create();
 
   /** The original DoFn under test. */
-  final DoFn origFn;
+  private final DoFn origFn;
 
   /** The side input values to provide to the DoFn under test. */
   private Map, Map> sideInputs =
@@ -651,23 +651,23 @@ public class DoFnTester {
   private Map accumulators;
 
   /** The output tags used by the DoFn under test. */
-  TupleTag mainOutputTag = new TupleTag<>();
+  private TupleTag mainOutputTag = new TupleTag<>();
 
   /** The original DoFn under test, if started. */
   DoFn fn;
 
   /** The ListOutputManager to examine the outputs. */
-  Map, List>> outputs;
+  private Map, List>> outputs;
 
   /** The state of processing of the DoFn under test. */
-  State state;
+  private State state;
 
-  DoFnTester(DoFn origFn) {
+  private DoFnTester(DoFn origFn) {
 this.origFn = origFn;
 resetState();
   }
 
-  void resetState() {
+  private void resetState() {
 fn = null;
 outputs = null;
 accumulators = null;
@@ -675,7 +675,7 @@ public class DoFnTester {
   }
 
   @SuppressWarnings("unchecked")
-  void initializeState() {
+  private void initializeState() {
 fn = (DoFn)
 SerializableUtils.deserializeFromByteArray(
 SerializableUtils.serializeToByteArray(origFn),

[4/4] incubator-beam git commit: This closes #659

[kenn]

This closes #659


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cb035693
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cb035693
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cb035693

Branch: refs/heads/master
Commit: cb03569326f29816d622329e2c3b1e4aad1751d1
Parents: 79c26d9 6044352
Author: Kenneth Knowles 
Authored: Thu Jul 14 17:14:00 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:14:00 2016 -0700

--
 .../runners/dataflow/DataflowRunnerTest.java|  10 --
 .../apache/beam/sdk/transforms/DoFnTester.java  | 102 +++
 .../beam/sdk/transforms/DoFnTesterTest.java |  74 +++---
 3 files changed, 99 insertions(+), 87 deletions(-)
--

[GitHub] incubator-beam pull request #659: Fix DoFnTester side inputs

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/659


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam git commit: Bump required Maven version to 3.3

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cb0356932 -> 9bd5c1a83


Bump required Maven version to 3.3


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4a70283e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4a70283e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4a70283e

Branch: refs/heads/master
Commit: 4a70283eeaa8f3fb8010d396f3dcfbe832049ecd
Parents: 84573e0
Author: Scott Wegner 
Authored: Thu Jul 14 15:50:12 2016 -0700
Committer: Scott Wegner 
Committed: Thu Jul 14 15:50:12 2016 -0700

--
 pom.xml | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4a70283e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 7089c2c..cf45e0e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -90,10 +90,6 @@
 
   
 
-  
-3.0.3
-  
-
   
 UTF-8
 
@@ -1017,7 +1013,7 @@
   [1.7,)
 
 
-  [3.0.3,)
+  [3.3,)
 
   
 

[2/3] incubator-beam git commit: Lower requirement from 3.3 to 3.2

[kenn]

Lower requirement from 3.3 to 3.2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/254a71dc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/254a71dc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/254a71dc

Branch: refs/heads/master
Commit: 254a71dce0323e7904b1313cf4b7c86ad5051b96
Parents: 4a70283
Author: Scott Wegner 
Authored: Thu Jul 14 16:01:00 2016 -0700
Committer: Scott Wegner 
Committed: Thu Jul 14 16:01:00 2016 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/254a71dc/pom.xml
--
diff --git a/pom.xml b/pom.xml
index cf45e0e..e074f6e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1013,7 +1013,7 @@
   [1.7,)
 
 
-  [3.3,)
+  [3.2,)
 
   
 

[3/3] incubator-beam git commit: This closes #660

[kenn]

This closes #660


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9bd5c1a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9bd5c1a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9bd5c1a8

Branch: refs/heads/master
Commit: 9bd5c1a83ff7c10536a000ad7e9c8d5d3f5641ad
Parents: cb03569 254a71d
Author: Kenneth Knowles 
Authored: Thu Jul 14 17:15:28 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:15:28 2016 -0700

--
 pom.xml | 6 +-
 1 file changed, 1 insertion(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9bd5c1a8/pom.xml
--

[GitHub] incubator-beam pull request #660: Bump required Maven version to 3.2

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/660


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_MavenVerify #889

[Apache Jenkins Server]

See 

Changes:

[klk] [BEAM-37] DoFnReflector: Add invoker interface and generate code

--
[...truncated 3553 lines...]
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-repository-metadata/3.0/maven-repository-metadata-3.0.jar
 (30 KB at 177.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-spi/1.7/aether-spi-1.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-aether-provider/3.0/maven-aether-provider-3.0.jar
 (50 KB at 276.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-api/1.7/aether-api-1.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-core/3.0/maven-core-3.0.jar
 (515 KB at 2586.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-model-builder/3.0/maven-model-builder-3.0.jar
 (145 KB at 722.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-spi/1.7/aether-spi-1.7.jar
 (14 KB at 61.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-impl/1.7/aether-impl-1.7.jar
 (104 KB at 473.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-compat/3.0/maven-compat-3.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-api/1.7/aether-api-1.7.jar
 (73 KB at 323.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.0/maven-artifact-3.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.14/plexus-interpolation-1.14.jar
 (60 KB at 242.4 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.2.3/plexus-classworlds-2.2.3.jar
 (46 KB at 170.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/5.0.2/asm-5.0.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
 (106 KB at 389.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-commons/5.0.2/asm-commons-5.0.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.0/maven-artifact-3.0.jar
 (51 KB at 185.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-tree/5.0.2/asm-tree-5.0.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
 (240 KB at 779.0 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/jdom/jdom/1.1/jdom-1.1.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-tree/5.0.2/asm-tree-5.0.2.jar
 (29 KB at 90.6 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-dependency-tree/2.2/maven-dependency-tree-2.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-commons/5.0.2/asm-commons-5.0.2.jar
 (41 KB at 126.5 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/5.0.2/asm-5.0.2.jar (52 KB 
at 160.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/vafer/jdependency/1.0/jdependency-1.0.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-compat/3.0/maven-compat-3.0.jar
 (279 KB at 811.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-analysis/5.0.3/asm-analysis-5.0.3.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-dependency-tree/2.2/maven-dependency-tree-2.2.jar
 (63 KB at 173.9 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-util/5.0.3/asm-util-5.0.3.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/jdom/jdom/1.1/jdom-1.1.jar (150 KB at 
414.2 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/11.0.2/guava-11.0.2.jar
[INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/vafer/jdependency/1.0/jdependency-1.0.jar
 (22 KB at 58.1 KB/sec)
[INFO] Downloading: 
https://repo.maven.apache.org/ma

Build failed in Jenkins: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: Core #889

[Apache Jenkins Server]

See 


--
[...truncated 1553 lines...]
[WARNING] 
:49:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.SortedSetMultimap;
[WARNING] ^
[WARNING] 
:50:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.TreeBasedTable;
[WARNING] ^
[WARNING] 
:51:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.TreeMultimap;
[WARNING] ^
[WARNING] 
:53:
 error: package com.fasterxml.jackson.annotation does not exist
[WARNING] import com.fasterxml.jackson.annotation.JsonIgnore;
[WARNING] ^
[WARNING] 
:54:
 error: package com.fasterxml.jackson.databind does not exist
[WARNING] import com.fasterxml.jackson.databind.JavaType;
[WARNING] ^
[WARNING] 
:55:
 error: package com.fasterxml.jackson.databind does not exist
[WARNING] import com.fasterxml.jackson.databind.ObjectMapper;
[WARNING] ^
[WARNING] 
:57:
 error: package org.slf4j does not exist
[WARNING] import org.slf4j.Logger;
[WARNING] ^
[WARNING] 
:58:
 error: package org.slf4j does not exist
[WARNING] import org.slf4j.LoggerFactory;
[WARNING] ^
[WARNING] 
:86:
 error: cannot find symbol
[WARNING] import javax.annotation.Nonnull;
[WARNING] ^
[WARNING] symbol:   class Nonnull
[WARNING] location: package javax.annotation
[WARNING] 
:30:
 error: package com.google.auto.value does not exist
[WARNING] import com.google.auto.value.AutoValue;
[WARNING] ^
[WARNING] 
:31:
 error: package com.google.common.base does not exist
[WARNING] import com.google.common.base.Defaults;
[WARNING] ^
[WARNING] 
:32:
 error: package com.google.common.base does not exist
[WARNING] import com.google.common.base.Function;
[WARNING] ^
[WARNING] 
:33:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.ClassToInstanceMap;
[WARNING] ^
[WARNING] 
:34:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.FluentIterable;
[WARNING] ^
[WARNING] 
:35:
 error: package com.google.common.collect does not exist
[WARNING] import com.google.common.collect.HashMultimap;
[WARNING] ^
[WARNING] 
:36:
 error: 

[jira] [Updated] (BEAM-445) Beam-examples-java build failed through local "mvn install"

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-445:
-
Assignee: Daniel Halperin  (was: Scott Wegner)

> Beam-examples-java build failed through local "mvn install"
> ---
>
> Key: BEAM-445
> URL: https://issues.apache.org/jira/browse/BEAM-445
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
> Environment: linux
>Reporter: Mark Liu
>Assignee: Daniel Halperin
>Priority: Critical
>
> Build project under beam/examples/java with command "mvn clean install 
> -DskipTests" failed with following error:
> [ERROR] Failed to execute goal on project beam-examples-java: Could not 
> resolve dependencies for project 
> org.apache.beam:beam-examples-java:jar:0.2.0-incubating-SNAPSHOT: Could not 
> transfer artifact 
> io.netty:netty-tcnative-boringssl-static:jar:${os.detected.classifier}:1.1.33.Fork13
>  from/to central (http://repo.maven.apache.org/maven2): Illegal character in 
> path at index 138: 
> http://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork13/netty-tcnative-boringssl-static-1.1.33.Fork13-${os.detected.classifier}.jar
> Reason: can't resolve ${os.detected.classifier} in 
> beam/sdks/java/io/google-cloud-platform/pom file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-445) Beam-examples-java build failed through local "mvn install"

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-445:
-
Assignee: Scott Wegner  (was: Daniel Halperin)

> Beam-examples-java build failed through local "mvn install"
> ---
>
> Key: BEAM-445
> URL: https://issues.apache.org/jira/browse/BEAM-445
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
> Environment: linux
>Reporter: Mark Liu
>Assignee: Scott Wegner
>Priority: Critical
>
> Build project under beam/examples/java with command "mvn clean install 
> -DskipTests" failed with following error:
> [ERROR] Failed to execute goal on project beam-examples-java: Could not 
> resolve dependencies for project 
> org.apache.beam:beam-examples-java:jar:0.2.0-incubating-SNAPSHOT: Could not 
> transfer artifact 
> io.netty:netty-tcnative-boringssl-static:jar:${os.detected.classifier}:1.1.33.Fork13
>  from/to central (http://repo.maven.apache.org/maven2): Illegal character in 
> path at index 138: 
> http://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork13/netty-tcnative-boringssl-static-1.1.33.Fork13-${os.detected.classifier}.jar
> Reason: can't resolve ${os.detected.classifier} in 
> beam/sdks/java/io/google-cloud-platform/pom file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-445) Beam-examples-java build failed through local "mvn install"

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378656#comment-15378656
 ] 

Kenneth Knowles commented on BEAM-445:
--

The build breakage itself is fixed - users are still required to include the 
{{os-maven-plugin}} in their build.

Three avenues, off the top of my head:

 - Get upstream upgraded to a version with an uber-jar..
 - Try a same-major-version override to get a working uber-jar (I understand 
that upstream is not using semantic versions, so this is a wildcard)
 - Produce classifiers ourselves; it doesn't absolve the user of including the 
plugin, but makes it less of a spooky issue.

> Beam-examples-java build failed through local "mvn install"
> ---
>
> Key: BEAM-445
> URL: https://issues.apache.org/jira/browse/BEAM-445
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
> Environment: linux
>Reporter: Mark Liu
>Assignee: Daniel Halperin
>Priority: Critical
>
> Build project under beam/examples/java with command "mvn clean install 
> -DskipTests" failed with following error:
> [ERROR] Failed to execute goal on project beam-examples-java: Could not 
> resolve dependencies for project 
> org.apache.beam:beam-examples-java:jar:0.2.0-incubating-SNAPSHOT: Could not 
> transfer artifact 
> io.netty:netty-tcnative-boringssl-static:jar:${os.detected.classifier}:1.1.33.Fork13
>  from/to central (http://repo.maven.apache.org/maven2): Illegal character in 
> path at index 138: 
> http://repo.maven.apache.org/maven2/io/netty/netty-tcnative-boringssl-static/1.1.33.Fork13/netty-tcnative-boringssl-static-1.1.33.Fork13-${os.detected.classifier}.jar
> Reason: can't resolve ${os.detected.classifier} in 
> beam/sdks/java/io/google-cloud-platform/pom file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #661: Update Dataflow container image and remove...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/661

Update Dataflow container image and remove runners-core dependency

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam new-container-image

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/661.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #661


commit 8030f0d53ce0b783734e7046c1521cdad7044fa2
Author: Daniel Halperin 
Date:   2016-07-14T15:48:37Z

Update Dataflow container image

commit 5de5de70209af4d7626a8be959e9e8de99cae414
Author: Kenneth Knowles 
Date:   2016-07-15T00:22:43Z

Remove runtime dependency on runners-core from Dataflow runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #655

[kenn]

This closes #655


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7a618034
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7a618034
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7a618034

Branch: refs/heads/master
Commit: 7a618034d308e32b9d2c1142ba637c5346fcdfb6
Parents: 9bd5c1a 8030f0d
Author: Kenneth Knowles 
Authored: Thu Jul 14 17:26:13 2016 -0700
Committer: Kenneth Knowles 
Committed: Thu Jul 14 17:26:13 2016 -0700

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: Update Dataflow container image

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 9bd5c1a83 -> 7a618034d


Update Dataflow container image

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8030f0d5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8030f0d5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8030f0d5

Branch: refs/heads/master
Commit: 8030f0d53ce0b783734e7046c1521cdad7044fa2
Parents: f0119b2
Author: Daniel Halperin 
Authored: Thu Jul 14 08:48:37 2016 -0700
Committer: GitHub 
Committed: Thu Jul 14 08:48:37 2016 -0700

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8030f0d5/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 1f2fdca..fa61587 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -218,9 +218,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160613";
+  = "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160714";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160613";
+  = "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160714";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[GitHub] incubator-beam pull request #655: Update Dataflow container image

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/655


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-446) Improve IOChannelUtils.resolve() to accept multiple paths at once

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu closed BEAM-446.
-

> Improve IOChannelUtils.resolve() to accept multiple paths at once
> -
>
> Key: BEAM-446
> URL: https://issues.apache.org/jira/browse/BEAM-446
> Project: Beam
>  Issue Type: Improvement
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Minor
> Fix For: 0.2.0-incubating
>
>
> Currently, IOChannelUtils.resolve() method can only resolve one path against 
> base path. 
> It's useful to have another method with arguments that includes one base path 
> and multiple others. The return string will be a directory that start with 
> base path and append rests which are separated by file separator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #650

[robertwb]

Closes #650


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a1a51c3c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a1a51c3c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a1a51c3c

Branch: refs/heads/python-sdk
Commit: a1a51c3c1a0edc215905920a0d8d300d263c80c6
Parents: c8cef2c 84fef46
Author: Robert Bradshaw 
Authored: Thu Jul 14 17:37:03 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 17:37:03 2016 -0700

--
 .../examples/complete/autocomplete_test.py  | 34 ++---
 .../examples/complete/estimate_pi.py| 19 +++--
 .../examples/complete/estimate_pi_test.py   | 36 +-
 .../examples/cookbook/coders_test.py| 33 +++--
 .../examples/cookbook/custom_ptransform.py  | 74 ++--
 .../examples/cookbook/custom_ptransform_test.py | 41 ---
 .../examples/cookbook/group_with_coder_test.py  | 70 --
 sdks/python/apache_beam/transforms/util.py  | 13 
 8 files changed, 139 insertions(+), 181 deletions(-)
--

[1/2] incubator-beam git commit: Update some of the example tests to use assert_that

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk c8cef2cba -> a1a51c3c1


Update some of the example tests to use assert_that


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/84fef464
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/84fef464
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/84fef464

Branch: refs/heads/python-sdk
Commit: 84fef464b0abea41e318c0fe983ac43874e5f6ad
Parents: c8cef2c
Author: Ahmet Altay 
Authored: Wed Jul 13 16:28:46 2016 -0700
Committer: Robert Bradshaw 
Committed: Thu Jul 14 17:37:02 2016 -0700

--
 .../examples/complete/autocomplete_test.py  | 34 ++---
 .../examples/complete/estimate_pi.py| 19 +++--
 .../examples/complete/estimate_pi_test.py   | 36 +-
 .../examples/cookbook/coders_test.py| 33 +++--
 .../examples/cookbook/custom_ptransform.py  | 74 ++--
 .../examples/cookbook/custom_ptransform_test.py | 41 ---
 .../examples/cookbook/group_with_coder_test.py  | 70 --
 sdks/python/apache_beam/transforms/util.py  | 13 
 8 files changed, 139 insertions(+), 181 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/84fef464/sdks/python/apache_beam/examples/complete/autocomplete_test.py
--
diff --git a/sdks/python/apache_beam/examples/complete/autocomplete_test.py 
b/sdks/python/apache_beam/examples/complete/autocomplete_test.py
index bd0a6cb..1b3ee5f 100644
--- a/sdks/python/apache_beam/examples/complete/autocomplete_test.py
+++ b/sdks/python/apache_beam/examples/complete/autocomplete_test.py
@@ -15,43 +15,17 @@
 # limitations under the License.
 #
 
-"""Test for the wordcount example."""
+"""Test for the autocomplete example."""
 
-import collections
 import unittest
 
-
 import apache_beam as beam
 from apache_beam.examples.complete import autocomplete
-from apache_beam.pvalue import AsIter
-
-# TODO(robertwb): Move to testing utilities.
-
-
-def assert_that(pcoll, matcher):
-  """Asserts that the give PCollection satisfies the constraints of the matcher
-  in a way that is runnable locally or on a remote service.
-  """
-  singleton = pcoll.pipeline | beam.Create('create_singleton', [None])
-
-  def check_matcher(_, side_value):
-assert matcher(side_value)
-return []
-  singleton | beam.FlatMap(check_matcher, AsIter(pcoll))  # pylint: 
disable=expression-not-assigned
-
-
-def contains_in_any_order(expected):
-  def matcher(value):
-vs = collections.Counter(value)
-es = collections.Counter(expected)
-if vs != es:
-  raise ValueError(
-  'extra: %s, missing: %s' % (vs - es, es - vs))
-return True
-  return matcher
+from apache_beam.transforms.util import assert_that
+from apache_beam.transforms.util import contains_in_any_order
 
 
-class WordCountTest(unittest.TestCase):
+class AutocompleteTest(unittest.TestCase):
 
   WORDS = ['this', 'this', 'that', 'to', 'to', 'to']
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/84fef464/sdks/python/apache_beam/examples/complete/estimate_pi.py
--
diff --git a/sdks/python/apache_beam/examples/complete/estimate_pi.py 
b/sdks/python/apache_beam/examples/complete/estimate_pi.py
index 8b0f202..3c4a2d9 100644
--- a/sdks/python/apache_beam/examples/complete/estimate_pi.py
+++ b/sdks/python/apache_beam/examples/complete/estimate_pi.py
@@ -85,6 +85,20 @@ class JsonCoder(object):
 return json.dumps(x)
 
 
+class EstimatePiTransform(beam.PTransform):
+  """Runs 10M trials, and combine the results to estimate pi."""
+
+  def __init__(self, label):
+super(EstimatePiTransform, self).__init__(label)
+
+  def apply(self, pcoll):
+# A hundred work items of a hundred thousand tries each.
+return (pcoll
+| beam.Create('Initialize', [10] * 100).with_output_types(int)
+| beam.Map('Run trials', run_trials)
+| beam.CombineGlobally('Sum', combine_results).without_defaults())
+
+
 def run(argv=None):
 
   parser = argparse.ArgumentParser()
@@ -94,11 +108,8 @@ def run(argv=None):
   known_args, pipeline_args = parser.parse_known_args(argv)
 
   p = beam.Pipeline(argv=pipeline_args)
-  # A thousand work items of a million tries each.
   (p  # pylint: disable=expression-not-assigned
-   | beam.Create('Initialize', [10] * 100).with_output_types(int)
-   | beam.Map('Run trials', run_trials)
-   | beam.CombineGlobally('Sum', combine_results).without_defaults()
+   | EstimatePiTransform('Estimate')
| beam.io.Write('Write',
beam.io.TextFileSink(known_args.output,
 coder=JsonCoder

[GitHub] incubator-beam pull request #650: Update some of the example tests to use as...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: Core #890

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_MavenVerify #890

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #890

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Handle HttpError in GCS upload thread

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk a1a51c3c1 -> d898d56ae


Handle HttpError in GCS upload thread

* break connection to the main thread and propagate the exception.
* Retry in auth _refresh() to guard against temporary errors in the
  metadata service.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a4267d26
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a4267d26
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a4267d26

Branch: refs/heads/python-sdk
Commit: a4267d264395706f12479aa876501a62d5b679b7
Parents: a1a51c3
Author: Ahmet Altay 
Authored: Fri Jul 8 16:28:07 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:15:34 2016 -0700

--
 sdks/python/apache_beam/internal/auth.py |  2 ++
 sdks/python/apache_beam/io/gcsio.py  | 23 ---
 2 files changed, 22 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a4267d26/sdks/python/apache_beam/internal/auth.py
--
diff --git a/sdks/python/apache_beam/internal/auth.py 
b/sdks/python/apache_beam/internal/auth.py
index 0081970..f324a2d 100644
--- a/sdks/python/apache_beam/internal/auth.py
+++ b/sdks/python/apache_beam/internal/auth.py
@@ -82,6 +82,8 @@ class GCEMetadataCredentials(OAuth2Credentials):
 None,  # token_uri
 user_agent)
 
+  @retry.with_exponential_backoff(
+  retry_filter=retry.retry_on_server_errors_and_timeout_filter)
   def _refresh(self, http_request):
 refresh_time = datetime.datetime.now()
 req = urllib2.Request('http://metadata.google.internal/computeMetadata/v1/'

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a4267d26/sdks/python/apache_beam/io/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcsio.py 
b/sdks/python/apache_beam/io/gcsio.py
index a01988b..c61f251 100644
--- a/sdks/python/apache_beam/io/gcsio.py
+++ b/sdks/python/apache_beam/io/gcsio.py
@@ -533,6 +533,7 @@ class GcsBufferedWriter(object):
 
 # Set up communication with uploading thread.
 parent_conn, child_conn = multiprocessing.Pipe()
+self.child_conn = child_conn
 self.conn = parent_conn
 
 # Set up uploader.
@@ -547,6 +548,7 @@ class GcsBufferedWriter(object):
 # Start uploading thread.
 self.upload_thread = threading.Thread(target=self._start_upload)
 self.upload_thread.daemon = True
+self.upload_thread.last_error = None
 self.upload_thread.start()
 
   # TODO(silviuc): Refactor so that retry logic can be applied.
@@ -560,7 +562,15 @@ class GcsBufferedWriter(object):
 #
 # The uploader by default transfers data in chunks of 1024 * 1024 bytes at
 # a time, buffering writes until that size is reached.
-self.client.objects.Insert(self.insert_request, upload=self.upload)
+try:
+  self.client.objects.Insert(self.insert_request, upload=self.upload)
+except HttpError as http_error:
+  logging.error(
+  'HTTP error while inserting file %s: %s', self.path, http_error)
+  self.upload_thread.last_error = http_error
+  raise
+finally:
+  self.child_conn.close()
 
   def write(self, data):
 """Write data to a GCS file.
@@ -574,8 +584,14 @@ class GcsBufferedWriter(object):
 self._check_open()
 if not data:
   return
-self.conn.send_bytes(data)
-self.position += len(data)
+try:
+  self.conn.send_bytes(data)
+  self.position += len(data)
+except IOError:
+  if self.upload_thread.last_error:
+raise self.upload_thread.last_error  # pylint: disable=raising-bad-type
+  else:
+raise
 
   def tell(self):
 """Return the total number of bytes passed to write() so far."""
@@ -583,6 +599,7 @@ class GcsBufferedWriter(object):
 
   def close(self):
 """Close the current GCS file."""
+self.closed = True
 self.conn.close()
 self.upload_thread.join()
 

[2/2] incubator-beam git commit: Closes #617

[dhalperi]

Closes #617


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d898d56a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d898d56a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d898d56a

Branch: refs/heads/python-sdk
Commit: d898d56aeb73cf58ce0f8978603d163601147b73
Parents: a1a51c3 a4267d2
Author: Dan Halperin 
Authored: Thu Jul 14 18:15:35 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:15:35 2016 -0700

--
 sdks/python/apache_beam/internal/auth.py |  2 ++
 sdks/python/apache_beam/io/gcsio.py  | 23 ---
 2 files changed, 22 insertions(+), 3 deletions(-)
--

[2/2] incubator-beam git commit: Closes #635

[dhalperi]

Closes #635


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/15ff85de
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/15ff85de
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/15ff85de

Branch: refs/heads/master
Commit: 15ff85de6362ffd70ec1348f24f631a354ad8658
Parents: 7a61803 f9d4c3a
Author: Dan Halperin 
Authored: Thu Jul 14 18:18:11 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:18:11 2016 -0700

--
 .../sdk/io/gcp/datastore/V1Beta3ReadIT.java | 114 ++
 .../io/gcp/datastore/V1Beta3TestOptions.java|  44 +++
 .../sdk/io/gcp/datastore/V1Beta3TestUtil.java   | 383 +++
 .../sdk/io/gcp/datastore/V1Beta3WriteIT.java|  85 
 4 files changed, 626 insertions(+)
--

[GitHub] incubator-beam pull request #635: BEAM-354: End-to-end tests for V1Beta3 Dat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/635


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Datastore e2e tests

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7a618034d -> 15ff85de6


Datastore e2e tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f9d4c3a7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f9d4c3a7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f9d4c3a7

Branch: refs/heads/master
Commit: f9d4c3a75095b215cf77b6d7be4f03c985622ca5
Parents: 7a61803
Author: Vikas Kedigehalli 
Authored: Mon Jul 11 11:31:48 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:18:10 2016 -0700

--
 .../sdk/io/gcp/datastore/V1Beta3ReadIT.java | 114 ++
 .../io/gcp/datastore/V1Beta3TestOptions.java|  44 +++
 .../sdk/io/gcp/datastore/V1Beta3TestUtil.java   | 383 +++
 .../sdk/io/gcp/datastore/V1Beta3WriteIT.java|  85 
 4 files changed, 626 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f9d4c3a7/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3ReadIT.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3ReadIT.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3ReadIT.java
new file mode 100644
index 000..ddb6d81
--- /dev/null
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1Beta3ReadIT.java
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.sdk.io.gcp.datastore;
+
+import static 
org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.deleteAllEntities;
+import static 
org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.getDatastore;
+import static 
org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.makeAncestorKey;
+import static org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.makeEntity;
+
+import org.apache.beam.sdk.Pipeline;
+import 
org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.UpsertMutationBuilder;
+import org.apache.beam.sdk.io.gcp.datastore.V1Beta3TestUtil.V1Beta3TestWriter;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.values.PCollection;
+
+import com.google.datastore.v1beta3.Entity;
+import com.google.datastore.v1beta3.Key;
+import com.google.datastore.v1beta3.Query;
+import com.google.datastore.v1beta3.client.Datastore;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+import java.util.UUID;
+
+/**
+ * End-to-end tests for Datastore V1Beta3.Read.
+ */
+@RunWith(JUnit4.class)
+public class V1Beta3ReadIT {
+  private V1Beta3TestOptions options;
+  private String ancestor;
+  private final long numEntities = 1000;
+
+  @Before
+  public void setup() {
+PipelineOptionsFactory.register(V1Beta3TestOptions.class);
+options = 
TestPipeline.testingPipelineOptions().as(V1Beta3TestOptions.class);
+ancestor = UUID.randomUUID().toString();
+  }
+
+  /**
+   * An end-to-end test for {@link V1Beta3.Read}.
+   *
+   * Write some test entities to datastore and then run a dataflow pipeline 
that
+   * reads and counts the total number of entities. Verify that the count 
matches the
+   * number of entities written.
+   */
+  @Test
+  public void testE2EV1Beta3Read() throws Exception {
+// Create entities and write them to datastore
+writeEntitiesToDatastore(options, ancestor, numEntities);
+
+// Read from datastore
+Query query = V1Beta3TestUtil.makeAncestorKindQuery(
+options.getKind(), options.getNamespace(), ancestor);
+
+V1Beta3.Read read = DatastoreIO.v1beta3().read()
+.withProjectId(options.getProject())
+.withQuery(query)
+.withNamespace(options.getNamespace()

[2/2] incubator-beam git commit: Remove runtime dependency on runners-core from Dataflow runner

[dhalperi]

Remove runtime dependency on runners-core from Dataflow runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5439a6e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5439a6e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5439a6e9

Branch: refs/heads/master
Commit: 5439a6e96cd45f5faf99366aa6ca475d028a8489
Parents: 15ff85d
Author: Kenneth Knowles 
Authored: Thu Jul 14 17:22:43 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:19:42 2016 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 6 --
 1 file changed, 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5439a6e9/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 9cd1fb4..ee8f56b 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -284,12 +284,6 @@
 
 
 
-  org.apache.beam
-  beam-runners-core-java
-  runtime
-
-
-
   com.google.api-client
   google-api-client
 

[jira] [Commented] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378698#comment-15378698
 ] 

ASF GitHub Bot commented on BEAM-354:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/635


> Modify DatastoreIO to use Datastore v1beta3 API
> ---
>
> Key: BEAM-354
> URL: https://issues.apache.org/jira/browse/BEAM-354
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 0.2.0-incubating
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the 
> DatastoreIO needs to be migrated to use the new version. Also in the process 
> of doing so, this is a good time to add a level of indirection via a 
> PTranform such that future changes in Datastore API would not result in 
> changing user/pipeline code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #661

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 15ff85de6 -> 695247138


Closes #661


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/69524713
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/69524713
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/69524713

Branch: refs/heads/master
Commit: 6952471388c17deefd449bbae75dfe5d42e4fca7
Parents: 15ff85d 5439a6e
Author: Dan Halperin 
Authored: Thu Jul 14 18:19:42 2016 -0700
Committer: Dan Halperin 
Committed: Thu Jul 14 18:19:42 2016 -0700

--
 runners/google-cloud-dataflow-java/pom.xml | 6 --
 1 file changed, 6 deletions(-)
--

[GitHub] incubator-beam pull request #661: Remove runners-core dependency from Datafl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/661


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_MavenVerify #891

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #891

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #617: [BEAM-391] Handle HttpError in GCS upload ...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/617


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-391) Exceptions in gcsio upload thread causes pipeline to stall

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378717#comment-15378717
 ] 

ASF GitHub Bot commented on BEAM-391:
-

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/617


> Exceptions in gcsio upload thread causes pipeline to stall
> --
>
> Key: BEAM-391
> URL: https://issues.apache.org/jira/browse/BEAM-391
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>
> gcsio got stuck with invalid bucket name
> GcsBufferedWriter._start_upload (gcsio.py) raises an exception if the bucket 
> does not exist. This causes upload thread to silenty fail. It logs exception 
> to the log but this does not stop the pipeline or closes the receiving end of 
> the multiprocessing.Pipe(). Later a call in to write() blocks at 
> self.conn.send_bytes(). Note that send may block if the buffer is full.
> Upload thread should have a finally clause to close the socket connection. Or 
> better propagating the exception to its parent. This is true for other types 
> of exceptions also.
> Another small issue in the GcsBufferedWriter.close(). It does not self 
> self.close to True.
> reproduction: python -m apache_beam.examples.wordcount --output 
> gs://no-such-thing/
> Prints the exception but goes on forever. Ctrl + C breaks the main thread 
> shows where it got stuck.
> Similarly reproducible on the service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #653: Fix min and max timestamp on 32-bit machin...

[charlesccychen]

Github user charlesccychen closed the pull request at:

https://github.com/apache/incubator-beam/pull/653


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-453) Improve efficiency of GCS operations by supporting batch requests

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-453:
---

 Summary: Improve efficiency of GCS operations by  supporting batch 
requests
 Key: BEAM-453
 URL: https://issues.apache.org/jira/browse/BEAM-453
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Chamikara Jayalath


Currently GCS file operations (in [1]) are performed one at a time. We should 
be able to improve performance by performing file operations in batches.

[1] 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/gcsio.py



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #662: [BEAM-115] WIP: JSON Schema definition of ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/662

[BEAM-115] WIP: JSON Schema definition of pipeline

This is a json-schema sketch of the concrete schema from the [Pipeline 
Runner API proposal document](https://s.apache.org/beam-runner-api). Because 
our [serialization tech 
discussion](http://mail-archives.apache.org/mod_mbox/beam-dev/201606.mbox/%3CCAN_Ypr2ZPQG3OgPWu==kf-zztg06k0v5i0ay3dabchjyver...@mail.gmail.com%3E)
 seemed to favor JSON on the front end and Proto on the backend, I made this 
quick port. The original Avro IDL definition is also on [a branch with a 
test](https://github.com/kennknowles/incubator-beam/blob/pipeline-model/model/pipeline/src/main/avro/org/apache/beam/model/pipeline/pipeline.avdl).

Notes & Caveats:
 - I did not try to flesh out any more details; this was a straight port. 
There's plenty to add, but a PR seems like a place that will attract a desired 
kind of concrete discussion even in the current state.
 - Typing this makes my hands hurt. Luckily, it should change exceedingly 
rarely. There are many libraries that can generate json-schema in various ways, 
including Jackson itself, but I'm not so sure any of them are applicable.
 - Reading this makes my eyes hurt. This is a real problem. We need a 
readable spec, not just a test suite for validation.
 - I am not so sure that [the schema 
library](https://github.com/daveclayton/json-schema-validator) I've used to 
build my smoke test is a good long term choice. I chose it because it was 
Jackson-based.
 - I've left comments in the JSON even though that is frowned upon, and 
taken advantage of Jackson's feature to allow them. They can also go into 
`"description"` fields.
 - Perhaps we could write YAML and convert to json-schema with no loss of 
precision?

Feel free to leave comments here about the schema or meta issues of e.g. 
where the schema should live and what libraries we might want to use.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
pipeline-json-schema

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/662.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #662


commit c5843ce10e782056c76157169eb5516bf18ed9e4
Author: Kenneth Knowles 
Date:   2016-06-10T15:51:02Z

WIP: add JSON Schema definition of pipeline




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15378823#comment-15378823
 ] 

ASF GitHub Bot commented on BEAM-115:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/662

[BEAM-115] WIP: JSON Schema definition of pipeline

This is a json-schema sketch of the concrete schema from the [Pipeline 
Runner API proposal document](https://s.apache.org/beam-runner-api). Because 
our [serialization tech 
discussion](http://mail-archives.apache.org/mod_mbox/beam-dev/201606.mbox/%3CCAN_Ypr2ZPQG3OgPWu==kf-zztg06k0v5i0ay3dabchjyver...@mail.gmail.com%3E)
 seemed to favor JSON on the front end and Proto on the backend, I made this 
quick port. The original Avro IDL definition is also on [a branch with a 
test](https://github.com/kennknowles/incubator-beam/blob/pipeline-model/model/pipeline/src/main/avro/org/apache/beam/model/pipeline/pipeline.avdl).

Notes & Caveats:
 - I did not try to flesh out any more details; this was a straight port. 
There's plenty to add, but a PR seems like a place that will attract a desired 
kind of concrete discussion even in the current state.
 - Typing this makes my hands hurt. Luckily, it should change exceedingly 
rarely. There are many libraries that can generate json-schema in various ways, 
including Jackson itself, but I'm not so sure any of them are applicable.
 - Reading this makes my eyes hurt. This is a real problem. We need a 
readable spec, not just a test suite for validation.
 - I am not so sure that [the schema 
library](https://github.com/daveclayton/json-schema-validator) I've used to 
build my smoke test is a good long term choice. I chose it because it was 
Jackson-based.
 - I've left comments in the JSON even though that is frowned upon, and 
taken advantage of Jackson's feature to allow them. They can also go into 
`"description"` fields.
 - Perhaps we could write YAML and convert to json-schema with no loss of 
precision?

Feel free to leave comments here about the schema or meta issues of e.g. 
where the schema should live and what libraries we might want to use.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
pipeline-json-schema

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/662.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #662


commit c5843ce10e782056c76157169eb5516bf18ed9e4
Author: Kenneth Knowles 
Date:   2016-06-10T15:51:02Z

WIP: add JSON Schema definition of pipeline




> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)