Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #187

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Improve GcsIO throughput by 10x

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk cce4331dc -> c1440f7aa


Improve GcsIO throughput by 10x

This change increases the read buffer used from 1M to 16M.  Previously,
the speed of reading an incompressible file were: (50 MB: 3.17 MB/s,
100 MB: 3.79 MB/s, 200 MB: 4.13 MB/s, 400 MB: 4.24 MB/s).

The speed is now improved to: (50 MB: 24.21 MB/s, 100 MB: 42.70 MB/s,
200 MB: 42.89 MB/s, 400 MB: 46.92 MB/s).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e4a332d9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e4a332d9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e4a332d9

Branch: refs/heads/python-sdk
Commit: e4a332d9de5eca941e08f23242cd63bb83148312
Parents: cce4331
Author: Charles Chen 
Authored: Thu Nov 17 11:46:44 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 21:53:26 2016 -0800

--
 sdks/python/apache_beam/io/gcsio.py | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e4a332d9/sdks/python/apache_beam/io/gcsio.py
--
diff --git a/sdks/python/apache_beam/io/gcsio.py 
b/sdks/python/apache_beam/io/gcsio.py
index 1b08994..4f310be 100644
--- a/sdks/python/apache_beam/io/gcsio.py
+++ b/sdks/python/apache_beam/io/gcsio.py
@@ -47,7 +47,23 @@ except ImportError:
   'Google Cloud Storage I/O not supported for this execution environment '
   '(could not import storage API client).')
 
-DEFAULT_READ_BUFFER_SIZE = 1024 * 1024
+# This is the size of each partial-file read operation from GCS.  This
+# parameter was chosen to give good throughput while keeping memory usage at
+# a reasonable level; the following table shows throughput reached when
+# reading files of a given size with a chosen buffer size and informed the
+# choice of the value, as of 11/2016:
+#
+# +---++-+-+-+
+# |   | 50 MB file | 100 MB file | 200 MB file | 400 MB file |
+# +---++-+-+-+
+# | 8 MB buffer   | 17.12 MB/s | 22.67 MB/s  | 23.81 MB/s  | 26.05 MB/s  |
+# | 16 MB buffer  | 24.21 MB/s | 42.70 MB/s  | 42.89 MB/s  | 46.92 MB/s  |
+# | 32 MB buffer  | 28.53 MB/s | 48.08 MB/s  | 54.30 MB/s  | 54.65 MB/s  |
+# | 400 MB buffer | 34.72 MB/s | 71.13 MB/s  | 79.13 MB/s  | 85.39 MB/s  |
+# +---++-+-+-+
+DEFAULT_READ_BUFFER_SIZE = 16 * 1024 * 1024
+
+# This is the size of chunks used when writing to GCS.
 WRITE_CHUNK_SIZE = 8 * 1024 * 1024
 
 
@@ -373,7 +389,7 @@ class GcsBufferedReader(object):
 # Initialize read buffer state.
 self.download_stream = StringIO.StringIO()
 self.downloader = transfer.Download(
-self.download_stream, auto_transfer=False)
+self.download_stream, auto_transfer=False, chunksize=buffer_size)
 self.client.objects.Get(get_request, download=self.downloader)
 self.position = 0
 self.buffer = ''

[2/2] incubator-beam git commit: Closes #1379

[robertwb]

Closes #1379


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c1440f7a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c1440f7a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c1440f7a

Branch: refs/heads/python-sdk
Commit: c1440f7aa69f0134d52463c4bcdcabce36b481d7
Parents: cce4331 e4a332d
Author: Robert Bradshaw 
Authored: Fri Nov 18 21:53:27 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 21:53:27 2016 -0800

--
 sdks/python/apache_beam/io/gcsio.py | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-101) Data-driven triggers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678424#comment-15678424
 ] 

ASF GitHub Bot commented on BEAM-101:
-

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/1392

[BEAM-101] A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1392


commit 5748647aa0d409828f9769dbc472bdb50f92a343
Author: Gus Katsiapis 
Date:   2016-11-19T02:31:20Z

A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.




> Data-driven triggers
> 
>
> Key: BEAM-101
> URL: https://issues.apache.org/jira/browse/BEAM-101
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Frances Perry
>
> For some applications, it's useful to declare a pane/window to be emitted (or 
> finished) based on its contents. The simplest of these is the AfterCount 
> trigger, but more sophisticated predicates could be constructed.
> The requirements for consistent trigger firing are essentially that the state 
> of the trigger form a lattice and that the "should fire?" question is a 
> monotonic predicate on the lattice. Basically it asks "are we high enough up 
> the lattice?"
> Because the element types may change between the application of Windowing and 
> the actuation of the trigger, one idea is to extract the relevant data from 
> the element at Windowing and pass it along implicitly where it can be 
> combined and inspected in a type safe way later (similar to how timestamps 
> and windows are implicitly passed with elements).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1392: [BEAM-101] A few improvements to Apache B...

[katsiapis]

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/1392

[BEAM-101] A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1392.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1392


commit 5748647aa0d409828f9769dbc472bdb50f92a343
Author: Gus Katsiapis 
Date:   2016-11-19T02:31:20Z

A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1015) Need language specific tabs that allow both regular text and code blocks

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1015:
--

 Summary: Need language specific tabs that allow both regular text 
and code blocks
 Key: BEAM-1015
 URL: https://issues.apache.org/jira/browse/BEAM-1015
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Melissa Pashniak
Assignee: James Malone


Right now, the Java/python/etc. toggle tabs work well for code blocks, but if 
you need to have some accompanying language specific text with your code block 
it becomes unwieldy, as everything in there is treated as code. Ideally we 
could have both regular text and code blocks inside the language toggle tabs.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1014) Anchor target ends up hidden behind the top nav bar

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1014:
--

 Summary: Anchor target ends up hidden behind the top nav bar
 Key: BEAM-1014
 URL: https://issues.apache.org/jira/browse/BEAM-1014
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Melissa Pashniak
Assignee: James Malone


When you jump to an anchor target, the target ends up being hidden behind the 
top nav bar instead of visible below the nav bar.
Example: click on any link in the Beam Programming Guide table of contents




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak updated BEAM-1013:
---
Issue Type: Bug  (was: Improvement)

> Recheck all existing programming guide code snippets for correctness
> 
>
> Key: BEAM-1013
> URL: https://issues.apache.org/jira/browse/BEAM-1013
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1013) Recheck all existing programming guide code snippets for correctness

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1013:
--

 Summary: Recheck all existing programming guide code snippets for 
correctness
 Key: BEAM-1013
 URL: https://issues.apache.org/jira/browse/BEAM-1013
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Melissa Pashniak
Assignee: Melissa Pashniak






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1012) Add data encoding section

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1012:
--

 Summary: Add data encoding section
 Key: BEAM-1012
 URL: https://issues.apache.org/jira/browse/BEAM-1012
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Melissa Pashniak
Assignee: Melissa Pashniak






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1011) Add Windowing/Triggers section

[Melissa Pashniak (JIRA)]

Melissa Pashniak created BEAM-1011:
--

 Summary: Add Windowing/Triggers section
 Key: BEAM-1011
 URL: https://issues.apache.org/jira/browse/BEAM-1011
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Melissa Pashniak
Assignee: Melissa Pashniak






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1010) Custom FileSinks should respect AUTO compression

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678338#comment-15678338
 ] 

ASF GitHub Bot commented on BEAM-1010:
--

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/1391


> Custom FileSinks should respect AUTO compression
> 
>
> Key: BEAM-1010
> URL: https://issues.apache.org/jira/browse/BEAM-1010
> Project: Beam
>  Issue Type: Bug
>Reporter: Konstantinos Katsiapis
>Assignee: Konstantinos Katsiapis
>
> Currently AUTO compression is respected by Native FileSinks but not Custom 
> FileSinks.
> Also, it would be good for Beam's _CompresedFile to support usage with Python 
> "with" clause (ie implement __enter__ and __exit__).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1391: [BEAM-1010] A few improvements to Apache ...

[katsiapis]

Github user katsiapis closed the pull request at:

https://github.com/apache/incubator-beam/pull/1391


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1391: [BEAM-1010] A few improvements to Apache ...

[katsiapis]

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/1391

[BEAM-1010] A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1391


commit aa8b2020e8cad004189373d7aca195c088d96040
Author: Gus Katsiapis 
Date:   2016-11-19T01:16:47Z

A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1010) Custom FileSinks should respect AUTO compression

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678335#comment-15678335
 ] 

ASF GitHub Bot commented on BEAM-1010:
--

GitHub user katsiapis opened a pull request:

https://github.com/apache/incubator-beam/pull/1391

[BEAM-1010] A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsiapis/incubator-beam fileio

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1391


commit aa8b2020e8cad004189373d7aca195c088d96040
Author: Gus Katsiapis 
Date:   2016-11-19T01:16:47Z

A few improvements to Apache Beam Python's FileIO.

- Ensuring that AUTO compression works properly for FileSinks.
- Introducing __enter__ and __exit__ in _CompressedFile to allow use
  of "with", and updating textio accordingly.




> Custom FileSinks should respect AUTO compression
> 
>
> Key: BEAM-1010
> URL: https://issues.apache.org/jira/browse/BEAM-1010
> Project: Beam
>  Issue Type: Bug
>Reporter: Konstantinos Katsiapis
>Assignee: Konstantinos Katsiapis
>
> Currently AUTO compression is respected by Native FileSinks but not Custom 
> FileSinks.
> Also, it would be good for Beam's _CompresedFile to support usage with Python 
> "with" clause (ie implement __enter__ and __exit__).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1010) Custom FileSinks should respect AUTO compression

[Konstantinos Katsiapis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Katsiapis updated BEAM-1010:
-
Description: 
Currently AUTO compression is respected by Native FileSinks but not Custom 
FileSinks.

Also, it would be good for Beam's _CompresedFile to support usage with Python 
"with" clause (ie implement __enter__ and __exit__).

> Custom FileSinks should respect AUTO compression
> 
>
> Key: BEAM-1010
> URL: https://issues.apache.org/jira/browse/BEAM-1010
> Project: Beam
>  Issue Type: Bug
>Reporter: Konstantinos Katsiapis
>Assignee: Konstantinos Katsiapis
>
> Currently AUTO compression is respected by Native FileSinks but not Custom 
> FileSinks.
> Also, it would be good for Beam's _CompresedFile to support usage with Python 
> "with" clause (ie implement __enter__ and __exit__).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1010) Custom FileSinks should respect AUTO compression

[Konstantinos Katsiapis (JIRA)]

Konstantinos Katsiapis created BEAM-1010:


 Summary: Custom FileSinks should respect AUTO compression
 Key: BEAM-1010
 URL: https://issues.apache.org/jira/browse/BEAM-1010
 Project: Beam
  Issue Type: Bug
Reporter: Konstantinos Katsiapis
Assignee: Konstantinos Katsiapis






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #186

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[Pei He (JIRA)]

Pei He created BEAM-1009:


 Summary: Upgrade from mockito-all 1 to mockito-core 2
 Key: BEAM-1009
 URL: https://issues.apache.org/jira/browse/BEAM-1009
 Project: Beam
  Issue Type: Test
Reporter: Pei He


Mockito 2 provides useful features, and the mockito-all module is no longer 
generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He reassigned BEAM-1009:


Assignee: Pei He

> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>Reporter: Pei He
>Assignee: Pei He
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1390: Jenkins DSL Config

[jasonkuster]

GitHub user jasonkuster opened a pull request:

https://github.com/apache/incubator-beam/pull/1390

Jenkins DSL Config

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jasonkuster/incubator-beam jenkins-config

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1390.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1390


commit 57d09258ed605f03fa88cde515d56204bf8f47f0
Author: Jason Kuster 
Date:   2016-11-14T23:35:40Z

Initial commit of jobs

Signed-off-by: Jason Kuster 

commit 6028c7ba6c560a400ea6b5f4b4521e35483b9c1c
Author: Jason Kuster 
Date:   2016-11-18T21:33:17Z

Update job properties

Signed-off-by: Jason Kuster 

commit 3c8bda55110a816691690dfdc721889714bfc6c6
Author: Jason Kuster 
Date:   2016-11-18T23:44:35Z

move files to final location

Signed-off-by: Jason Kuster 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1008) KeyedWorkItemCoder claims three coder arguments in getCoderArguments but only has two generic type variables

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1008:
-

 Summary: KeyedWorkItemCoder claims three coder arguments in 
getCoderArguments but only has two generic type variables
 Key: BEAM-1008
 URL: https://issues.apache.org/jira/browse/BEAM-1008
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles


The number of coder arguments should match the number of type variables. The 
purpose of the method is to associate the type variables with coders.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1605

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-23) User-facing state and timers for ParDo(DoFn)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678060#comment-15678060
 ] 

ASF GitHub Bot commented on BEAM-23:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1378


> User-facing state and timers for ParDo(DoFn)
> 
>
> Key: BEAM-23
> URL: https://issues.apache.org/jira/browse/BEAM-23
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> To a key-partitioned ParDo, add the ability for a user's DoFn to, on a 
> per-key-and-window basis:
>  - read and write consistent state
>  - set a timer to request a callback at a particular moment (in any time 
> domain)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/6] incubator-beam git commit: Separate ParDoTest cases and make them more flexible

[kenn]

Separate ParDoTest cases and make them more flexible

A number of excessively rigid name tests preclude runner expansion
of ParDo. This change makes them into independent unit tests
for better signal and makes them more accurate to the intent - the
name should have the relevant information, but may have other
content.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e92a157a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e92a157a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e92a157a

Branch: refs/heads/master
Commit: e92a157aa41746df225379b71cc88695c8e2d93f
Parents: 3548ffb
Author: Kenneth Knowles 
Authored: Thu Nov 17 10:14:07 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../apache/beam/sdk/transforms/ParDoTest.java   | 55 ++--
 1 file changed, 27 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e92a157a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
index 26f5570..3c3e266 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
@@ -27,6 +27,7 @@ import static 
org.apache.beam.sdk.util.StringUtils.byteArrayToJsonString;
 import static org.apache.beam.sdk.util.StringUtils.jsonStringToByteArray;
 import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.anyOf;
+import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.equalTo;
 import static 
org.hamcrest.collection.IsIterableContainingInAnyOrder.containsInAnyOrder;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
@@ -826,39 +827,37 @@ public class ParDoTest implements Serializable {
   }
 
   @Test
-  public void testParDoGetName() {
+  public void testParDoOutputNameBasedOnDoFnWithTrimmedSuffix() {
 Pipeline p = TestPipeline.create();
+PCollection output = p.apply(Create.of(1)).apply(ParDo.of(new 
TestOldDoFn()));
+assertThat(output.getName(), containsString("ParDo(Test)"));
+  }
 
-PCollection input =
-p.apply(Create.of(Arrays.asList(3, -42, 666)))
-.setName("MyInput");
-
-{
-  PCollection output1 = input.apply(ParDo.of(new TestOldDoFn()));
-  assertEquals("ParDo(Test).out", output1.getName());
-}
-
-{
-  PCollection output2 = input.apply("MyParDo", ParDo.of(new 
TestOldDoFn()));
-  assertEquals("MyParDo.out", output2.getName());
-}
-
-{
-  PCollection output4 = input.apply("TestOldDoFn", ParDo.of(new 
TestOldDoFn()));
-  assertEquals("TestOldDoFn.out", output4.getName());
-}
+  @Test
+  public void testParDoOutputNameBasedOnLabel() {
+Pipeline p = TestPipeline.create();
+PCollection output =
+p.apply(Create.of(1)).apply("MyParDo", ParDo.of(new TestOldDoFn()));
+assertThat(output.getName(), containsString("MyParDo"));
+  }
 
-{
-  PCollection output5 = input.apply(ParDo.of(new 
StrangelyNamedDoer()));
-  assertEquals("ParDo(StrangelyNamedDoer).out",
-  output5.getName());
-}
+  @Test
+  public void testParDoOutputNameBasedDoFnWithoutMatchingSuffix() {
+Pipeline p = TestPipeline.create();
+PCollection output = p.apply(Create.of(1)).apply(ParDo.of(new 
StrangelyNamedDoer()));
+assertThat(output.getName(), containsString("ParDo(StrangelyNamedDoer)"));
+  }
 
-assertEquals("ParDo(Printing)", ParDo.of(new PrintingDoFn()).getName());
+  @Test
+  public void testParDoTransformNameBasedDoFnWithTrimmedSuffix() {
+assertThat(ParDo.of(new PrintingDoFn()).getName(), 
containsString("ParDo(Printing)"));
+  }
 
-assertEquals(
-"ParMultiDo(SideOutputDummy)",
-ParDo.of(new SideOutputDummyFn(null)).withOutputTags(null, 
null).getName());
+  @Test
+  public void testParDoMultiNameBasedDoFnWithTrimmerSuffix() {
+assertThat(
+ParDo.of(new SideOutputDummyFn(null)).withOutputTags(null, 
null).getName(),
+containsString("ParMultiDo(SideOutputDummy)"));
   }
 
   @Test

[2/6] incubator-beam git commit: Use getNewFn for coder inferences in ParDo

[kenn]

Use getNewFn for coder inferences in ParDo


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/34e2a352
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/34e2a352
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/34e2a352

Branch: refs/heads/master
Commit: 34e2a3525643f98ddcee5889b2d0d232e2e05509
Parents: b0d46c2
Author: Kenneth Knowles 
Authored: Thu Nov 17 11:08:48 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/34e2a352/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
index ac1bccb..215ae6a 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
@@ -1094,7 +1094,7 @@ public class ParDo {
   Coder inputCoder = ((PCollection) input).getCoder();
   return input.getPipeline().getCoderRegistry().getDefaultCoder(
   output.getTypeDescriptor(),
-  getOldFn().getInputTypeDescriptor(),
+  getNewFn().getInputTypeDescriptor(),
   inputCoder);
   }
 

[5/6] incubator-beam git commit: DirectRunner: Expand ParDo.Bound into ParDo.BoundMulti

[kenn]

DirectRunner: Expand ParDo.Bound into ParDo.BoundMulti


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f86e98c9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f86e98c9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f86e98c9

Branch: refs/heads/master
Commit: f86e98c91cedbb1d9fd54e3268dfd8f014ac2f27
Parents: 34e2a35
Author: Kenneth Knowles 
Authored: Wed Nov 16 15:43:47 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   |   3 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  56 +--
 .../direct/ParDoMultiEvaluatorHooks.java|  55 ---
 .../direct/ParDoMultiOverrideFactory.java   |  51 +++
 .../runners/direct/ParDoOverrideFactory.java|  53 ---
 .../direct/ParDoSingleEvaluatorHooks.java   |  58 ---
 .../ParDoSingleViaMultiOverrideFactory.java |  66 +++
 .../direct/TransformEvaluatorRegistry.java  |   7 +-
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 ---
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 --
 10 files changed, 139 insertions(+), 984 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f86e98c9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
index cce73c3..0060e84 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectRunner.java
@@ -87,7 +87,8 @@ public class DirectRunner
   .put(GroupByKey.class, new DirectGroupByKeyOverrideFactory())
   .put(TestStream.class, new DirectTestStreamFactory())
   .put(Write.Bound.class, new WriteWithShardingFactory())
-  .put(ParDo.Bound.class, new ParDoOverrideFactory())
+  .put(ParDo.Bound.class, new ParDoSingleViaMultiOverrideFactory())
+  .put(ParDo.BoundMulti.class, new ParDoMultiOverrideFactory())
   .put(
   GBKIntoKeyedWorkItems.class,
   new DirectGBKIntoKeyedWorkItemsOverrideFactory())

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f86e98c9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
index ee4987f..f126000 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluatorFactory.java
@@ -24,49 +24,22 @@ import 
org.apache.beam.runners.direct.DirectExecutionContext.DirectStepContext;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.ParDo.BoundMulti;
 import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.POutput;
+import org.apache.beam.sdk.values.PCollectionTuple;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
-/**
- * A {@link TransformEvaluatorFactory} for {@link ParDo}-like primitive {@link 
PTransform
- * PTransforms}, parameterized by some {@link TransformHooks 
transform-specific handling}.
- */
-final class ParDoEvaluatorFactory<
-InputT,
-OutputT,
-TransformOutputT extends POutput,
-TransformT extends PTransform>
-implements TransformEvaluatorFactory {
-  interface TransformHooks<
-  InputT,
-  OutputT,
-  TransformOutputT extends POutput,
-  TransformT extends PTransform> {
-/** Returns the {@link DoFn} contained in the given {@link ParDo} 
transform. */
-DoFn getDoFn(TransformT transform);
-
-/** Configures and creates a {@link ParDoEvaluator} for the given {@link 
DoFn}. */
-ParDoEvaluator createParDoEvaluator(
-EvaluationContext evaluationContext,
-AppliedPTransform 
application,
-

[GitHub] incubator-beam pull request #1378: [BEAM-23] DirectRunner: rewrite single-ou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1378


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[6/6] incubator-beam git commit: Delegate getAggregators() in various DoFn adapters

[kenn]

Delegate getAggregators() in various DoFn adapters


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b0d46c2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b0d46c2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b0d46c2d

Branch: refs/heads/master
Commit: b0d46c2deb4318f8d0e55eeeb20e1d11ceadd218
Parents: 6fa8057
Author: Kenneth Knowles 
Authored: Thu Nov 17 15:50:17 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../main/java/org/apache/beam/sdk/transforms/DoFn.java   |  7 ++-
 .../org/apache/beam/sdk/transforms/DoFnAdapters.java |  6 ++
 .../java/org/apache/beam/sdk/transforms/OldDoFn.java |  7 ++-
 .../java/org/apache/beam/sdk/transforms/OldDoFnTest.java | 11 +++
 4 files changed, 25 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index 9978ef4..221d942 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -28,6 +28,8 @@ import java.lang.annotation.ElementType;
 import java.lang.annotation.Retention;
 import java.lang.annotation.RetentionPolicy;
 import java.lang.annotation.Target;
+import java.util.Collection;
+import java.util.Collections;
 import java.util.HashMap;
 import java.util.Map;
 import javax.annotation.Nullable;
@@ -236,7 +238,6 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
 
   aggregator.setDelegate(delegate);
 }
-
   }
 
   /**
@@ -298,6 +299,10 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
 
   protected Map aggregators = new 
HashMap<>();
 
+  Collection getAggregators() {
+return Collections.unmodifiableCollection(aggregators.values());
+  }
+
   /**
* Protects aggregators from being created after initialization.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index a3466bb..1a74ae7 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -18,6 +18,7 @@
 package org.apache.beam.sdk.transforms;
 
 import java.io.IOException;
+import java.util.Collection;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn.Context;
@@ -244,6 +245,11 @@ public class DoFnAdapters {
 }
 
 @Override
+Collection getAggregators() {
+  return fn.getAggregators();
+}
+
+@Override
 public Duration getAllowedTimestampSkew() {
   return fn.getAllowedTimestampSkew();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b0d46c2d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
index f16e0b3..9bf9003 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/OldDoFn.java
@@ -676,6 +676,11 @@ public abstract class OldDoFn implements 
Serializable, HasDispl
 }
 
 @Override
+Collection getAggregators() {
+  return OldDoFn.this.getAggregators();
+}
+
+@Override
 protected TypeDescriptor getOutputTypeDescriptor() {
   return OldDoFn.this.getOutputTypeDescriptor();
 }
@@ -683,7 +688,7 @@ public abstract class OldDoFn implements 
Serializable, HasDispl
 
   /**
* A {@link ProcessContext} for an {@link OldDoFn} that implements
-   * {@link OldDoFn.RequiresWindowAcccess}, via a context for a proper {@link 
DoFn}.
+   * {@link RequiresWindowAccess}, via a context for a proper {@link DoFn}.
*/
   private class 

[1/6] incubator-beam git commit: This closes #1378

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 3548ffb06 -> d93e9a88b


This closes #1378


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d93e9a88
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d93e9a88
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d93e9a88

Branch: refs/heads/master
Commit: d93e9a88b0f0b3c7dc2461cbd11147b9ad8327cf
Parents: 3548ffb f86e98c
Author: Kenneth Knowles 
Authored: Fri Nov 18 15:09:43 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../beam/runners/direct/DirectRunner.java   |   3 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  56 +--
 .../direct/ParDoMultiEvaluatorHooks.java|  55 ---
 .../direct/ParDoMultiOverrideFactory.java   |  51 +++
 .../runners/direct/ParDoOverrideFactory.java|  53 ---
 .../direct/ParDoSingleEvaluatorHooks.java   |  58 ---
 .../ParDoSingleViaMultiOverrideFactory.java |  66 +++
 .../direct/TransformEvaluatorRegistry.java  |   7 +-
 .../direct/ParDoMultiEvaluatorHooksTest.java| 439 ---
 .../direct/ParDoSingleEvaluatorHooksTest.java   | 335 --
 .../org/apache/beam/sdk/transforms/DoFn.java|   7 +-
 .../beam/sdk/transforms/DoFnAdapters.java   |   6 +
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   7 +-
 .../org/apache/beam/sdk/transforms/ParDo.java   |   2 +-
 .../beam/sdk/transforms/WithTimestamps.java |   5 +-
 .../apache/beam/sdk/transforms/OldDoFnTest.java |  11 +-
 .../apache/beam/sdk/transforms/ParDoTest.java   |  55 ++-
 17 files changed, 194 insertions(+), 1022 deletions(-)
--

[4/6] incubator-beam git commit: Do not override type descriptor in WithTimestamps

[kenn]

Do not override type descriptor in WithTimestamps


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6fa8057f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6fa8057f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6fa8057f

Branch: refs/heads/master
Commit: 6fa8057fe1db97615872915dcd81b2bca9b44e63
Parents: e92a157
Author: Kenneth Knowles 
Authored: Thu Nov 17 11:07:32 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:09:43 2016 -0800

--
 .../java/org/apache/beam/sdk/transforms/WithTimestamps.java | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6fa8057f/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
index 00ac8e4..64e7c45 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/WithTimestamps.java
@@ -99,9 +99,8 @@ public class WithTimestamps extends 
PTransform {

[2/2] incubator-beam git commit: This closes #1389

[kenn]

This closes #1389


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3548ffb0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3548ffb0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3548ffb0

Branch: refs/heads/master
Commit: 3548ffb0656c4729fc1475490cc7f0e21669c197
Parents: 2a7169b 73692b8
Author: Kenneth Knowles 
Authored: Fri Nov 18 15:00:13 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 15:00:13 2016 -0800

--
 .../beam/sdk/util/state/StateContexts.java  | 41 +++-
 1 file changed, 32 insertions(+), 9 deletions(-)
--

[GitHub] incubator-beam pull request #1389: Restore StateContexts.windowOnly for temp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1389


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Restore StateContexts.windowOnly for temporary compatibility

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2a7169b6f -> 3548ffb06


Restore StateContexts.windowOnly for temporary compatibility


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/73692b80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/73692b80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/73692b80

Branch: refs/heads/master
Commit: 73692b808aa4e0d85c6e20eb19a1bf4a4355a5c1
Parents: 2a7169b
Author: Kenneth Knowles 
Authored: Fri Nov 18 14:40:59 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:59:41 2016 -0800

--
 .../beam/sdk/util/state/StateContexts.java  | 41 +++-
 1 file changed, 32 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/73692b80/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
index 858d6fe..1d6e949 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateContexts.java
@@ -42,22 +42,45 @@ public class StateContexts {
 @Override
 public BoundedWindow window() {
   throw new IllegalArgumentException("cannot call window() in a null 
context");
-}};
+}
+  };
 
-  /**
-   * Returns a fake {@link StateContext}.
-   */
+  /** Returns a fake {@link StateContext}. */
   @SuppressWarnings("unchecked")
   public static  StateContext nullContext() {
 return (StateContext) NULL_CONTEXT;
   }
 
   /**
-   * Deprecated, do not use.
-   *
-   * This exists only for temporary compatibility with Dataflow worker and 
should be deleted
-   * once a worker image is released that uses runners-core build after
-   * https://github.com/apache/incubator-beam/pull/1353.
+   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
+   * deleted once a worker image is released that uses runners-core build 
after
+   * https://github.com/apache/incubator-beam/pull/1353.
+   */
+  @Deprecated
+  public static  StateContext windowOnly(final W 
window) {
+return new StateContext() {
+  @Override
+  public PipelineOptions getPipelineOptions() {
+throw new IllegalArgumentException(
+"cannot call getPipelineOptions() in a window only context");
+  }
+
+  @Override
+  public  T sideInput(PCollectionView view) {
+throw new IllegalArgumentException("cannot call sideInput() in a 
window only context");
+  }
+
+  @Override
+  public W window() {
+return window;
+  }
+};
+  }
+
+  /**
+   * @deprecated This exists only for temporary compatibility with Dataflow 
worker and should be
+   * deleted once a worker image is released that uses runners-core build 
after
+   * https://github.com/apache/incubator-beam/pull/1353.
*/
   @Deprecated
   public static  StateContext createFromComponents(

[jira] [Updated] (BEAM-1007) Runner Toggles in Quickstart.md Don't Work in Safari

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek updated BEAM-1007:
---
Attachment: Screen Shot 2016-11-19 at 00.00.45.png

> Runner Toggles in Quickstart.md Don't Work in Safari 
> -
>
> Key: BEAM-1007
> URL: https://issues.apache.org/jira/browse/BEAM-1007
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Aljoscha Krettek
>Assignee: Abdullah Bashir
>Priority: Minor
> Attachments: Screen Shot 2016-11-19 at 00.00.45.png
>
>
> I'm observing this with macOS Sierra 10.12.1 and Safari 10.0.1 
> (12602.2.14.0.7).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1007) Runner Toggles in Quickstart.md Don't Work in Safari

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678046#comment-15678046
 ] 

Aljoscha Krettek commented on BEAM-1007:


[~mabdullah353], I assigned to you because you initially implemented this and I 
have no idea of what might be going on there. Hope that's alright.

> Runner Toggles in Quickstart.md Don't Work in Safari 
> -
>
> Key: BEAM-1007
> URL: https://issues.apache.org/jira/browse/BEAM-1007
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Aljoscha Krettek
>Assignee: Abdullah Bashir
>Priority: Minor
> Attachments: Screen Shot 2016-11-19 at 00.00.45.png
>
>
> I'm observing this with macOS Sierra 10.12.1 and Safari 10.0.1 
> (12602.2.14.0.7).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1007) Runner Toggles in Quickstart.md Don't Work in Safari

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-1007:
--

 Summary: Runner Toggles in Quickstart.md Don't Work in Safari 
 Key: BEAM-1007
 URL: https://issues.apache.org/jira/browse/BEAM-1007
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Aljoscha Krettek
Assignee: Abdullah Bashir
Priority: Minor


I'm observing this with macOS Sierra 10.12.1 and Safari 10.0.1 (12602.2.14.0.7).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-795) org.apache.beam.sdk.io.ReadTest.testStreamingPrimitiveDisplayData failing in Dataflow RunnableOnService

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677994#comment-15677994
 ] 

Scott Wegner commented on BEAM-795:
---

This is done. [~jasonkuster] can you close?

> org.apache.beam.sdk.io.ReadTest.testStreamingPrimitiveDisplayData failing in 
> Dataflow RunnableOnService
> ---
>
> Key: BEAM-795
> URL: https://issues.apache.org/jira/browse/BEAM-795
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Scott Wegner
>Priority: Critical
>
> Test is failing in beam_PostCommit_RunnableOnService_GoogleCloudDataflow 
> since yesterday. Culprit change seems to be 
> https://github.com/apache/incubator-beam/commit/ad03d07ae783f054a31e8b2e14100afff8cdf747
> Affected Jenkins runs:
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/
> 1381 - 1386, 1388 as of bug creation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1389: Restore StateContexts.windowOnly for temp...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1389

Restore StateContexts.windowOnly for temporary compatibility

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This addresses the postcommit failure here: 
https://builds.apache.org/job/beam_PostCommit_RunnableOnService_GoogleCloudDataflow/1600/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.transforms/CombineTest/testSessionsCombineWithContext/

R: @davorbonaci @jkff 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam StateContexts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1389.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1389


commit bf469d0f9e668b87fe276c9c9bc6b230dd931a6e
Author: Kenneth Knowles 
Date:   2016-11-18T22:40:59Z

Restore StateContexts.windowOnly for temporary compatibility




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-767) Add DisplayData builder '.nested(..)' operation

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner closed BEAM-767.
-

> Add DisplayData builder '.nested(..)' operation
> ---
>
> Key: BEAM-767
> URL: https://issues.apache.org/jira/browse/BEAM-767
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
> Fix For: Not applicable
>
>
> See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515
> In the Java SDK, components can register display data directly, as well as 
> include display data from sub-components via the 'include(..)' operation. In 
> [PR #1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
> to display data metadata and also include a 'delegate(..)' action.
> During the change for paths, we noticed an emerging pattern where components 
> will typically add the .getClass() of a subcomponent before including it. We 
> can support this convention better by embedding it in a single configurable 
> operation, '.nested(..)'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-767) Add DisplayData builder '.nested(..)' operation

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677988#comment-15677988
 ] 

Scott Wegner commented on BEAM-767:
---

After chatting with @lukecwik, we decided that it was confusing to have a 
.nested(..) operation which does two things (effectively .add(key, 
comp.getClass()).include(key, comp)). We thought it would be surprising to have 
an automatically-added class item. Furthermore, the display data API today is 
purely additive, and it was confusing having configuration that could configure 
and subtract in the same fluent API 
(.nested(..).withLabel(..).withoutClassItem()).

We concluded that the current API is preferable because it is easier to 
understand and use, albeit being more syntax.

> Add DisplayData builder '.nested(..)' operation
> ---
>
> Key: BEAM-767
> URL: https://issues.apache.org/jira/browse/BEAM-767
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
> Fix For: Not applicable
>
>
> See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515
> In the Java SDK, components can register display data directly, as well as 
> include display data from sub-components via the 'include(..)' operation. In 
> [PR #1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
> to display data metadata and also include a 'delegate(..)' action.
> During the change for paths, we noticed an emerging pattern where components 
> will typically add the .getClass() of a subcomponent before including it. We 
> can support this convention better by embedding it in a single configurable 
> operation, '.nested(..)'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-767) Add DisplayData builder '.nested(..)' operation

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner resolved BEAM-767.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Add DisplayData builder '.nested(..)' operation
> ---
>
> Key: BEAM-767
> URL: https://issues.apache.org/jira/browse/BEAM-767
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
> Fix For: Not applicable
>
>
> See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515
> In the Java SDK, components can register display data directly, as well as 
> include display data from sub-components via the 'include(..)' operation. In 
> [PR #1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
> to display data metadata and also include a 'delegate(..)' action.
> During the change for paths, we noticed an emerging pattern where components 
> will typically add the .getClass() of a subcomponent before including it. We 
> can support this convention better by embedding it in a single configurable 
> operation, '.nested(..)'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677946#comment-15677946
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1386


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1386: [BEAM-498] Move DoFn.ArgumentProvider to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1386


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1386

[kenn]

This closes #1386


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2a7169b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2a7169b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2a7169b6

Branch: refs/heads/master
Commit: 2a7169b6f62de765fe1e4b28bad195f5e102aec7
Parents: bb9c386 33fb8c2
Author: Kenneth Knowles 
Authored: Fri Nov 18 14:20:21 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:20:21 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   7 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 122 ---
 .../beam/sdk/transforms/DoFnAdapters.java   |  10 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  41 +++
 .../sdk/transforms/reflect/DoFnInvoker.java | 121 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   4 +-
 .../sdk/transforms/reflect/OnTimerInvoker.java  |   8 +-
 .../transforms/reflect/DoFnInvokersTest.java|   5 +-
 .../transforms/reflect/OnTimerInvokersTest.java |   2 +-
 .../transforms/DoFnInvokersBenchmark.java   |   5 +-
 11 files changed, 161 insertions(+), 168 deletions(-)
--

[1/2] incubator-beam git commit: Move DoFn.ArgumentProvider to DoFnInvoker.ArgumentProvider

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master bb9c38664 -> 2a7169b6f


Move DoFn.ArgumentProvider to DoFnInvoker.ArgumentProvider

The arguments provided as a single object are an aspect of the
DoFnInvoker, not the DoFn. The DoFn itself is a specification
that may have other ways of being invoked, depending on the
circumstance.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33fb8c2d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33fb8c2d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33fb8c2d

Branch: refs/heads/master
Commit: 33fb8c2db8c64275f1b9b8ac6dfd12e92d7fb777
Parents: bb9c386
Author: Kenneth Knowles 
Authored: Thu Nov 17 23:04:55 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Nov 18 14:20:20 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   7 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 122 ---
 .../beam/sdk/transforms/DoFnAdapters.java   |  10 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java|  41 +++
 .../sdk/transforms/reflect/DoFnInvoker.java | 121 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   4 +-
 .../sdk/transforms/reflect/OnTimerInvoker.java  |   8 +-
 .../transforms/reflect/DoFnInvokersTest.java|   5 +-
 .../transforms/reflect/OnTimerInvokersTest.java |   2 +-
 .../transforms/DoFnInvokersBenchmark.java   |   5 +-
 11 files changed, 161 insertions(+), 168 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33fb8c2d/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 76aae8f..841e412 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -183,7 +183,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner extends DoFn.Context
-  implements DoFn.ArgumentProvider {
+  implements DoFnInvoker.ArgumentProvider {
 private static final int MAX_SIDE_OUTPUTS = 1000;
 
 final PipelineOptions options;
@@ -424,7 +424,7 @@ public class SimpleDoFnRunner implements 
DoFnRunner extends DoFn.ProcessContext
-  implements DoFn.ArgumentProvider {
+  implements DoFnInvoker.ArgumentProvider {
 
 final DoFn fn;
 final DoFnContext context;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/33fb8c2d/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
index 3003984..c38ab2f 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDo.java
@@ -392,10 +392,11 @@ public class SplittableParDo<
 }
 
 /**
- * Creates an {@link DoFn.ArgumentProvider} that provides the given 
tracker as well as the given
+ * Creates an {@link DoFnInvoker.ArgumentProvider} that provides the given 
tracker as well as
+ * the given
  * {@link ProcessContext} (which is also provided when a {@link Context} 
is requested.
  */
-private DoFn.ArgumentProvider wrapTracker(
+private DoFnInvoker.ArgumentProvider wrapTracker(
 TrackerT tracker, DoFn.ProcessContext processContext) 
{
 
   return new ArgumentProviderForTracker<>(tracker, processContext);
@@ -403,7 +404,7 @@ public class SplittableParDo<
 
 private static class ArgumentProviderForTracker<
 InputT, OutputT, TrackerT extends RestrictionTracker>
-implements DoFn.ArgumentProvider {
+implements DoFnInvoker.ArgumentProvider {
   private final 

[GitHub] incubator-beam pull request #1385: Fixes couple of issues of FileBasedSource...

[chamikaramj]

Github user chamikaramj closed the pull request at:

https://github.com/apache/incubator-beam/pull/1385


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1383: Remove redundant REQUIRED_TEST_PACKAGES

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1383


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1380

[robertwb]

Closes #1380


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cce4331d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cce4331d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cce4331d

Branch: refs/heads/python-sdk
Commit: cce4331dc7ed95aa32654e77d2cc170b63437183
Parents: 45b420d 99bcafe
Author: Robert Bradshaw 
Authored: Fri Nov 18 13:37:05 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:37:05 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py  |  5 +++
 sdks/python/apache_beam/io/fileio_test.py | 47 ++
 2 files changed, 52 insertions(+)
--

[1/2] incubator-beam git commit: Fix issue where batch GCS renames were not issued

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 45b420d82 -> cce4331dc


Fix issue where batch GCS renames were not issued


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99bcafe7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99bcafe7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/99bcafe7

Branch: refs/heads/python-sdk
Commit: 99bcafe7a02bbec5222d77abbad24f5eed8a687f
Parents: 45b420d
Author: Charles Chen 
Authored: Thu Nov 17 14:13:56 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:37:04 2016 -0800

--
 sdks/python/apache_beam/io/fileio.py  |  5 +++
 sdks/python/apache_beam/io/fileio_test.py | 47 ++
 2 files changed, 52 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99bcafe7/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 3b67c4f..4d0eea6 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -514,6 +514,7 @@ class ChannelFactory(object):
 gcs_batches = []
 gcs_current_batch = []
 for src, dest in src_dest_pairs:
+  gcs_current_batch.append((src, dest))
   if len(gcs_current_batch) == gcsio.MAX_BATCH_OPERATION_SIZE:
 gcs_batches.append(gcs_current_batch)
 gcs_current_batch = []
@@ -893,6 +894,8 @@ class FileSink(iobase.Sink):
   exception_infos = ChannelFactory.rename_batch(batch)
   for src, dest, exception in exception_infos:
 if exception:
+  logging.warning('Rename not successful: %s -> %s, %s', src, dest,
+  exception)
   should_report = True
   if isinstance(exception, IOError):
 # May have already been copied.
@@ -906,6 +909,8 @@ class FileSink(iobase.Sink):
 logging.warning(('Exception in _rename_batch. src: %s, '
  'dest: %s, err: %s'), src, dest, exception)
 exceptions.append(exception)
+else:
+  logging.debug('Rename successful: %s -> %s', src, dest)
   return exceptions
 
 # ThreadPool crashes in old versions of Python (< 2.7.5) if created from a

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99bcafe7/sdks/python/apache_beam/io/fileio_test.py
--
diff --git a/sdks/python/apache_beam/io/fileio_test.py 
b/sdks/python/apache_beam/io/fileio_test.py
index 63e71e0..9d1e424 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -28,6 +28,7 @@ import unittest
 import zlib
 
 import hamcrest as hc
+import mock
 
 import apache_beam as beam
 from apache_beam import coders
@@ -881,6 +882,52 @@ class TestFileSink(unittest.TestCase):
 with self.assertRaises(Exception):
   list(sink.finalize_write(init_token, [res1, res2]))
 
+  @mock.patch('apache_beam.io.fileio.ChannelFactory.rename')
+  @mock.patch('apache_beam.io.fileio.gcsio')
+  def test_rename_batch(self, *unused_args):
+# Prepare mocks.
+gcsio_mock = mock.MagicMock()
+fileio.gcsio.GcsIO = lambda: gcsio_mock
+fileio.ChannelFactory.rename = mock.MagicMock()
+to_rename = [
+('gs://bucket/from1', 'gs://bucket/to1'),
+('gs://bucket/from2', 'gs://bucket/to2'),
+('/local/from1', '/local/to1'),
+('gs://bucket/from3', 'gs://bucket/to3'),
+('/local/from2', '/local/to2'),
+]
+gcsio_mock.copy_batch.side_effect = [[
+('gs://bucket/from1', 'gs://bucket/to1', None),
+('gs://bucket/from2', 'gs://bucket/to2', None),
+('gs://bucket/from3', 'gs://bucket/to3', None),
+]]
+gcsio_mock.delete_batch.side_effect = [[
+('gs://bucket/from1', None),
+('gs://bucket/from2', None),
+('gs://bucket/from3', None),
+]]
+
+# Issue batch rename.
+fileio.ChannelFactory.rename_batch(to_rename)
+
+# Verify mocks.
+expected_local_rename_calls = [
+mock.call('/local/from1', '/local/to1'),
+mock.call('/local/from2', '/local/to2'),
+]
+self.assertEqual(fileio.ChannelFactory.rename.call_args_list,
+ expected_local_rename_calls)
+gcsio_mock.copy_batch.assert_called_once_with([
+('gs://bucket/from1', 'gs://bucket/to1'),
+('gs://bucket/from2', 'gs://bucket/to2'),
+('gs://bucket/from3', 'gs://bucket/to3'),
+])
+gcsio_mock.delete_batch.assert_called_once_with([
+'gs://bucket/from1',
+'gs://bucket/from2',
+'gs://bucket/from3',
+])
+
 
 if __name__ 

[1/2] incubator-beam git commit: Remove redundant REQUIRED_PACKAGES

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 3a0f01c8e -> 45b420d82


Remove redundant REQUIRED_PACKAGES


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/329be6e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/329be6e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/329be6e9

Branch: refs/heads/python-sdk
Commit: 329be6e9a16bfe865d61c3d6041ec5fb6707fc6a
Parents: 3a0f01c
Author: Ahmet Altay 
Authored: Thu Nov 17 15:20:08 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:35:47 2016 -0800

--
 sdks/python/setup.py | 3 ---
 1 file changed, 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/329be6e9/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index b8034af..1299bbf 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -96,9 +96,6 @@ REQUIRED_PACKAGES = [
 'python-gflags>=2.0,<4.0.0',
 'pyyaml>=3.10,<4.0.0',
 ]
-REQUIRED_TEST_PACKAGES = [
-'pyhamcrest>=1.9,<2.0',
-]
 
 
 REQUIRED_TEST_PACKAGES = [

[2/2] incubator-beam git commit: Closes #1383

[robertwb]

Closes #1383


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/45b420d8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/45b420d8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/45b420d8

Branch: refs/heads/python-sdk
Commit: 45b420d82aa6f47e3d37f5aa5ba98378cdc01e9c
Parents: 3a0f01c 329be6e
Author: Robert Bradshaw 
Authored: Fri Nov 18 13:35:48 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:35:48 2016 -0800

--
 sdks/python/setup.py | 3 ---
 1 file changed, 3 deletions(-)
--

[2/2] incubator-beam git commit: Closes #1385

[robertwb]

Closes #1385


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3a0f01c8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3a0f01c8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3a0f01c8

Branch: refs/heads/python-sdk
Commit: 3a0f01c8edd36fe525b8ad155011dfb759dad2b4
Parents: b83f12b 93c5233
Author: Robert Bradshaw 
Authored: Fri Nov 18 13:33:46 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:33:46 2016 -0800

--
 sdks/python/apache_beam/io/filebasedsource.py | 14 ++
 .../python/apache_beam/io/filebasedsource_test.py | 18 +-
 2 files changed, 27 insertions(+), 5 deletions(-)
--

[1/2] incubator-beam git commit: Fixes a couple of issues of FileBasedSource.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk b83f12b9f -> 3a0f01c8e


Fixes a couple of issues of FileBasedSource.

(1) Updates code so that a user-specified coder properly gets set to 
sub-sources.

(2) Currently each SingleFileSource takes a reference to FileBasedSource while  
FileBasedSource takes a reference to Concatsource.  ConcatSource has a 
reference to list of SingleFileSources. This results in quadratic space 
complexity when serializing splits of a FileBasedSource. This CL fixes this 
issue by making sure that FileBasedSource is cloned before taking a reference 
to  ConcatSource


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/93c5233a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/93c5233a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/93c5233a

Branch: refs/heads/python-sdk
Commit: 93c5233a1bf28e9b13412b909c2ee877bd6cf635
Parents: b83f12b
Author: Chamikara Jayalath 
Authored: Thu Nov 17 19:18:26 2016 -0800
Committer: Robert Bradshaw 
Committed: Fri Nov 18 13:33:33 2016 -0800

--
 sdks/python/apache_beam/io/filebasedsource.py | 14 ++
 .../python/apache_beam/io/filebasedsource_test.py | 18 +-
 2 files changed, 27 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93c5233a/sdks/python/apache_beam/io/filebasedsource.py
--
diff --git a/sdks/python/apache_beam/io/filebasedsource.py 
b/sdks/python/apache_beam/io/filebasedsource.py
index c7bc27e..7d8f686 100644
--- a/sdks/python/apache_beam/io/filebasedsource.py
+++ b/sdks/python/apache_beam/io/filebasedsource.py
@@ -109,6 +109,12 @@ class FileBasedSource(iobase.BoundedSource):
   file_names = [f for f in fileio.ChannelFactory.glob(self._pattern)]
   sizes = FileBasedSource._estimate_sizes_in_parallel(file_names)
 
+  # We create a reference for FileBasedSource that will be serialized along
+  # with each _SingleFileSource. To prevent this FileBasedSource from 
having
+  # a reference to ConcatSource (resulting in quadratic space complexity)
+  # we clone it here.
+  file_based_source_ref = pickler.loads(pickler.dumps(self))
+
   for index, file_name in enumerate(file_names):
 if sizes[index] == 0:
   continue  # Ignoring empty file.
@@ -123,7 +129,7 @@ class FileBasedSource(iobase.BoundedSource):
 splittable = False
 
 single_file_source = _SingleFileSource(
-self, file_name,
+file_based_source_ref, file_name,
 0,
 sizes[index],
 min_bundle_size=self._min_bundle_size,
@@ -194,9 +200,6 @@ class FileBasedSource(iobase.BoundedSource):
 return self._get_concat_source().get_range_tracker(start_position,
stop_position)
 
-  def default_output_coder(self):
-return self._get_concat_source().default_output_coder()
-
   def read_records(self, file_name, offset_range_tracker):
 """Returns a generator of records created by reading file 'file_name'.
 
@@ -315,3 +318,6 @@ class _SingleFileSource(iobase.BoundedSource):
 
   def read(self, range_tracker):
 return self._file_based_source.read_records(self._file_name, range_tracker)
+
+  def default_output_coder(self):
+return self._file_based_source.default_output_coder()

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93c5233a/sdks/python/apache_beam/io/filebasedsource_test.py
--
diff --git a/sdks/python/apache_beam/io/filebasedsource_test.py 
b/sdks/python/apache_beam/io/filebasedsource_test.py
index 7f4d8d3..a455cd3 100644
--- a/sdks/python/apache_beam/io/filebasedsource_test.py
+++ b/sdks/python/apache_beam/io/filebasedsource_test.py
@@ -533,6 +533,23 @@ class TestFileBasedSource(unittest.TestCase):
 assert_that(pcoll, equal_to(lines))
 pipeline.run()
 
+  def test_splits_get_coder_from_fbs(self):
+class DummyCoder(object):
+  val = 12345
+
+class FileBasedSourceWithCoder(LineSource):
+
+  def default_output_coder(self):
+return DummyCoder()
+
+pattern, expected_data = write_pattern([34, 66, 40, 24, 24, 12])
+self.assertEqual(200, len(expected_data))
+fbs = FileBasedSourceWithCoder(pattern)
+splits = [split for split in fbs.split(desired_bundle_size=50)]
+self.assertTrue(len(splits))
+for split in splits:
+  self.assertEqual(DummyCoder.val, split.source.default_output_coder().val)
+
 
 class TestSingleFileSource(unittest.TestCase):
 
@@ -685,7 +702,6 @@ class TestSingleFileSource(unittest.TestCase):
   

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1604

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1603

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1388: Move PerKeyCombineFnRunners to runners/co...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1388

Move PerKeyCombineFnRunners to runners/core.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam rm-keyed-combine-runners

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1388.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1388


commit fcfcdd6ed7315c1a47864c09433b0aec02d6a21c
Author: Pei He 
Date:   2016-11-14T22:15:35Z

Move PerKeyCombineFnRunners to runners/core.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1602

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-436) DirectRunner RunnableOnService tempLocation configuration insufficient

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-436:
--
Assignee: Scott Wegner  (was: Davor Bonaci)

> DirectRunner RunnableOnService tempLocation configuration insufficient
> --
>
> Key: BEAM-436
> URL: https://issues.apache.org/jira/browse/BEAM-436
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> The way we specify temp storage locations for RunnableOnService tests is not 
> sufficient, specifically for DirectRunner execution. Right now, 
> RunnableOnService tests are run for DirectRunner and DataflowRunner, who set 
> their temp locations differently:
> * DirectRunner doesn't specify a temp location directly, but test classes 
> will use a JUnit @Rule TemporaryDirectory. Individual tests set it as 
> necessary for tempLocation, and set a fake gs:// path for individual GCP IO 
> tests.
> * DataflowRunner tests pass an actual GCS path as tempRoot, and 
> TestDataflowRunner will initialize stagingLocation to this path.
> This setup makes it difficult to write RunnableOnService tests which pass for 
> both runners. We should separate temp location setup out of individual test 
> classes so that RunnableOnService tests "just work" on any runner.
> One solution would be to add logic inside TestPipeline#testingPipelineOptions:
> # If --tempRoot is specified, use it to set tempLocation and stagingLocation. 
> Otherwise, use a JUnit TemporaryDirectory to set it
> # If tempLocation is a GCS path, use it to set stagingLocation. Otherwise, 
> use a fake gcs path (i.e. gs://foo)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #1381

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4852d2ebf -> bb9c38664


Closes #1381


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bb9c3866
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bb9c3866
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bb9c3866

Branch: refs/heads/master
Commit: bb9c3866421863b257470ec9fb6bcb8328158045
Parents: 4852d2e 5fc68a5
Author: bchambers 
Authored: Fri Nov 18 09:55:29 2016 -0800
Committer: bchambers 
Committed: Fri Nov 18 09:55:29 2016 -0800

--
 .../org/apache/beam/sdk/metrics/MetricsContainer.java  | 13 -
 1 file changed, 13 deletions(-)
--

[GitHub] incubator-beam pull request #1381: Remove dead code from MetricsContainer

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1381


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Remove dead code from MetricsContainer

[bchambers]

Remove dead code from MetricsContainer


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5fc68a5c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5fc68a5c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5fc68a5c

Branch: refs/heads/master
Commit: 5fc68a5c109fdbe3ed8afb370c8fcee803e9f1d8
Parents: 4852d2e
Author: bchambers 
Authored: Thu Nov 17 14:40:52 2016 -0800
Committer: bchambers 
Committed: Fri Nov 18 09:55:29 2016 -0800

--
 .../org/apache/beam/sdk/metrics/MetricsContainer.java  | 13 -
 1 file changed, 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5fc68a5c/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
index 10032a2..ba5a343 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricsContainer.java
@@ -130,19 +130,6 @@ public class MetricsContainer {
* container.
*/
   public MetricUpdates getCumulative() {
-ImmutableList.Builder counterUpdates = 
ImmutableList.builder();
-for (Map.Entry counter : counters.entries()) {
-  counterUpdates.add(MetricUpdate.create(
-  MetricKey.create(stepName, counter.getKey()), 
counter.getValue().getCumulative()));
-}
-
-ImmutableList.Builder distributionUpdates =
-ImmutableList.builder();
-for (Map.Entry distribution : 
distributions.entries()) {
-  distributionUpdates.add(MetricUpdate.create(
-  MetricKey.create(stepName, distribution.getKey()),
-  distribution.getValue().getCumulative()));
-}
 return MetricUpdates.create(
 extractCumulatives(counters),
 extractCumulatives(distributions));

[jira] [Commented] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[David Billings (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677065#comment-15677065
 ] 

David Billings commented on BEAM-1006:
--

I've fixed in PR https://github.com/apache/incubator-beam/pull/1387

Works locally against my mongodb

> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: James Malone
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[David Billings (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Billings updated BEAM-1006:
-
Comment: was deleted

(was: I've fixed in PR https://github.com/apache/incubator-beam/pull/1387

Works locally against my mongodb)

> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: James Malone
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1387: [BEAM-1006] Fixing the splitKeysToFilters...

[david-billings]

GitHub user david-billings opened a pull request:

https://github.com/apache/incubator-beam/pull/1387

[BEAM-1006] Fixing the splitKeysToFilters methofd as calling toString on 
the Document prefixed …"Document"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…Document to the string, updated it to just get the value of ID. Also 
fixed a typo

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/david-billings/incubator-beam 
MongoDbIO-filter-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1387


commit b8fb7c638c47eb9a555b16c84b99b2b4e23466ae
Author: DavidB 
Date:   2016-11-18T15:11:47Z

Fixing the splitkey, calling to string on the BSON document prefixed 
Document to the string, updated it to just get the value of ID. Also fixed a 
typo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677058#comment-15677058
 ] 

ASF GitHub Bot commented on BEAM-1006:
--

GitHub user david-billings opened a pull request:

https://github.com/apache/incubator-beam/pull/1387

[BEAM-1006] Fixing the splitKeysToFilters methofd as calling toString on 
the Document prefixed …"Document"

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…Document to the string, updated it to just get the value of ID. Also fixed 
a typo

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/david-billings/incubator-beam 
MongoDbIO-filter-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1387.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1387


commit b8fb7c638c47eb9a555b16c84b99b2b4e23466ae
Author: DavidB 
Date:   2016-11-18T15:11:47Z

Fixing the splitkey, calling to string on the BSON document prefixed 
Document to the string, updated it to just get the value of ID. Also fixed a 
typo




> splitKeysToFilter method is creating a bad query string.
> 
>
> Key: BEAM-1006
> URL: https://issues.apache.org/jira/browse/BEAM-1006
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: 0.3.0-incubating
>Reporter: David Billings
>Assignee: James Malone
>Priority: Critical
>  Labels: mongodb
> Fix For: 0.4.0-incubating
>
>
> When you have a large number of documents in your mongo collection the 
> splitKeysToFilter method is invoked. 
> Currently Document{{_id= is prefixed to the query string
> {code}{ $and: [ 
> {"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}
> This causes the document parse to fail with the following exception: 
> Exception in thread "main" java.lang.IllegalArgumentException: invalid 
> hexadecimal representation of an ObjectId: 
> [Document{{_id=56e1ec546b2f0700015e61b2}}]
> This is caused by line 305 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> The code should use 
> {code}
> String splitKey = splitKeys.get(i).toString();
> {code}
> to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1006) splitKeysToFilter method is creating a bad query string.

[David Billings (JIRA)]

David Billings created BEAM-1006:


 Summary: splitKeysToFilter method is creating a bad query string.
 Key: BEAM-1006
 URL: https://issues.apache.org/jira/browse/BEAM-1006
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: 0.3.0-incubating
Reporter: David Billings
Assignee: James Malone
Priority: Critical
 Fix For: 0.4.0-incubating


When you have a large number of documents in your mongo collection the 
splitKeysToFilter method is invoked. 

Currently Document{{_id= is prefixed to the query string

{code}{ $and: [ 
{"_id":{$lte:Objectd("Document{{_id=564a0ad6e21b840001d75f5d}}")}} ]}{code}

This causes the document parse to fail with the following exception: 

Exception in thread "main" java.lang.IllegalArgumentException: invalid 
hexadecimal representation of an ObjectId: 
[Document{{_id=56e1ec546b2f0700015e61b2}}]

This is caused by line 305 
{code}
String splitKey = splitKeys.get(i).toString();
{code}

The code should use 
{code}
String splitKey = splitKeys.get(i).toString();
{code}

to just return the ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1354: Add IP configuration to Python SDK

[sammcveety]

Github user sammcveety closed the pull request at:

https://github.com/apache/incubator-beam/pull/1354


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-920) Support triggers and panes in streaming.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-920:
---
Description: Implement event-time based aggregation using triggers, panes 
and watermarks.  (was: Use Spark's {{mapWithState}} operator to maintain 
accumulated state across (micro) batches and the SDK to decide when to trigger 
and how to accumulate across panes. )

> Support triggers and panes in streaming.
> 
>
> Key: BEAM-920
> URL: https://issues.apache.org/jira/browse/BEAM-920
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Implement event-time based aggregation using triggers, panes and watermarks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-920) Support triggers, panes and watermarks.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-920:
---
Issue Type: New Feature  (was: Bug)

> Support triggers, panes and watermarks.
> ---
>
> Key: BEAM-920
> URL: https://issues.apache.org/jira/browse/BEAM-920
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> Implement event-time based aggregation using triggers, panes and watermarks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1600

[Apache Jenkins Server]

See