[GitHub] incubator-beam pull request #508: Dataflow runners core

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/508


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #508: Dataflow runners core

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/508

Dataflow runners core

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
dataflow-runners-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #508


commit 426d873d15d7f28603e026092bfbe64aeb727053
Author: Kenneth Knowles 
Date:   2016-06-20T22:05:59Z

Break dependency from Dataflow runner shim to runners-core

commit 3adccbdf8486a3cc8e67b94697d3bcecfd8c17ea
Author: Kenn Knowles 
Date:   2016-06-21T03:03:47Z

Move GABW shim from Dataflow runner to runners/core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-360) Add a framework for creating Python-SDK sources for new file types

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340898#comment-15340898
 ] 

ASF GitHub Bot commented on BEAM-360:
-

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/507

[BEAM-360] Implements a framework for developing Python SDK sources for new 
file types

Module 'filebasedsource' provides a framework for  creating sources for new 
file types. This framework implements several features common to many sources 
based on files.

Additionally, module 'avroio' contains a new source, 'AvroSource', that is 
implemented using the framework described above. 'AvroSource' is a source for 
reading Avro files.

Adds many unit tests for 'filebasedsource' and 'avroio' modules.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam filebasedsource

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #507


commit e51d4acf12133a79671c567c9ff709c941c54f8c
Author: Chamikara Jayalath 
Date:   2016-06-21T01:09:50Z

Implements a framework for developing sources for new file types.

Module 'filebasedsource' provides a framework for  creating sources for new 
file types. This framework readily implements several features common to many 
sources based on files.

Additionally, module 'avroio' contains a new source, 'AvroSource', that is 
implemented using the framework described above. 'AvroSource' is a source for 
reading Avro files.

Adds many unit tests for 'filebasedsource' and 'avroio' modules.




> Add a framework for creating Python-SDK sources for new file types
> --
>
> Key: BEAM-360
> URL: https://issues.apache.org/jira/browse/BEAM-360
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> We already have a framework for creating new sources for Beam Python SDK - 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/iobase.py#L326
> It would be great if we can add a framework on top of this that encapsulates 
> logic common to sources that are based on files. This framework can include 
> following features that are common to sources based on files.
> (1) glob expansion
> (2) support for new file-systems
> (3) dynamic work rebalancing based on byte offsets
> (4) support for reading compressed files.
> Java SDK has a similar framework and it's available at - 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #507: [BEAM-360] Implements a framework for deve...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/507

[BEAM-360] Implements a framework for developing Python SDK sources for new 
file types

Module 'filebasedsource' provides a framework for  creating sources for new 
file types. This framework implements several features common to many sources 
based on files.

Additionally, module 'avroio' contains a new source, 'AvroSource', that is 
implemented using the framework described above. 'AvroSource' is a source for 
reading Avro files.

Adds many unit tests for 'filebasedsource' and 'avroio' modules.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam filebasedsource

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #507


commit e51d4acf12133a79671c567c9ff709c941c54f8c
Author: Chamikara Jayalath 
Date:   2016-06-21T01:09:50Z

Implements a framework for developing sources for new file types.

Module 'filebasedsource' provides a framework for  creating sources for new 
file types. This framework readily implements several features common to many 
sources based on files.

Additionally, module 'avroio' contains a new source, 'AvroSource', that is 
implemented using the framework described above. 'AvroSource' is a source for 
reading Avro files.

Adds many unit tests for 'filebasedsource' and 'avroio' modules.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-357) Build doesn't pass on windows

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340738#comment-15340738
 ] 

Davor Bonaci commented on BEAM-357:
---

We also need to make sure to add tests, to prevent regressions in this scenario.

> Build doesn't pass on windows
> -
>
> Key: BEAM-357
> URL: https://issues.apache.org/jira/browse/BEAM-357
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, runner-spark, sdk-java-extensions
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
> Fix For: 0.2.0-incubating
>
> Attachments: bean-win.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-357) Build doesn't pass on windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-357:
--
Assignee: Romain Manni-Bucau  (was: Jean-Baptiste Onofré)

> Build doesn't pass on windows
> -
>
> Key: BEAM-357
> URL: https://issues.apache.org/jira/browse/BEAM-357
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, runner-spark, sdk-java-extensions
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
> Fix For: 0.2.0-incubating
>
> Attachments: bean-win.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-357) Build doesn't pass on windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-357:
--
Component/s: (was: beam-model)
 sdk-java-extensions
 runner-spark
 runner-flink

> Build doesn't pass on windows
> -
>
> Key: BEAM-357
> URL: https://issues.apache.org/jira/browse/BEAM-357
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, runner-spark, sdk-java-extensions
>Reporter: Romain Manni-Bucau
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.2.0-incubating
>
> Attachments: bean-win.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-357) Build doesn't pass on windows

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-357:
--
Fix Version/s: 0.2.0-incubating

> Build doesn't pass on windows
> -
>
> Key: BEAM-357
> URL: https://issues.apache.org/jira/browse/BEAM-357
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, runner-spark, sdk-java-extensions
>Reporter: Romain Manni-Bucau
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.2.0-incubating
>
> Attachments: bean-win.diff
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-252) Make Regex Transform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340697#comment-15340697
 ] 

ASF GitHub Bot commented on BEAM-252:
-

GitHub user eljefe6a opened a pull request:

https://github.com/apache/incubator-beam/pull/506

[BEAM-252] Make Regex Transform

Jira issue BEAM-252 Make Regex Transform. Adding a pre-built transform for 
Regex.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/eljefe6a/incubator-beam RegexTransform

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #506


commit 008ff6fc91ee3ed48764550cd3349afa2801b5e7
Author: Jesse Anderson 
Date:   2016-06-20T23:44:43Z

Added Regex Transform and test.




> Make Regex Transform
> 
>
> Key: BEAM-252
> URL: https://issues.apache.org/jira/browse/BEAM-252
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Kenneth Knowles
>
> There needs to be an easier way to run Regular Expressions as part of a 
> transform. This will make string-based ETL much easier.
> The transform should support using the matches and find methods. The 
> transform should allow you to choose a group in the regex to output. The 
> transform should allow single strings to be output or KV's of strings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-364) Integration Tests for Bigtable Read and Write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340694#comment-15340694
 ] 

ASF GitHub Bot commented on BEAM-364:
-

GitHub user ianzhou1 opened a pull request:

https://github.com/apache/incubator-beam/pull/505

[BEAM-364] End to end integration tests for Bigtable Read and Write

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ianzhou1/incubator-beam integration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit c2c0bea8a2200f68817248241f3b6c57fedc1fed
Author: Ian Zhou 
Date:   2016-06-17T21:21:05Z

Added integration tests for BigtableRead and BigtableWrite

commit bb3138f594593ad4cd8ce0c476003652d9291e97
Author: Ian Zhou 
Date:   2016-06-20T23:35:44Z

Added BigtableWrite and BigtableRead integration tests




> Integration Tests for Bigtable Read and Write
> -
>
> Key: BEAM-364
> URL: https://issues.apache.org/jira/browse/BEAM-364
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-gcp
>Reporter: Ian Zhou
>Assignee: Daniel Halperin
>Priority: Minor
>
> Integration tests should be added for BigtableIO.read and BigtableIO.write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-364) Integration Tests for Bigtable Read and Write

[Ian Zhou (JIRA)]

Ian Zhou created BEAM-364:
-

 Summary: Integration Tests for Bigtable Read and Write
 Key: BEAM-364
 URL: https://issues.apache.org/jira/browse/BEAM-364
 Project: Beam
  Issue Type: Test
  Components: sdk-java-gcp
Reporter: Ian Zhou
Assignee: Daniel Halperin
Priority: Minor


Integration tests should be added for BigtableIO.read and BigtableIO.write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #505: [BEAM-364] End to end integration tests fo...

[ianzhou1]

GitHub user ianzhou1 opened a pull request:

https://github.com/apache/incubator-beam/pull/505

[BEAM-364] End to end integration tests for Bigtable Read and Write

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ianzhou1/incubator-beam integration

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit c2c0bea8a2200f68817248241f3b6c57fedc1fed
Author: Ian Zhou 
Date:   2016-06-17T21:21:05Z

Added integration tests for BigtableRead and BigtableWrite

commit bb3138f594593ad4cd8ce0c476003652d9291e97
Author: Ian Zhou 
Date:   2016-06-20T23:35:44Z

Added BigtableWrite and BigtableRead integration tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #269: [BEAM-252] Make Regex Transform

[eljefe6a]

Github user eljefe6a closed the pull request at:

https://github.com/apache/incubator-beam/pull/269


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-252) Make Regex Transform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340664#comment-15340664
 ] 

ASF GitHub Bot commented on BEAM-252:
-

Github user eljefe6a closed the pull request at:

https://github.com/apache/incubator-beam/pull/269


> Make Regex Transform
> 
>
> Key: BEAM-252
> URL: https://issues.apache.org/jira/browse/BEAM-252
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Kenneth Knowles
>
> There needs to be an easier way to run Regular Expressions as part of a 
> transform. This will make string-based ETL much easier.
> The transform should support using the matches and find methods. The 
> transform should allow you to choose a group in the regex to output. The 
> transform should allow single strings to be output or KV's of strings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-363) CompressedSource splits when it shouldn't

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340657#comment-15340657
 ] 

ASF GitHub Bot commented on BEAM-363:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/504


> CompressedSource splits when it shouldn't
> -
>
> Key: BEAM-363
> URL: https://issues.apache.org/jira/browse/BEAM-363
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> CompressedSource is splitting when it should not. This can in some corner 
> cases lead to data loss during dynamic work rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #504

[dhalperi]

Closes #504


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b9906e8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b9906e8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b9906e8

Branch: refs/heads/master
Commit: 2b9906e8d97c254706656ac68eec85f29f946627
Parents: 682a19c 278a76e
Author: Dan Halperin 
Authored: Mon Jun 20 16:21:45 2016 -0700
Committer: Dan Halperin 
Committed: Mon Jun 20 16:21:45 2016 -0700

--
 .../apache/beam/sdk/io/CompressedSource.java|  2 +-
 .../beam/sdk/io/CompressedSourceTest.java   | 50 
 2 files changed, 51 insertions(+), 1 deletion(-)
--

[jira] [Closed] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-265.
-
   Resolution: Invalid
Fix Version/s: Not applicable

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-265) Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification Language Independence, Transo

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340628#comment-15340628
 ] 

Davor Bonaci commented on BEAM-265:
---

Closing / inactive.

> Dynamic Pipeline, Transformation, etc. / DAG Changes DSL | Full Specification 
> Language Independence, Transo
> ---
>
> Key: BEAM-265
> URL: https://issues.apache.org/jira/browse/BEAM-265
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
>
> There should conditional / dynamic changes to pipelines and DAGs. The Beam 
> specification should ideally be:
>  - fully language neutral to support many languages
>  - should be specified statically so code generation can work. E.g. pipeline 
> specified in Java but run as native. In this case the conversion would be 
> like that GWT does with Java.
> In addition the transformation code though expressed in a given language 
> should be transformable into an language independent form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-264) Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and Construction of Such Structures from Scalars, Other Structures and Queries

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340624#comment-15340624
 ] 

Davor Bonaci commented on BEAM-264:
---

Closing / inactive.

> Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and 
> Construction of Such Structures from Scalars, Other Structures and Queries
> -
>
> Key: BEAM-264
> URL: https://issues.apache.org/jira/browse/BEAM-264
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> Not all timeseries data is scalar values hence support other types of values 
> like vectors, matrices. Also ability to construct these constructs. E.g. if I 
> want a time series vector of asset prices as input to a pipeline. Also I will 
> have to choose what stocks are in the vector and also how to handle if the 
> choosing is time varying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-263) Generate Processing Infrastructure Code and Code for Runners in High Level Languages

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340622#comment-15340622
 ] 

Davor Bonaci commented on BEAM-263:
---

Closing / inactive.

> Generate Processing Infrastructure Code and Code for Runners in High Level 
> Languages
> 
>
> Key: BEAM-263
> URL: https://issues.apache.org/jira/browse/BEAM-263
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> In addition you can auto generate processing infrastructure code in languages 
> like C/C++, Java, Scala, C#, F#, Rust, Julia, Swift, D, Nim, etc.
> Auto generated code will be the runner itself and should not need any 
> additional infrastructure to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-264) Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and Construction of Such Structures from Scalars, Other Structures and Queries

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-264.
-
   Resolution: Invalid
Fix Version/s: Not applicable

> Matrix, Vector, Tensor Abstractions for Inputs, Outputs and Processing and 
> Construction of Such Structures from Scalars, Other Structures and Queries
> -
>
> Key: BEAM-264
> URL: https://issues.apache.org/jira/browse/BEAM-264
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> Not all timeseries data is scalar values hence support other types of values 
> like vectors, matrices. Also ability to construct these constructs. E.g. if I 
> want a time series vector of asset prices as input to a pipeline. Also I will 
> have to choose what stocks are in the vector and also how to handle if the 
> choosing is time varying.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-263) Generate Processing Infrastructure Code and Code for Runners in High Level Languages

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-263.
-
   Resolution: Invalid
Fix Version/s: Not applicable

> Generate Processing Infrastructure Code and Code for Runners in High Level 
> Languages
> 
>
> Key: BEAM-263
> URL: https://issues.apache.org/jira/browse/BEAM-263
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> In addition you can auto generate processing infrastructure code in languages 
> like C/C++, Java, Scala, C#, F#, Rust, Julia, Swift, D, Nim, etc.
> Auto generated code will be the runner itself and should not need any 
> additional infrastructure to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340615#comment-15340615
 ] 

Davor Bonaci commented on BEAM-262:
---

Closing / inactive.

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-262) Native Runners | Direct Compiler

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci closed BEAM-262.
-
   Resolution: Invalid
Fix Version/s: Not applicable

> Native Runners | Direct Compiler 
> -
>
> Key: BEAM-262
> URL: https://issues.apache.org/jira/browse/BEAM-262
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-ideas
>Reporter: Suminda Dharmasena
>Assignee: Davor Bonaci
> Fix For: Not applicable
>
>
> Having to depend on other frameworks to do the heavy lifting means that the 
> quakes, limitation and overhead of the other platform limits what can be 
> achieved. Hence is it possible to have Beam directly generate code for LLVM, 
> JVM and .Net platforms without dependence on any other platform.
> Also perhaps there can be code generation than directly native code in high 
> level languages like C/C++, Java, C#, F#, Rust, Julia, D, Nim, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-280) TestPipeline should be constructible without a runner

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-280.

Resolution: Fixed

> TestPipeline should be constructible without a runner
> -
>
> Key: BEAM-280
> URL: https://issues.apache.org/jira/browse/BEAM-280
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Today, one cannot create a {{Pipeline}} without a runner, as the runner is 
> wired in to do transform expansions. However, we want to remove the 
> {{DirectPipelineRunner}} from the SDK, so a {{TestPipeline}} should default 
> to a no-op runner that performs no expansion, but crashes upon {{run()}}, or 
> some such, in order to execute tests that do not really require a runner.
> (As soon as possible, this expansion wiring will be removed, but if we keep 
> the {{Pipeline.run()}} convenience method, we may still need some optional 
> runner set up)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #504: [BEAM-363] Return false for non-AUTO compr...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/504

[BEAM-363] Return false for non-AUTO compression mode



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam apache-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #504


commit 278a76e096ced0ba51b5ed0ae8205bbd32491c77
Author: Dan Halperin 
Date:   2016-06-20T22:44:54Z

[BEAM-363] Return false for non-AUTO compression mode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #498: Python sdk merge from master

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/498


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-363) CompressedSource splits when it shouldn't

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci updated BEAM-363:
--
Assignee: Daniel Halperin  (was: Davor Bonaci)

> CompressedSource splits when it shouldn't
> -
>
> Key: BEAM-363
> URL: https://issues.apache.org/jira/browse/BEAM-363
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> CompressedSource is splitting when it should not. This can in some corner 
> cases lead to data loss during dynamic work rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-363) CompressedSource splits when it shouldn't

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-363:


 Summary: CompressedSource splits when it shouldn't
 Key: BEAM-363
 URL: https://issues.apache.org/jira/browse/BEAM-363
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.1.0-incubating, 0.2.0-incubating
Reporter: Daniel Halperin
Assignee: Davor Bonaci


CompressedSource is splitting when it should not. This can in some corner cases 
lead to data loss during dynamic work rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[20/50] [abbrv] incubator-beam git commit: Rename InProcessPipelineRunner to DirectRunner

[davor]

Rename InProcessPipelineRunner to DirectRunner

Completes BEAM-243


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9400fc9a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9400fc9a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9400fc9a

Branch: refs/heads/python-sdk
Commit: 9400fc9a699f218a7948c21639428f5f00134ec5
Parents: d2ceaf5
Author: Thomas Groh 
Authored: Wed Jun 15 10:45:15 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 runners/direct-java/pom.xml |   2 +-
 .../direct/AbstractModelEnforcement.java|   2 +-
 .../direct/BoundedReadEvaluatorFactory.java |   4 +-
 .../beam/runners/direct/BundleFactory.java  |   4 +-
 .../beam/runners/direct/CommittedResult.java|   2 +-
 .../beam/runners/direct/CompletionCallback.java |   2 +-
 .../beam/runners/direct/DirectOptions.java  | 101 +
 .../beam/runners/direct/DirectRegistrar.java|  55 +++
 .../beam/runners/direct/DirectRunner.java   | 371 +++
 .../direct/EncodabilityEnforcementFactory.java  |   2 +-
 .../direct/ExecutorServiceParallelExecutor.java |   2 +-
 .../runners/direct/FlattenEvaluatorFactory.java |   6 +-
 .../ImmutabilityCheckingBundleFactory.java  |   4 +-
 .../direct/ImmutabilityEnforcementFactory.java  |   2 +-
 .../direct/InMemoryWatermarkManager.java|   2 +-
 .../runners/direct/InProcessBundleFactory.java  |   4 +-
 .../direct/InProcessBundleOutputManager.java|   6 +-
 .../direct/InProcessEvaluationContext.java  |  20 +-
 .../direct/InProcessExecutionContext.java   |   4 +-
 .../beam/runners/direct/InProcessExecutor.java  |   2 +-
 ...rocessGroupAlsoByWindowEvaluatorFactory.java |   4 +-
 ...InProcessGroupByKeyOnlyEvaluatorFactory.java |   6 +-
 .../direct/InProcessPipelineOptions.java| 101 -
 .../runners/direct/InProcessPipelineRunner.java | 370 --
 .../beam/runners/direct/InProcessRegistrar.java |  55 ---
 .../direct/InProcessTransformResult.java|   2 +-
 .../beam/runners/direct/ModelEnforcement.java   |   2 +-
 .../runners/direct/ModelEnforcementFactory.java |   2 +-
 .../runners/direct/ParDoInProcessEvaluator.java |   4 +-
 .../direct/ParDoMultiEvaluatorFactory.java  |   4 +-
 .../direct/ParDoSingleEvaluatorFactory.java |   4 +-
 .../direct/PassthroughTransformEvaluator.java   |   2 +-
 .../runners/direct/StepTransformResult.java |   2 +-
 .../beam/runners/direct/TransformEvaluator.java |   2 +-
 .../direct/TransformEvaluatorFactory.java   |   2 +-
 .../direct/TransformEvaluatorRegistry.java  |   2 +-
 .../beam/runners/direct/TransformExecutor.java  |   2 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |   4 +-
 .../runners/direct/ViewEvaluatorFactory.java|   6 +-
 .../runners/direct/WindowEvaluatorFactory.java  |   6 +-
 .../direct/AvroIOShardedWriteFactoryTest.java   |   2 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java |   4 +-
 .../runners/direct/CommittedResultTest.java |  12 +-
 .../runners/direct/DirectRegistrarTest.java |  74 
 .../beam/runners/direct/DirectRunnerTest.java   | 339 +
 .../EncodabilityEnforcementFactoryTest.java |   2 +-
 .../direct/FlattenEvaluatorFactoryTest.java |   4 +-
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |   4 +-
 .../ImmutabilityCheckingBundleFactoryTest.java  |   4 +-
 .../ImmutabilityEnforcementFactoryTest.java |   2 +-
 .../direct/InMemoryWatermarkManagerTest.java|   4 +-
 .../direct/InProcessBundleFactoryTest.java  |   4 +-
 .../direct/InProcessEvaluationContextTest.java  |  10 +-
 ...ocessGroupByKeyOnlyEvaluatorFactoryTest.java |   4 +-
 .../direct/InProcessPipelineRegistrarTest.java  |  74 
 .../direct/InProcessPipelineRunnerTest.java | 339 -
 .../direct/ParDoInProcessEvaluatorTest.java |   6 +-
 .../direct/ParDoMultiEvaluatorFactoryTest.java  |   4 +-
 .../direct/ParDoSingleEvaluatorFactoryTest.java |   4 +-
 .../direct/TextIOShardedWriteFactoryTest.java   |   2 +-
 .../runners/direct/TransformExecutorTest.java   |   2 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |   4 +-
 .../direct/ViewEvaluatorFactoryTest.java|   4 +-
 .../direct/WindowEvaluatorFactoryTest.java  |   4 +-
 .../translation/TransformTranslatorTest.java|   4 +-
 .../beam/sdk/options/PipelineOptions.java   |   2 +-
 testing/travis/test_wordcount.sh|   4 +-
 67 files changed, 1051 insertions(+), 1050 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml 

[10/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
new file mode 100644
index 000..7173129
--- /dev/null
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/WatermarkManagerTest.java
@@ -0,0 +1,1428 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct;
+
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.emptyIterable;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
+import org.apache.beam.runners.direct.WatermarkManager.FiredTimers;
+import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
+import 
org.apache.beam.runners.direct.WatermarkManager.TimerUpdate.TimerUpdateBuilder;
+import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Filter;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.WithKeys;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.apache.beam.sdk.util.TimerInternals.TimerData;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.state.StateNamespaces;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
+import org.apache.beam.sdk.values.PValue;
+import org.apache.beam.sdk.values.TimestampedValue;
+
+import com.google.common.collect.ImmutableList;
+
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.joda.time.Instant;
+import org.joda.time.ReadableInstant;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.annotation.Nullable;
+
+/**
+ * Tests for {@link WatermarkManager}.
+ */
+@RunWith(JUnit4.class)
+public class WatermarkManagerTest implements Serializable {
+  private transient MockClock clock;
+
+  private transient PCollection createdInts;
+
+  private transient PCollection filtered;
+  private transient PCollection filteredTimesTwo;
+  private transient PCollection> keyed;
+
+  private transient PCollection intsToFlatten;
+  private transient PCollection flattened;
+
+  private transient WatermarkManager manager;
+  private transient BundleFactory bundleFactory;
+
+  @Before
+  public void setup() {
+TestPipeline p = TestPipeline.create();
+
+createdInts = p.apply("createdInts", Create.of(1, 2, 3));
+
+filtered = createdInts.apply("filtered", Filter.greaterThan(1));
+filteredTimesTwo = filtered.apply("timesTwo", ParDo.of(new DoFn() {
+  @Override
+  public void processElement(DoFn.ProcessContext c) 
throws 

[15/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
index 4dd1475..ef31ba7 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
@@ -42,7 +42,7 @@ class TransformExecutor implements Runnable {
   public static  TransformExecutor create(
   TransformEvaluatorFactory factory,
   Iterable modelEnforcements,
-  InProcessEvaluationContext evaluationContext,
+  EvaluationContext evaluationContext,
   CommittedBundle inputBundle,
   AppliedPTransform transform,
   CompletionCallback completionCallback,
@@ -60,7 +60,7 @@ class TransformExecutor implements Runnable {
   private final TransformEvaluatorFactory evaluatorFactory;
   private final Iterable modelEnforcements;
 
-  private final InProcessEvaluationContext evaluationContext;
+  private final EvaluationContext evaluationContext;
 
   /** The transform that will be evaluated. */
   private final AppliedPTransform transform;
@@ -75,7 +75,7 @@ class TransformExecutor implements Runnable {
   private TransformExecutor(
   TransformEvaluatorFactory factory,
   Iterable modelEnforcements,
-  InProcessEvaluationContext evaluationContext,
+  EvaluationContext evaluationContext,
   CommittedBundle inputBundle,
   AppliedPTransform transform,
   CompletionCallback completionCallback,
@@ -117,7 +117,7 @@ class TransformExecutor implements Runnable {
 
   processElements(evaluator, enforcements);
 
-  InProcessTransformResult result = finishBundle(evaluator, enforcements);
+  TransformResult result = finishBundle(evaluator, enforcements);
 } catch (Throwable t) {
   onComplete.handleThrowable(inputBundle, t);
   if (t instanceof RuntimeException) {
@@ -155,13 +155,13 @@ class TransformExecutor implements Runnable {
* Finishes processing the input bundle and commit the result using the
* {@link CompletionCallback}, applying any {@link ModelEnforcement} if 
necessary.
*
-   * @return the {@link InProcessTransformResult} produced by
+   * @return the {@link TransformResult} produced by
* {@link TransformEvaluator#finishBundle()}
*/
-  private InProcessTransformResult finishBundle(
+  private TransformResult finishBundle(
   TransformEvaluator evaluator, Collection 
enforcements)
   throws Exception {
-InProcessTransformResult result = evaluator.finishBundle();
+TransformResult result = evaluator.finishBundle();
 CommittedResult outputs = onComplete.handleResult(inputBundle, result);
 for (ModelEnforcement enforcement : enforcements) {
   enforcement.afterFinish(inputBundle, result, outputs.getOutputs());

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformResult.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformResult.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformResult.java
new file mode 100644
index 000..c1e502d
--- /dev/null
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformResult.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct;
+
+import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
+import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
+import org.apache.beam.sdk.transforms.AppliedPTransform;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.WindowedValue;
+import 

[39/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
deleted file mode 100644
index 38d4c96..000
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerTest.java
+++ /dev/null
@@ -1,1417 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.containsString;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.hasItem;
-import static org.hamcrest.Matchers.instanceOf;
-import static org.hamcrest.Matchers.startsWith;
-import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Matchers.eq;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsList;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsMap;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsMultimap;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.BatchViewAsSingleton;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner.TransformedMap;
-import org.apache.beam.runners.dataflow.internal.IsmFormat;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.Pipeline.PipelineVisitor;
-import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
-import org.apache.beam.sdk.coders.BigEndianLongCoder;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.io.AvroIO;
-import org.apache.beam.sdk.io.AvroSource;
-import org.apache.beam.sdk.io.BigQueryIO;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.runners.TransformTreeNode;
-import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.DoFnTester;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.util.CoderUtils;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.NoopPathValidator;
-import org.apache.beam.sdk.util.ReleaseInfo;
-import org.apache.beam.sdk.util.TestCredential;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
-import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.gcsfs.GcsPath;
-import org.apache.beam.sdk.values.KV;
-import 

[41/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
new file mode 100644
index 000..91e34ac
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -0,0 +1,3229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
+import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
+import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification;
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TransformTranslator;
+import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TranslationContext;
+import org.apache.beam.runners.dataflow.internal.AssignWindows;
+import org.apache.beam.runners.dataflow.internal.DataflowAggregatorTransforms;
+import org.apache.beam.runners.dataflow.internal.IsmFormat;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
+import org.apache.beam.runners.dataflow.internal.ReadTranslator;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
+import org.apache.beam.runners.dataflow.util.DataflowTransport;
+import org.apache.beam.runners.dataflow.util.MonitoringUtil;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
+import org.apache.beam.sdk.PipelineResult.State;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.BigEndianLongCoder;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.ListCoder;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.coders.StandardCoder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.io.BigQueryIO;
+import org.apache.beam.sdk.io.FileBasedSink;
+import org.apache.beam.sdk.io.PubsubIO;
+import org.apache.beam.sdk.io.PubsubUnboundedSink;
+import org.apache.beam.sdk.io.PubsubUnboundedSource;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.ShardNameTemplate;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.io.Write;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsValidator;
+import org.apache.beam.sdk.options.StreamingOptions;
+import org.apache.beam.sdk.runners.AggregatorPipelineExtractor;
+import org.apache.beam.sdk.runners.PipelineRunner;
+import org.apache.beam.sdk.runners.TransformTreeNode;
+import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.Combine.CombineFn;
+import org.apache.beam.sdk.transforms.Create;

[44/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

Rename DataflowPipelineRunner to DataflowRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6d028ac6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6d028ac6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6d028ac6

Branch: refs/heads/python-sdk
Commit: 6d028ac6118e31ee2415f77cf744a99c59b31d41
Parents: 340d098
Author: Thomas Groh 
Authored: Fri Jun 17 11:42:19 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 .../beam/examples/DebuggingWordCount.java   |2 +-
 .../apache/beam/examples/MinimalWordCount.java  |4 +-
 .../apache/beam/examples/WindowedWordCount.java |2 +-
 .../org/apache/beam/examples/WordCount.java |2 +-
 .../examples/common/DataflowExampleUtils.java   |   18 +-
 .../beam/examples/complete/AutoComplete.java|4 +-
 .../apache/beam/examples/complete/TfIdf.java|2 +-
 .../examples/complete/TopWikipediaSessions.java |2 +-
 .../examples/cookbook/BigQueryTornadoes.java|2 +-
 .../cookbook/CombinePerKeyExamples.java |2 +-
 .../examples/cookbook/DatastoreWordCount.java   |4 +-
 .../beam/examples/cookbook/DeDupExample.java|2 +-
 .../beam/examples/cookbook/FilterExamples.java  |2 +-
 .../beam/examples/cookbook/JoinExamples.java|2 +-
 .../examples/cookbook/MaxPerKeyExamples.java|2 +-
 .../beam/examples/MinimalWordCountJava8.java|4 +-
 .../beam/examples/complete/game/GameStats.java  |2 +-
 .../examples/complete/game/HourlyTeamScore.java |2 +-
 .../examples/complete/game/LeaderBoard.java |2 +-
 .../beam/examples/complete/game/UserScore.java  |2 +-
 .../beam/runners/flink/FlinkPipelineRunner.java |2 +-
 .../BlockingDataflowPipelineRunner.java |  186 -
 .../dataflow/BlockingDataflowRunner.java|  186 +
 .../dataflow/DataflowJobCancelledException.java |2 +-
 .../dataflow/DataflowJobExecutionException.java |2 +-
 .../dataflow/DataflowJobUpdatedException.java   |2 +-
 .../runners/dataflow/DataflowPipelineJob.java   |2 +-
 .../dataflow/DataflowPipelineRegistrar.java |8 +-
 .../dataflow/DataflowPipelineRunner.java| 3229 --
 .../dataflow/DataflowPipelineRunnerHooks.java   |   39 -
 .../dataflow/DataflowPipelineTranslator.java|   10 +-
 .../beam/runners/dataflow/DataflowRunner.java   | 3229 ++
 .../runners/dataflow/DataflowRunnerHooks.java   |   39 +
 .../runners/dataflow/internal/package-info.java |2 +-
 .../BlockingDataflowPipelineOptions.java|6 +-
 .../options/DataflowPipelineOptions.java|4 +-
 .../DataflowPipelineWorkerPoolOptions.java  |6 +-
 .../testing/TestDataflowPipelineRunner.java |   10 +-
 .../BlockingDataflowPipelineRunnerTest.java |  304 --
 .../dataflow/BlockingDataflowRunnerTest.java|  304 ++
 .../dataflow/DataflowPipelineRegistrarTest.java |4 +-
 .../dataflow/DataflowPipelineRunnerTest.java| 1417 
 .../DataflowPipelineTranslatorTest.java |   50 +-
 .../runners/dataflow/DataflowRunnerTest.java| 1417 
 .../runners/dataflow/io/DataflowAvroIOTest.java |4 +-
 .../dataflow/io/DataflowPubsubIOTest.java   |4 +-
 .../runners/dataflow/io/DataflowTextIOTest.java |4 +-
 .../testing/TestDataflowPipelineRunnerTest.java |  601 
 .../testing/TestDataflowRunnerTest.java |  601 
 .../DataflowDisplayDataEvaluator.java   |   10 +-
 .../transforms/DataflowGroupByKeyTest.java  |8 +-
 .../dataflow/transforms/DataflowViewTest.java   |8 +-
 .../util/DataflowPathValidatorTest.java |8 +-
 .../beam/runners/spark/SimpleWordCountTest.java |2 +-
 .../spark/translation/SerializationTest.java|2 +-
 .../apache/beam/sdk/runners/package-info.java   |   12 +-
 .../apache/beam/sdk/testing/TestPipeline.java   |2 +-
 .../apache/beam/sdk/transforms/Aggregator.java  |2 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java|2 +-
 .../src/main/java/DebuggingWordCount.java   |2 +-
 .../src/main/java/MinimalWordCount.java |4 +-
 .../src/main/java/WindowedWordCount.java|2 +-
 .../src/main/java/WordCount.java|2 +-
 .../main/java/common/DataflowExampleUtils.java  |   14 +-
 .../src/main/java/StarterPipeline.java  |2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |2 +-
 sdks/java/pom.xml   |2 +-
 68 files changed, 5913 insertions(+), 5913 deletions(-)
--

[02/50] [abbrv] incubator-beam git commit: Remove the DirectPipelineRunner from the Core SDK

[davor]

Remove the DirectPipelineRunner from the Core SDK


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/99654ca4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/99654ca4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/99654ca4

Branch: refs/heads/python-sdk
Commit: 99654ca4bed6758d7128d0f0ad376e8b479d4eba
Parents: 45e57e0
Author: Thomas Groh 
Authored: Tue Jun 14 17:52:49 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../examples/common/DataflowExampleUtils.java   |   11 +-
 runners/spark/pom.xml   |6 +
 .../translation/TransformTranslatorTest.java|4 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   |1 -
 .../main/java/org/apache/beam/sdk/io/Read.java  |   44 -
 .../java/org/apache/beam/sdk/io/TextIO.java |1 -
 .../beam/sdk/options/DirectPipelineOptions.java |1 -
 .../sdk/runners/DirectPipelineRegistrar.java|   55 -
 .../beam/sdk/runners/DirectPipelineRunner.java  | 1298 --
 .../org/apache/beam/sdk/transforms/Flatten.java |   32 -
 .../org/apache/beam/sdk/transforms/ParDo.java   |  302 +---
 .../org/apache/beam/sdk/transforms/View.java|   24 -
 .../sdk/util/DirectModeExecutionContext.java|  130 --
 .../apache/beam/sdk/util/DoFnRunnerBase.java|1 -
 .../java/org/apache/beam/sdk/PipelineTest.java  |4 +-
 .../io/BoundedReadFromUnboundedSourceTest.java  |6 -
 .../runners/DirectPipelineRegistrarTest.java|   71 -
 .../sdk/runners/DirectPipelineRunnerTest.java   |  222 ---
 .../beam/sdk/runners/PipelineRunnerTest.java|9 +-
 .../apache/beam/sdk/transforms/CombineTest.java |   21 -
 .../beam/sdk/transforms/GroupByKeyTest.java |   13 +-
 .../apache/beam/sdk/transforms/ViewTest.java|   29 +-
 .../main/java/common/DataflowExampleUtils.java  |   13 +-
 testing/travis/test_wordcount.sh|4 +-
 24 files changed, 40 insertions(+), 2262 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99654ca4/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
index fb4f3bf..a0b7319 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.examples.common;
 
+import org.apache.beam.runners.dataflow.BlockingDataflowPipelineRunner;
 import org.apache.beam.runners.dataflow.DataflowPipelineJob;
 import org.apache.beam.runners.dataflow.DataflowPipelineRunner;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
@@ -25,7 +26,7 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.options.BigQueryOptions;
-import org.apache.beam.sdk.runners.DirectPipelineRunner;
+import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.IntraBundleParallelization;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.AttemptBoundedExponentialBackOff;
@@ -315,11 +316,13 @@ public class DataflowExampleUtils {
 
   /**
* Do some runner setup: check that the DirectPipelineRunner is not used in 
conjunction with
-   * streaming, and if streaming is specified, use the DataflowPipelineRunner. 
Return the streaming
-   * flag value.
+   * streaming, and if streaming is specified, use the DataflowPipelineRunner.
*/
   public void setupRunner() {
-if (options.isStreaming() && options.getRunner() != 
DirectPipelineRunner.class) {
+Class> runner = options.getRunner();
+if (options.isStreaming()
+&& (runner.equals(DataflowPipelineRunner.class)
+|| runner.equals(BlockingDataflowPipelineRunner.class))) {
   // In order to cancel the pipelines automatically,
   // {@literal DataflowPipelineRunner} is forced to be used.
   options.setRunner(DataflowPipelineRunner.class);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99654ca4/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index 4110689..e7d0834 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -118,6 +118,12 @@
   hamcrest-all
   test
 
+
+  org.apache.beam
+  beam-runners-direct-java
+  

[17/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InMemoryWatermarkManager.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InMemoryWatermarkManager.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InMemoryWatermarkManager.java
deleted file mode 100644
index fb8eb7c..000
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InMemoryWatermarkManager.java
+++ /dev/null
@@ -1,1420 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct;
-
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.transforms.AppliedPTransform;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.TimeDomain;
-import org.apache.beam.sdk.util.TimerInternals;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PValue;
-
-import com.google.auto.value.AutoValue;
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.Function;
-import com.google.common.base.MoreObjects;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.ComparisonChain;
-import com.google.common.collect.FluentIterable;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Ordering;
-import com.google.common.collect.SortedMultiset;
-import com.google.common.collect.TreeMultiset;
-
-import org.joda.time.Instant;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.EnumMap;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.NavigableSet;
-import java.util.Objects;
-import java.util.PriorityQueue;
-import java.util.Set;
-import java.util.TreeSet;
-import java.util.concurrent.ConcurrentLinkedQueue;
-import java.util.concurrent.atomic.AtomicReference;
-
-import javax.annotation.Nullable;
-
-/**
- * Manages watermarks of {@link PCollection PCollections} and input and output 
watermarks of
- * {@link AppliedPTransform AppliedPTransforms} to provide event-time and 
completion tracking for
- * in-memory execution. {@link InMemoryWatermarkManager} is designed to update 
and return a
- * consistent view of watermarks in the presence of concurrent updates.
- *
- * An {@link InMemoryWatermarkManager} is provided with the collection of 
root
- * {@link AppliedPTransform AppliedPTransforms} and a map of {@link 
PCollection PCollections} to
- * all the {@link AppliedPTransform AppliedPTransforms} that consume them at 
construction time.
- *
- * Whenever a root {@link AppliedPTransform transform} produces elements, 
the
- * {@link InMemoryWatermarkManager} is provided with the produced elements and 
the output watermark
- * of the producing {@link AppliedPTransform transform}. The
- * {@link InMemoryWatermarkManager watermark manager} is responsible for 
computing the watermarks
- * of all {@link AppliedPTransform transforms} that consume one or more
- * {@link PCollection PCollections}.
- *
- * Whenever a non-root {@link AppliedPTransform} finishes processing one or 
more in-flight
- * elements (referred to as the input {@link CommittedBundle bundle}), the 
following occurs
- * atomically:
- * 
- *  All of the in-flight elements are removed from the collection of 
pending elements for the
- *  {@link AppliedPTransform}.
- *  All of the elements produced by the {@link AppliedPTransform} are 
added to the collection
- *  of pending elements for each {@link AppliedPTransform} that consumes 
them.
- *  The input watermark for the {@link AppliedPTransform} becomes the 
maximum value of
- *
- *  the previous input watermark
- *  the 

[29/50] [abbrv] incubator-beam git commit: Remove the beam.examples dependency from flink.

[davor]

Remove the beam.examples dependency from flink.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e5812440
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e5812440
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e5812440

Branch: refs/heads/python-sdk
Commit: e5812440ef985a44316e0dde7c5fa19d38f91aa0
Parents: 93f9ef9
Author: Pei He 
Authored: Thu Jun 16 11:38:51 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 runners/flink/runner/pom.xml | 12 
 1 file changed, 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e5812440/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 2fdf61d..8933457 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -106,18 +106,6 @@
 
 
 
-  org.apache.beam
-  beam-examples-java
-  
-
-  org.slf4j
-  slf4j-jdk14
-
-  
-  test
-
-
-
   org.apache.flink
   flink-streaming-java_2.10
   ${flink.version}

[03/50] [abbrv] incubator-beam git commit: Update DataflowPipelineRunner worker container version

[davor]

Update DataflowPipelineRunner worker container version


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d2ceaf5e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d2ceaf5e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d2ceaf5e

Branch: refs/heads/python-sdk
Commit: d2ceaf5e5a778fad18472ab0d7c02a14259015d7
Parents: 99654ca
Author: Scott Wegner 
Authored: Tue Jun 14 09:00:49 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../org/apache/beam/runners/dataflow/DataflowPipelineRunner.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d2ceaf5e/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
index 2d34f23..1eb39ad 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
@@ -220,9 +220,9 @@ public class DataflowPipelineRunner extends 
PipelineRunner
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160427";
+  = "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160613";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160427";
+  = "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160613";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[21/50] [abbrv] incubator-beam git commit: Replace GcsPath by IOChannelFactory in WordCount.

[davor]

Replace GcsPath by IOChannelFactory in WordCount.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/60583307
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/60583307
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/60583307

Branch: refs/heads/python-sdk
Commit: 605833071a7034aa3b723776a0f9e24330f64c8b
Parents: e90a1b9
Author: Pei He 
Authored: Mon Jun 13 16:58:01 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../java/org/apache/beam/examples/WordCount.java | 19 +++
 1 file changed, 15 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/60583307/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java 
b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
index f5611d3..03ed5d7 100644
--- a/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
+++ b/examples/java/src/main/java/org/apache/beam/examples/WordCount.java
@@ -32,10 +32,15 @@ import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.transforms.Sum;
-import org.apache.beam.sdk.util.gcsfs.GcsPath;
+import org.apache.beam.sdk.util.IOChannelFactory;
+import org.apache.beam.sdk.util.IOChannelUtils;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
+import com.google.common.base.Strings;
+
+import java.io.IOException;
+
 /**
  * An example that counts words in Shakespeare and includes Beam best 
practices.
  *
@@ -176,9 +181,15 @@ public class WordCount {
 public static class OutputFactory implements DefaultValueFactory {
   @Override
   public String create(PipelineOptions options) {
-if (options.getTempLocation() != null) {
-  return GcsPath.fromUri(options.getTempLocation())
-  .resolve("counts.txt").toString();
+String tempLocation = options.getTempLocation();
+if (!Strings.isNullOrEmpty(tempLocation)) {
+  try {
+IOChannelFactory factory = IOChannelUtils.getFactory(tempLocation);
+return factory.resolve(tempLocation, "counts.txt");
+  } catch (IOException e) {
+throw new RuntimeException(
+String.format("Failed to resolve temp location: %s", 
tempLocation));
+  }
 } else {
   throw new IllegalArgumentException("Must specify --output or 
--tempLocation");
 }

[36/50] [abbrv] incubator-beam git commit: Turn on failOnWarning and ignore unused runners modules in example.

[davor]

Turn on failOnWarning and ignore unused runners modules in example.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c9550101
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c9550101
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c9550101

Branch: refs/heads/python-sdk
Commit: c9550101373c399c15c722e995bdbaf429eabf30
Parents: 6d028ac
Author: Pei He 
Authored: Thu Jun 16 15:50:21 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 examples/java/pom.xml | 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c9550101/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 9458962..5167810 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -59,6 +59,14 @@
   
 org.apache.maven.plugins
 maven-dependency-plugin
+
+  
+analyze-only
+
+  true
+
+  
+
   
 
   
@@ -216,6 +224,7 @@
   org.apache.beam
   beam-runners-direct-java
   ${project.version}
+  runtime
 
 
 

[37/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
deleted file mode 100644
index fbaf116..000
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunnerTest.java
+++ /dev/null
@@ -1,601 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow.testing;
-
-import static org.hamcrest.Matchers.containsString;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertSame;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Mockito.atLeastOnce;
-import static org.mockito.Mockito.doCallRealMethod;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.runners.dataflow.DataflowPipelineJob;
-import org.apache.beam.runners.dataflow.DataflowPipelineRunner;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil.JobMessagesHandler;
-import org.apache.beam.runners.dataflow.util.TimeUtil;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.testing.PAssert;
-import org.apache.beam.sdk.testing.SerializableMatcher;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.testing.TestPipelineOptions;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.NoopPathValidator;
-import org.apache.beam.sdk.util.TestCredential;
-import org.apache.beam.sdk.util.Transport;
-import org.apache.beam.sdk.values.PCollection;
-
-import com.google.api.client.http.LowLevelHttpResponse;
-import com.google.api.client.json.Json;
-import com.google.api.client.testing.http.MockHttpTransport;
-import com.google.api.client.testing.http.MockLowLevelHttpRequest;
-import com.google.api.client.testing.http.MockLowLevelHttpResponse;
-import com.google.api.services.dataflow.Dataflow;
-import com.google.api.services.dataflow.model.JobMessage;
-import com.google.api.services.dataflow.model.JobMetrics;
-import com.google.api.services.dataflow.model.MetricStructuredName;
-import com.google.api.services.dataflow.model.MetricUpdate;
-import com.google.common.base.Optional;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Lists;
-
-import org.hamcrest.BaseMatcher;
-import org.hamcrest.Description;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.Mockito;
-import org.mockito.MockitoAnnotations;
-import org.mockito.invocation.InvocationOnMock;
-import org.mockito.stubbing.Answer;
-
-import java.io.IOException;
-import java.math.BigDecimal;
-import java.util.Arrays;
-import java.util.concurrent.TimeUnit;
-
-/** Tests for {@link TestDataflowPipelineRunner}. */
-@RunWith(JUnit4.class)
-public class TestDataflowPipelineRunnerTest {
-  @Rule public ExpectedException expectedException = ExpectedException.none();
-  @Mock private MockHttpTransport transport;
-  @Mock private MockLowLevelHttpRequest request;
-  @Mock private GcsUtil mockGcsUtil;
-
-  private TestDataflowPipelineOptions options;
-  private Dataflow service;
-
-  @Before
-  public void setUp() 

[45/50] [abbrv] incubator-beam git commit: DataflowPipelineJob: Retry messages, metrics, and status polls

[davor]

DataflowPipelineJob: Retry messages, metrics, and status polls

At some point in the past, we decided to use a rawDataflowClient that
does not do retries when checking job status, because it was best-effort
reporting to users. The purported goal was to not clutter the log with
networking errors.

However, since that time, we have:
* Added the ability to suppress logs (emit only at DEBUG level or not at
  all) when retrying.
* Increased reliability of the job checking status so that these errors
  are less frequent and more indicative of quota or other issues.
* Started using the metrics in tests, where we do need to retry
  transient issues (BEAM-350).

So let's drop the raw transport client and just use the one that
retries.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d285e675
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d285e675
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d285e675

Branch: refs/heads/python-sdk
Commit: d285e675920cd790c68053291c9bf843c21fc493
Parents: 30d226a
Author: Dan Halperin 
Authored: Thu Jun 16 08:57:18 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 .../org/apache/beam/runners/dataflow/DataflowRunner.java |  4 ++--
 .../beam/runners/dataflow/util/DataflowTransport.java| 11 ---
 2 files changed, 2 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d285e675/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 91e34ac..5818ba5 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -633,12 +633,12 @@ public class DataflowRunner extends 
PipelineRunner {
 // regularly and need not be retried automatically.
 DataflowPipelineJob dataflowPipelineJob =
 new DataflowPipelineJob(options.getProject(), jobResult.getId(),
-DataflowTransport.newRawDataflowClient(options).build(), 
aggregatorTransforms);
+options.getDataflowClient(), aggregatorTransforms);
 
 // If the service returned client request id, the SDK needs to compare it
 // with the original id generated in the request, if they are not the same
 // (i.e., the returned job is not created by this request), throw
-// DataflowJobAlreadyExistsException or DataflowJobAlreadyUpdatedExcetpion
+// DataflowJobAlreadyExistsException or DataflowJobAlreadyUpdatedException
 // depending on whether this is a reload or not.
 if (jobResult.getClientRequestId() != null && 
!jobResult.getClientRequestId().isEmpty()
 && !jobResult.getClientRequestId().equals(requestId)) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d285e675/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
index 09fca94..f988749 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/DataflowTransport.java
@@ -92,17 +92,6 @@ public class DataflowTransport {
 .setGoogleClientRequestInitializer(options.getGoogleApiTrace());
   }
 
-  /**
-   * Returns a Dataflow client that does not automatically retry failed
-   * requests.
-   */
-  public static Dataflow.Builder
-  newRawDataflowClient(DataflowPipelineOptions options) {
-return newDataflowClient(options)
-.setHttpRequestInitializer(options.getGcpCredential())
-.setGoogleClientRequestInitializer(options.getGoogleApiTrace());
-  }
-
   private static HttpRequestInitializer chainHttpRequestInitializer(
   Credential credential, HttpRequestInitializer httpRequestInitializer) {
 if (credential == null) {

[16/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessExecutor.java
deleted file mode 100644
index 1cfa544..000
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessExecutor.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct;
-
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.sdk.transforms.AppliedPTransform;
-import org.apache.beam.sdk.transforms.PTransform;
-
-import java.util.Collection;
-
-/**
- * An executor that schedules and executes {@link AppliedPTransform 
AppliedPTransforms} for both
- * source and intermediate {@link PTransform PTransforms}.
- */
-interface InProcessExecutor {
-  /**
-   * Starts this executor. The provided collection is the collection of root 
transforms to
-   * initially schedule.
-   *
-   * @param rootTransforms
-   */
-  void start(Collection rootTransforms);
-
-  /**
-   * Blocks until the job being executed enters a terminal state. A job is 
completed after all
-   * root {@link AppliedPTransform AppliedPTransforms} have completed, and all
-   * {@link CommittedBundle Bundles} have been consumed. Jobs may also 
terminate abnormally.
-   *
-   * @throws Throwable whenever an executor thread throws anything, transfers 
the throwable to the
-   *   waiting thread and rethrows it
-   */
-  void awaitCompletion() throws Throwable;
-}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessGroupAlsoByWindowEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessGroupAlsoByWindowEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessGroupAlsoByWindowEvaluatorFactory.java
deleted file mode 100644
index 53b93d0..000
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/InProcessGroupAlsoByWindowEvaluatorFactory.java
+++ /dev/null
@@ -1,127 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct;
-
-import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetDoFn;
-import 
org.apache.beam.runners.direct.InProcessGroupByKey.InProcessGroupAlsoByWindow;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.transforms.AppliedPTransform;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.GroupByKeyViaGroupByKeyOnly;
-import org.apache.beam.sdk.util.GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly;
-import org.apache.beam.sdk.util.KeyedWorkItem;
-import org.apache.beam.sdk.util.SystemReduceFn;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
-import 

[43/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
deleted file mode 100644
index 1eb39ad..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunner.java
+++ /dev/null
@@ -1,3229 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import static org.apache.beam.sdk.util.StringUtils.approximatePTransformName;
-import static org.apache.beam.sdk.util.StringUtils.approximateSimpleName;
-import static org.apache.beam.sdk.util.WindowedValue.valueInEmptyWindows;
-
-import static com.google.common.base.Preconditions.checkArgument;
-import static com.google.common.base.Preconditions.checkState;
-
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TransformTranslator;
-import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.TranslationContext;
-import org.apache.beam.runners.dataflow.internal.AssignWindows;
-import org.apache.beam.runners.dataflow.internal.DataflowAggregatorTransforms;
-import org.apache.beam.runners.dataflow.internal.IsmFormat;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
-import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
-import org.apache.beam.runners.dataflow.internal.ReadTranslator;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
-import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
-import 
org.apache.beam.runners.dataflow.options.DataflowPipelineWorkerPoolOptions;
-import org.apache.beam.runners.dataflow.util.DataflowTransport;
-import org.apache.beam.runners.dataflow.util.MonitoringUtil;
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.Pipeline.PipelineVisitor;
-import org.apache.beam.sdk.PipelineResult.State;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.coders.AvroCoder;
-import org.apache.beam.sdk.coders.BigEndianLongCoder;
-import org.apache.beam.sdk.coders.CannotProvideCoderException;
-import org.apache.beam.sdk.coders.Coder;
-import org.apache.beam.sdk.coders.Coder.NonDeterministicException;
-import org.apache.beam.sdk.coders.CoderException;
-import org.apache.beam.sdk.coders.CoderRegistry;
-import org.apache.beam.sdk.coders.IterableCoder;
-import org.apache.beam.sdk.coders.KvCoder;
-import org.apache.beam.sdk.coders.ListCoder;
-import org.apache.beam.sdk.coders.MapCoder;
-import org.apache.beam.sdk.coders.SerializableCoder;
-import org.apache.beam.sdk.coders.StandardCoder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.coders.VarLongCoder;
-import org.apache.beam.sdk.io.AvroIO;
-import org.apache.beam.sdk.io.BigQueryIO;
-import org.apache.beam.sdk.io.FileBasedSink;
-import org.apache.beam.sdk.io.PubsubIO;
-import org.apache.beam.sdk.io.PubsubUnboundedSink;
-import org.apache.beam.sdk.io.PubsubUnboundedSource;
-import org.apache.beam.sdk.io.Read;
-import org.apache.beam.sdk.io.ShardNameTemplate;
-import org.apache.beam.sdk.io.TextIO;
-import org.apache.beam.sdk.io.UnboundedSource;
-import org.apache.beam.sdk.io.Write;
-import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.options.PipelineOptionsValidator;
-import org.apache.beam.sdk.options.StreamingOptions;
-import org.apache.beam.sdk.runners.AggregatorPipelineExtractor;
-import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.runners.TransformTreeNode;
-import org.apache.beam.sdk.transforms.Aggregator;
-import org.apache.beam.sdk.transforms.Combine;
-import org.apache.beam.sdk.transforms.Combine.CombineFn;
-import 

[27/50] [abbrv] incubator-beam git commit: Add BigIntegerCoder and tests

[davor]

Add BigIntegerCoder and tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8268f1d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8268f1d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8268f1d7

Branch: refs/heads/python-sdk
Commit: 8268f1d7ffdd1205a1904037f7dd1e1887a52f8d
Parents: 6460df1
Author: Kenneth Knowles 
Authored: Thu Jun 9 13:24:49 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../apache/beam/sdk/coders/BigIntegerCoder.java |  91 ++
 .../beam/sdk/coders/BigIntegerCoderTest.java| 119 +++
 2 files changed, 210 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8268f1d7/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
new file mode 100644
index 000..a41defe
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigIntegerCoder.java
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.coders;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.math.BigInteger;
+
+/**
+ * A {@link BigIntegerCoder} encodes a {@link BigInteger} as a byte array 
containing the big endian
+ * two's-complement representation, encoded via {@link ByteArrayCoder}.
+ */
+public class BigIntegerCoder extends AtomicCoder {
+
+  @JsonCreator
+  public static BigIntegerCoder of() {
+return INSTANCE;
+  }
+
+  /
+
+  private static final BigIntegerCoder INSTANCE = new BigIntegerCoder();
+
+  private BigIntegerCoder() {}
+
+  private final ByteArrayCoder byteArrayCoder = ByteArrayCoder.of();
+
+  @Override
+  public void encode(BigInteger value, OutputStream outStream, Context context)
+  throws IOException, CoderException {
+checkNotNull(value, String.format("cannot encode a null %s", 
BigInteger.class.getSimpleName()));
+byteArrayCoder.encode(value.toByteArray(), outStream, context);
+  }
+
+  @Override
+  public BigInteger decode(InputStream inStream, Context context)
+  throws IOException, CoderException {
+return new BigInteger(byteArrayCoder.decode(inStream, context));
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return {@code true}. This coder is injective.
+   */
+  @Override
+  public boolean consistentWithEquals() {
+return true;
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return {@code true}, because {@link #getEncodedElementByteSize} runs in 
constant time.
+   */
+  @Override
+  public boolean isRegisterByteSizeObserverCheap(BigInteger value, Context 
context) {
+return true;
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return the size of the encoding as a byte array according to {@link 
ByteArrayCoder}
+   */
+  @Override
+  protected long getEncodedElementByteSize(BigInteger value, Context context) 
throws Exception {
+checkNotNull(value, String.format("cannot encode a null %s", 
BigInteger.class.getSimpleName()));
+return byteArrayCoder.getEncodedElementByteSize(value.toByteArray(), 
context);
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8268f1d7/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/BigIntegerCoderTest.java
new file mode 100644
index 000..3934b03
--- 

[06/50] [abbrv] incubator-beam git commit: Rename DoFnTester#processBatch to processBundle

[davor]

Rename DoFnTester#processBatch to processBundle

DoFns process elements in bundles, not batches.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0065851b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0065851b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0065851b

Branch: refs/heads/python-sdk
Commit: 0065851b96644f2c75b8e51c95ebf0e79c5865f5
Parents: ecf6ab8
Author: Thomas Groh 
Authored: Tue Jun 14 09:27:55 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../org/apache/beam/examples/WordCountTest.java |  6 +++---
 .../cookbook/BigQueryTornadoesTest.java |  8 
 .../cookbook/CombinePerKeyExamplesTest.java |  4 ++--
 .../examples/cookbook/FilterExamplesTest.java   |  4 ++--
 .../examples/cookbook/JoinExamplesTest.java |  4 ++--
 .../cookbook/MaxPerKeyExamplesTest.java |  4 ++--
 .../examples/cookbook/TriggerExampleTest.java   |  4 ++--
 .../examples/complete/game/UserScoreTest.java   |  2 +-
 .../dataflow/DataflowPipelineRunnerTest.java| 20 ++--
 .../apache/beam/sdk/transforms/DoFnTester.java  | 16 
 .../beam/sdk/transforms/DoFnTesterTest.java | 10 +-
 .../sdk/transforms/join/CoGroupByKeyTest.java   |  2 +-
 .../src/test/java/WordCountTest.java|  6 +++---
 13 files changed, 45 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0065851b/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java 
b/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
index 1382612..ff117dc 100644
--- a/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
+++ b/examples/java/src/test/java/org/apache/beam/examples/WordCountTest.java
@@ -52,11 +52,11 @@ public class WordCountTest {
 DoFnTester extractWordsFn =
 DoFnTester.of(new ExtractWordsFn());
 
-Assert.assertThat(extractWordsFn.processBatch(" some  input  words "),
+Assert.assertThat(extractWordsFn.processBundle(" some  input  words "),
   CoreMatchers.hasItems("some", "input", "words"));
-Assert.assertThat(extractWordsFn.processBatch(" "),
+Assert.assertThat(extractWordsFn.processBundle(" "),
   CoreMatchers.hasItems());
-Assert.assertThat(extractWordsFn.processBatch(" some ", " input", " 
words"),
+Assert.assertThat(extractWordsFn.processBundle(" some ", " input", " 
words"),
   CoreMatchers.hasItems("some", "input", "words"));
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0065851b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
index 36be568..b986c0b 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/BigQueryTornadoesTest.java
@@ -45,7 +45,7 @@ public class BigQueryTornadoesTest {
   .set("tornado", true);
 DoFnTester extractWordsFn =
 DoFnTester.of(new ExtractTornadoesFn());
-Assert.assertThat(extractWordsFn.processBatch(row),
+Assert.assertThat(extractWordsFn.processBundle(row),
   CoreMatchers.hasItems(6));
   }
 
@@ -56,7 +56,7 @@ public class BigQueryTornadoesTest {
   .set("tornado", false);
 DoFnTester extractWordsFn =
 DoFnTester.of(new ExtractTornadoesFn());
-Assert.assertTrue(extractWordsFn.processBatch(row).isEmpty());
+Assert.assertTrue(extractWordsFn.processBundle(row).isEmpty());
   }
 
   @Test
@@ -65,12 +65,12 @@ public class BigQueryTornadoesTest {
 DoFnTester, TableRow> formatCountsFn =
 DoFnTester.of(new FormatCountsFn());
 KV empty[] = {};
-List results = formatCountsFn.processBatch(empty);
+List results = formatCountsFn.processBundle(empty);
 Assert.assertTrue(results.size() == 0);
 KV input[] = { KV.of(3, 0L),
KV.of(4, Long.MAX_VALUE),
KV.of(5, Long.MIN_VALUE) };
-results = formatCountsFn.processBatch(input);
+results = formatCountsFn.processBundle(input);
 Assert.assertEquals(results.size(), 3);
 

[12/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessEvaluationContextTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessEvaluationContextTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessEvaluationContextTest.java
deleted file mode 100644
index b1cbeb1..000
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/InProcessEvaluationContextTest.java
+++ /dev/null
@@ -1,545 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.direct;
-
-import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.not;
-import static org.junit.Assert.assertThat;
-
-import org.apache.beam.runners.direct.InMemoryWatermarkManager.FiredTimers;
-import org.apache.beam.runners.direct.InMemoryWatermarkManager.TimerUpdate;
-import 
org.apache.beam.runners.direct.InProcessExecutionContext.InProcessStepContext;
-import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
-import org.apache.beam.runners.direct.DirectRunner.PCollectionViewWriter;
-import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
-import org.apache.beam.sdk.coders.ByteArrayCoder;
-import org.apache.beam.sdk.coders.StringUtf8Coder;
-import org.apache.beam.sdk.coders.VarIntCoder;
-import org.apache.beam.sdk.io.CountingInput;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.testing.TestPipeline;
-import org.apache.beam.sdk.transforms.AppliedPTransform;
-import org.apache.beam.sdk.transforms.Create;
-import org.apache.beam.sdk.transforms.View;
-import org.apache.beam.sdk.transforms.WithKeys;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo;
-import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
-import org.apache.beam.sdk.util.SideInputReader;
-import org.apache.beam.sdk.util.TimeDomain;
-import org.apache.beam.sdk.util.TimerInternals.TimerData;
-import org.apache.beam.sdk.util.WindowedValue;
-import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.common.Counter;
-import org.apache.beam.sdk.util.common.Counter.AggregationKind;
-import org.apache.beam.sdk.util.common.CounterSet;
-import org.apache.beam.sdk.util.state.BagState;
-import org.apache.beam.sdk.util.state.CopyOnAccessInMemoryStateInternals;
-import org.apache.beam.sdk.util.state.StateNamespaces;
-import org.apache.beam.sdk.util.state.StateTag;
-import org.apache.beam.sdk.util.state.StateTags;
-import org.apache.beam.sdk.values.KV;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PCollection.IsBounded;
-import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PValue;
-
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.Iterables;
-
-import org.hamcrest.Matchers;
-import org.joda.time.Instant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-import java.util.Collection;
-import java.util.Collections;
-import java.util.Map;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.TimeUnit;
-
-/**
- * Tests for {@link InProcessEvaluationContext}.
- */
-@RunWith(JUnit4.class)
-public class InProcessEvaluationContextTest {
-  private TestPipeline p;
-  private InProcessEvaluationContext context;
-
-  private PCollection created;
-  private PCollection> downstream;
-  private PCollectionView view;
-  private PCollection unbounded;
-  private Collection rootTransforms;
-  private Map> valueToConsumers;
-
-  private BundleFactory bundleFactory;
-
-  

[25/50] [abbrv] incubator-beam git commit: Roll-forwards: Base PAssert on GBK instead of side inputs

[davor]

Roll-forwards: Base PAssert on GBK instead of side inputs

Previously PAssert - hence all RunnableOnService/NeedsRunner
tests - required side input support. This created a very steep
on ramp for new runners.

GroupByKey is a bit more fundamental and most backends will be
able to group by key in the global window very quickly. So switching
the primitive used to gather all the contents of a PCollection for
assertions should make it a bit easier to get early feedback during
runner development.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e90a1b9d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e90a1b9d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e90a1b9d

Branch: refs/heads/python-sdk
Commit: e90a1b9d74cbc06d7818bae8dfe2af81acd73222
Parents: 8278e5f
Author: Kenneth Knowles 
Authored: Wed Jun 8 15:07:52 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../testing/TestDataflowPipelineRunner.java |   3 +-
 .../org/apache/beam/sdk/testing/PAssert.java| 857 +++
 .../apache/beam/sdk/testing/PAssertTest.java|  27 -
 3 files changed, 491 insertions(+), 396 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e90a1b9d/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
index 3e8d903..c940e9a 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
@@ -166,7 +166,8 @@ public class TestDataflowPipelineRunner extends 
PipelineRunner OutputT apply(
   PTransform transform, InputT input) {
 if (transform instanceof PAssert.OneSideInputAssert
-|| transform instanceof PAssert.TwoSideInputAssert) {
+|| transform instanceof PAssert.GroupThenAssert
+|| transform instanceof PAssert.GroupThenAssertForSingleton) {
   expectedNumberOfAssertions += 1;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e90a1b9d/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index c2cd598..a29a56d 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.testing;
 
+import static com.google.common.base.Preconditions.checkState;
+
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.not;
@@ -34,46 +36,46 @@ import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Flatten;
+import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.transforms.View;
+import org.apache.beam.sdk.transforms.WithKeys;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
+import org.apache.beam.sdk.transforms.windowing.Never;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionList;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PDone;
 
-import com.google.common.base.Optional;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import java.io.IOException;
 import java.io.Serializable;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
-import 

[49/50] [abbrv] incubator-beam git commit: Remove Pipeline from TestDataflowPipelineRunner

[davor]

Remove Pipeline from TestDataflowPipelineRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a24e5570
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a24e5570
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a24e5570

Branch: refs/heads/python-sdk
Commit: a24e557090c7fb59846b332c0c9df9d49565c808
Parents: 90d0bcf
Author: Thomas Groh 
Authored: Fri Jun 17 16:36:22 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 examples/java/README.md |   4 +-
 .../beam/runners/flink/examples/TFIDF.java  |   2 +-
 .../testing/TestDataflowPipelineRunner.java | 271 ---
 .../dataflow/testing/TestDataflowRunner.java| 271 +++
 .../testing/TestDataflowRunnerTest.java |  40 +--
 5 files changed, 294 insertions(+), 294 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a24e5570/examples/java/README.md
--
diff --git a/examples/java/README.md b/examples/java/README.md
index ef3cf07..2b5edf5 100644
--- a/examples/java/README.md
+++ b/examples/java/README.md
@@ -64,7 +64,7 @@ the same pipeline on fully managed resources in Google Cloud 
Platform:
 -Dexec.mainClass=org.apache.beam.examples.WordCount \
 -Dexec.args="--project= \
 --tempLocation= \
---runner=BlockingDataflowPipelineRunner"
+--runner=BlockingDataflowRunner"
 
 Make sure to use your project id, not the project number or the descriptive 
name.
 The Cloud Storage location should be entered in the form of
@@ -86,7 +86,7 @@ Platform:
 org.apache.beam.examples.WordCount \
 --project= \
 --tempLocation= \
---runner=BlockingDataflowPipelineRunner
+--runner=BlockingDataflowRunner
 
 Other examples can be run similarly by replacing the `WordCount` class path 
with the example classpath, e.g.
 `org.apache.beam.examples.cookbook.BigQueryTornadoes`,

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a24e5570/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
index 0afde0a..876ecde 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
@@ -82,7 +82,7 @@ import java.util.Set;
  * {@code
  *   --project=YOUR_PROJECT_ID
  *   --stagingLocation=gs://YOUR_STAGING_DIRECTORY
- *   --runner=BlockingDataflowPipelineRunner
+ *   --runner=BlockingDataflowRunner
  * and an output prefix on GCS:
  *   --output=gs://YOUR_OUTPUT_PREFIX
  * }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a24e5570/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
deleted file mode 100644
index f83a139..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/testing/TestDataflowPipelineRunner.java
+++ /dev/null
@@ -1,271 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow.testing;
-
-import static org.hamcrest.MatcherAssert.assertThat;
-
-import org.apache.beam.runners.dataflow.DataflowJobExecutionException;
-import org.apache.beam.runners.dataflow.DataflowPipelineJob;
-import 

[07/50] [abbrv] incubator-beam git commit: Package javadoc for org.apache.beam.sdk.transforms.display

[davor]

Package javadoc for org.apache.beam.sdk.transforms.display


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/340fe3eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/340fe3eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/340fe3eb

Branch: refs/heads/python-sdk
Commit: 340fe3ebcfef0b57b163483d7d7243ad5456ae72
Parents: 90bb20e
Author: Scott Wegner 
Authored: Wed Jun 15 10:17:01 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../sdk/transforms/display/package-info.java| 26 
 1 file changed, 26 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/340fe3eb/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/package-info.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/package-info.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/package-info.java
new file mode 100644
index 000..4af3327
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/display/package-info.java
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Defines {@link org.apache.beam.sdk.transforms.display.HasDisplayData} for 
annotating components
+ * which provide {@link org.apache.beam.sdk.transforms.display.DisplayData 
display data} used
+ * within UIs and diagnostic tools.
+ *
+ * @see org.apache.beam.sdk.transforms.display.HasDisplayData
+ */
+package org.apache.beam.sdk.transforms.display;

[22/50] [abbrv] incubator-beam git commit: Fix type error in Eclipse

[davor]

Fix type error in Eclipse

This type error occurs in my Eclipse installation. It apparently
does not bother the various JDKs we test with. But this is an
accurate typing, so it may help other Eclipse-using contributors,
too.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cfa217a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cfa217a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cfa217a8

Branch: refs/heads/python-sdk
Commit: cfa217a894575f392f1dfe1612e10e393df5c7ab
Parents: babddbb
Author: Kenneth Knowles 
Authored: Tue Jun 14 16:12:11 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../beam/runners/direct/ExecutorServiceParallelExecutor.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfa217a8/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 4bb5021..78f3fe4 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -415,7 +415,7 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
   if (delivery.isEmpty()) {
 continue;
   }
-  KeyedWorkItem work =
+  KeyedWorkItem work =
   KeyedWorkItems.timersWorkItem(keyTimers.getKey().getKey(), 
delivery);
   @SuppressWarnings({"unchecked", "rawtypes"})
   CommittedBundle bundle =

[46/50] [abbrv] incubator-beam git commit: [BEAM-321] Fix Flink Comparators

[davor]

[BEAM-321] Fix Flink Comparators

KvCoderComparator and CoderComparator were hashing the key directly
while doing comparisons on the encoded form. This lead to
inconsistencies in GroupByKey results with large numbers of elements per
key.

This changes the comparators to hash on the encoded form and also adds
tests to verify the correct behavior.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90d0bcfa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90d0bcfa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90d0bcfa

Branch: refs/heads/python-sdk
Commit: 90d0bcfa74a0e99acb6721cc9c7623cf55e6626b
Parents: c955010
Author: Aljoscha Krettek 
Authored: Wed Jun 1 11:56:18 2016 +0200
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 runners/flink/runner/pom.xml|  10 +
 .../beam/runners/flink/FlinkRunnerResult.java   |   8 +
 .../FlinkBatchTransformTranslators.java |  49 +---
 .../FlinkBatchTranslationContext.java   |  13 +-
 .../translation/types/CoderComparator.java  | 217 
 .../translation/types/CoderTypeInformation.java |  14 +-
 .../translation/types/CoderTypeSerializer.java  |  37 +--
 .../types/EncodedValueComparator.java   | 197 ++
 .../types/EncodedValueSerializer.java   | 113 
 .../types/EncodedValueTypeInformation.java  | 111 
 .../translation/types/KvCoderComperator.java| 259 ---
 .../types/KvCoderTypeInformation.java   | 207 ---
 .../flink/translation/types/KvKeySelector.java  |  51 
 .../flink/EncodedValueComparatorTest.java   |  71 +
 .../beam/sdk/transforms/GroupByKeyTest.java | 206 ++-
 15 files changed, 799 insertions(+), 764 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90d0bcfa/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 8933457..33c13bf 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -50,6 +50,16 @@
   ${flink.version}
 
 
+
+
+  org.apache.flink
+  flink-core
+  ${flink.version}
+  test-jar
+  test
+
+
+
 
 
   org.apache.beam

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90d0bcfa/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
index 481d867..a8f4cac 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunnerResult.java
@@ -65,4 +65,12 @@ public class FlinkRunnerResult implements PipelineResult {
   new RuntimeException("Accumulator does not exist."));
 }
   }
+
+  @Override
+  public String toString() {
+return "FlinkRunnerResult{" +
+"aggregators=" + aggregators +
+", runtime=" + runtime +
+'}';
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90d0bcfa/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
index 200e4af..ac058b2 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkBatchTransformTranslators.java
@@ -27,7 +27,7 @@ import 
org.apache.beam.runners.flink.translation.functions.FlinkMultiOutputPruni
 import 
org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction;
 import org.apache.beam.runners.flink.translation.functions.FlinkReduceFunction;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
-import org.apache.beam.runners.flink.translation.types.KvCoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.KvKeySelector;
 import org.apache.beam.runners.flink.translation.wrappers.SourceInputFormat;
 import 

[34/50] [abbrv] incubator-beam git commit: Added BigDecimal coder and tests.

[davor]

Added BigDecimal coder and tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6460df19
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6460df19
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6460df19

Branch: refs/heads/python-sdk
Commit: 6460df195240dac4d488fcf111642e8706008690
Parents: 4f7a2ab
Author: Jesse Anderson 
Authored: Mon May 9 10:05:15 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../apache/beam/sdk/coders/BigDecimalCoder.java | 112 +++
 .../beam/sdk/coders/BigDecimalCoderTest.java| 105 +
 2 files changed, 217 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6460df19/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
new file mode 100644
index 000..3b723b9
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.coders;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+
+/**
+ * A {@link BigDecimalCoder} encodes {@link BigDecimal} in an integer and
+ * a byte array. The integer represents the scale and the byte array
+ * represents a {@link BigInteger}. The integer is in 4 bytes, big-endian.
+ */
+public class BigDecimalCoder extends AtomicCoder {
+
+  @JsonCreator
+  public static BigDecimalCoder of() {
+return INSTANCE;
+  }
+
+  /
+
+  private static final BigDecimalCoder INSTANCE = new BigDecimalCoder();
+
+  private BigDecimalCoder() {}
+
+  @Override
+  public void encode(BigDecimal value, OutputStream outStream, Context context)
+  throws IOException, CoderException {
+if (value == null) {
+  throw new CoderException("cannot encode a null BigDecimal");
+}
+
+byte[] bigIntBytes = value.unscaledValue().toByteArray();
+
+DataOutputStream dataOutputStream = new DataOutputStream(outStream);
+dataOutputStream.writeInt(value.scale());
+dataOutputStream.writeInt(bigIntBytes.length);
+dataOutputStream.write(bigIntBytes);
+  }
+
+  @Override
+  public BigDecimal decode(InputStream inStream, Context context)
+  throws IOException, CoderException {
+DataInputStream dataInputStream = new DataInputStream(inStream);
+int scale = dataInputStream.readInt();
+int bigIntBytesSize = dataInputStream.readInt();
+
+byte[] bigIntBytes = new byte[bigIntBytesSize];
+dataInputStream.readFully(bigIntBytes);
+
+BigInteger bigInteger = new BigInteger(bigIntBytes);
+BigDecimal bigDecimal = new BigDecimal(bigInteger, scale);
+
+return bigDecimal;
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return {@code true}. This coder is injective.
+   */
+  @Override
+  public boolean consistentWithEquals() {
+return true;
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return {@code true}, because {@link #getEncodedElementByteSize} runs in 
constant time.
+   */
+  @Override
+  public boolean isRegisterByteSizeObserverCheap(BigDecimal value, Context 
context) {
+return true;
+  }
+
+  /**
+   * {@inheritDoc}
+   *
+   * @return {@code 8} plus the size of the {@link BigInteger} bytes.
+   */
+  @Override
+  protected long getEncodedElementByteSize(BigDecimal value, Context context)
+  throws Exception {
+if (value == null) {
+  throw new CoderException("cannot encode a null 

[08/50] [abbrv] incubator-beam git commit: Explicitly set the Runner in TestFlinkPipelineRunner

[davor]

Explicitly set the Runner in TestFlinkPipelineRunner

This ensures that the created PipelineOptions are valid if the
DirectRunner is not on the classpath.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90bb20ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90bb20ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90bb20ee

Branch: refs/heads/python-sdk
Commit: 90bb20ee6738c57bc25f47e2d80690fb721b562e
Parents: 0065851
Author: Thomas Groh 
Authored: Tue Jun 14 15:49:34 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/90bb20ee/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java
index 139aebf..a55acb7 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/TestFlinkPipelineRunner.java
@@ -45,6 +45,7 @@ public class TestFlinkPipelineRunner extends 
PipelineRunner {
 
   public static TestFlinkPipelineRunner create(boolean streaming) {
 FlinkPipelineOptions flinkOptions = 
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
+flinkOptions.setRunner(TestFlinkPipelineRunner.class);
 flinkOptions.setStreaming(streaming);
 return TestFlinkPipelineRunner.fromOptions(flinkOptions);
   }

[11/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
index 8b8d44f..c378cf4 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoSingleEvaluatorFactoryTest.java
@@ -25,9 +25,9 @@ import static org.junit.Assert.assertThat;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
 
-import org.apache.beam.runners.direct.InMemoryWatermarkManager.TimerUpdate;
 import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
+import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
 import org.apache.beam.sdk.coders.StringUtf8Coder;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
@@ -66,7 +66,7 @@ import java.io.Serializable;
  */
 @RunWith(JUnit4.class)
 public class ParDoSingleEvaluatorFactoryTest implements Serializable {
-  private transient BundleFactory bundleFactory = 
InProcessBundleFactory.create();
+  private transient BundleFactory bundleFactory = 
ImmutableListBundleFactory.create();
 
   @Test
   public void testParDoInMemoryTransformEvaluator() throws Exception {
@@ -85,11 +85,11 @@ public class ParDoSingleEvaluatorFactoryTest implements 
Serializable {
 CommittedBundle inputBundle =
 bundleFactory.createRootBundle(input).commit(Instant.now());
 
-InProcessEvaluationContext evaluationContext = 
mock(InProcessEvaluationContext.class);
+EvaluationContext evaluationContext = mock(EvaluationContext.class);
 UncommittedBundle outputBundle = 
bundleFactory.createRootBundle(collection);
 when(evaluationContext.createBundle(inputBundle, 
collection)).thenReturn(outputBundle);
-InProcessExecutionContext executionContext =
-new InProcessExecutionContext(null, null, null, null);
+DirectExecutionContext executionContext =
+new DirectExecutionContext(null, null, null, null);
 
when(evaluationContext.getExecutionContext(collection.getProducingTransformInternal(),
 inputBundle.getKey())).thenReturn(executionContext);
 CounterSet counters = new CounterSet();
@@ -106,7 +106,7 @@ public class ParDoSingleEvaluatorFactoryTest implements 
Serializable {
 evaluator.processElement(
 WindowedValue.valueInGlobalWindow("bazam", 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
 
-InProcessTransformResult result = evaluator.finishBundle();
+TransformResult result = evaluator.finishBundle();
 assertThat(result.getOutputBundles(), 
Matchers.contains(outputBundle));
 assertThat(result.getWatermarkHold(), 
equalTo(BoundedWindow.TIMESTAMP_MAX_VALUE));
 assertThat(result.getCounters(), equalTo(counters));
@@ -137,11 +137,11 @@ public class ParDoSingleEvaluatorFactoryTest implements 
Serializable {
 CommittedBundle inputBundle =
 bundleFactory.createRootBundle(input).commit(Instant.now());
 
-InProcessEvaluationContext evaluationContext = 
mock(InProcessEvaluationContext.class);
+EvaluationContext evaluationContext = mock(EvaluationContext.class);
 UncommittedBundle outputBundle = 
bundleFactory.createRootBundle(collection);
 when(evaluationContext.createBundle(inputBundle, 
collection)).thenReturn(outputBundle);
-InProcessExecutionContext executionContext =
-new InProcessExecutionContext(null, null, null, null);
+DirectExecutionContext executionContext =
+new DirectExecutionContext(null, null, null, null);
 
when(evaluationContext.getExecutionContext(collection.getProducingTransformInternal(),
 inputBundle.getKey())).thenReturn(executionContext);
 CounterSet counters = new CounterSet();
@@ -158,7 +158,7 @@ public class ParDoSingleEvaluatorFactoryTest implements 
Serializable {
 evaluator.processElement(
 WindowedValue.valueInGlobalWindow("bazam", 
PaneInfo.ON_TIME_AND_ONLY_FIRING));
 
-InProcessTransformResult result = evaluator.finishBundle();
+TransformResult result = evaluator.finishBundle();
 assertThat(
 result.getOutputBundles(), 
Matchers.containsInAnyOrder(outputBundle));
 assertThat(result.getWatermarkHold(), 
equalTo(BoundedWindow.TIMESTAMP_MAX_VALUE));
@@ -198,13 +198,13 @@ public class ParDoSingleEvaluatorFactoryTest implements 
Serializable {
 CommittedBundle inputBundle =
 bundleFactory.createRootBundle(input).commit(Instant.now());
 
-InProcessEvaluationContext evaluationContext 

[35/50] [abbrv] incubator-beam git commit: Remove references to javax.servlet.

[davor]

Remove references to javax.servlet.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/09bf9b37
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/09bf9b37
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/09bf9b37

Branch: refs/heads/python-sdk
Commit: 09bf9b3720f08acc9e94784461f2482ab371cd90
Parents: 6a41da8
Author: Pei He 
Authored: Fri Jun 17 13:02:58 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 examples/java/pom.xml  | 6 --
 .../org/apache/beam/examples/common/DataflowExampleUtils.java  | 6 +++---
 2 files changed, 3 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/09bf9b37/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 5211b80..9458962 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -285,12 +285,6 @@
   runtime
 
 
-
-  javax.servlet
-  javax.servlet-api
-  3.1.0
-
-
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/09bf9b37/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
index a0b7319..5b1af6d 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
@@ -62,8 +62,6 @@ import java.util.List;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
-import javax.servlet.http.HttpServletResponse;
-
 /**
  * The utility class that sets up and tears down external resources, starts 
the Google Cloud Pub/Sub
  * injector, and cancels the streaming and the injector pipelines once the 
program terminates.
@@ -72,6 +70,8 @@ import javax.servlet.http.HttpServletResponse;
  */
 public class DataflowExampleUtils {
 
+  private static final int SC_NOT_FOUND = 404;
+
   private final DataflowPipelineOptions options;
   private Bigquery bigQueryClient = null;
   private Pubsub pubsubClient = null;
@@ -481,7 +481,7 @@ public class DataflowExampleUtils {
 try {
   return request.execute();
 } catch (GoogleJsonResponseException e) {
-  if (e.getStatusCode() == HttpServletResponse.SC_NOT_FOUND) {
+  if (e.getStatusCode() == SC_NOT_FOUND) {
 return null;
   } else {
 throw e;

[38/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
new file mode 100644
index 000..e094d0d
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
@@ -0,0 +1,1417 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import static org.apache.beam.sdk.util.WindowedValue.valueInGlobalWindow;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasItem;
+import static org.hamcrest.Matchers.instanceOf;
+import static org.hamcrest.Matchers.startsWith;
+import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsList;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsMap;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsMultimap;
+import org.apache.beam.runners.dataflow.DataflowRunner.BatchViewAsSingleton;
+import org.apache.beam.runners.dataflow.DataflowRunner.TransformedMap;
+import org.apache.beam.runners.dataflow.internal.IsmFormat;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecord;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.IsmRecordCoder;
+import org.apache.beam.runners.dataflow.internal.IsmFormat.MetadataKeyCoder;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineDebugOptions;
+import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.Pipeline.PipelineVisitor;
+import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
+import org.apache.beam.sdk.coders.BigEndianLongCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.VarLongCoder;
+import org.apache.beam.sdk.io.AvroIO;
+import org.apache.beam.sdk.io.AvroSource;
+import org.apache.beam.sdk.io.BigQueryIO;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.io.TextIO;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.runners.TransformTreeNode;
+import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFnTester;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.PaneInfo;
+import org.apache.beam.sdk.util.CoderUtils;
+import org.apache.beam.sdk.util.GcsUtil;
+import org.apache.beam.sdk.util.NoopPathValidator;
+import org.apache.beam.sdk.util.ReleaseInfo;
+import org.apache.beam.sdk.util.TestCredential;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.FullWindowedValueCoder;
+import org.apache.beam.sdk.util.WindowingStrategy;
+import org.apache.beam.sdk.util.gcsfs.GcsPath;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import 

[47/50] [abbrv] incubator-beam git commit: Rename FlinkPipelineRunner to FlinkRunner

[davor]

Rename FlinkPipelineRunner to FlinkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4b417680
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4b417680
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4b417680

Branch: refs/heads/python-sdk
Commit: 4b417680d09da9f9f1c990a4f235572689efdf93
Parents: d285e67
Author: Thomas Groh 
Authored: Fri Jun 17 09:57:25 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 README.md   |   2 +-
 runners/flink/README.md |   4 +-
 .../beam/runners/flink/examples/TFIDF.java  |   4 +-
 .../beam/runners/flink/examples/WordCount.java  |   4 +-
 .../flink/examples/streaming/AutoComplete.java  |   4 +-
 .../flink/examples/streaming/JoinExamples.java  |   4 +-
 .../examples/streaming/KafkaIOExamples.java |   4 +-
 .../KafkaWindowedWordCountExample.java  |   4 +-
 .../examples/streaming/WindowedWordCount.java   |   4 +-
 runners/flink/runner/pom.xml|   4 +-
 .../beam/runners/flink/FlinkPipelineRunner.java | 180 ---
 .../apache/beam/runners/flink/FlinkRunner.java  | 179 ++
 .../runners/flink/FlinkRunnerRegistrar.java |   4 +-
 .../runners/flink/TestFlinkPipelineRunner.java  |  81 -
 .../beam/runners/flink/TestFlinkRunner.java |  81 +
 .../streaming/io/UnboundedFlinkSource.java  |   6 +-
 .../streaming/state/FlinkStateInternals.java|   2 +-
 .../runners/flink/FlinkRunnerRegistrarTest.java |   8 +-
 .../beam/runners/flink/FlinkTestPipeline.java   |   4 +-
 19 files changed, 291 insertions(+), 292 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4b417680/README.md
--
diff --git a/README.md b/README.md
index b34d933..c4a9155 100644
--- a/README.md
+++ b/README.md
@@ -68,7 +68,7 @@ Beam supports executing programs on multiple distributed 
processing backends thr
 
 - The `DirectRunner` runs the pipeline on your local machine.
 - The `DataflowPipelineRunner` submits the pipeline to the [Google Cloud 
Dataflow](http://cloud.google.com/dataflow/).
-- The `FlinkPipelineRunner` runs the pipeline on an Apache Flink cluster. The 
code has been donated from 
[dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) 
and is now part of Beam.
+- The `FlinkRunner` runs the pipeline on an Apache Flink cluster. The code has 
been donated from 
[dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) 
and is now part of Beam.
 - The `SparkPipelineRunner` runs the pipeline on an Apache Spark cluster. The 
code has been donated from 
[cloudera/spark-dataflow](https://github.com/cloudera/spark-dataflow) and is 
now part of Beam.
 
 Have ideas for new Runners? See the 
[Jira](https://issues.apache.org/jira/browse/BEAM/component/12328916/).

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4b417680/runners/flink/README.md
--
diff --git a/runners/flink/README.md b/runners/flink/README.md
index 69e2abb..3348119 100644
--- a/runners/flink/README.md
+++ b/runners/flink/README.md
@@ -107,14 +107,14 @@ Flink-Runner is now installed in your local maven 
repository.
 
 Next, let's run the classic WordCount example. It's semantically identically to
 the example provided with Apache Beam. Only this time, we chose the
-`FlinkPipelineRunner` to execute the WordCount on top of Flink.
+`FlinkRunner` to execute the WordCount on top of Flink.
 
 Here's an excerpt from the WordCount class file:
 
 ```java
 Options options = PipelineOptionsFactory.fromArgs(args).as(Options.class);
 // yes, we want to run WordCount with Flink
-options.setRunner(FlinkPipelineRunner.class);
+options.setRunner(FlinkRunner.class);
 
 Pipeline p = Pipeline.create(options);
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4b417680/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
index 876ecde..af920aa 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/TFIDF.java
@@ -18,7 +18,7 @@
 package org.apache.beam.runners.flink.examples;
 
 import org.apache.beam.runners.flink.FlinkPipelineOptions;
-import 

[14/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/babddbbc/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTimerInternalsTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTimerInternalsTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTimerInternalsTest.java
new file mode 100644
index 000..c038910
--- /dev/null
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectTimerInternalsTest.java
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.direct;
+
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Mockito.when;
+
+import org.apache.beam.runners.direct.WatermarkManager.TimerUpdate;
+import 
org.apache.beam.runners.direct.WatermarkManager.TimerUpdate.TimerUpdateBuilder;
+import org.apache.beam.runners.direct.WatermarkManager.TransformWatermarks;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.apache.beam.sdk.util.TimerInternals.TimerData;
+import org.apache.beam.sdk.util.state.StateNamespaces;
+
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+/**
+ * Tests for {@link DirectTimerInternals}.
+ */
+@RunWith(JUnit4.class)
+public class DirectTimerInternalsTest {
+  private MockClock clock;
+  @Mock private TransformWatermarks watermarks;
+
+  private TimerUpdateBuilder timerUpdateBuilder;
+
+  private DirectTimerInternals internals;
+
+  @Before
+  public void setup() {
+MockitoAnnotations.initMocks(this);
+clock = MockClock.fromInstant(new Instant(0));
+
+timerUpdateBuilder = TimerUpdate.builder(StructuralKey.of(1234, 
VarIntCoder.of()));
+
+internals = DirectTimerInternals.create(clock, watermarks, 
timerUpdateBuilder);
+  }
+
+  @Test
+  public void setTimerAddsToBuilder() {
+TimerData eventTimer =
+TimerData.of(StateNamespaces.global(), new Instant(20145L), 
TimeDomain.EVENT_TIME);
+TimerData processingTimer =
+TimerData.of(StateNamespaces.global(), new Instant(12555L), 
TimeDomain.PROCESSING_TIME);
+TimerData synchronizedProcessingTimer =
+TimerData.of(
+StateNamespaces.global(),
+new Instant(98745632189L),
+TimeDomain.SYNCHRONIZED_PROCESSING_TIME);
+internals.setTimer(eventTimer);
+internals.setTimer(processingTimer);
+internals.setTimer(synchronizedProcessingTimer);
+
+assertThat(
+internals.getTimerUpdate().getSetTimers(),
+containsInAnyOrder(eventTimer, synchronizedProcessingTimer, 
processingTimer));
+  }
+
+  @Test
+  public void deleteTimerDeletesOnBuilder() {
+TimerData eventTimer =
+TimerData.of(StateNamespaces.global(), new Instant(20145L), 
TimeDomain.EVENT_TIME);
+TimerData processingTimer =
+TimerData.of(StateNamespaces.global(), new Instant(12555L), 
TimeDomain.PROCESSING_TIME);
+TimerData synchronizedProcessingTimer =
+TimerData.of(
+StateNamespaces.global(),
+new Instant(98745632189L),
+TimeDomain.SYNCHRONIZED_PROCESSING_TIME);
+internals.deleteTimer(eventTimer);
+internals.deleteTimer(processingTimer);
+internals.deleteTimer(synchronizedProcessingTimer);
+
+assertThat(
+internals.getTimerUpdate().getDeletedTimers(),
+containsInAnyOrder(eventTimer, synchronizedProcessingTimer, 
processingTimer));
+  }
+
+  @Test
+  public void getProcessingTimeIsClockNow() {
+assertThat(internals.currentProcessingTime(), equalTo(clock.now()));
+Instant oldProcessingTime = internals.currentProcessingTime();
+
+clock.advance(Duration.standardHours(12));
+
+assertThat(internals.currentProcessingTime(), equalTo(clock.now()));
+assertThat(
+

[24/50] [abbrv] incubator-beam git commit: [Spark] Elide assigning windows when WindowFn is null

[davor]

[Spark] Elide assigning windows when WindowFn is null

Previously, when translating a Window.Bound transform, the case
where the WindowFn was null was missed, resulting in a
NullPointerException.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8278e5f7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8278e5f7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8278e5f7

Branch: refs/heads/python-sdk
Commit: 8278e5f78f36fb48fae994ee7abcc1485db84189
Parents: 0a7246d
Author: Kenneth Knowles 
Authored: Wed Jun 15 10:42:59 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../spark/translation/TransformTranslator.java | 17 ++---
 .../apache/beam/sdk/util/AssignWindowsDoFn.java|  9 -
 2 files changed, 18 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8278e5f7/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
index ebceb6b..34a0ede 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java
@@ -689,8 +689,6 @@ public final class TransformTranslator {
 rdd.saveAsNewAPIHadoopFile(outputDir, keyClass, valueClass, formatClass, 
conf);
   }
 
-  private static final FieldGetter WINDOW_FG = new 
FieldGetter(Window.Bound.class);
-
   private static  
TransformEvaluator window() {
 return new TransformEvaluator() {
   @Override
@@ -698,14 +696,19 @@ public final class TransformTranslator {
 @SuppressWarnings("unchecked")
 JavaRDDLike inRDD =
 (JavaRDDLike) context.getInputRDD(transform);
-WindowFn windowFn = WINDOW_FG.get("windowFn", transform);
+
+@SuppressWarnings("unchecked")
+WindowFn windowFn = (WindowFn) 
transform.getWindowFn();
+
 // Avoid running assign windows if both source and destination are 
global window
-if (context.getInput(transform).getWindowingStrategy().getWindowFn()
-instanceof GlobalWindows
-&& windowFn instanceof GlobalWindows) {
+// or if the user has not specified the WindowFn (meaning they are 
just messing
+// with triggering or allowed lateness)
+if (windowFn == null
+|| 
(context.getInput(transform).getWindowingStrategy().getWindowFn()
+instanceof GlobalWindows
+&& windowFn instanceof GlobalWindows)) {
   context.setOutputRDD(transform, inRDD);
 } else {
-  @SuppressWarnings("unchecked")
   DoFn addWindowsDoFn = new AssignWindowsDoFn<>(windowFn);
   DoFnFunction dofn =
   new DoFnFunction<>(addWindowsDoFn, context.getRuntimeContext(), 
null);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8278e5f7/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AssignWindowsDoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AssignWindowsDoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AssignWindowsDoFn.java
index e71a47e..caec40e 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AssignWindowsDoFn.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/AssignWindowsDoFn.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.util;
 
+import static com.google.common.base.Preconditions.checkNotNull;
+
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
@@ -37,7 +39,12 @@ public class AssignWindowsDoFn 
extends DoFn {
   private WindowFn fn;
 
   public AssignWindowsDoFn(WindowFn fn) {
-this.fn = fn;
+this.fn =
+checkNotNull(
+fn,
+"%s provided to %s cannot be null",
+WindowFn.class.getSimpleName(),
+AssignWindowsDoFn.class.getSimpleName());
   }
 
   @Override

[32/50] [abbrv] incubator-beam git commit: Finish removing DirectPipelineRunner references

[davor]

Finish removing DirectPipelineRunner references


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/340d0984
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/340d0984
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/340d0984

Branch: refs/heads/python-sdk
Commit: 340d09845959340f73577512437ebe0939bdeff9
Parents: 09bf9b3
Author: Thomas Groh 
Authored: Fri Jun 17 13:22:26 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../examples/common/DataflowExampleUtils.java   |  4 +-
 .../examples/cookbook/DatastoreWordCount.java   |  4 +-
 .../translation/TransformTranslatorTest.java|  2 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java   |  2 +-
 .../java/org/apache/beam/sdk/io/TextIO.java |  2 +-
 .../beam/sdk/options/DirectPipelineOptions.java | 74 
 .../beam/sdk/options/PipelineOptions.java   |  7 +-
 .../beam/sdk/util/BigQueryTableRowIterator.java |  2 +-
 .../apache/beam/sdk/util/DoFnRunnerBase.java|  2 +-
 .../beam/sdk/runners/PipelineRunnerTest.java| 24 ---
 .../main/java/common/DataflowExampleUtils.java  |  2 +-
 .../src/main/java/StarterPipeline.java  |  2 +-
 .../src/main/java/it/pkg/StarterPipeline.java   |  2 +-
 13 files changed, 28 insertions(+), 101 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/340d0984/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
index 5b1af6d..46b8af3 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/common/DataflowExampleUtils.java
@@ -315,7 +315,7 @@ public class DataflowExampleUtils {
   }
 
   /**
-   * Do some runner setup: check that the DirectPipelineRunner is not used in 
conjunction with
+   * Do some runner setup: check that the DirectRunner is not used in 
conjunction with
* streaming, and if streaming is specified, use the DataflowPipelineRunner.
*/
   public void setupRunner() {
@@ -413,7 +413,7 @@ public class DataflowExampleUtils {
   }
 } else {
   // Do nothing if the given PipelineResult doesn't support waitToFinish(),
-  // such as EvaluationResults returned by DirectPipelineRunner.
+  // such as EvaluationResults returned by DirectRunner.
   tearDown();
   printPendingMessages();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/340d0984/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
index bfaecdf..2d1f88c 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/cookbook/DatastoreWordCount.java
@@ -193,7 +193,7 @@ public class DatastoreWordCount {
 
   /**
* An example that creates a pipeline to populate DatastoreIO from a
-   * text input.  Forces use of DirectPipelineRunner for local execution mode.
+   * text input.  Forces use of DirectRunner for local execution mode.
*/
   public static void writeDataToDatastore(Options options) {
   Pipeline p = Pipeline.create(options);
@@ -247,7 +247,7 @@ public class DatastoreWordCount {
 
   /**
* An example to demo how to use {@link DatastoreIO}.  The runner here is
-   * customizable, which means users could pass either {@code 
DirectPipelineRunner}
+   * customizable, which means users could pass either {@code DirectRunner}
* or {@code DataflowPipelineRunner} in the pipeline options.
*/
   public static void main(String args[]) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/340d0984/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
index 5fdfb49..b593316 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/TransformTranslatorTest.java
+++ 

[33/50] [abbrv] incubator-beam git commit: Fix overflow in ReduceFnRunner garbage collection times

[davor]

Fix overflow in ReduceFnRunner garbage collection times


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4f7a2ab4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4f7a2ab4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4f7a2ab4

Branch: refs/heads/python-sdk
Commit: 4f7a2ab47c5fdd9b3de5f091a40128e68ddd11a3
Parents: 5bf732c
Author: Kenneth Knowles 
Authored: Tue Jun 14 16:10:09 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../org/apache/beam/sdk/util/ReduceFnRunner.java | 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4f7a2ab4/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
index 34208da..864e8e7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReduceFnRunner.java
@@ -936,16 +936,21 @@ public class ReduceFnRunner {
   }
 
   /**
-   * Return when {@code window} should be garbage collected. If the window is 
the GlobalWindow,
-   * that will be the end of the window. Otherwise, add the allowed lateness 
to the end of
-   * the window.
+   * Return when {@code window} should be garbage collected. If the window's 
expiration time is on
+   * or after the end of the global window, it will be truncated to the end of 
the global window.
*/
   private Instant garbageCollectionTime(W window) {
-Instant maxTimestamp = window.maxTimestamp();
-if (maxTimestamp.isBefore(GlobalWindow.INSTANCE.maxTimestamp())) {
-  return maxTimestamp.plus(windowingStrategy.getAllowedLateness());
+
+// If the end of the window + allowed lateness is beyond the "end of time" 
aka the end of the
+// global window, then we truncate it. The conditional is phrased like it 
is because the
+// addition of EOW + allowed lateness might even overflow the maximum 
allowed Instant
+if (GlobalWindow.INSTANCE
+.maxTimestamp()
+.minus(windowingStrategy.getAllowedLateness())
+.isBefore(window.maxTimestamp())) {
+  return GlobalWindow.INSTANCE.maxTimestamp();
 } else {
-  return maxTimestamp;
+  return 
window.maxTimestamp().plus(windowingStrategy.getAllowedLateness());
 }
   }
 

[18/50] [abbrv] incubator-beam git commit: Remove InProcess Prefixes

[davor]

Remove InProcess Prefixes

These prefixes are out of date with the rename of the runner. Most of
the prefixes are be droped in their entirety, as the classes are scoped
to the direct runner module.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/babddbbc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/babddbbc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/babddbbc

Branch: refs/heads/python-sdk
Commit: babddbbc8247bc7322c3fd519a5bf0fa23c57064
Parents: 9400fc9
Author: Thomas Groh 
Authored: Wed Jun 15 11:21:41 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../direct/AbstractModelEnforcement.java|2 +-
 .../direct/BoundedReadEvaluatorFactory.java |   12 +-
 .../beam/runners/direct/BundleFactory.java  |4 +-
 .../beam/runners/direct/CommittedResult.java|4 +-
 .../beam/runners/direct/CompletionCallback.java |2 +-
 .../runners/direct/DirectExecutionContext.java  |  106 ++
 .../beam/runners/direct/DirectGroupByKey.java   |  132 ++
 .../direct/DirectGroupByKeyOverrideFactory.java |   41 +
 .../beam/runners/direct/DirectRegistrar.java|   16 +-
 .../beam/runners/direct/DirectRunner.java   |   29 +-
 .../runners/direct/DirectTimerInternals.java|   84 ++
 .../runners/direct/EmptyTransformEvaluator.java |4 +-
 .../beam/runners/direct/EvaluationContext.java  |  429 ++
 .../beam/runners/direct/EvaluatorKey.java   |4 +-
 .../direct/ExecutorServiceParallelExecutor.java |   28 +-
 .../runners/direct/FlattenEvaluatorFactory.java |   12 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |  127 ++
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |  186 +++
 .../direct/ImmutabilityEnforcementFactory.java  |2 +-
 .../direct/ImmutableListBundleFactory.java  |  163 ++
 .../direct/InMemoryWatermarkManager.java| 1420 -
 .../runners/direct/InProcessBundleFactory.java  |  161 --
 .../direct/InProcessBundleOutputManager.java|   51 -
 .../direct/InProcessEvaluationContext.java  |  429 --
 .../direct/InProcessExecutionContext.java   |  105 --
 .../beam/runners/direct/InProcessExecutor.java  |   48 -
 ...rocessGroupAlsoByWindowEvaluatorFactory.java |  127 --
 .../runners/direct/InProcessGroupByKey.java |  132 --
 ...InProcessGroupByKeyOnlyEvaluatorFactory.java |  185 ---
 .../InProcessGroupByKeyOverrideFactory.java |   41 -
 .../direct/InProcessSideInputContainer.java |  277 
 .../runners/direct/InProcessTimerInternals.java |   84 --
 .../direct/InProcessTransformResult.java|   84 --
 .../beam/runners/direct/ModelEnforcement.java   |6 +-
 .../beam/runners/direct/ParDoEvaluator.java |  186 +++
 .../runners/direct/ParDoInProcessEvaluator.java |  186 ---
 .../direct/ParDoMultiEvaluatorFactory.java  |6 +-
 .../direct/ParDoSingleEvaluatorFactory.java |6 +-
 .../direct/PassthroughTransformEvaluator.java   |2 +-
 .../beam/runners/direct/PipelineExecutor.java   |   48 +
 .../beam/runners/direct/SideInputContainer.java |  277 
 .../runners/direct/StepTransformResult.java |6 +-
 ...readLocalInvalidatingTransformEvaluator.java |2 +-
 .../beam/runners/direct/TransformEvaluator.java |4 +-
 .../direct/TransformEvaluatorFactory.java   |2 +-
 .../direct/TransformEvaluatorRegistry.java  |   10 +-
 .../beam/runners/direct/TransformExecutor.java  |   14 +-
 .../beam/runners/direct/TransformResult.java|   84 ++
 .../direct/UnboundedReadEvaluatorFactory.java   |   12 +-
 .../direct/UncommittedBundleOutputManager.java  |   51 +
 .../runners/direct/ViewEvaluatorFactory.java|   16 +-
 .../beam/runners/direct/WatermarkManager.java   | 1420 +
 .../runners/direct/WindowEvaluatorFactory.java  |   12 +-
 .../direct/BoundedReadEvaluatorFactoryTest.java |   10 +-
 .../runners/direct/CommittedResultTest.java |2 +-
 .../runners/direct/DirectRegistrarTest.java |   18 +-
 .../direct/DirectTimerInternalsTest.java|  134 ++
 .../EncodabilityEnforcementFactoryTest.java |2 +-
 .../runners/direct/EvaluationContextTest.java   |  545 +++
 .../direct/FlattenEvaluatorFactoryTest.java |   12 +-
 .../direct/GroupByKeyEvaluatorFactoryTest.java  |9 +-
 .../GroupByKeyOnlyEvaluatorFactoryTest.java |  197 +++
 .../ImmutabilityCheckingBundleFactoryTest.java  |2 +-
 .../ImmutabilityEnforcementFactoryTest.java |2 +-
 .../direct/ImmutableListBundleFactoryTest.java  |  231 +++
 .../direct/InMemoryWatermarkManagerTest.java| 1428 --
 .../direct/InProcessBundleFactoryTest.java  |  231 ---
 .../direct/InProcessEvaluationContextTest.java  |  545 ---
 ...ocessGroupByKeyOnlyEvaluatorFactoryTest.java |  196 ---
 

[40/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
new file mode 100644
index 000..05297ec
--- /dev/null
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunnerHooks.java
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow;
+
+import org.apache.beam.sdk.annotations.Experimental;
+
+import com.google.api.services.dataflow.model.Environment;
+
+/**
+ * An instance of this class can be passed to the
+ * {@link DataflowRunner} to add user defined hooks to be
+ * invoked at various times during pipeline execution.
+ */
+@Experimental
+public class DataflowRunnerHooks {
+  /**
+   * Allows the user to modify the environment of their job before their job 
is submitted
+   * to the service for execution.
+   *
+   * @param environment The environment of the job. Users can make change to 
this instance in order
+   * to change the environment with which their job executes on the 
service.
+   */
+  public void modifyEnvironmentBeforeSubmission(Environment environment) {}
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
index f2e8459..ea83bfb 100755
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/internal/package-info.java
@@ -16,6 +16,6 @@
  * limitations under the License.
  */
 /**
- * Implementation of the {@link 
org.apache.beam.runners.dataflow.DataflowPipelineRunner}.
+ * Implementation of the {@link 
org.apache.beam.runners.dataflow.DataflowRunner}.
  */
 package org.apache.beam.runners.dataflow.internal;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
index 7fa5ad6..809df35 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/BlockingDataflowPipelineOptions.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.dataflow.options;
 
-import org.apache.beam.runners.dataflow.BlockingDataflowPipelineRunner;
+import org.apache.beam.runners.dataflow.BlockingDataflowRunner;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.DefaultValueFactory;
 import org.apache.beam.sdk.options.Description;
@@ -29,9 +29,9 @@ import com.fasterxml.jackson.annotation.JsonIgnore;
 import java.io.PrintStream;
 
 /**
- * Options that are used to configure the {@link 
BlockingDataflowPipelineRunner}.
+ * Options that are used to configure the {@link BlockingDataflowRunner}.
  */
-@Description("Configure options on the BlockingDataflowPipelineRunner.")
+@Description("Configure options on the BlockingDataflowRunner.")
 public interface BlockingDataflowPipelineOptions 

[31/50] [abbrv] incubator-beam git commit: Remove last vestige of the words DirectPipeline

[davor]

Remove last vestige of the words DirectPipeline


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a41da85
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a41da85
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a41da85

Branch: refs/heads/python-sdk
Commit: 6a41da853537e152613fb17bed782bc16d767c57
Parents: e581244
Author: Thomas Groh 
Authored: Fri Jun 17 10:25:44 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6a41da85/README.md
--
diff --git a/README.md b/README.md
index fa00e82..b34d933 100644
--- a/README.md
+++ b/README.md
@@ -66,7 +66,7 @@ Have ideas for new SDKs or DSLs? See the 
[Jira](https://issues.apache.org/jira/b
 
 Beam supports executing programs on multiple distributed processing backends 
through PipelineRunners. Currently, the following PipelineRunners are available:
 
-- The `DirectPipelineRunner` runs the pipeline on your local machine.
+- The `DirectRunner` runs the pipeline on your local machine.
 - The `DataflowPipelineRunner` submits the pipeline to the [Google Cloud 
Dataflow](http://cloud.google.com/dataflow/).
 - The `FlinkPipelineRunner` runs the pipeline on an Apache Flink cluster. The 
code has been donated from 
[dataArtisans/flink-dataflow](https://github.com/dataArtisans/flink-dataflow) 
and is now part of Beam.
 - The `SparkPipelineRunner` runs the pipeline on an Apache Spark cluster. The 
code has been donated from 
[cloudera/spark-dataflow](https://github.com/cloudera/spark-dataflow) and is 
now part of Beam.

[26/50] [abbrv] incubator-beam git commit: Touch up BigDecimalCoder and tests

[davor]

Touch up BigDecimalCoder and tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6491100a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6491100a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6491100a

Branch: refs/heads/python-sdk
Commit: 6491100a5d655cb9f6c702767d6354269208f650
Parents: 8268f1d
Author: Kenneth Knowles 
Authored: Thu Jun 9 13:24:28 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../apache/beam/sdk/coders/BigDecimalCoder.java |  56 --
 .../beam/sdk/coders/BigDecimalCoderTest.java| 105 +--
 2 files changed, 95 insertions(+), 66 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6491100a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
index 3b723b9..c4b7cd1 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/BigDecimalCoder.java
@@ -17,20 +17,22 @@
  */
 package org.apache.beam.sdk.coders;
 
+import static com.google.common.base.Preconditions.checkNotNull;
+
 import com.fasterxml.jackson.annotation.JsonCreator;
 
-import java.io.DataInputStream;
-import java.io.DataOutputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.math.BigDecimal;
 import java.math.BigInteger;
+import java.math.MathContext;
 
 /**
- * A {@link BigDecimalCoder} encodes {@link BigDecimal} in an integer and
- * a byte array. The integer represents the scale and the byte array
- * represents a {@link BigInteger}. The integer is in 4 bytes, big-endian.
+ * A {@link BigDecimalCoder} encodes a {@link BigDecimal} as an integer scale 
encoded with
+ * {@link VarIntCoder} and a {@link BigInteger} encoded using {@link 
BigIntegerCoder}. The
+ * {@link BigInteger}, when scaled (with unlimited precision, aka {@link 
MathContext#UNLIMITED}),
+ * yields the expected {@link BigDecimal}.
  */
 public class BigDecimalCoder extends AtomicCoder {
 
@@ -43,37 +45,25 @@ public class BigDecimalCoder extends 
AtomicCoder {
 
   private static final BigDecimalCoder INSTANCE = new BigDecimalCoder();
 
+  private final VarIntCoder integerCoder = VarIntCoder.of();
+  private final BigIntegerCoder bigIntegerCoder = BigIntegerCoder.of();
+
   private BigDecimalCoder() {}
 
   @Override
   public void encode(BigDecimal value, OutputStream outStream, Context context)
   throws IOException, CoderException {
-if (value == null) {
-  throw new CoderException("cannot encode a null BigDecimal");
-}
-
-byte[] bigIntBytes = value.unscaledValue().toByteArray();
-
-DataOutputStream dataOutputStream = new DataOutputStream(outStream);
-dataOutputStream.writeInt(value.scale());
-dataOutputStream.writeInt(bigIntBytes.length);
-dataOutputStream.write(bigIntBytes);
+checkNotNull(value, String.format("cannot encode a null %s", 
BigDecimal.class.getSimpleName()));
+integerCoder.encode(value.scale(), outStream, context.nested());
+bigIntegerCoder.encode(value.unscaledValue(), outStream, context.nested());
   }
 
   @Override
   public BigDecimal decode(InputStream inStream, Context context)
   throws IOException, CoderException {
-DataInputStream dataInputStream = new DataInputStream(inStream);
-int scale = dataInputStream.readInt();
-int bigIntBytesSize = dataInputStream.readInt();
-
-byte[] bigIntBytes = new byte[bigIntBytesSize];
-dataInputStream.readFully(bigIntBytes);
-
-BigInteger bigInteger = new BigInteger(bigIntBytes);
-BigDecimal bigDecimal = new BigDecimal(bigInteger, scale);
-
-return bigDecimal;
+int scale = integerCoder.decode(inStream, context.nested());
+BigInteger bigInteger = bigIntegerCoder.decode(inStream, context.nested());
+return new BigDecimal(bigInteger, scale);
   }
 
   /**
@@ -99,14 +89,14 @@ public class BigDecimalCoder extends 
AtomicCoder {
   /**
* {@inheritDoc}
*
-   * @return {@code 8} plus the size of the {@link BigInteger} bytes.
+   * @return {@code 4} (the size of an integer denoting the scale) plus {@code 
4} (the size of an
+   * integer length prefix for the following bytes) plus the size of the 
two's-complement
+   * representation of the {@link BigInteger} that, when scaled, equals the 
given value.
*/
   @Override
-  protected long getEncodedElementByteSize(BigDecimal value, Context context)
-  throws 

[19/50] [abbrv] incubator-beam git commit: Rename InProcessPipelineRunner to DirectRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluator.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluator.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluator.java
index ba9815b..1fec9d8 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluator.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluator.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
-import org.apache.beam.runners.direct.InProcessPipelineRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.util.WindowedValue;
 
 /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorFactory.java
index f2d577e..b12a34c 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorFactory.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
-import org.apache.beam.runners.direct.InProcessPipelineRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.DoFn;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
index 81d2520..dfc1753 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformEvaluatorRegistry.java
@@ -19,7 +19,7 @@ package org.apache.beam.runners.direct;
 
 import 
org.apache.beam.runners.direct.InProcessGroupByKey.InProcessGroupAlsoByWindow;
 import 
org.apache.beam.runners.direct.InProcessGroupByKey.InProcessGroupByKeyOnly;
-import org.apache.beam.runners.direct.InProcessPipelineRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.Flatten.FlattenPCollectionList;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
index fb637b4..4dd1475 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutor.java
@@ -19,7 +19,7 @@ package org.apache.beam.runners.direct;
 
 import static com.google.common.base.Preconditions.checkState;
 
-import org.apache.beam.runners.direct.InProcessPipelineRunner.CommittedBundle;
+import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
 import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.util.WindowedValue;
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9400fc9a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
index fceb20c..5030730 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java
@@ -17,8 +17,8 @@
  */
 package org.apache.beam.runners.direct;
 
-import 

[09/50] [abbrv] incubator-beam git commit: Add test for ReduceFnRunner GC time overflow

[davor]

Add test for ReduceFnRunner GC time overflow


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5bf732cd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5bf732cd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5bf732cd

Branch: refs/heads/python-sdk
Commit: 5bf732cd3e598321a5c51e1239eda0fe2877a65d
Parents: 6058330
Author: Kenneth Knowles 
Authored: Tue Jun 14 16:04:10 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../org/apache/beam/sdk/WindowMatchers.java |  5 ++
 .../beam/sdk/util/ReduceFnRunnerTest.java   | 68 
 2 files changed, 73 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5bf732cd/sdks/java/core/src/test/java/org/apache/beam/sdk/WindowMatchers.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/WindowMatchers.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/WindowMatchers.java
index b47c32c..7a5e2fb 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/WindowMatchers.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/WindowMatchers.java
@@ -47,6 +47,11 @@ public class WindowMatchers {
 return new WindowedValueMatcher<>(valueMatcher, timestampMatcher, 
Matchers.anything());
   }
 
+  public static  Matcher isWindowedValue(
+  Matcher valueMatcher) {
+return new WindowedValueMatcher<>(valueMatcher, Matchers.anything(), 
Matchers.anything());
+  }
+
   public static  Matcher isSingleWindowedValue(
   T value, long timestamp, long windowStart, long windowEnd) {
 return WindowMatchers.isSingleWindowedValue(

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5bf732cd/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReduceFnRunnerTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReduceFnRunnerTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReduceFnRunnerTest.java
index 0df4bc6..b7ec540 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReduceFnRunnerTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReduceFnRunnerTest.java
@@ -18,6 +18,7 @@
 package org.apache.beam.sdk.util;
 
 import static org.apache.beam.sdk.WindowMatchers.isSingleWindowedValue;
+import static org.apache.beam.sdk.WindowMatchers.isWindowedValue;
 
 import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
@@ -51,6 +52,7 @@ import org.apache.beam.sdk.transforms.windowing.FixedWindows;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
 import org.apache.beam.sdk.transforms.windowing.GlobalWindows;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Never;
 import org.apache.beam.sdk.transforms.windowing.OutputTimeFns;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo.Timing;
@@ -59,6 +61,7 @@ import org.apache.beam.sdk.transforms.windowing.Sessions;
 import org.apache.beam.sdk.transforms.windowing.SlidingWindows;
 import org.apache.beam.sdk.transforms.windowing.Trigger;
 import org.apache.beam.sdk.transforms.windowing.Window.ClosingBehavior;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
 import org.apache.beam.sdk.util.WindowingStrategy.AccumulationMode;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TimestampedValue;
@@ -79,6 +82,7 @@ import org.mockito.MockitoAnnotations;
 import org.mockito.invocation.InvocationOnMock;
 import org.mockito.stubbing.Answer;
 
+import java.util.Collection;
 import java.util.Iterator;
 import java.util.List;
 
@@ -226,6 +230,70 @@ public class ReduceFnRunnerTest {
 tester.assertHasOnlyGlobalAndFinishedSetsFor(firstWindow);
   }
 
+  /**
+   * Tests that the garbage collection time for a fixed window does not 
overflow the end of time.
+   */
+  @Test
+  public void testFixedWindowEndOfTimeGarbageCollection() throws Exception {
+
+Duration allowedLateness = Duration.standardDays(365);
+Duration windowSize = Duration.millis(10);
+WindowFn windowFn = FixedWindows.of(windowSize);
+
+// This timestamp falls into a window where the end of the window is 
before the end of the
+// global window - the "end of time" - yet its expiration time is after.
+final Instant elementTimestamp =
+GlobalWindow.INSTANCE.maxTimestamp().minus(allowedLateness).plus(1);
+
+IntervalWindow window = Iterables.getOnlyElement(
+

[01/50] [abbrv] incubator-beam git commit: Remove the DirectPipelineRunner from the Core SDK

[davor]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 0e5c662b4 -> f166b16b8


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99654ca4/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
--
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
index ea708e5..8abfb05 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/PipelineTest.java
@@ -28,8 +28,8 @@ import 
org.apache.beam.sdk.Pipeline.PipelineExecutionException;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptions.CheckEnabled;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.runners.DirectPipelineRunner;
 import org.apache.beam.sdk.runners.PipelineRunner;
+import org.apache.beam.sdk.testing.CrashingRunner;
 import org.apache.beam.sdk.testing.ExpectedLogs;
 import org.apache.beam.sdk.testing.NeedsRunner;
 import org.apache.beam.sdk.testing.PAssert;
@@ -157,7 +157,7 @@ public class PipelineTest {
   @Test
   public void testToString() {
 PipelineOptions options = PipelineOptionsFactory.as(PipelineOptions.class);
-options.setRunner(DirectPipelineRunner.class);
+options.setRunner(CrashingRunner.class);
 Pipeline pipeline = Pipeline.create(options);
 assertEquals("Pipeline#" + pipeline.hashCode(), pipeline.toString());
   }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99654ca4/sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java
index 774968f..cabfc21 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/BoundedReadFromUnboundedSourceTest.java
@@ -18,14 +18,12 @@
 package org.apache.beam.sdk.io;
 
 import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.includesDisplayDataFrom;
-
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.assertTrue;
 
 import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.runners.DirectPipelineRunner;
 import org.apache.beam.sdk.runners.dataflow.TestCountingSource;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.RunnableOnService;
@@ -123,10 +121,6 @@ public class BoundedReadFromUnboundedSourceTest implements 
Serializable{
   private void test(boolean dedup, boolean timeBound) throws Exception {
 Pipeline p = TestPipeline.create();
 
-if (p.getOptions().getRunner() == DirectPipelineRunner.class) {
-  finalizeTracker = new ArrayList<>();
-  TestCountingSource.setFinalizeTracker(finalizeTracker);
-}
 TestCountingSource source = new 
TestCountingSource(Integer.MAX_VALUE).withoutSplitting();
 if (dedup) {
   source = source.withDedup();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/99654ca4/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/DirectPipelineRegistrarTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/DirectPipelineRegistrarTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/DirectPipelineRegistrarTest.java
deleted file mode 100644
index 92c4835..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/DirectPipelineRegistrarTest.java
+++ /dev/null
@@ -1,71 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.runners;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
-
-import 

[42/50] [abbrv] incubator-beam git commit: Rename DataflowPipelineRunner to DataflowRunner

[davor]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
deleted file mode 100644
index 4d37966..000
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineRunnerHooks.java
+++ /dev/null
@@ -1,39 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow;
-
-import org.apache.beam.sdk.annotations.Experimental;
-
-import com.google.api.services.dataflow.model.Environment;
-
-/**
- * An instance of this class can be passed to the
- * {@link DataflowPipelineRunner} to add user defined hooks to be
- * invoked at various times during pipeline execution.
- */
-@Experimental
-public class DataflowPipelineRunnerHooks {
-  /**
-   * Allows the user to modify the environment of their job before their job 
is submitted
-   * to the service for execution.
-   *
-   * @param environment The environment of the job. Users can make change to 
this instance in order
-   * to change the environment with which their job executes on the 
service.
-   */
-  public void modifyEnvironmentBeforeSubmission(Environment environment) {}
-}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6d028ac6/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index f5fefc0..e15b9d2 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -30,7 +30,7 @@ import static org.apache.beam.sdk.util.Structs.getString;
 
 import static com.google.common.base.Preconditions.checkArgument;
 
-import 
org.apache.beam.runners.dataflow.DataflowPipelineRunner.GroupByKeyAndSortValuesOnly;
+import 
org.apache.beam.runners.dataflow.DataflowRunner.GroupByKeyAndSortValuesOnly;
 import org.apache.beam.runners.dataflow.internal.ReadTranslator;
 import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions;
 import org.apache.beam.runners.dataflow.util.DoFnInfo;
@@ -139,7 +139,7 @@ public class DataflowPipelineTranslator {
*/
   public JobSpecification translate(
   Pipeline pipeline,
-  DataflowPipelineRunner runner,
+  DataflowRunner runner,
   List packages) {
 
 Translator translator = new Translator(pipeline, runner);
@@ -224,7 +224,7 @@ public class DataflowPipelineTranslator {
 
   /**
* The interface provided to registered callbacks for interacting
-   * with the {@link DataflowPipelineRunner}, including reading and writing the
+   * with the {@link DataflowRunner}, including reading and writing the
* values of {@link PCollection}s and side inputs ({@link PCollectionView}s).
*/
   public interface TranslationContext {
@@ -350,7 +350,7 @@ public class DataflowPipelineTranslator {
 private final Pipeline pipeline;
 
 /** The runner which will execute the pipeline. */
-private final DataflowPipelineRunner runner;
+private final DataflowRunner runner;
 
 /** The Cloud Dataflow Job representation. */
 private final Job job = new Job();
@@ -385,7 +385,7 @@ public class DataflowPipelineTranslator {
  * Constructs a Translator that will translate the specified
  * Pipeline into Dataflow objects.
  */
-public Translator(Pipeline pipeline, DataflowPipelineRunner runner) {
+public Translator(Pipeline pipeline, 

[30/50] [abbrv] incubator-beam git commit: CrashingRunner: cleanup some code

[davor]

CrashingRunner: cleanup some code

make it final, fix an error message, remove unused code


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/93f9ef92
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/93f9ef92
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/93f9ef92

Branch: refs/heads/python-sdk
Commit: 93f9ef92dcdcdec4f481e996b02f256cb18dc628
Parents: 3d87f8b
Author: Dan Halperin 
Authored: Thu Jun 16 10:15:58 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../apache/beam/sdk/testing/CrashingRunner.java | 33 +++-
 .../apache/beam/sdk/testing/TestPipeline.java   |  3 +-
 2 files changed, 6 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93f9ef92/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java
index 975facc..b3360ae 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/CrashingRunner.java
@@ -21,17 +21,15 @@ package org.apache.beam.sdk.testing;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.PipelineResult;
 import org.apache.beam.sdk.options.PipelineOptions;
-import org.apache.beam.sdk.runners.AggregatorRetrievalException;
-import org.apache.beam.sdk.runners.AggregatorValues;
 import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.transforms.Aggregator;
 
 /**
  * A {@link PipelineRunner} that applies no overrides and throws an exception 
on calls to
  * {@link Pipeline#run()}. For use in {@link TestPipeline} to construct but 
not execute pipelines.
  */
-public class CrashingRunner extends PipelineRunner{
+public final class CrashingRunner extends PipelineRunner{
 
+  @SuppressWarnings("unused") // used by reflection
   public static CrashingRunner fromOptions(PipelineOptions opts) {
 return new CrashingRunner();
   }
@@ -41,32 +39,11 @@ public class CrashingRunner extends 
PipelineRunner{
 throw new IllegalArgumentException(String.format("Cannot call 
#run(Pipeline) on an instance "
 + "of %s. %s should only be used as the default to construct a 
Pipeline "
 + "using %s, and cannot execute Pipelines. Instead, specify a %s "
-+ "by providing PipelineOptions in the environment variable '%s'.",
-getClass().getSimpleName(),
-getClass().getSimpleName(),
++ "by providing PipelineOptions in the system property '%s'.",
+CrashingRunner.class.getSimpleName(),
+CrashingRunner.class.getSimpleName(),
 TestPipeline.class.getSimpleName(),
 PipelineRunner.class.getSimpleName(),
 TestPipeline.PROPERTY_BEAM_TEST_PIPELINE_OPTIONS));
   }
-
-  private static class TestPipelineResult implements PipelineResult {
-private TestPipelineResult() {
-  // Should never be instantiated by the enclosing class
-  throw new UnsupportedOperationException(String.format("Forbidden to 
instantiate %s",
-  getClass().getSimpleName()));
-}
-
-@Override
-public State getState() {
-  throw new UnsupportedOperationException(String.format("Forbidden to 
instantiate %s",
-  getClass().getSimpleName()));
-}
-
-@Override
-public  AggregatorValues getAggregatorValues(Aggregator 
aggregator)
-throws AggregatorRetrievalException {
-  throw new AssertionError(String.format("Forbidden to instantiate %s",
-  getClass().getSimpleName()));
-}
-  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/93f9ef92/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
index 4618e33..0dba043 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/TestPipeline.java
@@ -141,8 +141,7 @@ public class TestPipeline extends Pipeline {
   Strings.isNullOrEmpty(beamTestPipelineOptions)
   ? PipelineOptionsFactory.create()
   : PipelineOptionsFactory.fromArgs(
-  MAPPER.readValue(
-  
System.getProperty(PROPERTY_BEAM_TEST_PIPELINE_OPTIONS), String[].class))
+   

[48/50] [abbrv] incubator-beam git commit: Configure RunnableOnService tests for Spark runner, batch mode

[davor]

Configure RunnableOnService tests for Spark runner, batch mode


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/30d226a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/30d226a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/30d226a3

Branch: refs/heads/python-sdk
Commit: 30d226a3ae547c4a2d890d1d42487862323a4ae3
Parents: a24e557
Author: Kenneth Knowles 
Authored: Thu May 5 15:11:07 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:31 2016 -0700

--
 runners/spark/pom.xml   | 112 +--
 .../runners/spark/SparkRunnerRegistrar.java |   3 +-
 .../runners/spark/TestSparkPipelineRunner.java  |  77 +
 .../runners/spark/SparkRunnerRegistrarTest.java |   2 +-
 4 files changed, 155 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30d226a3/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index e7d0834..747464e 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -37,6 +37,62 @@
 1.6.1
   
 
+  
+
+  jacoco
+  
+
+  
+org.jacoco
+jacoco-maven-plugin
+  
+
+  
+
+
+
+  
+  runnable-on-service-tests
+  false
+  
+
+  
+
+  org.apache.maven.plugins
+  maven-surefire-plugin
+  
+
+  runnable-on-service-tests
+  
+
org.apache.beam.sdk.testing.RunnableOnService
+none
+true
+
+  org.apache.beam:java-sdk-all
+
+
+  org.apache.beam.sdk.io.BoundedReadFromUnboundedSourceTest
+
+
+  
+[
+  
"--runner=org.apache.beam.runners.spark.TestSparkPipelineRunner",
+  "--streaming=false"
+]
+  
+  
true
+
+  
+
+  
+
+  
+
+  
+
+  
+
   
 
   org.apache.spark
@@ -122,6 +178,25 @@
   org.apache.beam
   beam-runners-direct-java
   0.2.0-incubating-SNAPSHOT
+
+
+
+
+  org.apache.beam
+  beam-sdks-java-core
+  tests
+  test
+  
+
+  org.slf4j
+  slf4j-jdk14
+
+  
+
+
+
+  org.mockito
+  mockito-all
   test
 
   
@@ -237,41 +312,4 @@
 
   
 
-  
-
-  jacoco
-  
-
-  
-org.jacoco
-jacoco-maven-plugin
-  
-
-  
-
-
-
-  disable-runnable-on-service-tests
-  
-true
-  
-  
-
-  
-org.apache.maven.plugins
-maven-surefire-plugin
-
-  
-runnable-on-service-tests
-
-  true
-
-  
-
-  
-
-  
-
-  
-
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30d226a3/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
index 9537ec6..baa2241 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/SparkRunnerRegistrar.java
@@ -43,7 +43,8 @@ public final class SparkRunnerRegistrar {
   public static class Runner implements PipelineRunnerRegistrar {
 @Override
 public Iterable> getPipelineRunners() {
-  return ImmutableList.>of(SparkPipelineRunner.class);
+  return ImmutableList.>of(
+  SparkPipelineRunner.class, TestSparkPipelineRunner.class);
 }
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/30d226a3/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkPipelineRunner.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkPipelineRunner.java
 

[28/50] [abbrv] incubator-beam git commit: [BEAM-342] Implement Filter#greaterThan, etc with Filter#byPredicate

[davor]

[BEAM-342] Implement Filter#greaterThan,etc with Filter#byPredicate


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3d87f8b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3d87f8b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3d87f8b9

Branch: refs/heads/python-sdk
Commit: 3d87f8b987e243c6b3d99ab67142301af7b65743
Parents: 6491100
Author: manuzhang 
Authored: Wed Jun 15 16:02:35 2016 +0800
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:30 2016 -0700

--
 .../beam/examples/complete/AutoComplete.java|   2 +-
 .../examples/complete/AutoCompleteTest.java |  14 +-
 .../beam/examples/MinimalWordCountJava8.java|   2 +-
 .../examples/complete/game/HourlyTeamScore.java |   6 +-
 .../examples/MinimalWordCountJava8Test.java |   2 +-
 .../complete/game/HourlyTeamScoreTest.java  |   2 +-
 .../flink/examples/streaming/AutoComplete.java  |  12 +-
 .../org/apache/beam/sdk/transforms/Filter.java  | 128 +++
 .../apache/beam/sdk/transforms/FilterTest.java  |  63 +++--
 .../beam/sdk/transforms/FilterJava8Test.java|   8 +-
 10 files changed, 89 insertions(+), 150 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d87f8b9/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
index d725e0a..3e4440c 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/AutoComplete.java
@@ -235,7 +235,7 @@ public class AutoComplete {
 .of(larger.get(1).apply(ParDo.of(new FlattenTops(
 // ...together with those (previously excluded) candidates of 
length
 // exactly minPrefix...
-.and(input.apply(Filter.byPredicate(
+.and(input.apply(Filter.by(
 new SerializableFunction() {
   @Override
   public Boolean apply(CompletionCandidate c) {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d87f8b9/examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteTest.java
--
diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteTest.java
 
b/examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteTest.java
index 93dd0be..b2ed9a2 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteTest.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/complete/AutoCompleteTest.java
@@ -85,13 +85,13 @@ public class AutoCompleteTest implements Serializable {
 
 PCollection> output =
   input.apply(new ComputeTopCompletions(2, recursive))
-   .apply(Filter.byPredicate(
-new SerializableFunction, Boolean>() {
-  @Override
-  public Boolean apply(KV element) {
-return element.getKey().length() <= 2;
-  }
-  }));
+   .apply(Filter.by(
+   new SerializableFunction, 
Boolean>() {
+ @Override
+ public Boolean apply(KV 
element) {
+   return element.getKey().length() <= 2;
+ }
+   }));
 
 PAssert.that(output).containsInAnyOrder(
 KV.of("a", parseList("apple:2", "apricot:1")),

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3d87f8b9/examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
--
diff --git 
a/examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
 
b/examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
index d491741..0ad1a04 100644
--- 
a/examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
+++ 
b/examples/java8/src/main/java/org/apache/beam/examples/MinimalWordCountJava8.java
@@ -55,7 +55,7 @@ public class MinimalWordCountJava8 {
 p.apply(TextIO.Read.from("gs://dataflow-samples/shakespeare/*"))
  .apply(FlatMapElements.via((String word) -> 
Arrays.asList(word.split("[^a-zA-Z']+")))
  .withOutputType(TypeDescriptors.strings()))
- 

[04/50] [abbrv] incubator-beam git commit: Run NeedsRunner tests in Runner Core on the DirectRunner

[davor]

Run NeedsRunner tests in Runner Core on the DirectRunner

This ensures that all runner tests in runners/core-java are executed in
the standard maven build.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6ada1a63
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6ada1a63
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6ada1a63

Branch: refs/heads/python-sdk
Commit: 6ada1a635382fcddc42a7580e74e755839f7172e
Parents: d5e3dfa
Author: Thomas Groh 
Authored: Wed Jun 15 12:01:56 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 runners/core-java/pom.xml   | 13 +
 runners/direct-java/pom.xml |  1 +
 2 files changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ada1a63/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index cd8c2d1..8ede60b 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -41,6 +41,19 @@
 
   
 org.apache.maven.plugins
+maven-surefire-plugin
+
+  
+org.apache.beam.sdk.testing.NeedsRunner
+  
+  
+true
+  
+
+  
+
+  
+org.apache.maven.plugins
 maven-dependency-plugin
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6ada1a63/runners/direct-java/pom.xml
--
diff --git a/runners/direct-java/pom.xml b/runners/direct-java/pom.xml
index b2cb607..8667aee 100644
--- a/runners/direct-java/pom.xml
+++ b/runners/direct-java/pom.xml
@@ -97,6 +97,7 @@
   true
   
 org.apache.beam:beam-sdks-java-core
+org.apache.beam:beam-runners-java-core
   
   
 

[23/50] [abbrv] incubator-beam git commit: Improve BigQueryIO validation for streaming WriteDisposition

[davor]

Improve BigQueryIO validation for streaming WriteDisposition


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0a7246d2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0a7246d2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0a7246d2

Branch: refs/heads/python-sdk
Commit: 0a7246d268969cb1b7f46149e38361802c95e70a
Parents: cfa217a
Author: Scott Wegner 
Authored: Mon Jun 13 11:05:52 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:29 2016 -0700

--
 .../core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java  | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0a7246d2/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
index 38009bf..1c666ed 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/BigQueryIO.java
@@ -201,8 +201,8 @@ import javax.annotation.Nullable;
  *
  * See {@link BigQueryIO.Write} for details on how to specify if a write 
should
  * append to an existing table, replace the table, or verify that the table is
- * empty. Note that the dataset being written to must already exist. Write
- * dispositions are not supported in streaming mode.
+ * empty. Note that the dataset being written to must already exist. Unbounded 
PCollections can only
+ * be written using {@link WriteDisposition#WRITE_EMPTY} or {@link 
WriteDisposition#WRITE_APPEND}.
  *
  * Sharding BigQuery output tables
  * A common use case is to dynamically generate BigQuery table names based 
on
@@ -1674,7 +1674,7 @@ public class BigQueryIO {
   }
 }
 
-if (options.isStreaming() || tableRefFunction != null) {
+if (input.isBounded() == PCollection.IsBounded.UNBOUNDED || 
tableRefFunction != null) {
   // We will use BigQuery's streaming write API -- validate supported 
dispositions.
   checkArgument(
   createDisposition != CreateDisposition.CREATE_NEVER,

[05/50] [abbrv] incubator-beam git commit: Fix NullPointerException in AfterWatermark display data

[davor]

Fix NullPointerException in AfterWatermark display data

Window transforms register display data for the associated trigger
function by calling its .toString() method. The AfterWatermark
trigger .toString() method was not properly handling cases where
there is no late firings registered.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d5e3dfaa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d5e3dfaa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d5e3dfaa

Branch: refs/heads/python-sdk
Commit: d5e3dfaa864744ec9a011c51707d15f1ab68a734
Parents: 340fe3e
Author: Scott Wegner 
Authored: Wed Jun 15 09:51:59 2016 -0700
Committer: Davor Bonaci 
Committed: Mon Jun 20 15:14:28 2016 -0700

--
 .../sdk/transforms/windowing/AfterWatermark.java | 14 +-
 .../transforms/windowing/AfterWatermarkTest.java | 19 +++
 2 files changed, 28 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d5e3dfaa/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index e48cc44..019a68d 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -31,6 +31,8 @@ import org.joda.time.Instant;
 import java.util.List;
 import java.util.Objects;
 
+import javax.annotation.Nullable;
+
 /**
  * {@code AfterWatermark} triggers fire based on progress of the system 
watermark. This time is a
  * lower-bound, sometimes heuristically established, on event times that have 
been fully processed
@@ -106,6 +108,7 @@ public class AfterWatermark {
 private static final int LATE_INDEX = 1;
 
 private final OnceTrigger earlyTrigger;
+@Nullable
 private final OnceTrigger lateTrigger;
 
 @SuppressWarnings("unchecked")
@@ -226,7 +229,6 @@ public class AfterWatermark {
 public String toString() {
   StringBuilder builder = new StringBuilder(TO_STRING);
 
-  Trigger earlyTrigger = subTriggers.get(EARLY_INDEX);
   if (!(earlyTrigger instanceof Never.NeverTrigger)) {
 builder
 .append(".withEarlyFirings(")
@@ -234,10 +236,12 @@ public class AfterWatermark {
 .append(")");
   }
 
-  builder
-  .append(".withLateFirings(")
-  .append(subTriggers.get(LATE_INDEX))
-  .append(")");
+  if (lateTrigger != null && !(lateTrigger instanceof Never.NeverTrigger)) 
{
+builder
+.append(".withLateFirings(")
+.append(lateTrigger)
+.append(")");
+  }
 
   return builder.toString();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d5e3dfaa/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java
index be0ec1c..d692cbf 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/windowing/AfterWatermarkTest.java
@@ -345,6 +345,15 @@ public class AfterWatermarkTest {
   }
 
   @Test
+  public void testEarlyFiringsToString() {
+Trigger trigger = AfterWatermark.pastEndOfWindow()
+.withEarlyFirings(StubTrigger.named("t1"))
+.buildTrigger();
+
+assertEquals("AfterWatermark.pastEndOfWindow().withEarlyFirings(t1)", 
trigger.toString());
+  }
+
+  @Test
   public void testLateFiringsToString() {
 Trigger trigger = AfterWatermark.pastEndOfWindow()
 .withLateFirings(StubTrigger.named("t1"))
@@ -363,4 +372,14 @@ public class AfterWatermarkTest {
 
assertEquals("AfterWatermark.pastEndOfWindow().withEarlyFirings(t1).withLateFirings(t2)",
 trigger.toString());
   }
+
+  @Test
+  public void testToStringExcludesNeverTrigger() {
+Trigger trigger = AfterWatermark.pastEndOfWindow()
+.withEarlyFirings(Never.ever())
+.withLateFirings(Never.ever())
+.buildTrigger();
+
+assertEquals("AfterWatermark.pastEndOfWindow()", trigger.toString());
+  }
 }

[3/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index 06992fd..eee24a0 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -99,7 +99,7 @@
 
   
 Apache Beam Capability 
Matrix
-Last updated: 2016-06-14 18:36 
PDT
+Last updated: 2016-06-20 14:17 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf;>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 
@@ -157,19 +157,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -178,19 +182,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+~
+
 
   
   
@@ -199,19 +207,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -220,19 +232,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -241,19 +257,23 @@
 
 
 
-
+
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
   
   
@@ -262,19 +282,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+~ (https://issues.apache.org/jira/browse/BEAM-102;>BEAM-102)
+
 
 
 
-
+~
+
 
   
   
@@ -283,19 +307,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -304,19 +332,23 @@
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
   
   
@@ -325,19 +357,23 @@
 
 
 
-
+ (https://issues.apache.org/jira/browse/BEAM-25;>BEAM-25)
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -366,19 +402,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -387,19 +427,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+~
+
 
   
   
@@ -408,19 +452,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+~
+
 
   
   
@@ -429,19 +477,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -450,19 +502,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -471,19 +527,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -492,19 +552,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -533,19 +597,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -554,19 +622,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -575,19 +647,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -596,19 +672,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -617,19 +697,23 @@
 
 
 
-
+ (https://issues.apache.org/jira/browse/BEAM-101;>BEAM-101)
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -638,19 +722,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -659,19 +747,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -680,19 +772,23 @@
 
 
 
-
+ (https://issues.apache.org/jira/browse/BEAM-27;>BEAM-27)
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -721,19 +817,23 @@
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
 
 
-
+
+
 
   
   
@@ -742,19 

[2/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index fcfa7b2..8dd4a36 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -6,9 +6,9 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml; rel="self" 
type="application/rss+xml"/>
-Tue, 14 Jun 2016 18:36:16 -0700
-Tue, 14 Jun 2016 18:36:16 -0700
-Jekyll v2.5.0
+Mon, 20 Jun 2016 14:17:33 -0700
+Mon, 20 Jun 2016 14:17:33 -0700
+Jekyll v3.1.6
 
   
 The first release of Apache Beam!
@@ -59,20 +59,20 @@ to us via a 
href=/mailing_lists/user’s mailing list/a&
 
 !--more--
 
-pBefore we start, though, let’s quickly talk about the execution of 
Beam programs and how this is relevant to today’s post. A Beam pipeline can 
contain bounded and unbounded sources. If the pipeline only contains bounded 
sources it can be executed in a batch fashion, if it contains some unbounded 
sources it must be executed in a streaming fashion. When executing a Beam 
pipeline on Flink, you don’t have to choose the execution mode. Internally, 
the Flink runner either translates the pipeline to a Flink 
codeDataSet/code program or a 
codeDataStream/code program, depending on whether unbounded 
sources are used in the pipeline. In the following, when we say “Batch 
runner” what we are really talking about is the Flink runner being in batch 
execution mode./p
+pBefore we start, though, let’s quickly talk about the execution of 
Beam programs and how this is relevant to today’s post. A Beam pipeline can 
contain bounded and unbounded sources. If the pipeline only contains bounded 
sources it can be executed in a batch fashion, if it contains some unbounded 
sources it must be executed in a streaming fashion. When executing a Beam 
pipeline on Flink, you don’t have to choose the execution mode. Internally, 
the Flink runner either translates the pipeline to a Flink code 
class=highlighter-rougeDataSet/code program or a 
code class=highlighter-rougeDataStream/code 
program, depending on whether unbounded sources are used in the pipeline. In 
the following, when we say “Batch runner” what we are really talking about 
is the Flink runner being in batch execution mode./p
 
 h2 id=what-does-this-mean-for-usersWhat does this mean for 
users?/h2
 
 pSupport for windowing was the last missing puzzle piece for making 
the Flink Batch runner compatible with the Beam model. With the latest change 
to the Batch runner users can now run any pipeline that only contains bounded 
sources and be certain that the results match those of the original 
reference-implementation runners that were provided by Google as part of the 
initial code drop coming from the Google Dataflow SDK./p
 
-pThe most obvious part of the change is that windows can now be 
assigned to elements and that the runner respects these windows for the 
codeGroupByKey/code and codeCombine/code 
operations. A not-so-obvious change concerns side-inputs. In the Beam model, 
side inputs respect windows; when a value of the main input is being processed 
only the side input that corresponds to the correct window is available to the 
processing function, the codeDoFn/code./p
+pThe most obvious part of the change is that windows can now be 
assigned to elements and that the runner respects these windows for the 
code class=highlighter-rougeGroupByKey/code and 
code class=highlighter-rougeCombine/code 
operations. A not-so-obvious change concerns side-inputs. In the Beam model, 
side inputs respect windows; when a value of the main input is being processed 
only the side input that corresponds to the correct window is available to the 
processing function, the code 
class=highlighter-rougeDoFn/code./p
 
 pGetting side-input semantics right is an important milestone in 
it’s own because it allows to use a big suite of unit tests for verifying the 
correctness of a runner implementation. These tests exercise every obscure 
detail of the Beam programming model and verify that the results produced by a 
runner match what you would expect from a correct implementation. In the suite, 
side inputs are used to compare the expected result to the actual result. With 
these tests being executed regularly we can now be more confident that the 
implementation produces correct results for user-specified pipelines./p
 
 h2 id=under-the-hoodUnder the Hood/h2
-pThe basis for the changes is the introduction of 
codeWindowedValue/code in the generated Flink transformations. 
Before, a Beam codePCollectionlt;Tgt;/code would be 
transformed to a codeDataSetlt;Tgt;/code. Now, we 
instead create a 
codeDataSetlt;WindowedValuelt;Tgt;gt;/code. 
The codeWindowedValuelt;Tgt;/code stores meta data 
about the value, such as the timestamp and the windows to which it was 
assigned./p
+pThe basis for the changes is the 

[5/5] incubator-beam-site git commit: Cleanup and tests

[jamesmalone]

Cleanup and tests

This closes #23


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0d0ba1b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0d0ba1b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0d0ba1b9

Branch: refs/heads/asf-site
Commit: 0d0ba1b90bb870f7939864a4cbf3cdcdd18b670a
Parents: 2c0a36d 2a61d38
Author: James Malone 
Authored: Mon Jun 20 15:13:33 2016 -0700
Committer: James Malone 
Committed: Mon Jun 20 15:13:33 2016 -0700

--
 .gitignore  |   1 +
 Gemfile |   3 +
 Rakefile|  10 +
 _config.yml |   2 +-
 _includes/capability-matrix-row-blog.md |   2 +-
 _includes/capability-matrix-row-full.md |   2 +-
 _includes/capability-matrix-row-summary.md  |   2 +-
 _pages/blog.md  |   2 +-
 _pages/material.md  |   4 +-
 _pages/public-meetings.md   |   2 +-
 _posts/2016-02-22-beam-has-a-logo.markdown  |   4 +-
 coming-soon.md  |  13 +-
 .../2016/03/17/capability-matrix.html   | 324 ++---
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   4 +-
 .../blog/2016/05/18/splitAtFraction-method.html |   2 +-
 .../05/27/where-is-my-pcollection-dot-map.html  |  12 +-
 .../06/13/flink-batch-runner-milestone.html |   8 +-
 content/blog/index.html |  19 +-
 content/capability-matrix/index.html| 650 ---
 content/coming-soon.html|  13 +-
 content/contribution-guide/index.html   |  45 +-
 content/docs/index.html |  12 +-
 content/feed.xml| 360 ++
 content/index.html  |  13 +-
 content/material/index.html |  36 +-
 content/presentation-materials/index.html   |  10 +-
 content/public-meetings/index.html  |   4 +-
 content/releases/index.html |   7 +-
 content/source_repository/index.html|   9 +-
 docs/index.md   |  12 +-
 index.md|  13 +-
 32 files changed, 1024 insertions(+), 578 deletions(-)
--

[1/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 2c0a36d70 -> 0d0ba1b90


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/releases/index.html
--
diff --git a/content/releases/index.html b/content/releases/index.html
index f438d41..0a936cf 100644
--- a/content/releases/index.html
+++ b/content/releases/index.html
@@ -104,10 +104,10 @@
 http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22;>Maven
 Central Repository.
 
 For example, if you are developing using Maven and want to use the SDK for
-Java with the DirectRunner, add the following dependencies to your
-pom.xml file:
+Java with the DirectRunner, add the 
following dependencies to your
+pom.xml file:
 
-dependency
+dependency
   groupIdorg.apache.beam/groupId
   artifactIdbeam-sdks-java-core/artifactId
   version0.1.0-incubating/version
@@ -119,6 +119,7 @@ Java with the DirectRunner, add the following 
dependencies to your
   scoperuntime/scope
 /dependency
 
+
 
 Additionally, you may want to depend on additional SDK modules, such as IO
 connectors or other extensions, and additional runners to execute your pipeline

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/source_repository/index.html
--
diff --git a/content/source_repository/index.html 
b/content/source_repository/index.html
index 7bcf6e6..18603e9 100644
--- a/content/source_repository/index.html
+++ b/content/source_repository/index.html
@@ -110,20 +110,23 @@
 Web Browser Access
 The following is a link to a browsable version of the source repository:
 
-https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Anonymous Access
 The source can be checked out anonymously from Git with this command (See 
http://git-scm.com/docs/git-clone):
 
-$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Developer Access
 Only project developers can access the Git tree via this method (See 
http://git-scm.com/docs/git-clone).
 
-$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Access from Behind a Firewall
 Refer to the documentation of the SCM used for more information about 
access behind a firewall.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/docs/index.md
--
diff --git a/docs/index.md b/docs/index.md
index 8ca1550..e825ed3 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,13 +1,11 @@
 ---
 layout: default
 ---
-
-  
-  
-  
-  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
-  
-
+
+
+
+The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
+
 
 # Apache Beam Documentation
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/index.md
--
diff --git a/index.md b/index.md
index cee6606..db5d6f0 100644
--- a/index.md
+++ b/index.md
@@ -1,13 +1,11 @@
 ---
 layout: default
 ---
-
-  
-  
-  
-  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
-  
-
+
+
+
+The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
+
 
 # Apache Beam (incubating)
 
@@ -95,4 +93,3 @@ Interested in working with Apache Beam? Great! Here's how to 
get started:
 ## Apache Project
 Apache Beam is an [Apache Software Foundation project](http://www.apache.org),
 available under the Apache v2 license.
-

[4/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

Fixed HTML errors; added link and HTML tests via rake

Adding gemfile.lock to gitignore

Removing Gemfile.lock

Remove Rakefile from static build


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/2a61d388
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/2a61d388
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/2a61d388

Branch: refs/heads/asf-site
Commit: 2a61d388f762b02cf40071a84211a55d20821685
Parents: 2c0a36d
Author: James Malone 
Authored: Mon Jun 20 13:40:21 2016 -0700
Committer: James Malone 
Committed: Mon Jun 20 14:54:05 2016 -0700

--
 .gitignore  |   1 +
 Gemfile |   3 +
 Rakefile|  10 +
 _config.yml |   2 +-
 _includes/capability-matrix-row-blog.md |   2 +-
 _includes/capability-matrix-row-full.md |   2 +-
 _includes/capability-matrix-row-summary.md  |   2 +-
 _pages/blog.md  |   2 +-
 _pages/material.md  |   4 +-
 _pages/public-meetings.md   |   2 +-
 _posts/2016-02-22-beam-has-a-logo.markdown  |   4 +-
 coming-soon.md  |  13 +-
 .../2016/03/17/capability-matrix.html   | 324 ++---
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   4 +-
 .../blog/2016/05/18/splitAtFraction-method.html |   2 +-
 .../05/27/where-is-my-pcollection-dot-map.html  |  12 +-
 .../06/13/flink-batch-runner-milestone.html |   8 +-
 content/blog/index.html |  19 +-
 content/capability-matrix/index.html| 650 ---
 content/coming-soon.html|  13 +-
 content/contribution-guide/index.html   |  45 +-
 content/docs/index.html |  12 +-
 content/feed.xml| 360 ++
 content/index.html  |  13 +-
 content/material/index.html |  36 +-
 content/presentation-materials/index.html   |  10 +-
 content/public-meetings/index.html  |   4 +-
 content/releases/index.html |   7 +-
 content/source_repository/index.html|   9 +-
 docs/index.md   |  12 +-
 index.md|  13 +-
 32 files changed, 1024 insertions(+), 578 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/.gitignore
--
diff --git a/.gitignore b/.gitignore
index 45c1505..666de10 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,4 @@
 _site
 .sass-cache
 .jekyll-metadata
+Gemfile.lock

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/Gemfile
--
diff --git a/Gemfile b/Gemfile
new file mode 100644
index 000..bb67505
--- /dev/null
+++ b/Gemfile
@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gem 'jekyll'
+gem 'html-proofer'

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/Rakefile
--
diff --git a/Rakefile b/Rakefile
new file mode 100644
index 000..9778c98
--- /dev/null
+++ b/Rakefile
@@ -0,0 +1,10 @@
+require 'html-proofer'
+
+task :test do
+  sh "bundle exec jekyll build"
+  HTMLProofer.check_directory("./content", {
+:allow_hash_href => true,
+:check_html => true,
+:file_ignore => [/javadoc/]
+}).run
+end

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_config.yml
--
diff --git a/_config.yml b/_config.yml
index ed94c3c..697a7f8 100644
--- a/_config.yml
+++ b/_config.yml
@@ -38,7 +38,7 @@ collections:
 - beam_team
 
 # Things to ignore in the build
-exclude: ['README.md', 'Gemfile.lock', 'Gemfile']
+exclude: ['README.md', 'Gemfile.lock', 'Gemfile', 'Rakefile']
 
 # Downloads directory
 downloads: downloads

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_includes/capability-matrix-row-blog.md
--
diff --git a/_includes/capability-matrix-row-blog.md 
b/_includes/capability-matrix-row-blog.md
index bd3da68..5b43d9f 100644
--- a/_includes/capability-matrix-row-blog.md
+++ b/_includes/capability-matrix-row-blog.md
@@ -1 +1 @@
-{% if val.l1 == 'Yes' %}{% elsif val.l1 == 'Partially' 
%}~{% else %}{% endif %}
+{% if val.l1 == 

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340502#comment-15340502
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/503


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #503: [BEAM-362] Move some easy stuff into runne...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/503


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340445#comment-15340445
 ] 

ASF GitHub Bot commented on BEAM-362:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/503

[BEAM-362] Move some easy stuff into runners/core-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This change moves a set of classes with no dependents, leaving them in the 
same Java packages for now.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam runners-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #503


commit 45f47111234ee8c6dd1a89c3c4b1ce5a334b0f07
Author: Kenneth Knowles 
Date:   2016-06-20T18:54:20Z

Move some easy stuff into runners/core-java

This change moves a set of classes with no dependents,
leaving them in the same Java packages for now.




> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #503: [BEAM-362] Move some easy stuff into runne...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/503

[BEAM-362] Move some easy stuff into runners/core-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This change moves a set of classes with no dependents, leaving them in the 
same Java packages for now.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam runners-core

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #503


commit 45f47111234ee8c6dd1a89c3c4b1ce5a334b0f07
Author: Kenneth Knowles 
Date:   2016-06-20T18:54:20Z

Move some easy stuff into runners/core-java

This change moves a set of classes with no dependents,
leaving them in the same Java packages for now.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-362:


 Summary: Move shared runner functionality out of SDK and into 
runners/core-java
 Key: BEAM-362
 URL: https://issues.apache.org/jira/browse/BEAM-362
 Project: Beam
  Issue Type: Improvement
  Components: runner-core
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #500: Clone the Checkpoint Mark before resuming

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/500


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #502: Move allowsDynamicSplitting to Reader, and...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/502

Move allowsDynamicSplitting to Reader, and set it in CompressedSource.





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam compression

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/502.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #502


commit bff7dea2fa6184c6778c9bf046cbbcd080a79f2e
Author: Pei He 
Date:   2016-06-18T01:24:06Z

Move allowsDynamicSplitting to Reader, and set it in CompressedSource.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #500: Clone the Checkpoint Mark before resuming

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/500

Clone the Checkpoint Mark before resuming

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

The documentation for CheckpointMark#finalizeCheckpoint specifies that a
checkpoint instance that the reader is restarted from will have been
deserialized from the serialized form of the earlier checkpoint with the
CheckpointMarkCoder. This brings the direct runner in line with this
documentation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam unbounded_read_clone

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #500


commit 7a42205a665ccab08d9378c45b6e47b154c5150b
Author: Thomas Groh 
Date:   2016-06-20T20:51:11Z

Clone the Checkpoint Mark before resuming

The documentation for CheckpointMark#finalizeCheckpoint specifies that a
checkpoint instance that the reader is restarted from will have been
deserialized from the serialized form of the earlier checkpoint with the
CheckpointMarkCoder. This brings the direct runner in line with this
documentation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-361) Batch Pub/Sub sink does not set idLabel

[Daniel Mills (JIRA)]

Daniel Mills created BEAM-361:
-

 Summary: Batch Pub/Sub sink does not set idLabel
 Key: BEAM-361
 URL: https://issues.apache.org/jira/browse/BEAM-361
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Daniel Mills
Assignee: Daniel Halperin


The transform should generate a unique token (must be stable across retries), 
and set it on outgoing pubsub messages if an idLabel has been set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-354) Modify DatastoreIO to use Datastore v1beta3 API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340149#comment-15340149
 ] 

ASF GitHub Bot commented on BEAM-354:
-

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/499

[BEAM-354]: Modify DatastoreIO to use the v1beta3 API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Use v1beta3 datastore API and remove all v1beta2 dependencies
- Modify tests to pass. 
  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam 
vikasrk/datastoreIO-v1beta3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #499


commit af35d9829581a450245153dbfb7db23f9ed9fff0
Author: Vikas Kedigehalli 
Date:   2016-06-16T20:57:43Z

DatastoreIO: Update datastore API to v1beta3

commit 88b34f746e6332aec69cf1c60212971aae0f9fc0
Author: Vikas Kedigehalli 
Date:   2016-06-20T17:38:43Z

DatastoreIO: Update comments

commit 834b87394f5834f7c6475032cafaed89be977ecb
Author: Vikas Kedigehalli 
Date:   2016-06-20T18:39:14Z

DatastoreIO: remove unused imports




> Modify DatastoreIO to use Datastore v1beta3 API
> ---
>
> Key: BEAM-354
> URL: https://issues.apache.org/jira/browse/BEAM-354
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> Datastore v1beta2 API is getting deprecated in favor of v1beta3. Hence the 
> DatastoreIO needs to be migrated to use the new version. Also in the process 
> of doing so, this is a good time to add a level of indirection via a 
> PTranform such that future changes in Datastore API would not result in 
> changing user/pipeline code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #499: [BEAM-354]: Modify DatastoreIO to use the ...

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/incubator-beam/pull/499

[BEAM-354]: Modify DatastoreIO to use the v1beta3 API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

- Use v1beta3 datastore API and remove all v1beta2 dependencies
- Modify tests to pass. 
  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam 
vikasrk/datastoreIO-v1beta3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #499


commit af35d9829581a450245153dbfb7db23f9ed9fff0
Author: Vikas Kedigehalli 
Date:   2016-06-16T20:57:43Z

DatastoreIO: Update datastore API to v1beta3

commit 88b34f746e6332aec69cf1c60212971aae0f9fc0
Author: Vikas Kedigehalli 
Date:   2016-06-20T17:38:43Z

DatastoreIO: Update comments

commit 834b87394f5834f7c6475032cafaed89be977ecb
Author: Vikas Kedigehalli 
Date:   2016-06-20T18:39:14Z

DatastoreIO: remove unused imports




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-360) Add a framework for creating Python-SDK sources for new file types

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-360:
---

 Summary: Add a framework for creating Python-SDK sources for new 
file types
 Key: BEAM-360
 URL: https://issues.apache.org/jira/browse/BEAM-360
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


We already have a framework for creating new sources for Beam Python SDK - 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/iobase.py#L326

It would be great if we can add a framework on top of this that encapsulates 
logic common to sources that are based on files. This framework can include 
following features that are common to sources based on files.
(1) glob expansion
(2) support for new file-systems
(3) dynamic work rebalancing based on byte offsets
(4) support for reading compressed files.

Java SDK has a similar framework and it's available at - 
https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-358) JAXB Coder is not thread safe

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-358.
--
Resolution: Fixed

> JAXB Coder is not thread safe
> -
>
> Key: BEAM-358
> URL: https://issues.apache.org/jira/browse/BEAM-358
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Marshaller and Unmarshaller are cached for reuse in an instance variable. 
> These objects are not thread safe 
> (http://stackoverflow.com/questions/7400422/jaxb-creating-context-and-marshallers-cost),
>  so they should be accessed in a single-threaded manner, either via use of a 
> ThreadLocal or creating a new instance in calls to encode/decode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-358) JAXB Coder is not thread safe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340120#comment-15340120
 ] 

ASF GitHub Bot commented on BEAM-358:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/497


> JAXB Coder is not thread safe
> -
>
> Key: BEAM-358
> URL: https://issues.apache.org/jira/browse/BEAM-358
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Marshaller and Unmarshaller are cached for reuse in an instance variable. 
> These objects are not thread safe 
> (http://stackoverflow.com/questions/7400422/jaxb-creating-context-and-marshallers-cost),
>  so they should be accessed in a single-threaded manner, either via use of a 
> ThreadLocal or creating a new instance in calls to encode/decode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #497

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6f52ff940 -> 682a19c4e


Closes #497


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/682a19c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/682a19c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/682a19c4

Branch: refs/heads/master
Commit: 682a19c4e67512a912fd12c591d1ff0b73252984
Parents: 6f52ff9 071d9c7
Author: Dan Halperin 
Authored: Mon Jun 20 11:26:32 2016 -0700
Committer: Dan Halperin 
Committed: Mon Jun 20 11:26:32 2016 -0700

--
 .../org/apache/beam/sdk/coders/JAXBCoder.java   | 28 +---
 .../apache/beam/sdk/coders/JAXBCoderTest.java   | 69 +++-
 2 files changed, 84 insertions(+), 13 deletions(-)
--

[jira] [Created] (BEAM-359) AvroCoder should be able to handle anonymous classes as schemas

[Daniel Mills (JIRA)]

Daniel Mills created BEAM-359:
-

 Summary: AvroCoder should be able to handle anonymous classes as 
schemas
 Key: BEAM-359
 URL: https://issues.apache.org/jira/browse/BEAM-359
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 0.1.0-incubating, 0.2.0-incubating
Reporter: Daniel Mills
Assignee: Ben Chambers


Currently, the determinism checker NPEs with:
java.lang.IllegalArgumentException: Unable to get field id from class null
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.getField(AvroCoder.java:710)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.checkRecord(AvroCoder.java:548)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.doCheck(AvroCoder.java:477)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.recurse(AvroCoder.java:453)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.checkRecord(AvroCoder.java:567)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.doCheck(AvroCoder.java:477)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.recurse(AvroCoder.java:453)
at 
com.google.cloud.dataflow.sdk.coders.AvroCoder$AvroDeterminismChecker.check(AvroCoder.java:430)
at com.google.cloud.dataflow.sdk.coders.AvroCoder.(AvroCoder.java:189)
at com.google.cloud.dataflow.sdk.coders.AvroCoder.of(AvroCoder.java:144)
at mypackage.GenericsTest$1.create(GenericsTest.java:102)
at 
com.google.cloud.dataflow.sdk.coders.CoderRegistry.getDefaultCoderFromFactory(CoderRegistry.java:797)
at 
com.google.cloud.dataflow.sdk.coders.CoderRegistry.getDefaultCoder(CoderRegistry.java:748)
at 
com.google.cloud.dataflow.sdk.coders.CoderRegistry.getDefaultCoder(CoderRegistry.java:719)
at 
com.google.cloud.dataflow.sdk.coders.CoderRegistry.getDefaultCoder(CoderRegistry.java:696)
at 
com.google.cloud.dataflow.sdk.coders.CoderRegistry.getDefaultCoder(CoderRegistry.java:178)
at 
com.google.cloud.dataflow.sdk.values.TypedPValue.inferCoderOrFail(TypedPValue.java:147)
at 
com.google.cloud.dataflow.sdk.values.TypedPValue.getCoder(TypedPValue.java:48)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #498: Python sdk merge from master

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/498

Python sdk merge from master

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #498


commit 9e797e3d3a6bda2ff42b8c3acf8fd60fec179aa4
Author: Thomas Groh 
Date:   2016-05-20T22:18:11Z

Execute NeedsRunner tests in the Direct Runner

commit ee1297e21a481fbea52475c0732526a0441d03cb
Author: Thomas Groh 
Date:   2016-06-08T00:50:38Z

Use a DirectExecutor for Watermark Callbacks

This fixes a resource leak where the executor service is not properly
shut down with the rest of the DirectRunner.

commit 82912196080b7ae79b39e355d10077ad8654eda6
Author: Kenn Knowles 
Date:   2016-06-10T02:26:23Z

This closes #392

commit de49d032730dd21691e6e4358fdcfef249aef46f
Author: Thomas Groh 
Date:   2016-06-10T21:36:42Z

Update Pipeline Execution Style in WindowedWordCountTest

This sets the runner a Pipeline creation time rather than sending a
(potentially rewritten) pipeline to a new runner instance.

commit a8a33b19933326c28522dee530974c96d4aef0cb
Author: Thomas Groh 
Date:   2016-06-10T21:38:36Z

Update Direct Module tests to explicitly set Pipeline

commit f73bd73caa5e8222946cfc20491fd2806edd1d2b
Author: Thomas Groh 
Date:   2016-06-10T21:41:06Z

Use TestPipeline#testingPipelineOptions in IO Tests

commit f2fb59c65119d5da56df5dd4e64fa1873c6ccbbb
Author: Thomas Groh 
Date:   2016-06-10T21:43:10Z

Move GcsUtil TextIO Tests to TextIOTest

These tests are not a test of the DataflowRunner, nor any
DataflowRunner specific behavior, so they should be part of TextIOTest

commit a2dbc41b406e35794b1856796c32593653dc903c
Author: Dan Halperin 
Date:   2016-06-10T23:12:00Z

[BEAM-334] DataflowPipelineRunner: bump environment major version

commit f0e266bbd1f5781f5280703876b9fd76c7836090
Author: Jesse Anderson 
Date:   2016-05-16T20:58:08Z

Changed Word Counts to use TypeDescriptors.

commit 50c0ed04784cff0454d89fcacab24e2a4f051edb
Author: Jesse Anderson 
Date:   2016-05-16T21:09:18Z

Updated complete examples to use TypeDescriptors.

commit 60964b611638178787106d422063a2cdfe6b9902
Author: Dan Halperin 
Date:   2016-06-12T19:33:01Z

Closes #447

commit dd07cba31ef1686495562914de13f1ca24ff9ce2
Author: Kenneth Knowles 
Date:   2016-06-13T01:29:46Z

Add success/failure counters to new PAssert mechanism

commit 99c596ebcfa481c043c3f26b8085ddb4f0867ef5
Author: manuzhang 
Date:   2016-06-13T03:09:38Z

[BEAM-336] update examples-java README

commit a2abc6a249cdc4e6000d1539df6c3b5cde8d39b0
Author: Maximilian Michels 
Date:   2016-06-10T12:26:45Z

[flink] fix potential NPE in ParDoWrapper

commit be05942da0f09a247e195d1d29513ae40e1a95e0
Author: Maximilian Michels 
Date:   2016-06-13T12:57:31Z

This closes #450

commit 51030829ef53001eb7db608fd56bcf1b4bea4249
Author: Scott Wegner 
Date:   2016-06-10T19:52:43Z

Port cleanupDaemonThreads fix to archetype module

commit fe5b8db6cfefc67e8d33d4673f94611c70a0df7e
Author: Luke Cwik 
Date:   2016-06-13T16:48:09Z

[BEAM-330] Port cleanupDaemonThreads fix to archetype module

This closes #444.

commit a3feeefa2db3f759f88cacd8cb89268263ff955a
Author: Kenneth Knowles 
Date:   2016-06-13T17:11:22Z

This closes #448

commit dbddb9f3cda5b8052e5a0e5117a1d57508b229f9
Author: Kenneth Knowles 
Date:   2016-06-13T18:04:25Z

This closes #393

commit 9e0d7d650a04a63c7e9b392f938990cea90b108e
Author: Kenneth Knowles 
Date:   2016-06-13T18:16:46Z

This closes #449

commit 335202a033ced6f30f1b0e5df9da047241abc750
Author: Scott Wegner 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #687

[Apache Jenkins Server]

See