[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261297#comment-15261297
 ] 

Luke Cwik commented on BEAM-151:


Note there are a few more objects that could move over such as CloudObject but 
they are so tightly integrated with Coders and other Beam primitives that at 
this point there is not much that is feasible to move. Closing this as done, a 
future refactor/clean-up issue can be opened as the goals of this issue have 
been met.

> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261292#comment-15261292
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/243


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240134#comment-15240134
 ] 

Luke Cwik commented on BEAM-151:


No, I still need to update package-info.java and make another pass over the 
remaining classes to see if there are any stragglers.

> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15240107#comment-15240107
 ] 

Davor Bonaci commented on BEAM-151:
---

Done?

> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238553#comment-15238553
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/173


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238494#comment-15238494
 ] 

ASF GitHub Bot commented on BEAM-151:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/173

[BEAM-151] Add support for RunnableOnService validation tests to runners 
module

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Default implementation for Dataflow requires you to specify the 
runnableOnServicePipelineOptions
system property with a valid project/runner/... to be able to execute on 
Google Cloud Dataflow.

Default implementation for Flink and Spark are to have it disabled.

Note the whitespace formatting fix on spark pom.xml

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #173


commit 60d5471f74df1da5979dc31a6a7ee11c6fe1a850
Author: Luke Cwik 
Date:   2016-04-13T00:49:07Z

[BEAM-151] Add support for RunnableOnService validation tests to runners 
package

Default implementation for Flink and Spark are to have it disabled.
Default implementation for Dataflow requires you to specify the 
runnableOnServicePipelineOptions
system property with a valid project/runner/... to be able to execute on 
Google Cloud Dataflow.

commit 86e61c5f91f02340b42e3608b0b949f6384ae3d9
Author: Luke Cwik 
Date:   2016-04-13T02:50:49Z

[BEAM-151] !fixup Add the runnable-on-service-tests execution

commit c1b4f3350b830f726e16cfe6db0d6b337022
Author: Luke Cwik 
Date:   2016-04-13T02:54:45Z

[BEAM-151] !fixup Fix comment in TestPipeline




> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237780#comment-15237780
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user lukecwik closed the pull request at:

https://github.com/apache/incubator-beam/pull/170


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236435#comment-15236435
 ] 

ASF GitHub Bot commented on BEAM-151:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/170

[BEAM-151] Enable integration tests underneath google-cloud-dataflow-java

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Added surefire.dependencyToScan and surefire.testPipelineOptions 
configuration options

surefire.dependencyToScan adds an additional dependency used to find tests.
This allows for java-sdk-all to be added to all runners for running
the @Category(RunnableOnService.class) tests.

surefire.testPipelineOptions configures the pipeline options used while
constructing instances of a TestPipeline. A json list of arguments
is expected, for example:
[ "--runner=MyRunner", "--myRunnerOption=myRunnerValue" ]

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #170


commit ec90f511c3e3308ceaa06a34666c2b654b7ef59d
Author: Luke Cwik 
Date:   2016-04-12T01:58:00Z

[BEAM-151] Enable integration tests underneath google-cloud-dataflow-java

Added surefire.dependencyToScan and surefire.testPipelineOptions 
configuration options

surefire.dependencyToScan adds an additional dependency used to find tests.
This allows for java-sdk-all to be added to all runners for running
the @Category(RunnableOnService.class) tests.

surefire.testPipelineOptions configures the pipeline options used while
constructing instances of a TestPipeline. A json list of arguments
is expected, for example:
[ "--runner=MyRunner", "--myRunnerOption=myRunnerValue" ]




> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15236210#comment-15236210
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/165


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15235857#comment-15235857
 ] 

ASF GitHub Bot commented on BEAM-151:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/165

[BEAM-151] Move Dataflow runner to new Dataflow runner maven module

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Note that Flink runner and Java examples both depend on the Dataflow runner 
still. So I needed to make the Dataflow runner a top level component to satisfy 
these dependencies.

Also, we are currently shading within the Dataflow runner and using the 
same namespace to reduce the jar size.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #165


commit 0c45173b206b97825b6a9c24d095efba110eb405
Author: Luke Cwik 
Date:   2016-04-11T19:49:07Z

[BEAM-151] Move a large portion of the Dataflow runner to separate maven 
module

Note that Flink runner and Java examples both depend on the Dataflow runner
still. So I needed to make the Dataflow runner a top level component to
satisfy these dependencies.

commit 31ca6c214e0e90a5acd0cfc895b01a97b20a165c
Author: Luke Cwik 
Date:   2016-04-11T19:55:53Z

[BEAM-151] !fixup Add comment explaining build order change




> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15234418#comment-15234418
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/158


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233285#comment-15233285
 ] 

ASF GitHub Bot commented on BEAM-151:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/158

[BEAM-151] Move maven archetypes build order to be after runners

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace "" in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
sdks/java/maven-archtypes has several dependencies on the 
DataflowPipelineRunner. Until these are refactored out or a released artifact 
exists, we need to modify the build order.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam dataflow-runner-5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/158.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #158


commit ba4cdc5f3f753d6a3fd26f6b6d4e2f9c3d8f7e70
Author: Luke Cwik 
Date:   2016-04-09T01:37:46Z

[BEAM-151] Move maven archetypes build order to be after runners

Currently sdks/java/maven-archetypes has several dependencies
on the DataflowPipelineRunner. This is mainly throw the utility
class called DataflowExampleUtils.

This is a temporary workaround till examples get updated
to use only beam SDK.




> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15233272#comment-15233272
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/154


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15230764#comment-15230764
 ] 

ASF GitHub Bot commented on BEAM-151:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/87


> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-151) Create Dataflow Runner Package

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15212523#comment-15212523
 ] 

ASF GitHub Bot commented on BEAM-151:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/78

[BEAM-151] Clean up reference to DataflowWorkerHarnessOptions

This prevents moving DataflowWorkerHarnessOptions to Dataflow Runner maven
module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam dataflow-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/78.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #78


commit e8abd3f93aa0c37b7a96712276310c6c8d7b5381
Author: Luke Cwik 
Date:   2016-03-25T22:28:43Z

[BEAM-151] Clean up reference to DataflowWorkerHarnessOptions

This prevents moving DataflowWorkerHarnessOptions to Dataflow Runner maven
module.




> Create Dataflow Runner Package
> --
>
> Key: BEAM-151
> URL: https://issues.apache.org/jira/browse/BEAM-151
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Move Dataflow runner out of SDK core and into new Dataflow runner maven 
> module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)