[GitHub] incubator-beam pull request #852: Correctly type collections in PipelineOpti...
GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/852 Correctly type collections in PipelineOptions. @bjchambers @lukecwik I would expect this to work, but the cast to ParameterizedType fails. Luke, were you suggesting something else? You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/fix_generic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/852.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #852 commit bbb8e5a704564b989c8a19941b610e14c685e3c5 Author: sammcveety Date: 2016-08-19T00:49:05Z Correctly type collections in PipelineOptions. commit 9dc397915d2055b8be89abb92ccb2c8c7872a294 Author: sammcveety Date: 2016-08-19T00:53:29Z Minor fix. commit 77c77f0010142c65f896aa4a431bd50ab7e45f96 Author: sammcveety Date: 2016-08-19T00:54:16Z Minor fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-568) DataflowRunner incompatible with DoFn setup and teardown
Ian Zhou created BEAM-568: - Summary: DataflowRunner incompatible with DoFn setup and teardown Key: BEAM-568 URL: https://issues.apache.org/jira/browse/BEAM-568 Project: Beam Issue Type: Bug Components: sdk-java-gcp Reporter: Ian Zhou Assignee: Daniel Halperin The DataflowRunner is currently incompatible with the DoFn setup and teradown methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-124) Testing -- End to End WordCount Batch and Streaming Tests
[ https://issues.apache.org/jira/browse/BEAM-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427375#comment-15427375 ] ASF GitHub Bot commented on BEAM-124: - GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/851 [BEAM-124] Modify example dependencies to only add runners as optional dependencies Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark Upfront setup: ``` mvn clean install -pl examples/java,examples/java8 -am ``` Dataflow: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]' ``` Flink: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.flink.TestFlinkRunner" ]' -Dit.test=WordCountIT ``` Spark: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=/tmp", "--runner=org.apache.beam.runners.spark.SparkRunner" ]' -Dit.test=WordCountIT ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam examples-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/851.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #851 commit 41273bcc40551d8bb36127ee775d2e21bf9194cd Author: Luke Cwik Date: 2016-08-18T23:35:33Z Modify example dependencies to only add runners as optional dependencies. Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark > Testing -- End to End WordCount Batch and Streaming Tests > - > > Key: BEAM-124 > URL: https://issues.apache.org/jira/browse/BEAM-124 > Project: Beam > Issue Type: New Feature > Components: testing >Reporter: Steve Wheeler >Assignee: Mark Liu > > Set up testing infrastructure so that an end to end test for WordCount (both > batch and streaming) will be run periodically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request #851: [BEAM-124] Modify example dependencies to ...
GitHub user lukecwik opened a pull request: https://github.com/apache/incubator-beam/pull/851 [BEAM-124] Modify example dependencies to only add runners as optional dependencies Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark Upfront setup: ``` mvn clean install -pl examples/java,examples/java8 -am ``` Dataflow: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]' ``` Flink: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=gs://clouddfe-testing-temp-storage", "--runner=org.apache.beam.runners.flink.TestFlinkRunner" ]' -Dit.test=WordCountIT ``` Spark: ``` mvn clean verify -pl examples/java -DskipITs=false -DintegrationTestPipelineOptions='[ "--tempRoot=/tmp", "--runner=org.apache.beam.runners.spark.SparkRunner" ]' -Dit.test=WordCountIT ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/lukecwik/incubator-beam examples-cleanup Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/851.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #851 commit 41273bcc40551d8bb36127ee775d2e21bf9194cd Author: Luke Cwik Date: 2016-08-18T23:35:33Z Modify example dependencies to only add runners as optional dependencies. Also support excluding runners by disabling the include-runners profile. Tested ITs BQ and WordCount pass on Dataflow WordCount passes on Flink and Spark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-567) FlinkRunner pollutes beam-examples-java[8] modules with SLF4J backend dependencies which are not optional (e.g. test/provided/system scoped, or marked optional)
Luke Cwik created BEAM-567: -- Summary: FlinkRunner pollutes beam-examples-java[8] modules with SLF4J backend dependencies which are not optional (e.g. test/provided/system scoped, or marked optional) Key: BEAM-567 URL: https://issues.apache.org/jira/browse/BEAM-567 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Luke Cwik Priority: Minor Exposed SLF4J backends: org.slf4j:slf4j-log4j12:jar:1.7.7:compile mvn dependency:tree -pl runners/flink/runner/ [INFO] [INFO] Building Apache Beam :: Runners :: Flink :: Core 0.3.0-incubating-SNAPSHOT [INFO] [INFO] [INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ beam-runners-flink_2.10 --- [INFO] org.apache.beam:beam-runners-flink_2.10:jar:0.3.0-incubating-SNAPSHOT [INFO] +- org.apache.flink:flink-streaming-java_2.10:jar:1.0.3:compile [INFO] | +- org.apache.flink:flink-core:jar:1.0.3:compile [INFO] | +- org.apache.flink:flink-runtime_2.10:jar:1.0.3:compile [INFO] | | +- io.netty:netty-all:jar:4.0.27.Final:compile [INFO] | | +- org.javassist:javassist:jar:3.18.2-GA:compile [INFO] | | +- org.scala-lang:scala-library:jar:2.10.4:compile [INFO] | | +- com.typesafe.akka:akka-actor_2.10:jar:2.3.7:compile [INFO] | | | \- com.typesafe:config:jar:1.2.1:compile [INFO] | | +- com.typesafe.akka:akka-remote_2.10:jar:2.3.7:compile [INFO] | | | +- io.netty:netty:jar:3.8.0.Final:compile [INFO] | | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile [INFO] | | +- com.typesafe.akka:akka-slf4j_2.10:jar:2.3.7:compile [INFO] | | +- org.clapper:grizzled-slf4j_2.10:jar:1.0.2:compile [INFO] | | +- com.github.scopt:scopt_2.10:jar:3.2.0:compile [INFO] | | +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile [INFO] | | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.0:compile [INFO] | | +- io.dropwizard.metrics:metrics-json:jar:3.1.0:compile [INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile [INFO] | | | \- jline:jline:jar:0.9.94:compile [INFO] | | \- com.twitter:chill_2.10:jar:0.7.4:compile [INFO] | | \- com.twitter:chill-java:jar:0.7.4:compile [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile [INFO] | +- org.apache.sling:org.apache.sling.commons.json:jar:2.0.6:compile [INFO] | +- org.apache.commons:commons-math3:jar:3.5:compile [INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile [INFO] | +- org.slf4j:slf4j-api:jar:1.7.14:compile [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.7:compile [INFO] | +- log4j:log4j:jar:1.2.17:compile [INFO] | \- org.apache.flink:force-shading:jar:1.0.3:compile [INFO] +- org.apache.flink:flink-java:jar:1.0.3:compile [INFO] | \- org.apache.flink:flink-shaded-hadoop2:jar:1.0.3:compile [INFO] | +- xmlenc:xmlenc:jar:0.52:compile [INFO] | +- commons-codec:commons-codec:jar:1.4:compile [INFO] | +- commons-io:commons-io:jar:2.4:compile [INFO] | +- commons-net:commons-net:jar:3.1:compile [INFO] | +- commons-collections:commons-collections:jar:3.2.1:compile [INFO] | +- javax.servlet:servlet-api:jar:2.5:compile [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:compile [INFO] | +- commons-el:commons-el:jar:1.0:runtime [INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile [INFO] | +- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile [INFO] | +- commons-lang:commons-lang:jar:2.6:compile [INFO] | +- commons-configuration:commons-configuration:jar:1.7:compile [INFO] | +- commons-digester:commons-digester:jar:1.8.1:compile [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile [INFO] | +- com.jcraft:jsch:jar:0.1.42:compile [INFO] | +- org.tukaani:xz:jar:1.0:compile [INFO] | +- commons-beanutils:commons-beanutils-bean-collections:jar:1.8.3:compile [INFO] | +- commons-daemon:commons-daemon:jar:1.0.13:compile [INFO] | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] | +- javax.xml.stream:stax-api:jar:1.0-2:compile [INFO] | +- javax.activation:activation:jar:1.1:compile [INFO] | +- com.google.inject:guice:jar:3.0:compile [INFO] | +- javax.inject:javax.inject:jar:1:compile [INFO] | \- aopalliance:aopalliance:jar:1.0:compile [INFO] +- org.apache.flink:flink-clients_2.10:jar:1.0.3:compile [INFO] | +- org.apache.flink:flink-optimizer_2.10:jar:1.0.3:compile [INFO] | \- commons-cli:commons-cli:jar:1.2:compile [INFO] +- org.apache.flink:flink-core:test-jar:tests:1.0.3:test [INFO] | +- org.apache.flink:flink-annotations:jar:1.0.3:compile [INFO] | +- com.esotericsoftware.k
[GitHub] incubator-beam pull request #850: Incorporate Private IP change to Beam
GitHub user sammcveety opened a pull request: https://github.com/apache/incubator-beam/pull/850 Incorporate Private IP change to Beam @dhalperi , please take a look --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/sammcveety/incubator-beam sgmc/pip Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/850.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #850 commit 32e72da9e4682dfed781a0e9d3ce6b03adf5e88a Author: sammcveety Date: 2016-08-18T22:02:06Z Update pom.xml commit c967ca3c4967a097d5f47635a8319594e6d95fa3 Author: sammcveety Date: 2016-08-18T22:02:52Z Update DataflowPipelineWorkerPoolOptions.java commit a9989ecc8c8d7b566c3337df7adc4afaa0e63071 Author: sammcveety Date: 2016-08-18T22:03:50Z Update DataflowPipelineTranslator.java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-beam pull request #849: [BEAM-556] Fix typo in comment
GitHub user fyellin opened a pull request: https://github.com/apache/incubator-beam/pull/849 [BEAM-556] Fix typo in comment You can merge this pull request into a Git repository by running: $ git pull https://github.com/fyellin/incubator-beam beam-556 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/849.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #849 commit 52b92029ec8953f957a4a9cc4476ba41ac9ee5e7 Author: Frank Yellin Date: 2016-08-17T06:03:56Z Fix typo in comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (BEAM-556) typo in documentation
[ https://issues.apache.org/jira/browse/BEAM-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427221#comment-15427221 ] ASF GitHub Bot commented on BEAM-556: - GitHub user fyellin opened a pull request: https://github.com/apache/incubator-beam/pull/849 [BEAM-556] Fix typo in comment You can merge this pull request into a Git repository by running: $ git pull https://github.com/fyellin/incubator-beam beam-556 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/849.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #849 commit 52b92029ec8953f957a4a9cc4476ba41ac9ee5e7 Author: Frank Yellin Date: 2016-08-17T06:03:56Z Fix typo in comment > typo in documentation > - > > Key: BEAM-556 > URL: https://issues.apache.org/jira/browse/BEAM-556 > Project: Beam > Issue Type: Bug > Components: sdk-py >Reporter: Frank Yellin >Assignee: Frank Yellin >Priority: Trivial > Original Estimate: 2m > Remaining Estimate: 2m > > transform.py: > ergument -> argument > in documentation for parse_label_and_args -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (BEAM-561) Add WindowedWordCountIT
[ https://issues.apache.org/jira/browse/BEAM-561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Liu reassigned BEAM-561: - Assignee: Mark Liu > Add WindowedWordCountIT > --- > > Key: BEAM-561 > URL: https://issues.apache.org/jira/browse/BEAM-561 > Project: Beam > Issue Type: Bug >Reporter: Jason Kuster >Assignee: Mark Liu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request #841: [Beam-556] Fix typo in documentation
Github user fyellin closed the pull request at: https://github.com/apache/incubator-beam/pull/841 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (BEAM-566) Implement proposal process
Frances Perry created BEAM-566: -- Summary: Implement proposal process Key: BEAM-566 URL: https://issues.apache.org/jira/browse/BEAM-566 Project: Beam Issue Type: Bug Components: website Reporter: Frances Perry Assignee: Frances Perry As discussed on the dev list... - Update contribution guide to explain what the design doc / proposal should include (like is done in https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) - Clearly track the open proposals (potentially in JIRA with a known label and incrementing proposal IDs). - Set expectations around the timelines for proposals -- both to ensure enough feedback is gathered and perhaps inactive proposals are archived. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request #848: Use an AtomicReference instead of Locking ...
GitHub user tgroh opened a pull request: https://github.com/apache/incubator-beam/pull/848 Use an AtomicReference instead of Locking in JAXBCoder Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/tgroh/incubator-beam jaxb_context_thread_safe Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/848.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #848 commit 888f9a850eb68799a5e9bc6b0c25a040dea626c7 Author: Thomas Groh Date: 2016-08-18T21:18:01Z Use an AtomicReference instead of Double-Locking in JAXBCoder --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (BEAM-548) Combine$PerKeyWithHotKeyFanout should propagate Display Data
[ https://issues.apache.org/jira/browse/BEAM-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427132#comment-15427132 ] ASF GitHub Bot commented on BEAM-548: - GitHub user ianzhou1 opened a pull request: https://github.com/apache/incubator-beam/pull/847 [BEAM-548] Fixed Combine display data Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/ianzhou1/incubator-beam CombinePerKeyWithHotKeyFanout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/847.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #847 commit 8018044d479cc66330e115e9ea8e533c817c0fd0 Author: Ian Zhou Date: 2016-08-18T20:50:52Z Fixed Combine display data > Combine$PerKeyWithHotKeyFanout should propagate Display Data > > > Key: BEAM-548 > URL: https://issues.apache.org/jira/browse/BEAM-548 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Ben Chambers >Assignee: Ian Zhou >Priority: Minor > > When using hot key combines, we produce two anonymous CombineFn's that wrap > the original CombineFn. These currently suppress the display data of the > delegate, instead showing up as "Combine$PerKeyWithHotKeyFanout$1". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request #847: [BEAM-548] Fixed Combine display data
GitHub user ianzhou1 opened a pull request: https://github.com/apache/incubator-beam/pull/847 [BEAM-548] Fixed Combine display data Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/ianzhou1/incubator-beam CombinePerKeyWithHotKeyFanout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/847.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #847 commit 8018044d479cc66330e115e9ea8e533c817c0fd0 Author: Ian Zhou Date: 2016-08-18T20:50:52Z Fixed Combine display data --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (BEAM-156) Implement Quiescence Signalling in the InProcessPipelineRunner
[ https://issues.apache.org/jira/browse/BEAM-156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Groh resolved BEAM-156. -- Resolution: Fixed Fix Version/s: 0.3.0-incubating The DirectRunner drives progress via quiescence. Although it does not pass an explicit flag that the pipeline has quiesced, it will only invoke root transforms if no progress can be made with the elements currently in the Pipeline. > Implement Quiescence Signalling in the InProcessPipelineRunner > -- > > Key: BEAM-156 > URL: https://issues.apache.org/jira/browse/BEAM-156 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Thomas Groh >Assignee: Thomas Groh > Fix For: 0.3.0-incubating > > > A pipeline is quiescent when the following two properties hold: > There are no triggers that can fire, given the current processing time and > watermark > All pending elements cannot make progress until a side input produces > additional output > This is approximately equivalent to: If no more input is received, the > pipeline will not perform any additional processing absent advances in > processing time or event time > See also: > https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4/edit# -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-beam pull request #846: Move native TextFileWriter to use GcsIO fo...
GitHub user charlesccychen opened a pull request: https://github.com/apache/incubator-beam/pull/846 Move native TextFileWriter to use GcsIO for writing You can merge this pull request into a Git repository by running: $ git pull https://github.com/charlesccychen/incubator-beam native-text-gcsio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/846.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #846 commit ac3a963e3666c5cad93b42f9d0894283b993fddf Author: Charles Chen Date: 2016-08-18T08:01:06Z Move native TextFileWriter to use GcsIO for writing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
