[GitHub] [beam] youngoli commented on pull request #13348: Go redundant type cleanup.

2020-11-13 Thread GitBox
youngoli commented on pull request #13348: URL: https://github.com/apache/beam/pull/13348#issuecomment-727156193 R: @lostluck (feel free to merge if there's no comments) This is an automated message from the Apache Git

[GitHub] [beam] youngoli opened a new pull request #13348: Go redundant type cleanup.

2020-11-13 Thread GitBox
youngoli opened a new pull request #13348: URL: https://github.com/apache/beam/pull/13348 Cleaning up some stylistic warnings about redundant types. No actual changes to behavior. Thank you for your contribution! Follow this checklist to help us

[GitHub] [beam] amaliujia edited a comment on pull request #13306: [BEAM-10925] Create interface for SQL Java aggregate function.

2020-11-13 Thread GitBox
amaliujia edited a comment on pull request #13306: URL: https://github.com/apache/beam/pull/13306#issuecomment-726559395 Also from potential user's requetes, there are existing CombinFn implementation and they want use those as UDAF. This is a different case from `Scalar Fn` that rewrite

[GitHub] [beam] piotr-szuberski removed a comment on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
piotr-szuberski removed a comment on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-727145037 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] piotr-szuberski commented on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-727154152 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] codecov[bot] edited a comment on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
codecov[bot] edited a comment on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-683611200 # [Codecov](https://codecov.io/gh/apache/beam/pull/12611?src=pr=h1) Report > Merging [#12611](https://codecov.io/gh/apache/beam/pull/12611?src=pr=desc) (1c43284)

[GitHub] [beam] youngoli merged pull request #13325: [BEAM-11188] Refactor Go xlang expansion from runners to core.

2020-11-13 Thread GitBox
youngoli merged pull request #13325: URL: https://github.com/apache/beam/pull/13325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] youngoli commented on a change in pull request #13325: [BEAM-11188] Refactor Go xlang expansion from runners to core.

2020-11-13 Thread GitBox
youngoli commented on a change in pull request #13325: URL: https://github.com/apache/beam/pull/13325#discussion_r523383346 ## File path: sdks/go/pkg/beam/xlang.go ## @@ -87,61 +84,16 @@ func TryCrossLanguage(s Scope, ext *graph.ExternalTransform, ins []*graph.Inboun

[GitHub] [beam] youngoli commented on a change in pull request #13325: [BEAM-11188] Refactor Go xlang expansion from runners to core.

2020-11-13 Thread GitBox
youngoli commented on a change in pull request #13325: URL: https://github.com/apache/beam/pull/13325#discussion_r523383319 ## File path: sdks/go/pkg/beam/core/runtime/xlangx/expand.go ## @@ -18,19 +18,79 @@ package xlangx import ( "context" +

[GitHub] [beam] youngoli commented on a change in pull request #13325: [BEAM-11188] Refactor Go xlang expansion from runners to core.

2020-11-13 Thread GitBox
youngoli commented on a change in pull request #13325: URL: https://github.com/apache/beam/pull/13325#discussion_r523383268 ## File path: sdks/go/pkg/beam/core/runtime/graphx/xlang.go ## @@ -16,11 +16,222 @@ package graphx import ( + "fmt" +

[GitHub] [beam] piotr-szuberski edited a comment on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski edited a comment on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-724889360 @timrobertson100 @iemejia Could I ask you to take a look at this PR? I screwed up a bit the previous PR (update Hadoop to 2.10.1 and test with 2.8.5) because I

[GitHub] [beam] piotr-szuberski removed a comment on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski removed a comment on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-724886649 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] piotr-szuberski commented on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-727145816 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] piotr-szuberski removed a comment on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski removed a comment on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-726004319 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] piotr-szuberski commented on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-727145888 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] piotr-szuberski commented on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-727145836 Run Java HadoopFormatIO Performance Test This is an automated message from the Apache Git Service. To

[GitHub] [beam] piotr-szuberski commented on pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #13287: URL: https://github.com/apache/beam/pull/13287#issuecomment-727145750 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] piotr-szuberski removed a comment on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit

2020-11-13 Thread GitBox
piotr-szuberski removed a comment on pull request #13026: URL: https://github.com/apache/beam/pull/13026#issuecomment-724694752 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] piotr-szuberski commented on pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #13026: URL: https://github.com/apache/beam/pull/13026#issuecomment-727145508 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [beam] piotr-szuberski removed a comment on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
piotr-szuberski removed a comment on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-727022314 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] piotr-szuberski commented on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
piotr-szuberski commented on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-727145037 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] codecov[bot] edited a comment on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

2020-11-13 Thread GitBox
codecov[bot] edited a comment on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-683611200 # [Codecov](https://codecov.io/gh/apache/beam/pull/12611?src=pr=h1) Report > Merging [#12611](https://codecov.io/gh/apache/beam/pull/12611?src=pr=desc) (1c43284)

[GitHub] [beam] tvalentyn merged pull request #13303: [BEAM-11196] Ensure parent of fused stages is not one of its transforms

2020-11-13 Thread GitBox
tvalentyn merged pull request #13303: URL: https://github.com/apache/beam/pull/13303 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] TheNeuralBit commented on pull request #13128: [BEAM-11265] Update quickstart-java.md

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727119983 Failure looks like flake BEAM-4136 This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13345: [BEAM-4136] Keep strong reference to loggers to avoid potential NPE

2020-11-13 Thread GitBox
TheNeuralBit commented on a change in pull request #13345: URL: https://github.com/apache/beam/pull/13345#discussion_r523336104 ## File path: sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java ## @@ -146,20 +147,25 @@ public void

[GitHub] [beam] TheNeuralBit opened a new pull request #13347: [release-2.26.0][BEAM-11264] Add Reshuffle in pd.read_*

2020-11-13 Thread GitBox
TheNeuralBit opened a new pull request #13347: URL: https://github.com/apache/beam/pull/13347 Another fix for using DataframeTransform at scale. Without this it's not possible to distribute reading distinct files on distinct workers. Post-Commit Tests Status (on master branch)

[GitHub] [beam] TheNeuralBit commented on pull request #13347: [release-2.26.0][BEAM-11264] Add Reshuffle in pd.read_*

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13347: URL: https://github.com/apache/beam/pull/13347#issuecomment-727117498 R: @robertwb R: @lostluck This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] TheNeuralBit merged pull request #13346: [BEAM-11264] Add Reshuffle in pd.read_*

2020-11-13 Thread GitBox
TheNeuralBit merged pull request #13346: URL: https://github.com/apache/beam/pull/13346 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] yifanmai commented on pull request #13303: [BEAM-11196] Ensure parent of fused stages is not one of its transforms

2020-11-13 Thread GitBox
yifanmai commented on pull request #13303: URL: https://github.com/apache/beam/pull/13303#issuecomment-727109859 Checks look good now; windows-latest, 3.7, py37 failure looks unrelated: `PermissionError: [WinError 32] The process cannot access the file because it is being used by another

[GitHub] [beam] TheNeuralBit commented on pull request #13128: [BEAM-11265] Update quickstart-java.md

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727109764 Unfortunately since we're just after the 2.26.0 release cut we won't be able to make the website change until 2.27.0 is out (probably early 2021). Filed BEAM-11265 to keep

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
TheNeuralBit commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523326593 ## File path: website/www/site/content/en/get-started/quickstart-java.md ## @@ -266,7 +266,7 @@ For Unix shells (Instructions currently only

[GitHub] [beam] veblush commented on pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6

2020-11-13 Thread GitBox
veblush commented on pull request #13311: URL: https://github.com/apache/beam/pull/13311#issuecomment-727108530 @kennknowles I found that issue when running the wordcount example on dataflow with `--experiments=use_grpc_for_gcs` option.

[GitHub] [beam] davidwrede edited a comment on pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede edited a comment on pull request #13317: URL: https://github.com/apache/beam/pull/13317#issuecomment-727102822 @robertwb I implemented some of your changes, but unfortunately, someone else will have to address the rest. I'm happy to help merge in the PR as is, or hand it

[GitHub] [beam] TheNeuralBit commented on pull request #13340: [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13340: URL: https://github.com/apache/beam/pull/13340#issuecomment-727107604 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] pcoet commented on pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
pcoet commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727106066 Okay, I think we should be good to go. I left the sample in, but removed the header and excluded it. After the next Beam release, I'll update the docs so that the inputFile points

[GitHub] [beam] aaltay commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
aaltay commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523307893 ## File path: sdks/java/maven-archetypes/examples/sample.txt ## @@ -0,0 +1,117 @@ +#Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] TheNeuralBit opened a new pull request #13346: Add Reshuffle in pd.read_*

2020-11-13 Thread GitBox
TheNeuralBit opened a new pull request #13346: URL: https://github.com/apache/beam/pull/13346 Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow | Flink | Samza |

[GitHub] [beam] pcoet commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
pcoet commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523304147 ## File path: sdks/java/maven-archetypes/examples/sample.txt ## @@ -0,0 +1,117 @@ +#Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] davidwrede commented on pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on pull request #13317: URL: https://github.com/apache/beam/pull/13317#issuecomment-727102822 @robertwb I implemented some of your changes, but ran out of time to get to the rest. Someone else will have to address them.

[GitHub] [beam] robertwb merged pull request #13343: Allow null next index for last report.

2020-11-13 Thread GitBox
robertwb merged pull request #13343: URL: https://github.com/apache/beam/pull/13343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523302254 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523302017 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523301916 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] aaltay commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
aaltay commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523301546 ## File path: sdks/java/maven-archetypes/examples/sample.txt ## @@ -0,0 +1,117 @@ +#Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] aaltay commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
aaltay commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523301546 ## File path: sdks/java/maven-archetypes/examples/sample.txt ## @@ -0,0 +1,117 @@ +#Licensed to the Apache Software Foundation (ASF) under one or more

[GitHub] [beam] pcoet commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
pcoet commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523301162 ## File path: website/www/site/content/en/get-started/quickstart-java.md ## @@ -283,7 +283,7 @@ We are working on adding the instruction for this runner!

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523300408 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523299584 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523299391 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523299244 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] apilloud commented on pull request #13340: [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest

2020-11-13 Thread GitBox
apilloud commented on pull request #13340: URL: https://github.com/apache/beam/pull/13340#issuecomment-727098604 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523298936 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,280 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
TheNeuralBit commented on a change in pull request #13128: URL: https://github.com/apache/beam/pull/13128#discussion_r523294633 ## File path: website/www/site/content/en/get-started/quickstart-java.md ## @@ -283,7 +283,7 @@ We are working on adding the instruction for this

[GitHub] [beam] TheNeuralBit commented on pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727093417 Looks like the flake is already filed as BEAM-7827 This is an automated message from the Apache Git

[GitHub] [beam] tysonjh commented on a change in pull request #13170: [BEAM-9650] Adding support for ReadAll from BigQuery transform

2020-11-13 Thread GitBox
tysonjh commented on a change in pull request #13170: URL: https://github.com/apache/beam/pull/13170#discussion_r523292482 ## File path: sdks/python/apache_beam/io/gcp/bigquery.py ## @@ -79,6 +79,41 @@ `ReadFromBigQuery`, you can use the flag `use_json_exports` to export

[GitHub] [beam] hostirosti edited a comment on pull request #4080: [BEAM-2500] Add S3 FileSystem to Java SDK

2020-11-13 Thread GitBox
hostirosti edited a comment on pull request #4080: URL: https://github.com/apache/beam/pull/4080#issuecomment-727086784 Solution seems to be to useĀ  ```Java options.setAwsRegion(); FileSystems.setDefaultPipelineOptions(options); ``` at runtime before using

[GitHub] [beam] hostirosti commented on pull request #4080: [BEAM-2500] Add S3 FileSystem to Java SDK

2020-11-13 Thread GitBox
hostirosti commented on pull request #4080: URL: https://github.com/apache/beam/pull/4080#issuecomment-727086784 Solution seems to be to useĀ  `FileSystems.setDefaultPipelineOptions(options);` at runtime before using FileSystems.create(..)

[GitHub] [beam] TheNeuralBit opened a new pull request #13345: [BEAM-4136] Keep strong reference to loggers to avoid potential NPE

2020-11-13 Thread GitBox
TheNeuralBit opened a new pull request #13345: URL: https://github.com/apache/beam/pull/13345 I haven't found evidence that this is the cause but I strongly suspect the cause of the NPE on this line:

[GitHub] [beam] rohdesamuel commented on pull request #13215: [BEAM-11151] Adds the ToStringFnRunner to Java

2020-11-13 Thread GitBox
rohdesamuel commented on pull request #13215: URL: https://github.com/apache/beam/pull/13215#issuecomment-727082892 R: @robertwb This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] rohdesamuel commented on pull request #13262: [BEAM-11151] Adds the ToStringFn Bundle Processor to Python

2020-11-13 Thread GitBox
rohdesamuel commented on pull request #13262: URL: https://github.com/apache/beam/pull/13262#issuecomment-727082809 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] tysonjh edited a comment on pull request #13259: Fix Java ValidatesRunner V2 task dependency.

2020-11-13 Thread GitBox
tysonjh edited a comment on pull request #13259: URL: https://github.com/apache/beam/pull/13259#issuecomment-727082642 Failed due to https://issues.apache.org/jira/browse/BEAM-11263 for which I've made a fix in https://github.com/apache/beam/pull/13344. All other tests pass. I'm not

[GitHub] [beam] tysonjh commented on pull request #13259: Fix Java ValidatesRunner V2 task dependency.

2020-11-13 Thread GitBox
tysonjh commented on pull request #13259: URL: https://github.com/apache/beam/pull/13259#issuecomment-727082642 Failed due to https://issues.apache.org/jira/browse/BEAM-11263 for which I've made a fix in https://github.com/apache/beam/pull/13344. All other tests pass. I'm not going

[GitHub] [beam] tysonjh commented on pull request #13344: [BEAM-11263] Java cleanUpDockerImages now force removes container images.

2020-11-13 Thread GitBox
tysonjh commented on pull request #13344: URL: https://github.com/apache/beam/pull/13344#issuecomment-727082068 R: @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] tysonjh opened a new pull request #13344: [BEAM-11263] Java cleanUpDockerImages now force removes containers.

2020-11-13 Thread GitBox
tysonjh opened a new pull request #13344: URL: https://github.com/apache/beam/pull/13344 Jenkins jobs were failing, rarely, when attempting to delete the container image because the container was not removed first. This is likely due to a race between shutting down the container and

[GitHub] [beam] aaltay commented on pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
aaltay commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727081597 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] aaltay commented on pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
aaltay commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727081551 Java precommit - I think this is a flaky test : "org.apache.beam.sdk.metrics.MetricsTest$AttemptedMetricTests.testAllAttemptedMetrics". @tysonjh @kennknowles - could we deflake

[GitHub] [beam] pcoet commented on pull request #13128: Update quickstart-java.md

2020-11-13 Thread GitBox
pcoet commented on pull request #13128: URL: https://github.com/apache/beam/pull/13128#issuecomment-727080918 Shakespeare is licensed! I'm not sure why "Run Java PreCommit" is failing... This is an automated message from the

[GitHub] [beam] kennknowles commented on pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6

2020-11-13 Thread GitBox
kennknowles commented on pull request #13311: URL: https://github.com/apache/beam/pull/13311#issuecomment-727080429 Is there a way to reproduce the NPE in a Beam test? This is an automated message from the Apache Git

[GitHub] [beam] kennknowles commented on pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6

2020-11-13 Thread GitBox
kennknowles commented on pull request #13311: URL: https://github.com/apache/beam/pull/13311#issuecomment-727080341 run java postcommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] amaliujia commented on pull request #13339: [BEAM-11192] Add use_runner_v2 to experiments.

2020-11-13 Thread GitBox
amaliujia commented on pull request #13339: URL: https://github.com/apache/beam/pull/13339#issuecomment-727079850 thanks for this change! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] robertwb commented on pull request #13343: Allow null next index for last report.

2020-11-13 Thread GitBox
robertwb commented on pull request #13343: URL: https://github.com/apache/beam/pull/13343#issuecomment-727078781 R: @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] robertwb opened a new pull request #13343: Allow null next index for last report.

2020-11-13 Thread GitBox
robertwb opened a new pull request #13343: URL: https://github.com/apache/beam/pull/13343 These should be OK; avoid spamming the logs. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[GitHub] [beam] boyuanzz merged pull request #13339: [BEAM-11192] Add use_runner_v2 to experiments.

2020-11-13 Thread GitBox
boyuanzz merged pull request #13339: URL: https://github.com/apache/beam/pull/13339 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz commented on pull request #13339: [BEAM-11192] Add use_runner_v2 to experiments.

2020-11-13 Thread GitBox
boyuanzz commented on pull request #13339: URL: https://github.com/apache/beam/pull/13339#issuecomment-727071689 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] kennknowles merged pull request #13315: [BEAM-11228] Add guava to dependency management in generated pom

2020-11-13 Thread GitBox
kennknowles merged pull request #13315: URL: https://github.com/apache/beam/pull/13315 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] kennknowles commented on pull request #13315: [BEAM-11228] Add guava to dependency management in generated pom

2020-11-13 Thread GitBox
kennknowles commented on pull request #13315: URL: https://github.com/apache/beam/pull/13315#issuecomment-727070635 > Run Gradle Publish I did not know that was an option! This is an automated message from the Apache

[GitHub] [beam] TheNeuralBit commented on pull request #13340: [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest

2020-11-13 Thread GitBox
TheNeuralBit commented on pull request #13340: URL: https://github.com/apache/beam/pull/13340#issuecomment-727069687 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] kennknowles commented on a change in pull request #13342: [BEAM-11260][BEAM-11261] Remove inappropriate assumptions about repo from linkage check script

2020-11-13 Thread GitBox
kennknowles commented on a change in pull request #13342: URL: https://github.com/apache/beam/pull/13342#discussion_r523262786 ## File path: sdks/java/build-tools/beam-linkage-check.sh ## @@ -36,46 +36,60 @@ set -o pipefail set -e # These default artifacts are common

[GitHub] [beam] kennknowles commented on pull request #13342: [BEAM-11260][BEAM-11261] Remove inappropriate assumptions about repo from linkage check script

2020-11-13 Thread GitBox
kennknowles commented on pull request #13342: URL: https://github.com/apache/beam/pull/13342#issuecomment-727062832 R: @suztomo Especially curious if you see the question about having a bash for loop vs doing the linkage check on a list of artifacts. Do we need both?

[GitHub] [beam] tvalentyn closed pull request #13334: [DO NOT MERGE] This is a clone of #13316 [BEAM-9980] do not hardcode Python version for cross language tests and make it switchable.

2020-11-13 Thread GitBox
tvalentyn closed pull request #13334: URL: https://github.com/apache/beam/pull/13334 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] tvalentyn merged pull request #13316: [BEAM-9980] do not hardcode Python version for cross language tests and make it switchable.

2020-11-13 Thread GitBox
tvalentyn merged pull request #13316: URL: https://github.com/apache/beam/pull/13316 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] kennknowles commented on pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6

2020-11-13 Thread GitBox
kennknowles commented on pull request #13311: URL: https://github.com/apache/beam/pull/13311#issuecomment-727060995 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] boyuanzz merged pull request #13313: [BEAM-10480] Update SDF related documentation.

2020-11-13 Thread GitBox
boyuanzz merged pull request #13313: URL: https://github.com/apache/beam/pull/13313 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] boyuanzz commented on pull request #13313: [BEAM-10480] Update SDF related documentation.

2020-11-13 Thread GitBox
boyuanzz commented on pull request #13313: URL: https://github.com/apache/beam/pull/13313#issuecomment-727060117 Thanks for your help! This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] kennknowles commented on pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6

2020-11-13 Thread GitBox
kennknowles commented on pull request #13311: URL: https://github.com/apache/beam/pull/13311#issuecomment-727060096 I fixed it in https://github.com/apache/beam/pull/13342. With those changes I could run: /bin/bash sdks/java/build-tools/beam-linkage-check.sh github/master

[GitHub] [beam] udim commented on pull request #13315: [BEAM-11228] Add guava to dependency management in generated pom

2020-11-13 Thread GitBox
udim commented on pull request #13315: URL: https://github.com/apache/beam/pull/13315#issuecomment-727059667 https://ci-beam.apache.org/job/beam_PostRelease_NightlySnapshot/1175/ is successful This is an automated message

[GitHub] [beam] TheNeuralBit merged pull request #13003: [BEAM-10989] Add @SchemaCaseFormat annotation

2020-11-13 Thread GitBox
TheNeuralBit merged pull request #13003: URL: https://github.com/apache/beam/pull/13003 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523256644 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523255051 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523254273 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523252451 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] kennknowles commented on a change in pull request #13342: [BEAM-11260][BEAM-11261] Remove inappropriate assumptions about repo from linkage check script

2020-11-13 Thread GitBox
kennknowles commented on a change in pull request #13342: URL: https://github.com/apache/beam/pull/13342#discussion_r523252500 ## File path: sdks/java/build-tools/beam-linkage-check.sh ## @@ -36,46 +36,60 @@ set -o pipefail set -e # These default artifacts are common

[GitHub] [beam] kennknowles opened a new pull request #13342: [BEAM-11260][BEAM-11261] Remove inappropriate assumptions about repo from linkage check script

2020-11-13 Thread GitBox
kennknowles opened a new pull request #13342: URL: https://github.com/apache/beam/pull/13342 This updates the beam linkage check script to be simpler to use and also removes incorrect assumptions. Now you just pass the refs you want to compare and that is that. A few minor fixes, too.

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523250127 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] tvalentyn commented on a change in pull request #13303: [BEAM-11196] Ensure parent of fused stages is not one of its transforms

2020-11-13 Thread GitBox
tvalentyn commented on a change in pull request #13303: URL: https://github.com/apache/beam/pull/13303#discussion_r523250343 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py ## @@ -1212,6 +1213,22 @@ def get_ancestors(name): return

[GitHub] [beam] rohdesamuel commented on pull request #13335: [BEAM-10921]: Fix BEAM-10921 and underlying issues

2020-11-13 Thread GitBox
rohdesamuel commented on pull request #13335: URL: https://github.com/apache/beam/pull/13335#issuecomment-727051791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523249222 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] amaliujia commented on a change in pull request #13319: [BEAM-11173] Add Bigtable table with read operation

2020-11-13 Thread GitBox
amaliujia commented on a change in pull request #13319: URL: https://github.com/apache/beam/pull/13319#discussion_r523246921 ## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableRowToBeamRow.java ## @@ -0,0

[GitHub] [beam] davidwrede commented on a change in pull request #13317: [BEAM-11254] New documentation on multi-language pipelines

2020-11-13 Thread GitBox
davidwrede commented on a change in pull request #13317: URL: https://github.com/apache/beam/pull/13317#discussion_r523248814 ## File path: website/www/site/content/en/documentation/programming-guide.md ## @@ -5431,4 +5431,282 @@ use case. {{< highlight py >}} {{<

[GitHub] [beam] tvalentyn commented on pull request #13303: [BEAM-11196] Ensure parent of fused stages is not one of its transforms

2020-11-13 Thread GitBox
tvalentyn commented on pull request #13303: URL: https://github.com/apache/beam/pull/13303#issuecomment-727049909 > Will defer to your judgement. Discussed offline, we can fix-forward and CP. Adverse behavior (if any) should happen at pipeline launch, so this should not cause silent

  1   2   >