[GitHub] [beam] tvalentyn merged pull request #13451: Update python versions in pre-requisites

2020-11-30 Thread GitBox
tvalentyn merged pull request #13451: URL: https://github.com/apache/beam/pull/13451 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on pull request #13451: Update python versions in pre-requisites

2020-11-30 Thread GitBox
tvalentyn commented on pull request #13451: URL: https://github.com/apache/beam/pull/13451#issuecomment-736248033 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [beam] amaliujia merged pull request #13319: [BEAM-11173] Add Bigtable table with read operation

2020-11-30 Thread GitBox
amaliujia merged pull request #13319: URL: https://github.com/apache/beam/pull/13319 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] amaliujia commented on pull request #13319: [BEAM-11173] Add Bigtable table with read operation

2020-11-30 Thread GitBox
amaliujia commented on pull request #13319: URL: https://github.com/apache/beam/pull/13319#issuecomment-736247818 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

[GitHub] [beam] boyuanzz commented on pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
boyuanzz commented on pull request #13431: URL: https://github.com/apache/beam/pull/13431#issuecomment-736211057 > When I looked at the programming guide, it pointed to the python docs for the details, which have been updated. I mean we may also want to provide the code example that

[GitHub] [beam] Aliraza-N commented on pull request #13450: Ignoring DicomIOReadIT

2020-11-30 Thread GitBox
Aliraza-N commented on pull request #13450: URL: https://github.com/apache/beam/pull/13450#issuecomment-736208044 Hi, sorry for the delay. @ibzib @pabloem This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] TheNeuralBit opened a new pull request #13451: Update python versions in pre-requisites

2020-11-30 Thread GitBox
TheNeuralBit opened a new pull request #13451: URL: https://github.com/apache/beam/pull/13451 R: @tvalentyn Post-Commit Tests Status (on master branch) Lang | SDK | Dataflow |

[GitHub] [beam] Aliraza-N opened a new pull request #13450: Ignoring DicomIOReadIT

2020-11-30 Thread GitBox
Aliraza-N opened a new pull request #13450: URL: https://github.com/apache/beam/pull/13450 PR https://github.com/apache/beam/pull/13137 caused Post-commit to break because of DicomIOReadIT. Ignoring the test while fixing. Thank you for your contributi

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on a change in pull request #13308: URL: https://github.com/apache/beam/pull/13308#discussion_r532898302 ## File path: start-build-env.sh ## @@ -0,0 +1,131 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13448: [BEAM-9187] Address flake in loadBalancesBundles

2020-11-30 Thread GitBox
TheNeuralBit commented on a change in pull request #13448: URL: https://github.com/apache/beam/pull/13448#discussion_r533049574 ## File path: runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java ## @@ -459,27 +461

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13448: [BEAM-9187] Address flake in loadBalancesBundles

2020-11-30 Thread GitBox
TheNeuralBit commented on a change in pull request #13448: URL: https://github.com/apache/beam/pull/13448#discussion_r533048304 ## File path: runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java ## @@ -459,27 +461

[GitHub] [beam] bntnam commented on pull request #13439: [Website revamp][11179 - 11180 - 11183] Implemented Quotes component, Header and Navbar

2020-11-30 Thread GitBox
bntnam commented on pull request #13439: URL: https://github.com/apache/beam/pull/13439#issuecomment-736185851 Hmm. I wonder why the Java check is failed. 🤔 This is an automated message from the Apache Git Service. To respon

[GitHub] [beam] Sealina opened a new pull request #13449: [BEAM-11365] Direct Runner: Handle pipeline option string quotes under windows os

2020-11-30 Thread GitBox
Sealina opened a new pull request #13449: URL: https://github.com/apache/beam/pull/13449 r: @TobKed Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s

[GitHub] [beam] apilloud commented on a change in pull request #13448: [BEAM-9187] Address flake in loadBalancesBundles

2020-11-30 Thread GitBox
apilloud commented on a change in pull request #13448: URL: https://github.com/apache/beam/pull/13448#discussion_r533024977 ## File path: runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactoryTest.java ## @@ -459,27 +461,28

[GitHub] [beam] boyuanzz merged pull request #13446: Revert "Add use_unified_worker to pipeline options."

2020-11-30 Thread GitBox
boyuanzz merged pull request #13446: URL: https://github.com/apache/beam/pull/13446 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] TheNeuralBit opened a new pull request #13448: [BEAM-9187] Address flake in loadBalancesBundles

2020-11-30 Thread GitBox
TheNeuralBit opened a new pull request #13448: URL: https://github.com/apache/beam/pull/13448 See my comment on BEAM-9187 for an explanation of what seems to have caused the flakes: > It seems that, in the error case, what's happening is as soon as the forked thread releases the sem

[GitHub] [beam] y1chi commented on pull request #13421: [BEAM-9602] Add timer family support for python SDK

2020-11-30 Thread GitBox
y1chi commented on pull request #13421: URL: https://github.com/apache/beam/pull/13421#issuecomment-736162895 R: @robertwb @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] robertwb commented on pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
robertwb commented on pull request #13431: URL: https://github.com/apache/beam/pull/13431#issuecomment-736160446 When I looked at the programming guide, it pointed to the python docs for the details, which have been updated. ---

[GitHub] [beam] robertwb commented on a change in pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
robertwb commented on a change in pull request #13431: URL: https://github.com/apache/beam/pull/13431#discussion_r533016487 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py ## @@ -510,6 +510,21 @@ def process( actual = (p | beam.

[GitHub] [beam] xinbinhuang commented on pull request #13239: Update doc/examples: BigQuerySource to ReadFromBigQuery

2020-11-30 Thread GitBox
xinbinhuang commented on pull request #13239: URL: https://github.com/apache/beam/pull/13239#issuecomment-736158679 @pabloem Kindly reminder This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] tvalentyn commented on a change in pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
tvalentyn commented on a change in pull request #13420: URL: https://github.com/apache/beam/pull/13420#discussion_r533011437 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -17,54 +17,169 @@ limitations under the License. # Container env

[GitHub] [beam] y1chi edited a comment on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
y1chi edited a comment on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736154645 > Thanks. It seems that caching may improve the startup time, and be useful for users who frequently launch the same pipeline. However I think caching may result in

[GitHub] [beam] y1chi commented on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
y1chi commented on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736154645 > Thanks. It seems that caching may improve the startup time, and be useful for users who frequently launch the same pipeline. However I think caching may result in a d

[GitHub] [beam] yifanmai commented on a change in pull request #13432: [BEAM-11355] Fix topological ordering of pipeline_from_stages after sort_stages

2020-11-30 Thread GitBox
yifanmai commented on a change in pull request #13432: URL: https://github.com/apache/beam/pull/13432#discussion_r532998837 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py ## @@ -557,8 +557,13 @@ def add_parent(child, parent):

[GitHub] [beam] yifanmai commented on a change in pull request #13432: [BEAM-11355] Fix topological ordering of pipeline_from_stages after sort_stages

2020-11-30 Thread GitBox
yifanmai commented on a change in pull request #13432: URL: https://github.com/apache/beam/pull/13432#discussion_r532998267 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/translations.py ## @@ -557,8 +557,13 @@ def add_parent(child, parent):

[GitHub] [beam] emilymye commented on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
emilymye commented on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736140858 My main worry with this is similar to Valentyn's, that Kaniko detects changes based on the Dockerfile and the majority of the time our changes to the image are not to the actual

[GitHub] [beam] y1chi commented on a change in pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
y1chi commented on a change in pull request #13399: URL: https://github.com/apache/beam/pull/13399#discussion_r532995065 ## File path: sdks/python/apache_beam/runners/portability/sdk_container_builder.py ## @@ -236,10 +235,21 @@ def invoke_docker_build_and_push(self, container

[GitHub] [beam] tvalentyn edited a comment on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
tvalentyn edited a comment on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736125575 Thanks. It seems that caching may improve the startup time, and be useful for users who frequently launch the same pipeline. However I think caching may result in a diff

[GitHub] [beam] emilymye commented on a change in pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
emilymye commented on a change in pull request #13399: URL: https://github.com/apache/beam/pull/13399#discussion_r532993637 ## File path: sdks/python/apache_beam/runners/portability/sdk_container_builder.py ## @@ -236,10 +235,21 @@ def invoke_docker_build_and_push(self, contai

[GitHub] [beam] emilymye commented on a change in pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
emilymye commented on a change in pull request #13399: URL: https://github.com/apache/beam/pull/13399#discussion_r532993637 ## File path: sdks/python/apache_beam/runners/portability/sdk_container_builder.py ## @@ -236,10 +235,21 @@ def invoke_docker_build_and_push(self, contai

[GitHub] [beam] boyuanzz commented on a change in pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
boyuanzz commented on a change in pull request #13431: URL: https://github.com/apache/beam/pull/13431#discussion_r532992606 ## File path: sdks/python/apache_beam/runners/portability/fn_api_runner/fn_runner_test.py ## @@ -510,6 +510,21 @@ def process( actual = (p | beam.

[GitHub] [beam] ibzib commented on a change in pull request #13440: [BEAM-11073] Added a line about DICOM IO in CHANGES.md

2020-11-30 Thread GitBox
ibzib commented on a change in pull request #13440: URL: https://github.com/apache/beam/pull/13440#discussion_r532992723 ## File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/DicomIOReadIT.java ## @@ -25,11 +25,7 @@ import org.ap

[GitHub] [beam] ibzib commented on pull request #13440: [BEAM-11073] Added a line about DICOM IO in CHANGES.md

2020-11-30 Thread GitBox
ibzib commented on pull request #13440: URL: https://github.com/apache/beam/pull/13440#issuecomment-736136880 > @ibzib Hi, there was an issue with an IT related to the DICOM IO connector. I've added a work around (simple ignore) to this PR. If it looks good and the checks pass, could you p

[GitHub] [beam] chamikaramj commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
chamikaramj commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532987921 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --jo

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736128843 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] lostluck merged pull request #13447: [BEAM-11200] Update useragent version of Go SDK

2020-11-30 Thread GitBox
lostluck merged pull request #13447: URL: https://github.com/apache/beam/pull/13447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] tvalentyn commented on a change in pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
tvalentyn commented on a change in pull request #13420: URL: https://github.com/apache/beam/pull/13420#discussion_r532981982 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -25,19 +25,22 @@ Prebuilt SDK container images are released per sup

[GitHub] [beam] tvalentyn commented on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
tvalentyn commented on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736125811 cc: @emilymye who may also have feedback. This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] tvalentyn commented on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
tvalentyn commented on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736125575 Thanks. It seems that caching may improve the startup time, and be useful for users who frequently launch the same pipeline. However I think caching may result in a difference

[GitHub] [beam] emilymye commented on a change in pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
emilymye commented on a change in pull request #13420: URL: https://github.com/apache/beam/pull/13420#discussion_r532961476 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -87,77 +202,28 @@ python -m apache_beam.examples.wordcount \ --outp

[GitHub] [beam] tvalentyn closed pull request #12890: [DO NOT MERGE] [BEAM-7372][BEAM-9372] Disable Py2,Py35 tasks in respective postcommit jobs.

2020-11-30 Thread GitBox
tvalentyn closed pull request #12890: URL: https://github.com/apache/beam/pull/12890 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736112246 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tvalentyn commented on pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
tvalentyn commented on pull request #13420: URL: https://github.com/apache/beam/pull/13420#issuecomment-736112118 (some of my comments got hidden under `Show Resolved` due to new push, hope you can still find them :) ) This

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736111894 Run Java_Examples_Dataflow PreCommit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736111858 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [beam] tvalentyn commented on a change in pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
tvalentyn commented on a change in pull request #13420: URL: https://github.com/apache/beam/pull/13420#discussion_r532955364 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -15,56 +15,168 @@ See the License for the specific language governi

[GitHub] [beam] lostluck commented on pull request #13447: [BEAM-11200] Update useragent version of Go SDK

2020-11-30 Thread GitBox
lostluck commented on pull request #13447: URL: https://github.com/apache/beam/pull/13447#issuecomment-736110417 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] lostluck opened a new pull request #13447: [BEAM-11200] Update useragent version of Go SDK

2020-11-30 Thread GitBox
lostluck opened a new pull request #13447: URL: https://github.com/apache/beam/pull/13447 Patching a variant of https://github.com/apache/beam/pull/13279 into the release to unbreak peak versions sooner. Remaining work on BEAM-11200 is to include a go file in the updates so this hap

[GitHub] [beam] ananvay commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
ananvay commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532961047 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --job_na

[GitHub] [beam] boyuanzz commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
boyuanzz commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532960617 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --job_n

[GitHub] [beam] boyuanzz opened a new pull request #13446: Revert "Add use_unified_worker to pipeline options."

2020-11-30 Thread GitBox
boyuanzz opened a new pull request #13446: URL: https://github.com/apache/beam/pull/13446 This reverts commit ab2e94d3 It seems like `use_runner_v2` is enough, which is as expected. r: @chamikaramj cc: @aaltay Thank you for your contr

[GitHub] [beam] kennknowles commented on pull request #12326: [BEAM-9865] Cleanup Jenkins WS on successful jobs

2020-11-30 Thread GitBox
kennknowles commented on pull request #12326: URL: https://github.com/apache/beam/pull/12326#issuecomment-736101571 Since you pointed me here, the problem is that there are important transparent caches in the workspace, such as the maven dep downloads. It is slow and wasteful for all parti

[GitHub] [beam] TheNeuralBit commented on pull request #13440: [BEAM-11073] Added a line about DICOM IO in CHANGES.md

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13440: URL: https://github.com/apache/beam/pull/13440#issuecomment-736101103 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [beam] kennknowles edited a comment on pull request #9200: [BEAM-7788] Add git scm extension to wipe workspace before cloning project

2020-11-30 Thread GitBox
kennknowles edited a comment on pull request #9200: URL: https://github.com/apache/beam/pull/9200#issuecomment-736100156 If that is true, then this change is fine. But `wipeOutWorkspace` in Jenkins context typically means the Jenkins workspace, which would include `src/..`. Hermetici

[GitHub] [beam] kennknowles commented on pull request #9200: [BEAM-7788] Add git scm extension to wipe workspace before cloning project

2020-11-30 Thread GitBox
kennknowles commented on pull request #9200: URL: https://github.com/apache/beam/pull/9200#issuecomment-736100156 If that is true, then this change is fine. But `wipeOutWorkspace` in Jenkins context typically means the Jenkins workspace, which would include `src/..`. Hermeticity is g

[GitHub] [beam] Aliraza-N commented on pull request #13440: [BEAM-11073] Added a line about DICOM IO in CHANGES.md

2020-11-30 Thread GitBox
Aliraza-N commented on pull request #13440: URL: https://github.com/apache/beam/pull/13440#issuecomment-736099622 @ibzib Hi, there was an issue with an IT related to the DICOM IO connector. I've added a work around (simple ignore) to this PR. If it looks good and the checks pass, could you

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736097418 Run Python 3.7 PostCommit This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736097325 Run Java Flink PortableValidatesRunner Batch This is an automated message from the Apache Git Service. To respon

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736097182 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to t

[GitHub] [beam] y1chi commented on pull request #13399: [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_conta…

2020-11-30 Thread GitBox
y1chi commented on pull request #13399: URL: https://github.com/apache/beam/pull/13399#issuecomment-736097097 > > Could you please add some details to the PR description about Kaniko and why are we want to use it? Thanks. > > Done. PTAL --

[GitHub] [beam] Aliraza-N commented on pull request #13137: [BEAM-11073] Dicom IO Connector for Java

2020-11-30 Thread GitBox
Aliraza-N commented on pull request #13137: URL: https://github.com/apache/beam/pull/13137#issuecomment-736096808 This PR has the change needed to ignore the test. At first glance, it appears that the test input file is not available during the run. https://github.com/apache/beam/pull/13

[GitHub] [beam] udim commented on pull request #9200: [BEAM-7788] Add git scm extension to wipe workspace before cloning project

2020-11-30 Thread GitBox
udim commented on pull request #9200: URL: https://github.com/apache/beam/pull/9200#issuecomment-736096668 To make sure we're on the same page: this change only wipes the `src/` subdirectory in the workspace. There is also another later change that wipes the whole workspace after the jo

[GitHub] [beam] tvalentyn commented on a change in pull request #13442: [BEAM-7372] Update python version support in Python quickstart

2020-11-30 Thread GitBox
tvalentyn commented on a change in pull request #13442: URL: https://github.com/apache/beam/pull/13442#discussion_r532943780 ## File path: website/www/site/content/en/get-started/quickstart-py.md ## @@ -23,7 +23,7 @@ If you're interested in contributing to the Apache Beam Pyth

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736092162 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736092101 Run Go PostCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] TheNeuralBit commented on pull request #13137: [BEAM-11073] Dicom IO Connector for Java

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13137: URL: https://github.com/apache/beam/pull/13137#issuecomment-736089962 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [beam] lostluck commented on pull request #13275: [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

2020-11-30 Thread GitBox
lostluck commented on pull request #13275: URL: https://github.com/apache/beam/pull/13275#issuecomment-736089091 Retest This Please This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736087395 Thanks @lostluck - I actually just checked out this PR locally and came here to say that I can't repro the goVet issue, it runs fine for me. I'm running a full :check now.

[GitHub] [beam] Aliraza-N edited a comment on pull request #13137: [BEAM-11073] Dicom IO Connector for Java

2020-11-30 Thread GitBox
Aliraza-N edited a comment on pull request #13137: URL: https://github.com/apache/beam/pull/13137#issuecomment-736086841 > I think this broke Java Postcommit: https://ci-beam.apache.org/job/beam_PostCommit_Java/6904/ > > ``` > java.nio.file.NoSuchFileException: src/test/resources

[GitHub] [beam] Aliraza-N commented on pull request #13137: [BEAM-11073] Dicom IO Connector for Java

2020-11-30 Thread GitBox
Aliraza-N commented on pull request #13137: URL: https://github.com/apache/beam/pull/13137#issuecomment-736086841 > I think this broke Java Postcommit: https://ci-beam.apache.org/job/beam_PostCommit_Java/6904/ > > ``` > java.nio.file.NoSuchFileException: src/test/resources/DICOM/

[GitHub] [beam] lostluck commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
lostluck commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736085775 The vet problem is the same as the linked BEAM-4831 JIRA. It's got everything to do with the way gogradle caches everything in vendor directories, and the go vet tests checking

[GitHub] [beam] ananvay commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
ananvay commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532935140 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --job_na

[GitHub] [beam] TheNeuralBit commented on pull request #13137: [BEAM-11073] Dicom IO Connector for Java

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13137: URL: https://github.com/apache/beam/pull/13137#issuecomment-736080729 I think this broke Java Postcommit: https://ci-beam.apache.org/job/beam_PostCommit_Java/6904/ ``` java.nio.file.NoSuchFileException: src/test/resources/DICOM/testDi

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736077630 > I'll rebase and squash to 1 commit on master so this can be merged cleanly. You don't need to do that unless you'd prefer to, when we merge we can have GitHub create

[GitHub] [beam] chamikaramj commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
chamikaramj commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532924602 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --jo

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736029890 I just read the dev list and what you wrote. I think committing it as it is now and improve from there is a good idea. That will make it a lot easier for the go experts to

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736022373 I created https://issues.apache.org/jira/browse/BEAM-11363 for the IT issue This is an automated message from

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736020937 > So I assume does that mean we should make that a separate issue? Yeah we can handle that separately. I would like to get to the bottom of the goVet issue in t

[GitHub] [beam] TheNeuralBit commented on pull request #13444: Create `:sdks:java:extensions:ml:integrationTest` target, run in Java PostCommit

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13444: URL: https://github.com/apache/beam/pull/13444#issuecomment-736019046 Run Spotless PreCommit This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] kennknowles commented on pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
kennknowles commented on pull request #13431: URL: https://github.com/apache/beam/pull/13431#issuecomment-736016678 Drive by comment: Java users who want to build instances of DoFns programmatically (such as DSL authors) suffer when annotations are the only way to define something. Ideally

[GitHub] [beam] TheNeuralBit merged pull request #13442: [BEAM-7372] Update python version support in Python quickstart

2020-11-30 Thread GitBox
TheNeuralBit merged pull request #13442: URL: https://github.com/apache/beam/pull/13442 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] nielsbasjes commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
nielsbasjes commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-736012865 @TheNeuralBit Looks like this code was added as part of https://issues.apache.org/jira/browse/BEAM-9147 So I assume does that mean we should make that a separate issue?

[GitHub] [beam] TheNeuralBit commented on a change in pull request #13427: [BEAM-11343] Make ExpansionServiceClient accesible outside of core-construction-java

2020-11-30 Thread GitBox
TheNeuralBit commented on a change in pull request #13427: URL: https://github.com/apache/beam/pull/13427#discussion_r532871012 ## File path: runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/External.java ## @@ -78,7 +78,7 @@ private st

[GitHub] [beam] TheNeuralBit opened a new pull request #13445: [BEAM-11343] Run spotlessApply

2020-11-30 Thread GitBox
TheNeuralBit opened a new pull request #13445: URL: https://github.com/apache/beam/pull/13445 R: @iemejia R: @ibzib Post-Commit Tests Status (on master branch) Lang | SDK |

[GitHub] [beam] TheNeuralBit edited a comment on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit edited a comment on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-735992625 The `extensions:ml` build.gradle has explicitly added *IT.java to the `:test` task: https://github.com/apache/beam/blob/40f517f9da8b2f0914f06493f8e042eeb2739578/sd

[GitHub] [beam] TheNeuralBit commented on pull request #13444: Create `:sdks:java:extensions:ml:integrationTest` target, run in Java PostCommit

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13444: URL: https://github.com/apache/beam/pull/13444#issuecomment-736009468 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, ple

[GitHub] [beam] TheNeuralBit opened a new pull request #13444: Create `:sdks:java:extensions:ml:integrationTest` target, run in Java PostCommit

2020-11-30 Thread GitBox
TheNeuralBit opened a new pull request #13444: URL: https://github.com/apache/beam/pull/13444 Instead of running IT classes in the `:test`, this PR runs them in a separate `:integrationTest` task, which is run by the Java PostCommit. Post-Commit Tests Status (on master branch) --

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-735992625 The `extensions:ml` build.gradle has explicitly added *IT.java to the `:test` task: https://github.com/apache/beam/blob/40f517f9da8b2f0914f06493f8e042eeb2739578/sdks/java

[GitHub] [beam] ibzib commented on a change in pull request #13440: [BEAM-11073] Added a line about DICOM IO in CHANGES.md

2020-11-30 Thread GitBox
ibzib commented on a change in pull request #13440: URL: https://github.com/apache/beam/pull/13440#discussion_r532838779 ## File path: CHANGES.md ## @@ -57,6 +57,7 @@ * ReadFromMongoDB can now be used with MongoDB Atlas (Python) ([BEAM-11266](https://issues.apache.org/jira/br

[GitHub] [beam] robertwb commented on pull request #13443: [BEAM-11361] Dynamic splitting of SDF IOs.

2020-11-30 Thread GitBox
robertwb commented on pull request #13443: URL: https://github.com/apache/beam/pull/13443#issuecomment-735986687 R: @boyuanzz CC: @TheNeuralBit This is an automated message from the Apache Git Service. To respond to the

[GitHub] [beam] robertwb commented on pull request #13431: [BEAM-11354] Allow DoFn itself to be used as the restriction provider.

2020-11-30 Thread GitBox
robertwb commented on pull request #13431: URL: https://github.com/apache/beam/pull/13431#issuecomment-735986540 I've updated the documentation. Java provides meaning to these specific methods by using annotations, but I still think we want a type in Python (unless we wanted to make

[GitHub] [beam] pabloem commented on pull request #13441: Handle mark expiration for blobs in azure

2020-11-30 Thread GitBox
pabloem commented on pull request #13441: URL: https://github.com/apache/beam/pull/13441#issuecomment-735979148 Run Spotless PreCommit This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [beam] pabloem commented on pull request #13441: Handle mark expiration for blobs in azure

2020-11-30 Thread GitBox
pabloem commented on pull request #13441: URL: https://github.com/apache/beam/pull/13441#issuecomment-735977260 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] pabloem merged pull request #13332: Fixes silent fail in bigtableio.py by logging ImportError

2020-11-30 Thread GitBox
pabloem merged pull request #13332: URL: https://github.com/apache/beam/pull/13332 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] robertwb opened a new pull request #13443: [BEAM-11361] Dynamic splitting of SDF IOs.

2020-11-30 Thread GitBox
robertwb opened a new pull request #13443: URL: https://github.com/apache/beam/pull/13443 For now, only json lines are supported. CSV files are more complicated. This is based on top of https://github.com/apache/beam/pull/13431 as well. Thank you

[GitHub] [beam] boyuanzz commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
boyuanzz commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532823715 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --job_n

[GitHub] [beam] boyuanzz commented on a change in pull request #13396: Add use_unified_worker to pipeline options.

2020-11-30 Thread GitBox
boyuanzz commented on a change in pull request #13396: URL: https://github.com/apache/beam/pull/13396#discussion_r532823715 ## File path: sdks/python/apache_beam/examples/kafkataxi/README.md ## @@ -185,5 +185,7 @@ python -m apache_beam.examples.kafkataxi.kafka_taxi \ --job_n

[GitHub] [beam] TheNeuralBit commented on pull request #13308: [BEAM-10891] Standardized developer build environment using Docker

2020-11-30 Thread GitBox
TheNeuralBit commented on pull request #13308: URL: https://github.com/apache/beam/pull/13308#issuecomment-735970866 @lostluck, @youngoli any advice for the goVet errors? For the GCP permissions issue, if this is an IT I don't think it should be run as part of `:check`, maybe its ge

[GitHub] [beam] pabloem commented on pull request #13395: Add FHIR Search to io/gcp/healthcare/FhirIO

2020-11-30 Thread GitBox
pabloem commented on pull request #13395: URL: https://github.com/apache/beam/pull/13395#issuecomment-735959926 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please lo

[GitHub] [beam] tysonjh commented on a change in pull request #13420: Rewrite container docs for custom containers

2020-11-30 Thread GitBox
tysonjh commented on a change in pull request #13420: URL: https://github.com/apache/beam/pull/13420#discussion_r532768803 ## File path: website/www/site/content/en/documentation/runtime/environments.md ## @@ -15,56 +15,168 @@ See the License for the specific language governing

  1   2   >