[GitHub] [beam] piotr-szuberski edited a comment on pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-10-08 Thread GitBox
piotr-szuberski edited a comment on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-706002739 Sorry, I forgot there is some additional setup needed to run it locally. You also need to build the jar of the runner you are using - example for Flink (you

[GitHub] [beam] piotr-szuberski commented on pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-10-08 Thread GitBox
piotr-szuberski commented on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-706002739 Sorry, I forgot there is an additional setup needed. You also need to build the jar of the runner you are using - example for Flink (you don't need it for DirectRun

[GitHub] [beam] codecov[bot] edited a comment on pull request #13048: [BEAM-3736] Add CombineFn.setup and CombineFn.teardown to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13048: URL: https://github.com/apache/beam/pull/13048#issuecomment-705515661 # [Codecov](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=h1) Report > Merging [#13048](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=desc) into

[GitHub] [beam] ihji commented on pull request #13025: [BEAM-10967] adding validate runner for Dataflow runner v2 to Java SDK

2020-10-08 Thread GitBox
ihji commented on pull request #13025: URL: https://github.com/apache/beam/pull/13025#issuecomment-705924279 Run Java Dataflow V2 ValidatesRunner This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] ihji commented on a change in pull request #13025: [BEAM-10967] adding validate runner for Dataflow runner v2 to Java SDK

2020-10-08 Thread GitBox
ihji commented on a change in pull request #13025: URL: https://github.com/apache/beam/pull/13025#discussion_r502132567 ## File path: .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_V2.groovy ## @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundati

[GitHub] [beam] ihji commented on a change in pull request #13025: [BEAM-10967] adding validate runner for Dataflow runner v2 to Java SDK

2020-10-08 Thread GitBox
ihji commented on a change in pull request #13025: URL: https://github.com/apache/beam/pull/13025#discussion_r502132001 ## File path: sdks/java/core/src/main/java/org/apache/beam/sdk/testing/RunnerV2Incompatible.java ## @@ -0,0 +1,24 @@ +/* + * Licensed to the Apache Software

[GitHub] [beam] ihji commented on pull request #13025: [BEAM-10967] adding validate runner for Dataflow runner v2 to Java SDK

2020-10-08 Thread GitBox
ihji commented on pull request #13025: URL: https://github.com/apache/beam/pull/13025#issuecomment-705922784 run seed job This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [beam] dhercher opened a new pull request #13055: [BEAM-11006] BigQuery failsafe function

2020-10-08 Thread GitBox
dhercher opened a new pull request #13055: URL: https://github.com/apache/beam/pull/13055 Implementing a new function to allow BigQueryIO write to have customized Failsafe handling for Streaming Writes. Adding a new withFailsafeFormatFunction() which formats the original PCollection

[GitHub] [beam] dhercher commented on pull request #13055: [BEAM-11006] BigQuery failsafe function

2020-10-08 Thread GitBox
dhercher commented on pull request #13055: URL: https://github.com/apache/beam/pull/13055#issuecomment-705920722 :R @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] codecov[bot] edited a comment on pull request #13054: [BEAM-11021] Add unsupported BundleFinalizationHandler to portable batch Flink.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13054: URL: https://github.com/apache/beam/pull/13054#issuecomment-705898005 # [Codecov](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=h1) Report > Merging [#13054](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13024: URL: https://github.com/apache/beam/pull/13024#issuecomment-705900291 # [Codecov](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=h1) Report > Merging [#13024](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13024: URL: https://github.com/apache/beam/pull/13024#issuecomment-705900291 # [Codecov](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=h1) Report > Merging [#13024](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13054: [BEAM-11021] Add unsupported BundleFinalizationHandler to portable batch Flink.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13054: URL: https://github.com/apache/beam/pull/13054#issuecomment-705898005 # [Codecov](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=h1) Report > Merging [#13054](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] commented on pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

2020-10-08 Thread GitBox
codecov[bot] commented on pull request #13024: URL: https://github.com/apache/beam/pull/13024#issuecomment-705900291 # [Codecov](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=h1) Report > Merging [#13024](https://codecov.io/gh/apache/beam/pull/13024?src=pr&el=desc) into [master

[GitHub] [beam] codecov[bot] edited a comment on pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12971: URL: https://github.com/apache/beam/pull/12971#issuecomment-701054283 # [Codecov](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=h1) Report > Merging [#12971](https://codecov.io/gh/apache/beam/pull/12971?src=pr&el=desc) into

[GitHub] [beam] boyuanzz commented on pull request #13054: [BEAM-11021] Add unsupported BundleFinalizationHandler to portable batch Flink.

2020-10-08 Thread GitBox
boyuanzz commented on pull request #13054: URL: https://github.com/apache/beam/pull/13054#issuecomment-705899062 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] codecov[bot] commented on pull request #13054: [BEAM-11021] Add unsupported BundleFinalizationHandler to portable batch Flink.

2020-10-08 Thread GitBox
codecov[bot] commented on pull request #13054: URL: https://github.com/apache/beam/pull/13054#issuecomment-705898005 # [Codecov](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=h1) Report > Merging [#13054](https://codecov.io/gh/apache/beam/pull/13054?src=pr&el=desc) into [master

[GitHub] [beam] robertwb commented on a change in pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-10-08 Thread GitBox
robertwb commented on a change in pull request #12971: URL: https://github.com/apache/beam/pull/12971#discussion_r502090932 ## File path: sdks/python/apache_beam/dataframe/transforms.py ## @@ -166,16 +173,40 @@ def expand(self, pcolls): partitioned_pcoll = next(pcoll

[GitHub] [beam] chamikaramj commented on pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

2020-10-08 Thread GitBox
chamikaramj commented on pull request #13024: URL: https://github.com/apache/beam/pull/13024#issuecomment-705897134 Refactored code to not to depend on an exception path. PTAL. This is an automated message from the Apache Git

[GitHub] [beam] codecov[bot] edited a comment on pull request #13048: [BEAM-3736] Add CombineFn.setup and CombineFn.teardown to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13048: URL: https://github.com/apache/beam/pull/13048#issuecomment-705515661 # [Codecov](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=h1) Report > Merging [#13048](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=desc) into

[GitHub] [beam] boyuanzz opened a new pull request #13054: [BEAM-11021] Add unsupported BundleFinalizationHandler to portable batch Flink.

2020-10-08 Thread GitBox
boyuanzz opened a new pull request #13054: URL: https://github.com/apache/beam/pull/13054 This PR added an unsupported BundleFinalizationHandler to portable Flink in batch mode. This will give users a more detailed error message like: ``` ERROR:root:java.lang.UnsupportedOperat

[GitHub] [beam] codecov[bot] edited a comment on pull request #13052: [BEAM-11045] Fix screendiff integration tests

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13052: URL: https://github.com/apache/beam/pull/13052#issuecomment-705867197 # [Codecov](https://codecov.io/gh/apache/beam/pull/13052?src=pr&el=h1) Report > Merging [#13052](https://codecov.io/gh/apache/beam/pull/13052?src=pr&el=desc) into

[GitHub] [beam] suztomo commented on a change in pull request #11156: [BEAM-9444] Use GCP Libraries BOM for Google Cloud Dependencies

2020-10-08 Thread GitBox
suztomo commented on a change in pull request #11156: URL: https://github.com/apache/beam/pull/11156#discussion_r502063556 ## File path: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy ## @@ -1210,6 +1222,18 @@ class BeamModulePlugin implements Plugin {

[GitHub] [beam] codecov[bot] commented on pull request #13052: [BEAM-11045] Fix screendiff integration tests

2020-10-08 Thread GitBox
codecov[bot] commented on pull request #13052: URL: https://github.com/apache/beam/pull/13052#issuecomment-705867197 # [Codecov](https://codecov.io/gh/apache/beam/pull/13052?src=pr&el=h1) Report > Merging [#13052](https://codecov.io/gh/apache/beam/pull/13052?src=pr&el=desc) into [master

[GitHub] [beam] kennknowles opened a new pull request #13053: WIP: Resolve beam_fn_api experiment for side inputs, using portable expansion always

2020-10-08 Thread GitBox
kennknowles opened a new pull request #13053: URL: https://github.com/apache/beam/pull/13053 Do not review; it doesn't work yet! - The SDK now uses the portable expansion always - Runners override to restore pre-portable expansion as needed Uploading to have an easy UI to

[GitHub] [beam] kennknowles commented on pull request #12870: [BEAM-2146] Continuously test Dataflow ValidatesRunner forcing streaming

2020-10-08 Thread GitBox
kennknowles commented on pull request #12870: URL: https://github.com/apache/beam/pull/12870#issuecomment-705858621 OK I think the path of least resistance here is probably to merge first so that the job is established and see how it runs on Jenkins. I will fix the commit history before me

[GitHub] [beam] KevinGG opened a new pull request #13052: [BEAM-11045] Fix screendiff integration tests

2020-10-08 Thread GitBox
KevinGG opened a new pull request #13052: URL: https://github.com/apache/beam/pull/13052 1. Removed the server daemon process from screen diff tests. Replaced it with a lightweight daemon thread. 2. Upgraded chromedriver-binary version to 85 to match the newest chrome browser

[GitHub] [beam] aaltay commented on pull request #13048: [BEAM-3736] Add CombineFn.setup and CombineFn.teardown to Python SDK

2020-10-08 Thread GitBox
aaltay commented on pull request #13048: URL: https://github.com/apache/beam/pull/13048#issuecomment-705855829 R: @tvalentyn @yifanmai /cc @chuanyu This is an automated message from the Apache Git Service. To respond to

[GitHub] [beam] codecov[bot] edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-698423583 # [Codecov](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=h1) Report > Merging [#12930](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=desc) into

[GitHub] [beam] iemejia commented on pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia commented on pull request #13050: URL: https://github.com/apache/beam/pull/13050#issuecomment-705839832 @lukecwik It seems there is a new test failing on the Structured Streaming runner after we merged the SDF read by default change. Not sure if it could be related, do you see any

[GitHub] [beam] iemejia edited a comment on pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia edited a comment on pull request #13050: URL: https://github.com/apache/beam/pull/13050#issuecomment-705823383 Please remember than there are two GbK tests failing on the VR suite for this runner, so this is just to validate that the now unexcluded test passes correctly.

[GitHub] [beam] iemejia merged pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia merged pull request #13050: URL: https://github.com/apache/beam/pull/13050 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] iemejia commented on pull request #13051: Enable UsesStrictTimerOrdering ValidatesRunner tests on Spark Classic Runner

2020-10-08 Thread GitBox
iemejia commented on pull request #13051: URL: https://github.com/apache/beam/pull/13051#issuecomment-705836897 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] iemejia opened a new pull request #13051: Enable UsesStrictTimerOrdering ValidatesRunner tests on Spark Classic Runner

2020-10-08 Thread GitBox
iemejia opened a new pull request #13051: URL: https://github.com/apache/beam/pull/13051 Seems we were excluding this set of tests even if they were passing for the Batch case. R: @aromanenko-dev This is an automated mes

[GitHub] [beam] boyuanzz commented on a change in pull request #13043: [BEAM-11042] Remove DirectRunner dependency on deprecated CreatePCollectionView primitive

2020-10-08 Thread GitBox
boyuanzz commented on a change in pull request #13043: URL: https://github.com/apache/beam/pull/13043#discussion_r502014361 ## File path: runners/direct-java/src/main/java/org/apache/beam/runners/direct/CreateViewNoopEvaluatorFactory.java ## @@ -0,0 +1,51 @@ +/* + * Licensed t

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-705825814 All tests have passed. I'm going to merge this PR. Thanks for all your help! This is an automated message from t

[GitHub] [beam] boyuanzz merged pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
boyuanzz merged pull request #12952: URL: https://github.com/apache/beam/pull/12952 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] iemejia commented on pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia commented on pull request #13050: URL: https://github.com/apache/beam/pull/13050#issuecomment-705823383 Please remember than there are two GbK/Combine tests failing on the VR suite for this runner, so this is just to validate that the now unexcluded test passes correctly. ---

[GitHub] [beam] iemejia commented on pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia commented on pull request #13050: URL: https://github.com/apache/beam/pull/13050#issuecomment-705822867 Run Spark StructuredStreaming ValidatesRunner This is an automated message from the Apache Git Service. To respon

[GitHub] [beam] iemejia opened a new pull request #13050: Enable SDF testPairWithIndexWindowedTimestampedBounded in Spark Structured Streaming runner

2020-10-08 Thread GitBox
iemejia opened a new pull request #13050: URL: https://github.com/apache/beam/pull/13050 This passes now after some recent changes in core. A really minor improvement after reviewing the PR on SDF Bounded reads for Spark runner R: @lukecwik

[GitHub] [beam] iemejia commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
iemejia commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-705822583 > @iemejia How the number of partitions is calculated different during the SDF initial split then what we do with SourceRDD. If I understood correctly the initial split is

[GitHub] [beam] codecov[bot] edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-698423583 # [Codecov](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=h1) Report > Merging [#12930](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=desc) into

[GitHub] [beam] iemejia edited a comment on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
iemejia edited a comment on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-705812846 Yes definitely! For the watermark part of my comment what I was expecting is that because we now need to [deal with WatermarkEstimator + ProcessContinuation](https:

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-705815486 > LGTM Excellent improvement. Hopefully we can get rid of the native translations rapidly in the future. Thanks! We can remove it for each runner once they match parity wi

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-705813659 @iemejia How the number of partitions is calculated different during the SDF initial split then what we do with SourceRDD. -

[GitHub] [beam] iemejia merged pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
iemejia merged pull request #13021: URL: https://github.com/apache/beam/pull/13021 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] iemejia commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

2020-10-08 Thread GitBox
iemejia commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-705812846 Yes definitely! For the watermark part of my comment what I was eexpecting is that because we now need to [deal with WatermarkEstimator + ProcessContinuation](https://gith

[GitHub] [beam] kennknowles merged pull request #13037: [BEAM-11031] Fix flaking in ULR ValidatesRunner suite

2020-10-08 Thread GitBox
kennknowles merged pull request #13037: URL: https://github.com/apache/beam/pull/13037 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [beam] kennknowles commented on pull request #13037: [BEAM-11031] Fix flaking in ULR ValidatesRunner suite

2020-10-08 Thread GitBox
kennknowles commented on pull request #13037: URL: https://github.com/apache/beam/pull/13037#issuecomment-705807191 Py on windows failures are unrelated and known issue. This is an automated message from the Apache Git Servic

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-10-08 Thread GitBox
TheNeuralBit commented on a change in pull request #12971: URL: https://github.com/apache/beam/pull/12971#discussion_r501273437 ## File path: sdks/python/apache_beam/dataframe/transforms.py ## @@ -321,17 +352,100 @@ def expr_to_pcoll(expr): else: return stage_to

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12971: [BEAM-10988] Partition dataframes according to size estimates.

2020-10-08 Thread GitBox
TheNeuralBit commented on a change in pull request #12971: URL: https://github.com/apache/beam/pull/12971#discussion_r501970588 ## File path: sdks/python/apache_beam/dataframe/transforms.py ## @@ -321,17 +352,100 @@ def expr_to_pcoll(expr): else: return stage_to

[GitHub] [beam] cameron-p-m opened a new pull request #13049: [BEAM-9052] Make SchemaCoder UUID deterministic

2020-10-08 Thread GitBox
cameron-p-m opened a new pull request #13049: URL: https://github.com/apache/beam/pull/13049 See issue for more details. I have made the same UUID be generated deterministically if the Schema names, typeNames, and nullability are equivalent. Discussion sent to the dev mailing

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] pabloem commented on pull request #13045: [BEAM-11039] Resolve conflicts between TFMA and Facets imports

2020-10-08 Thread GitBox
pabloem commented on pull request #13045: URL: https://github.com/apache/beam/pull/13045#issuecomment-705770305 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [beam] pabloem merged pull request #13045: [BEAM-11039] Resolve conflicts between TFMA and Facets imports

2020-10-08 Thread GitBox
pabloem merged pull request #13045: URL: https://github.com/apache/beam/pull/13045 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] ajamato commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
ajamato commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501954641 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] ajamato commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
ajamato commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501954641 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] ajamato commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
ajamato commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501954641 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] ajamato commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
ajamato commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501954641 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] ajamato commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
ajamato commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501954641 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] angoenka commented on pull request #13037: [BEAM-11031] Fix flaking in ULR ValidatesRunner suite

2020-10-08 Thread GitBox
angoenka commented on pull request #13037: URL: https://github.com/apache/beam/pull/13037#issuecomment-705768101 Thanks for the fix. LGTM This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [beam] aaltay merged pull request #12953: Minor programming guide improvements

2020-10-08 Thread GitBox
aaltay merged pull request #12953: URL: https://github.com/apache/beam/pull/12953 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] kileys commented on pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-10-08 Thread GitBox
kileys commented on pull request #12972: URL: https://github.com/apache/beam/pull/12972#issuecomment-705764275 The initial jenkins job was causing failures, but it's already fixed and checked in. Forgot to close this out Thi

[GitHub] [beam] kileys closed pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-10-08 Thread GitBox
kileys closed pull request #12972: URL: https://github.com/apache/beam/pull/12972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] lukecwik commented on a change in pull request #12984: [BEAM-10994] Add ability for HotKeyLogger to log a key

2020-10-08 Thread GitBox
lukecwik commented on a change in pull request #12984: URL: https://github.com/apache/beam/pull/12984#discussion_r501948250 ## File path: runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/HotKeyLoggerTest.java ## @@ -50,11 +50,11 @

[GitHub] [beam] pabloem commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

2020-10-08 Thread GitBox
pabloem commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-705760739 @epicfaace there's a number of failing tests. (Python Docs and Lint) - can you fix those? This is an automated m

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-705760762 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] pabloem commented on a change in pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

2020-10-08 Thread GitBox
pabloem commented on a change in pull request #13017: URL: https://github.com/apache/beam/pull/13017#discussion_r501944163 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -271,6 +276,11 @@ def __init__(self, client=None): # randomized prefix for row ID

[GitHub] [beam] aaltay commented on pull request #12389: [BEAM-10587] Support Maps in BigQuery

2020-10-08 Thread GitBox
aaltay commented on pull request #12389: URL: https://github.com/apache/beam/pull/12389#issuecomment-705755708 @rworley-monster - What is the next step on this PR? This is an automated message from the Apache Git Service. To

[GitHub] [beam] aaltay commented on pull request #12779: [BEAM-10856] Support for NestedValueProvider for Python SDK

2020-10-08 Thread GitBox
aaltay commented on pull request #12779: URL: https://github.com/apache/beam/pull/12779#issuecomment-705755411 What is the next step on this PR? This is an automated message from the Apache Git Service. To respond to the mess

[GitHub] [beam] aaltay merged pull request #12746: [BEAM-10845] Add Union support to AvroUtils

2020-10-08 Thread GitBox
aaltay merged pull request #12746: URL: https://github.com/apache/beam/pull/12746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] aaltay commented on pull request #12746: [BEAM-10845] Add Union support to AvroUtils

2020-10-08 Thread GitBox
aaltay commented on pull request #12746: URL: https://github.com/apache/beam/pull/12746#issuecomment-705754971 Merging this. Both @reuvenlax and @kanterov approved and tests are passing. This is an automated message from the

[GitHub] [beam] aaltay commented on pull request #12972: Revert "[BEAM-8024] Add JPMS E2E test"

2020-10-08 Thread GitBox
aaltay commented on pull request #12972: URL: https://github.com/apache/beam/pull/12972#issuecomment-705753842 What is reason for reverting this change? Is it causing failures? This is an automated message from the Apache Git

[GitHub] [beam] aaltay commented on pull request #12938: Integrated Nebula plugin

2020-10-08 Thread GitBox
aaltay commented on pull request #12938: URL: https://github.com/apache/beam/pull/12938#issuecomment-705752427 What is Nebula plugin? Why does Beam need this? This is an automated message from the Apache Git Service. To respo

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12705: [BEAM-10720] Finish implementing StringMethods (cat, repeat)

2020-10-08 Thread GitBox
TheNeuralBit commented on a change in pull request #12705: URL: https://github.com/apache/beam/pull/12705#discussion_r501926434 ## File path: sdks/python/apache_beam/dataframe/transforms_test.py ## @@ -266,6 +266,17 @@ def check(actual): lambda x: {'res': 3 * x}, pro

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12705: [BEAM-10720] Finish implementing StringMethods (cat, repeat)

2020-10-08 Thread GitBox
TheNeuralBit commented on a change in pull request #12705: URL: https://github.com/apache/beam/pull/12705#discussion_r501926434 ## File path: sdks/python/apache_beam/dataframe/transforms_test.py ## @@ -266,6 +266,17 @@ def check(actual): lambda x: {'res': 3 * x}, pro

[GitHub] [beam] TheNeuralBit commented on a change in pull request #12705: [BEAM-10720] Finish implementing StringMethods (cat, repeat)

2020-10-08 Thread GitBox
TheNeuralBit commented on a change in pull request #12705: URL: https://github.com/apache/beam/pull/12705#discussion_r501925345 ## File path: sdks/python/apache_beam/dataframe/transforms_test.py ## @@ -266,6 +266,18 @@ def check(actual): lambda x: {'res': 3 * x}, pro

[GitHub] [beam] codecov[bot] edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-698423583 # [Codecov](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=h1) Report > Merging [#12930](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] kennknowles commented on pull request #12568: [BEAM-9456] Upgrade to Gradle 6.6.0

2020-10-08 Thread GitBox
kennknowles commented on pull request #12568: URL: https://github.com/apache/beam/pull/12568#issuecomment-705737074 I think it makes sense to have separate tickets for deprecation warnings. If there are a few categories, maybe a few tickets. This way we can do it in small pieces. ---

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-705727264 > Compilation for Spark fails, looks like this could be related to changes in fn-execution? https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2774/ Yes. The spark

[GitHub] [beam] codecov[bot] edited a comment on pull request #12930: [BEAM-10962] Add Multiple PubSub reader to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12930: URL: https://github.com/apache/beam/pull/12930#issuecomment-698423583 # [Codecov](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=h1) Report > Merging [#12930](https://codecov.io/gh/apache/beam/pull/12930?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] aaltay commented on pull request #12953: Minor programming guide improvements

2020-10-08 Thread GitBox
aaltay commented on pull request #12953: URL: https://github.com/apache/beam/pull/12953#issuecomment-705725442 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please l

[GitHub] [beam] yuanhunglo commented on pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

2020-10-08 Thread GitBox
yuanhunglo commented on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-705720340 I tried that and received this error. I'll wait till 2.25.0 release to try again. Thanks! > _InactiveRpcError: <_InactiveRpcError of RPC that terminated with: st

[GitHub] [beam] rosetn edited a comment on pull request #12953: Minor programming guide improvements

2020-10-08 Thread GitBox
rosetn edited a comment on pull request #12953: URL: https://github.com/apache/beam/pull/12953#issuecomment-705711800 Looking closer at the file, I see that there already is alt text that I've missed. Sorry about that! On this site and flavor of markdown, you can specify the alt text direc

[GitHub] [beam] rosetn commented on pull request #12953: Minor programming guide improvements

2020-10-08 Thread GitBox
rosetn commented on pull request #12953: URL: https://github.com/apache/beam/pull/12953#issuecomment-705711800 Looking closer at the file, I see that there already is alt text that I've missed. Sorry about that! On this site and flavor of markdown, you can specify the alt text directly in

[GitHub] [beam] lukecwik merged pull request #12994: [BEAM-11044] Add support for gracefully aborting workers.

2020-10-08 Thread GitBox
lukecwik merged pull request #12994: URL: https://github.com/apache/beam/pull/12994 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lukecwik commented on a change in pull request #12994: Add support for gracefully aborting workers.

2020-10-08 Thread GitBox
lukecwik commented on a change in pull request #12994: URL: https://github.com/apache/beam/pull/12994#discussion_r501855713 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/GroupingShuffleReader.java ## @@ -402,6 +43

[GitHub] [beam] lostluck merged pull request #13028: [BEAM-8017] Plumb errors and remove panics from package graphx

2020-10-08 Thread GitBox
lostluck merged pull request #13028: URL: https://github.com/apache/beam/pull/13028 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] kennknowles commented on pull request #13043: [BEAM-11042] Remove DirectRunner dependency on deprecated CreatePCollectionView primitive

2020-10-08 Thread GitBox
kennknowles commented on pull request #13043: URL: https://github.com/apache/beam/pull/13043#issuecomment-705663096 Fixed up the issue. It was the heavily mocked out direct runner tests that needed to be adjusted. Please take a look.

[GitHub] [beam] codecov[bot] edited a comment on pull request #12955: Add Relative Path Sensitivity

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12955: URL: https://github.com/apache/beam/pull/12955#issuecomment-700102215 # [Codecov](https://codecov.io/gh/apache/beam/pull/12955?src=pr&el=h1) Report > Merging [#12955](https://codecov.io/gh/apache/beam/pull/12955?src=pr&el=desc) into

[GitHub] [beam] mxm commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
mxm commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-705630679 Compilation for Spark fails, looks like this could be related to changes in fn-execution? https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2774/ ---

[GitHub] [beam] codecov[bot] edited a comment on pull request #13048: [BEAM-3736] Add CombineFn.setup and CombineFn.teardown to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13048: URL: https://github.com/apache/beam/pull/13048#issuecomment-705515661 # [Codecov](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=h1) Report > Merging [#13048](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > Merging [#12952](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12955: Add Relative Path Sensitivity

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #12955: URL: https://github.com/apache/beam/pull/12955#issuecomment-700102215 # [Codecov](https://codecov.io/gh/apache/beam/pull/12955?src=pr&el=h1) Report > Merging [#12955](https://codecov.io/gh/apache/beam/pull/12955?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13048: [BEAM-3736] Add CombineFn.setup and CombineFn.teardown to Python SDK

2020-10-08 Thread GitBox
codecov[bot] edited a comment on pull request #13048: URL: https://github.com/apache/beam/pull/13048#issuecomment-705515661 # [Codecov](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=h1) Report > Merging [#13048](https://codecov.io/gh/apache/beam/pull/13048?src=pr&el=desc) into

[GitHub] [beam] kamilwu commented on pull request #12955: Add Relative Path Sensitivity

2020-10-08 Thread GitBox
kamilwu commented on pull request #12955: URL: https://github.com/apache/beam/pull/12955#issuecomment-705609417 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please

  1   2   >