[GitHub] [beam] codecov[bot] edited a comment on pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

codecov[bot] edited a comment on pull request #13019: URL: https://github.com/apache/beam/pull/13019#issuecomment-704487289 # [Codecov](https://codecov.io/gh/apache/beam/pull/13019?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@a27faca`). [Click

[GitHub] [beam] iemejia merged pull request #13023: [BEAM-11027] Avoid excessive logging of ZetaSQL on Nexmark run

iemejia merged pull request #13023: URL: https://github.com/apache/beam/pull/13023 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] chamikaramj commented on pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

chamikaramj commented on pull request #13024: URL: https://github.com/apache/beam/pull/13024#issuecomment-704653764 R: @pabloem This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [beam] chamikaramj opened a new pull request #13024: [BEAM-11033] Updates Dataflow Metrics processor for portable job submission

chamikaramj opened a new pull request #13024: URL: https://github.com/apache/beam/pull/13024 Currently, Dataflow metrics processor expects Dataflow internal step names generated for v1beta3 job description in metrics returned by Dataflow service. But with portable job submission, Dat

[GitHub] [beam] youngoli commented on a change in pull request #13013: [BEAM-11019] Fixing Go SDF progress reporting with multiple windows.

youngoli commented on a change in pull request #13013: URL: https://github.com/apache/beam/pull/13013#discussion_r500674070 ## File path: sdks/go/pkg/beam/core/runtime/exec/sdf_test.go ## @@ -400,6 +400,90 @@ func TestAsSplittableUnit(t *testing.T) { if err != nil {

[GitHub] [beam] youngoli commented on a change in pull request #13013: [BEAM-11019] Fixing Go SDF progress reporting with multiple windows.

youngoli commented on a change in pull request #13013: URL: https://github.com/apache/beam/pull/13013#discussion_r500671989 ## File path: sdks/go/pkg/beam/core/runtime/exec/pardo.go ## @@ -134,40 +134,40 @@ func (n *ParDo) processMainInput(mainIn *MainInput) error { //

[GitHub] [beam] pabloem commented on a change in pull request #13001: Matching job creation and error messaged for BigQuery jobs between Java

pabloem commented on a change in pull request #13001: URL: https://github.com/apache/beam/pull/13001#discussion_r500664349 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -421,8 +418,36 @@ def _insert_load_job( ), jobReference=refe

[GitHub] [beam] yuanhunglo commented on pull request #12297: [BEAM-10137] Add KinesisIO for cross-language usage with python wrapper

yuanhunglo commented on pull request #12297: URL: https://github.com/apache/beam/pull/12297#issuecomment-704581960 I'm getting this error when trying `ReadDataFromKinesis`, > RuntimeError: Unable to fetch remote job server jar at https://repo.maven.apache.org/maven2/org/apache/beam/

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] ibzib commented on a change in pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

ibzib commented on a change in pull request #13019: URL: https://github.com/apache/beam/pull/13019#discussion_r500612311 ## File path: sdks/python/apache_beam/runners/portability/spark_runner_test.py ## @@ -35,10 +35,9 @@ # Run as # -# pytest spark_runner_test.py \ -# [

[GitHub] [beam] ibzib commented on a change in pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

ibzib commented on a change in pull request #13019: URL: https://github.com/apache/beam/pull/13019#discussion_r500611605 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -56,10 +56,9 @@ # Run as # -# pytest flink_runner_test.py \ -# [

[GitHub] [beam] boyuanzz commented on a change in pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

boyuanzz commented on a change in pull request #13019: URL: https://github.com/apache/beam/pull/13019#discussion_r500610728 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -56,10 +56,9 @@ # Run as # -# pytest flink_runner_test.py \ -#

[GitHub] [beam] iemejia commented on pull request #13023: [BEAM-11027] Avoid excessive logging of ZetaSQL on Nexmark run

iemejia commented on pull request #13023: URL: https://github.com/apache/beam/pull/13023#issuecomment-704564595 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] iemejia removed a comment on pull request #13023: [BEAM-11027] Avoid excessive logging of ZetaSQL on Nexmark run

iemejia removed a comment on pull request #13023: URL: https://github.com/apache/beam/pull/13023#issuecomment-704564202 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the mes

[GitHub] [beam] iemejia commented on pull request #13023: [BEAM-11027] Avoid excessive logging of ZetaSQL on Nexmark run

iemejia commented on pull request #13023: URL: https://github.com/apache/beam/pull/13023#issuecomment-704564202 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] iemejia opened a new pull request #13023: [BEAM-11027] Avoid excessive logging of ZetaSQL on Nexmark run

iemejia opened a new pull request #13023: URL: https://github.com/apache/beam/pull/13023 Notice this while waiting for the Spark Nexmark run to finish on #13021 R: @lukecwik CC: @apilloud This is an automated mess

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] aaltay merged pull request #12892: [BEAM-10937] Add first introductory notebook

aaltay merged pull request #12892: URL: https://github.com/apache/beam/pull/12892 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] aaltay merged pull request #13020: [BEAM-11025] Make PCollection Visualization tests robust

aaltay merged pull request #13020: URL: https://github.com/apache/beam/pull/13020 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-704554139 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To re

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] robinyqiu merged pull request #13022: [BEAM-10670] Cherrypick #13015 to 2.25.0: "[BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users."

robinyqiu merged pull request #13022: URL: https://github.com/apache/beam/pull/13022 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lukecwik commented on pull request #13022: [BEAM-10670] Cherrypick #13015 to 2.25.0: "[BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users."

lukecwik commented on pull request #13022: URL: https://github.com/apache/beam/pull/13022#issuecomment-704548479 R: @robinyqiu This is an automated message from the Apache Git Service. To respond to the message, please log o

[GitHub] [beam] lukecwik opened a new pull request #13022: [BEAM-10670] Cherrypick #13015 to 2.25.0: "[BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users."

lukecwik opened a new pull request #13022: URL: https://github.com/apache/beam/pull/13022 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://b

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704542329 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704542454 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [beam] iemejia commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

iemejia commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704528075 Run Spark Runner Nexmark Tests This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704517805 Run Spark StructuredStreaming ValidatesRunner This is an automated message from the Apache Git Service. To respo

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704517720 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [beam] codecov[bot] edited a comment on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

codecov[bot] edited a comment on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-699286177 # [Codecov](https://codecov.io/gh/apache/beam/pull/12952?src=pr&el=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@82cfa29`). [Click

[GitHub] [beam] lukecwik commented on pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik commented on pull request #13021: URL: https://github.com/apache/beam/pull/13021#issuecomment-704513685 R: @iemejia Part 2 of https://github.com/apache/beam/pull/12603 Note that experimenting with watermark holds will likely take a few days to see if some of my ideas

[GitHub] [beam] lukecwik edited a comment on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

lukecwik edited a comment on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704435983 > I see, so it is the full switch from Read.Bounded/Unbounded to SDF by default. Can you get this one green so we can test it and then merge it, I would like to see if th

[GitHub] [beam] lukecwik opened a new pull request #13021: [BEAM-10670] Make Spark by default execute Read.Bounded using SplittableDoFn.

lukecwik opened a new pull request #13021: URL: https://github.com/apache/beam/pull/13021 Instructions to use the old implementation are in CHANGES.md, latest version was merged in https://github.com/apache/beam/pull/13015 Thank you for your contributi

[GitHub] [beam] lukecwik merged pull request #13015: [BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users.

lukecwik merged pull request #13015: URL: https://github.com/apache/beam/pull/13015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lukecwik commented on pull request #13015: [BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users.

lukecwik commented on pull request #13015: URL: https://github.com/apache/beam/pull/13015#issuecomment-704512052 @robinyqiu For posterity it would make sense even though this change won't impact the built artifacts. This is

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-704505768 Run Java Flink PortableValidatesRunner Streaming This is an automated message from the Apache Git Service. To re

[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704500066 Excellent idea to break the PR into little ones, first one merged. waiting for the next! This is an automated me

[GitHub] [beam] iemejia commented on pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

iemejia commented on pull request #13018: URL: https://github.com/apache/beam/pull/13018#issuecomment-704499220 Closing since already merged manually. This is an automated message from the Apache Git Service. To respond to th

[GitHub] [beam] iemejia closed pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

iemejia closed pull request #13018: URL: https://github.com/apache/beam/pull/13018 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] iemejia commented on a change in pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

iemejia commented on a change in pull request #13018: URL: https://github.com/apache/beam/pull/13018#discussion_r500538346 ## File path: runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingTransformTranslator.java ## @@ -115,6 +119,31 @@ pu

[GitHub] [beam] codecov[bot] edited a comment on pull request #12995: [BEAM-2914] Add portable merging window support to Python.

codecov[bot] edited a comment on pull request #12995: URL: https://github.com/apache/beam/pull/12995#issuecomment-703015426 # [Codecov](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=h1) Report > Merging [#12995](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12995: [BEAM-2914] Add portable merging window support to Python.

codecov[bot] edited a comment on pull request #12995: URL: https://github.com/apache/beam/pull/12995#issuecomment-703015426 # [Codecov](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=h1) Report > Merging [#12995](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

codecov[bot] edited a comment on pull request #13019: URL: https://github.com/apache/beam/pull/13019#issuecomment-704487289 # [Codecov](https://codecov.io/gh/apache/beam/pull/13019?src=pr&el=h1) Report > Merging [#13019](https://codecov.io/gh/apache/beam/pull/13019?src=pr&el=desc) into

[GitHub] [beam] KevinGG commented on pull request #13020: [BEAM-11025] Make PCollection Visualization tests robust

KevinGG commented on pull request #13020: URL: https://github.com/apache/beam/pull/13020#issuecomment-704488502 R: @ajamato This is an automated message from the Apache Git Service. To respond to the message, please log on t

[GitHub] [beam] KevinGG opened a new pull request #13020: [BEAM-11025] Make PCollection Visualization tests robust

KevinGG opened a new pull request #13020: URL: https://github.com/apache/beam/pull/13020 1. Added 2 patches to affected tests to make sure get_ipython does not return None and is_in_notebook returns True. 2. Added 1 new test to test the failure path when is_in_notebook returns

[GitHub] [beam] codecov[bot] commented on pull request #13019: [BEAM-10527] Update the test invocation cmd for portable runner

codecov[bot] commented on pull request #13019: URL: https://github.com/apache/beam/pull/13019#issuecomment-704487289 # [Codecov](https://codecov.io/gh/apache/beam/pull/13019?src=pr&el=h1) Report > Merging [#13019](https://codecov.io/gh/apache/beam/pull/13019?src=pr&el=desc) into [master

[GitHub] [beam] ibzib commented on a change in pull request #13019: [BEAM-10527] Update the test invocation cmd for poratble runner

ibzib commented on a change in pull request #13019: URL: https://github.com/apache/beam/pull/13019#discussion_r500527061 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -56,10 +56,9 @@ # Run as # -# pytest flink_runner_test.py \ -# [

[GitHub] [beam] codecov[bot] edited a comment on pull request #12995: [BEAM-2914] Add portable merging window support to Python.

codecov[bot] edited a comment on pull request #12995: URL: https://github.com/apache/beam/pull/12995#issuecomment-703015426 # [Codecov](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=h1) Report > Merging [#12995](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12995: [BEAM-2914] Add portable merging window support to Python.

codecov[bot] edited a comment on pull request #12995: URL: https://github.com/apache/beam/pull/12995#issuecomment-703015426 # [Codecov](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=h1) Report > Merging [#12995](https://codecov.io/gh/apache/beam/pull/12995?src=pr&el=desc) into

[GitHub] [beam] boyuanzz opened a new pull request #13019: Update the test invocation cmd for poratble runner

boyuanzz opened a new pull request #13019: URL: https://github.com/apache/beam/pull/13019 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quick

[GitHub] [beam] tysonjh commented on a change in pull request #12870: [BEAM-2146] Continuously test Dataflow ValidatesRunner forcing streaming

tysonjh commented on a change in pull request #12870: URL: https://github.com/apache/beam/pull/12870#discussion_r500511794 ## File path: runners/google-cloud-dataflow-java/build.gradle ## @@ -151,33 +161,34 @@ def commonExcludeCategories = [ // For the following test tasks u

[GitHub] [beam] lukecwik commented on pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

lukecwik commented on pull request #13018: URL: https://github.com/apache/beam/pull/13018#issuecomment-704459244 > Run Spark StructuredStreaming ValidatesRunner Note that spark structured streaming hasn't passed for ~26 days: https://ci-beam.apache.org/job/beam_PostCommit_Java_Valida

[GitHub] [beam] ihji commented on pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

ihji commented on pull request #13017: URL: https://github.com/apache/beam/pull/13017#issuecomment-704458418 R: @ajamato @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [beam] boyuanzz commented on pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

boyuanzz commented on pull request #12952: URL: https://github.com/apache/beam/pull/12952#issuecomment-704457703 > Thanks @boyuanzz, looks good to me. Could we now enable the excluded tests here? > > https://github.com/apache/beam/blob/b7ae7bb1dc28a7c8f26e9f48682e781a74e2d3c4/runner

[GitHub] [beam] iemejia commented on pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

iemejia commented on pull request #13018: URL: https://github.com/apache/beam/pull/13018#issuecomment-70448 Run Spark StructuredStreaming ValidatesRunner This is an automated message from the Apache Git Service. To re

[GitHub] [beam] boyuanzz commented on a change in pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

boyuanzz commented on a change in pull request #12952: URL: https://github.com/apache/beam/pull/12952#discussion_r500496949 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -427,11 +427,29 @@ def test_sql(self): class FlinkRunnerTestStre

[GitHub] [beam] lukecwik commented on pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

lukecwik commented on pull request #13018: URL: https://github.com/apache/beam/pull/13018#issuecomment-704445382 R: @iemejia This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [beam] lukecwik commented on pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

lukecwik commented on pull request #13018: URL: https://github.com/apache/beam/pull/13018#issuecomment-704445624 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [beam] lukecwik edited a comment on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

lukecwik edited a comment on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704435983 > I see, so it is the full switch from Read.Bounded/Unbounded to SDF by default. Can you get this one green so we can test it and then merge it, I would like to see if th

[GitHub] [beam] lukecwik opened a new pull request #13018: [BEAM-10670] Support impulse within non-portable Spark implementations.

lukecwik opened a new pull request #13018: URL: https://github.com/apache/beam/pull/13018 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https

[GitHub] [beam] tvalentyn merged pull request #13011: Update Dataflow Python container to beam-master-20201005

tvalentyn merged pull request #13011: URL: https://github.com/apache/beam/pull/13011 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] lukecwik commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

lukecwik commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704435983 > I see, so it is the full switch from Read.Bounded/Unbounded to SDF by default. Can you get this one green so we can test it and then merge it, I would like to see if there is

[GitHub] [beam] chadrik commented on pull request #13004: [BEAM-7746] Get mypy passing on transforms

chadrik commented on pull request #13004: URL: https://github.com/apache/beam/pull/13004#issuecomment-704434754 I think this PR is ready to go This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [beam] codecov[bot] edited a comment on pull request #13011: Update Dataflow Python container to beam-master-20201005

codecov[bot] edited a comment on pull request #13011: URL: https://github.com/apache/beam/pull/13011#issuecomment-703834488 # [Codecov](https://codecov.io/gh/apache/beam/pull/13011?src=pr&el=h1) Report > Merging [#13011](https://codecov.io/gh/apache/beam/pull/13011?src=pr&el=desc) into

[GitHub] [beam] lukecwik commented on a change in pull request #13015: [BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users.

lukecwik commented on a change in pull request #13015: URL: https://github.com/apache/beam/pull/13015#discussion_r500453246 ## File path: CHANGES.md ## @@ -80,7 +80,7 @@ ## Highlights -* Splittable DoFn is opt-out for Java based runners (Direct, Flink, Jet, Samza, Twister

[GitHub] [beam] ajamato commented on pull request #13011: Update Dataflow Python container to beam-master-20201005

ajamato commented on pull request #13011: URL: https://github.com/apache/beam/pull/13011#issuecomment-704401191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] kennknowles commented on a change in pull request #12870: [BEAM-2146] Continuously test Dataflow ValidatesRunner forcing streaming

kennknowles commented on a change in pull request #12870: URL: https://github.com/apache/beam/pull/12870#discussion_r500441921 ## File path: runners/google-cloud-dataflow-java/build.gradle ## @@ -151,33 +161,34 @@ def commonExcludeCategories = [ // For the following test tas

[GitHub] [beam] tysonjh commented on pull request #12915: [BEAM-7386] Introduce temporal inner join.

tysonjh commented on pull request #12915: URL: https://github.com/apache/beam/pull/12915#issuecomment-704405996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[GitHub] [beam] tysonjh closed pull request #12915: [BEAM-7386] Introduce temporal inner join.

tysonjh closed pull request #12915: URL: https://github.com/apache/beam/pull/12915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[GitHub] [beam] kennknowles commented on a change in pull request #13015: [BEAM-10670] Improve messaging in CHANGES.md giving clearer instructions to users.

kennknowles commented on a change in pull request #13015: URL: https://github.com/apache/beam/pull/13015#discussion_r500443627 ## File path: CHANGES.md ## @@ -80,7 +80,7 @@ ## Highlights -* Splittable DoFn is opt-out for Java based runners (Direct, Flink, Jet, Samza, Twis

[GitHub] [beam] davidcavazos commented on pull request #12892: [BEAM-10937] Add first introductory notebook

davidcavazos commented on pull request #12892: URL: https://github.com/apache/beam/pull/12892#issuecomment-704416036 @aaltay -- comments addressed, and staged link updated. This should be ready for review This is an automate

[GitHub] [beam] runningcode commented on pull request #12955: Add Relative Path Sensitivity

runningcode commented on pull request #12955: URL: https://github.com/apache/beam/pull/12955#issuecomment-704386534 @kamilwu those are good questions. 1. You can use a build scan to check if the build cache is being re-used. The build scan will tell you if a task was pulled from cac

[GitHub] [beam] iemejia commented on pull request #12603: [WIP][BEAM-10670] Make SparkRunner opt-out for using an SDF powered Read transform.

iemejia commented on pull request #12603: URL: https://github.com/apache/beam/pull/12603#issuecomment-704389742 I see, so it is the full switch from Read.Bounded/Unbounded to SDF by default. Can you get this one green so we can test it and then merge it, I would like to see if there is som

[GitHub] [beam] kamilwu commented on pull request #12955: Add Relative Path Sensitivity

kamilwu commented on pull request #12955: URL: https://github.com/apache/beam/pull/12955#issuecomment-704381394 Hello, a couple of questions from me: - How can we verify that the build cache can be shared after this change? - Is there any task left that needs relative path sensitivity?

[GitHub] [beam] chamikaramj commented on a change in pull request #13001: Matching job creation and error messaged for BigQuery jobs between Java

chamikaramj commented on a change in pull request #13001: URL: https://github.com/apache/beam/pull/13001#discussion_r500416620 ## File path: sdks/python/apache_beam/io/gcp/bigquery_tools.py ## @@ -421,8 +418,36 @@ def _insert_load_job( ), jobReference=

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider

piotr-szuberski commented on a change in pull request #12827: URL: https://github.com/apache/beam/pull/12827#discussion_r500404021 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableAvroTest.java ## @@ -0,0

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider

piotr-szuberski commented on a change in pull request #12827: URL: https://github.com/apache/beam/pull/12827#discussion_r500403269 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableTest.java ## @@ -0,0 +1,

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider

piotr-szuberski commented on a change in pull request #12827: URL: https://github.com/apache/beam/pull/12827#discussion_r500402676 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableTest.java ## @@ -0,0 +1,

[GitHub] [beam] davidcavazos commented on a change in pull request #12892: [BEAM-10937] Add first introductory notebook

davidcavazos commented on a change in pull request #12892: URL: https://github.com/apache/beam/pull/12892#discussion_r500400418 ## File path: examples/notebooks/tour-of-beam/getting-started.ipynb ## @@ -0,0 +1,1016 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata":

[GitHub] [beam] piotr-szuberski commented on a change in pull request #12827: [BEAM-10885] Add Avro support to Kafka table provider

piotr-szuberski commented on a change in pull request #12827: URL: https://github.com/apache/beam/pull/12827#discussion_r500337338 ## File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTableTest.java ## @@ -0,0 +1,

[GitHub] [beam] piotr-szuberski commented on pull request #12611: [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper

piotr-szuberski commented on pull request #12611: URL: https://github.com/apache/beam/pull/12611#issuecomment-704255878 @nielm Could you take a look at this thread? https://github.com/apache/beam/pull/12611#discussion_r480429441

[GitHub] [beam] mxm commented on a change in pull request #12952: [BEAM-11022] Support bundle finalization for portable Flink in streaming.

mxm commented on a change in pull request #12952: URL: https://github.com/apache/beam/pull/12952#discussion_r500295025 ## File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py ## @@ -427,11 +427,29 @@ def test_sql(self): class FlinkRunnerTestStreaming

[GitHub] [beam] iemejia commented on pull request #12937: [BEAM-10969] Upgrade Spark to version 2.4.7

iemejia commented on pull request #12937: URL: https://github.com/apache/beam/pull/12937#issuecomment-704275344 I wonder if we should get this one merged then given that the failing tests are unrelated to the version upgrade. Opinions? @aromanenko-dev @ibzib --

[GitHub] [beam] iemejia edited a comment on pull request #12937: [BEAM-10969] Upgrade Spark to version 2.4.7

iemejia edited a comment on pull request #12937: URL: https://github.com/apache/beam/pull/12937#issuecomment-704273168 Just for info it seems the errors on Spark Structured Streaming runner are not related to this PR, it seems the runner has its Validates Runner tests broken in master sinc

[GitHub] [beam] iemejia commented on pull request #12937: [BEAM-10969] Upgrade Spark to version 2.4.7

iemejia commented on pull request #12937: URL: https://github.com/apache/beam/pull/12937#issuecomment-704273168 Just for info it seems the errors on Spark Structured Streaming runner are not related to this PR, it seems the runner has its Validates Runner tests broken since https://ci-bea

[GitHub] [beam] epicfaace commented on pull request #11642: Replace call to .checkpoint() in SDF direct runner to .try_claim(0)

epicfaace commented on pull request #11642: URL: https://github.com/apache/beam/pull/11642#issuecomment-704219078 Still need to take a look at this This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] codecov[bot] edited a comment on pull request #12975: Update Beam's io dependencies

codecov[bot] edited a comment on pull request #12975: URL: https://github.com/apache/beam/pull/12975#issuecomment-701452806 # [Codecov](https://codecov.io/gh/apache/beam/pull/12975?src=pr&el=h1) Report > Merging [#12975](https://codecov.io/gh/apache/beam/pull/12975?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #12975: Update Beam's io dependencies

codecov[bot] edited a comment on pull request #12975: URL: https://github.com/apache/beam/pull/12975#issuecomment-701452806 # [Codecov](https://codecov.io/gh/apache/beam/pull/12975?src=pr&el=h1) Report > Merging [#12975](https://codecov.io/gh/apache/beam/pull/12975?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

codecov[bot] edited a comment on pull request #13017: URL: https://github.com/apache/beam/pull/13017#issuecomment-703976988 # [Codecov](https://codecov.io/gh/apache/beam/pull/13017?src=pr&el=h1) Report > Merging [#13017](https://codecov.io/gh/apache/beam/pull/13017?src=pr&el=desc) into

[GitHub] [beam] codecov[bot] edited a comment on pull request #13017: [BEAM-11018] Use metric for Python BigQuery streaming insert API late…

codecov[bot] edited a comment on pull request #13017: URL: https://github.com/apache/beam/pull/13017#issuecomment-703976988 # [Codecov](https://codecov.io/gh/apache/beam/pull/13017?src=pr&el=h1) Report > Merging [#13017](https://codecov.io/gh/apache/beam/pull/13017?src=pr&el=desc) into