Re: [PR] Automatically enable Dataflow Runner v2 for pipeliens that use the transform upgrade feature [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30061: URL: https://github.com/apache/beam/pull/30061#issuecomment-1901729855 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

[PR] Automatically enable Dataflow Runner v2 for pipeliens that use the transform upgrade feature [beam]

2024-01-19 Thread via GitHub
chamikaramj opened a new pull request, #30061: URL: https://github.com/apache/beam/pull/30061 PR Also includes following two updates. * Add an annotation to upgraded transforms so that they can be easily identified by looking at the pipeline proto. * Prevent Dataflow from performin

Re: [PR] [YAML] add required parameters and required windowing to providers [beam]

2024-01-19 Thread via GitHub
Polber commented on PR #30044: URL: https://github.com/apache/beam/pull/30044#issuecomment-1901563961 > I'm not sure how much validation we should try to build into the wrappers vs. let the application of the transforms themselves do validation (as that seems to lend itself to an unbounded

Re: [PR] [YAML] Require numStreams for unbounded BigQueryStorageWriteApiSchemaTransform xlang transform [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30059: URL: https://github.com/apache/beam/pull/30059#issuecomment-1901563629 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [YAML] Require numStreams for unbounded BigQueryStorageWriteApiSchemaTransform xlang transform [beam]

2024-01-19 Thread via GitHub
Polber commented on PR #30059: URL: https://github.com/apache/beam/pull/30059#issuecomment-1901561520 R: @robertwb What about this instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

[PR] [YAML] Require numStreams for unbounded BigQueryStorageWriteApiSchemaTransform xlang transform [beam]

2024-01-19 Thread via GitHub
Polber opened a new pull request, #30059: URL: https://github.com/apache/beam/pull/30059 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in

Re: [PR] Add Java 17 to docker image for self-hosted runners [beam]

2024-01-19 Thread via GitHub
kennknowles commented on PR #29992: URL: https://github.com/apache/beam/pull/29992#issuecomment-1901274436 I'd probably decouple "default for CI" from "is already installed" so that we can toggle default for CI just by editing a version somewhere. -- This is an automated message from the

Re: [PR] Add Java 17 to docker image for self-hosted runners [beam]

2024-01-19 Thread via GitHub
kennknowles closed pull request #29992: Add Java 17 to docker image for self-hosted runners URL: https://github.com/apache/beam/pull/29992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Add Java 17 to docker image for self-hosted runners [beam]

2024-01-19 Thread via GitHub
kennknowles commented on code in PR #29992: URL: https://github.com/apache/beam/pull/29992#discussion_r1459964873 ## .github/gh-actions-self-hosted-runners/arc/images/Dockerfile: ## @@ -41,6 +41,9 @@ RUN curl -OL https://cdn.azul.com/zulu/bin/zulu8.70.0.23-ca-jdk8.0.372-linux_x

Re: [PR] Update windmill proto definition [beam]

2024-01-19 Thread via GitHub
m-trieu commented on PR #30046: URL: https://github.com/apache/beam/pull/30046#issuecomment-1901252920 > R: @scwhittle > > (Just to get the bot off the hook. The "R" needs to be R and not r.) ack thanks! -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Centralize exec stage urn string [beam]

2024-01-19 Thread via GitHub
tvalentyn merged PR #29947: URL: https://github.com/apache/beam/pull/29947 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on code in PR #30045: URL: https://github.com/apache/beam/pull/30045#discussion_r1459880528 ## sdks/python/apache_beam/io/gcp/bigquery.py: ## @@ -2244,18 +2250,65 @@ def find_in_nested_dict(schema): table = bigquery_tools.get_hashable_destination(self

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on code in PR #30045: URL: https://github.com/apache/beam/pull/30045#discussion_r1459876470 ## sdks/python/apache_beam/io/gcp/bigquery.py: ## @@ -2244,18 +2250,65 @@ def find_in_nested_dict(schema): table = bigquery_tools.get_hashable_destination(self

Re: [PR] Heartbeats [beam]

2024-01-19 Thread via GitHub
acrites commented on code in PR #29963: URL: https://github.com/apache/beam/pull/29963#discussion_r1459866899 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java: ## @@ -1363,6 +1385,10 @@ private void c

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on code in PR #30045: URL: https://github.com/apache/beam/pull/30045#discussion_r1459862152 ## sdks/python/apache_beam/io/gcp/bigquery.py: ## @@ -2244,18 +2250,65 @@ def find_in_nested_dict(schema): table = bigquery_tools.get_hashable_destination(self

Re: [PR] [YAML] Expose flatten implementation from Java. [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30057: URL: https://github.com/apache/beam/pull/30057#issuecomment-1901167956 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [YAML] Expose flatten implementation from Java. [beam]

2024-01-19 Thread via GitHub
robertwb commented on PR #30057: URL: https://github.com/apache/beam/pull/30057#issuecomment-1901165200 R: @Polber I do not think the failures are related. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [I] [Bug]: Usage of logical types breaks Beam YAML Sql [beam]

2024-01-19 Thread via GitHub
Abacn closed issue #29825: [Bug]: Usage of logical types breaks Beam YAML Sql URL: https://github.com/apache/beam/issues/29825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] [Bug]: Usage of logical types breaks Beam YAML Sql [beam]

2024-01-19 Thread via GitHub
Abacn commented on issue #29825: URL: https://github.com/apache/beam/issues/29825#issuecomment-1901163962 This one sounds already done in 2.53.0 release. 2.54.0 milestone is an effect of close and then re-open this after 2.53.0 cut -- This is an automated message from the Apache Git Servi

Re: [I] [Bug]: testHotKeyCombineWithSideInputs permared on Spark SparkStructuredStreaming runner [beam]

2024-01-19 Thread via GitHub
Abacn closed issue #29972: [Bug]: testHotKeyCombineWithSideInputs permared on Spark SparkStructuredStreaming runner URL: https://github.com/apache/beam/issues/29972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Exclude newly added failing validation runner tests [beam]

2024-01-19 Thread via GitHub
Abacn merged PR #30030: URL: https://github.com/apache/beam/pull/30030 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn merged PR #30051: URL: https://github.com/apache/beam/pull/30051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] [YAML] Expose flatten implementation from Java. [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30057: URL: https://github.com/apache/beam/pull/30057#issuecomment-1901145334 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] [Python BQ] Substitute final destination schema when no input schema is specified [beam]

2024-01-19 Thread via GitHub
ahmedabu98 merged PR #30015: URL: https://github.com/apache/beam/pull/30015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apac

Re: [PR] Time at max threads fix [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30041: URL: https://github.com/apache/beam/pull/30041#issuecomment-1901114565 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @robertwb added as fallback since no labels match configuration

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
damccorm commented on code in PR #30051: URL: https://github.com/apache/beam/pull/30051#discussion_r1459769646 ## buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy: ## @@ -574,6 +574,10 @@ class BeamModulePlugin implements Plugin { return project.con

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn commented on code in PR #30051: URL: https://github.com/apache/beam/pull/30051#discussion_r1459748353 ## buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy: ## @@ -574,6 +574,10 @@ class BeamModulePlugin implements Plugin { return project.contai

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn commented on code in PR #30051: URL: https://github.com/apache/beam/pull/30051#discussion_r1459748353 ## buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy: ## @@ -574,6 +574,10 @@ class BeamModulePlugin implements Plugin { return project.contai

Re: [PR] Improve varint encoding throughput with unrolled loop [beam]

2024-01-19 Thread via GitHub
sjvanrossum commented on PR #29689: URL: https://github.com/apache/beam/pull/29689#issuecomment-1901085338 I ran both of them on my laptop (macOS on M1) and workstation (Linux on AMD) and they completed successfully. -- This is an automated message from the Apache Git Service. To respond

[PR] [YAML] Expose flatten implementation from Java. [beam]

2024-01-19 Thread via GitHub
robertwb opened a new pull request, #30057: URL: https://github.com/apache/beam/pull/30057 Technically Flatten will produce exactly the same result no matter where it is expanded, but simply having this in Java avoids complexities around dealing with affinity optimization. --

Re: [PR] [Python BQ] Substitute final destination schema when no input schema is specified [beam]

2024-01-19 Thread via GitHub
shunping commented on code in PR #30015: URL: https://github.com/apache/beam/pull/30015#discussion_r1459716006 ## sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py: ## @@ -903,6 +903,68 @@ def setUp(self): _LOGGER.info( "Created dataset %s in project %s",

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
damccorm commented on code in PR #30051: URL: https://github.com/apache/beam/pull/30051#discussion_r1459712497 ## buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy: ## @@ -574,6 +574,10 @@ class BeamModulePlugin implements Plugin { return project.con

Re: [PR] Time at max threads fix [beam]

2024-01-19 Thread via GitHub
edman124 commented on PR #30041: URL: https://github.com/apache/beam/pull/30041#issuecomment-1901057572 ptal: @m-trieu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] [YAML] Allow windowing to be done in Java or Python. [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30055: URL: https://github.com/apache/beam/pull/30055#issuecomment-1901026230 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [Python BQ] Substitute final destination schema when no input schema is specified [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on code in PR #30015: URL: https://github.com/apache/beam/pull/30015#discussion_r1459664421 ## sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py: ## @@ -903,6 +903,68 @@ def setUp(self): _LOGGER.info( "Created dataset %s in project %s"

[PR] [YAML] Allow windowing to be done in Java or Python. [beam]

2024-01-19 Thread via GitHub
robertwb opened a new pull request, #30055: URL: https://github.com/apache/beam/pull/30055 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in yo

Re: [PR] [YAML] Allow windowing to be done in Java or Python. [beam]

2024-01-19 Thread via GitHub
robertwb commented on PR #30055: URL: https://github.com/apache/beam/pull/30055#issuecomment-1901024876 R: @Polber -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsub

Re: [I] [Bug]: FlinkRunner: KafkaIO emits MAX_WATEMARK [beam]

2024-01-19 Thread via GitHub
Abacn commented on issue #30050: URL: https://github.com/apache/beam/issues/30050#issuecomment-1901022086 Does this related to #29902 or would integrating #29993 resolve the issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Update windmill proto definition [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30046: URL: https://github.com/apache/beam/pull/30046#issuecomment-1901003553 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] Update windmill proto definition [beam]

2024-01-19 Thread via GitHub
lostluck commented on PR #30046: URL: https://github.com/apache/beam/pull/30046#issuecomment-1901001364 R: @scwhittle (Just to get the bot off the hook. The "R" needs to be R and not r.) -- This is an automated message from the Apache Git Service. To respond to the message, please l

Re: [PR] Add Java 17 to docker image for self-hosted runners [beam]

2024-01-19 Thread via GitHub
damccorm commented on code in PR #29992: URL: https://github.com/apache/beam/pull/29992#discussion_r1459591477 ## .github/gh-actions-self-hosted-runners/arc/images/Dockerfile: ## @@ -41,6 +41,9 @@ RUN curl -OL https://cdn.azul.com/zulu/bin/zulu8.70.0.23-ca-jdk8.0.372-linux_x64

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30051: URL: https://github.com/apache/beam/pull/30051#issuecomment-1900969648 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] Add Java 17 to docker image for self-hosted runners [beam]

2024-01-19 Thread via GitHub
damccorm commented on code in PR #29992: URL: https://github.com/apache/beam/pull/29992#discussion_r1459589823 ## .github/gh-actions-self-hosted-runners/arc/images/Dockerfile: ## @@ -41,6 +41,9 @@ RUN curl -OL https://cdn.azul.com/zulu/bin/zulu8.70.0.23-ca-jdk8.0.372-linux_x64

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #30051: URL: https://github.com/apache/beam/pull/30051#issuecomment-1900967907 R: @tvalentyn @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Use official jamm [beam]

2024-01-19 Thread via GitHub
Abacn merged PR #30034: URL: https://github.com/apache/beam/pull/30034 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Use official jamm [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #30034: URL: https://github.com/apache/beam/pull/30034#issuecomment-1900964024 Also manually tested a dataflow wordcount `2024-01-19_11_04_21-10729576904645109680`, checked the worker log there is no warning about jamm, and the job succeeded merging for now

[I] [Bug]: JMSIO drops messages when autoscaling down. [beam]

2024-01-19 Thread via GitHub
jlampek opened a new issue, #30054: URL: https://github.com/apache/beam/issues/30054 ### What happened? **Steps to Reproduce** 1. Create a Dataflow job that uses JMSIO to read from an MQ queue. 2. Configure the Dataflow job to use stream engine with number of workers initially

Re: [PR] Add support for current active work budget. [beam]

2024-01-19 Thread via GitHub
m-trieu commented on code in PR #30048: URL: https://github.com/apache/beam/pull/30048#discussion_r1459562668 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ActiveWorkState.java: ## @@ -238,6 +239,23 @@ private static

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #30051: URL: https://github.com/apache/beam/pull/30051#issuecomment-1900934994 Tested `./gradlew :sdks:java:container:java8:docker` with both M1 mac and linux amd64 locally -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] [Bug]: FlinkRunner: KafkaIO emits MAX_WATEMARK [beam]

2024-01-19 Thread via GitHub
je-ik commented on issue #30050: URL: https://github.com/apache/beam/issues/30050#issuecomment-1900929180 > Probably related to #29563. Reverting the commit seems not to resolve the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Add logging to help understand errors from #28131. [beam]

2024-01-19 Thread via GitHub
tvalentyn merged PR #30052: URL: https://github.com/apache/beam/pull/30052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

Re: [PR] Undeprecate reshuffle [beam]

2024-01-19 Thread via GitHub
damccorm merged PR #30049: URL: https://github.com/apache/beam/pull/30049 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Undeprecate reshuffle [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30049: URL: https://github.com/apache/beam/pull/30049#issuecomment-1900914150 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Add logging to help understand errors from #28131. [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30052: URL: https://github.com/apache/beam/pull/30052#issuecomment-1900914066 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label python. Available commands: - `stop

Re: [PR] [YAML] add time unit suffix to windowing [beam]

2024-01-19 Thread via GitHub
robertwb merged PR #30035: URL: https://github.com/apache/beam/pull/30035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Update to libraries-bom 26.30.0 [beam]

2024-01-19 Thread via GitHub
damccorm merged PR #30028: URL: https://github.com/apache/beam/pull/30028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Improve varint encoding throughput with unrolled loop [beam]

2024-01-19 Thread via GitHub
damondouglas commented on PR #29689: URL: https://github.com/apache/beam/pull/29689#issuecomment-1900848505 @sjvanrossum Could you confirm this as well? The two failing tests referenced from https://ge.apache.org/s/jwl3x4lvysvzc: 1) `:sdks:java:fn-execution:test` and 2) `:runners:spar

Re: [PR] Fixes an upgrade compatibility breakage for the BQ write transform [beam]

2024-01-19 Thread via GitHub
chamikaramj commented on code in PR #30032: URL: https://github.com/apache/beam/pull/30032#discussion_r1459436295 ## runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformUpgrader.java: ## @@ -204,11 +211,22 @@ RunnerApi.Pipeline updateT

[PR] Add logging to help understand errors from #28131. [beam]

2024-01-19 Thread via GitHub
tvalentyn opened a new pull request, #30052: URL: https://github.com/apache/beam/pull/30052 I came across a couple of reports where SDK was not able to parse and execute a request received from the runner like in #28131. Adding some logs so that we can better understand cause of the error.

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
damondouglas commented on code in PR #30045: URL: https://github.com/apache/beam/pull/30045#discussion_r1459401849 ## sdks/python/apache_beam/io/gcp/bigquery.py: ## @@ -2244,18 +2250,65 @@ def find_in_nested_dict(schema): table = bigquery_tools.get_hashable_destination(se

Re: [PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
codecov[bot] commented on PR #30051: URL: https://github.com/apache/beam/pull/30051#issuecomment-1900806307 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/30051?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] [Python BQ] Substitute final destination schema when no input schema is specified [beam]

2024-01-19 Thread via GitHub
shunping commented on PR #30015: URL: https://github.com/apache/beam/pull/30015#issuecomment-1900801002 Thanks! LGTM with some suggestion about adding comments in the code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

[PR] Fix buildx always been true [beam]

2024-01-19 Thread via GitHub
Abacn opened a new pull request, #30051: URL: https://github.com/apache/beam/pull/30051 It has been found `DockerExtension.ext.buildx` parameter is always true even for local testing. The issue is ``` def useBuildx = project.containerPlatforms() != [project.nativeArchitecture()] `

Re: [PR] [Python BQ] Substitute final destination schema when no input schema is specified [beam]

2024-01-19 Thread via GitHub
shunping commented on code in PR #30015: URL: https://github.com/apache/beam/pull/30015#discussion_r1459385613 ## sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py: ## @@ -903,6 +903,68 @@ def setUp(self): _LOGGER.info( "Created dataset %s in project %s",

Re: [PR] Undeprecate reshuffle [beam]

2024-01-19 Thread via GitHub
kennknowles commented on code in PR #30049: URL: https://github.com/apache/beam/pull/30049#discussion_r1459377380 ## sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java: ## @@ -44,18 +44,18 @@ * * A {@link PTransform} that returns a {@link PCollection}

Re: [I] [Bug]: FlinkRunner: KafkaIO emits MAX_WATEMARK [beam]

2024-01-19 Thread via GitHub
je-ik commented on issue #30050: URL: https://github.com/apache/beam/issues/30050#issuecomment-1900786023 Probably related to https://github.com/apache/beam/pull/29563. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #30045: URL: https://github.com/apache/beam/pull/30045#issuecomment-1900784046 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on PR #30045: URL: https://github.com/apache/beam/pull/30045#issuecomment-1900782397 R: @chamikaramj R: @johnjcasey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] [Task]: Update the minor version of google-cloud-storage library prior to Beam release. [beam]

2024-01-19 Thread via GitHub
shunping commented on issue #27326: URL: https://github.com/apache/beam/issues/27326#issuecomment-1900764422 I just checked for the latest gcsio (after the recent migration), we still rely on the internal variable `_responses` mentioned in the description: * https://github.com/apache/bea

Re: [I] [Task]: Update the minor version of google-cloud-storage library prior to Beam release. [beam]

2024-01-19 Thread via GitHub
shunping commented on issue #27326: URL: https://github.com/apache/beam/issues/27326#issuecomment-1900757406 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Fix BigtableConfigTranslator [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #30039: URL: https://github.com/apache/beam/pull/30039#issuecomment-1900724105 GCP IO Direct PreCommit `testHL7v2IO_ListHL7v2Messages` known flaky test not related -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] [Feature Request]: Allow a flag so that "with pipeline" will not wait_until_finish so that this paradigm can be used for streaming jobs [beam]

2024-01-19 Thread via GitHub
damccorm commented on issue #29440: URL: https://github.com/apache/beam/issues/29440#issuecomment-1900705781 > Please help me understand this a bit more, so that I can write the doc better. I don't understand this case where a watermark can advance to the end of the global window. Ba

[PR] Undeprecate reshuffle [beam]

2024-01-19 Thread via GitHub
damccorm opened a new pull request, #30049: URL: https://github.com/apache/beam/pull/30049 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quick

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on PR #30045: URL: https://github.com/apache/beam/pull/30045#issuecomment-1900688713 Xlang GCP Direct postcommit: https://github.com/ahmedabu98/beam/actions/runs/7586206846 Xlang GCP Dataflow postcommit: https://github.com/ahmedabu98/beam/actions/runs/7586212776 --

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on PR #30045: URL: https://github.com/apache/beam/pull/30045#issuecomment-1900684382 Run Python_Xlang_Gcp_Direct PostCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Support dynamic destinations with Python Storage API [beam]

2024-01-19 Thread via GitHub
ahmedabu98 commented on PR #30045: URL: https://github.com/apache/beam/pull/30045#issuecomment-1900684523 Run Python_Xlang_Gcp_Dataflow PostCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Revert #28614: Add UseDataStreamForBatch option to Flink runner [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #29993: URL: https://github.com/apache/beam/pull/29993#issuecomment-1900638692 Hi @noster-dev have you been able to confirm this would fix the problem? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Use official jamm [beam]

2024-01-19 Thread via GitHub
Abacn commented on PR #30034: URL: https://github.com/apache/beam/pull/30034#issuecomment-1900631061 > GrpcLoggingServiceTest The rerun is green I saw the timeout test observed before: #14474 it has higher probability when gradle cache is disabled. Similarly cron jobs appear to

Re: [PR] added custom watermark for kinesis reader [beam]

2024-01-19 Thread via GitHub
damccorm commented on PR #28763: URL: https://github.com/apache/beam/pull/28763#issuecomment-1900608036 @johnjcasey could you please take another look at this one/help take this pr to completion? -- This is an automated message from the Apache Git Service. To respond to the message, pleas

Re: [I] [Feature Request]: Allow a flag so that "with pipeline" will not wait_until_finish so that this paradigm can be used for streaming jobs [beam]

2024-01-19 Thread via GitHub
lazarillo commented on issue #29440: URL: https://github.com/apache/beam/issues/29440#issuecomment-1900542855 > +1 on adding docs, I agree this could be clearer. OK, I'll try to work on it this weekend. > > context managers should not be used for streaming > > This isn't

Re: [PR] Use official jamm [beam]

2024-01-19 Thread via GitHub
damccorm commented on PR #30034: URL: https://github.com/apache/beam/pull/30034#issuecomment-1900517048 > Java PreCommit stuck test `org.apache.beam.runners.fnexecution.logging.GrpcLoggingServiceTest. testMultipleClientsFailingIsHandledGracefullyByServer` not related Doesn't look lik

Re: [PR] Add support for current active work budget. [beam]

2024-01-19 Thread via GitHub
scwhittle commented on code in PR #30048: URL: https://github.com/apache/beam/pull/30048#discussion_r1458915472 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ActiveWorkState.java: ## @@ -238,6 +239,23 @@ private stat

Re: [PR] Windowing Support for the DaskRunner [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #27618: URL: https://github.com/apache/beam/pull/27618#issuecomment-1900308592 Reminder, please take a look at this pr: @riteshghorse @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] added custom watermark for kinesis reader [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #28763: URL: https://github.com/apache/beam/pull/28763#issuecomment-1900308471 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @robertwb f

Re: [PR] Generalize file based source output capabilities [beam]

2024-01-19 Thread via GitHub
github-actions[bot] commented on PR #29566: URL: https://github.com/apache/beam/pull/29566#issuecomment-1900308337 Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`: R: @Abacn for

Re: [PR] Update windmill proto definition [beam]

2024-01-19 Thread via GitHub
scwhittle commented on code in PR #30046: URL: https://github.com/apache/beam/pull/30046#discussion_r1458695977 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/WindmillEndpoints.java: ## @@ -90,31 +137,20 @@ public stat

Re: [PR] Heartbeats [beam]

2024-01-19 Thread via GitHub
scwhittle commented on code in PR #29963: URL: https://github.com/apache/beam/pull/29963#discussion_r1458591244 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java: ## @@ -1363,6 +1385,10 @@ private void

Re: [PR] Heartbeats [beam]

2024-01-19 Thread via GitHub
scwhittle commented on code in PR #29963: URL: https://github.com/apache/beam/pull/29963#discussion_r1458589403 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java: ## @@ -1873,14 +1915,20 @@ private voi

Re: [PR] Add support for current active work budget. [beam]

2024-01-19 Thread via GitHub
m-trieu commented on PR #30048: URL: https://github.com/apache/beam/pull/30048#issuecomment-1899938844 run java precommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To