Re: [I] [Bug]: KinesisIO source on FlinkRunner initializes the same splits twice [beam]

2024-05-23 Thread via GitHub
je-ik commented on issue #31313: URL: https://github.com/apache/beam/issues/31313#issuecomment-2128718496 I suppose this is (similar, but) different issue, probably caused by the same underlying bug. #30903 fixed Impulse only. Does using `--experiments=beam_fn_api` fix the issue?

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.54.3 in /sdks [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31389: URL: https://github.com/apache/beam/pull/31389#issuecomment-2128615640 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.16.21 in /sdks [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31388: URL: https://github.com/apache/beam/pull/31388#issuecomment-2128565065 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Bump commons-cli:commons-cli from 1.6.0 to 1.8.0 [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31387: URL: https://github.com/apache/beam/pull/31387#issuecomment-2128565111 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.54.2 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] closed pull request #31326: Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.54.2 in /sdks URL: https://github.com/apache/beam/pull/31326 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.54.2 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] commented on PR #31326: URL: https://github.com/apache/beam/pull/31326#issuecomment-2128547408 Superseded by #31389. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

[PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.42.2 to 1.54.3 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] opened a new pull request, #31389: URL: https://github.com/apache/beam/pull/31389 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.42.2 to 1.54.3. Commits https://github.com/aws/aws-sdk-go-v2/commit/8abec4c1a7b9326c05

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.16.20 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] commented on PR #31325: URL: https://github.com/apache/beam/pull/31325#issuecomment-2128538715 Superseded by #31388. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.16.20 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] closed pull request #31325: Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.16.20 in /sdks URL: https://github.com/apache/beam/pull/31325 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[PR] Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.13.8 to 1.16.21 in /sdks [beam]

2024-05-23 Thread via GitHub
dependabot[bot] opened a new pull request, #31388: URL: https://github.com/apache/beam/pull/31388 Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.13.8 to 1.16.21. Commits https://github.com/aws/aws-sdk-go-v2/commit/b08ae8434

[PR] Bump commons-cli:commons-cli from 1.6.0 to 1.8.0 [beam]

2024-05-23 Thread via GitHub
dependabot[bot] opened a new pull request, #31387: URL: https://github.com/apache/beam/pull/31387 Bumps commons-cli:commons-cli from 1.6.0 to 1.8.0. [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=commons-cli:co

Re: [I] [Bug]: KinesisIO source on FlinkRunner initializes the same splits twice [beam]

2024-05-23 Thread via GitHub
akashk99 commented on issue #31313: URL: https://github.com/apache/beam/issues/31313#issuecomment-2128405423 @je-ik is this the same issue as this https://github.com/apache/beam/issues/30903 I noticed you fixed it and the problem statement seems to be similar, but please let me know

Re: [PR] Don't install TF on Python 3.12, since there is no compatible version atm. [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31386: URL: https://github.com/apache/beam/pull/31386#issuecomment-2128318875 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @liferoad for label python. Available commands: - `stop

Re: [PR] Don't install TF on Python 3.12, since there is no compatible version atm. [beam]

2024-05-23 Thread via GitHub
liferoad commented on PR #31386: URL: https://github.com/apache/beam/pull/31386#issuecomment-2128293343 Not sure the failed ML test is a concern. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] Don't install TF on Python 3.12, since there is no compatible version atm. [beam]

2024-05-23 Thread via GitHub
tvalentyn opened a new pull request, #31386: URL: https://github.com/apache/beam/pull/31386 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quic

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
tvalentyn merged PR #30828: URL: https://github.com/apache/beam/pull/30828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

Re: [I] [Task]: Support Python 3.12 [beam]

2024-05-23 Thread via GitHub
tvalentyn closed issue #29149: [Task]: Support Python 3.12 URL: https://github.com/apache/beam/issues/29149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-ma

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
damccorm commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612283952 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8 -

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
damccorm commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612283952 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8 -

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
tvalentyn commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612267356 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8

Re: [PR] move playground ci nightly to highmem due to mem exaustion [beam]

2024-05-23 Thread via GitHub
damccorm merged PR #31384: URL: https://github.com/apache/beam/pull/31384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] move playground ci nightly to highmem due to mem exaustion [beam]

2024-05-23 Thread via GitHub
volatilemolotov commented on PR #31384: URL: https://github.com/apache/beam/pull/31384#issuecomment-2127936993 R: @damccorm this seems to help on my fork -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[PR] move playground ci nightly to highmem due to mem exaustion [beam]

2024-05-23 Thread via GitHub
volatilemolotov opened a new pull request, #31384: URL: https://github.com/apache/beam/pull/31384 Move due to java job failing Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: -

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
damccorm commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612228056 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8 -

Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]

2024-05-23 Thread via GitHub
tonytanger commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127899927 I added some unit tests for the ReadChangeStream config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
tvalentyn commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612213019 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8

Re: [PR] Fix beam quest link [beam]

2024-05-23 Thread via GitHub
svetakvsundhar merged PR #31383: URL: https://github.com/apache/beam/pull/31383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.

Re: [PR] [WIP] Add in redistribute option for Kafka Read [beam]

2024-05-23 Thread via GitHub
kennknowles commented on code in PR #31347: URL: https://github.com/apache/beam/pull/31347#discussion_r1612123815 ## sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java: ## @@ -2295,6 +2391,15 @@ public ReadSourceDescriptors withProcessingTime() {

Re: [PR] remove processing/scheduling logic from StreamingDataflowWorker [beam]

2024-05-23 Thread via GitHub
m-trieu commented on code in PR #31317: URL: https://github.com/apache/beam/pull/31317#discussion_r1612107606 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java: ## @@ -201,6 +202,24 @@ public voi

[PR] Fix beam quest link [beam]

2024-05-23 Thread via GitHub
damccorm opened a new pull request, #31383: URL: https://github.com/apache/beam/pull/31383 R: @svetakvsundhar Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the

Re: [PR] Add Backlog Metrics to Kafka Splittable DoFn Implementation [beam]

2024-05-23 Thread via GitHub
johnjcasey commented on PR #31281: URL: https://github.com/apache/beam/pull/31281#issuecomment-2127718810 LGTM! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

Re: [PR] Add Python 3.12 support [beam]

2024-05-23 Thread via GitHub
damccorm commented on code in PR #30828: URL: https://github.com/apache/beam/pull/30828#discussion_r1612039017 ## .github/workflows/beam_PostCommit_TransformService_Direct.yml: ## @@ -77,7 +77,7 @@ jobs: java-version: 11 python-version: | 3.8 -

Re: [PR] Reapply "[#30789] Add support for Flink 1.18 (#31062)" [beam]

2024-05-23 Thread via GitHub
je-ik merged PR #31328: URL: https://github.com/apache/beam/pull/31328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Memoize some dataframes analysis operations. [beam]

2024-05-23 Thread via GitHub
tvalentyn commented on code in PR #31377: URL: https://github.com/apache/beam/pull/31377#discussion_r1611976017 ## sdks/python/apache_beam/dataframe/transforms.py: ## @@ -302,6 +302,7 @@ def __repr__(self, indent=0): self.outputs)) # First define some hel

Re: [PR] Fixed the extension name in Java SDK Ordered Processing extension [beam]

2024-05-23 Thread via GitHub
damccorm merged PR #31382: URL: https://github.com/apache/beam/pull/31382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Update README.md (fix typo) [beam]

2024-05-23 Thread via GitHub
Abacn merged PR #31380: URL: https://github.com/apache/beam/pull/31380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127438489 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @m-trieu for label java. R: @johnjcasey for label io. R: @ig

Re: [PR] Add option to disable validation of cloud bigtable change stream IO [beam]

2024-05-23 Thread via GitHub
tonytanger commented on PR #31376: URL: https://github.com/apache/beam/pull/31376#issuecomment-2127413740 @jackdingilian for Bigtable review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] [Bug]: Watermarks and Windowing Not Working with FlinkRunner and KinesisIO Read Transform [beam]

2024-05-23 Thread via GitHub
je-ik commented on issue #31085: URL: https://github.com/apache/beam/issues/31085#issuecomment-2127409080 Yeah. Thia narrows it down to the source API. I was not able to identify a comnit to 2.52.0 that could cause this, but I'll use your test locally. Thanks. 👍 -- This is an automated

Re: [PR] Fixed the extension name in Java SDK Ordered Processing extension [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31382: URL: https://github.com/apache/beam/pull/31382#issuecomment-2127376731 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damondouglas for label java. Available commands: - `sto

[PR] [WIP] try new protobuf [beam]

2024-05-23 Thread via GitHub
riteshghorse opened a new pull request, #30556: URL: https://github.com/apache/beam/pull/30556 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution q

Re: [I] [Bug]: Watermarks and Windowing Not Working with FlinkRunner and KinesisIO Read Transform [beam]

2024-05-23 Thread via GitHub
yelianevich commented on issue #31085: URL: https://github.com/apache/beam/issues/31085#issuecomment-2127349579 @je-ik I do not specify any additional flags in the test, so I think it uses defaults. I run the same test using `--experiments=use_deprecated_read` and it didn't change a

[PR] Fixed the extension name in Java SDK Ordered Processing extension [beam]

2024-05-23 Thread via GitHub
slilichenko opened a new pull request, #31382: URL: https://github.com/apache/beam/pull/31382 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution qu

Re: [I] [Bug]: Watermarks and Windowing Not Working with FlinkRunner and KinesisIO Read Transform [beam]

2024-05-23 Thread via GitHub
je-ik commented on issue #31085: URL: https://github.com/apache/beam/issues/31085#issuecomment-2127105551 @yelianevich Is this with default flags? Does this change when using `--experiments=use_deprecated_read` or `--experiments=beam_fn_api`? -- This is an automated message from the Apach

Re: [I] [Bug]: Watermarks and Windowing Not Working with FlinkRunner and KinesisIO Read Transform [beam]

2024-05-23 Thread via GitHub
je-ik commented on issue #31085: URL: https://github.com/apache/beam/issues/31085#issuecomment-2127016131 Great, thanks! Looks like empty sources (before emitting first element) do not update downstream watermark. This is consistent with the observation above. I will look into it. -- Thi

Re: [PR] [WIP] try new protobuf [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #30556: URL: https://github.com/apache/beam/pull/30556#issuecomment-2127009883 This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. -- This is

Re: [PR] [WIP] try new protobuf [beam]

2024-05-23 Thread via GitHub
github-actions[bot] closed pull request #30556: [WIP] try new protobuf URL: https://github.com/apache/beam/pull/30556 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubsc

Re: [PR] Bump tqdm from 4.66.2 to 4.66.3 in /sdks/python/container/py38 [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31179: URL: https://github.com/apache/beam/pull/31179#issuecomment-2126958634 Reminder, please take a look at this pr: @riteshghorse -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Clean up stale code in BigtableService [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #30172: URL: https://github.com/apache/beam/pull/30172#issuecomment-2126958710 Reminder, please take a look at this pr: @Abacn @Abacn @igorbernstein2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on t

Re: [PR] use work processing context in work processing [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31148: URL: https://github.com/apache/beam/pull/31148#issuecomment-2126958664 Reminder, please take a look at this pr: @johnjcasey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] remove processing/scheduling logic from StreamingDataflowWorker [beam]

2024-05-23 Thread via GitHub
scwhittle commented on code in PR #31317: URL: https://github.com/apache/beam/pull/31317#discussion_r1611422879 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java: ## @@ -201,6 +202,24 @@ public v

Re: [I] [Bug]: Watermarks and Windowing Not Working with FlinkRunner and KinesisIO Read Transform [beam]

2024-05-23 Thread via GitHub
yelianevich commented on issue #31085: URL: https://github.com/apache/beam/issues/31085#issuecomment-2126656952 > I don't know if there are any workarounds, as the described behavior seems to be (unknown) bug. It needs further investigation. Could you please provide a simplified pipeline th

Re: [PR] Add Histogram transform and combiner [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31379: URL: https://github.com/apache/beam/pull/31379#issuecomment-2126426456 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Update README.md (fix typo) [beam]

2024-05-23 Thread via GitHub
github-actions[bot] commented on PR #31380: URL: https://github.com/apache/beam/pull/31380#issuecomment-2126426362 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @Abacn added as fallback since no labels match configuration

Re: [PR] remove processing/scheduling logic from StreamingDataflowWorker [beam]

2024-05-23 Thread via GitHub
m-trieu commented on code in PR #31317: URL: https://github.com/apache/beam/pull/31317#discussion_r1611146062 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/ComputationState.java: ## @@ -116,6 +118,11 @@ public boolea

Re: [PR] Reapply "[#30789] Add support for Flink 1.18 (#31062)" [beam]

2024-05-23 Thread via GitHub
je-ik commented on PR #31328: URL: https://github.com/apache/beam/pull/31328#issuecomment-2126407935 @Abacn [this][1] fixes the issue with the test. [1]: https://github.com/apache/beam/pull/31328/commits/c4ea518a247f325ade7c094c1d4d1d69c3671a6c -- This is an automated message fro

[PR] Update README.md (fix typo) [beam]

2024-05-23 Thread via GitHub
paulk-asert opened a new pull request, #31380: URL: https://github.com/apache/beam/pull/31380 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution qu