Re: [PR] adds pre-commit hook to standardize whitespaces, adds EditorConfig to set the indents [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35564: URL: https://github.com/apache/beam/pull/35564#issuecomment-3060835907 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] [Dataflow Streaming] Fix grpc commit stream test [beam]

2025-07-10 Thread via GitHub
arunpandianp commented on PR #35552: URL: https://github.com/apache/beam/pull/35552#issuecomment-3060799012 Run Java PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] updates Hazelcast website link in the documentation [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35565: URL: https://github.com/apache/beam/pull/35565#issuecomment-3060561517 Assigning reviewers: R: @damccorm for label website. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

Re: [PR] updates Hazelcast website link in the documentation [beam]

2025-07-10 Thread via GitHub
nyoungstudios commented on code in PR #35565: URL: https://github.com/apache/beam/pull/35565#discussion_r2199484036 ## README.md: ## @@ -19,7 +19,7 @@ # Apache Beam -[Apache Beam](http://beam.apache.org/) is a unified model for defining both batch and streaming data-parall

Re: [PR] [YAML] A Streaming Inference Pipeline - YouTube Comments Sentiment Analysis [beam]

2025-07-10 Thread via GitHub
chamikaramj commented on code in PR #35375: URL: https://github.com/apache/beam/pull/35375#discussion_r2199386982 ## sdks/python/apache_beam/yaml/examples/transforms/ml/inference/streaming_sentiment_analysis.yaml: ## @@ -0,0 +1,257 @@ +# coding=utf-8 +# +# Licensed to the Apache

Re: [PR] removes Hazelcast Jet runner links from the documentation [beam]

2025-07-10 Thread via GitHub
Abacn commented on code in PR #35565: URL: https://github.com/apache/beam/pull/35565#discussion_r2199424427 ## README.md: ## @@ -19,7 +19,7 @@ # Apache Beam -[Apache Beam](http://beam.apache.org/) is a unified model for defining both batch and streaming data-parallel proce

[PR] removes Hazelcast Jet runner links from the documentation [beam]

2025-07-10 Thread via GitHub
nyoungstudios opened a new pull request, #35565: URL: https://github.com/apache/beam/pull/35565 **Please** add a meaningful description for your change here removes Hazelcast Jet runner links from the documentation since the URL is no longer valid

[PR] adds pre-commit hook to standardize whitespaces, adds EditorConfig to set the indents [beam]

2025-07-10 Thread via GitHub
nyoungstudios opened a new pull request, #35564: URL: https://github.com/apache/beam/pull/35564 **Please** add a meaningful description for your change here Adds a pre-commit hook to standardize whitespaces, so we don't have stray new lines or missing new line at the end of the file,

Re: [I] [Bug]: Python experiments and dataflow_service_options do not handle comma separated options [beam]

2025-07-10 Thread via GitHub
Abacn commented on issue #35563: URL: https://github.com/apache/beam/issues/35563#issuecomment-3060124536 To fix this one needs to implement a custom argparse action and set it here https://github.com/apache/beam/blob/9039608560c514fdae6034f2534120cbb16ac090/sdks/python/apache_beam/op

[I] [Bug]: Python experiments and dataflow_service_options do not handle comma separated options [beam]

2025-07-10 Thread via GitHub
Abacn opened a new issue, #35563: URL: https://github.com/apache/beam/issues/35563 ### What happened? In Beam Java SDK, one can add experiments `--experiments=abc,def`, `--dataflowServiceOptions=abc=true,def=true`, but in Python SDK, `--experiments=abc,def` will end up with an experi

Re: [PR] Another attempt to fix jdbc timestamp logical type. [beam]

2025-07-10 Thread via GitHub
Abacn commented on PR #35426: URL: https://github.com/apache/beam/pull/35426#issuecomment-3060094218 Consider run https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml?query=event%3Aschedule which contains more relevant tests (e.g. bigquery IO write xl

Re: [PR] Mysql embeddings [beam]

2025-07-10 Thread via GitHub
Abacn commented on PR #35393: URL: https://github.com/apache/beam/pull/35393#issuecomment-3060048221 This has increased the number of test of https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml?query=event%3Aschedule from 45 -> 62 causing tests oft

Re: [PR] Move remaining workflow to Java11 [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35559: URL: https://github.com/apache/beam/pull/35559#issuecomment-3059909767 Assigning reviewers: R: @damccorm for label python. R: @lostluck for label go. R: @damccorm for label build. Note: If you would like to opt out of this re

Re: [PR] Move remaining workflow to Java11 [beam]

2025-07-10 Thread via GitHub
Abacn commented on PR #35559: URL: https://github.com/apache/beam/pull/35559#issuecomment-3059648783 Tested: PostCommit XVR Samza: https://github.com/apache/beam/actions/runs/16206535143 PostCommit Python ValidatesRunner Samza: https://github.com/apache/beam/actions/runs/162014

Re: [PR] [Dataflow Streaming] Fix grpc commit stream test [beam]

2025-07-10 Thread via GitHub
arunpandianp commented on PR #35552: URL: https://github.com/apache/beam/pull/35552#issuecomment-3059459807 Run Java PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [PR] [Dataflow Streaming] Fix grpc commit stream test [beam]

2025-07-10 Thread via GitHub
arunpandianp commented on PR #35552: URL: https://github.com/apache/beam/pull/35552#issuecomment-3059457681 Run PreCommit Java -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] [Dataflow Streaming] Fix grpc commit stream test [beam]

2025-07-10 Thread via GitHub
arunpandianp commented on PR #35552: URL: https://github.com/apache/beam/pull/35552#issuecomment-3059458707 The failing tests look un-releated to the change. Rerunning them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] [Iceberg SQL] Add iceberg CDC table [beam]

2025-07-10 Thread via GitHub
codecov[bot] commented on PR #35562: URL: https://github.com/apache/beam/pull/35562#issuecomment-3059416288 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/35562?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term

[PR] [Iceberg SQL] Add iceberg CDC table [beam]

2025-07-10 Thread via GitHub
ahmedabu98 opened a new pull request, #35562: URL: https://github.com/apache/beam/pull/35562 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [I] [Failing Test]: GoogleCloudStorageImpl constructor mismatch in GcsUtil [beam]

2025-07-10 Thread via GitHub
Abacn commented on issue #35560: URL: https://github.com/apache/beam/issues/35560#issuecomment-3059243583 Upgrade to com.google.cloud.bigdataoss:gcsio major version 3 on Beam mainline is blocked due to it dropped Java8 support. On Java8 client at runtime, it will fail with ``` Exc

Re: [PR] [GrowableOffsetRangeTracker] Use UnsignedLong instead of BigDecimal to calculate progress [beam]

2025-07-10 Thread via GitHub
mohamedawnallah commented on PR #35561: URL: https://github.com/apache/beam/pull/35561#issuecomment-3059202568 /gemini review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] draft [beam]

2025-07-10 Thread via GitHub
portikCoder closed pull request #35549: draft URL: https://github.com/apache/beam/pull/35549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsu

Re: [PR] draft [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #35549: URL: https://github.com/apache/beam/pull/35549#issuecomment-3059136805 Closed in favour of: https://github.com/apache/beam/pull/35558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Draft: Phun fix copy job writetruncate on identical tableid tmp fix pipelines v2.66 [beam]

2025-07-10 Thread via GitHub
portikCoder closed pull request #35550: Draft: Phun fix copy job writetruncate on identical tableid tmp fix pipelines v2.66 URL: https://github.com/apache/beam/pull/35550 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [PR] Draft: Phun fix copy job writetruncate on identical tableid tmp fix pipelines v2.66 [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #35550: URL: https://github.com/apache/beam/pull/35550#issuecomment-3059135083 Closed in favour of: https://github.com/apache/beam/pull/35558 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] [GrowableOffsetRangeTracker] Use UnsignedLong instead of BigDecimal to calculate progress [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35561: URL: https://github.com/apache/beam/pull/35561#issuecomment-3058992199 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
liferoad closed pull request #34248: [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) URL: https://github.com/apache/beam/pull/34248 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
liferoad commented on PR #34248: URL: https://github.com/apache/beam/pull/34248#issuecomment-3058762404 we generally do not do the patch release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #34248: URL: https://github.com/apache/beam/pull/34248#issuecomment-3058746250 > > why you try to merge the PR to [apache:release-2.63.0-postrelease](https://github.com/apache/beam/tree/release-2.63.0-postrelease)? You should rebase to the master branch. >

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #35558: URL: https://github.com/apache/beam/pull/35558#issuecomment-3058749132 I opened this to master instead, here is why: https://github.com/apache/beam/pull/34248#issuecomment-3058746250 -- This is an automated message from the Apache Git Service. To res

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35558: URL: https://github.com/apache/beam/pull/35558#issuecomment-3058740295 Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

[PR] Use unsigned integer math to calculate differences between range start, consumed and estimated end positions. The difference between two values does not exceed the maximum value of unsigned longs

2025-07-10 Thread via GitHub
sjvanrossum opened a new pull request, #35561: URL: https://github.com/apache/beam/pull/35561 Use unsigned integer math to calculate differences between range start, consumed and estimated end positions. The difference between two values does not exceed the maximum value of unsigned longs s

[I] [Failing Test]: GoogleCloudStorageImpl constructor mismatch in GcsUtil [beam]

2025-07-10 Thread via GitHub
jinseopkim0 opened a new issue, #35560: URL: https://github.com/apache/beam/issues/35560 ### What happened? https://github.com/apache/beam/pull/35175 was drafted to check compatibility of the candidate versions of LTS 9. Test failures show that a dependency conflict was encountered r

Re: [I] [Bug]: DebeziumIO and RequestResponseIO in different package than all other IOs [beam]

2025-07-10 Thread via GitHub
mdanowar3 commented on issue #35557: URL: https://github.com/apache/beam/issues/35557#issuecomment-3058467819 Thanks for the update on the greatest -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Avoid unreasonably long stage names for @ptransform_fn. [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35547: URL: https://github.com/apache/beam/pull/35547#issuecomment-3058436563 Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #35558: URL: https://github.com/apache/beam/pull/35558#issuecomment-3058377079 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Arnav Arora BigTablePR [beam]

2025-07-10 Thread via GitHub
arnavarora2004 commented on code in PR #35435: URL: https://github.com/apache/beam/pull/35435#discussion_r2198170975 ## sdks/python/apache_beam/yaml/tests/bigTable.yaml: ## @@ -0,0 +1,124 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[PR] Move remaining workflow to Java11 [beam]

2025-07-10 Thread via GitHub
Abacn opened a new pull request, #35559: URL: https://github.com/apache/beam/pull/35559 * Honor testJavaVersion property in toxTask * Honor testJavaVersion property in Go Validation Runner Task **Please** add a meaningful description for your change here -

Re: [PR] [Dataflow Streaming] Fix grpc commit stream test [beam]

2025-07-10 Thread via GitHub
liferoad commented on PR #35552: URL: https://github.com/apache/beam/pull/35552#issuecomment-3057982161 ``` org.apache.beam.sdk.io.TextIOWriteTest > testWriteUnboundedWithCustomBatchParameters STANDARD_ERROR [direct-runner-worker] INFO org.apache.beam.sdk.io.WriteFiles - Opening w

Re: [PR] Arnav Arora BigTablePR [beam]

2025-07-10 Thread via GitHub
ahmedabu98 commented on code in PR #35435: URL: https://github.com/apache/beam/pull/35435#discussion_r2197875878 ## sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java: ## @@ -0,0 +1,211 @@ +/* + *

[PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
portikCoder opened a new pull request, #35558: URL: https://github.com/apache/beam/pull/35558 I was thinking first to use only datasetId+tableId, but i changed my mind, since it makes better sense to use the full reference instead. All the other details can be found in the mentioned bug fil

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
portikCoder commented on PR #34248: URL: https://github.com/apache/beam/pull/34248#issuecomment-3057898450 > why you try to merge the PR to [apache:release-2.63.0-postrelease](https://github.com/apache/beam/tree/release-2.63.0-postrelease)? You should rebase to the master branch. See

Re: [I] [Bug]: Error in SolaceIO during autoscaling events ("Tried to start a closed message consumer") [beam]

2025-07-10 Thread via GitHub
stankiewicz commented on issue #35304: URL: https://github.com/apache/beam/issues/35304#issuecomment-3057824415 Paweł, would you be able to share some logs from the worker that had this exception? especially if there are logs like: ```SolaceIO.Read: Closing session for the reader

Re: [PR] Fix failed rows conversion missing 'as_dict' error [beam]

2025-07-10 Thread via GitHub
claudevdm commented on code in PR #35533: URL: https://github.com/apache/beam/pull/35533#discussion_r2197974988 ## sdks/python/apache_beam/io/gcp/bigquery.py: ## @@ -2723,11 +2723,12 @@ def expand(self, input): lambda row_and_error: row_and_error[0]) if not is_rows

Re: [I] [Bug]: Error in SolaceIO during autoscaling events ("Tried to start a closed message consumer") [beam]

2025-07-10 Thread via GitHub
stankiewicz commented on issue #35304: URL: https://github.com/apache/beam/issues/35304#issuecomment-3057736835 thanks Pawel, looking into this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [Bug]: Breaking change in Coder introduced in 2.63 is leading to incompatiblities [beam]

2025-07-10 Thread via GitHub
stankiewicz closed issue #34933: [Bug]: Breaking change in Coder introduced in 2.63 is leading to incompatiblities URL: https://github.com/apache/beam/issues/34933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [I] [Bug]: Breaking change in Coder introduced in 2.63 is leading to incompatiblities [beam]

2025-07-10 Thread via GitHub
stankiewicz commented on issue #34933: URL: https://github.com/apache/beam/issues/34933#issuecomment-3057728588 this is resolved on backend with additional flag. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Fix PostCommit Python ValidatesContainer Dataflow With RC job [beam]

2025-07-10 Thread via GitHub
mohamedawnallah commented on PR #35556: URL: https://github.com/apache/beam/pull/35556#issuecomment-3057690613 /gemini review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Fix javadoc workflow [beam]

2025-07-10 Thread via GitHub
Abacn commented on code in PR #35551: URL: https://github.com/apache/beam/pull/35551#discussion_r2197895116 ## sdks/java/javadoc/build.gradle: ## @@ -83,4 +81,22 @@ task aggregateJavadoc(type: Javadoc) { links createJavadocIOUrlForDependency(dep) } } + + // certa

[I] [Bug]: DebeziumIO and RequestResponseIO in different package than all other IOs [beam]

2025-07-10 Thread via GitHub
Abacn opened a new issue, #35557: URL: https://github.com/apache/beam/issues/35557 ### What happened? while most IOs live in package `org.apache.beam.sdk.io`, DebeziumIO and RequestResponseIO live in `org.apache.beam.io`. It is most likely not intended. However, moving them is a brea

Re: [PR] [Python] Fix WriteToBigQuery transform using CopyJob does not work with WRITE_TRUNCATE write disposition (#34247) [beam]

2025-07-10 Thread via GitHub
liferoad commented on PR #34248: URL: https://github.com/apache/beam/pull/34248#issuecomment-3057513436 why you try to merge the PR to [apache:release-2.63.0-postrelease](https://github.com/apache/beam/tree/release-2.63.0-postrelease)? You should rebase to the master branch. -- This is a

Re: [PR] Fix javadoc workflow [beam]

2025-07-10 Thread via GitHub
damccorm commented on code in PR #35551: URL: https://github.com/apache/beam/pull/35551#discussion_r2197739703 ## sdks/java/javadoc/build.gradle: ## @@ -83,4 +81,22 @@ task aggregateJavadoc(type: Javadoc) { links createJavadocIOUrlForDependency(dep) } } + + // ce

Re: [I] [Bug]: ParDoLifeCycleTest in Stateful DoFn fails on Samza PVR runner [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on issue #32520: URL: https://github.com/apache/beam/issues/32520#issuecomment-3057311600 This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue

Re: [I] [prism] Smarter "globally" aware dynamic splits. [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on issue #32538: URL: https://github.com/apache/beam/issues/32538#issuecomment-3057311538 This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue

Re: [PR] [KafkaIO] Report unknown backlog size when latest offset lags behind next offset [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35086: URL: https://github.com/apache/beam/pull/35086#issuecomment-3057204230 Reminder, please take a look at this pr: @kennknowles @liferoad -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

Re: [PR] Distinguishing bigquery logging failures severities [beam]

2025-07-10 Thread via GitHub
github-actions[bot] commented on PR #35373: URL: https://github.com/apache/beam/pull/35373#issuecomment-3057204136 Reminder, please take a look at this pr: @liferoad -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the