Re: [PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.87.2 to 1.87.3 in /sdks [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36042: URL: https://github.com/apache/beam/pull/36042#issuecomment-3250885020 Assigning reviewers: R: @jrmccluskey for label go. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available comma

[PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.87.2 to 1.87.3 in /sdks [beam]

2025-09-03 Thread via GitHub
dependabot[bot] opened a new pull request, #36042: URL: https://github.com/apache/beam/pull/36042 Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.87.2 to 1.87.3. Commits https://github.com/aws/aws-sdk-go-v2/commit/640b80872b2c49ad52

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319915328 ## sdks/python/apache_beam/io/jdbc.py: ## @@ -96,10 +95,11 @@ from apache_beam.transforms.external import BeamJarExpansionService from apache_beam.transforms.external

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
liferoad commented on PR #36036: URL: https://github.com/apache/beam/pull/36036#issuecomment-3250653558 /gemini review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
gemini-code-assist[bot] commented on code in PR #36036: URL: https://github.com/apache/beam/pull/36036#discussion_r2320095874 ## sdks/python/apache_beam/yaml/examples/transforms/blueprint/iceberg_to_alloydb.yaml: ## @@ -0,0 +1,51 @@ +# coding=utf-8 +# +# Licensed to the Apache S

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
liferoad commented on code in PR #36036: URL: https://github.com/apache/beam/pull/36036#discussion_r2320093761 ## sdks/python/apache_beam/yaml/examples/transforms/blueprint/iceberg_to_alloydb.yaml: ## @@ -0,0 +1,55 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Foundat

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
derrickaw commented on code in PR #36036: URL: https://github.com/apache/beam/pull/36036#discussion_r2320023053 ## sdks/python/apache_beam/yaml/examples/transforms/blueprint/iceberg_to_alloydb.yaml: ## @@ -0,0 +1,55 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Founda

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
tarun-google commented on code in PR #36036: URL: https://github.com/apache/beam/pull/36036#discussion_r2320011622 ## sdks/python/apache_beam/yaml/examples/transforms/blueprint/iceberg_to_alloydb.yaml: ## @@ -0,0 +1,55 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Fou

Re: [I] [Bug]: JDBC logical type not found when running a Python pipeline in docker env [beam]

2025-09-03 Thread via GitHub
shunping closed issue #36013: [Bug]: JDBC logical type not found when running a Python pipeline in docker env URL: https://github.com/apache/beam/issues/36013 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319951676 ## sdks/python/apache_beam/io/jdbc.py: ## @@ -96,10 +95,11 @@ from apache_beam.transforms.external import BeamJarExpansionService from apache_beam.transforms.external

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping merged PR #36014: URL: https://github.com/apache/beam/pull/36014 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
derrickaw commented on code in PR #36036: URL: https://github.com/apache/beam/pull/36036#discussion_r2319948717 ## sdks/python/apache_beam/yaml/examples/transforms/blueprint/iceberg_to_alloydb.yaml: ## @@ -0,0 +1,55 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Founda

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
Abacn commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319939241 ## sdks/python/apache_beam/io/jdbc.py: ## @@ -96,10 +95,11 @@ from apache_beam.transforms.external import BeamJarExpansionService from apache_beam.transforms.external im

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
Abacn commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319910050 ## sdks/python/apache_beam/io/jdbc.py: ## @@ -96,10 +95,11 @@ from apache_beam.transforms.external import BeamJarExpansionService from apache_beam.transforms.external im

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
claudevdm commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319903383 ## sdks/python/apache_beam/io/jdbc.py: ## @@ -96,10 +95,11 @@ from apache_beam.transforms.external import BeamJarExpansionService from apache_beam.transforms.externa

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on PR #36034: URL: https://github.com/apache/beam/pull/36034#issuecomment-3250373545 > > where is this translation called > > when converting the transform to its proto representation: > > https://github.com/apache/beam/blob/d28fd643096d6bdb9db164684f58388a04

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on PR #36034: URL: https://github.com/apache/beam/pull/36034#issuecomment-3250343555 > > Will take a look later, but from a quick skim, it looks like we're missing translation logic in this PR > > Thanks for taking a look. I've been able to call both the managed and

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
ahmedabu98 commented on PR #36034: URL: https://github.com/apache/beam/pull/36034#issuecomment-3250362173 > where is this translation called when converting the transform to its proto representation: https://github.com/apache/beam/blob/d28fd643096d6bdb9db164684f58388a04a5b9b5/sdks/jav

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
ahmedabu98 commented on code in PR #36034: URL: https://github.com/apache/beam/pull/36034#discussion_r2319873888 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto: ## @@ -76,6 +76,10 @@ message ManagedTransforms { "beam:schematra

Re: [PR] feat(build): add validation task for CHANGES.md formatting [beam]

2025-09-03 Thread via GitHub
damccorm commented on code in PR #36041: URL: https://github.com/apache/beam/pull/36041#discussion_r2319798344 ## build.gradle.kts: ## @@ -637,6 +637,127 @@ tasks.register("formatChanges") { } } +tasks.register("validateChanges") { Review Comment: Ok, that works --

Re: [I] [Bug]: Prism isn't correctly used as the default for streaming pipelines. [beam]

2025-09-03 Thread via GitHub
damccorm commented on issue #34768: URL: https://github.com/apache/beam/issues/34768#issuecomment-3250270023 This has been fixed as part of making prism the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [I] [Bug]: Prism isn't correctly used as the default for streaming pipelines. [beam]

2025-09-03 Thread via GitHub
damccorm closed issue #34768: [Bug]: Prism isn't correctly used as the default for streaming pipelines. URL: https://github.com/apache/beam/issues/34768 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [I] [Task]: Enrich ML features such as RAG, Anomaly Detection [beam]

2025-09-03 Thread via GitHub
damccorm commented on issue #34553: URL: https://github.com/apache/beam/issues/34553#issuecomment-3250268790 I think we can call this done. We've added anomaly detection and RAG primitives. We will probably add more, but we've hit a reasonable level of completeness. -- This is an automat

Re: [I] [Task]: Enrich ML features such as RAG, Anomaly Detection [beam]

2025-09-03 Thread via GitHub
damccorm closed issue #34553: [Task]: Enrich ML features such as RAG, Anomaly Detection URL: https://github.com/apache/beam/issues/34553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] [Task]: Make Prism default [beam]

2025-09-03 Thread via GitHub
damccorm commented on issue #34549: URL: https://github.com/apache/beam/issues/34549#issuecomment-3250259135 I think we can call this done. There are corner cases we can burn down, but the core work is complete -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] [Task]: Make Prism default [beam]

2025-09-03 Thread via GitHub
damccorm closed issue #34549: [Task]: Make Prism default URL: https://github.com/apache/beam/issues/34549 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail

Re: [PR] feat(pubsub): support batch mode in WriteToPubSub transform [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36027: URL: https://github.com/apache/beam/pull/36027#issuecomment-3250255571 Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available c

Re: [I] [Beam I/O Standards] Avoid partially-built PTransform objects [beam]

2025-09-03 Thread via GitHub
damccorm closed issue #21510: [Beam I/O Standards] Avoid partially-built PTransform objects URL: https://github.com/apache/beam/issues/21510 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] [Beam I/O Standards] Avoid partially-built PTransform objects [beam]

2025-09-03 Thread via GitHub
damccorm commented on issue #21510: URL: https://github.com/apache/beam/issues/21510#issuecomment-3250252645 I think we probably will not pick this up, it doesn't seem super high value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [PR] feat(build): add validation task for CHANGES.md formatting [beam]

2025-09-03 Thread via GitHub
liferoad merged PR #36041: URL: https://github.com/apache/beam/pull/36041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] feat(build): add validation task for CHANGES.md formatting [beam]

2025-09-03 Thread via GitHub
liferoad commented on code in PR #36041: URL: https://github.com/apache/beam/pull/36041#discussion_r2319789019 ## build.gradle.kts: ## @@ -637,6 +637,127 @@ tasks.register("formatChanges") { } } +tasks.register("validateChanges") { Review Comment: sounds good. I will d

Re: [PR] add generics support to AutoValueUtils helpers [beam]

2025-09-03 Thread via GitHub
Abacn commented on PR #32977: URL: https://github.com/apache/beam/pull/32977#issuecomment-3250237374 related to #34672 (acutally this PR is the one referred in Beam 3 milestone doc https://s.apache.org/beam3-milestones: https://docs.google.com/document/d/1KfTMok67-bZBUcCCYeHkdktyQZ6arWQJEP

Re: [PR] feat(build): add validation task for CHANGES.md formatting [beam]

2025-09-03 Thread via GitHub
damccorm commented on code in PR #36041: URL: https://github.com/apache/beam/pull/36041#discussion_r2319774407 ## build.gradle.kts: ## @@ -637,6 +637,127 @@ tasks.register("formatChanges") { } } +tasks.register("validateChanges") { Review Comment: Could we make this a

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36036: URL: https://github.com/apache/beam/pull/36036#issuecomment-3250172906 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewer

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
tarun-google commented on PR #36036: URL: https://github.com/apache/beam/pull/36036#issuecomment-3250169853 R: @claudevdm @derrickaw @chamikaramj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[PR] feat(build): add validation task for CHANGES.md formatting [beam]

2025-09-03 Thread via GitHub
liferoad opened a new pull request, #36041: URL: https://github.com/apache/beam/pull/36041 Add a new gradle task 'validateChanges' that checks CHANGES.md follows required formatting rules including: - Language references must use parentheses - Each entry must have a proper issue link

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319658873 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = lo

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
Abacn commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319643908 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = logic

Re: [I] Get rid of jackson to avoid the continuous flow of CVEs in Jackson [beam]

2025-09-03 Thread via GitHub
Abacn closed issue #19707: Get rid of jackson to avoid the continuous flow of CVEs in Jackson URL: https://github.com/apache/beam/issues/19707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [I] Get rid of jackson to avoid the continuous flow of CVEs in Jackson [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19707: URL: https://github.com/apache/beam/issues/19707#issuecomment-3250035334 I think this is now obsolete. Beam and some of its popular dependencies (calcite, iceberg, hadoop) still use jackson databind. Even Beam itself moved out of jackson-databind user pipeli

Re: [I] [Task]: Enrich ML features such as RAG, Anomaly Detection [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34553: URL: https://github.com/apache/beam/issues/34553#issuecomment-3249996976 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Consider reducing dependencies in core [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34554: URL: https://github.com/apache/beam/issues/34554#issuecomment-3249997636 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Tracking Umbrella] Prism Runner areas for contribution. [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #29650: URL: https://github.com/apache/beam/issues/29650#issuecomment-3249994607 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Feature Request]: Update the default behaviour in `BigQueryIO.Write.Method.DEFAULT` to `STORAGE_API_AT_LEAST_ONCE` [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #31827: URL: https://github.com/apache/beam/issues/31827#issuecomment-3249998869 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Deprecate Java8 [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #31678: URL: https://github.com/apache/beam/issues/31678#issuecomment-3250012234 Move to milestone 3 as Java 8 has been deprecated in Beam 2.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

Re: [I] [Bug]: Prism isn't correctly used as the default for streaming pipelines. [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34768: URL: https://github.com/apache/beam/issues/34768#issuecomment-3250010462 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Remove support of Samza runner in new Beam versions [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #35448: URL: https://github.com/apache/beam/issues/35448#issuecomment-3250009464 Move to milestone 3 as Samza runner has been deprecated for Beam 2.x -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] [Task]: Upgrade avro to the latest version after Java 8 is deprecated [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34993: URL: https://github.com/apache/beam/issues/34993#issuecomment-3250006131 moved to Milestone 3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] [Task]: Upgrade avro to the latest version after Java 8 is deprecated [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34993: URL: https://github.com/apache/beam/issues/34993#issuecomment-3250005776 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Scope the work about improving Beam Python Typehints [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34671: URL: https://github.com/apache/beam/issues/34671#issuecomment-3250003746 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Bug]: Docker worker containers are confusingly named. [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34350: URL: https://github.com/apache/beam/issues/34350#issuecomment-3250005020 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Improve the schema support for Beam 3.0 [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34672: URL: https://github.com/apache/beam/issues/34672#issuecomment-3250004328 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Deduplicate Coders [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34637: URL: https://github.com/apache/beam/issues/34637#issuecomment-325408 This is essentially same FR as #18645 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] [Task]: Deduplicate Coders [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34637: URL: https://github.com/apache/beam/issues/34637#issuecomment-324597 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Default option changes for IO connectors (e.g., BigQuery IO), Runners (e.g., SA vs SE) [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34550: URL: https://github.com/apache/beam/issues/34550#issuecomment-3249996647 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Enrich Managed IO by onboarding more IOs and making Iceberg feature complete [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34552: URL: https://github.com/apache/beam/issues/34552#issuecomment-3249995996 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Bug]: Inline exception propagation with fused DoFns can lead to unexpected behavior [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34419: URL: https://github.com/apache/beam/issues/34419#issuecomment-3249993561 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Task]: Make Prism default [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #34549: URL: https://github.com/apache/beam/issues/34549#issuecomment-3249994052 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Obtain metadata field at once in file system's IO connectors [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #21551: URL: https://github.com/apache/beam/issues/21551#issuecomment-3249992103 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Remove TextIO deprecated API [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19389: URL: https://github.com/apache/beam/issues/19389#issuecomment-3249989420 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Unify serviceAccount pipeline option name [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19278: URL: https://github.com/apache/beam/issues/19278#issuecomment-3249988408 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Remove AvroIO deprecated API [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19388: URL: https://github.com/apache/beam/issues/19388#issuecomment-3249989160 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] [Beam I/O Standards] Avoid partially-built PTransform objects [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #21510: URL: https://github.com/apache/beam/issues/21510#issuecomment-3249990633 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Get rid of jackson to avoid the continuous flow of CVEs in Jackson [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19707: URL: https://github.com/apache/beam/issues/19707#issuecomment-3249990104 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Namespace for pipeline options [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19331: URL: https://github.com/apache/beam/issues/19331#issuecomment-3249988820 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Migrate to java.time lib instead of joda-time [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #19215: URL: https://github.com/apache/beam/issues/19215#issuecomment-3249987975 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Use JSON-B instead of an hardcoded Jackson [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #18868: URL: https://github.com/apache/beam/issues/18868#issuecomment-3249987148 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [I] Ensure coders are updatable partially in a cluster (aka add versions to coders) [beam]

2025-09-03 Thread via GitHub
Abacn commented on issue #18645: URL: https://github.com/apache/beam/issues/18645#issuecomment-3249986022 This Issue is marked as Beam 3 - Milestone 1: No Code and ML focus. Please help us burning down the open blocker tasks. * If this issue is solved, close it. If it is obsolete, clo

Re: [PR] feat(pubsub): support batch mode in WriteToPubSub transform [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36027: URL: https://github.com/apache/beam/pull/36027#issuecomment-3249980492 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] [KafkaIO] Only retrieve the shared caches once during setup in ReadFromKafkaDoFn [beam]

2025-09-03 Thread via GitHub
Abacn merged PR #36028: URL: https://github.com/apache/beam/pull/36028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36036: URL: https://github.com/apache/beam/pull/36036#issuecomment-3249880306 Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available comm

Re: [I] [Feature Request]: Support Protobuf 6.x in Python SDK [beam]

2025-09-03 Thread via GitHub
damccorm closed issue #35432: [Feature Request]: Support Protobuf 6.x in Python SDK URL: https://github.com/apache/beam/issues/35432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

Re: [PR] Support Protobuf 6.x. [beam]

2025-09-03 Thread via GitHub
damccorm commented on PR #35477: URL: https://github.com/apache/beam/pull/35477#issuecomment-3249862824 Merging since the cut has passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Support Protobuf 6.x. [beam]

2025-09-03 Thread via GitHub
damccorm merged PR #35477: URL: https://github.com/apache/beam/pull/35477 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Cloudpickle deterministic coder changes.md. [beam]

2025-09-03 Thread via GitHub
damccorm merged PR #36037: URL: https://github.com/apache/beam/pull/36037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319345662 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = lo

Re: [PR] feat(pubsub): support batch mode in WriteToPubSub transform [beam]

2025-09-03 Thread via GitHub
liferoad commented on code in PR #36027: URL: https://github.com/apache/beam/pull/36027#discussion_r2319087592 ## sdks/python/apache_beam/io/gcp/pubsub.py: ## @@ -541,11 +546,67 @@ def is_bounded(self): return False -# TODO(BEAM-27443): Remove in favor of a proper Write

Re: [PR] Add Iceberg to AlloyDB Blueprint [beam]

2025-09-03 Thread via GitHub
tarun-google commented on PR #36036: URL: https://github.com/apache/beam/pull/36036#issuecomment-3249695723 Run Python_Transforms PreCommit 3.10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [I] [Bug]: gprcio limitation to < 1.66 in Python is problematic [beam]

2025-09-03 Thread via GitHub
liferoad commented on issue #34081: URL: https://github.com/apache/beam/issues/34081#issuecomment-3249661919 This can be moved to 2.69 since we need the fix from grpcio. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [I] [Task]: Support more Beam portable schema types as Python types [beam]

2025-09-03 Thread via GitHub
VIKTORVAV99 commented on issue #25946: URL: https://github.com/apache/beam/issues/25946#issuecomment-3249616428 > > It's possible to work around in several ways but it just adds more code that should not be needed. > > @VIKTORVAV99 curious to know what approach you're taking! Are you

Re: [I] [Task]: Support more Beam portable schema types as Python types [beam]

2025-09-03 Thread via GitHub
ahmedabu98 commented on issue #25946: URL: https://github.com/apache/beam/issues/25946#issuecomment-3249608107 > It's possible to work around in several ways but it just adds more code that should not be needed. @VIKTORVAV99 curious to know what approach you're taking! Are you working

Re: [I] [Task]: Support more Beam portable schema types as Python types [beam]

2025-09-03 Thread via GitHub
ahmedabu98 commented on issue #25946: URL: https://github.com/apache/beam/issues/25946#issuecomment-3249604963 I've created a doc to outline new logical types for Timestamp and DateTime, with configurable precision levels: https://s.apache.org/beam-timestamp-strategy -- This is an automat

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
claudevdm commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319216933 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = l

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36034: URL: https://github.com/apache/beam/pull/36034#discussion_r2319173650 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto: ## @@ -76,6 +76,10 @@ message ManagedTransforms { "beam:schematrans

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36034: URL: https://github.com/apache/beam/pull/36034#discussion_r2319173650 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto: ## @@ -76,6 +76,10 @@ message ManagedTransforms { "beam:schematrans

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36034: URL: https://github.com/apache/beam/pull/36034#discussion_r2319173650 ## model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto: ## @@ -76,6 +76,10 @@ message ManagedTransforms { "beam:schematrans

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319163872 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = lo

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319163872 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = lo

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
shunping commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319163872 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = lo

Re: [PR] Bump cloud.google.com/go/spanner from 1.83.0 to 1.85.0 in /sdks [beam]

2025-09-03 Thread via GitHub
jrmccluskey merged PR #36005: URL: https://github.com/apache/beam/pull/36005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
claudevdm commented on code in PR #36014: URL: https://github.com/apache/beam/pull/36014#discussion_r2319125732 ## sdks/python/apache_beam/typehints/schemas.py: ## @@ -690,7 +690,16 @@ def add(self, urn, logical_type): self.by_language_type[logical_type.language_type()] = l

Re: [I] [Feature Request]: PrismRunner should support transform overrides [beam]

2025-09-03 Thread via GitHub
liferoad closed issue #36011: [Feature Request]: PrismRunner should support transform overrides URL: https://github.com/apache/beam/issues/36011 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [I] [Feature Request]: PrismRunner should support transform overrides [beam]

2025-09-03 Thread via GitHub
liferoad commented on issue #36011: URL: https://github.com/apache/beam/issues/36011#issuecomment-3249415149 After talking to @damccorm, we do not need overrides for PrismRunner for now since it will fall back to DirectRunner. -- This is an automated message from the Apache Git Service.

Re: [PR] Fix jdbc logical type not found when python sdk worker running in docker env [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36014: URL: https://github.com/apache/beam/pull/36014#issuecomment-3249413450 Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

Re: [PR] feat(pubsub): support batch mode in WriteToPubSub transform [beam]

2025-09-03 Thread via GitHub
liferoad commented on code in PR #36027: URL: https://github.com/apache/beam/pull/36027#discussion_r2319087592 ## sdks/python/apache_beam/io/gcp/pubsub.py: ## @@ -541,11 +546,67 @@ def is_bounded(self): return False -# TODO(BEAM-27443): Remove in favor of a proper Write

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
shunping commented on PR #36034: URL: https://github.com/apache/beam/pull/36034#issuecomment-3249381526 > Will take a look later, but from a quick skim, it looks like we're missing translation logic in this PR Thanks for taking a look. I've been able to call both the managed and unma

Re: [PR] feat(pubsub): add batch mode support for WriteToPubSub in DataflowRunner [beam]

2025-09-03 Thread via GitHub
liferoad commented on PR #36001: URL: https://github.com/apache/beam/pull/36001#issuecomment-3249379381 close this. https://github.com/apache/beam/pull/36027 is the recommended way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] feat(pubsub): add batch mode support for WriteToPubSub in DataflowRunner [beam]

2025-09-03 Thread via GitHub
liferoad closed pull request #36001: feat(pubsub): add batch mode support for WriteToPubSub in DataflowRunner URL: https://github.com/apache/beam/pull/36001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] Support managed jdbc io (Postgres) [beam]

2025-09-03 Thread via GitHub
github-actions[bot] commented on PR #36034: URL: https://github.com/apache/beam/pull/36034#issuecomment-3249303335 Assigning reviewers: R: @liferoad for label python. R: @Abacn for label java. Note: If you would like to opt out of this review, comment `assign to next rev

Re: [I] [Bug]: beam.io.gcp.bigquery.ReadFromBigQuery failing with permission issue [beam]

2025-09-03 Thread via GitHub
liferoad commented on issue #35957: URL: https://github.com/apache/beam/issues/35957#issuecomment-3249244730 https://beam.apache.org/blog/beam-2.66.0/ Python: Argument abbreviation is no longer enabled within Beam. If you previously abbreviated arguments (e.g. --r for --runner), you w

  1   2   3   4   5   6   7   8   9   10   >