Re: [PR] Move PeriodicStream into periodicsequence. [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35412: URL: https://github.com/apache/beam/pull/35412#issuecomment-2998785288 Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available c

Re: [PR] improved the error message when encoding_positions_set has errors [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35410: URL: https://github.com/apache/beam/pull/35410#issuecomment-2998674053 Assigning reviewers: R: @damccorm for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available comm

[PR] Move PeriodicStream into periodicsequence. [beam]

2025-06-23 Thread via GitHub
shunping opened a new pull request, #35412: URL: https://github.com/apache/beam/pull/35412 A follow-up PR of #35300 to address the concern at https://github.com/apache/beam/pull/35300#issuecomment-2997517699 -- This is an automated message from the Apache Git Service. To respond to the me

Re: [I] [Bug]: Got error 'ValueError: Schema with id has encoding_positions_set=True, but not all fields have encoding_position set' for no apparent reason [beam]

2025-06-23 Thread via GitHub
jonathaningram commented on issue #35318: URL: https://github.com/apache/beam/issues/35318#issuecomment-2998592037 It could be related, but I'm not really sure since I've also seen the same error on different occasions per the original messages above. -- This is an automated message from

Re: [PR] Deserialize proto based (mutable key range) change stream return records [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35408: URL: https://github.com/apache/beam/pull/35408#issuecomment-2998575749 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Add new change record action functions [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35409: URL: https://github.com/apache/beam/pull/35409#issuecomment-2998477156 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [I] [Bug]: Got error 'ValueError: Schema with id has encoding_positions_set=True, but not all fields have encoding_position set' for no apparent reason [beam]

2025-06-23 Thread via GitHub
liferoad commented on issue #35318: URL: https://github.com/apache/beam/issues/35318#issuecomment-2998474410 Good idea. Put a PR https://github.com/apache/beam/pull/35410 to improve the error message. -- This is an automated message from the Apache Git Service. To respond to the message,

[PR] improved the error message when encoding_positions_set has errors [beam]

2025-06-23 Thread via GitHub
liferoad opened a new pull request, #35410: URL: https://github.com/apache/beam/pull/35410 Addresses https://github.com/apache/beam/issues/35318#issuecomment-2998390616 Thank you for your contribution! Follow this checklist to help us incorporate your c

Re: [I] How do I compress the output of using the WriteToFiles method? [beam]

2025-06-23 Thread via GitHub
serratedserenade commented on issue #35363: URL: https://github.com/apache/beam/issues/35363#issuecomment-2998435255 Ok, just going to report that the code sample you provided for gzip didn't really work. It does write but I'm unable to decompress it from CLI or in Python with the gzip libr

Re: [I] How do I compress the output of using the WriteToFiles method? [beam]

2025-06-23 Thread via GitHub
serratedserenade commented on issue #35363: URL: https://github.com/apache/beam/issues/35363#issuecomment-2998398294 > Doesn't `shard_name_template` work for you? No, what I like about `WriteToFiles` is that you can make the files go to different outputs based on the record being writ

Re: [I] [Bug]: Got error 'ValueError: Schema with id has encoding_positions_set=True, but not all fields have encoding_position set' for no apparent reason [beam]

2025-06-23 Thread via GitHub
jonathaningram commented on issue #35318: URL: https://github.com/apache/beam/issues/35318#issuecomment-2998393501 For the record, my current workaround is to drain the existing pipeline via the Console, then wait 5 mins or so until it's drained. Then I apply my terraform again to force a n

Re: [I] [Bug]: Got error 'ValueError: Schema with id has encoding_positions_set=True, but not all fields have encoding_position set' for no apparent reason [beam]

2025-06-23 Thread via GitHub
jonathaningram commented on issue #35318: URL: https://github.com/apache/beam/issues/35318#issuecomment-2998390616 This happened again when adding `timestamp_attribute` to my Pub/Sub write stage. Before: ```yaml - type: WriteToPubSub name: WriteToPubSub input: MyIn

[PR] Add new change record action functions [beam]

2025-06-23 Thread via GitHub
changliiu opened a new pull request, #35409: URL: https://github.com/apache/beam/pull/35409 Add new change recored actions include: 1. PartitionStartRecord -> create new partitions in the metadata tables to be scheduled. Update watermark 2. PartitionEndRecord, PartitionEventRecord -> u

[PR] Deserialize proto based (mutable key range) change stream return records [beam]

2025-06-23 Thread via GitHub
changliiu opened a new pull request, #35408: URL: https://github.com/apache/beam/pull/35408 Parse proto type change stream return records into connector models. Inlcuding: 1. DataChangeRecord 2. HeartbeatRecord 3. PartitionStartRecord 4. PartitionEndRecord 5. PartitionEv

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
Abacn merged PR #35258: URL: https://github.com/apache/beam/pull/35258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

[PR] add a thread to flush buffered metrics [beam]

2025-06-23 Thread via GitHub
Naireen opened a new pull request, #35407: URL: https://github.com/apache/beam/pull/35407 Moved metrics flushing to a background thread so it could be done less frequently Exposed property of metrics collection so we could check if it needed to be reinitialized to store new metrics.

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
dedocibula commented on code in PR #35258: URL: https://github.com/apache/beam/pull/35258#discussion_r2162660571 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/model/PartitionEventRecordTest.java: ## @@ -0,0 +1,44 @@ +/* + *

Re: [I] [Bug]: Prism firing windows incorrectly when sliding windows are used [beam]

2025-06-23 Thread via GitHub
lostluck closed issue #35384: [Bug]: Prism firing windows incorrectly when sliding windows are used URL: https://github.com/apache/beam/issues/35384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

[PR] ]Cherrypick] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
Amar3tto opened a new pull request, #35402: URL: https://github.com/apache/beam/pull/35402 (cherry picked from commit 2feb407ea0ef8b5962ca6061955d14acb404803a) Thank you for your contribution! Follow this checklist to help us incorporate your contri

[I] Performance Regression or Improvement: sideinpts_python_batch_10gb_1kb_10workers_1000window_first_iterable:runtime [beam]

2025-06-23 Thread via GitHub
github-actions[bot] opened a new issue, #35406: URL: https://github.com/apache/beam/issues/35406 Performance change found in the test: `sideinpts_python_batch_10gb_1kb_10workers_1000window_first_iterable` for the metric: `runtime`. For more information on how to triage t

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2998097476 Run Python_Examples PreCommit 3.10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2998094219 Run Python_Examples PreCommit 3.10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t

Re: [PR] [Cherrypick] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35402: URL: https://github.com/apache/beam/pull/35402#issuecomment-2997720448 Assigning reviewers: R: @m-trieu for label java. R: @Dippatel98 for label kafka. Note: If you would like to opt out of this review, comment `assign to next

Re: [PR] Sliding window ptransform for time series data [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35379: URL: https://github.com/apache/beam/pull/35379#discussion_r2162019919 ## sdks/python/apache_beam/ml/ts/ordered_sliding_window.py: ## @@ -0,0 +1,154 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] Improve LogElements to show pane_info and timestamps in seconds. [beam]

2025-06-23 Thread via GitHub
shunping merged PR #35387: URL: https://github.com/apache/beam/pull/35387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [I] [Feature Request]: Add the job name to userAgentString for Spanner IO [beam]

2025-06-23 Thread via GitHub
stephenmw commented on issue #35401: URL: https://github.com/apache/beam/issues/35401#issuecomment-2997832314 I talked with @liferoad offline. I think we should use job ID instead of job name. Job IDs are globally unique making them easier to work with and contain no user generated content.

[PR] Reduce issues induced by ordering [beam]

2025-06-23 Thread via GitHub
damccorm opened a new pull request, #35405: URL: https://github.com/apache/beam/pull/35405 Ordering was introduced into this workflow to prevent race conditions as multiple jobs write to svn. However, the wrong ordering was applied, and staging python and prism to svn are blocked on pushing

[I] [Feature Request]: Add the job name to userAgentString for Spanner IO [beam]

2025-06-23 Thread via GitHub
liferoad opened a new issue, #35401: URL: https://github.com/apache/beam/issues/35401 ### What would you like to happen? https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerAccessor.java#L231 We c

Re: [PR] Sliding window ptransform for time series data [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35379: URL: https://github.com/apache/beam/pull/35379#issuecomment-2997720721 Assigning reviewers: R: @claudevdm for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

Re: [PR] Bump google.golang.org/api from 0.237.0 to 0.238.0 in /sdks [beam]

2025-06-23 Thread via GitHub
lostluck merged PR #35368: URL: https://github.com/apache/beam/pull/35368 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
changliiu commented on code in PR #35258: URL: https://github.com/apache/beam/pull/35258#discussion_r2162436865 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/model/PartitionEventRecordTest.java: ## @@ -0,0 +1,44 @@ +/* + * L

Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]

2025-06-23 Thread via GitHub
damccorm commented on code in PR #34912: URL: https://github.com/apache/beam/pull/34912#discussion_r2162327101 ## website/www/site/content/en/documentation/programming-guide.md: ## @@ -1183,6 +1183,100 @@ func init() { > parameters to a single `emitter function`. +{{< parag

Re: [PR] Disable dependency logging in local envs [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35404: URL: https://github.com/apache/beam/pull/35404#issuecomment-2997875077 Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available com

Re: [PR] Add PeriodicStream in the new time series folder. [beam]

2025-06-23 Thread via GitHub
shunping commented on PR #35300: URL: https://github.com/apache/beam/pull/35300#issuecomment-2997958630 > I just came across this code and I have a couple of concerns: > > 1. Why are we creating a new transform instead of extending periodicImpulse? It seems like the same goals could b

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
dedocibula commented on code in PR #35258: URL: https://github.com/apache/beam/pull/35258#discussion_r2162423800 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/model/PartitionEventRecordTest.java: ## @@ -0,0 +1,44 @@ +/* + *

Re: [PR] reconnect when publishMessage in JMS fails [beam]

2025-06-23 Thread via GitHub
Abacn commented on code in PR #35386: URL: https://github.com/apache/beam/pull/35386#discussion_r2162461490 ## sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java: ## @@ -1226,8 +1226,6 @@ void connect() throws JMSException { } else if (spec.getTopic(

Re: [PR] Add PeriodicStream in the new time series folder. [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35300: URL: https://github.com/apache/beam/pull/35300#issuecomment-2997517699 I just came across this code and I have a couple of concerns: 1) Why are we creating a new transform instead of extending periodicImpulse? It seems like the same goals could be pret

Re: [PR] Reduce issues induced by ordering [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35405: URL: https://github.com/apache/beam/pull/35405#issuecomment-2997874901 Assigning reviewers: R: @Abacn for label build. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available commands

Re: [PR] Add checkpoint during progress reporting. [beam]

2025-06-23 Thread via GitHub
lostluck commented on PR #34828: URL: https://github.com/apache/beam/pull/34828#issuecomment-2997803433 waiting on author (since the bot doesn't know how drafts work) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Fix The PostCommit Python Xlang Gcp Direct job is flaky [beam]

2025-06-23 Thread via GitHub
Amar3tto commented on PR #35396: URL: https://github.com/apache/beam/pull/35396#issuecomment-2997728884 I don't think we want to fix one issue and undo two other fixes, so it's better to try fix them all -- This is an automated message from the Apache Git Service. To respond to the messag

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
dedocibula commented on code in PR #35258: URL: https://github.com/apache/beam/pull/35258#discussion_r2162423800 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/model/PartitionEventRecordTest.java: ## @@ -0,0 +1,44 @@ +/* + *

Re: [PR] Add change stream new record models and tests. [beam]

2025-06-23 Thread via GitHub
dedocibula commented on code in PR #35258: URL: https://github.com/apache/beam/pull/35258#discussion_r2162423800 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/model/PartitionEventRecordTest.java: ## @@ -0,0 +1,44 @@ +/* + *

Re: [PR] [Cherrypick] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
Amar3tto merged PR #35402: URL: https://github.com/apache/beam/pull/35402 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Improve exception raised from cache [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35403: URL: https://github.com/apache/beam/pull/35403#issuecomment-2997720297 Assigning reviewers: R: @liferoad for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available comm

[PR] Disable dependency logging in local envs [beam]

2025-06-23 Thread via GitHub
damccorm opened a new pull request, #35404: URL: https://github.com/apache/beam/pull/35404 Context: https://lists.apache.org/thread/rrpdj3b32cz8qhr23x82q9fbg0rn8crr Thank you for your contribution! Follow this checklist to help us incorporate your contr

Re: [I] [Feature Request]: Add library stubs or py.typed marker [beam]

2025-06-23 Thread via GitHub
cnsgsz commented on issue #27906: URL: https://github.com/apache/beam/issues/27906#issuecomment-2997658007 +1, like https://pypi.org/project/pandas-stubs/1.2.0.39/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

Re: [PR] Refactor: separate SplittableTruncateSizedRestrictions [beam]

2025-06-23 Thread via GitHub
kennknowles commented on code in PR #35021: URL: https://github.com/apache/beam/pull/35021#discussion_r2159157147 ## sdks/java/harness/src/main/java/org/apache/beam/fn/harness/SplittableTruncateSizedRestrictionsDoFnRunner.java: ## @@ -0,0 +1,1016 @@ +/* + * Licensed to the Apach

Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]

2025-06-23 Thread via GitHub
damccorm commented on code in PR #34912: URL: https://github.com/apache/beam/pull/34912#discussion_r2162326649 ## website/www/site/content/en/documentation/programming-guide.md: ## @@ -1184,6 +1184,101 @@ func init() { +{{< paragraph class="language-python">}} +Proper Use

Re: [PR] Enable prism by default (where supported) [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #34612: URL: https://github.com/apache/beam/pull/34612#issuecomment-2997595530 Current persistent (or possibly persistent) test failures: **[beam_PreCommit_Python_ML](https://github.com/apache/beam/actions/runs/1583506/job/44623727892?pr=34612#logs)** and

Re: [PR] Improve LogElements to show pane_info and timestamps in seconds. [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35387: URL: https://github.com/apache/beam/pull/35387#discussion_r2162321085 ## sdks/python/apache_beam/transforms/util.py: ## @@ -1445,30 +1445,50 @@ class LogElements(PTransform): level: (optional) The logging level for the output (e.g. `

Re: [PR] Improve LogElements to show pane_info and timestamps in seconds. [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35387: URL: https://github.com/apache/beam/pull/35387#discussion_r2162319641 ## sdks/python/apache_beam/transforms/util.py: ## @@ -1445,30 +1445,50 @@ class LogElements(PTransform): level: (optional) The logging level for the output (e.g. `

Re: [PR] ✨[Feature]: Add pip-based install support for JupyterLab side panel extension [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35399: URL: https://github.com/apache/beam/pull/35399#issuecomment-2997167346 Assigning reviewers: R: @jrmccluskey for label python. Note: If you would like to opt out of this review, comment `assign to next reviewer`. Available c

Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]

2025-06-23 Thread via GitHub
YashaswiniTB commented on PR #34912: URL: https://github.com/apache/beam/pull/34912#issuecomment-2997502090 > waiting on author Thanks @damccorm! I believe I’ve addressed all the comments. Please let me know if anything else is required -- This is an automated message from the Apac

Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #34912: URL: https://github.com/apache/beam/pull/34912#issuecomment-2997413093 waiting on author -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add BigQuery Storage Read API Enrichment Handler [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35197: URL: https://github.com/apache/beam/pull/35197#issuecomment-2997397422 > > > I’ve noticed that wherever this module is used, the Beam pipeline is skipping those tests > > > > > > @pandasanjay I found it depends mostly on the GitHub runner/workflow

[PR] Improve exception raised from cache [beam]

2025-06-23 Thread via GitHub
damccorm opened a new pull request, #35403: URL: https://github.com/apache/beam/pull/35403 When we fail an instruction, we tell the instruction cache that we've failed it, but we don't provide details. This makes it harder to raise a good exception, but exceptions can still bubble up from t

Re: [PR] Sliding window ptransform for time series data [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35379: URL: https://github.com/apache/beam/pull/35379#discussion_r2162073434 ## sdks/python/apache_beam/ml/ts/ordered_sliding_window.py: ## @@ -0,0 +1,166 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] ✨[Feature]: Add pip-based install support for JupyterLab side panel extension [beam]

2025-06-23 Thread via GitHub
liferoad commented on PR #35399: URL: https://github.com/apache/beam/pull/35399#issuecomment-2997352197 Please fix the failed workflows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Exactly one of Topic or Subscription must be set in pubsubio.go [beam]

2025-06-23 Thread via GitHub
lostluck commented on code in PR #35369: URL: https://github.com/apache/beam/pull/35369#discussion_r2162118152 ## sdks/go/pkg/beam/io/pubsubio/pubsubio.go: ## @@ -52,32 +52,50 @@ func init() { // ReadOptions represents options for reading from PubSub. type ReadOptions struct

Re: [I] [Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job [beam]

2025-06-23 Thread via GitHub
liferoad commented on issue #34270: URL: https://github.com/apache/beam/issues/34270#issuecomment-2997342012 Thanks, @quentin-sommer, for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Fix The PostCommit Python Xlang Gcp Direct job is flaky [beam]

2025-06-23 Thread via GitHub
Abacn commented on PR #35396: URL: https://github.com/apache/beam/pull/35396#issuecomment-2997115380 This one is in Beam 2.66 release branch. We probably need to cherry pick also cc: @Amar3tto @damccorm -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
Amar3tto merged PR #35394: URL: https://github.com/apache/beam/pull/35394 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Fix a logical type issue about JdbcDateType and JdbcTimeType [beam]

2025-06-23 Thread via GitHub
claudevdm commented on code in PR #35243: URL: https://github.com/apache/beam/pull/35243#discussion_r2162113182 ## sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py: ## @@ -255,10 +253,6 @@ def test_xlang_jdbc_write_read(self, database): classpath=config

Re: [PR] Fix the issue of sliding window firing early. [beam]

2025-06-23 Thread via GitHub
lostluck merged PR #35385: URL: https://github.com/apache/beam/pull/35385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

[PR] Minstant [beam]

2025-06-23 Thread via GitHub
claudevdm opened a new pull request, #35400: URL: https://github.com/apache/beam/pull/35400 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quic

Re: [I] [Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job [beam]

2025-06-23 Thread via GitHub
quentin-sommer commented on issue #34270: URL: https://github.com/apache/beam/issues/34270#issuecomment-2997240568 my workaround was to be more careful with the data I sent. Now hopefully this will be fixed -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] Sliding window ptransform for time series data [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35379: URL: https://github.com/apache/beam/pull/35379#discussion_r2162079827 ## sdks/python/apache_beam/ml/ts/ordered_sliding_window.py: ## @@ -0,0 +1,166 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [I] The PostCommit Python Xlang IO Direct job is flaky [beam]

2025-06-23 Thread via GitHub
Amar3tto commented on issue #32809: URL: https://github.com/apache/beam/issues/32809#issuecomment-2997195541 Job has been stable for 3 days. Closing as resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Extend Schema Registry Support on Managed Kafka I/O to Google's Managed Schema Registry Solution [beam]

2025-06-23 Thread via GitHub
Abacn commented on code in PR #35085: URL: https://github.com/apache/beam/pull/35085#discussion_r2162065687 ## sdks/java/io/kafka/build.gradle: ## @@ -107,6 +107,9 @@ dependencies { testImplementation library.java.testcontainers_kafka testRuntimeOnly library.java.slf4j_jdk

Re: [I] The PostCommit Python Xlang IO Direct job is flaky [beam]

2025-06-23 Thread via GitHub
Amar3tto closed issue #32809: The PostCommit Python Xlang IO Direct job is flaky URL: https://github.com/apache/beam/issues/32809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Extend Schema Registry Support on Managed Kafka I/O to Google's Managed Schema Registry Solution [beam]

2025-06-23 Thread via GitHub
Abacn commented on code in PR #35085: URL: https://github.com/apache/beam/pull/35085#discussion_r2162024328 ## sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java: ## @@ -178,16 +180,45 @@ public PCollectionRowTuple expand(PCollect

Re: [I] The LoadTests Java PubsubIO job is flaky [beam]

2025-06-23 Thread via GitHub
Abacn closed issue #35194: The LoadTests Java PubsubIO job is flaky URL: https://github.com/apache/beam/issues/35194 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscr

Re: [PR] Fix PubsubIOLT [beam]

2025-06-23 Thread via GitHub
Abacn merged PR #35372: URL: https://github.com/apache/beam/pull/35372 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2997087359 > If you have an example of the tests running on ubuntu-latest that you can link to eventually, that will help as well This one was on `ubuntu-latest` which is ubuntu version

Re: [PR] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
damccorm commented on code in PR #35394: URL: https://github.com/apache/beam/pull/35394#discussion_r2161978451 ## examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/SchemasUtils.java: ## @@ -88,13 +87,16 @@ private void parseJson(String jsonSche

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2997048891 If you have an example of the tests running on ubuntu-latest that you can link to eventually, that will help as well -- This is an automated message from the Apache Git Service. To resp

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2997045270 > Hey @damccorm, > > Most of the main tasks in this PR are nearly complete. However, the `beam_PreCommit_Python_ML` job is currently failing due to an issue related to Docker conta

Re: [PR] Sliding window ptransform for time series data [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35379: URL: https://github.com/apache/beam/pull/35379#discussion_r2161955731 ## sdks/python/apache_beam/ml/ts/ordered_sliding_window.py: ## @@ -0,0 +1,154 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] Fix suppressed Spotbugs issues [beam]

2025-06-23 Thread via GitHub
Amar3tto commented on code in PR #35394: URL: https://github.com/apache/beam/pull/35394#discussion_r2161937397 ## examples/java/src/main/java/org/apache/beam/examples/complete/datatokenization/utils/SchemasUtils.java: ## @@ -88,13 +87,16 @@ private void parseJson(String jsonSche

Re: [PR] Try reenabling flaky test [beam]

2025-06-23 Thread via GitHub
damccorm closed pull request #35398: Try reenabling flaky test URL: https://github.com/apache/beam/pull/35398 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] Try reenabling flaky test [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35398: URL: https://github.com/apache/beam/pull/35398#issuecomment-2996941674 Still failing - https://github.com/apache/beam/actions/runs/15828254640/job/44614043093?pr=35398 - though it passed on the first run. So there is likely still a bug here cc/ @lostl

Re: [I] [Bug]: [Python] Respect BigQuery insert byte size limit when writing batched rows [beam]

2025-06-23 Thread via GitHub
liferoad closed issue #27363: [Bug]: [Python] Respect BigQuery insert byte size limit when writing batched rows URL: https://github.com/apache/beam/issues/27363 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [I] [Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job [beam]

2025-06-23 Thread via GitHub
liferoad closed issue #34270: [Bug]: Bigquery python streaming insertAll SSLError leads to stuck streaming job URL: https://github.com/apache/beam/issues/34270 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Estimate req size in BQ streaming when auto sharding is true [beam]

2025-06-23 Thread via GitHub
liferoad merged PR #35212: URL: https://github.com/apache/beam/pull/35212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2996927141 > This runs fine on a typical GitHub hosted Ubuntu runner I have written a workflow for this `Run Milvus Integration Tests` and all pass on GitHub hosted ubuntu runner versio

Re: [PR] Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.80.3 to 1.81.0 in /sdks [beam]

2025-06-23 Thread via GitHub
jrmccluskey merged PR #35366: URL: https://github.com/apache/beam/pull/35366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

Re: [PR] Try reenabling flaky test [beam]

2025-06-23 Thread via GitHub
damccorm commented on PR #35398: URL: https://github.com/apache/beam/pull/35398#issuecomment-2996898567 Looks like https://github.com/apache/beam/pull/35337 probably fixed this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[PR] ✨[Feature]: Add pip-based install support for JupyterLab side panel extension [beam]

2025-06-23 Thread via GitHub
Chenzo1001 opened a new pull request, #35399: URL: https://github.com/apache/beam/pull/35399 Summary This PR enables the `apache-beam-jupyterlab-sidepanel` JupyterLab extension to be installed via `pip` directly, without requiring the use of `jupyter labextension install`. Fixing the iss

[PR] Try reenabling flaky test [beam]

2025-06-23 Thread via GitHub
damccorm opened a new pull request, #35398: URL: https://github.com/apache/beam/pull/35398 Seeing if recent fixes solved the flakiness problem Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and ea

Re: [I] How do I compress the output of using the WriteToFiles method? [beam]

2025-06-23 Thread via GitHub
liferoad commented on issue #35363: URL: https://github.com/apache/beam/issues/35363#issuecomment-2996720960 Doesn't `shard_name_template` work for you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [I] [Feature Request]: Installing Jupyter Sidepanel by pip [beam]

2025-06-23 Thread via GitHub
Chenzo1001 commented on issue #35397: URL: https://github.com/apache/beam/issues/35397#issuecomment-2996711767 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

[I] [Feature Request]: Installing Jupyter Sidepanel by pip [beam]

2025-06-23 Thread via GitHub
Chenzo1001 opened a new issue, #35397: URL: https://github.com/apache/beam/issues/35397 ### What would you like to happen? Currently, the apache-beam-jupyterlab-sidepanel extension is installed via the deprecated jupyter labextension install mechanism. Since JupyterLab 4 officially r

Re: [PR] Estimate req size in BQ streaming when auto sharding is true [beam]

2025-06-23 Thread via GitHub
liferoad commented on PR #35212: URL: https://github.com/apache/beam/pull/35212#issuecomment-2996708734 > When I run > > ``` > pytest apache_beam/io/gcp/bigquery_test.py > ``` > > I don't reproduce the error in the action log line [9576](https://github.com/apache/beam/act

Re: [PR] Fix a logical type issue about JdbcDateType and JdbcTimeType [beam]

2025-06-23 Thread via GitHub
shunping commented on code in PR #35243: URL: https://github.com/apache/beam/pull/35243#discussion_r2161737747 ## sdks/python/apache_beam/io/external/xlang_jdbcio_it_test.py: ## @@ -255,10 +253,6 @@ def test_xlang_jdbc_write_read(self, database): classpath=config[

Re: [I] CVE-2025-4565 in protobuf<6.31.1 used by Python SDK [beam]

2025-06-23 Thread via GitHub
tpwo2 closed issue #35328: CVE-2025-4565 in protobuf<6.31.1 used by Python SDK URL: https://github.com/apache/beam/issues/35328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] Fix The PostCommit Python Xlang Gcp Direct job is flaky [beam]

2025-06-23 Thread via GitHub
github-actions[bot] commented on PR #35396: URL: https://github.com/apache/beam/pull/35396#issuecomment-2996646532 Assigning reviewers: R: @shunping for label python. R: @Abacn for label build. Note: If you would like to opt out of this review, comment `assign to next re

Re: [PR] Improve LogElements to show pane_info and timestamps in seconds. [beam]

2025-06-23 Thread via GitHub
damccorm commented on code in PR #35387: URL: https://github.com/apache/beam/pull/35387#discussion_r2161683527 ## sdks/python/apache_beam/transforms/util.py: ## @@ -1445,30 +1445,50 @@ class LogElements(PTransform): level: (optional) The logging level for the output (e.g. `

Re: [PR] sdks/python: enrich data with Milvus Search [Vector, Keyword, Hybrid] [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35216: URL: https://github.com/apache/beam/pull/35216#issuecomment-2996596518 Hey @damccorm, Most of the main tasks in this PR are nearly complete. However, the `beam_PreCommit_Python_ML` job is currently failing due to an issue related to Docker cont

Re: [PR] sdks/python: Check if ML RAG Integration Tests Ever Run in the CI [beam]

2025-06-23 Thread via GitHub
claudevdm commented on PR #35395: URL: https://github.com/apache/beam/pull/35395#issuecomment-2996572024 Actually, it runs as part of .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Direct.json Its not super obvious where tests run so thanks for double checking https://git

Re: [PR] sdks/python: Check if ML RAG Integration Tests Ever Run in the CI [beam]

2025-06-23 Thread via GitHub
mohamedawnallah commented on PR #35395: URL: https://github.com/apache/beam/pull/35395#issuecomment-2996567504 > I think since these are marked as integration tests (file _it_test.py) they run in the PostCommit python suite. They dont really rely on the ML deps. It is similar to other enric

  1   2   >