Re: [I] Can't use side inputs with Combine.PerKeyWithHotKeyFanout [beam]

2023-10-05 Thread via GitHub
marc7806 commented on issue #20637: URL: https://github.com/apache/beam/issues/20637#issuecomment-1750064914 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Setup Java 21 container [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28833: URL: https://github.com/apache/beam/pull/28833#issuecomment-1750009917 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Bump golang.org/x/sync from 0.3.0 to 0.4.0 in /sdks [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28856: URL: https://github.com/apache/beam/pull/28856#issuecomment-1750009851 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @jrmccluskey for label go. Available commands: - `stop r

Re: [PR] Bump golang.org/x/sync from 0.3.0 to 0.4.0 in /sdks [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28856: URL: https://github.com/apache/beam/pull/28856#issuecomment-1750001467 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28856?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] Bump golang.org/x/net from 0.15.0 to 0.16.0 in /sdks [beam]

2023-10-05 Thread via GitHub
lostluck commented on PR #28858: URL: https://github.com/apache/beam/pull/28858#issuecomment-1749996118 It looks like I ended up merging a the dependabot change, and then fat fingered a merge of a revert commit. Dependabot will generate a new PR, to resolve this. -- This is an auto

Re: [PR] Bump golang.org/x/net from 0.15.0 to 0.16.0 in /sdks [beam]

2023-10-05 Thread via GitHub
lostluck merged PR #28858: URL: https://github.com/apache/beam/pull/28858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Bump golang.org/x/sync from 0.3.0 to 0.4.0 in /sdks [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28856: URL: https://github.com/apache/beam/pull/28856#issuecomment-1749990210 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] Bump github.com/proullon/ramsql from 0.1.2 to 0.1.3 in /sdks [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28857: URL: https://github.com/apache/beam/pull/28857#issuecomment-1749990139 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @riteshghorse for label go. Available commands: - `stop

Re: [PR] Bump golang.org/x/net from 0.15.0 to 0.16.0 in /sdks [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28858: URL: https://github.com/apache/beam/pull/28858#issuecomment-1749990077 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @lostluck for label go. Available commands: - `stop revi

Re: [PR] Bump github.com/proullon/ramsql from 0.1.2 to 0.1.3 in /sdks [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28857: URL: https://github.com/apache/beam/pull/28857#issuecomment-1749972185 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28857?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] Bump golang.org/x/net from 0.15.0 to 0.16.0 in /sdks [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28858: URL: https://github.com/apache/beam/pull/28858#issuecomment-1749970557 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28858?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[PR] Bump golang.org/x/net from 0.15.0 to 0.16.0 in /sdks [beam]

2023-10-05 Thread via GitHub
dependabot[bot] opened a new pull request, #28858: URL: https://github.com/apache/beam/pull/28858 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.15.0 to 0.16.0. Commits https://github.com/golang/net/commit/88194ad8ab44a02ea952c169883c3f57db6cf9f4";>88194ad go.m

[PR] Bump github.com/proullon/ramsql from 0.1.2 to 0.1.3 in /sdks [beam]

2023-10-05 Thread via GitHub
dependabot[bot] opened a new pull request, #28857: URL: https://github.com/apache/beam/pull/28857 Bumps [github.com/proullon/ramsql](https://github.com/proullon/ramsql) from 0.1.2 to 0.1.3. Release notes Sourced from https://github.com/proullon/ramsql/releases";>github.com/proullon

[PR] Bump golang.org/x/sync from 0.3.0 to 0.4.0 in /sdks [beam]

2023-10-05 Thread via GitHub
dependabot[bot] opened a new pull request, #28856: URL: https://github.com/apache/beam/pull/28856 Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.3.0 to 0.4.0. Commits https://github.com/golang/sync/commit/22ba2078e183beec12908ea94f1d899c53dbf02c";>22ba207 sin

Re: [PR] [#20970] Fix gRPC leak by closing ResidualSource at BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker [beam]

2023-10-05 Thread via GitHub
baeminbo commented on PR #28548: URL: https://github.com/apache/beam/pull/28548#issuecomment-1749926173 Run Java_GCP_IO_Direct PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on code in PR #28385: URL: https://github.com/apache/beam/pull/28385#discussion_r1347863097 ## sdks/python/scripts/run_tox.sh: ## @@ -53,18 +53,27 @@ if [[ "$JENKINS_HOME" != "" ]]; then export PY_COLORS=1 fi -if [[ ! -z $2 ]]; then +# Determine if th

Re: [PR] Bump snappy-java version to 1.1.10.4 [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28655: URL: https://github.com/apache/beam/pull/28655#issuecomment-1749903894 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @Abacn for label build. Available commands: - `stop revi

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
Abacn merged PR #28846: URL: https://github.com/apache/beam/pull/28846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Setup Java 21 container [beam]

2023-10-05 Thread via GitHub
Abacn commented on code in PR #28833: URL: https://github.com/apache/beam/pull/28833#discussion_r1348175837 ## sdks/java/container/java21/option-jamm.json: ## @@ -0,0 +1,10 @@ +{ Review Comment: although same options are added as java17, it seems jamm still not working on J

Re: [PR] [Draft] Setup Java 21 container [beam]

2023-10-05 Thread via GitHub
Abacn commented on code in PR #28833: URL: https://github.com/apache/beam/pull/28833#discussion_r1348174284 ## .github/workflows/beam_LoadTests_Java_GBK_Dataflow_V2_Batch_Java11.yml: ## @@ -95,7 +95,7 @@ jobs: -PloadTest.mainClass=org.apache.beam.sdk.loadtests.Grou

Re: [PR] Bump snappy-java version to 1.1.10.4 [beam]

2023-10-05 Thread via GitHub
bvolpato commented on PR #28655: URL: https://github.com/apache/beam/pull/28655#issuecomment-1749887254 Run Java_Pulsar_IO_Direct PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spe

Re: [PR] [Draft] Setup Java 21 container [beam]

2023-10-05 Thread via GitHub
Abacn commented on PR #28833: URL: https://github.com/apache/beam/pull/28833#issuecomment-1749886259 Example wordcount job (maven project) on Dataflow: https://console.cloud.google.com/dataflow/jobs/us-central1/2023-10-05_17_20_31-16427189156223040319?project=google.com%3Aclouddfe ![

Re: [PR] [YAML] Schemify avroio and add a yaml provider. [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28855: URL: https://github.com/apache/beam/pull/28855#issuecomment-1749843924 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28855?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] [YAML] Schemify avroio and add a yaml provider. [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28855: URL: https://github.com/apache/beam/pull/28855#issuecomment-1749834171 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [YAML] Schemify avroio and add a yaml provider. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28855: URL: https://github.com/apache/beam/pull/28855#issuecomment-1749833181 R: @riteshghorse -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] [BEAM-28684] [YAML] Add Parquet reading and writing. [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28850: URL: https://github.com/apache/beam/pull/28850#issuecomment-1749831775 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [BEAM-28684] [YAML] Add Parquet reading and writing. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28850: URL: https://github.com/apache/beam/pull/28850#issuecomment-1749829092 R: @anandinguva -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] [YAML] Java PubSubIO [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28782: URL: https://github.com/apache/beam/pull/28782#issuecomment-1749828070 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] [YAML] Java PubSubIO [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28782: URL: https://github.com/apache/beam/pull/28782#issuecomment-1749825355 R: @chamikaramj -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [PR] [YAML] Schemify avroio and add a yaml provider. [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28855: URL: https://github.com/apache/beam/pull/28855#issuecomment-1749823243 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] [YAML] Add JSON parsing to PubSub IO. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28754: URL: https://github.com/apache/beam/pull/28754#issuecomment-1749817986 R: @liferoad -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsu

Re: [PR] Populate top-level display data in yaml main. [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28512: URL: https://github.com/apache/beam/pull/28512#issuecomment-1749815770 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

Re: [PR] Populate top-level display data in yaml main. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28512: URL: https://github.com/apache/beam/pull/28512#issuecomment-1749814530 R: @polber -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Add Java variants of WriteTo{Csv,Json}. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28380: URL: https://github.com/apache/beam/pull/28380#issuecomment-1749813567 R: @johnjcasey I think the Jenkins failures are infra issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Refactor StateFetcher [beam]

2023-10-05 Thread via GitHub
m-trieu commented on code in PR #28755: URL: https://github.com/apache/beam/pull/28755#discussion_r1348108297 ## runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/sideinput/SideInputStateFetcher.java: ## @@ -0,0 +1,215 @@ +

[PR] [YAML] Schemify avroio and add a yaml provider. [beam]

2023-10-05 Thread via GitHub
robertwb opened a new pull request, #28855: URL: https://github.com/apache/beam/pull/28855 This fixes https://github.com/apache/beam/issues/28852 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and

Re: [I] TextIO.write .withWindowedWrites does not work on windows [beam]

2023-10-05 Thread via GitHub
DwaipayanSom commented on issue #21317: URL: https://github.com/apache/beam/issues/21317#issuecomment-1749659633 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [I] TextIO.write .withWindowedWrites does not work on windows [beam]

2023-10-05 Thread via GitHub
DwaipayanSom commented on issue #21317: URL: https://github.com/apache/beam/issues/21317#issuecomment-1749658865 .take-issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Do not rewrite project version in BeamModulePlugin [beam]

2023-10-05 Thread via GitHub
kennknowles merged PR #28843: URL: https://github.com/apache/beam/pull/28843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

Re: [PR] Do not rewrite project version in BeamModulePlugin [beam]

2023-10-05 Thread via GitHub
kennknowles commented on PR #28843: URL: https://github.com/apache/beam/pull/28843#issuecomment-1749654881 Yup, passed global presubmit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_GBK_Dataflow_* [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28827: URL: https://github.com/apache/beam/pull/28827#issuecomment-1749641609 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label build. Available commands: - `stop r

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28846: URL: https://github.com/apache/beam/pull/28846#issuecomment-1749641418 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @Abacn for label build. Available commands: - `stop revi

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_GBK_Dataflow_* [beam]

2023-10-05 Thread via GitHub
magicgoody commented on code in PR #28827: URL: https://github.com/apache/beam/pull/28827#discussion_r1347960762 ## .github/workflows/beam_LoadTests_Python_GBK_Dataflow_Streaming.yml: ## @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# c

Re: [PR] Bump google.golang.org/api from 0.143.0 to 0.144.0 in /sdks [beam]

2023-10-05 Thread via GitHub
lostluck merged PR #28839: URL: https://github.com/apache/beam/pull/28839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
magicgoody commented on code in PR #28846: URL: https://github.com/apache/beam/pull/28846#discussion_r1347948365 ## .github/workflows/beam_LoadTests_Python_Combine_Flink_Streaming.yml: ## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[PR] Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_Smoke and job_LoadTests_Java_GBK_Smoke [beam]

2023-10-05 Thread via GitHub
aleksandr-dudko opened a new pull request, #28854: URL: https://github.com/apache/beam/pull/28854 PR migrates Jenkins Jobs to Github Actions Workflow: ".github/workflows/beam_LoadTests_Python_Smoke.yml" ".github/workflows/beam_LoadTests_Java_GBK_Smoke.yml" Jenkins Jobs: https

Re: [I] [Bug]: Ongoing operation in Bigtable write step [beam]

2023-10-05 Thread via GitHub
chcornwell commented on issue #28562: URL: https://github.com/apache/beam/issues/28562#issuecomment-1749598674 Hey @mutianf, I see this every time we run the job, it usually runs fine for anywhere from a few hours to a day or so and then gets stuck. -- This is an automated message from th

Re: [PR] Add new resource hint to all sdks for number of cpus per worker machine [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on code in PR #28848: URL: https://github.com/apache/beam/pull/28848#discussion_r1347930679 ## sdks/python/apache_beam/transforms/resources.py: ## @@ -177,6 +178,20 @@ def get_merged_value( ResourceHint.register_resource_hint('minRam', MinRamHint) +cla

Re: [PR] Add Publish Docker Snapshots workflow [beam]

2023-10-05 Thread via GitHub
damccorm merged PR #28794: URL: https://github.com/apache/beam/pull/28794 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

Re: [PR] Add Publish Docker Snapshots workflow [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28794: URL: https://github.com/apache/beam/pull/28794#issuecomment-1749570452 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label build. Available commands: - `stop r

Re: [PR] Add new resource hint to all sdks for number of cpus per worker machine [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28848: URL: https://github.com/apache/beam/pull/28848#issuecomment-1749570249 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @AnandInguva for label python. R: @Abacn for label java. R:

Re: [I] Add CouchbaseIO [beam]

2023-10-05 Thread via GitHub
deniswsrosa commented on issue #18381: URL: https://github.com/apache/beam/issues/18381#issuecomment-1749560140 @kennknowles Could you please point me to an example that I could follow? Maybe I could make that happen. -- This is an automated message from the Apache Git Service. To respond

Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]

2023-10-05 Thread via GitHub
Abacn closed issue #28793: [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" URL: https://github.com/apache/beam/issues/28793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] BigtableIO: Force minimum value of `desiredBundleSizeBytes` to be 1 [beam]

2023-10-05 Thread via GitHub
Abacn merged PR #28798: URL: https://github.com/apache/beam/pull/28798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

Re: [PR] Add Publish Beam SDK Snapshots workflow [beam]

2023-10-05 Thread via GitHub
Amar3tto commented on code in PR #28795: URL: https://github.com/apache/beam/pull/28795#discussion_r1347886725 ## .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml: ## @@ -0,0 +1,88 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor licen

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on code in PR #28385: URL: https://github.com/apache/beam/pull/28385#discussion_r1347863097 ## sdks/python/scripts/run_tox.sh: ## @@ -53,18 +53,27 @@ if [[ "$JENKINS_HOME" != "" ]]; then export PY_COLORS=1 fi -if [[ ! -z $2 ]]; then +# Determine if th

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on code in PR #28385: URL: https://github.com/apache/beam/pull/28385#discussion_r1347863097 ## sdks/python/scripts/run_tox.sh: ## @@ -53,18 +53,27 @@ if [[ "$JENKINS_HOME" != "" ]]; then export PY_COLORS=1 fi -if [[ ! -z $2 ]]; then +# Determine if th

Re: [PR] Enable typevar-name-mismatch check [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28849: URL: https://github.com/apache/beam/pull/28849#issuecomment-1749486847 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @liferoad for label python. Available commands: - `stop

Re: [PR] [BEAM-28684] [YAML] Add Parquet reading and writing. [beam]

2023-10-05 Thread via GitHub
github-actions[bot] commented on PR #28850: URL: https://github.com/apache/beam/pull/28850#issuecomment-1749486750 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

Re: [PR] BigtableIO: Force minimum value of `desiredBundleSizeBytes` to be 1 [beam]

2023-10-05 Thread via GitHub
Abacn commented on PR #28798: URL: https://github.com/apache/beam/pull/28798#issuecomment-1749474003 whitespace issue was an issue on master and now fixed #28845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on PR #28385: URL: https://github.com/apache/beam/pull/28385#issuecomment-1749470427 retest this please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[PR] Fix Reshuffle implementation in Java SDK [beam]

2023-10-05 Thread via GitHub
kennknowles opened a new pull request, #28853: URL: https://github.com/apache/beam/pull/28853 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution qu

Re: [PR] Upgrade flink to 1.17 [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28851: URL: https://github.com/apache/beam/pull/28851#issuecomment-1749468890 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28851?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] Update release guide somewhat according to 2.51.0 release experience [beam]

2023-10-05 Thread via GitHub
kennknowles commented on PR #28836: URL: https://github.com/apache/beam/pull/28836#issuecomment-1749465271 That was a brain-o. Incorrectly combined _cutting_ the release branch with stabilizing and then cutting an RC. It has one section that covered all that. Now cutting the branch is a pha

Re: [PR] [#20970] Fix gRPC leak by closing ResidualSource at BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker [beam]

2023-10-05 Thread via GitHub
kennknowles commented on PR #28548: URL: https://github.com/apache/beam/pull/28548#issuecomment-1749459982 OK the failure we see has been fixed at HEAD. Can you rebase against the master branch and we can re-run to get green. -- This is an automated message from the Apache Git Service. To

Re: [PR] Add Bigtable connector reviewers [beam]

2023-10-05 Thread via GitHub
ahmedabu98 merged PR #28834: URL: https://github.com/apache/beam/pull/28834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apac

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
AnandInguva commented on PR #28385: URL: https://github.com/apache/beam/pull/28385#issuecomment-1749446520 Run Seed Job -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To u

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_fanout_4_times_with_2GB_10byte_records_total:runtime [beam]

2023-10-05 Thread via GitHub
tvalentyn closed issue #28807: Performance Regression or Improvement: gbk_python_batch_load_test_fanout_4_times_with_2GB_10byte_records_total:runtime URL: https://github.com/apache/beam/issues/28807 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [I] Performance Regression or Improvement: gbk_python_batch_load_test_fanout_8_times_with_2GB_10byte_records_total:runtime [beam]

2023-10-05 Thread via GitHub
tvalentyn closed issue #28808: Performance Regression or Improvement: gbk_python_batch_load_test_fanout_8_times_with_2GB_10byte_records_total:runtime URL: https://github.com/apache/beam/issues/28808 -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] Enable typevar-name-mismatch check [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28849: URL: https://github.com/apache/beam/pull/28849#issuecomment-1749436262 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28849?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[PR] Upgrade flink to 1.17 [beam]

2023-10-05 Thread via GitHub
kkdoon opened a new pull request, #28851: URL: https://github.com/apache/beam/pull/28851 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly

Re: [PR] Add rotate credentials workflows [beam]

2023-10-05 Thread via GitHub
damccorm commented on code in PR #28815: URL: https://github.com/apache/beam/pull/28815#discussion_r1347809604 ## .github/workflows/beam_IODatastoresCredentialsRotation.yml: ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

Re: [PR] Add new resource hint to all sdks for number of cpus per worker machine [beam]

2023-10-05 Thread via GitHub
codecov[bot] commented on PR #28848: URL: https://github.com/apache/beam/pull/28848#issuecomment-1749416711 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28848?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

Re: [PR] Add Cleanup Dataproc Resources workflow [beam]

2023-10-05 Thread via GitHub
damccorm commented on code in PR #28814: URL: https://github.com/apache/beam/pull/28814#discussion_r1347804280 ## .github/workflows/beam_CleanUpDataprocResources.yml: ## @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

Re: [PR] Add Publish Beam SDK Snapshots workflow [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28795: URL: https://github.com/apache/beam/pull/28795#discussion_r1347800793 ## .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml: ## @@ -0,0 +1,88 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributo

Re: [PR] Retrieve workflow runs for GA Post-Commits Status dashboard [beam]

2023-10-05 Thread via GitHub
damccorm commented on code in PR #28844: URL: https://github.com/apache/beam/pull/28844#discussion_r1347794495 ## .test-infra/metrics/sync/github/sync_workflows.py: ## @@ -1,185 +1,203 @@ -# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor lic

Re: [PR] Do not rewrite project version in BeamModulePlugin [beam]

2023-10-05 Thread via GitHub
kennknowles commented on PR #28843: URL: https://github.com/apache/beam/pull/28843#issuecomment-1749398914 Running some internal tests too, to check that it plays nice with Google's build system wrapper of Beam's build (and I'll just fix internal breakages, since IMO this is obviously corre

Re: [PR] Add rotate credentials workflows [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28815: URL: https://github.com/apache/beam/pull/28815#discussion_r1347793365 ## .github/workflows/beam_IODatastoresCredentialsRotation.yml: ## @@ -0,0 +1,78 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contr

Re: [PR] [BEAM-28684] [YAML] Add Parquet reading and writing. [beam]

2023-10-05 Thread via GitHub
robertwb commented on PR #28850: URL: https://github.com/apache/beam/pull/28850#issuecomment-1749394631 This fixes https://github.com/apache/beam/issues/28684 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[PR] [BEAM-28684] [YAML] Add Parquet reading and writing. [beam]

2023-10-05 Thread via GitHub
robertwb opened a new pull request, #28850: URL: https://github.com/apache/beam/pull/28850 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate issue in yo

[PR] Enable typevar-name-mismatch check [beam]

2023-10-05 Thread via GitHub
jrmccluskey opened a new pull request, #28849: URL: https://github.com/apache/beam/pull/28849 Enables the typevar-name-mismatch pylint check and skips the check on the one instance the rule is violated within the Python sdk (which was already skipped for mypy checks.) Fixes #28241

Re: [PR] Add LoadTests Java ParDo Dataflow V2 workflows [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28744: URL: https://github.com/apache/beam/pull/28744#discussion_r1347782646 ## .github/workflows/beam_LoadTests_Java_ParDo_Dataflow_V2_Batch_Java11.yml: ## @@ -0,0 +1,130 @@ +# Licensed to the Apache Software Foundation (ASF) under one o

[PR] Add new resource hint to all sdks for number of cpus per worker machine [beam]

2023-10-05 Thread via GitHub
kerrydc opened a new pull request, #28848: URL: https://github.com/apache/beam/pull/28848 This resource hint allows for further specifcation of user needs at the transfor or pipeline level. GitHub Actions Tests Status (on master branch) ---

Re: [PR] Bump pillow from 9.3.0 to 10.0.1 in /sdks/python/apache_beam/examples/ml-orchestration/kfp/components/preprocessing [beam]

2023-10-05 Thread via GitHub
jrmccluskey merged PR #28810: URL: https://github.com/apache/beam/pull/28810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_GBK_Dataflow_* [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28827: URL: https://github.com/apache/beam/pull/28827#discussion_r1347778751 ## .github/workflows/beam_LoadTests_Python_GBK_Dataflow_Streaming.yml: ## @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more

Re: [PR] Bump github.com/fsouza/fake-gcs-server from 1.47.4 to 1.47.5 in /sdks [beam]

2023-10-05 Thread via GitHub
jrmccluskey merged PR #28812: URL: https://github.com/apache/beam/pull/28812 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28846: URL: https://github.com/apache/beam/pull/28846#discussion_r134616 ## .github/workflows/beam_LoadTests_Python_Combine_Flink_Streaming.yml: ## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28846: URL: https://github.com/apache/beam/pull/28846#discussion_r1347770869 ## .github/workflows/beam_LoadTests_Python_Combine_Flink_Streaming.yml: ## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] Add observed=False to be explicit for Pandas 2.1 [beam]

2023-10-05 Thread via GitHub
tvalentyn merged PR #28823: URL: https://github.com/apache/beam/pull/28823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

Re: [PR] Add map function and skip applymap for 2.1 [beam]

2023-10-05 Thread via GitHub
tvalentyn merged PR #28818: URL: https://github.com/apache/beam/pull/28818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

Re: [PR] [Python BQ] Retry get_table for quota errors [beam]

2023-10-05 Thread via GitHub
ahmedabu98 commented on code in PR #28820: URL: https://github.com/apache/beam/pull/28820#discussion_r1347769950 ## sdks/python/apache_beam/utils/retry.py: ## @@ -42,9 +43,11 @@ try: from apitools.base.py.exceptions import HttpError from google.api_core.exceptions import

Re: [PR] Github Workflow Replacement for Jenkins Jobs, beam_LoadTests_Python_Combine_Flink_* [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28846: URL: https://github.com/apache/beam/pull/28846#discussion_r1347770869 ## .github/workflows/beam_LoadTests_Python_Combine_Flink_Streaming.yml: ## @@ -0,0 +1,128 @@ +# Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] [Python BQ] Retry get_table for quota errors [beam]

2023-10-05 Thread via GitHub
ahmedabu98 commented on PR #28820: URL: https://github.com/apache/beam/pull/28820#issuecomment-1749366938 Thanks for the review @satybald, PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Add GitHub Workflow Replacements for Jenkins job_LoadTests_Python_SideInput_Dataflow_Batch [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28826: URL: https://github.com/apache/beam/pull/28826#discussion_r1347768356 ## .github/workflows/beam_LoadTests_Python_SideInput_Dataflow_Batch.yml: ## @@ -0,0 +1,184 @@ +# Licensed to the Apache Software Foundation (ASF) under one or mo

Re: [PR] Add edge segment size to filter out change points that are observed on the data edge [beam]

2023-10-05 Thread via GitHub
tvalentyn commented on code in PR #28780: URL: https://github.com/apache/beam/pull/28780#discussion_r1347767100 ## sdks/python/apache_beam/testing/analyzers/perf_analysis_utils.py: ## @@ -170,7 +170,22 @@ def find_latest_change_point_index(metric_values: List[Union[float, int]]

Re: [PR] Add GitHub Workflow Replacements for Jenkins job_LoadTests_Java_CoGBK [beam]

2023-10-05 Thread via GitHub
andreydevyatkin commented on code in PR #28742: URL: https://github.com/apache/beam/pull/28742#discussion_r1347743319 ## .github/workflows/beam_LoadTests_Java_CoGBK_Dataflow_Batch_Java.yml: ## @@ -0,0 +1,133 @@ +# Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] [Python BQ] Retry get_table for quota errors [beam]

2023-10-05 Thread via GitHub
ahmedabu98 commented on code in PR #28820: URL: https://github.com/apache/beam/pull/28820#discussion_r1347756779 ## sdks/python/apache_beam/utils/retry.py: ## @@ -57,6 +60,18 @@ # pylint: enable=wrong-import-order, wrong-import-position _LOGGER = logging.getLogger(__name__)

Re: [PR] Add pyproject.toml [beam]

2023-10-05 Thread via GitHub
tvalentyn commented on code in PR #28385: URL: https://github.com/apache/beam/pull/28385#discussion_r1347748955 ## sdks/python/setup.py: ## @@ -159,15 +159,14 @@ def generate_protos_first(): # Pyproject toml build happens in isolated environemnts. In those envs, # gen_

Re: [PR] [Python BQ] Retry get_table for quota errors [beam]

2023-10-05 Thread via GitHub
ahmedabu98 commented on code in PR #28820: URL: https://github.com/apache/beam/pull/28820#discussion_r1347747794 ## sdks/python/apache_beam/utils/retry.py: ## @@ -168,15 +183,29 @@ def retry_on_server_errors_and_timeout_filter(exception): def retry_on_server_errors_timeout_or_q

Re: [PR] [Python BQ] Retry get_table for quota errors [beam]

2023-10-05 Thread via GitHub
ahmedabu98 commented on code in PR #28820: URL: https://github.com/apache/beam/pull/28820#discussion_r1347747381 ## sdks/python/apache_beam/utils/retry.py: ## @@ -168,15 +183,29 @@ def retry_on_server_errors_and_timeout_filter(exception): def retry_on_server_errors_timeout_or_q

  1   2   >