[GitHub] [beam] Amar3tto opened a new pull request, #28044: Add Java IO Direct jobs: Hadoop, Kinesis, Pulsar

2023-08-16 Thread via GitHub
Amar3tto opened a new pull request, #28044: URL: https://github.com/apache/beam/pull/28044 The following jenkins jobs have been migrated: https://ci-beam.apache.org/job/beam_PreCommit_Java_Hadoop_IO_Direct_Commit/ https://ci-beam.apache.org/job/beam_PreCommit_Java_Hadoop_IO_Direct_C

[GitHub] [beam] github-actions[bot] commented on pull request #28021: Add Java IO Direct jobs: JDBC, Jms, Snowflake

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28021: URL: https://github.com/apache/beam/pull/28021#issuecomment-1681666832 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @Abacn for label build. Available commands: - `stop revi

[GitHub] [beam] github-actions[bot] commented on pull request #27966: Add Java IO Direct jobs: MongoDb, Mqtt, Parquet, RabbitMq, SingleStore

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #27966: URL: https://github.com/apache/beam/pull/27966#issuecomment-1681664241 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] Amar3tto commented on pull request #27966: Add Java IO Direct jobs: MongoDb, Mqtt, Parquet, RabbitMq, SingleStore

2023-08-16 Thread via GitHub
Amar3tto commented on PR #27966: URL: https://github.com/apache/beam/pull/27966#issuecomment-1681662861 R: @Abacn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

[GitHub] [beam] magicgoody commented on a diff in pull request #28027: Add GitHub Workflow Replacement for Jenkins job_PreCommit_Java_Debezium_IO_Direct and job_PreCommit_Java_ElasticSearch_IO_Direct

2023-08-16 Thread via GitHub
magicgoody commented on code in PR #28027: URL: https://github.com/apache/beam/pull/28027#discussion_r1296679614 ## .github/workflows/beam_PreCommit_Java_Debezium_IO_Direct.yml: ## @@ -0,0 +1,95 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contribu

[GitHub] [beam] gabrywu commented on a diff in pull request #28007: remove fastjson library

2023-08-16 Thread via GitHub
gabrywu commented on code in PR #28007: URL: https://github.com/apache/beam/pull/28007#discussion_r1296629045 ## CHANGES.md: ## @@ -89,6 +89,7 @@ * Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2. * [Python] Dataflow Runner will no

[GitHub] [beam] gabrywu commented on pull request #28007: remove fastjson library

2023-08-16 Thread via GitHub
gabrywu commented on PR #28007: URL: https://github.com/apache/beam/pull/28007#issuecomment-1681546196 > I will add `ALLOW_TRAILING_COMMA` flag -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to g

[GitHub] [beam] github-actions[bot] commented on pull request #28043: [Do not review] Temp change to re-run java17 ARM tests

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28043: URL: https://github.com/apache/beam/pull/28043#issuecomment-1681426277 Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers` -- This is an automated me

[GitHub] [beam] rszper commented on a diff in pull request #28042: New MLTransform documentation

2023-08-16 Thread via GitHub
rszper commented on code in PR #28042: URL: https://github.com/apache/beam/pull/28042#discussion_r1296544269 ## website/www/site/content/en/documentation/ml/preprocess-data.md: ## @@ -0,0 +1,227 @@ +--- +title: "Preprocess data" +--- + + +# Preprocess data with MLTransform + +Us

[GitHub] [beam] github-actions[bot] commented on pull request #28042: New MLTransform documentation

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28042: URL: https://github.com/apache/beam/pull/28042#issuecomment-1681401487 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] github-actions[bot] commented on pull request #28042: New MLTransform documentation

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28042: URL: https://github.com/apache/beam/pull/28042#issuecomment-1681398541 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @lostluck added as fallback since no labels match configuration

[GitHub] [beam] codecov[bot] commented on pull request #28043: [Do not review] Temp change to re-run java17 ARM tests

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28043: URL: https://github.com/apache/beam/pull/28043#issuecomment-1681396533 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28043?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] tvalentyn merged pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python containers

2023-08-16 Thread via GitHub
tvalentyn merged PR #28036: URL: https://github.com/apache/beam/pull/28036 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apach

[GitHub] [beam] tvalentyn closed issue #28035: [Bug]: Multi-arch Beam SDK Python 3.11 container missing libgeos-dev for ARM

2023-08-16 Thread via GitHub
tvalentyn closed issue #28035: [Bug]: Multi-arch Beam SDK Python 3.11 container missing libgeos-dev for ARM URL: https://github.com/apache/beam/issues/28035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [beam] tvalentyn commented on pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python containers

2023-08-16 Thread via GitHub
tvalentyn commented on PR #28036: URL: https://github.com/apache/beam/pull/28036#issuecomment-1681380271 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe

[GitHub] [beam] celeste-zeng opened a new pull request, #28043: [Do not review] Temp change to re-run java17 ARM tests

2023-08-16 Thread via GitHub
celeste-zeng opened a new pull request, #28043: URL: https://github.com/apache/beam/pull/28043 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appropriate iss

[GitHub] [beam] github-actions[bot] commented on pull request #28040: Update build_release_candidate.yml publish_java_artifacts to be self hosted.

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28040: URL: https://github.com/apache/beam/pull/28040#issuecomment-1681379092 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @damccorm for label build. Available commands: - `stop r

[GitHub] [beam] lostluck commented on pull request #28040: Update build_release_candidate.yml publish_java_artifacts to be self hosted.

2023-08-16 Thread via GitHub
lostluck commented on PR #28040: URL: https://github.com/apache/beam/pull/28040#issuecomment-1681368560 The action worked with self hosted! https://github.com/apache/beam/actions/runs/5883552047/job/15956443420 -- This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] tvalentyn commented on a diff in pull request #26059: Add a simple UnionCoder

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #26059: URL: https://github.com/apache/beam/pull/26059#discussion_r1296509290 ## sdks/python/apache_beam/coders/union_coder.py: ## @@ -0,0 +1,94 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license a

[GitHub] [beam] kerrydc commented on a diff in pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
kerrydc commented on code in PR #27959: URL: https://github.com/apache/beam/pull/27959#discussion_r1296459640 ## sdks/python/apache_beam/options/pipeline_options_validator_test.py: ## @@ -111,69 +111,65 @@ def get_validator(temp_location, staging_location): validator = Pi

[GitHub] [beam] kerrydc commented on a diff in pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
kerrydc commented on code in PR #27959: URL: https://github.com/apache/beam/pull/27959#discussion_r1296458776 ## sdks/python/apache_beam/options/pipeline_options_validator_test.py: ## @@ -479,20 +470,17 @@ def test_experiment_region_and_worker_region_mutually_exclusive(self):

[GitHub] [beam] github-actions[bot] commented on pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python containers

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28036: URL: https://github.com/apache/beam/pull/28036#issuecomment-1681300209 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @tvalentyn for label python. Available commands: - `stop

[GitHub] [beam] github-actions[bot] commented on pull request #28039: Better manage transform service lifecycle for tests

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28039: URL: https://github.com/apache/beam/pull/28039#issuecomment-1681300107 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @jrmccluskey for label python. R: @Abacn for label build.

[GitHub] [beam] github-actions[bot] commented on pull request #28041: Update python-pipeline-dependencies.md

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28041: URL: https://github.com/apache/beam/pull/28041#issuecomment-1681296417 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] tvalentyn commented on a diff in pull request #28041: Update python-pipeline-dependencies.md

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28041: URL: https://github.com/apache/beam/pull/28041#discussion_r1296448388 ## website/www/site/content/en/documentation/sdks/python-pipeline-dependencies.md: ## @@ -235,4 +235,4 @@ The Apache Beam version and the Python major.minor versions

[GitHub] [beam] tvalentyn commented on pull request #28041: Update python-pipeline-dependencies.md

2023-08-16 Thread via GitHub
tvalentyn commented on PR #28041: URL: https://github.com/apache/beam/pull/28041#issuecomment-1681295108 R: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [beam] tvalentyn opened a new pull request, #28041: Update python-pipeline-dependencies.md

2023-08-16 Thread via GitHub
tvalentyn opened a new pull request, #28041: URL: https://github.com/apache/beam/pull/28041 Update a broken link. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [beam] celeste-zeng commented on a diff in pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python containers

2023-08-16 Thread via GitHub
celeste-zeng commented on code in PR #28036: URL: https://github.com/apache/beam/pull/28036#discussion_r1296438161 ## sdks/python/container/Dockerfile: ## @@ -80,6 +80,11 @@ RUN \ # Remove pip cache. rm -rf /root/.cache/pip +# Required for using Beam Python SDK on A

[GitHub] [beam] tvalentyn commented on a diff in pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28036: URL: https://github.com/apache/beam/pull/28036#discussion_r1296435296 ## sdks/python/container/Dockerfile: ## @@ -80,6 +80,11 @@ RUN \ # Remove pip cache. rm -rf /root/.cache/pip +# Required for using Beam Python SDK on ARM

[GitHub] [beam] tvalentyn commented on a diff in pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28036: URL: https://github.com/apache/beam/pull/28036#discussion_r1296435073 ## sdks/python/container/Dockerfile: ## @@ -80,6 +80,11 @@ RUN \ # Remove pip cache. rm -rf /root/.cache/pip +# Required for using Beam Python SDK on ARM

[GitHub] [beam] tvalentyn commented on a diff in pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28036: URL: https://github.com/apache/beam/pull/28036#discussion_r1296433672 ## sdks/python/container/Dockerfile: ## @@ -80,6 +80,11 @@ RUN \ # Remove pip cache. rm -rf /root/.cache/pip +# Required for using Beam Python SDK on ARM

[GitHub] [beam] tvalentyn commented on a diff in pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28036: URL: https://github.com/apache/beam/pull/28036#discussion_r1296433672 ## sdks/python/container/Dockerfile: ## @@ -80,6 +80,11 @@ RUN \ # Remove pip cache. rm -rf /root/.cache/pip +# Required for using Beam Python SDK on ARM

[GitHub] [beam] codecov[bot] commented on pull request #28037: [WIP] Replace deprecated pkg_resources module

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28037: URL: https://github.com/apache/beam/pull/28037#issuecomment-1681266620 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28037?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] codecov[bot] commented on pull request #28039: Better manage transform service lifecycle for tests

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28039: URL: https://github.com/apache/beam/pull/28039#issuecomment-1681262127 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28039?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] lostluck commented on pull request #28040: Update build_release_candidate.yml publish_java_artifacts to be self hosted.

2023-08-16 Thread via GitHub
lostluck commented on PR #28040: URL: https://github.com/apache/beam/pull/28040#issuecomment-1681227945 cc: @damccorm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To un

[GitHub] [beam] lostluck opened a new pull request, #28040: Update build_release_candidate.yml publish_java_artifacts to be self hosted.

2023-08-16 Thread via GitHub
lostluck opened a new pull request, #28040: URL: https://github.com/apache/beam/pull/28040 Fix for #28038 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Mention the appro

[GitHub] [beam] damccorm commented on issue #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
damccorm commented on issue #28038: URL: https://github.com/apache/beam/issues/28038#issuecomment-1681225945 Yeah, we're in the process of migrating all workflows which started with stabilizing our self hosted runners -- This is an automated message from the Apache Git Service. To respond

[GitHub] [beam] lostluck commented on issue #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
lostluck commented on issue #28038: URL: https://github.com/apache/beam/issues/28038#issuecomment-1681218749 As a practical matter, if it's a workflow change I can point the action to, I'm happy to try that out instead of locally. My laptop is currently busy with the docker pushes th

[GitHub] [beam] lostluck commented on issue #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
lostluck commented on issue #28038: URL: https://github.com/apache/beam/issues/28038#issuecomment-1681216434 I did > Interestingly, this happened once in solr io - https://github.com/apache/beam/actions/runs/5872911524/job/15925292759 and once in pulsar io - https://github.com/apach

[GitHub] [beam] chamikaramj opened a new pull request, #28039: Better manage transform service lifecycle for tests

2023-08-16 Thread via GitHub
chamikaramj opened a new pull request, #28039: URL: https://github.com/apache/beam/pull/28039 Wait for the transform service to start/stop in tests. Start unique transform service instance per test task. Cleanup transform service temp directory after during the clanup phase of

[GitHub] [beam] ahmedabu98 commented on pull request #28009: [BQ Python StorageWriteAPI] Enable writing rows with missing nullable fields

2023-08-16 Thread via GitHub
ahmedabu98 commented on PR #28009: URL: https://github.com/apache/beam/pull/28009#issuecomment-1681214769 It's an anitpattern. We don't have similar checks with native Java and Python implementations. Instead we output failed rows to DLQ when appropriate or entirely fail in the case of `FIL

[GitHub] [beam] damccorm commented on issue #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
damccorm commented on issue #28038: URL: https://github.com/apache/beam/issues/28038#issuecomment-1681210670 One thing we could try here is changing the workflow to run on our self hosted runners (I'd thought it already was) - https://github.com/apache/beam/blob/master/.github/workflows/bui

[GitHub] [beam] damccorm commented on issue #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
damccorm commented on issue #28038: URL: https://github.com/apache/beam/issues/28038#issuecomment-1681207492 Interestingly, this happened once in solr io - https://github.com/apache/beam/actions/runs/5872911524/job/15925292759 and once in pulsar io - https://github.com/apache/beam/action

[GitHub] [beam] github-actions[bot] commented on pull request #28014: Fetch executor state when call direct runner getState

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28014: URL: https://github.com/apache/beam/pull/28014#issuecomment-1681202916 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @riteshghorse added as fallback since no labels match configuratio

[GitHub] [beam] lostluck opened a new issue, #28038: [Bug]: Github Action build_release_candidate publish_java_artifacts job fails with GC overhead limit exceeded.

2023-08-16 Thread via GitHub
lostluck opened a new issue, #28038: URL: https://github.com/apache/beam/issues/28038 ### What happened? The build_release_candidate workflow is not currently usable to actually build release candidates. This occurs when running the build_release_candidate workflow publish_jav

[GitHub] [beam] ahmedabu98 commented on pull request #27998: Use Periodic Impulse for BQ SchemaUpdate tests

2023-08-16 Thread via GitHub
ahmedabu98 commented on PR #27998: URL: https://github.com/apache/beam/pull/27998#issuecomment-1681191398 > Just note that there is a GenerateSequence transform can emit integers with given interval, from pipeline startup That's true, but GenerateSequence as a streaming doesn't have

[GitHub] [beam] Abacn merged pull request #28028: Fix Pytest Unknown Marker Error for Xlang Kafka IT Test

2023-08-16 Thread via GitHub
Abacn merged PR #28028: URL: https://github.com/apache/beam/pull/28028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.or

[GitHub] [beam] kennknowles commented on a diff in pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
kennknowles commented on code in PR #27959: URL: https://github.com/apache/beam/pull/27959#discussion_r1296357589 ## sdks/python/apache_beam/options/pipeline_options.py: ## @@ -864,10 +864,6 @@ def validate(self, validator): else: setattr(self, 'temp_location

[GitHub] [beam] codecov[bot] commented on pull request #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28036: URL: https://github.com/apache/beam/pull/28036#issuecomment-1681181491 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28036?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] jrmccluskey commented on a diff in pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
jrmccluskey commented on code in PR #27959: URL: https://github.com/apache/beam/pull/27959#discussion_r1296352735 ## sdks/python/apache_beam/options/pipeline_options_validator_test.py: ## @@ -479,20 +470,17 @@ def test_experiment_region_and_worker_region_mutually_exclusive(self

[GitHub] [beam] Abacn commented on pull request #27998: Use Periodic Impulse for BQ SchemaUpdate tests

2023-08-16 Thread via GitHub
Abacn commented on PR #27998: URL: https://github.com/apache/beam/pull/27998#issuecomment-1681163387 Just note that there is a GenerateSequence transform can emit integers with given interval, from pipeline startup (do not flush backlogs at the beginning) -- This is an automated message f

[GitHub] [beam] jrmccluskey opened a new pull request, #28037: [WIP] Replace deprecated pkg_resources module

2023-08-16 Thread via GitHub
jrmccluskey opened a new pull request, #28037: URL: https://github.com/apache/beam/pull/28037 Starts to replace usage of the deprecated pkg_resources module with importlib.resources, importlib.metadata, and packaging packages. DO NOT MERGE Thank

[GitHub] [beam] kerrydc commented on pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
kerrydc commented on PR #27959: URL: https://github.com/apache/beam/pull/27959#issuecomment-1681154646 Finally all tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [beam] johnjcasey commented on pull request #28009: [BQ Python StorageWriteAPI] Enable writing rows with missing nullable fields

2023-08-16 Thread via GitHub
johnjcasey commented on PR #28009: URL: https://github.com/apache/beam/pull/28009#issuecomment-1681151605 Should we remove the entire check, or is there a way to keep the check that is more relaxed? -- This is an automated message from the Apache Git Service. To respond to the message, pl

[GitHub] [beam] celeste-zeng opened a new pull request, #28036: Add libgeos-dev for ARM components of multi-arch Beam SDK Python 3.11 containers

2023-08-16 Thread via GitHub
celeste-zeng opened a new pull request, #28036: URL: https://github.com/apache/beam/pull/28036 - Addressing https://github.com/apache/beam/issues/28035 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quick

[GitHub] [beam] tvalentyn commented on a diff in pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28033: URL: https://github.com/apache/beam/pull/28033#discussion_r1296323781 ## sdks/python/container/build.gradle: ## @@ -42,8 +42,33 @@ tasks.register("buildAll") { dependsOn ':sdks:python:container:py311:docker' } +for(int i=8; i<=11;

[GitHub] [beam] tvalentyn commented on a diff in pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28033: URL: https://github.com/apache/beam/pull/28033#discussion_r1296315822 ## sdks/python/container/build.gradle: ## @@ -42,8 +42,33 @@ tasks.register("buildAll") { dependsOn ':sdks:python:container:py311:docker' } +for(int i=8; i<=11;

[GitHub] [beam] tvalentyn commented on a diff in pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
tvalentyn commented on code in PR #28033: URL: https://github.com/apache/beam/pull/28033#discussion_r1296314936 ## sdks/python/container/build.gradle: ## @@ -42,8 +42,33 @@ tasks.register("buildAll") { dependsOn ':sdks:python:container:py311:docker' } +for(int i=8; i<=11;

[GitHub] [beam] ahmedabu98 commented on a diff in pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
ahmedabu98 commented on code in PR #28032: URL: https://github.com/apache/beam/pull/28032#discussion_r1296311554 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java: ## @@ -302,15 +303,15 @@ public void testWriteDynami

[GitHub] [beam] ahmedabu98 commented on a diff in pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
ahmedabu98 commented on code in PR #28032: URL: https://github.com/apache/beam/pull/28032#discussion_r1296309909 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java: ## @@ -1329,8 +1330,12 @@ static class SchemaPojo {

[GitHub] [beam] celeste-zeng opened a new issue, #28035: [Bug]: Multi-arch Beam SDK Python 3.11 container missing libgeos-dev for ARM

2023-08-16 Thread via GitHub
celeste-zeng opened a new issue, #28035: URL: https://github.com/apache/beam/issues/28035 ### What happened? Used a multi-arch Beam SDK Python 3.11 container to run a Dataflow pipeline on ARM and the container failed to start due to the error `OSError: Could not find lib geos_c or lo

[GitHub] [beam] jrmccluskey merged pull request #28029: Replace usage of deprecated pandas.Int64Index()

2023-08-16 Thread via GitHub
jrmccluskey merged PR #28029: URL: https://github.com/apache/beam/pull/28029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apa

[GitHub] [beam] lostluck commented on pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
lostluck commented on PR #28033: URL: https://github.com/apache/beam/pull/28033#issuecomment-1681084073 FYI: @celeste-zeng as the original author and @tvalentyn as the reviewr of the PR. Apparently the python container ordering there was load bearing. -- This is an automated message from

[GitHub] [beam] johnjcasey commented on a diff in pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
johnjcasey commented on code in PR #28032: URL: https://github.com/apache/beam/pull/28032#discussion_r1296280323 ## sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteTest.java: ## @@ -302,15 +303,15 @@ public void testWriteDynami

[GitHub] [beam] lostluck commented on pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
lostluck commented on PR #28033: URL: https://github.com/apache/beam/pull/28033#issuecomment-1681082106 https://github.com/apache/beam/actions/runs/5882454184 Running to validate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [beam] rafaelsms opened a new issue, #28034: [Bug]: Datastore's Read fails Pipeline validation and throws on DataflowRunner

2023-08-16 Thread via GitHub
rafaelsms opened a new issue, #28034: URL: https://github.com/apache/beam/issues/28034 ### What happened? Hello! Sorry if this is a duplicate. To be honest, I don't know much about Apache Beam and Dataflow, so I am still learning and might be doing something wrong, let me know

[GitHub] [beam] codecov[bot] commented on pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28033: URL: https://github.com/apache/beam/pull/28033#issuecomment-1681073487 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28033?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] github-actions[bot] commented on pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28033: URL: https://github.com/apache/beam/pull/28033#issuecomment-1681062225 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] damccorm commented on pull request #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
damccorm commented on PR #28033: URL: https://github.com/apache/beam/pull/28033#issuecomment-1681060544 R: @lostluck -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

[GitHub] [beam] damccorm opened a new pull request, #28033: Add back docker prune step between python container pushes

2023-08-16 Thread via GitHub
damccorm opened a new pull request, #28033: URL: https://github.com/apache/beam/pull/28033 This ordering/intermediate prune on releases got removed in https://github.com/apache/beam/pull/27836/files but the intermediate prune is still needed Thank you

[GitHub] [beam] github-actions[bot] commented on pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28032: URL: https://github.com/apache/beam/pull/28032#issuecomment-1681046244 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] ahmedabu98 commented on pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
ahmedabu98 commented on PR #28032: URL: https://github.com/apache/beam/pull/28032#issuecomment-1681042542 R: @johnjcasey -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [beam] damccorm commented on issue #28030: [Bug]: Build Release Candidate stage_docker job fails with out of space error.

2023-08-16 Thread via GitHub
damccorm commented on issue #28030: URL: https://github.com/apache/beam/issues/28030#issuecomment-1681041139 With previous instances of the same problem, we've worked around it by pruning the images in the middle - https://github.com/apache/beam/blob/a3b1d0b4d2c915b5ea261448ae2e8a70f56bcc5c

[GitHub] [beam] damccorm merged pull request #28031: Refactored ReadMe

2023-08-16 Thread via GitHub
damccorm merged PR #28031: URL: https://github.com/apache/beam/pull/28031 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

[GitHub] [beam] ahmedabu98 commented on pull request #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
ahmedabu98 commented on PR #28032: URL: https://github.com/apache/beam/pull/28032#issuecomment-1681035286 Run Kotlin_Examples PreCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

[GitHub] [beam] damccorm commented on a diff in pull request #28031: Refactored ReadMe

2023-08-16 Thread via GitHub
damccorm commented on code in PR #28031: URL: https://github.com/apache/beam/pull/28031#discussion_r1296240009 ## README.md: ## @@ -138,3 +121,12 @@ Here are some resources actively maintained by the Beam community to help you ge + +## Contact Us + +To get involved with

[GitHub] [beam] ahmedabu98 opened a new pull request, #28032: Add more batch Storage Write API tests

2023-08-16 Thread via GitHub
ahmedabu98 opened a new pull request, #28032: URL: https://github.com/apache/beam/pull/28032 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[GitHub] [beam] github-actions[bot] commented on pull request #28029: Replace usage of deprecated pandas.Int64Index()

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28029: URL: https://github.com/apache/beam/pull/28029#issuecomment-1681025064 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] github-actions[bot] commented on pull request #28029: Replace usage of deprecated pandas.Int64Index()

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28029: URL: https://github.com/apache/beam/pull/28029#issuecomment-1681018604 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @AnandInguva for label python. Available commands: - `st

[GitHub] [beam] Abacn commented on pull request #28028: Fix Pytest Unknown Marker Error for Xlang Kafka IT Test

2023-08-16 Thread via GitHub
Abacn commented on PR #28028: URL: https://github.com/apache/beam/pull/28028#issuecomment-1681015962 Test passed: https://ci-beam.apache.org/job/beam_PostCommit_Python_Xlang_IO_Dataflow_PR/9/testReport/apache_beam.io.external.xlang_kafkaio_it_test/CrossLanguageKafkaIOTest/ -- This is an a

[GitHub] [beam] github-actions[bot] commented on pull request #28031: Refactored ReadMe

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28031: URL: https://github.com/apache/beam/pull/28031#issuecomment-1681003701 Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control -- This is an automated message from the Apache Git Serv

[GitHub] [beam] Abacn commented on pull request #28007: remove fastjson library

2023-08-16 Thread via GitHub
Abacn commented on PR #28007: URL: https://github.com/apache/beam/pull/28007#issuecomment-1681002476 There are test failures: ``` org.apache.beam.sdk.extensions.sql.meta.provider.kafka.KafkaTableProviderIT.testFake2[0] Error Message java.lang.RuntimeException: illegal tab

[GitHub] [beam] svetakvsundhar opened a new pull request, #28031: Refactored ReadMe

2023-08-16 Thread via GitHub
svetakvsundhar opened a new pull request, #28031: URL: https://github.com/apache/beam/pull/28031 This PR: 1) Removes the `Getting Started` section, as it's redundant with the `Learn More` Section. 2) Moves the `Contact Us` section to the end of the file. 3) Removes the links for

[GitHub] [beam] github-actions[bot] commented on pull request #28028: Fix Pytest Unknown Marker Error for Xlang Kafka IT Test

2023-08-16 Thread via GitHub
github-actions[bot] commented on PR #28028: URL: https://github.com/apache/beam/pull/28028#issuecomment-1680970172 Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`: R: @riteshghorse for label python. Available commands: - `s

[GitHub] [beam] Abacn commented on issue #28030: [Bug]: Build Release Candidate stage_docker job fails with out of space error.

2023-08-16 Thread via GitHub
Abacn commented on issue #28030: URL: https://github.com/apache/beam/issues/28030#issuecomment-1680927105 multiarch double the space needed for containers, now github action fails halfway, when build python3.10 container. Runners (flink etc) already pushed to dockerhub, e.g. https://hub.doc

[GitHub] [beam] lostluck commented on issue #28030: [Bug]: Build Release Candidate stage_docker job fails with out of space error.

2023-08-16 Thread via GitHub
lostluck commented on issue #28030: URL: https://github.com/apache/beam/issues/28030#issuecomment-1680926145 cc: @damccorm for triage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [beam] lostluck opened a new issue, #28030: [Bug]: Build Release Candidate stage_docker job fails with out of space error.

2023-08-16 Thread via GitHub
lostluck opened a new issue, #28030: URL: https://github.com/apache/beam/issues/28030 ### What happened? https://github.com/apache/beam/actions/workflows/build_release_candidate.yml stage_docker job fails ``` #13 117.0 --- Logging error --- 4340#13 117.0 Traceback (

[GitHub] [beam] codecov[bot] commented on pull request #27959: Updates validation on --dataflow_endpoint to accept any legal url.

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #27959: URL: https://github.com/apache/beam/pull/27959#issuecomment-1680917660 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/27959?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] codecov[bot] commented on pull request #28029: Replace usage of deprecated pandas.Int64Index()

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28029: URL: https://github.com/apache/beam/pull/28029#issuecomment-1680908905 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28029?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] codecov[bot] commented on pull request #28028: Fix Pytest Unknown Marker Error for Xlang Kafka IT Test

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28028: URL: https://github.com/apache/beam/pull/28028#issuecomment-1680886296 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28028?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] damccorm commented on pull request #28026: Add per key inference integration test

2023-08-16 Thread via GitHub
damccorm commented on PR #28026: URL: https://github.com/apache/beam/pull/28026#issuecomment-1680881514 Run Python 3.9 PostCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

[GitHub] [beam] jrmccluskey opened a new pull request, #28029: Replace usage of deprecated pandas.Int64Index()

2023-08-16 Thread via GitHub
jrmccluskey opened a new pull request, #28029: URL: https://github.com/apache/beam/pull/28029 `pandas.Int64Index()` has been deprecated in favor of using `pandas.Index(dtype='int64')` and the deprecation warnings are clogging testing output. This resolves that issue and should reduce test l

[GitHub] [beam] lostluck merged pull request #28005: Bump google.golang.org/api from 0.136.0 to 0.137.0 in /sdks

2023-08-16 Thread via GitHub
lostluck merged PR #28005: URL: https://github.com/apache/beam/pull/28005 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache

[GitHub] [beam] jrmccluskey opened a new pull request, #28028: Fix Pytest Unknown Marker Error for Xlang Kafka IT Test

2023-08-16 Thread via GitHub
jrmccluskey opened a new pull request, #28028: URL: https://github.com/apache/beam/pull/28028 The Kafka Xlang IT tests were seemingly not running because the `uses_io_expansion_service` tag was not defined. This should fix those warnings and enable those tests. -

[GitHub] [beam] Abacn commented on a diff in pull request #28021: Add Java IO Direct jobs: JDBC, Jms, Snowflake

2023-08-16 Thread via GitHub
Abacn commented on code in PR #28021: URL: https://github.com/apache/beam/pull/28021#discussion_r1296109942 ## .github/workflows/beam_PreCommit_Java_JDBC_IO_Direct.yml: ## @@ -0,0 +1,102 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor lice

[GitHub] [beam] codecov[bot] commented on pull request #28026: Add per key inference integration test

2023-08-16 Thread via GitHub
codecov[bot] commented on PR #28026: URL: https://github.com/apache/beam/pull/28026#issuecomment-1680855451 ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28026?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report

[GitHub] [beam] aleksandr-dudko opened a new pull request, #28027: Add GitHub Workflow Replacement for Jenkins job_PreCommit_Java_Debezium_IO_Direct and job_PreCommit_Java_ElasticSearch_IO_Direct

2023-08-16 Thread via GitHub
aleksandr-dudko opened a new pull request, #28027: URL: https://github.com/apache/beam/pull/28027 The following jenkins jobs have been migrated: https://ci-beam.apache.org/view/PreCommit/job/beam_PreCommit_Java_ElasticSearch_IO_Direct_Commit/ https://ci-beam.apache.org/view/PreCom

[GitHub] [beam] Abacn commented on issue #28012: [Task]: Upload JUnit test report for github action runs

2023-08-16 Thread via GitHub
Abacn commented on issue #28012: URL: https://github.com/apache/beam/issues/28012#issuecomment-1680837032 #27996 added reports/tests to actions/upload-artifact@v3 but it does not work -- "Warning: No files were found with the provided path: "**/build/test-results/**/*.xml" "**/build/repo

[GitHub] [beam] damccorm closed pull request #27880: [WIP] Add example and integration test for model per key inference

2023-08-16 Thread via GitHub
damccorm closed pull request #27880: [WIP] Add example and integration test for model per key inference URL: https://github.com/apache/beam/pull/27880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [beam] damccorm commented on pull request #28026: Add per key inference integration test

2023-08-16 Thread via GitHub
damccorm commented on PR #28026: URL: https://github.com/apache/beam/pull/28026#issuecomment-1680828765 Run Python 3.9 PostCommit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm

  1   2   >