(beam) branch users/damccorm/jobName deleted (was 8f8c12d4e7f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/jobName in repository https://gitbox.apache.org/repos/asf/beam.git was 8f8c12d4e7f Improve wheels job name The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Improve wheels job name (#32644)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new decc997ea57 Improve wheels job name (#32644)
decc997ea57 is described below
commit decc997ea57a61449062d1d7ac47d8ac53b04d9d
Author: Danny McCormick
AuthorDate: Fri Oct 4 14:27:36 2024 -0400
Improve wheels job name (#32644)
---
.github/workflows/build_wheels.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 25030231a5e..d1e99f2bd57 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -211,7 +211,7 @@ jobs:
run: gsutil cp -r -a public-read source/* ${{ env.GCP_PATH }}
build_wheels:
-name: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os
}}
+name: Build python ${{matrix.py_version}} wheels on
${{matrix.os_python.arch}} for ${{ matrix.os_python.os }}
needs:
- check_env_variables
- build_source
(beam) branch master updated: Update groupbykey.py (#32359)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6561abf7beb Update groupbykey.py (#32359)
6561abf7beb is described below
commit 6561abf7beb3d19e4ff4fa9726c4351b9019c1ab
Author: Christoph Grotz
AuthorDate: Thu Oct 3 17:37:56 2024 +0200
Update groupbykey.py (#32359)
The original example was not actually counting the produce but grouping the
produce per season. Maybe it's better to rename the variables to reflect this,
in order to not confuse the reader.
---
.../examples/snippets/transforms/aggregation/groupbykey.py| 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py
b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py
index aca39f6a219..a2d32b564a3 100644
---
a/sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py
+++
b/sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py
@@ -40,9 +40,9 @@ def groupbykey(test=None):
import apache_beam as beam
with beam.Pipeline() as pipeline:
-produce_counts = (
+produce_per_season = (
pipeline
-| 'Create produce counts' >> beam.Create([
+| 'Create produce list' >> beam.Create([
('spring', '🍓'),
('spring', '🥕'),
('spring', '🍆'),
@@ -54,12 +54,12 @@ def groupbykey(test=None):
('fall', '🍅'),
('winter', '🍆'),
])
-| 'Group counts per produce' >> beam.GroupByKey()
+| 'Group produce per season' >> beam.GroupByKey()
| beam.MapTuple(lambda k, vs: (k, sorted(vs))) # sort and format
| beam.Map(print))
# [END groupbykey]
if test:
- test(produce_counts)
+ test(produce_per_season)
if __name__ == '__main__':
(beam) 01/01: Improve wheels job name
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/jobName
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8f8c12d4e7f24f9d959ee8dbdead66afcf389dd3
Author: Danny McCormick
AuthorDate: Thu Oct 3 11:14:30 2024 -0400
Improve wheels job name
---
.github/workflows/build_wheels.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 25030231a5e..d1e99f2bd57 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -211,7 +211,7 @@ jobs:
run: gsutil cp -r -a public-read source/* ${{ env.GCP_PATH }}
build_wheels:
-name: Build python wheels on ${{matrix.arch}} for ${{ matrix.os_python.os
}}
+name: Build python ${{matrix.py_version}} wheels on
${{matrix.os_python.arch}} for ${{ matrix.os_python.os }}
needs:
- check_env_variables
- build_source
(beam) branch users/damccorm/jobName created (now 8f8c12d4e7f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/jobName in repository https://gitbox.apache.org/repos/asf/beam.git at 8f8c12d4e7f Improve wheels job name This branch includes the following new commits: new 8f8c12d4e7f Improve wheels job name The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: Rag opensearch usecase with Beam's MLTransform (#32018)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 1d7b0bcde9e Rag opensearch usecase with Beam's MLTransform (#32018) 1d7b0bcde9e is described below commit 1d7b0bcde9e5f6954f00845a93827d89532ac2c3 Author: Ayush Pandey <32012449+itsayushpan...@users.noreply.github.com> AuthorDate: Thu Oct 3 09:10:12 2024 -0400 Rag opensearch usecase with Beam's MLTransform (#32018) * Adding insertion and enrichment pipeline * Enhanced Data Schema * Added Apache Licensed to the notebook * Adding Chunking Strategy * removed unused imports * Modified insertion logic in redis for incorporating chunking strategy * refacted redis code * code review changes * Added chunking code in notebook * Added code review changes * Code review changes: using chunking strategy as enum * Added Code Review Changes * Code review changes * Added code review changes * Added Code Review Changes * Code review changes * Ingestion and Enrichment pipeline for OpenSearch Vector DB * Added logic for reading password from .env file * Added opensearch vector notebook * Update credentials.env * Added code review changes * Added Description in opensearch notebook * Added description in opensearch notebook * Code review changes --- .../beam-ml/rag_usecase/opensearch_connector.py| 372 + .../beam-ml/rag_usecase/opensearch_enrichment.py | 134 ++ .../rag_usecase/opensearch_rag_pipeline.ipynb | 1719 3 files changed, 2225 insertions(+) diff --git a/examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py b/examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py new file mode 100644 index 000..fc83c8d443c --- /dev/null +++ b/examples/notebooks/beam-ml/rag_usecase/opensearch_connector.py @@ -0,0 +1,372 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import absolute_import + +import apache_beam as beam + +from apache_beam.transforms import DoFn +from apache_beam.transforms import PTransform +from apache_beam.transforms import Reshuffle + +from typing import Optional +from opensearchpy import OpenSearch + +import os +from dotenv import load_dotenv + +load_dotenv() + +# Set the logging level to reduce verbose information +import logging + +logging.root.setLevel(logging.INFO) +logger = logging.getLogger(__name__) + +__all__ = ['InsertDocInOpenSearch', 'InsertEmbeddingInOpenSearch'] + +"""This module implements IO classes to read document in Opensearch. + + +Insert Doc in OpenSearch: +- +:class:`InsertDocInOpenSearch` is a ``PTransform`` that writes key and values to a +configured sink, and the write is conducted through a Opensearch pipeline. + +The ptransform works by getting the first and second elements from the input, +this means that inputs like `[k,v]` or `(k,v)` are valid. + +Example usage:: + + pipeline | InsertDocInOpenSearch(host='localhost', + port=6379, + username='admin', + password='admin' + batch_size=100) + + +No backward compatibility guarantees. Everything in this module is experimental. +""" + + +class InsertDocInOpenSearch(PTransform): +"""InsertDocInOpensearch is a ``PTransform`` that writes a ``PCollection`` of +key, value tuple or 2-element array into a Opensearch server. +""" + +def __init__(self, + host: str, + port: int, + username: Optional[str], + password: Optional[str], + batch_size: int = 100 + ): +""" +Args: +host (str): The opensearch host +port (int): The opensearch port +us
(beam) branch master updated: Call out breaking assert_that change more explicitly (#32624)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d9ffc23ae13 Call out breaking assert_that change more explicitly (#32624) d9ffc23ae13 is described below commit d9ffc23ae13742788d1c38c5109ef518a24d89cb Author: Danny McCormick AuthorDate: Wed Oct 2 09:52:50 2024 -0400 Call out breaking assert_that change more explicitly (#32624) --- CHANGES.md | 1 + sdks/python/apache_beam/testing/util.py | 7 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index d92639d626b..3d224157e59 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -75,6 +75,7 @@ ## Breaking Changes +* In Python, assert_that now throws if it is not in a pipeline context instead of silently succeeding ([#30771](https://github.com/apache/beam/pull/30771)) * In Python and YAML, ReadFromJson now override the dtype from None to an explicit False. Most notably, string values like `"123"` are preserved as strings rather than silently coerced (and possibly truncated) to numeric diff --git a/sdks/python/apache_beam/testing/util.py b/sdks/python/apache_beam/testing/util.py index f7fabde43d4..8532d1c1f97 100644 --- a/sdks/python/apache_beam/testing/util.py +++ b/sdks/python/apache_beam/testing/util.py @@ -266,7 +266,12 @@ def assert_that( # The pipeline was already run. The user most likely called assert_that # after the pipeleline context. raise RuntimeError( -'assert_that must be used within a beam.Pipeline context') +'assert_that must be used within a beam.Pipeline context. ' + +'Prior to Beam 2.60.0, asserts outside of the context of a pipeline ' + +'were silently ignored, starting with Beam 2.60.0 this is no longer ' + +'allowed. To fix, move your assert_that call into your pipeline ' + +'context so that it is added before the pipeline is run. For more ' + +'information, see https://github.com/apache/beam/pull/30771') # Usually, the uniqueness of the label is left to the pipeline # writer to guarantee. Since we're in a testing context, we'll
(beam) branch users/damccorm/breaking deleted (was 05489829fc4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/breaking in repository https://gitbox.apache.org/repos/asf/beam.git was 05489829fc4 Call out breaking assert_that change more explicitly The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) 01/01: Call out breaking assert_that change more explicitly
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/breaking in repository https://gitbox.apache.org/repos/asf/beam.git commit 05489829fc4c5a7f8f0b103551f9c9b20365e89f Author: Danny McCormick AuthorDate: Wed Oct 2 09:00:28 2024 -0400 Call out breaking assert_that change more explicitly --- CHANGES.md | 1 + sdks/python/apache_beam/testing/util.py | 7 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index d92639d626b..3d224157e59 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -75,6 +75,7 @@ ## Breaking Changes +* In Python, assert_that now throws if it is not in a pipeline context instead of silently succeeding ([#30771](https://github.com/apache/beam/pull/30771)) * In Python and YAML, ReadFromJson now override the dtype from None to an explicit False. Most notably, string values like `"123"` are preserved as strings rather than silently coerced (and possibly truncated) to numeric diff --git a/sdks/python/apache_beam/testing/util.py b/sdks/python/apache_beam/testing/util.py index f7fabde43d4..8532d1c1f97 100644 --- a/sdks/python/apache_beam/testing/util.py +++ b/sdks/python/apache_beam/testing/util.py @@ -266,7 +266,12 @@ def assert_that( # The pipeline was already run. The user most likely called assert_that # after the pipeleline context. raise RuntimeError( -'assert_that must be used within a beam.Pipeline context') +'assert_that must be used within a beam.Pipeline context. ' + +'Prior to Beam 2.60.0, asserts outside of the context of a pipeline ' + +'were silently ignored, starting with Beam 2.60.0 this is no longer ' + +'allowed. To fix, move your assert_that call into your pipeline ' + +'context so that it is added before the pipeline is run. For more ' + +'information, see https://github.com/apache/beam/pull/30771') # Usually, the uniqueness of the label is left to the pipeline # writer to guarantee. Since we're in a testing context, we'll
(beam) branch users/damccorm/breaking created (now 05489829fc4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/breaking in repository https://gitbox.apache.org/repos/asf/beam.git at 05489829fc4 Call out breaking assert_that change more explicitly This branch includes the following new commits: new 05489829fc4 Call out breaking assert_that change more explicitly The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/bq_fix deleted (was bc0dde5fc0c)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_fix in repository https://gitbox.apache.org/repos/asf/beam.git was bc0dde5fc0c feedback The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Force BQIO to output elements in the correct row (#32584)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e640b2578fb Force BQIO to output elements in the correct row (#32584)
e640b2578fb is described below
commit e640b2578fb1d31ac5823fc95cceff346624d4b3
Author: Danny McCormick
AuthorDate: Mon Sep 30 12:43:17 2024 -0400
Force BQIO to output elements in the correct row (#32584)
* Fix bqio
* import fix
* syntax
* feedback
---
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
...tCommit_Python_ValidatesContainer_Dataflow.json | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 59 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 1 +
4 files changed, 38 insertions(+), 27 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index 30ee463ad4e..1eb60f6e495 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run.",
- "modification": 2
+ "modification": 3
}
diff --git
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
index d6c608f6dab..4897480d69a 100644
---
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
+++
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
@@ -1,3 +1,4 @@
{
-"comment": "Modify this file in a trivial way to cause this test suite to
run"
+"comment": "Modify this file in a trivial way to cause this test suite to
run",
+"modification": 1
}
\ No newline at end of file
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py
b/sdks/python/apache_beam/io/gcp/bigquery.py
index b897df2d32a..2cb64742f26 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -418,7 +418,6 @@ from apache_beam.transforms.external import
SchemaAwareExternalTransform
from apache_beam.transforms.sideinputs import SIDE_INPUT_PREFIX
from apache_beam.transforms.sideinputs import get_sideinput_index
from apache_beam.transforms.util import ReshufflePerKey
-from apache_beam.transforms.window import GlobalWindows
from apache_beam.typehints.row_type import RowTypeConstraint
from apache_beam.typehints.schemas import schema_from_element_type
from apache_beam.utils import retry
@@ -1581,7 +1580,8 @@ class BigQueryWriteFn(DoFn):
additional_create_parameters=self.additional_bq_parameters)
_KNOWN_TABLES.add(str_table_reference)
- def process(self, element, *schema_side_inputs):
+ def process(
+ self, element, window_value=DoFn.WindowedValueParam,
*schema_side_inputs):
destination = bigquery_tools.get_hashable_destination(element[0])
if callable(self.schema):
@@ -1608,12 +1608,11 @@ class BigQueryWriteFn(DoFn):
return [
pvalue.TaggedOutput(
BigQueryWriteFn.FAILED_ROWS_WITH_ERRORS,
-GlobalWindows.windowed_value(
+window_value.with_value(
(destination, row_and_insert_id[0], error))),
pvalue.TaggedOutput(
BigQueryWriteFn.FAILED_ROWS,
-GlobalWindows.windowed_value(
-(destination, row_and_insert_id[0])))
+window_value.with_value((destination, row_and_insert_id[0])))
]
# Flush current batch first if adding this row will exceed our limits
@@ -1624,11 +1623,11 @@ class BigQueryWriteFn(DoFn):
flushed_batch = self._flush_batch(destination)
# After flushing our existing batch, we now buffer the current row
# for the next flush
-self._rows_buffer[destination].append(row_and_insert_id)
+self._rows_buffer[destination].append((row_and_insert_id,
window_value))
self._destination_buffer_byte_size[destination] = row_byte_size
return flushed_batch
- self._rows_buffer[destination].append(row_and_insert_id)
+ self._rows_buffer[destination].append((row_and_insert_id, window_value))
self._destination_buffer_byte_size[destination] += row_byte_size
self._total_buffered_rows += 1
if self._total_buffered_rows >= self._max_buffered_rows:
@@ -1636,7 +1635,8 @@ class BigQueryWriteFn(DoFn):
else:
# The input is already batched per destination, flush the rows now.
batched_rows = element[1]
- self._rows_buffer[destination].extend(batched_rows)
+ for r in batched_rows:
+self._rows_buffer[destination].appen
(beam) branch users/damccorm/bq_fix updated (c19dc93ad00 -> bc0dde5fc0c)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_fix in repository https://gitbox.apache.org/repos/asf/beam.git from c19dc93ad00 syntax add bc0dde5fc0c feedback No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery.py | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
(beam) branch users/damccorm/bq_fix updated (e41c2eccb9f -> c19dc93ad00)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_fix in repository https://gitbox.apache.org/repos/asf/beam.git from e41c2eccb9f import fix add c19dc93ad00 syntax No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery.py | 36 +- 1 file changed, 16 insertions(+), 20 deletions(-)
(beam) branch users/damccorm/bq_fix updated (62bd70e852a -> e41c2eccb9f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_fix in repository https://gitbox.apache.org/repos/asf/beam.git from 62bd70e852a Fix bqio add e41c2eccb9f import fix No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Fix bqio
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/bq_fix
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 62bd70e852a99c3b5c878ba3e598148cc172746c
Author: Danny McCormick
AuthorDate: Fri Sep 27 15:59:57 2024 -0400
Fix bqio
---
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
...tCommit_Python_ValidatesContainer_Dataflow.json | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 41 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 1 +
4 files changed, 29 insertions(+), 18 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index 30ee463ad4e..1eb60f6e495 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run.",
- "modification": 2
+ "modification": 3
}
diff --git
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
index d6c608f6dab..4897480d69a 100644
---
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
+++
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
@@ -1,3 +1,4 @@
{
-"comment": "Modify this file in a trivial way to cause this test suite to
run"
+"comment": "Modify this file in a trivial way to cause this test suite to
run",
+"modification": 1
}
\ No newline at end of file
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py
b/sdks/python/apache_beam/io/gcp/bigquery.py
index b897df2d32a..9ecf9e635c9 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -1581,7 +1581,11 @@ class BigQueryWriteFn(DoFn):
additional_create_parameters=self.additional_bq_parameters)
_KNOWN_TABLES.add(str_table_reference)
- def process(self, element, *schema_side_inputs):
+ def process(
+ self,
+ element,
+ window_value=beam.DoFn.WindowedValueParam,
+ *schema_side_inputs):
destination = bigquery_tools.get_hashable_destination(element[0])
if callable(self.schema):
@@ -1608,12 +1612,11 @@ class BigQueryWriteFn(DoFn):
return [
pvalue.TaggedOutput(
BigQueryWriteFn.FAILED_ROWS_WITH_ERRORS,
-GlobalWindows.windowed_value(
+window_value.with_value(
(destination, row_and_insert_id[0], error))),
pvalue.TaggedOutput(
BigQueryWriteFn.FAILED_ROWS,
-GlobalWindows.windowed_value(
-(destination, row_and_insert_id[0])))
+window_value.with_value((destination, row_and_insert_id[0])))
]
# Flush current batch first if adding this row will exceed our limits
@@ -1624,11 +1627,11 @@ class BigQueryWriteFn(DoFn):
flushed_batch = self._flush_batch(destination)
# After flushing our existing batch, we now buffer the current row
# for the next flush
-self._rows_buffer[destination].append(row_and_insert_id)
+self._rows_buffer[destination].append((row_and_insert_id,
window_value))
self._destination_buffer_byte_size[destination] = row_byte_size
return flushed_batch
- self._rows_buffer[destination].append(row_and_insert_id)
+ self._rows_buffer[destination].append((row_and_insert_id, window_value))
self._destination_buffer_byte_size[destination] += row_byte_size
self._total_buffered_rows += 1
if self._total_buffered_rows >= self._max_buffered_rows:
@@ -1636,7 +1639,8 @@ class BigQueryWriteFn(DoFn):
else:
# The input is already batched per destination, flush the rows now.
batched_rows = element[1]
- self._rows_buffer[destination].extend(batched_rows)
+ for r in batched_rows:
+self._rows_buffer[destination].append((r, window_value))
return self._flush_batch(destination)
def finish_bundle(self):
@@ -1659,7 +1663,7 @@ class BigQueryWriteFn(DoFn):
def _flush_batch(self, destination):
# Flush the current batch of rows to BigQuery.
-rows_and_insert_ids = self._rows_buffer[destination]
+rows_and_insert_ids_with_windows = self._rows_buffer[destination]
table_reference = bigquery_tools.parse_table_reference(destination)
if table_reference.projectId is None:
table_reference.projectId = vp.RuntimeValueProvider.get_value(
@@ -1668,9 +1672,11 @@ class BigQueryWriteFn(DoFn):
_LOGGER.debug(
'Flushing data to %s. Total %s rows.',
destination,
-len(rows_and_insert_ids))
-
(beam) branch users/damccorm/bq_fix created (now 62bd70e852a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_fix in repository https://gitbox.apache.org/repos/asf/beam.git at 62bd70e852a Fix bqio This branch includes the following new commits: new 62bd70e852a Fix bqio The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/bq_writes deleted (was 6f8f9a9be16)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_writes in repository https://gitbox.apache.org/repos/asf/beam.git was 6f8f9a9be16 fmt The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch users/damccorm/bq_writes created (now d674a8ed417)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_writes in repository https://gitbox.apache.org/repos/asf/beam.git at d674a8ed417 Always write to BQ from global window This branch includes the following new commits: new d674a8ed417 Always write to BQ from global window The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/bq_writes updated (d674a8ed417 -> 6f8f9a9be16)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_writes in repository https://gitbox.apache.org/repos/asf/beam.git from d674a8ed417 Always write to BQ from global window add 6f8f9a9be16 fmt No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) 01/01: Always write to BQ from global window
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/bq_writes
in repository https://gitbox.apache.org/repos/asf/beam.git
commit d674a8ed41764f6d37f8a6f1981c8dea8c2530ae
Author: Danny McCormick
AuthorDate: Fri Sep 27 13:12:56 2024 -0400
Always write to BQ from global window
---
.github/trigger_files/beam_PostCommit_Python.json | 2 +-
.../beam_PostCommit_Python_ValidatesContainer_Dataflow.json | 3 ++-
sdks/python/apache_beam/io/gcp/bigquery.py| 3 +++
sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 4 ++--
4 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index 30ee463ad4e..1eb60f6e495 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run.",
- "modification": 2
+ "modification": 3
}
diff --git
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
index d6c608f6dab..4897480d69a 100644
---
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
+++
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
@@ -1,3 +1,4 @@
{
-"comment": "Modify this file in a trivial way to cause this test suite to
run"
+"comment": "Modify this file in a trivial way to cause this test suite to
run",
+"modification": 1
}
\ No newline at end of file
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py
b/sdks/python/apache_beam/io/gcp/bigquery.py
index b897df2d32a..a5f84dc441e 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -1864,6 +1864,9 @@ class _StreamToBigQuery(PTransform):
return (
tagged_data
| 'FromHashableTableRef' >> beam.Map(_restore_table_ref)
+# Use global window for writes since we're outputting back into the
+# global window.
+| 'Window into Global Window' >> beam.WindowInto(GlobalWindows())
| 'StreamInsertRows' >> ParDo(
bigquery_write_fn, *self.schema_side_inputs).with_outputs(
BigQueryWriteFn.FAILED_ROWS,
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
index b0140793cf7..bb5c36a3e9b 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
@@ -470,7 +470,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
input_data = [{
'number': 1,
'str': 'some_string',
-}]
+}]*500
table_schema = {
"fields": [{
@@ -483,7 +483,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
bq_result_errors = [({
'number': 1,
'str': 'some_string',
-}, "Not Found")]
+}, "Not Found")]*500
args = self.test_pipeline.get_full_options_as_args()
(beam) branch users/damccorm/bq_sampling updated (4e0ad877043 -> 82a05fee9fe)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 4e0ad877043 return type add 82a05fee9fe lint No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/utils/windowed_value_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch users/damccorm/bq_sampling updated (67c8349a024 -> 4e0ad877043)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 67c8349a024 fix to_global_window add 4e0ad877043 return type No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/coders/coder_impl.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
(beam) branch users/damccorm/bq_sampling updated (674d3caba0a -> 67c8349a024)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 674d3caba0a ssmall fix add 67c8349a024 fix to_global_window No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch users/damccorm/bq_sampling updated (a67a9d85189 -> e76d4c65a05)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from a67a9d85189 Add hash function add e76d4c65a05 use member functions instead of inheritance No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/coders/coder_impl.py | 21 +-- sdks/python/apache_beam/transforms/window.py | 30 +++- 2 files changed, 27 insertions(+), 24 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (e76d4c65a05 -> 674d3caba0a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from e76d4c65a05 use member functions instead of inheritance add 674d3caba0a ssmall fix No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window.py | 10 ++ 1 file changed, 6 insertions(+), 4 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (8e76e1aee1b -> a67a9d85189)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 8e76e1aee1b revert setup changes add a67a9d85189 Add hash function No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/coders/coder_impl.py | 2 +- sdks/python/apache_beam/transforms/window.py | 17 + 2 files changed, 10 insertions(+), 9 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (948eb33fa8a -> 280691bec5a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 948eb33fa8a Simpler repro of problem add 280691bec5a Fix coder issues No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window.py | 4 ++-- sdks/python/apache_beam/utils/windowed_value_test.py | 9 + sdks/python/setup.py | 5 +++-- 3 files changed, 10 insertions(+), 8 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (280691bec5a -> fed480e2983)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 280691bec5a Fix coder issues add fed480e2983 clean up equals No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/window.py | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (e2705dde452 -> 8e76e1aee1b)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from e2705dde452 fully translate back add 8e76e1aee1b revert setup changes No new revisions were added by this update. Summary of changes: sdks/python/setup.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (fed480e2983 -> e2705dde452)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from fed480e2983 clean up equals add e2705dde452 fully translate back No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/coders/coder_impl.py | 7 +++ 1 file changed, 7 insertions(+)
(beam) branch users/damccorm/bq_sampling updated (65e382594e3 -> 948eb33fa8a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 65e382594e3 Got the repro add 948eb33fa8a Simpler repro of problem No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json| 2 +- ...am_PostCommit_Python_ValidatesContainer_Dataflow.json | 3 +-- sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 1 - sdks/python/apache_beam/utils/windowed_value_test.py | 16 4 files changed, 18 insertions(+), 4 deletions(-)
(beam) branch users/damccorm/bq_sampling updated (3a472e03401 -> 65e382594e3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git from 3a472e03401 More suites add 65e382594e3 Got the repro No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam) branch users/damccorm/bq_sampling updated: More suites
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/bq_sampling
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/users/damccorm/bq_sampling by
this push:
new 3a472e03401 More suites
3a472e03401 is described below
commit 3a472e0340149954ef9c56e2657f6fd873a72549
Author: Danny McCormick
AuthorDate: Thu Sep 26 10:13:35 2024 -0400
More suites
---
.../beam_PostCommit_Python_ValidatesContainer_Dataflow.json| 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
index d6c608f6dab..a88b73a2acd 100644
---
a/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
+++
b/.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json
@@ -1,3 +1,4 @@
{
-"comment": "Modify this file in a trivial way to cause this test suite to
run"
+"comment": "Modify this file in a trivial way to cause this test suite to
run",
+"modification": "1"
}
\ No newline at end of file
(beam) branch users/damccorm/bq_sampling created (now 3d3168773b2)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/bq_sampling in repository https://gitbox.apache.org/repos/asf/beam.git at 3d3168773b2 Try to repro failure This branch includes the following new commits: new 3d3168773b2 Try to repro failure The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Try to repro failure
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/bq_sampling
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 3d3168773b29edab2a7de19facdd0219fdc98c75
Author: Danny McCormick
AuthorDate: Thu Sep 26 10:12:13 2024 -0400
Try to repro failure
---
.github/trigger_files/beam_PostCommit_Python.json| 2 +-
sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index 2934a91b84b..30ee463ad4e 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to
run.",
- "modification": 1
+ "modification": 2
}
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
index b0140793cf7..bb5c36a3e9b 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
@@ -470,7 +470,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
input_data = [{
'number': 1,
'str': 'some_string',
-}]
+}]*500
table_schema = {
"fields": [{
@@ -483,7 +483,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
bq_result_errors = [({
'number': 1,
'str': 'some_string',
-}, "Not Found")]
+}, "Not Found")]*500
args = self.test_pipeline.get_full_options_as_args()
(beam) branch damccorm-patch-1 deleted (was 6a46b0de0db)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git was 6a46b0de0db Take version from default The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (2d5e72c615d -> 5d8b0418cef)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 2d5e72c615d Bump go.mongodb.org/mongo-driver from 1.16.1 to 1.17.0 in /sdks (#32497) add 5d8b0418cef Update build.gradle to use 3.12 for metrics report (#32547) No new revisions were added by this update. Summary of changes: .test-infra/jenkins/build.gradle | 3 --- 1 file changed, 3 deletions(-)
(beam) branch master updated: Vllm model handler (#32410)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 97cb452e4b9 Vllm model handler (#32410)
97cb452e4b9 is described below
commit 97cb452e4b92603ea0a28a225c6ff1d60200840e
Author: Danny McCormick
AuthorDate: Tue Sep 24 14:00:43 2024 -0400
Vllm model handler (#32410)
* Vllm first pass [wip]
* Example for integration tests wip
* Still wip
* Test changes
* Dockerfile improvements
* Remove bad change
* Clean up test args
* clean up invocation
* string fix
* string fix
* clean up
* lint
* Get tests working with 5xx driver
* cleanup
* Fixes, everything is now working
* Batching
* lint
* Feedback + CHANGES.md
---
.github/trigger_files/beam_PostCommit_Python.json | 4 +-
CHANGES.md | 15 +-
build.gradle.kts | 1 +
.../apache_beam/examples/inference/README.md | 80 ++
.../examples/inference/vllm_text_completion.py | 162 +++
.../ml/inference/test_resources/vllm.dockerfile| 47
.../apache_beam/ml/inference/vllm_inference.py | 312 +
sdks/python/setup.py | 1 +
sdks/python/test-suites/dataflow/common.gradle | 39 +++
9 files changed, 646 insertions(+), 15 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python.json
b/.github/trigger_files/beam_PostCommit_Python.json
index d01a47e7265..30ee463ad4e 100644
--- a/.github/trigger_files/beam_PostCommit_Python.json
+++ b/.github/trigger_files/beam_PostCommit_Python.json
@@ -1,5 +1,5 @@
{
- "comment": "modify this file in a trivial way to cause this test suite to
run.",
- "modification": 1
+ "comment": "Modify this file in a trivial way to cause this test suite to
run.",
+ "modification": 2
}
diff --git a/CHANGES.md b/CHANGES.md
index d58ceffeb41..c123a8e1a4d 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -57,18 +57,13 @@
## Highlights
-* New highly anticipated feature X added to Python SDK
([#X](https://github.com/apache/beam/issues/X)).
-* New highly anticipated feature Y added to Java SDK
([#Y](https://github.com/apache/beam/issues/Y)).
-
-## I/Os
-
-* Support for X source added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
+* Added support for using vLLM in the RunInference transform (Python)
([#32528](https://github.com/apache/beam/issues/32528))
## New Features / Improvements
* Dataflow worker can install packages from Google Artifact Registry Python
repositories (Python) ([#32123](https://github.com/apache/beam/issues/32123)).
* Added support for Zstd codec in SerializableAvroCodecFactory (Java)
([#32349](https://github.com/apache/beam/issues/32349))
-* X feature added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
+* Added support for using vLLM in the RunInference transform (Python)
([#32528](https://github.com/apache/beam/issues/32528))
## Breaking Changes
@@ -77,11 +72,9 @@
as strings rather than silently coerced (and possibly truncated) to numeric
values. To retain the old behavior, pass `dtype=True` (or any other value
accepted by `pandas.read_json`).
-* X behavior was changed ([#X](https://github.com/apache/beam/issues/X)).
## Deprecations
-* X behavior is deprecated and will be removed in X versions
([#X](https://github.com/apache/beam/issues/X)).
* Python 3.8 is reaching EOL and support is being removed in Beam 2.61.0. The
2.60.0 release will warn users
when running on 3.8. ([#31192](https://github.com/apache/beam/issues/31192))
@@ -92,10 +85,6 @@ when running on 3.8.
([#31192](https://github.com/apache/beam/issues/31192))
## Security Fixes
* Fixed (CVE--)[https://www.cve.org/CVERecord?id=CVE--]
(Java/Python/Go) ([#X](https://github.com/apache/beam/issues/X)).
-## Known Issues
-
-* ([#X](https://github.com/apache/beam/issues/X)).
-
# [2.59.0] - 2024-09-11
## Highlights
diff --git a/build.gradle.kts b/build.gradle.kts
index d74cae3267e..38b58b6979e 100644
--- a/build.gradle.kts
+++ b/build.gradle.kts
@@ -542,6 +542,7 @@ tasks.register("python312PostCommit") {
dependsOn(":sdks:python:test-suites:direct:py312:postCommitIT")
dependsOn(":sdks:python:test-suites:direct:py312:hdfsIntegrationTest")
dependsOn(":sdks:python:test-suites:portable:py312:postCommitPy312")
+
dependsOn(":sdks:python:test-suites:dataflow:py312:inferencePostCommitITPy312")
}
tasks.register("portablePythonPreCommit") {
diff --git a/sdks/python/apache_beam/examples/inference/README.md
b/sdks/pyth
(beam) branch damccorm-patch-1 updated (81541f0a0e9 -> 6a46b0de0db)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git from 81541f0a0e9 Update build.gradle to use 3.12 add 6a46b0de0db Take version from default No new revisions were added by this update. Summary of changes: .test-infra/jenkins/build.gradle | 3 --- 1 file changed, 3 deletions(-)
(beam) 01/01: Update build.gradle to use 3.12
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch damccorm-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 81541f0a0e9f336a6c99db8271d14004de3b7006
Author: Danny McCormick
AuthorDate: Tue Sep 24 12:37:52 2024 -0400
Update build.gradle to use 3.12
---
.test-infra/jenkins/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.test-infra/jenkins/build.gradle b/.test-infra/jenkins/build.gradle
index 37c9c4d8d6a..9f4acf8b791 100644
--- a/.test-infra/jenkins/build.gradle
+++ b/.test-infra/jenkins/build.gradle
@@ -23,7 +23,7 @@ applyGroovyNature()
applyPythonNature()
// TODO(https://github.com/apache/beam/issues/20209): Don't hardcode this
version, take the value provided by Python nature.
-pythonVersion = '3.8'
+pythonVersion = '3.12'
task generateMetricsReport {
dependsOn setupVirtualenv
(beam) branch damccorm-patch-1 created (now 81541f0a0e9)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git at 81541f0a0e9 Update build.gradle to use 3.12 This branch includes the following new commits: new 81541f0a0e9 Update build.gradle to use 3.12 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/assert deleted (was 149f355cc84)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/assert in repository https://gitbox.apache.org/repos/asf/beam.git was 149f355cc84 Fix elements The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Add schrodinger logo (#32525)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6adb1453890 Add schrodinger logo (#32525) 6adb1453890 is described below commit 6adb1453890e804d7dd62be38e3ba367ceff2eb0 Author: Hai Joey Tran AuthorDate: Sun Sep 22 16:09:38 2024 -0400 Add schrodinger logo (#32525) * Add schrodinger card * move png --- .../www/site/content/en/case-studies/schrodinger.md | 19 +++ .../static/images/logos/powered-by/schrodinger.png | Bin 0 -> 145837 bytes 2 files changed, 19 insertions(+) diff --git a/website/www/site/content/en/case-studies/schrodinger.md b/website/www/site/content/en/case-studies/schrodinger.md new file mode 100644 index 000..178f87a768d --- /dev/null +++ b/website/www/site/content/en/case-studies/schrodinger.md @@ -0,0 +1,19 @@ +--- +title: "Schrodinger" +icon: /images/logos/powered-by/schrodinger.png +hasNav: false +hasLink: "https://www.schrodinger.com/"; +--- + diff --git a/website/www/site/static/images/logos/powered-by/schrodinger.png b/website/www/site/static/images/logos/powered-by/schrodinger.png new file mode 100644 index 000..83a95c752a9 Binary files /dev/null and b/website/www/site/static/images/logos/powered-by/schrodinger.png differ
(beam) branch users/damccorm/assert updated (50dd0487724 -> 149f355cc84)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/assert in repository https://gitbox.apache.org/repos/asf/beam.git from 50dd0487724 Update beam_PostCommit_Python.json add 149f355cc84 Fix elements No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
(beam) branch users/damccorm/assert updated (855bd54bba1 -> 50dd0487724)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/assert in repository https://gitbox.apache.org/repos/asf/beam.git from 855bd54bba1 Move assert into pipeline add 50dd0487724 Update beam_PostCommit_Python.json No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch users/damccorm/assert created (now 855bd54bba1)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/assert in repository https://gitbox.apache.org/repos/asf/beam.git at 855bd54bba1 Move assert into pipeline This branch includes the following new commits: new 855bd54bba1 Move assert into pipeline The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Move assert into pipeline
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/assert in repository https://gitbox.apache.org/repos/asf/beam.git commit 855bd54bba17be7bcdde19914cef863819f21cb7 Author: Danny McCormick AuthorDate: Sat Sep 21 11:53:31 2024 -0400 Move assert into pipeline --- sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 22 +++--- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/sdks/python/apache_beam/ml/gcp/visionml_test_it.py b/sdks/python/apache_beam/ml/gcp/visionml_test_it.py index ea3fc9768ff..cbd59307c24 100644 --- a/sdks/python/apache_beam/ml/gcp/visionml_test_it.py +++ b/sdks/python/apache_beam/ml/gcp/visionml_test_it.py @@ -64,17 +64,17 @@ class VisionMlTestIT(unittest.TestCase): context_side_input=beam.pvalue.AsDict(contexts)) | beam.ParDo(extract)) -assert_that( -output, -equal_to([ -'WAITING?\nPLEASE\nTURN OFF\nYOUR\nENGINE', -'WAITING?', -'PLEASE', -'TURN', -'OFF', -'YOUR', -'ENGINE' -])) + assert_that( + output, + equal_to([ + 'WAITING?\nPLEASE\nTURN OFF\nYOUR\nENGINE', + 'WAITING?', + 'PLEASE', + 'TURN', + 'OFF', + 'YOUR', + 'ENGINE' + ])) if __name__ == '__main__':
(beam) branch master updated (7474e6a7427 -> 6a095456aa3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 7474e6a7427 Rollback Bigtable throttling counter (#32442) add 6a095456aa3 [Python] Add a couple quality-of-life improvemenets to `testing.util.assert_that` (#30771) No new revisions were added by this update. Summary of changes: .../transforms/aggregation/groupby_expr.py | 5 +- .../aggregation/groupby_global_aggregate.py| 5 +- .../aggregation/groupby_simple_aggregate.py| 5 +- .../transforms/aggregation/groupby_test.py | 125 + sdks/python/apache_beam/testing/util.py| 17 +++ sdks/python/apache_beam/testing/util_test.py | 13 +++ sdks/python/apache_beam/transforms/trigger_test.py | 11 +- sdks/python/apache_beam/transforms/util_test.py| 8 +- 8 files changed, 121 insertions(+), 68 deletions(-)
(beam) branch liferoad-patch-2 deleted (was 4a00cd48b11)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch liferoad-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git was 4a00cd48b11 Update run_inference_huggingface.ipynb The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Publish Java Expansion Service SNAPSHOT containers (#32491)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new bb96ac0456c Publish Java Expansion Service SNAPSHOT containers (#32491) bb96ac0456c is described below commit bb96ac0456cb002908e1e723cc5677039e34f7ce Author: Chamikara Jayalath AuthorDate: Thu Sep 19 07:02:32 2024 -0700 Publish Java Expansion Service SNAPSHOT containers (#32491) --- .github/workflows/beam_Publish_Beam_SDK_Snapshots.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml index 72de1f15229..61ef31a0023 100644 --- a/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml +++ b/.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml @@ -71,6 +71,7 @@ jobs: - "python:container:py310" - "python:container:py311" - "python:container:py312" + - "java:expansion-service:container" steps: - uses: actions/checkout@v4 - name: Setup repository
(beam) branch master updated (c4e2e950531 -> 85581a3c02f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c4e2e950531 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#32397) add 85581a3c02f Update run_inference_huggingface.ipynb (#32488) No new revisions were added by this update. Summary of changes: examples/notebooks/beam-ml/run_inference_huggingface.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) branch master updated: [yaml] Add examples for Spanner IO in YAML (#32288)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 9f8a4b2d177 [yaml] Add examples for Spanner IO in YAML (#32288) 9f8a4b2d177 is described below commit 9f8a4b2d177c6308da112411c976b8ed1425e412 Author: Reeba Qureshi <64488642+reeba...@users.noreply.github.com> AuthorDate: Tue Sep 17 02:12:51 2024 +0600 [yaml] Add examples for Spanner IO in YAML (#32288) * Add example for spanner read * Add example for spanner write * move spanner examples * minor update * minor changes 1. Add good element in spanner write example to pass checks. 2. Remove spanner examples from examples_test.py for the time being. * add license --- .../apache_beam/yaml/examples/io/spanner_read.yaml | 80 ++ .../yaml/examples/io/spanner_write.yaml| 53 ++ 2 files changed, 133 insertions(+) diff --git a/sdks/python/apache_beam/yaml/examples/io/spanner_read.yaml b/sdks/python/apache_beam/yaml/examples/io/spanner_read.yaml new file mode 100644 index 000..c86d42c1e0c --- /dev/null +++ b/sdks/python/apache_beam/yaml/examples/io/spanner_read.yaml @@ -0,0 +1,80 @@ +# coding=utf-8 +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +pipeline: + transforms: + + # Reading data from a Spanner database. The table used here has the following columns: + # shipment_id (String), customer_id (String), shipment_date (String), shipment_cost (Float64), customer_name (String), customer_email (String) + # ReadFromSpanner transform is called using project_id, instance_id, database_id and a query + # A table with a list of columns can also be specified instead of a query +- type: ReadFromSpanner + name: ReadShipments + config: +project_id: 'apache-beam-testing' +instance_id: 'shipment-test' +database_id: 'shipment' +query: 'SELECT * FROM shipments' + + # Filtering the data based on a specific condition + # Here, the condition is used to keep only the rows where the customer_id is 'C1' +- type: Filter + name: FilterShipments + input: ReadShipments + config: +language: python +keep: "customer_id == 'C1'" + + # Mapping the data fields and applying transformations + # A new field 'shipment_cost_category' is added with a custom transformation + # A callable is defined to categorize shipment cost +- type: MapToFields + name: MapFieldsForSpanner + input: FilterShipments + config: +language: python +fields: + shipment_id: shipment_id + customer_id: customer_id + shipment_date: shipment_date + shipment_cost: shipment_cost + customer_name: customer_name + customer_email: customer_email + shipment_cost_category: +callable: | + def categorize_cost(row): + cost = float(row[3]) + if cost < 50: + return 'Low Cost' + elif cost < 200: + return 'Medium Cost' + else: + return 'High Cost' + + # Writing the transformed data to a CSV file +- type: WriteToCsv + name: WriteBig + input: MapFieldsForSpanner + config: +path: shipments.csv + + + # On executing the above pipeline, a new CSV file is created with the following records + +# Expected: +# Row(shipment_id='S1', customer_id='C1', shipment_date='2023-05-01', shipment_cost=150.0, customer_name='Alice', customer_email='al...@example.com', shipment_cost_category='Medium Cost') +# Row(shipment_id='S3', customer_id='C1', shipment_date='2023-05-10', shipment_cost=20.0, customer_name='Alice', customer_email='al...@example.com', sh
(beam) branch users/damccorm/beamsummit deleted (was 45036736e0a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/beamsummit in repository https://gitbox.apache.org/repos/asf/beam.git was 45036736e0a Remove beam summit banner The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump spanner max bound back to <4 (#32411)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new ed84acb2967 Bump spanner max bound back to <4 (#32411) ed84acb2967 is described below commit ed84acb2967cf4ffa30a41bd53398ed770271f47 Author: Danny McCormick AuthorDate: Thu Sep 12 13:41:15 2024 -0400 Bump spanner max bound back to <4 (#32411) --- sdks/python/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/setup.py b/sdks/python/setup.py index dd03c590ce3..f9eb9cd8173 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -441,7 +441,7 @@ if __name__ == '__main__': 'google-cloud-bigquery-storage>=2.6.3,<3', 'google-cloud-core>=2.0.0,<3', 'google-cloud-bigtable>=2.19.0,<3', - 'google-cloud-spanner>=3.0.0,<3.48', + 'google-cloud-spanner>=3.0.0,<4', # GCP Packages required by ML functionality 'google-cloud-dlp>=3.0.0,<4', 'google-cloud-language>=2.0,<3',
(beam-starter-python) branch main updated: Bump apache-beam from 2.58.1 to 2.59.0 (#36)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/beam-starter-python.git The following commit(s) were added to refs/heads/main by this push: new cc1f2b7 Bump apache-beam from 2.58.1 to 2.59.0 (#36) cc1f2b7 is described below commit cc1f2b787b5182b0762309c5483f915fd39c8932 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Thu Sep 12 09:34:34 2024 -0400 Bump apache-beam from 2.58.1 to 2.59.0 (#36) Bumps [apache-beam](https://github.com/apache/beam) from 2.58.1 to 2.59.0. - [Release notes](https://github.com/apache/beam/releases) - [Changelog](https://github.com/apache/beam/blob/master/CHANGES.md) - [Commits](https://github.com/apache/beam/compare/v2.58.1...v2.59.0) --- updated-dependencies: - dependency-name: apache-beam dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index c7f74c3..f6bbd90 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1 +1 @@ -apache-beam==2.58.1 +apache-beam==2.59.0
(beam) branch master updated: Temporarily fix pip check failure on xgboost and grpcio (#32432)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 17718a97eaf Temporarily fix pip check failure on xgboost and grpcio
(#32432)
17718a97eaf is described below
commit 17718a97eafe46796d0eaf0ac38f134a3adbe2bf
Author: Shunping Huang
AuthorDate: Thu Sep 12 09:34:01 2024 -0400
Temporarily fix pip check failure on xgboost and grpcio (#32432)
* Pin pip version to 24.1 so pip check wont't fail
* Trigger precomit python ml tests to verify the fix.
* Pin pip version in pyproject.toml
* Try to enforce pip version in tox.
* Disable xgboost tests by removing xgboost dependency.
* Add tox env for macos
* Revert some unnecessary changes
---
.github/trigger_files/beam_PreCommit_Python_ML.json | 0
.github/workflows/python_tests.yml | 8 ++--
sdks/python/setup.py| 5 -
sdks/python/tox.ini | 15 +++
4 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/.github/trigger_files/beam_PreCommit_Python_ML.json
b/.github/trigger_files/beam_PreCommit_Python_ML.json
new file mode 100644
index 000..e69de29bb2d
diff --git a/.github/workflows/python_tests.yml
b/.github/workflows/python_tests.yml
index 3ef9a5fe26b..e7461be2507 100644
--- a/.github/workflows/python_tests.yml
+++ b/.github/workflows/python_tests.yml
@@ -113,10 +113,14 @@ jobs:
python-version: ${{ matrix.params.py_ver }}
- name: Install tox
run: pip install tox
- - name: Run tests basic unix
-if: startsWith(matrix.os, 'ubuntu') || startsWith(matrix.os, 'macos')
+ - name: Run tests basic linux
+if: startsWith(matrix.os, 'ubuntu')
working-directory: ./sdks/python
run: tox -c tox.ini run -e ${{ matrix.params.tox_env }}
+ - name: Run tests basic macos
+if: startsWith(matrix.os, 'macos')
+working-directory: ./sdks/python
+run: tox -c tox.ini run -e ${{ matrix.params.tox_env }}-macos
- name: Run tests basic windows
if: startsWith(matrix.os, 'windows')
working-directory: ./sdks/python
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 7bcff2bacfd..dd03c590ce3 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -498,7 +498,10 @@ if __name__ == '__main__':
'tf2onnx',
'torch',
'transformers',
- 'xgboost<2.0', # https://github.com/apache/beam/issues/31252
+ # Comment out xgboost as it is breaking presubmit python ml
+ # tests due to tag check introduced since pip 24.2
+ # https://github.com/apache/beam/issues/31285
+ # 'xgboost<2.0', # https://github.com/apache/beam/issues/31252
],
'aws': ['boto3>=1.9,<2'],
'azure': [
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index aa0200f7500..da4cde009ce 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -68,6 +68,21 @@ commands_post =
commands = false {envname} is misconfigured
[testenv:py{38,39,310,311,312}]
+commands_pre =
+ python --version
+ pip --version
+ pip check
+ bash {toxinidir}/scripts/run_tox_cleanup.sh
+commands =
+ python apache_beam/examples/complete/autocomplete_test.py
+ bash {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
+
+[testenv:py{38,39,310,311,312}-macos]
+commands_pre =
+ python --version
+ pip --version
+ # pip check
+ bash {toxinidir}/scripts/run_tox_cleanup.sh
commands =
python apache_beam/examples/complete/autocomplete_test.py
bash {toxinidir}/scripts/run_pytest.sh {envname} "{posargs}"
(beam) branch master updated: Dont run flaky test on windows (#32419)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new ebcb2dbd160 Dont run flaky test on windows (#32419) ebcb2dbd160 is described below commit ebcb2dbd160cd729ccc683d3130b0380bbae6efb Author: Danny McCormick AuthorDate: Tue Sep 10 14:55:23 2024 -0400 Dont run flaky test on windows (#32419) * Revert "Try deflaking test timing (#32351)" This reverts commit 7d6f6fb55bc9ffed3468828487132b039daacd3a. * Dont run flaky test on windows --- sdks/python/apache_beam/ml/inference/base_test.py | 13 +++-- 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sdks/python/apache_beam/ml/inference/base_test.py b/sdks/python/apache_beam/ml/inference/base_test.py index 76772132784..31f02c9c61c 100644 --- a/sdks/python/apache_beam/ml/inference/base_test.py +++ b/sdks/python/apache_beam/ml/inference/base_test.py @@ -878,21 +878,22 @@ class RunInferenceBaseTest(unittest.TestCase): bad_without_error, equal_to(expected_bad), label='assert:failures') @unittest.skipIf( - sys.version_info < (3, 11), + sys.platform == "win32" or sys.version_info < (3, 11), "This test relies on the __del__ lifecycle method, but __del__ does " + - "not get invoked in the same way on older versions of Python, " + - "breaking this test. See " + + "not get invoked in the same way on older versions of Python or on " + + "windows, breaking this test. See " + "github.com/python/cpython/issues/87950#issuecomment-1807570983 " + "for example.") def test_run_inference_timeout_does_garbage_collection(self): with tempfile.TemporaryDirectory() as tmp_dirname: tmp_path = os.path.join(tmp_dirname, 'tmp_filename') + expected_file_contents = 'Deleted FakeSlowModel' with TestPipeline() as pipeline: # Start with bad example which gets timed out. # Then provide plenty of time for GC to happen. -examples = [20] + [1] * 15 + [20, 20, 20] +examples = [20] + [1] * 15 expected_good = [1] * 15 -expected_bad = [20, 20, 20, 20] +expected_bad = [20] pcoll = pipeline | 'start' >> beam.Create(examples) main, other = pcoll | base.RunInference( FakeSlowModelHandler( @@ -909,7 +910,7 @@ class RunInferenceBaseTest(unittest.TestCase): with open(tmp_path) as f: s = f.read() -self.assertNotEqual(s, '') +self.assertEqual(s, expected_file_contents) def test_run_inference_impl_inference_args(self): with TestPipeline() as pipeline:
(beam) 02/02: Dont run flaky test on windows
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/test_fix in repository https://gitbox.apache.org/repos/asf/beam.git commit 15fe088cbaba64cca3ac8c2036576bbf4f9e5cc6 Author: Danny McCormick AuthorDate: Tue Sep 10 13:04:23 2024 -0400 Dont run flaky test on windows --- sdks/python/apache_beam/ml/inference/base_test.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/python/apache_beam/ml/inference/base_test.py b/sdks/python/apache_beam/ml/inference/base_test.py index 359a372bc5b..31f02c9c61c 100644 --- a/sdks/python/apache_beam/ml/inference/base_test.py +++ b/sdks/python/apache_beam/ml/inference/base_test.py @@ -878,10 +878,10 @@ class RunInferenceBaseTest(unittest.TestCase): bad_without_error, equal_to(expected_bad), label='assert:failures') @unittest.skipIf( - sys.version_info < (3, 11), + sys.platform == "win32" or sys.version_info < (3, 11), "This test relies on the __del__ lifecycle method, but __del__ does " + - "not get invoked in the same way on older versions of Python, " + - "breaking this test. See " + + "not get invoked in the same way on older versions of Python or on " + + "windows, breaking this test. See " + "github.com/python/cpython/issues/87950#issuecomment-1807570983 " + "for example.") def test_run_inference_timeout_does_garbage_collection(self):
(beam) branch users/damccorm/spanner created (now 3719c1e7eea)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/spanner in repository https://gitbox.apache.org/repos/asf/beam.git at 3719c1e7eea Bump spanner max bound back to <4 This branch includes the following new commits: new 3719c1e7eea Bump spanner max bound back to <4 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: [yaml] Adding Spanner IO Providers for Beam YAML (#31987)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 173dd486842 [yaml] Adding Spanner IO Providers for Beam YAML (#31987)
173dd486842 is described below
commit 173dd486842cec312a975530a9867b54f9f44cf4
Author: Reeba Qureshi <64488642+reeba...@users.noreply.github.com>
AuthorDate: Mon Sep 9 19:58:04 2024 +0530
[yaml] Adding Spanner IO Providers for Beam YAML (#31987)
* Add Spanner IO providers to YAML SDK
* add handling logic for more datatypes
* delete examples
* minor changes
* minor change
* add integration test
* add docs
* minor change
* minor changes
1. Removed serialiazability from ErrorHandling.java
2. Removed double map definitions from MutationUtils.java
3. Added checkNotNull in spanner write provider
4. Modified some variables in spanner wrapper
5. Change instance id in integration tests
* Update spanner_wrapper.py
import retry
* minor changes
1. replace checknotnull with checkargument in spanner read provider
2. use the correct table name (tmp_table) in integration test
* minor changes
1. Added serializable to error handling
2. Corrected validation methods in spanner read
3. Added retry import and removed default project name in spanner wrapper
4. Corrected instance and database names in spanner integration test
5. Corrected table name in query
* formatting
* Update SpannerWriteSchemaTransformProvider.java
* correct lint failures
* correct lint failures
* correct lint failures
* Update SpannerReadSchemaTransformProvider.java
* correct lint failures
* Update SpannerWriteSchemaTransformProvider.java
* spanner version update
---
.../transforms/providers/ErrorHandling.java| 3 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 60 ++
.../SpannerReadSchemaTransformProvider.java| 235 +
.../SpannerWriteSchemaTransformProvider.java | 160 +++---
sdks/python/apache_beam/io/gcp/spanner_wrapper.py | 76 +++
sdks/python/apache_beam/yaml/integration_tests.py | 17 ++
sdks/python/apache_beam/yaml/standard_io.yaml | 27 +++
sdks/python/apache_beam/yaml/tests/spanner.yaml| 95 +
sdks/python/setup.py | 2 +-
9 files changed, 645 insertions(+), 30 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/ErrorHandling.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/ErrorHandling.java
index 7fa29708c9f..053521dbfb3 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/ErrorHandling.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/providers/ErrorHandling.java
@@ -18,13 +18,14 @@
package org.apache.beam.sdk.schemas.transforms.providers;
import com.google.auto.value.AutoValue;
+import java.io.Serializable;
import javax.annotation.Nullable;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.schemas.annotations.SchemaFieldDescription;
import org.apache.beam.sdk.values.Row;
@AutoValue
-public abstract class ErrorHandling {
+public abstract class ErrorHandling implements Serializable {
@SchemaFieldDescription("The name of the output PCollection containing
failed writes.")
public abstract String getOutput();
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
index c0654b2cb05..5a106a34b0c 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtils.java
@@ -26,8 +26,11 @@ import com.google.cloud.ByteArray;
import com.google.cloud.Timestamp;
import com.google.cloud.spanner.Key;
import com.google.cloud.spanner.Mutation;
+import com.google.cloud.spanner.Value;
import java.math.BigDecimal;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.stream.StreamSupport;
import org.apache.beam.sdk.schemas.Schema;
import org.apache.beam.sdk.transforms.SerializableFunction;
@@ -351,4 +354,61 @@ final class MutationUtils {
beamIterableType.getTypeName()));
}
}
+
+ public static Row createRowFromMutation(Schema schema, Mutation mutation) {
+Map mutationHashMap = new HashMap<>();
+mutation
+.asMap()
+.forEach(
+
(beam) branch damccorm-patch-1 deleted (was fb6db657d8a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git was fb6db657d8a Merge branch 'master' of https://github.com/apache/beam into damccorm-patch-1 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (696978018ed -> 3b4ecd7a987)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 696978018ed Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396) add 3b4ecd7a987 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 (#32350) No new revisions were added by this update. Summary of changes: release/build.gradle.kts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) 01/01: Undo part of artifact action upgrade to fix workflow
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/wheels_fix
in repository https://gitbox.apache.org/repos/asf/beam.git
commit f52a535ef334397aa1061fa6b30d1381272971d1
Author: Danny McCormick
AuthorDate: Thu Sep 5 09:29:12 2024 -0700
Undo part of artifact action upgrade to fix workflow
---
.github/workflows/build_wheels.yml | 12
1 file changed, 4 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 3b718b0..f97f4de50ae 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -127,13 +127,11 @@ jobs:
# https://github.com/pypa/setuptools/issues/4300 changed naming. Match
both old and new names.
run: mv $(ls | grep "apache-beam-\|apache_beam-") apache-beam-source
- name: Upload source as artifact
-# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v4
with:
name: source
path: sdks/python/apache-beam-source
- name: Upload compressed sources as artifacts
-# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v4
with:
name: source_zip
@@ -172,14 +170,12 @@ jobs:
run: mv $(ls | grep "apache-beam-\|apache_beam-") apache-beam-source-rc
- name: Upload RC source as artifact
if: steps.is_rc.outputs.is_rc == 1
-# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v4
with:
name: source_rc${{ steps.get_rc_version.outputs.RC_NUM }}
path: sdks/python/apache-beam-source-rc
- name: Upload compressed RC sources as artifacts
if: steps.is_rc.outputs.is_rc == 1
-# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v4
with:
name: source_zip_rc${{ steps.get_rc_version.outputs.RC_NUM }}
@@ -283,8 +279,8 @@ jobs:
shell: bash
- name: Upload wheels as artifacts
if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
- # Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/upload-artifact@v4
+ # Pinned to v3 because of
https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes
+ uses: actions/upload-artifact@v3
with:
name: wheelhouse-${{ matrix.os_python.os }}${{ (matrix.os_python.arch
== 'aarch64' && '-aarch64') || '' }}
path: apache-beam-source/wheelhouse/
@@ -333,8 +329,8 @@ jobs:
arch: aarch64
steps:
- name: Download wheels from artifacts
- # Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v4.1.8
+ # Pinned to v3 because of
https://github.com/actions/upload-artifact?tab=readme-ov-file#breaking-changes
+ uses: actions/download-artifact@v3
with:
name: wheelhouse-${{ matrix.os }}${{ (matrix.arch == 'aarch64' &&
'-aarch64') || '' }}
path: wheelhouse/
(beam) branch master updated: Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 696978018ed Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396)
696978018ed is described below
commit 696978018eddf32d8ed57b5c00829360f1e7476c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Thu Sep 5 08:15:21 2024 -0700
Bump actions/download-artifact from 4.1.7 to 4.1.8 (#32396)
Bumps
[actions/download-artifact](https://github.com/actions/download-artifact) from
4.1.7 to 4.1.8.
- [Release notes](https://github.com/actions/download-artifact/releases)
-
[Commits](https://github.com/actions/download-artifact/compare/v4.1.7...v4.1.8)
---
updated-dependencies:
- dependency-name: actions/download-artifact
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
.github/workflows/build_wheels.yml | 8
.github/workflows/python_tests.yml | 2 +-
2 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index ebac61eec83..3b718b0 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -207,7 +207,7 @@ jobs:
steps:
- name: Download compressed sources from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/download-artifact@v4.1.7
+uses: actions/download-artifact@v4.1.8
with:
name: source_zip
path: source/
@@ -238,14 +238,14 @@ jobs:
steps:
- name: Download python source distribution from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v4.1.7
+ uses: actions/download-artifact@v4.1.8
with:
name: source
path: apache-beam-source
- name: Download Python SDK RC source distribution from artifacts
if: ${{ needs.build_source.outputs.is_rc == 1 }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v4.1.7
+ uses: actions/download-artifact@v4.1.8
with:
name: source_rc${{ needs.build_source.outputs.rc_num }}
path: apache-beam-source-rc
@@ -334,7 +334,7 @@ jobs:
steps:
- name: Download wheels from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v4.1.7
+ uses: actions/download-artifact@v4.1.8
with:
name: wheelhouse-${{ matrix.os }}${{ (matrix.arch == 'aarch64' &&
'-aarch64') || '' }}
path: wheelhouse/
diff --git a/.github/workflows/python_tests.yml
b/.github/workflows/python_tests.yml
index 7a086e1ebea..3ef9a5fe26b 100644
--- a/.github/workflows/python_tests.yml
+++ b/.github/workflows/python_tests.yml
@@ -172,7 +172,7 @@ jobs:
python-version: ${{ matrix.python }}
go-version: default
- name: Download source from artifacts
-uses: actions/download-artifact@v4.1.7
+uses: actions/download-artifact@v4.1.8
with:
name: python_sdk_source
path: apache-beam-source
(beam) branch master updated: Upgrade artifact actions to v4 (#32391)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 34a7c4f4198 Upgrade artifact actions to v4 (#32391)
34a7c4f4198 is described below
commit 34a7c4f41984a3789cb6fb64973b951e698c15f1
Author: Danny McCormick
AuthorDate: Tue Sep 3 17:33:39 2024 -0700
Upgrade artifact actions to v4 (#32391)
* Upgrade artifact actions to v4
* Fix typo
---
.github/workflows/build_wheels.yml | 20 ++--
.github/workflows/python_tests.yml | 2 +-
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 97a350c3fcc..ebac61eec83 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -128,13 +128,13 @@ jobs:
run: mv $(ls | grep "apache-beam-\|apache_beam-") apache-beam-source
- name: Upload source as artifact
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source
path: sdks/python/apache-beam-source
- name: Upload compressed sources as artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_zip
path: sdks/python/dist
@@ -173,14 +173,14 @@ jobs:
- name: Upload RC source as artifact
if: steps.is_rc.outputs.is_rc == 1
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_rc${{ steps.get_rc_version.outputs.RC_NUM }}
path: sdks/python/apache-beam-source-rc
- name: Upload compressed RC sources as artifacts
if: steps.is_rc.outputs.is_rc == 1
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_zip_rc${{ steps.get_rc_version.outputs.RC_NUM }}
path: sdks/python/dist
@@ -207,7 +207,7 @@ jobs:
steps:
- name: Download compressed sources from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/download-artifact@v3
+uses: actions/download-artifact@v4.1.7
with:
name: source_zip
path: source/
@@ -238,14 +238,14 @@ jobs:
steps:
- name: Download python source distribution from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7
with:
name: source
path: apache-beam-source
- name: Download Python SDK RC source distribution from artifacts
if: ${{ needs.build_source.outputs.is_rc == 1 }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7
with:
name: source_rc${{ needs.build_source.outputs.rc_num }}
path: apache-beam-source-rc
@@ -284,7 +284,7 @@ jobs:
- name: Upload wheels as artifacts
if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: wheelhouse-${{ matrix.os_python.os }}${{ (matrix.os_python.arch
== 'aarch64' && '-aarch64') || '' }}
path: apache-beam-source/wheelhouse/
@@ -310,7 +310,7 @@ jobs:
- name: Upload RC wheels as artifacts
if: ${{ needs.build_source.outputs.is_rc == 1 }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: wheelhouse-rc${{ needs.build_source.outputs.rc_num }}-${{
matrix.os_python.os }}${{ (matrix.arch == 'aarch64' && '-aarch64') || '' }}
path: apache-beam-source-rc/wheelhouse/
@@ -334,7 +334,7 @@ jobs:
steps:
- name: Download wheels from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7
with:
name: wheelhouse-${{ matrix.os }}${{ (matrix.arch
(beam) branch users/damccorm/artifact_versions deleted (was 57874a0fda4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/artifact_versions in repository https://gitbox.apache.org/repos/asf/beam.git was 57874a0fda4 Fix typo The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch users/damccorm/artifact_versions updated (3327296894e -> 57874a0fda4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/artifact_versions in repository https://gitbox.apache.org/repos/asf/beam.git from 3327296894e Upgrade artifact actions to v4 add 57874a0fda4 Fix typo No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 8 1 file changed, 4 insertions(+), 4 deletions(-)
(beam) 01/01: Upgrade artifact actions to v4
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/artifact_versions
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 3327296894edd3eeecbdc5bed554317d15c70993
Author: Danny McCormick
AuthorDate: Tue Sep 3 16:49:11 2024 -0700
Upgrade artifact actions to v4
---
.github/workflows/build_wheels.yml | 20 ++--
.github/workflows/python_tests.yml | 2 +-
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 97a350c3fcc..d956e2702e9 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -128,13 +128,13 @@ jobs:
run: mv $(ls | grep "apache-beam-\|apache_beam-") apache-beam-source
- name: Upload source as artifact
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source
path: sdks/python/apache-beam-source
- name: Upload compressed sources as artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_zip
path: sdks/python/dist
@@ -173,14 +173,14 @@ jobs:
- name: Upload RC source as artifact
if: steps.is_rc.outputs.is_rc == 1
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_rc${{ steps.get_rc_version.outputs.RC_NUM }}
path: sdks/python/apache-beam-source-rc
- name: Upload compressed RC sources as artifacts
if: steps.is_rc.outputs.is_rc == 1
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/upload-artifact@v3
+uses: actions/upload-artifact@v4
with:
name: source_zip_rc${{ steps.get_rc_version.outputs.RC_NUM }}
path: sdks/python/dist
@@ -207,7 +207,7 @@ jobs:
steps:
- name: Download compressed sources from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
-uses: actions/download-artifact@v3
+uses: actions/download-artifact@v4.1.7.1.7
with:
name: source_zip
path: source/
@@ -238,14 +238,14 @@ jobs:
steps:
- name: Download python source distribution from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7.1.7
with:
name: source
path: apache-beam-source
- name: Download Python SDK RC source distribution from artifacts
if: ${{ needs.build_source.outputs.is_rc == 1 }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7.1.7
with:
name: source_rc${{ needs.build_source.outputs.rc_num }}
path: apache-beam-source-rc
@@ -284,7 +284,7 @@ jobs:
- name: Upload wheels as artifacts
if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: wheelhouse-${{ matrix.os_python.os }}${{ (matrix.os_python.arch
== 'aarch64' && '-aarch64') || '' }}
path: apache-beam-source/wheelhouse/
@@ -310,7 +310,7 @@ jobs:
- name: Upload RC wheels as artifacts
if: ${{ needs.build_source.outputs.is_rc == 1 }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: wheelhouse-rc${{ needs.build_source.outputs.rc_num }}-${{
matrix.os_python.os }}${{ (matrix.arch == 'aarch64' && '-aarch64') || '' }}
path: apache-beam-source-rc/wheelhouse/
@@ -334,7 +334,7 @@ jobs:
steps:
- name: Download wheels from artifacts
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4.1.7.1.7
with:
name: wheelhouse-${{ matrix.os }}${{ (matrix.arch == 'aarch64' &&
'-aarch64') || '' }}
path: wheelhouse/
diff --git a/.github/workflows/python_tests.yml
b/.github/workflows/python_tests.yml
index
(beam) branch users/damccorm/artifact_versions created (now 3327296894e)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/artifact_versions in repository https://gitbox.apache.org/repos/asf/beam.git at 3327296894e Upgrade artifact actions to v4 This branch includes the following new commits: new 3327296894e Upgrade artifact actions to v4 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/ri_test deleted (was 1b1bdd8b176)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git was 1b1bdd8b176 Merge branch 'master' of https://github.com/apache/beam into users/damccorm/ri_test The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (55678b2edcf -> 7d6f6fb55bc)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 55678b2edcf Pause delete images in public AR (#32354) add 7d6f6fb55bc Try deflaking test timing (#32351) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/ml/inference/base_test.py | 7 +++ 1 file changed, 3 insertions(+), 4 deletions(-)
(beam) branch damccorm-patch-1 updated (6060a7213d6 -> fb6db657d8a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git from 6060a7213d6 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 add 923fde0f070 Fix pr building of wheels (#32353) add fb6db657d8a Merge branch 'master' of https://github.com/apache/beam into damccorm-patch-1 No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 ++ 1 file changed, 2 insertions(+)
(beam) branch users/damccorm/ri_test updated (cbc792fbc89 -> 1b1bdd8b176)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git from cbc792fbc89 Simpler technique add 923fde0f070 Fix pr building of wheels (#32353) add 1b1bdd8b176 Merge branch 'master' of https://github.com/apache/beam into users/damccorm/ri_test No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 ++ 1 file changed, 2 insertions(+)
(beam) branch damccorm-patch-2 deleted (was 9c39d23f3d3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git was 9c39d23f3d3 Fix pr building of wheels The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (399112b2fab -> 923fde0f070)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 399112b2fab Fix BigtableIO.write() client sharing (#32340) add 923fde0f070 Fix pr building of wheels (#32353) No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 ++ 1 file changed, 2 insertions(+)
(beam) 01/01: Fix pr building of wheels
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch damccorm-patch-2
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 9c39d23f3d34d40e7397ad6ab0cb16867df56093
Author: Danny McCormick
AuthorDate: Wed Aug 28 20:32:12 2024 +0100
Fix pr building of wheels
---
.github/workflows/build_wheels.yml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 69ffec573fc..97a350c3fcc 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -274,6 +274,7 @@ jobs:
if: startsWith(matrix.os_python.os, 'macos')
run: brew install coreutils
- name: Add checksums
+ if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
working-directory: apache-beam-source/wheelhouse/
run: |
for file in *.whl; do
@@ -281,6 +282,7 @@ jobs:
done
shell: bash
- name: Upload wheels as artifacts
+ if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v3
with:
(beam) branch damccorm-patch-2 created (now 9c39d23f3d3)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git at 9c39d23f3d3 Fix pr building of wheels This branch includes the following new commits: new 9c39d23f3d3 Fix pr building of wheels The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/ri_test updated (fc2e1bbfbf3 -> cbc792fbc89)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git from fc2e1bbfbf3 Add another potential timeout add cbc792fbc89 Simpler technique No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/ml/inference/base_test.py | 9 - 1 file changed, 4 insertions(+), 5 deletions(-)
(beam) branch users/damccorm/ri_test updated: Add another potential timeout
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/users/damccorm/ri_test by this push: new fc2e1bbfbf3 Add another potential timeout fc2e1bbfbf3 is described below commit fc2e1bbfbf38f3d7f9ef074517cd16ed8c36a2ac Author: Danny McCormick AuthorDate: Wed Aug 28 20:15:39 2024 +0100 Add another potential timeout --- sdks/python/apache_beam/ml/inference/base_test.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sdks/python/apache_beam/ml/inference/base_test.py b/sdks/python/apache_beam/ml/inference/base_test.py index 00f6563d5f1..cbe12443b31 100644 --- a/sdks/python/apache_beam/ml/inference/base_test.py +++ b/sdks/python/apache_beam/ml/inference/base_test.py @@ -891,9 +891,9 @@ class RunInferenceBaseTest(unittest.TestCase): with TestPipeline() as pipeline: # Start with bad example which gets timed out. # Then provide plenty of time for GC to happen. -examples = [20] + [1] * 60 -expected_good = [1] * 60 -expected_bad = [20] +examples = [20] + [1] * 30 + [20] +expected_good = [1] * 30 +expected_bad = [20, 20] pcoll = pipeline | 'start' >> beam.Create(examples) main, other = pcoll | base.RunInference( FakeSlowModelHandler(
(beam) branch users/damccorm/ri_test created (now 2fc6e1cb06a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git at 2fc6e1cb06a Try deflaking test timing This branch includes the following new commits: new 2fc6e1cb06a Try deflaking test timing The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Try deflaking test timing
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/ri_test in repository https://gitbox.apache.org/repos/asf/beam.git commit 2fc6e1cb06a4c8e95b037e8556a0c3e6682fe2c1 Author: Danny McCormick AuthorDate: Wed Aug 28 20:14:14 2024 +0100 Try deflaking test timing --- sdks/python/apache_beam/ml/inference/base_test.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/ml/inference/base_test.py b/sdks/python/apache_beam/ml/inference/base_test.py index 359a372bc5b..00f6563d5f1 100644 --- a/sdks/python/apache_beam/ml/inference/base_test.py +++ b/sdks/python/apache_beam/ml/inference/base_test.py @@ -891,8 +891,8 @@ class RunInferenceBaseTest(unittest.TestCase): with TestPipeline() as pipeline: # Start with bad example which gets timed out. # Then provide plenty of time for GC to happen. -examples = [20] + [1] * 15 -expected_good = [1] * 15 +examples = [20] + [1] * 60 +expected_good = [1] * 60 expected_bad = [20] pcoll = pipeline | 'start' >> beam.Create(examples) main, other = pcoll | base.RunInference(
(beam) branch damccorm-patch-1 created (now 6060a7213d6)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git at 6060a7213d6 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 This branch includes the following new commits: new 6060a7213d6 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Bump commons-cli:commons-cli from 1.8.0 to 1.9.0
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git commit 6060a7213d63e343bc31d29e0bf6eecf39228dfa Author: Danny McCormick AuthorDate: Wed Aug 28 20:09:07 2024 +0100 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 --- release/build.gradle.kts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/release/build.gradle.kts b/release/build.gradle.kts index 3b55542f939..ca1c152c9eb 100644 --- a/release/build.gradle.kts +++ b/release/build.gradle.kts @@ -29,8 +29,8 @@ val library = project.extensions.extraProperties["library"] as Map
(beam) branch master updated: Fix remote execution test flakiness in tearDown (#32328)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a6cef9210b9 Fix remote execution test flakiness in tearDown (#32328)
a6cef9210b9 is described below
commit a6cef9210b92e2006c29c2f6fe90545a7abe2c8d
Author: Yi Hu
AuthorDate: Wed Aug 28 14:36:29 2024 -0400
Fix remote execution test flakiness in tearDown (#32328)
---
.../beam/runners/fnexecution/control/RemoteExecutionTest.java | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
index 9bdd089c0cd..874748d7b97 100644
---
a/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
+++
b/runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/RemoteExecutionTest.java
@@ -264,10 +264,12 @@ public class RemoteExecutionTest implements Serializable {
try {
sdkHarnessExecutorFuture.get();
} catch (ExecutionException e) {
- if (e.getCause() instanceof RuntimeException
- && e.getCause().getCause() instanceof InterruptedException) {
-// expected
- } else {
+ Throwable ex = e.getCause();
+ while (ex instanceof RuntimeException) {
+ex = ex.getCause();
+ }
+ // InterruptedException at call tree is expected
+ if (!(ex instanceof InterruptedException)) {
throw e;
}
}
(beam) branch master updated: Filter out old actions runs from dashboard (#32347)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a85b0a636dc Filter out old actions runs from dashboard (#32347)
a85b0a636dc is described below
commit a85b0a636dc54483e20f981e3ee5b2b3369578c6
Author: Danny McCormick
AuthorDate: Wed Aug 28 17:26:58 2024 +0100
Filter out old actions runs from dashboard (#32347)
---
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
index e8fe60b4a7d..fd4a5e713fe 100644
--- a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
+++ b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
@@ -307,6 +307,8 @@ async def fetch_workflow_runs():
number_of_entries_per_page = 100 # The number of results per page (max
100)
params = {"branch": "master", "page": page, "per_page":
number_of_entries_per_page}
concurrent_requests = 30 # Number of requests to send simultaneously
+start = datetime.now() - timedelta(days=90)
+earliest_run_creation_date = start.strftime('%Y-%m-%d')
semaphore = asyncio.Semaphore(concurrent_requests)
print("Start fetching recent workflow runs")
@@ -338,6 +340,7 @@ async def fetch_workflow_runs():
"page": page,
"per_page": number_of_entries_per_page,
"exclude_pull_requests": "true",
+"created": f'>={earliest_run_creation_date}',
}
workflow_run_tasks.append(fetch(runs_url, semaphore, params,
headers))
page += 1
(beam) branch users/damccorm/grafanaSync deleted (was 20d27727ba4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/grafanaSync in repository https://gitbox.apache.org/repos/asf/beam.git was 20d27727ba4 Filter out old actions runs from dashboard The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch dependabot/gradle/commons-cli-commons-cli-1.9.0 updated (d02ac289fb0 -> e7e18aad2b1)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch dependabot/gradle/commons-cli-commons-cli-1.9.0 in repository https://gitbox.apache.org/repos/asf/beam.git from d02ac289fb0 Bump commons-cli:commons-cli from 1.8.0 to 1.9.0 add e7e18aad2b1 Fix pr building of wheels (#32348) No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
(beam) branch damccorm-patch-1 deleted (was 278904a32eb)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git was 278904a32eb Fix pr building of wheels The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) 01/01: Fix pr building of wheels
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch damccorm-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 278904a32eb60849890708e75aca9e34df06ce88
Author: Danny McCormick
AuthorDate: Wed Aug 28 15:08:42 2024 +0100
Fix pr building of wheels
---
.github/workflows/build_wheels.yml | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index 69ffec573fc..ccea89a4c20 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -274,6 +274,7 @@ jobs:
if: startsWith(matrix.os_python.os, 'macos')
run: brew install coreutils
- name: Add checksums
+ if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
working-directory: apache-beam-source/wheelhouse/
run: |
for file in *.whl; do
@@ -281,6 +282,7 @@ jobs:
done
shell: bash
- name: Upload wheels as artifacts
+ if: ${{ contains(matrix.os_python.python, matrix.py_version) }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v3
with:
@@ -298,7 +300,7 @@ jobs:
run: cibuildwheel --print-build-identifiers && cibuildwheel --output-dir
wheelhouse
shell: bash
- name: Add RC checksums
- if: ${{ needs.build_source.outputs.is_rc == 1 }}
+ if: ${{ needs.build_source.outputs.is_rc == 1 &&
contains(matrix.os_python.python, matrix.py_version) }}
working-directory: apache-beam-source-rc/wheelhouse/
run: |
for file in *.whl; do
@@ -306,7 +308,7 @@ jobs:
done
shell: bash
- name: Upload RC wheels as artifacts
- if: ${{ needs.build_source.outputs.is_rc == 1 }}
+ if: ${{ needs.build_source.outputs.is_rc == 1 &&
contains(matrix.os_python.python, matrix.py_version) }}
# Pinned to v3 because of
https://github.com/actions/download-artifact/issues/249
uses: actions/upload-artifact@v3
with:
(beam) branch damccorm-patch-1 created (now 278904a32eb)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git at 278904a32eb Fix pr building of wheels This branch includes the following new commits: new 278904a32eb Fix pr building of wheels The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch users/damccorm/grafanaSync created (now 20d27727ba4)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/grafanaSync in repository https://gitbox.apache.org/repos/asf/beam.git at 20d27727ba4 Filter out old actions runs from dashboard This branch includes the following new commits: new 20d27727ba4 Filter out old actions runs from dashboard The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Filter out old actions runs from dashboard
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/grafanaSync
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 20d27727ba46fd4b920e7519f03f0d44ce015f3f
Author: Danny McCormick
AuthorDate: Wed Aug 28 14:34:55 2024 +0100
Filter out old actions runs from dashboard
---
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
index e8fe60b4a7d..fd4a5e713fe 100644
--- a/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
+++ b/.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py
@@ -307,6 +307,8 @@ async def fetch_workflow_runs():
number_of_entries_per_page = 100 # The number of results per page (max
100)
params = {"branch": "master", "page": page, "per_page":
number_of_entries_per_page}
concurrent_requests = 30 # Number of requests to send simultaneously
+start = datetime.now() - timedelta(days=90)
+earliest_run_creation_date = start.strftime('%Y-%m-%d')
semaphore = asyncio.Semaphore(concurrent_requests)
print("Start fetching recent workflow runs")
@@ -338,6 +340,7 @@ async def fetch_workflow_runs():
"page": page,
"per_page": number_of_entries_per_page,
"exclude_pull_requests": "true",
+"created": f'>={earliest_run_creation_date}',
}
workflow_run_tasks.append(fetch(runs_url, semaphore, params,
headers))
page += 1
(beam) branch users/damccorm/pulsar deleted (was b6be53bb5b0)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/pulsar in repository https://gitbox.apache.org/repos/asf/beam.git was b6be53bb5b0 spotless The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Add quality warnings to pulsar (#32346)
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8cc80ff44c5 Add quality warnings to pulsar (#32346)
8cc80ff44c5 is described below
commit 8cc80ff44c56125d59e010b4299582ecef6b40ae
Author: Danny McCormick
AuthorDate: Wed Aug 28 14:10:37 2024 +0100
Add quality warnings to pulsar (#32346)
* Add quality warnings to pulsar
* spotless
---
.../main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java | 15 +++
.../org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java | 5 +
.../org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java | 5 +
.../java/org/apache/beam/sdk/io/pulsar/package-info.java | 6 +-
4 files changed, 30 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
index 6d0f0a08a01..aaff08a96d3 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
@@ -31,12 +31,22 @@ import org.apache.pulsar.client.api.PulsarClient;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.joda.time.Instant;
+/**
+ * Class for reading and writing from Apache Pulsar. Support is currently
incomplete, and there may
+ * be bugs; see https://github.com/apache/beam/issues/31078 for more info, and
comment in that issue
+ * if you run into issues with this IO.
+ */
@SuppressWarnings({"rawtypes", "nullness"})
public class PulsarIO {
/** Static class, prevent instantiation. */
private PulsarIO() {}
+ /**
+ * Read from Apache Pulsar. Support is currently incomplete, and there may
be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment in
that issue if you run
+ * into issues with this IO.
+ */
public static Read read() {
return new AutoValue_PulsarIO_Read.Builder()
.setPulsarClient(PulsarIOUtils.PULSAR_CLIENT_SERIALIZABLE_FUNCTION)
@@ -144,6 +154,11 @@ public class PulsarIO {
}
}
+ /**
+ * Write to Apache Pulsar. Support is currently incomplete, and there may be
bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment in
that issue if you run
+ * into issues with this IO.
+ */
public static Write write() {
return new AutoValue_PulsarIO_Write.Builder().build();
}
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
index 97a8dcd6e2b..a4e0a78e634 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
@@ -43,6 +43,11 @@ import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+/**
+ * Transform for reading from Apache Pulsar. Support is currently incomplete,
and there may be bugs;
+ * see https://github.com/apache/beam/issues/31078 for more info, and comment
in that issue if you
+ * run into issues with this IO.
+ */
@DoFn.UnboundedPerElement
@SuppressWarnings({"rawtypes", "nullness"})
public class ReadFromPulsarDoFn extends DoFn {
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
index 9659940e02b..375e8ce92a3 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
@@ -23,6 +23,11 @@ import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
+/**
+ * Transform for writing to Apache Pulsar. Support is currently incomplete,
and there may be bugs;
+ * see https://github.com/apache/beam/issues/31078 for more info, and comment
in that issue if you
+ * run into issues with this IO.
+ */
@DoFn.UnboundedPerElement
@SuppressWarnings({"rawtypes", "nullness"})
public class WriteToPulsarDoFn extends DoFn {
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
index 8b7ab101a72..ffa15257fe5 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
@@ -15,5 +15,9 @@
* See the
(beam) branch users/damccorm/pulsar updated (523f999d31f -> b6be53bb5b0)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/pulsar in repository https://gitbox.apache.org/repos/asf/beam.git from 523f999d31f Add quality warnings to pulsar add b6be53bb5b0 spotless No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/io/pulsar/PulsarIO.java| 30 +++--- .../beam/sdk/io/pulsar/ReadFromPulsarDoFn.java | 10 .../beam/sdk/io/pulsar/WriteToPulsarDoFn.java | 10 .../apache/beam/sdk/io/pulsar/package-info.java| 10 4 files changed, 30 insertions(+), 30 deletions(-)
(beam) branch users/damccorm/pulsar created (now 523f999d31f)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/pulsar in repository https://gitbox.apache.org/repos/asf/beam.git at 523f999d31f Add quality warnings to pulsar This branch includes the following new commits: new 523f999d31f Add quality warnings to pulsar The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Add quality warnings to pulsar
This is an automated email from the ASF dual-hosted git repository.
damccorm pushed a commit to branch users/damccorm/pulsar
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 523f999d31f240c6e9418b53a756e62b252ccfea
Author: Danny McCormick
AuthorDate: Wed Aug 28 13:50:00 2024 +0100
Add quality warnings to pulsar
---
.../main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java | 15 +++
.../org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java | 5 +
.../org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java | 5 +
.../java/org/apache/beam/sdk/io/pulsar/package-info.java | 6 +-
4 files changed, 30 insertions(+), 1 deletion(-)
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
index 6d0f0a08a01..e5cde0aec01 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/PulsarIO.java
@@ -31,12 +31,22 @@ import org.apache.pulsar.client.api.PulsarClient;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.joda.time.Instant;
+/** Class for reading and writing from Apache Pulsar. Support is
+ * currently incomplete, and there may be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment
+ * in that issue if you run into issues with this IO.
+*/
@SuppressWarnings({"rawtypes", "nullness"})
public class PulsarIO {
/** Static class, prevent instantiation. */
private PulsarIO() {}
+ /** Read from Apache Pulsar. Support is currently incomplete, and there may
+ * be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment
+ * in that issue if you run into issues with this IO.
+ */
public static Read read() {
return new AutoValue_PulsarIO_Read.Builder()
.setPulsarClient(PulsarIOUtils.PULSAR_CLIENT_SERIALIZABLE_FUNCTION)
@@ -144,6 +154,11 @@ public class PulsarIO {
}
}
+ /** Write to Apache Pulsar. Support is currently incomplete, and there may
+ * be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment
+ * in that issue if you run into issues with this IO.
+ */
public static Write write() {
return new AutoValue_PulsarIO_Write.Builder().build();
}
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
index 97a8dcd6e2b..14ca4b4d5b6 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/ReadFromPulsarDoFn.java
@@ -43,6 +43,11 @@ import org.joda.time.Instant;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+/** Transform for reading from Apache Pulsar. Support is
+ * currently incomplete, and there may be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment
+ * in that issue if you run into issues with this IO.
+*/
@DoFn.UnboundedPerElement
@SuppressWarnings({"rawtypes", "nullness"})
public class ReadFromPulsarDoFn extends DoFn {
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
index 9659940e02b..3199bf792bd 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/WriteToPulsarDoFn.java
@@ -23,6 +23,11 @@ import org.apache.pulsar.client.api.Producer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.PulsarClientException;
+/** Transform for writing to Apache Pulsar. Support is
+ * currently incomplete, and there may be bugs; see
+ * https://github.com/apache/beam/issues/31078 for more info, and comment
+ * in that issue if you run into issues with this IO.
+*/
@DoFn.UnboundedPerElement
@SuppressWarnings({"rawtypes", "nullness"})
public class WriteToPulsarDoFn extends DoFn {
diff --git
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
index 8b7ab101a72..27effe2e077 100644
---
a/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
+++
b/sdks/java/io/pulsar/src/main/java/org/apache/beam/sdk/io/pulsar/package-info.java
@@ -15,5 +15,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-/** Transforms for reading and writing from Apache Pulsar. */
+/** Transforms for reading and writing from Apache Pulsar. Support is
+
(beam) 01/01: Remove beam summit banner
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch users/damccorm/beamsummit in repository https://gitbox.apache.org/repos/asf/beam.git commit 45036736e0a2e4cf8562392be6bb0dcffe99bae1 Author: Danny McCormick AuthorDate: Tue Aug 27 18:00:15 2024 +0100 Remove beam summit banner --- website/www/site/layouts/partials/header.html | 6 -- 1 file changed, 6 deletions(-) diff --git a/website/www/site/layouts/partials/header.html b/website/www/site/layouts/partials/header.html index a01ce5de084..139ae9bc885 100644 --- a/website/www/site/layouts/partials/header.html +++ b/website/www/site/layouts/partials/header.html @@ -207,12 +207,6 @@ - - https://beamsummit.org/register/";> - - - - https://tour.beam.apache.org";>
(beam) branch users/damccorm/beamsummit created (now 45036736e0a)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/beamsummit in repository https://gitbox.apache.org/repos/asf/beam.git at 45036736e0a Remove beam summit banner This branch includes the following new commits: new 45036736e0a Remove beam summit banner The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
