(beam) branch master updated: sdks/python/apache_beam/io: add note on `output_fn` ctor arg to indicate that It is unused and only retained for backward compatibility (#32635)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d4dd58b2c4c sdks/python/apache_beam/io: add note on `output_fn` ctor arg to indicate that It is unused and only retained for backward compatibility (#32635) d4dd58b2c4c is described below commit d4dd58b2c4c4b5867ace4bdd34e1bcc32de963cc Author: Mohamed Awnallah <69568555+mohamedawnal...@users.noreply.github.com> AuthorDate: Sat Oct 5 01:36:44 2024 +0300 sdks/python/apache_beam/io: add note on `output_fn` ctor arg to indicate that It is unused and only retained for backward compatibility (#32635) In this commit, we add unused note on `output_fn` constructor argument in `WriteToFiles` class and only retained for backward compatibility. Signed-off-by: Mohamed Awnallah --- sdks/python/apache_beam/io/fileio.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sdks/python/apache_beam/io/fileio.py b/sdks/python/apache_beam/io/fileio.py index 378ecf71920..d9b2a204067 100644 --- a/sdks/python/apache_beam/io/fileio.py +++ b/sdks/python/apache_beam/io/fileio.py @@ -557,6 +557,8 @@ class WriteToFiles(beam.PTransform): class signature or an instance of FileSink to this parameter. If none is provided, a ``TextSink`` is used. shards (int): The number of shards per destination and trigger firing. + output_fn (callable, optional): A callable to process the output. This +parameter is currently unused and retained for backward compatibility. max_writers_per_bundle (int): The number of writers that can be open concurrently in a single worker that's processing one bundle. """
(beam) branch master updated: Fix counter metrics for ParDo#with_exception_handling(timeout). (#32571)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0a71499f3f5 Fix counter metrics for
ParDo#with_exception_handling(timeout). (#32571)
0a71499f3f5 is described below
commit 0a71499f3f5e20a8214e164f3f505c874cfdfbda
Author: claudevdm <33973061+claude...@users.noreply.github.com>
AuthorDate: Thu Oct 3 14:51:27 2024 -0400
Fix counter metrics for ParDo#with_exception_handling(timeout). (#32571)
Co-authored-by: Claude
---
sdks/python/apache_beam/transforms/core.py | 18 ---
.../apache_beam/transforms/ptransform_test.py | 26 ++
2 files changed, 41 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/transforms/core.py
b/sdks/python/apache_beam/transforms/core.py
index e7180bc093b..91ca4c8e33c 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -2611,11 +2611,23 @@ class _TimeoutDoFn(DoFn):
def process(self, *args, **kwargs):
if self._pool is None:
self._pool = concurrent.futures.ThreadPoolExecutor(10)
+
+# Import here to avoid circular dependency
+from apache_beam.runners.worker.statesampler import get_current_tracker,
set_current_tracker
+
+# State sampler/tracker is stored as a thread local variable, and is used
+# when incrementing counter metrics.
+dispatching_thread_state_sampler = get_current_tracker()
+
+def wrapped_process():
+ """Makes the dispatching thread local state sampler available to child
+ thread"""
+ set_current_tracker(dispatching_thread_state_sampler)
+ return list(self._fn.process(*args, **kwargs))
+
# Ensure we iterate over the entire output list in the given amount of
time.
try:
- return self._pool.submit(
- lambda: list(self._fn.process(*args, **kwargs))).result(
- self._timeout)
+ return self._pool.submit(wrapped_process).result(self._timeout)
except TimeoutError:
self._pool.shutdown(wait=False)
self._pool = None
diff --git a/sdks/python/apache_beam/transforms/ptransform_test.py
b/sdks/python/apache_beam/transforms/ptransform_test.py
index 2fdec14651f..d760ef74fb1 100644
--- a/sdks/python/apache_beam/transforms/ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/ptransform_test.py
@@ -2780,6 +2780,32 @@ class DeadLettersTest(unittest.TestCase):
('slow', 'TimeoutError()')]),
label='CheckBad')
+ def test_increment_counter(self):
+# Counters are not currently supported for
+# ParDo#with_exception_handling(use_subprocess=True).
+if (self.use_subprocess):
+ return
+
+class CounterDoFn(beam.DoFn):
+ def __init__(self):
+self.records_counter = Metrics.counter(self.__class__,
'recordsCounter')
+
+ def process(self, element):
+self.records_counter.inc()
+
+with TestPipeline() as p:
+ _, _ = (
+ (p | beam.Create([1,2,3])) | beam.ParDo(CounterDoFn())
+ .with_exception_handling(
+use_subprocess=self.use_subprocess, timeout=1))
+results = p.result
+metric_results = results.metrics().query(
+MetricsFilter().with_name("recordsCounter"))
+records_counter = metric_results['counters'][0]
+
+self.assertEqual(records_counter.key.metric.name, 'recordsCounter')
+self.assertEqual(records_counter.result, 3)
+
def test_lifecycle(self):
die = type(self).die
(beam) branch master updated: Revert "Deepcopy combine_fn in PrecombineFn and PostCombineFn." (#32634)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new fbeba899071 Revert "Deepcopy combine_fn in PrecombineFn and
PostCombineFn." (#32634)
fbeba899071 is described below
commit fbeba899071acadaa7eba6066a5eb6ed67a2d9f2
Author: claudevdm <33973061+claude...@users.noreply.github.com>
AuthorDate: Wed Oct 2 15:57:37 2024 -0400
Revert "Deepcopy combine_fn in PrecombineFn and PostCombineFn." (#32634)
This reverts commit eaf53e531b963fe9fc43fb3bd95809cc3c22fe66.
---
.../transforms/combinefn_lifecycle_test.py | 11 ++
sdks/python/apache_beam/transforms/core.py | 43 +-
2 files changed, 22 insertions(+), 32 deletions(-)
diff --git a/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
b/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
index 2a86f0251e7..62dbbc5fb77 100644
--- a/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
+++ b/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
@@ -53,18 +53,15 @@ class CombineFnLifecycleTest(unittest.TestCase):
@parameterized_class([
-{'runner': direct_runner.BundleBasedDirectRunner, 'pickler': 'dill'},
-{'runner': direct_runner.BundleBasedDirectRunner, 'pickler':
'cloudpickle'},
-{'runner': fn_api_runner.FnApiRunner, 'pickler': 'dill'},
-{'runner': fn_api_runner.FnApiRunner, 'pickler': 'cloudpickle'},
-]) # yapf: disable
+{'runner': direct_runner.BundleBasedDirectRunner},
+{'runner': fn_api_runner.FnApiRunner},
+]) # yapf: disable
class LocalCombineFnLifecycleTest(unittest.TestCase):
def tearDown(self):
CallSequenceEnforcingCombineFn.instances.clear()
def test_combine(self):
-test_options = PipelineOptions(flags=[f"--pickle_library={self.pickler}"])
-run_combine(TestPipeline(runner=self.runner(), options=test_options))
+run_combine(TestPipeline(runner=self.runner()))
self._assert_teardown_called()
def test_non_liftable_combine(self):
diff --git a/sdks/python/apache_beam/transforms/core.py
b/sdks/python/apache_beam/transforms/core.py
index 953b6cfe627..e7180bc093b 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -3158,40 +3158,33 @@ class _CombinePerKeyWithHotKeyFanout(PTransform):
yield pvalue.TaggedOutput('hot', ((self._nonce % fanout, key),
value))
class PreCombineFn(CombineFn):
- def __init__(self):
-# Deepcopy of the combine_fn to avoid sharing state between lifted
-# stages when using cloudpickle.
-self._combine_fn_copy = copy.deepcopy(combine_fn)
-self.setup = self._combine_fn_copy.setup
-self.create_accumulator = self._combine_fn_copy.create_accumulator
-self.add_input = self._combine_fn_copy.add_input
-self.merge_accumulators = self._combine_fn_copy.merge_accumulators
-self.compact = self._combine_fn_copy.compact
-self.teardown = self._combine_fn_copy.teardown
-
@staticmethod
def extract_output(accumulator):
# Boolean indicates this is an accumulator.
return (True, accumulator)
+ setup = combine_fn.setup
+ create_accumulator = combine_fn.create_accumulator
+ add_input = combine_fn.add_input
+ merge_accumulators = combine_fn.merge_accumulators
+ compact = combine_fn.compact
+ teardown = combine_fn.teardown
+
class PostCombineFn(CombineFn):
- def __init__(self):
-# Deepcopy of the combine_fn to avoid sharing state between lifted
-# stages when using cloudpickle.
-self._combine_fn_copy = copy.deepcopy(combine_fn)
-self.setup = self._combine_fn_copy.setup
-self.create_accumulator = self._combine_fn_copy.create_accumulator
-self.merge_accumulators = self._combine_fn_copy.merge_accumulators
-self.compact = self._combine_fn_copy.compact
-self.extract_output = self._combine_fn_copy.extract_output
-self.teardown = self._combine_fn_copy.teardown
-
- def add_input(self, accumulator, element):
+ @staticmethod
+ def add_input(accumulator, element):
is_accumulator, value = element
if is_accumulator:
- return self._combine_fn_copy.merge_accumulators([accumulator, value])
+ return combine_fn.merge_accumulators([accumulator, value])
else:
- return self._combine_fn_copy.add_input(accumulator, value)
+ return combine_fn.add_input(accumulator, value)
+
+ setup = combine_fn.setup
+ create_accumulator = combine_fn.create_accumulator
+ merge_accu
(beam) branch master updated: Deepcopy combine_fn in PrecombineFn and PostCombineFn. (#32598)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eaf53e531b9 Deepcopy combine_fn in PrecombineFn and PostCombineFn.
(#32598)
eaf53e531b9 is described below
commit eaf53e531b963fe9fc43fb3bd95809cc3c22fe66
Author: claudevdm <33973061+claude...@users.noreply.github.com>
AuthorDate: Tue Oct 1 15:50:06 2024 -0400
Deepcopy combine_fn in PrecombineFn and PostCombineFn. (#32598)
Co-authored-by: Claude
---
.../transforms/combinefn_lifecycle_test.py | 11 --
sdks/python/apache_beam/transforms/core.py | 43 +-
2 files changed, 32 insertions(+), 22 deletions(-)
diff --git a/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
b/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
index 62dbbc5fb77..2a86f0251e7 100644
--- a/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
+++ b/sdks/python/apache_beam/transforms/combinefn_lifecycle_test.py
@@ -53,15 +53,18 @@ class CombineFnLifecycleTest(unittest.TestCase):
@parameterized_class([
-{'runner': direct_runner.BundleBasedDirectRunner},
-{'runner': fn_api_runner.FnApiRunner},
-]) # yapf: disable
+{'runner': direct_runner.BundleBasedDirectRunner, 'pickler': 'dill'},
+{'runner': direct_runner.BundleBasedDirectRunner, 'pickler':
'cloudpickle'},
+{'runner': fn_api_runner.FnApiRunner, 'pickler': 'dill'},
+{'runner': fn_api_runner.FnApiRunner, 'pickler': 'cloudpickle'},
+]) # yapf: disable
class LocalCombineFnLifecycleTest(unittest.TestCase):
def tearDown(self):
CallSequenceEnforcingCombineFn.instances.clear()
def test_combine(self):
-run_combine(TestPipeline(runner=self.runner()))
+test_options = PipelineOptions(flags=[f"--pickle_library={self.pickler}"])
+run_combine(TestPipeline(runner=self.runner(), options=test_options))
self._assert_teardown_called()
def test_non_liftable_combine(self):
diff --git a/sdks/python/apache_beam/transforms/core.py
b/sdks/python/apache_beam/transforms/core.py
index e7180bc093b..953b6cfe627 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -3158,33 +3158,40 @@ class _CombinePerKeyWithHotKeyFanout(PTransform):
yield pvalue.TaggedOutput('hot', ((self._nonce % fanout, key),
value))
class PreCombineFn(CombineFn):
+ def __init__(self):
+# Deepcopy of the combine_fn to avoid sharing state between lifted
+# stages when using cloudpickle.
+self._combine_fn_copy = copy.deepcopy(combine_fn)
+self.setup = self._combine_fn_copy.setup
+self.create_accumulator = self._combine_fn_copy.create_accumulator
+self.add_input = self._combine_fn_copy.add_input
+self.merge_accumulators = self._combine_fn_copy.merge_accumulators
+self.compact = self._combine_fn_copy.compact
+self.teardown = self._combine_fn_copy.teardown
+
@staticmethod
def extract_output(accumulator):
# Boolean indicates this is an accumulator.
return (True, accumulator)
- setup = combine_fn.setup
- create_accumulator = combine_fn.create_accumulator
- add_input = combine_fn.add_input
- merge_accumulators = combine_fn.merge_accumulators
- compact = combine_fn.compact
- teardown = combine_fn.teardown
-
class PostCombineFn(CombineFn):
- @staticmethod
- def add_input(accumulator, element):
+ def __init__(self):
+# Deepcopy of the combine_fn to avoid sharing state between lifted
+# stages when using cloudpickle.
+self._combine_fn_copy = copy.deepcopy(combine_fn)
+self.setup = self._combine_fn_copy.setup
+self.create_accumulator = self._combine_fn_copy.create_accumulator
+self.merge_accumulators = self._combine_fn_copy.merge_accumulators
+self.compact = self._combine_fn_copy.compact
+self.extract_output = self._combine_fn_copy.extract_output
+self.teardown = self._combine_fn_copy.teardown
+
+ def add_input(self, accumulator, element):
is_accumulator, value = element
if is_accumulator:
- return combine_fn.merge_accumulators([accumulator, value])
+ return self._combine_fn_copy.merge_accumulators([accumulator, value])
else:
- return combine_fn.add_input(accumulator, value)
-
- setup = combine_fn.setup
- create_accumulator = combine_fn.create_accumulator
- merge_accumulators = combine_fn.merge_accumulators
- compact = combine_fn.compact
- extract_output = combine_fn.extract_output
- teardown = combine_fn.teardown
+ return self._combine_fn_copy.add_input(accumulator, value)
def StripNonce(nonce_key_value):
(_, key), value = nonce_key_value
(beam) branch master updated: Fix a bug in _get_function_body_without_inners for module sdks.python.transforms.core (#32591)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aadb695348d Fix a bug in _get_function_body_without_inners for module
sdks.python.transforms.core (#32591)
aadb695348d is described below
commit aadb695348de77f4791b7b0f947552ffc6bff02f
Author: DKPHUONG <82434977+dk...@users.noreply.github.com>
AuthorDate: Tue Oct 1 07:12:54 2024 +0700
Fix a bug in _get_function_body_without_inners for module
sdks.python.transforms.core (#32591)
---
sdks/python/apache_beam/transforms/core.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/transforms/core.py
b/sdks/python/apache_beam/transforms/core.py
index c6560ee4357..e7180bc093b 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -1461,7 +1461,7 @@ class CallableWrapperPartitionFn(PartitionFn):
def _get_function_body_without_inners(func):
source_lines = inspect.getsourcelines(func)[0]
- source_lines = dropwhile(lambda x: x.startswith("@"), source_lines)
+ source_lines = dropwhile(lambda x: x.strip().startswith("@"), source_lines)
first_def_line = next(source_lines).strip()
if first_def_line.startswith("def "):
last_def_line_without_comment = first_def_line.split("#")[0] \
(beam) branch tvalentyn-patch-2 created (now 40a5524d65d)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git at 40a5524d65d Update environments.py This branch includes the following new commits: new 40a5524d65d Update environments.py The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Update environments.py
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch tvalentyn-patch-2
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 40a5524d65dd1c9c1c0798e41440c92b870fd111
Author: tvalentyn
AuthorDate: Mon Sep 30 14:18:21 2024 -0700
Update environments.py
Reduce spammy logs
---
sdks/python/apache_beam/transforms/environments.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/transforms/environments.py
b/sdks/python/apache_beam/transforms/environments.py
index 43542dd99b6..dbb22780292 100644
--- a/sdks/python/apache_beam/transforms/environments.py
+++ b/sdks/python/apache_beam/transforms/environments.py
@@ -311,14 +311,14 @@ class DockerEnvironment(Environment):
):
super().__init__(capabilities, artifacts, resource_hints)
if container_image:
- logging.info(
+ logging.debug(
'Using provided Python SDK container image: %s' % (container_image))
self.container_image = container_image
else:
- logging.info('No image given, using default Python SDK image')
+ logging.debug('No image given, using default Python SDK image')
self.container_image = self.default_docker_image()
-logging.info(
+logging.debug(
'Python SDK container image set to "%s" for Docker environment' %
(self.container_image))
(beam) branch master updated (06e2170c1ab -> aeead3f7cd3)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 06e2170c1ab Update python SDK container (#32501) add aeead3f7cd3 Disable a permared suite. (#32493) No new revisions were added by this update. Summary of changes: .github/build.gradle | 4 ++-- .../workflows/beam_PostCommit_XVR_GoUsingJava_Dataflow.yml| 11 +++ 2 files changed, 9 insertions(+), 6 deletions(-)
(beam) branch tvalentyn-gha updated (bd438478b34 -> fe298682849)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git omit bd438478b34 downgrade grpc add bcf404238ab downgrade grpc (#32449) add 45036736e0a Remove beam summit banner add 98410caac19 Merge pull request #32331 from apache/users/damccorm/beamsummit add 9a524cc004e Add Lineage metrics to Python PubsubIO, BigtableIO, FileIO (#32430) add 83ae551cea1 Update Playground and other location usage of Go to use latest released version by default. (#32444) add ed8dc5df618 fix: use threadsafe data structure thats shared between threads (#32452) add 25196614070 fix: ensure that non ApiExceptions in handleMutationException are not swallowed (#32455) add a6f11c89667 Bump Python SDK container for Dataflow add 1c307088a86 Merge pull request #32453: Bump Python SDK container for Dataflow add fb5424e5888 Bump cloud.google.com/go/pubsub from 1.41.0 to 1.43.0 in /sdks (#32415) add 7a6121a3231 fix docker compose command (#32459) add e259f472add Don't override existing consumer config values KafkaIO (#32443) add 12974595cd0 Bump golang.org/x/text from 0.17.0 to 0.18.0 in /sdks (#32466) add 6514136c17e fix: ensure that all outstanding future have completed in FinishBundle (#32454) add bbd09d60665 Remove Python 3.8 GitHub Actions Runs (#32429) add 8d1c8d2cb9d Add the script to extract docs from the dev list add 56cb2c22e02 Add the script to extract docs from the dev list add dd511fb22a6 Merge remote-tracking branch 'refs/remotes/origin/discussion-docs' into discussion-docs add f2992149035 update license add a65bed2ea23 updated the md content add 6b3043ba100 run this for all years add b808e9e90c4 fixed some author name errors add c2b09e26ae6 standardized the urls add e00559140df Merge pull request #32460 from liferoad/discussion-docs add f8475c952c1 Fix macos tox.ini definition (#32472) add 9f8a4b2d177 [yaml] Add examples for Spanner IO in YAML (#32288) add d20c0b1bbb2 Add information on python metrics to the programming guide (#32464) add b949ac2ca43 Bump golang.org/x/sys from 0.24.0 to 0.25.0 in /sdks (#32479) add 1b2d21acf8b Fix Kafka with Redistribute and commits enabled (#32344) add b1fd221a6d8 Restore mypy ignore[call-arg] in yaml_io.py (#32475) add ad7d7bc973c Update Dataflow internal client test (#32471) add 77913c59a17 Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#32480) add 5ebed2a8039 Bump google.golang.org/api from 0.192.0 to 0.197.0 in /sdks (#32426) add ff35dcbe03f `@JsonIgnore` causes emulator endpoint to be missing in DoFn's (#32450) add c4e2e950531 Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#32397) add 85581a3c02f Update run_inference_huggingface.ipynb (#32488) add 3cc00799c76 Fix non-utf8 encoding problem in ReadFromCsv and WritetoCsv. add f81c475f02a Apply yapf add 8088ecfffb3 Set default file mode to read for _TruncatingFileHandle. add ec37750639b Flush the underlying file-like object. add 4b8cc5fe082 Merge branch 'master' into readfromcsv-encoding add fc4db69dea6 Merge pull request #32463 from shunping/readfromcsv-encoding add 8690ddd6bfa Update python SDK container (#32487) add fe298682849 Disable a permared suite. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (bd438478b34) \ N -- N -- N refs/heads/tvalentyn-gha (fe298682849) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../test-properties.json | 18 +- .../actions/setup-environment-action/action.yml| 2 +- .../workflows/beam_CloudML_Benchmarks_Dataflow.yml | 2 +- .../beam_Inference_Python_Benchmarks_Dataflow.yml | 10 +- .../beam_LoadTests_Python_CoGBK_Dataflow_Batch.yml | 8 +- ...m_LoadTests_Python_CoGBK_Dataflow_Streaming.yml | 8 +- ...eam_LoadTests_Python_Combine_Dataflow_Batch.yml | 6 +- ...LoadTests_Python_Combine_Dataflow_Streaming.yml | 6 +- .../beam_LoadTests_Python_Combine_Flink_Batch.yml | 6 +- ...am_LoadTests_Python_Combine_Flink_Streaming.yml | 4 +- ...LoadTests_Python_FnApiRunner_Microbenchmark.yml | 2 +- .../be
(beam) branch master updated: downgrade grpc (#32449)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new bcf404238ab downgrade grpc (#32449) bcf404238ab is described below commit bcf404238abe8aaba57a52a7f3dcd7f1e5b5ac39 Author: tvalentyn AuthorDate: Thu Sep 12 17:49:13 2024 -0700 downgrade grpc (#32449) --- sdks/python/container/py310/base_image_requirements.txt | 2 +- sdks/python/container/py311/base_image_requirements.txt | 2 +- sdks/python/container/py312/base_image_requirements.txt | 2 +- sdks/python/container/py38/base_image_requirements.txt | 2 +- sdks/python/container/py39/base_image_requirements.txt | 2 +- sdks/python/setup.py| 3 ++- 6 files changed, 7 insertions(+), 6 deletions(-) diff --git a/sdks/python/container/py310/base_image_requirements.txt b/sdks/python/container/py310/base_image_requirements.txt index c89f35e243f..ec7a6d123d6 100644 --- a/sdks/python/container/py310/base_image_requirements.txt +++ b/sdks/python/container/py310/base_image_requirements.txt @@ -78,7 +78,7 @@ googleapis-common-protos==1.65.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.66.1 +grpcio==1.65.5 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 diff --git a/sdks/python/container/py311/base_image_requirements.txt b/sdks/python/container/py311/base_image_requirements.txt index fda2e214618..39d014a699c 100644 --- a/sdks/python/container/py311/base_image_requirements.txt +++ b/sdks/python/container/py311/base_image_requirements.txt @@ -76,7 +76,7 @@ googleapis-common-protos==1.65.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.66.1 +grpcio==1.65.5 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 diff --git a/sdks/python/container/py312/base_image_requirements.txt b/sdks/python/container/py312/base_image_requirements.txt index c027191bc6a..4c445ba0d2a 100644 --- a/sdks/python/container/py312/base_image_requirements.txt +++ b/sdks/python/container/py312/base_image_requirements.txt @@ -75,7 +75,7 @@ googleapis-common-protos==1.65.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.66.1 +grpcio==1.65.5 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 diff --git a/sdks/python/container/py38/base_image_requirements.txt b/sdks/python/container/py38/base_image_requirements.txt index 1796e4e83bf..32d77b7d53f 100644 --- a/sdks/python/container/py38/base_image_requirements.txt +++ b/sdks/python/container/py38/base_image_requirements.txt @@ -79,7 +79,7 @@ googleapis-common-protos==1.65.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.66.1 +grpcio==1.65.5 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 diff --git a/sdks/python/container/py39/base_image_requirements.txt b/sdks/python/container/py39/base_image_requirements.txt index aa2b4f3bb38..08fec552c2f 100644 --- a/sdks/python/container/py39/base_image_requirements.txt +++ b/sdks/python/container/py39/base_image_requirements.txt @@ -78,7 +78,7 @@ googleapis-common-protos==1.65.0 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.66.1 +grpcio==1.65.5 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 diff --git a/sdks/python/setup.py b/sdks/python/setup.py index f9eb9cd8173..21669416890 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -353,7 +353,8 @@ if __name__ == '__main__': 'cloudpickle~=2.2.1', 'fastavro>=0.23.6,<2', 'fasteners>=0.3,<1.0', - 'grpcio>=1.33.1,<2,!=1.48.0,!=1.59.*,!=1.60.*,!=1.61.*,!=1.62.0,!=1.62.1', # pylint: disable=line-too-long + # TODO(https://github.com/grpc/grpc/issues/37710): Unpin grpc + 'grpcio>=1.33.1,<2,!=1.48.0,!=1.59.*,!=1.60.*,!=1.61.*,!=1.62.0,!=1.62.1,<1.66.0', # pylint: disable=line-too-long 'hdfs>=2.1.0,<3.0.0', 'httplib2>=0.8,<0.23.0', 'jsonschema>=4.0.0,<5.0.0',
(beam) branch tvalentyn-gha updated (31695368017 -> bd438478b34)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 31695368017 Try to reduce feedback loop
omit a852a2b0e96 reduce scope of xlang suite
omit b0896dac512 Disable permared go xlang tests.
add a806bfc0526 [Python] Add the "no_wait_until_finish" option without
waiting fo the job completion
add e8c6a8c3aec Merge pull request #32409 from liferoad/wait_until_finish
add e76b51feb3a Fix RowCoderGenerator to use the encodingPositions when
encoding and decoding the bit set representing null fields. (#32389)
add 21abfb23f36 Disable permared go xlang tests. (#32423)
add 9c31e45d595 2.59.0 Beam site updates. (#32308)
add ef1bb52d3d3 Fixes link typo in 2.59, and fixes archive links. (#32437)
add 17718a97eaf Temporarily fix pip check failure on xgboost and grpcio
(#32432)
add ed84acb2967 Bump spanner max bound back to <4 (#32411)
add 4ee26065d9b Accept runner and options in ib.collect. (#32434)
add 02af7d4a218 Unit Testing in Beam Blog Post (#32412)
add bd438478b34 downgrade grpc
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (31695368017)
\
N -- N -- N refs/heads/tvalentyn-gha (bd438478b34)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
...low_Java.json => beam_PreCommit_Python_ML.json} | 0
.github/workflows/python_tests.yml | 8 +-
CHANGES.md | 16 +-
examples/notebooks/blog/unittests_in_beam.ipynb| 258 +++--
sdks/go/test/build.gradle | 14 +-
sdks/go/test/integration/xlang/xlang_test.go | 2 +
sdks/go/test/run_validatesrunner_tests.sh | 9 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 8 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 131 +--
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 7 +-
.../org/apache/beam/sdk/coders/RowCoderTest.java | 134 ++-
.../python/apache_beam/options/pipeline_options.py | 8 +
sdks/python/apache_beam/pipeline.py| 8 +-
sdks/python/apache_beam/pipeline_test.py | 12 +-
.../display/pcoll_visualization_test.py| 2 +-
.../runners/interactive/interactive_beam.py| 12 +-
.../interactive/non_interactive_runner_test.py | 30 +++
.../runners/interactive/pipeline_fragment.py | 12 +-
.../runners/interactive/recording_manager.py | 17 +-
.../container/py310/base_image_requirements.txt| 2 +-
.../container/py311/base_image_requirements.txt| 2 +-
.../container/py312/base_image_requirements.txt| 2 +-
.../container/py38/base_image_requirements.txt | 2 +-
.../container/py39/base_image_requirements.txt | 2 +-
sdks/python/setup.py | 10 +-
sdks/python/tox.ini| 15 ++
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.59.0.md| 76 ++
.../site/content/en/blog/unit-testing-in-beam.md | 198
.../www/site/content/en/get-started/downloads.md | 25 +-
30 files changed, 830 insertions(+), 194 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_Examples_Dataflow_Java.json
=> beam_PreCommit_Python_ML.json} (100%)
create mode 100644 website/www/site/content/en/blog/beam-2.59.0.md
create mode 100644 website/www/site/content/en/blog/unit-testing-in-beam.md
(beam) branch tvalentyn-gha updated (b0896dac512 -> 31695368017)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from b0896dac512 Disable permared go xlang tests. add a852a2b0e96 reduce scope of xlang suite add 31695368017 Try to reduce feedback loop No new revisions were added by this update. Summary of changes: sdks/go/test/build.gradle| 14 +++--- sdks/go/test/integration/xlang/xlang_test.go | 2 -- sdks/go/test/run_validatesrunner_tests.sh| 9 + 3 files changed, 12 insertions(+), 13 deletions(-)
(beam) branch master updated: Disable permared go xlang tests. (#32423)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 21abfb23f36 Disable permared go xlang tests. (#32423)
21abfb23f36 is described below
commit 21abfb23f36278ea2d60b0be658f7a171febf5d6
Author: tvalentyn
AuthorDate: Wed Sep 11 10:29:45 2024 -0700
Disable permared go xlang tests. (#32423)
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 50f94da2590..7b27cb8e140 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -2824,7 +2824,8 @@ class BeamModulePlugin implements Plugin {
// CrossLanguageValidatesRunnerTask is setup under python sdk but also
runs tasks not involving
// python versions. set 'skipNonPythonTask' property to avoid duplicated
run of these tasks.
if (!(project.hasProperty('skipNonPythonTask') &&
project.skipNonPythonTask == 'true')) {
-mainTask.configure { dependsOn goTask }
+System.err.println 'GoUsingJava tests have been disabled:
https://github.com/apache/beam/issues/30517#issuecomment-2341881604.'
+// mainTask.configure { dependsOn goTask }
}
cleanupTask.configure { mustRunAfter goTask }
config.cleanupJobServer.configure { mustRunAfter goTask }
(beam) branch tvalentyn-gha updated (64e6194b948 -> b0896dac512)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 64e6194b948 Preserve existing linter comments. add 1e873f42e14 Use highmem runner for beam_PostRelease_NightlySnapshot.yml (#31749) add b68b29a3ff6 Basic yaml-defined provider. add d2df083a029 Refactor jinja templatiziation to common location. add 3212688e2e6 Merge pull request #31684 Basic yaml-defined provider. add d6139904db8 Use GCP libraries-bom version for all grpc (#31760) add f15ca986811 Use ByteBuffer instead of BytesString which is unsupported in Schema Coders (#31746) add dbe72830b11 Add a test for getting state with MultimapSideInput StateKey (#31757) add f45b5d88e1d Add SerializableSupplier to the core beam.sdk.util package (#31766) add 54b882d285e Replace LGPL dep in Go SDK with an MIT alternative (#31769) add c1ca5156e64 Polish DoFn.Setup add dea440f46d5 Merge pull request #31764 from liferoad/polish-dofn add a1a22835710 Fix CHANGES.md from #31769 which incorrectly added to a released version (#31770) add 0c89a0edb9c Fix playground snippets (#31778) add a5eee589697 Fix flaky StreamingDataflowWorkerTest which wasn't waiting for enough commits. (#31781) add c08afeae60d Enable MapState and SetState for dataflow streaming engine pipelines with legacy runner by building on top of MultimapState. (#31453) add d1df1d7ecc9 Bump cloud.google.com/go/storage from 1.41.0 to 1.43.0 in /sdks (#31772) add 02600f55d21 Set Snowflake escape char to backslash since it is the default used by CSVParser (fixes #24467) (#31779) add ac423af5699 Pass-through IcebergIO catalog properties (#31726) add 631d40d0e79 Stage PrismRunner implementation and dependencies (#31794) add 8a88f1583f0 Solace Read connector: adding implementations of SempClient and SempClientFactory (#31542) add 746f3c5557e Use go 1.22 for self-hosted GHAs (#31767) add de4645d4507 Add support for StringSet metric in Java SDK. (#31789) add 516bbc77ef3 Add support for WindowStrategy Pane and AllowedLatness features (#31806) add dc10b77ce00 Update Go Version to 1.22.5 (#31812) add 9721aca8f50 Fix PostCommit Java ValidatesRunner Samza job (#31773) add 1db2373debc correctly close the javadoc tag in JmsIO.Write (#31801) add 88aa25391ec Solace Read connector: integration tests with testcontainers (#31543) add 9cbdda1b4e5 add in redistribute option for Kafka Read add cf37997d1dd Merge pull request #31347: Add in redistribute option for Kafka Read add e15cd9e040e Don't cache when building release candidates (#31810) add ef143aed418 Add link to security model (#31811) add 78bab0dd15e Avoid length-prefix-bytes substitutions for Flink boundaries. add dda0fbf57be Merge pull request #31579 Avoid length-prefix-bytes substitutions for Flink boundaries. add fa9c618cdbd Allow pr-bot to re-assign reviewers when stopped (#31436) add a4558dfd8c8 Bump certifi from 2024.2.2 to 2024.7.4 in /sdks/python/container/py38 (#31790) add 566a3ca96c4 Publish and export Javadoc for Solace (#31809) add 81538672cfe Support class executes the Prism binary (#31795) add b12943380b5 Exclude StringSet tests from portable runners and Dataflow LegacyRunner (#31818) add f72f6ce0e81 Remove CsvIOParseResult (#31819) add 5579a16de7d Introduce support for emitting lineage in BiqQueryIOs add dded4f06d82 Be spotless add 024692647b4 A couple improvements to BQ source lineage. add c827bbac387 Update contains test. add c9adc8ee6c6 Merge pull request #31805 Introduce support for emitting lineage in BQ Source. add 018bcdf592c Add missing params to Python Bigtable MutationsBatcher (#31791) add a2260949431 Avoid publishing string set metrics on the Dataflow legacy runner. (#31825) add b9a44126622 Add changelog notes regarding Solace read connector (#31826) add 6c829db657b Bump google.golang.org/grpc from 1.64.0 to 1.64.1 in /sdks (#31817) add 4df89c704b0 Allow Firestore project to be configurable (#31808) add 080c80a9573 Moving to 2.59.0-SNAPSHOT on master branch. add a0ba8dea7d8 isort add 36961405769 Merge pull request #31755 Modernize type hints. add dd0912460c4 add doc warning against using icebergio directly (#31833) add 7c0cf39001a Merge pull request #31823 Add lineage information for BigQuery sinks. add 8d5c3b5ee2c Locate and download Prism binary (#31796) add 00bf1c6d036 Change orphan file log to warning (#31835) add 041ccdbe5d0 playground python image update openjdk to 17 (#31843) add 9ee961fc0c2 Use fileNameTemplate attribute for file prefix (#31844) add b34c014888b Dedup SerializableSupplier (#31829) add e646c28d2ac [CsvIO] Implemented CsvIOParseHelpers:parseCell (#31802) add 5
(beam) branch revert-32345-update-docs-py-dependencies deleted (was 2518c7032e7)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch revert-32345-update-docs-py-dependencies in repository https://gitbox.apache.org/repos/asf/beam.git was 2518c7032e7 Revert "docs: modernize py dependencies docs and example (#32345)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (cfe8feee7c5 -> 511f294db15)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from cfe8feee7c5 Improve BatchElements documentation (#32082)
add 511f294db15 Revert "docs: modernize py dependencies docs and example"
(#32382)
No new revisions were added by this update.
Summary of changes:
.../juliaset/{src => juliaset}/__init__.py | 0
.../juliaset/{src => }/juliaset/juliaset.py| 0
.../juliaset/{src => }/juliaset/juliaset_test.py | 2 +-
.../{src => }/juliaset/juliaset_test_it.py | 2 +-
.../examples/complete/juliaset/pyproject.toml | 33 -
.../examples/complete/juliaset/setup.py| 26 +++
.../complete/juliaset/src/juliaset/__init__.py | 16 ---
.../sdks/python-pipeline-dependencies.md | 54 +-
8 files changed, 42 insertions(+), 91 deletions(-)
rename sdks/python/apache_beam/examples/complete/juliaset/{src =>
juliaset}/__init__.py (100%)
rename sdks/python/apache_beam/examples/complete/juliaset/{src =>
}/juliaset/juliaset.py (100%)
rename sdks/python/apache_beam/examples/complete/juliaset/{src =>
}/juliaset/juliaset_test.py (97%)
rename sdks/python/apache_beam/examples/complete/juliaset/{src =>
}/juliaset/juliaset_test_it.py (96%)
delete mode 100644
sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
delete mode 100644
sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/__init__.py
(beam) 01/01: Revert "docs: modernize py dependencies docs and example (#32345)"
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch revert-32345-update-docs-py-dependencies
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 2518c7032e7bca52800b44a2965a9c23ad3416c2
Author: tvalentyn
AuthorDate: Fri Aug 30 10:44:10 2024 -0700
Revert "docs: modernize py dependencies docs and example (#32345)"
This reverts commit 28f2d47662a89b2e01d8f9046932201b8eba18e6.
---
.../juliaset/{src => juliaset}/__init__.py | 0
.../juliaset/{src => }/juliaset/juliaset.py| 0
.../juliaset/{src => }/juliaset/juliaset_test.py | 2 +-
.../{src => }/juliaset/juliaset_test_it.py | 2 +-
.../examples/complete/juliaset/pyproject.toml | 33 -
.../examples/complete/juliaset/setup.py| 26 +++
.../complete/juliaset/src/juliaset/__init__.py | 16 ---
.../sdks/python-pipeline-dependencies.md | 54 +-
8 files changed, 42 insertions(+), 91 deletions(-)
diff --git a/sdks/python/apache_beam/examples/complete/juliaset/src/__init__.py
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/__init__.py
similarity index 100%
rename from sdks/python/apache_beam/examples/complete/juliaset/src/__init__.py
rename to
sdks/python/apache_beam/examples/complete/juliaset/juliaset/__init__.py
diff --git
a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset.py
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
similarity index 100%
rename from
sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset.py
rename to
sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset.py
diff --git
a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test.py
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
similarity index 97%
rename from
sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test.py
rename to
sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
index b371c88d360..6416831f426 100644
---
a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test.py
+++
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test.py
@@ -27,7 +27,7 @@ import unittest
import pytest
-from apache_beam.examples.complete.juliaset.src.juliaset import juliaset
+from apache_beam.examples.complete.juliaset.juliaset import juliaset
from apache_beam.testing.util import open_shards
diff --git
a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test_it.py
similarity index 96%
rename from
sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py
rename to
sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test_it.py
index 0aac9f9cffb..a2a3262a1fb 100644
---
a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py
+++
b/sdks/python/apache_beam/examples/complete/juliaset/juliaset/juliaset_test_it.py
@@ -27,7 +27,7 @@ import uuid
import pytest
from hamcrest.core.core.allof import all_of
-from apache_beam.examples.complete.juliaset.src.juliaset import juliaset
+from apache_beam.examples.complete.juliaset.juliaset import juliaset
from apache_beam.io.filesystems import FileSystems
from apache_beam.runners.runner import PipelineState
from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
diff --git a/sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
b/sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
deleted file mode 100644
index 6c865974cbd..000
--- a/sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
+++ /dev/null
@@ -1,33 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-#http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[project]
-name = "juliaset"
-version = "0.0.1"
-description = "Julia set workflow package."
-
-# Configure the required packages and scripts to install.
-# Note that the Python Dataflow containers come with numpy
(beam) branch revert-32345-update-docs-py-dependencies created (now 2518c7032e7)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch revert-32345-update-docs-py-dependencies in repository https://gitbox.apache.org/repos/asf/beam.git at 2518c7032e7 Revert "docs: modernize py dependencies docs and example (#32345)" This branch includes the following new commits: new 2518c7032e7 Revert "docs: modernize py dependencies docs and example (#32345)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: Improve BatchElements documentation (#32082)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new cfe8feee7c5 Improve BatchElements documentation (#32082)
cfe8feee7c5 is described below
commit cfe8feee7c5e180bd2671d15330bc46e228ea384
Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com>
AuthorDate: Fri Aug 30 11:03:03 2024 -0400
Improve BatchElements documentation (#32082)
* Imporve BatchElements documentation
* Add link to new documentation
* Update sdks/python/apache_beam/transforms/util.py
Co-authored-by: Jonathan Sabbagh
<108473809+jbsabb...@users.noreply.github.com>
* linting
* Apply suggestions from code review
Co-authored-by: tvalentyn
* line-too-long
* Update sdks/python/apache_beam/transforms/util.py
-
Co-authored-by: Jonathan Sabbagh
<108473809+jbsabb...@users.noreply.github.com>
Co-authored-by: tvalentyn
---
sdks/python/apache_beam/transforms/util.py | 14 ++
1 file changed, 14 insertions(+)
diff --git a/sdks/python/apache_beam/transforms/util.py
b/sdks/python/apache_beam/transforms/util.py
index 750d98f0789..a27c7aca9e2 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -802,6 +802,20 @@ class BatchElements(PTransform):
corresponding to its contents. Each batch is emitted with a timestamp at
the end of their window.
+ When the max_batch_duration_secs arg is provided, a stateful implementation
+ of BatchElements is used to batch elements across bundles. This is most
+ impactful in streaming applications where many bundles only contain one
+ element. Larger max_batch_duration_secs values `might` reduce the throughput
+ of the transform, while smaller values might improve the throughput but
+ make it more likely that batches are smaller than the target batch size.
+
+ As a general recommendation, start with low values (e.g. 0.005 aka 5ms) and
+ increase as needed to get the desired tradeoff between target batch size
+ and latency or throughput.
+
+ For more information on tuning parameters to this transform, see
+ https://beam.apache.org/documentation/patterns/batch-elements
+
Args:
min_batch_size: (optional) the smallest size of a batch
max_batch_size: (optional) the largest size of a batch
(beam) 01/01: Update juliaset_test_it.py
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch tvalentyn-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git commit 72595741db8513647a5e0dad6e73d97a11da22a1 Author: tvalentyn AuthorDate: Fri Aug 30 07:56:58 2024 -0700 Update juliaset_test_it.py --- .../examples/complete/juliaset/src/juliaset/juliaset_test_it.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py b/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py index 0aac9f9cffb..de4f8e9408c 100644 --- a/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py +++ b/sdks/python/apache_beam/examples/complete/juliaset/src/juliaset/juliaset_test_it.py @@ -48,7 +48,7 @@ class JuliaSetTestIT(unittest.TestCase): 'coordinate_output': coordinate_output, 'grid_size': self.GRID_SIZE, 'setup_file': os.path.normpath( -os.path.join(os.path.dirname(__file__), '..', 'setup.py')), +os.path.join(os.path.dirname(__file__), '../..', 'setup.py')), 'on_success_matcher': all_of(PipelineStateMatcher(PipelineState.DONE)), } args = pipeline.get_full_options_as_args(**extra_args)
(beam) branch tvalentyn-patch-1 created (now 72595741db8)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git at 72595741db8 Update juliaset_test_it.py This branch includes the following new commits: new 72595741db8 Update juliaset_test_it.py The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: docs: modernize py dependencies docs and example (#32345)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 28f2d47662a docs: modernize py dependencies docs and example (#32345)
28f2d47662a is described below
commit 28f2d47662a89b2e01d8f9046932201b8eba18e6
Author: Nito Buendia
AuthorDate: Fri Aug 30 06:42:19 2024 +0800
docs: modernize py dependencies docs and example (#32345)
* feat: update Python multifile docs
A more common approach to packaging Python package is leveraging
pyproject.toml files and having a src directory (instead of a flat
directory). This change intends to update the documentation and examples
to match this way of packaging Python packages.
* fix: fix juliaset package path
* cleanup: move main file outside src
* docs: address feedback #32345
Add build-system to pyproject.toml.
Improve wording on documentation.
Add extra step when using custom images.
* fix: fix juliaset path
* nit: remove extra space
* lint: format setup.py
* nit: reorder entries in pyproject.toml
* update the description
-
Co-authored-by: tvalentyn
---
.../{juliaset/__init__.py => pyproject.toml} | 17 +++
.../examples/complete/juliaset/setup.py| 26 ---
.../juliaset/{juliaset => src}/__init__.py | 0
.../juliaset/{ => src}/juliaset/__init__.py| 0
.../juliaset/{ => src}/juliaset/juliaset.py| 0
.../juliaset/{ => src}/juliaset/juliaset_test.py | 2 +-
.../{ => src}/juliaset/juliaset_test_it.py | 2 +-
.../sdks/python-pipeline-dependencies.md | 54 +-
8 files changed, 59 insertions(+), 42 deletions(-)
diff --git
a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/__init__.py
b/sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
similarity index 63%
copy from
sdks/python/apache_beam/examples/complete/juliaset/juliaset/__init__.py
copy to sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
index cce3acad34a..6c865974cbd 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/juliaset/__init__.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/pyproject.toml
@@ -14,3 +14,20 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+
+[project]
+name = "juliaset"
+version = "0.0.1"
+description = "Julia set workflow package."
+
+# Configure the required packages and scripts to install.
+# Note that the Python Dataflow containers come with numpy already installed
+# so this dependency will not trigger anything to be installed unless a version
+# restriction is specified.
+dependencies = [
+ "numpy"
+]
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
diff --git a/sdks/python/apache_beam/examples/complete/juliaset/setup.py
b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
index c3a9fe04376..649a5be7db7 100644
--- a/sdks/python/apache_beam/examples/complete/juliaset/setup.py
+++ b/sdks/python/apache_beam/examples/complete/juliaset/setup.py
@@ -15,14 +15,16 @@
# limitations under the License.
#
-"""Setup.py module for the workflow's worker utilities.
+"""setup.py module for the pipeline package.
-All the workflow related code is gathered in a package that will be built as a
-source distribution, staged in the staging area for the workflow being run and
-then installed in the workers when they start running.
+In this example, the pipeline code is gathered in a package that can be built
+as source distribution and installed on the workers. The package is defined
+in the pyproject.toml file. You can use setup.py file for defining
+configuration that needs to be determined programatically, for example,
+custom commands to run when a package is installed.
-This behavior is triggered by specifying the --setup_file command line option
-when running the workflow for remote execution.
+You can install this package into the workers at runtime by using
+the --setup_file pipeline option.
"""
# pytype: skip-file
@@ -107,19 +109,7 @@ class CustomCommands(setuptools.Command):
self.RunCustomCommand(command)
-# Configure the required packages and scripts to install.
-# Note that the Python Dataflow containers come with numpy already installed
-# so this dependency will not trigger anything to be installed unless a version
-# restriction is specified.
-REQUIRED_PACKAGES = [
-'numpy',
-]
-
setuptools.setup(
-name='juliaset',
-version='0.0.1',
-description='Julia set workflow package.',
-
(beam) branch weekly_update_python_dependencies_1724546078 deleted (was b0d9e46dfaf)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch weekly_update_python_dependencies_1724546078 in repository https://gitbox.apache.org/repos/asf/beam.git was b0d9e46dfaf Update Python Dependencies The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Update Python Dependencies (#32310)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new d3f38bec406 Update Python Dependencies (#32310) d3f38bec406 is described below commit d3f38bec4066153b18ca58e6e03dedfc0b7f90ae Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Wed Aug 28 16:46:41 2024 -0700 Update Python Dependencies (#32310) Co-authored-by: damccorm --- .../container/py310/base_image_requirements.txt| 30 +-- .../container/py311/base_image_requirements.txt| 30 +-- .../container/py312/base_image_requirements.txt| 32 ++-- .../container/py38/base_image_requirements.txt | 34 +++--- .../container/py39/base_image_requirements.txt | 32 ++-- 5 files changed, 79 insertions(+), 79 deletions(-) diff --git a/sdks/python/container/py310/base_image_requirements.txt b/sdks/python/container/py310/base_image_requirements.txt index c2ba9315bc6..2c7a1b07c78 100644 --- a/sdks/python/container/py310/base_image_requirements.txt +++ b/sdks/python/container/py310/base_image_requirements.txt @@ -27,7 +27,7 @@ attrs==24.2.0 beautifulsoup4==4.12.3 bs4==0.0.2 build==1.2.1 -cachetools==5.4.0 +cachetools==5.5.0 certifi==2024.7.4 cffi==1.17.0 charset-normalizer==3.3.2 @@ -36,7 +36,7 @@ cloudpickle==2.2.1 cramjam==2.8.3 crcmod==1.7 cryptography==43.0.0 -Cython==3.0.10 +Cython==3.0.11 deprecation==2.1.0 dill==0.3.1.1 dnspython==2.6.1 @@ -50,17 +50,17 @@ fasteners==0.19 freezegun==1.5.1 future==1.0.0 google-api-core==2.19.1 -google-api-python-client==2.140.0 +google-api-python-client==2.142.0 google-apitools==0.5.31 -google-auth==2.33.0 +google-auth==2.34.0 google-auth-httplib2==0.2.0 -google-cloud-aiplatform==1.61.0 +google-cloud-aiplatform==1.63.0 google-cloud-bigquery==3.25.0 google-cloud-bigquery-storage==2.25.0 -google-cloud-bigtable==2.25.0 +google-cloud-bigtable==2.26.0 google-cloud-core==2.4.1 -google-cloud-datastore==2.20.0 -google-cloud-dlp==3.21.0 +google-cloud-datastore==2.20.1 +google-cloud-dlp==3.22.0 google-cloud-language==2.14.0 google-cloud-profiler==4.1.0 google-cloud-pubsub==2.23.0 @@ -77,13 +77,13 @@ googleapis-common-protos==1.63.2 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.65.4 +grpcio==1.66.0 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 httplib2==0.22.0 -hypothesis==6.110.1 -idna==3.7 +hypothesis==6.111.2 +idna==3.8 iniconfig==2.0.0 Jinja2==3.0.3 joblib==1.4.2 @@ -132,16 +132,16 @@ PyYAML==6.0.2 redis==5.0.8 referencing==0.35.1 regex==2024.7.24 -requests==2.32.0 +requests==2.32.3 requests-mock==1.12.1 rpds-py==0.20.0 rsa==4.9 scikit-learn==1.5.1 -scipy==1.14.0 -shapely==2.0.5 +scipy==1.14.1 +shapely==2.0.6 six==1.16.0 sortedcontainers==2.4.0 -soupsieve==2.5 +soupsieve==2.6 SQLAlchemy==2.0.32 sqlparse==0.5.1 tenacity==8.5.0 diff --git a/sdks/python/container/py311/base_image_requirements.txt b/sdks/python/container/py311/base_image_requirements.txt index 18704be6ec4..4d125bdfc92 100644 --- a/sdks/python/container/py311/base_image_requirements.txt +++ b/sdks/python/container/py311/base_image_requirements.txt @@ -26,7 +26,7 @@ attrs==24.2.0 beautifulsoup4==4.12.3 bs4==0.0.2 build==1.2.1 -cachetools==5.4.0 +cachetools==5.5.0 certifi==2024.7.4 cffi==1.17.0 charset-normalizer==3.3.2 @@ -35,7 +35,7 @@ cloudpickle==2.2.1 cramjam==2.8.3 crcmod==1.7 cryptography==43.0.0 -Cython==3.0.10 +Cython==3.0.11 deprecation==2.1.0 dill==0.3.1.1 dnspython==2.6.1 @@ -48,17 +48,17 @@ fasteners==0.19 freezegun==1.5.1 future==1.0.0 google-api-core==2.19.1 -google-api-python-client==2.140.0 +google-api-python-client==2.142.0 google-apitools==0.5.31 -google-auth==2.33.0 +google-auth==2.34.0 google-auth-httplib2==0.2.0 -google-cloud-aiplatform==1.61.0 +google-cloud-aiplatform==1.63.0 google-cloud-bigquery==3.25.0 google-cloud-bigquery-storage==2.25.0 -google-cloud-bigtable==2.25.0 +google-cloud-bigtable==2.26.0 google-cloud-core==2.4.1 -google-cloud-datastore==2.20.0 -google-cloud-dlp==3.21.0 +google-cloud-datastore==2.20.1 +google-cloud-dlp==3.22.0 google-cloud-language==2.14.0 google-cloud-profiler==4.1.0 google-cloud-pubsub==2.23.0 @@ -75,13 +75,13 @@ googleapis-common-protos==1.63.2 greenlet==3.0.3 grpc-google-iam-v1==0.13.1 grpc-interceptor==0.15.4 -grpcio==1.65.4 +grpcio==1.66.0 grpcio-status==1.62.3 guppy3==3.1.4.post1 hdfs==2.7.3 httplib2==0.22.0 -hypothesis==6.110.1 -idna==3.7 +hypothesis==6.111.2 +idna==3.8 iniconfig==2.0.0 Jinja2==3.0.3 joblib==1.4.2 @@ -130,16 +130,16 @@ PyYAML==6.0.2 redis==5.0.8 referencing==0.35.1 regex==2024.7.24 -requests==2.32.0 +requests==2.32.3 requests-mock==1.12.1 rpds-py==0.20.0 rsa==4.9 scikit-learn==1.5.1 -scipy==1.14.0 -shapely==2.0.5
(beam) branch master updated: Update nltk version to 3.9.1 (#32256)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 71e3eedcd65 Update nltk version to 3.9.1 (#32256) 71e3eedcd65 is described below commit 71e3eedcd650e4d69eecfad856324e380301ba53 Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com> AuthorDate: Tue Aug 20 18:16:00 2024 -0400 Update nltk version to 3.9.1 (#32256) --- sdks/python/container/py310/base_image_requirements.txt | 2 +- sdks/python/container/py311/base_image_requirements.txt | 2 +- sdks/python/container/py312/base_image_requirements.txt | 2 +- sdks/python/container/py38/base_image_requirements.txt | 2 +- sdks/python/container/py39/base_image_requirements.txt | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/sdks/python/container/py310/base_image_requirements.txt b/sdks/python/container/py310/base_image_requirements.txt index 31cf194fbbc..d28ce0ca830 100644 --- a/sdks/python/container/py310/base_image_requirements.txt +++ b/sdks/python/container/py310/base_image_requirements.txt @@ -94,7 +94,7 @@ jsonschema-specifications==2023.12.1 MarkupSafe==2.1.5 mmh3==4.1.0 mock==5.1.0 -nltk==3.8.1 +nltk==3.9.1 nose==1.3.7 numpy==1.26.4 oauth2client==4.1.3 diff --git a/sdks/python/container/py311/base_image_requirements.txt b/sdks/python/container/py311/base_image_requirements.txt index 44b8a1edc8d..c5d0fce872e 100644 --- a/sdks/python/container/py311/base_image_requirements.txt +++ b/sdks/python/container/py311/base_image_requirements.txt @@ -92,7 +92,7 @@ jsonschema-specifications==2023.12.1 MarkupSafe==2.1.5 mmh3==4.1.0 mock==5.1.0 -nltk==3.8.1 +nltk==3.9.1 nose==1.3.7 numpy==1.26.4 oauth2client==4.1.3 diff --git a/sdks/python/container/py312/base_image_requirements.txt b/sdks/python/container/py312/base_image_requirements.txt index 76553ad6ab6..9c068e3fdae 100644 --- a/sdks/python/container/py312/base_image_requirements.txt +++ b/sdks/python/container/py312/base_image_requirements.txt @@ -91,7 +91,7 @@ jsonschema-specifications==2023.12.1 MarkupSafe==2.1.5 mmh3==4.1.0 mock==5.1.0 -nltk==3.8.1 +nltk==3.9.1 nose==1.3.7 numpy==1.26.4 oauth2client==4.1.3 diff --git a/sdks/python/container/py38/base_image_requirements.txt b/sdks/python/container/py38/base_image_requirements.txt index 8e6d6f8d059..86bf428b63b 100644 --- a/sdks/python/container/py38/base_image_requirements.txt +++ b/sdks/python/container/py38/base_image_requirements.txt @@ -97,7 +97,7 @@ jsonschema-specifications==2023.12.1 MarkupSafe==2.1.5 mmh3==4.1.0 mock==5.1.0 -nltk==3.8.1 +nltk==3.9.1 nose==1.3.7 numpy==1.24.4 oauth2client==4.1.3 diff --git a/sdks/python/container/py39/base_image_requirements.txt b/sdks/python/container/py39/base_image_requirements.txt index 8e07b087a6c..252073368a5 100644 --- a/sdks/python/container/py39/base_image_requirements.txt +++ b/sdks/python/container/py39/base_image_requirements.txt @@ -95,7 +95,7 @@ jsonschema-specifications==2023.12.1 MarkupSafe==2.1.5 mmh3==4.1.0 mock==5.1.0 -nltk==3.8.1 +nltk==3.9.1 nose==1.3.7 numpy==1.26.4 oauth2client==4.1.3
svn commit: r70949 [2/2] - in /release/beam: 2.58.0/ 2.58.1/ 2.58.1/prism/ 2.58.1/prism/darwin/ 2.58.1/prism/darwin/amd64/ 2.58.1/prism/darwin/arm64/ 2.58.1/prism/linux/ 2.58.1/prism/linux/amd64/ 2.58
Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc == --- release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc (added) +++ release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.asc Fri Aug 16 19:32:47 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAma+q5gYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQikhwQAIJHpH8vsbusu6Smu7hd2pWg +DD5FgRs/RUoD0hcR5Z9mwwZuAMVDpasiqS7JcHFqeqcf0cQQnQzPKcwtnmD/5cKA +0cnfHhBjJVsH/XHTIIbTFgp/MIUnaA4VCBR0A1De6hF3XQKNdgEbcfp/m9dVVlNf +Q30GsCmeVLSmPkTHSohAZtaQdw9ai/UnpwkJsOjaXvFnnf/trxHzmJ1tbAE/Ymus +Izmh7iQHNcNeYlhzA0gV4CvBs2udQ2yV3rlo/c7vdZXdtjZUiYQNXkXp3mmT88+A +MMns15xDBskVNBi9Je/sBxDXY84a7rj1G3tZh5ZPTdJ1WTbl0IKimqNPsQp/nOnO +zw9EwMkqUAk4lpn0UIk1vDnxXP1xL7UENaIlW8khji9C9QpA84iagQ7Fy+RGyVB8 +pubowPAf0x39oXL1lrPYk12xkwZlAExOWM+kxlvhk0CDaTok3R6EFd8V4ptREOoA +5Pko5exJa47oIfH1PPCr8LGGTnXMTFXkHEHtZNVRpV5/AXmDNQu9vpfRal4pdxxN +9wkcftPunkwfHKqovPAnWO8Pyquai0Nt0Q9KaVKIi1MptIdUNnhXzLGHJAYCadaZ +9McLBkq+S4as8a91zPeUhGjHF2H4j/a2bv94/ejPl4q/uGqBGSF7F7jp10wnnsTR +x0TVacnfu0oLhgA6zm/h +=9/Dq +-END PGP SIGNATURE- Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 == --- release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 (added) +++ release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 Fri Aug 16 19:32:47 2024 @@ -0,0 +1 @@ +15b502177a6f070569d6cc5d4aa8c7d813dbb10b8a4f333ef3ecde2058d726cd625b9a1cd4d811616665d2760ac0712a39d2c3d4ecfaf834c17bcb14b22dce23 apache_beam-2.58.1-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl == Binary file - no diff available. Propchange: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc == --- release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc (added) +++ release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc Fri Aug 16 19:32:47 2024 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAma+q5gYHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQiYLMP/3ECH/OaDpucO4OlyOLaovcD +NOjFYH9bjDuyhqN/s9SWR9hjYAkdoHT9MrzLLJ1V1rgc+3dw//qUIND5tzooZD5j +uXaHBrHjtxIZwBos3hjaDc1Nhc5aFC61sJAV0X5+8mP1U/3bBYRB5jF6Nqb/JiRv +ebEjHuTJgokfETqMMbVTVW2U3M26AphGHHuAx2HSS0hCjDCOfClPR8dxAJFDEYE5 +oxb5XQQjwIwUNvHcWPpv82suQolSBprIA66PvRI3EIWe+XlMeWs7NJN+2XLzpMnN +WDIsrFjdJbwkLE/n/kDUUGXOirGfZXxVpgjSr6jZdpfzvuf4e3cFsAG+2XT0aVud +Nwiy5K/IVqTP4xYg1KStFaPzUYfj0YmxTTR/p1gfvsnsYmY4NUgt7efHCwh/CZnt +Obbw9mi/m0HA5sHGWOILod7hqTYgOfm6H0UHOUUlRf0K5Yr1A2yAnCnnMRlKU0G8 +IgUW9mPjiYtog933II1RRQZyKQUqzC4tuEW1uofk14X32sbfGSGU+ijUFRLffEyD +pjoj8Zt5rV8oP3oRvIiH8u2fx+PayUA8zOwUKytgvoZylIExzy5vn9Q7W6tNEzsf +asC0cNOc7x/bNZNDXCUnFvPW6IlOY/UenjFUAb5FhQ7Z50sfYWmhHbTqo6ZvjXlX +bOwP338crlbmnMp9249j +=FxBc +-END PGP SIGNATURE- Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 == --- release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 (added) +++ release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 Fri Aug 16 19:32:47 2024 @@ -0,0 +1 @@ +55d365a36ce261c0b5a1a062dc273e761522a531004f52b8144e3868d22063ae344044ca9c88e2ca44a019c998f228cd6339c9138764f8bfd13df9c8a01f510a apache_beam-2.58.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-win32.whl == Binary file - no diff available. Propchange: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-win32.whl -- svn:mime-type = application/octet-stream Added: release/beam/2.58.1/python/apache_beam-2.58.1-cp39-cp39-win32.whl.asc ==
svn commit: r70949 [1/2] - in /release/beam: 2.58.0/ 2.58.1/ 2.58.1/prism/ 2.58.1/prism/darwin/ 2.58.1/prism/darwin/amd64/ 2.58.1/prism/darwin/arm64/ 2.58.1/prism/linux/ 2.58.1/prism/linux/amd64/ 2.58
Author: tvalentyn Date: Fri Aug 16 19:32:47 2024 New Revision: 70949 Log: Adding artifacts for the 2.58.1 release and removing old artifacts Added: release/beam/2.58.1/ release/beam/2.58.1/apache-beam-2.58.1-source-release.zip (with props) release/beam/2.58.1/apache-beam-2.58.1-source-release.zip.asc release/beam/2.58.1/apache-beam-2.58.1-source-release.zip.sha512 release/beam/2.58.1/prism/ release/beam/2.58.1/prism/darwin/ release/beam/2.58.1/prism/darwin/amd64/ release/beam/2.58.1/prism/darwin/amd64/apache_beam-v2.58.1-prism-darwin-amd64.zip (with props) release/beam/2.58.1/prism/darwin/amd64/apache_beam-v2.58.1-prism-darwin-amd64.zip.asc release/beam/2.58.1/prism/darwin/amd64/apache_beam-v2.58.1-prism-darwin-amd64.zip.sha512 release/beam/2.58.1/prism/darwin/arm64/ release/beam/2.58.1/prism/darwin/arm64/apache_beam-v2.58.1-prism-darwin-arm64.zip (with props) release/beam/2.58.1/prism/darwin/arm64/apache_beam-v2.58.1-prism-darwin-arm64.zip.asc release/beam/2.58.1/prism/darwin/arm64/apache_beam-v2.58.1-prism-darwin-arm64.zip.sha512 release/beam/2.58.1/prism/linux/ release/beam/2.58.1/prism/linux/amd64/ release/beam/2.58.1/prism/linux/amd64/apache_beam-v2.58.1-prism-linux-amd64.zip (with props) release/beam/2.58.1/prism/linux/amd64/apache_beam-v2.58.1-prism-linux-amd64.zip.asc release/beam/2.58.1/prism/linux/amd64/apache_beam-v2.58.1-prism-linux-amd64.zip.sha512 release/beam/2.58.1/prism/linux/arm64/ release/beam/2.58.1/prism/linux/arm64/apache_beam-v2.58.1-prism-linux-arm64.zip (with props) release/beam/2.58.1/prism/linux/arm64/apache_beam-v2.58.1-prism-linux-arm64.zip.asc release/beam/2.58.1/prism/linux/arm64/apache_beam-v2.58.1-prism-linux-arm64.zip.sha512 release/beam/2.58.1/prism/windows/ release/beam/2.58.1/prism/windows/amd64/ release/beam/2.58.1/prism/windows/amd64/apache_beam-v2.58.1-prism-windows-amd64.zip (with props) release/beam/2.58.1/prism/windows/amd64/apache_beam-v2.58.1-prism-windows-amd64.zip.asc release/beam/2.58.1/prism/windows/amd64/apache_beam-v2.58.1-prism-windows-amd64.zip.sha512 release/beam/2.58.1/prism/windows/arm64/ release/beam/2.58.1/prism/windows/arm64/apache_beam-v2.58.1-prism-windows-arm64.zip (with props) release/beam/2.58.1/prism/windows/arm64/apache_beam-v2.58.1-prism-windows-arm64.zip.asc release/beam/2.58.1/prism/windows/arm64/apache_beam-v2.58.1-prism-windows-arm64.zip.sha512 release/beam/2.58.1/python/ release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-macosx_10_9_x86_64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-macosx_10_9_x86_64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-macosx_10_9_x86_64.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win32.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win32.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win32.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win_amd64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win_amd64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp310-cp310-win_amd64.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311-macosx_10_9_x86_64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311-macosx_10_9_x86_64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311-macosx_10_9_x86_64.whl.sha512 release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc release/beam/2.58.1/python/apache_beam-2.58.1-cp311-cp311
(beam) branch master updated: Bump torch from 1.13.1 to 2.2.0 in /sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train (#31983)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 99672af7fe1 Bump torch from 1.13.1 to 2.2.0 in /sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train (#31983) 99672af7fe1 is described below commit 99672af7fe12f72b562289d0c9449e4711b973c2 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Aug 6 17:38:40 2024 -0700 Bump torch from 1.13.1 to 2.2.0 in /sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train (#31983) Bumps [torch](https://github.com/pytorch/pytorch) from 1.13.1 to 2.2.0. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](https://github.com/pytorch/pytorch/compare/v1.13.1...v2.2.0) --- updated-dependencies: - dependency-name: torch dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .../examples/ml-orchestration/kfp/components/train/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/requirements.txt b/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/requirements.txt index 2e65f0fba24..ba1103dd1ef 100644 --- a/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/requirements.txt +++ b/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/requirements.txt @@ -13,6 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -torch==1.13.1 +torch==2.2.0 numpy==1.22.4 Pillow==10.2.0 \ No newline at end of file
(beam) branch dependabot/pip/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/torch-2.2.0 deleted (was 1aa20749b59)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch dependabot/pip/sdks/python/apache_beam/examples/ml-orchestration/kfp/components/train/torch-2.2.0 in repository https://gitbox.apache.org/repos/asf/beam.git was 1aa20749b59 Bump torch The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Exclude a not yet implemented pandas op from dataframe tests. (#32066)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 9b564ef925b Exclude a not yet implemented pandas op from dataframe
tests. (#32066)
9b564ef925b is described below
commit 9b564ef925b83ca040c46d54314c600f5e65940c
Author: tvalentyn
AuthorDate: Tue Aug 6 17:33:29 2024 -0700
Exclude a not yet implemented pandas op from dataframe tests. (#32066)
---
sdks/python/apache_beam/dataframe/pandas_doctests_test.py | 1 +
sdks/python/setup.py | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/dataframe/pandas_doctests_test.py
b/sdks/python/apache_beam/dataframe/pandas_doctests_test.py
index a4bd0d0a812..c7ea908a933 100644
--- a/sdks/python/apache_beam/dataframe/pandas_doctests_test.py
+++ b/sdks/python/apache_beam/dataframe/pandas_doctests_test.py
@@ -557,6 +557,7 @@ class DoctestTest(unittest.TestCase):
'pandas.core.series.Series': ['ser.iloc[0] = 999'],
},
not_implemented_ok={
+'pandas.core.series.Series.case_when': ['*'],
'pandas.core.series.Series.transform': [
# str arg not supported. Tested with np.sum in
# frames_test.py::DeferredFrameTest::test_groupby_transform_sum
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index c9b2d087d04..756c952b010 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -155,7 +155,7 @@ else:
# Exclude 1.5.0 and 1.5.1 because of
# https://github.com/pandas-dev/pandas/issues/45725
dataframe_dependency = [
-'pandas>=1.4.3,!=1.5.0,!=1.5.1,<2.2;python_version>="3.8"',
+'pandas>=1.4.3,!=1.5.0,!=1.5.1,<2.3;python_version>="3.8"',
]
(beam) branch master updated (b61ef7591fe -> 2824944530f)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b61ef7591fe update document in AwsOptions (#32036) add 2824944530f Remove `--impersonate_service_account` whenever PipelineOptions are serialized (#32031) No new revisions were added by this update. Summary of changes: CHANGES.md | 3 ++- sdks/python/apache_beam/options/pipeline_options.py | 14 ++ website/www/site/content/en/blog/beam-2.49.0.md | 1 + 3 files changed, 17 insertions(+), 1 deletion(-)
(beam) branch master updated: Requirements_cache shouldn't create a cache folder when skipped. (#31961)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new b9a0c2b72ac Requirements_cache shouldn't create a cache folder when skipped. (#31961) b9a0c2b72ac is described below commit b9a0c2b72acbd6d86ab1f64c8bed8f46c731e389 Author: Eddie Phillips <85954305+eddie...@users.noreply.github.com> AuthorDate: Thu Jul 25 09:45:53 2024 -0700 Requirements_cache shouldn't create a cache folder when skipped. (#31961) * If the requirements cache is skipped then we shouldn't generate a requirements_cache folder. * pre-commit - Co-authored-by: eddiephillips --- sdks/python/apache_beam/runners/portability/stager.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/portability/stager.py b/sdks/python/apache_beam/runners/portability/stager.py index 48dabe18aa3..98c0e3176f7 100644 --- a/sdks/python/apache_beam/runners/portability/stager.py +++ b/sdks/python/apache_beam/runners/portability/stager.py @@ -214,7 +214,8 @@ class Stager(object): os.path.join(tempfile.gettempdir(), 'dataflow-requirements-cache') if (setup_options.requirements_cache is None) else setup_options.requirements_cache) - if not os.path.exists(requirements_cache_path): + if (setup_options.requirements_cache != SKIP_REQUIREMENTS_CACHE and + not os.path.exists(requirements_cache_path)): os.makedirs(requirements_cache_path) # Stage a requirements file if present.
(beam) branch dependabot/npm_and_yarn/scripts/ci/pr-bot/braces-3.0.3 deleted (was 90535bf8231)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch dependabot/npm_and_yarn/scripts/ci/pr-bot/braces-3.0.3 in repository https://gitbox.apache.org/repos/asf/beam.git was 90535bf8231 Bump braces from 3.0.2 to 3.0.3 in /scripts/ci/pr-bot The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump braces from 3.0.2 to 3.0.3 in /scripts/ci/pr-bot (#31886)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3e7614a50e5 Bump braces from 3.0.2 to 3.0.3 in /scripts/ci/pr-bot
(#31886)
3e7614a50e5 is described below
commit 3e7614a50e56cce87dc7b24e60829b1770d92ed7
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Mon Jul 15 14:32:25 2024 -0700
Bump braces from 3.0.2 to 3.0.3 in /scripts/ci/pr-bot (#31886)
Bumps [braces](https://github.com/micromatch/braces) from 3.0.2 to 3.0.3.
- [Changelog](https://github.com/micromatch/braces/blob/master/CHANGELOG.md)
- [Commits](https://github.com/micromatch/braces/compare/3.0.2...3.0.3)
---
updated-dependencies:
- dependency-name: braces
dependency-type: indirect
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
scripts/ci/pr-bot/package-lock.json | 28 ++--
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/scripts/ci/pr-bot/package-lock.json
b/scripts/ci/pr-bot/package-lock.json
index 336a8d45677..7cb764a4379 100644
--- a/scripts/ci/pr-bot/package-lock.json
+++ b/scripts/ci/pr-bot/package-lock.json
@@ -273,12 +273,12 @@
}
},
"node_modules/braces": {
- "version": "3.0.2",
- "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz";,
- "integrity":
"sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==",
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz";,
+ "integrity":
"sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==",
"dev": true,
"dependencies": {
-"fill-range": "^7.0.1"
+"fill-range": "^7.1.1"
},
"engines": {
"node": ">=8"
@@ -469,9 +469,9 @@
}
},
"node_modules/fill-range": {
- "version": "7.0.1",
- "resolved":
"https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz";,
- "integrity":
"sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==",
+ "version": "7.1.1",
+ "resolved":
"https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz";,
+ "integrity":
"sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==",
"dev": true,
"dependencies": {
"to-regex-range": "^5.0.1"
@@ -1421,12 +1421,12 @@
}
},
"braces": {
- "version": "3.0.2",
- "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz";,
- "integrity":
"sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==",
+ "version": "3.0.3",
+ "resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz";,
+ "integrity":
"sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==",
"dev": true,
"requires": {
-"fill-range": "^7.0.1"
+"fill-range": "^7.1.1"
}
},
"browser-stdout": {
@@ -1563,9 +1563,9 @@
"dev": true
},
"fill-range": {
- "version": "7.0.1",
- "resolved":
"https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz";,
- "integrity":
"sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==",
+ "version": "7.1.1",
+ "resolved":
"https://registry.npmjs.org/fill-range/-/fill-range-7.1.1.tgz";,
+ "integrity":
"sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==",
"dev": true,
"requires": {
"to-regex-range": "^5.0.1"
(beam) branch tvalentyn-gha updated (919bbea3bbd -> 64e6194b948)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git omit 919bbea3bbd pin the lower bound for typing-extensions add 37b8c8a87b8 Add Iceberg workflows (#31401) add 1f714760219 Add option to disable validation of cloud bigtable change stream IO add 10929b9ee7d Fix ordering of the validations add 13a3e46f87f Add unit test for ReadChangeStream config and validation add aa831b47330 Add additionl comment clarifying the effect of withoutValidation add 944273fe341 Merge pull request #31376 from tonytanger/validate_cbt_change_stream add bb4c1e6ded5 Include Python 3.12 in release validation steps. (#31415) add fd4368f1c4a Add Backlog Metrics to Kafka Splittable DoFn Implementation (#31281) add e488f41b9bd Memoize some dataframes analysis operations. (#31377) add 305e75359ad Better mixed Java/Python error messages for external transforms. (#31284) add 93cc6a521ee add PR trigger files (#31424) add 791ead6a05b add pull_request_target event (#31426) add 28443f803c8 Remove bad dialect option. (#31429) add df067dba5dc pin the lower bound for typing-extensions (#31425) add f7519774e3c BigQueryIO read throttling detection python (#31404) add 7d281558dd8 [#29697] Add prism artifact building workflow. (#31369) add 4daedbf5a8a [#30083] Add synthetic processing time to prism. (#30492) add 49a4290426d Add options to specify read and write http timeout for gcs as well as lower batching limit for rewrite operations which are copying. (#31410) add b1a6eb06051 [YAML] Fix simple YAML mappings type hinting (#31427) add 6842136e0c9 Add SDK capability to detect if the SDK Fn Harness data channel is busy. add ad841c6004f Regenerate Go protos. add 8b33e1f65c3 Merge pull request #31442 SDK protocol to detect if the SDK Fn Harness data channel is busy add df8bead5945 Refactor RowMutationInformation to use string type (#31323) add 06e103d87e8 Add ApplyBucketsWithInterpolation TFTransform (#31291) add 8d77c8fad07 Add try-excepts around data sampler encoding (#31396) add 0b5ffd7d153 Add SDK capability to detect if the SDK Fn Harness data channel is busy or not (#31420) add 19630e576fe Add in-memory variants of side inputs. (#31232) add 80d85aa38ff Add docs for YAML AssertThat. (#31448) add 90f020921c1 Update bigquery_tools.py (#31444) add 16d62827551 Update bigquery.py documentation (#31443) add 74f014ee1b9 SpannerIO: support max commit delay add 4a0849b15ee Merge pull request #31008: SpannerIO: support max commit delay add 9f3f1c93a60 Don't re-encode byte[] values in SortValues transform (#31025) add 3cadc83d348 flink portable client configurations (#31188) add c22678b4efb Emit a warning when large elements are detected. (#31363) add c90fd8320a5 Use bytes instead of mibs. (#31457) add 5454489e29b Report prism changes. (#31433) add b50ad0fe8fc [ManagedIO] pass underlying transform URN as an annotation (#31398) add 6911a1af720 Updating Commons Compress to 1.26.2 add f410b71c076 Merge pull request #31464: Updating Commons Compress to 1.26.2 add daf81143388 Merge pull request #31449 Pass through docs (and configs) for SqlProviders. add 19d57d01266 Change configurations for stress tests. Fix BigQueryIOST (#31439) add 7b6f9415c10 Updates Expansion Service Container to support upgrading using the schema-transform ID (#31451) add f93a67a6ae8 remove processing/scheduling logic from StreamingDataflowWorker (#31317) add 51525d34cfa Set retry policy to DEFAULT_RETRY when initializing BlobWriter. (#31468) add fa4ead99a6b Bump Jackson databind to 2.15 (#31473) add 349d7b9f9c6 add support for ConnectionFactory ProviderFn in JmsIO (#31264) add fc003513805 Update JdbcIO.java documentation (#31454) add 85650449827 Remove excess space from Programming Guide (#31469) add bb4ad994304 Fix one small typo in build script. (#31413) add fbe9427b0a9 Add Trigger_Never to Prepare features check (#31472) add 44550eea721 Add test of Select type inference. add 5914f272e01 yapf add 64d9794f444 Merge pull request #31428 Add test of Select type inference. add f54fda9df73 Add a test for sql dialect. add 59b0fdc9a3c Merge branch 'master' into yaml-sql-dialect add 6dd32a0aa65 Merge pull request #31276 Add a test for sql dialect. add 9cce412e302 Ensure that BigtableIO closes the stream after reading (#31478) add 4ad7037b55f Parse YAML ExpansionService configs directly using SnakeYAML (#31406) add b680fd58ffe Fix mangled license headers. add f98b6617b1b Merge pull request #31480 Fix mangled license headers. add bd17b74f442 Remove setup gcp credential step for beam_ workflows running on self-hosted runner (#
(beam) branch master updated (a1a22835710 -> 0c89a0edb9c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a1a22835710 Fix CHANGES.md from #31769 which incorrectly added to a released version (#31770) add 0c89a0edb9c Fix playground snippets (#31778) No new revisions were added by this update. Summary of changes: .../content/en/documentation/transforms/python/elementwise/flatmap.md | 2 +- .../content/en/documentation/transforms/python/elementwise/map.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
(beam) branch master updated: Fix CHANGES.md from #31769 which incorrectly added to a released version (#31770)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a1a22835710 Fix CHANGES.md from #31769 which incorrectly added to a released version (#31770) a1a22835710 is described below commit a1a22835710e23d85c423c304e602fd56b4bbd35 Author: Oliver Cardoza AuthorDate: Wed Jul 3 23:48:08 2024 -0400 Fix CHANGES.md from #31769 which incorrectly added to a released version (#31770) --- CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index d516f1244e8..38fa6e44b73 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -67,6 +67,7 @@ ## New Features / Improvements * Multiple RunInference instances can now share the same model instance by setting the model_identifier parameter (Python) ([#31665](https://github.com/apache/beam/issues/31665)). +* Removed a 3rd party LGPL dependency from the Go SDK ([#31765](https://github.com/apache/beam/issues/31765)). ## Breaking Changes @@ -117,7 +118,6 @@ * ProcessingTime is now handled synthetically with TestStream pipelines and Non-TestStream pipelines, for fast test pipeline execution by default. ([#30083](https://github.com/apache/beam/issues/30083)). * Prism does NOT yet support "real time" execution for this release. * Improve processing for large elements to reduce the chances for exceeding 2GB protobuf limits (Python)([https://github.com/apache/beam/issues/31607]). -* Removed a 3rd party LGPL dependency from the Go SDK ([#31765](https://github.com/apache/beam/issues/31765)). ## Breaking Changes
(beam) branch dependabot/pip/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/scikit-learn-1.5.0 deleted (was c66f3dbfdd0)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch dependabot/pip/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/scikit-learn-1.5.0 in repository https://gitbox.apache.org/repos/asf/beam.git was c66f3dbfdd0 Bump scikit-learn The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated: Bump scikit-learn from 1.0.2 to 1.5.0 in /sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline (#31629)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new aabf7d0fb46 Bump scikit-learn from 1.0.2 to 1.5.0 in /sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline (#31629) aabf7d0fb46 is described below commit aabf7d0fb46bb8b7ffb3b2f61db3cac75a6f972a Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Jun 18 14:29:58 2024 -0700 Bump scikit-learn from 1.0.2 to 1.5.0 in /sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline (#31629) Bumps [scikit-learn](https://github.com/scikit-learn/scikit-learn) from 1.0.2 to 1.5.0. - [Release notes](https://github.com/scikit-learn/scikit-learn/releases) - [Commits](https://github.com/scikit-learn/scikit-learn/compare/1.0.2...1.5.0) --- updated-dependencies: - dependency-name: scikit-learn dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .../inference/anomaly_detection/anomaly_detection_pipeline/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/setup.py b/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/setup.py index 72a65ab7281..365b6634d1a 100644 --- a/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/setup.py +++ b/sdks/python/apache_beam/examples/inference/anomaly_detection/anomaly_detection_pipeline/setup.py @@ -31,7 +31,7 @@ from setuptools import find_packages REQUIREMENTS = [ "apache-beam[gcp]==2.41.0", "hdbscan==0.8.28", -"scikit-learn==1.0.2", +"scikit-learn==1.5.0", "transformers==4.36.0", "torch==1.13.1", "pandas==1.3.5",
(beam) branch master updated: Update changes.md with a recent improvement. (#31609)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 0d1cd695010 Update changes.md with a recent improvement. (#31609) 0d1cd695010 is described below commit 0d1cd69501034e038977760fa0675525f5dc5634 Author: tvalentyn AuthorDate: Fri Jun 14 19:48:09 2024 -0700 Update changes.md with a recent improvement. (#31609) --- CHANGES.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGES.md b/CHANGES.md index f970a408788..0d4de2e3d58 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -116,6 +116,7 @@ * Pre-built Prism binaries are now part of the release and are available via the Github release page. ([#29697](https://github.com/apache/beam/issues/29697)). * ProcessingTime is now handled synthetically with TestStream pipelines and Non-TestStream pipelines, for fast test pipeline execution by default. ([#30083](https://github.com/apache/beam/issues/30083)). * Prism does NOT yet support "real time" execution for this release. +* Improve processing for large elements to reduce the chances for exceeding 2GB protobuf limits (Python)([https://github.com/apache/beam/issues/31607]). ## Breaking Changes
(beam) branch master updated: Add try-except block to guard against NotFound exception from GCS delete (#31603)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 481c71ac6a6 Add try-except block to guard against NotFound exception
from GCS delete (#31603)
481c71ac6a6 is described below
commit 481c71ac6a651a055082d37ecd502bbf1fa6ba16
Author: Shunping Huang
AuthorDate: Fri Jun 14 20:54:03 2024 -0400
Add try-except block to guard against NotFound exception from GCS delete
(#31603)
* Add try-except block to guard against NotFound exception from GCS delete.
* Skip test if the package cannot be imported.
---
sdks/python/apache_beam/io/gcp/gcsio_integration_test.py | 13 -
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
index ed4dd7e401e..fad63813680 100644
--- a/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
+++ b/sdks/python/apache_beam/io/gcp/gcsio_integration_test.py
@@ -44,6 +44,11 @@ try:
except ImportError:
gcsio = None # type: ignore
+try:
+ from google.api_core.exceptions import NotFound
+except ImportError:
+ NotFound = None
+
@unittest.skipIf(gcsio is None, 'GCP dependencies are not installed')
class GcsIOIntegrationTest(unittest.TestCase):
@@ -145,6 +150,7 @@ class GcsIOIntegrationTest(unittest.TestCase):
@pytest.mark.it_postcommit
@mock.patch('apache_beam.io.gcp.gcsio.default_gcs_bucket_name')
+ @unittest.skipIf(NotFound is None, 'GCP dependencies are not installed')
def test_create_default_bucket(self, mock_default_gcs_bucket_name):
google_cloud_options = self.test_pipeline.options.view_as(
GoogleCloudOptions)
@@ -168,7 +174,12 @@ class GcsIOIntegrationTest(unittest.TestCase):
# remove the existing bucket with the same name as the default bucket
existing_bucket = self.gcsio.get_bucket(overridden_bucket_name)
if existing_bucket:
- existing_bucket.delete()
+ try:
+existing_bucket.delete()
+ except NotFound:
+# Bucket existence check from get_bucket may be inaccurate due to gcs
+# cache or delay
+pass
bucket = gcsio.get_or_create_default_gcs_bucket(google_cloud_options)
self.assertIsNotNone(bucket)
(beam) branch master updated (30cce4450e3 -> 635372f67c0)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 30cce4450e3 Clone dataflow containers (#31591) add 635372f67c0 Limit the size of bundles of elements emitted by SDK into the data output stream. (#31581) No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json| 4 ++-- sdks/python/apache_beam/runners/worker/data_plane.py | 12 2 files changed, 10 insertions(+), 6 deletions(-)
(beam) branch master updated: Add try-except for process.send_signal so it can work on Windows. (#31574)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 0bf43078130 Add try-except for process.send_signal so it can work on
Windows. (#31574)
0bf43078130 is described below
commit 0bf43078130d7a258a0f1638a921d6d5287ca01e
Author: James Roseman
AuthorDate: Wed Jun 12 23:38:57 2024 +0100
Add try-except for process.send_signal so it can work on Windows. (#31574)
The current subprocess_server uses process.send_signal to terminate the
process. On Windows, this will raise a ValueError, as .terminate() should be
used instead.
---
sdks/python/apache_beam/utils/subprocess_server.py | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/utils/subprocess_server.py
b/sdks/python/apache_beam/utils/subprocess_server.py
index 7d9cf50d853..b8668c2b61b 100644
--- a/sdks/python/apache_beam/utils/subprocess_server.py
+++ b/sdks/python/apache_beam/utils/subprocess_server.py
@@ -243,7 +243,11 @@ class SubprocessServer(object):
if process.poll() is not None:
break
logging.debug("Sending SIGINT to process")
- process.send_signal(signal.SIGINT)
+ try:
+process.send_signal(signal.SIGINT)
+ except ValueError:
+# process.send_signal raises a ValueError on Windows.
+process.terminate()
time.sleep(1)
if process.poll() is None:
process.kill()
(beam) branch master updated (e26eb125f9f -> 2ddfcfb1c18)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from e26eb125f9f Merge pull request #31412 Remove SpannerIO pipeline options override add 2ddfcfb1c18 Correct test name to HashStringsTest (#31497) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/ml/transforms/tft_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated: Use bytes instead of mibs. (#31457)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c90fd8320a5 Use bytes instead of mibs. (#31457)
c90fd8320a5 is described below
commit c90fd8320a571573e9375318818bdb1a7c15d421
Author: tvalentyn
AuthorDate: Thu May 30 11:21:53 2024 -0700
Use bytes instead of mibs. (#31457)
* Use bytes.
* Update sdks/python/apache_beam/runners/worker/data_plane.py
---
sdks/python/apache_beam/runners/worker/data_plane.py | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/sdks/python/apache_beam/runners/worker/data_plane.py
b/sdks/python/apache_beam/runners/worker/data_plane.py
index ad4790a59e1..3dd6bdbe9ae 100644
--- a/sdks/python/apache_beam/runners/worker/data_plane.py
+++ b/sdks/python/apache_beam/runners/worker/data_plane.py
@@ -135,12 +135,12 @@ class
SizeBasedBufferingClosableOutputStream(ClosableOutputStream):
close_callback=None, # type: Optional[Callable[[bytes], None]]
flush_callback=None, # type: Optional[Callable[[bytes], None]]
size_flush_threshold=_DEFAULT_SIZE_FLUSH_THRESHOLD, # type: int
- large_buffer_warn_threshold_mib = 512 # type: int
+ large_buffer_warn_threshold_bytes = 512 << 20 # type: int
):
super().__init__(close_callback)
self._flush_callback = flush_callback
self._size_flush_threshold = size_flush_threshold
-self._large_buffer_warn_threshold_mib = large_buffer_warn_threshold_mib
+self._large_buffer_warn_threshold_bytes = large_buffer_warn_threshold_bytes
# This must be called explicitly to avoid flushing partial elements.
def maybe_flush(self):
@@ -152,8 +152,8 @@ class
SizeBasedBufferingClosableOutputStream(ClosableOutputStream):
# type: () -> None
if self._flush_callback:
size = self.size()
- if (self._large_buffer_warn_threshold_mib and
- size > self._large_buffer_warn_threshold_mib << 20):
+ if (self._large_buffer_warn_threshold_bytes and
+ size > self._large_buffer_warn_threshold_bytes):
if size > _FLUSH_MAX_SIZE:
raise ValueError(
f'Buffer size {size} exceeds GRPC limit {_FLUSH_MAX_SIZE}. '
@@ -167,7 +167,7 @@ class
SizeBasedBufferingClosableOutputStream(ClosableOutputStream):
if self._large_flush_last_observed_timestamp + 600 < time.time():
self._large_flush_last_observed_timestamp = time.time()
_LOGGER.warning(
- 'Data output stream buffer size %s exceeds %s MB. '
+ 'Data output stream buffer size %s exceeds %s bytes. '
'This is likely due to a large element in a PCollection. '
'Large elements increase pipeline RAM requirements and '
'can cause runtime errors. '
@@ -176,7 +176,7 @@ class
SizeBasedBufferingClosableOutputStream(ClosableOutputStream):
'storage systems, and use PCollections to pass their metadata, '
'or use a custom coder that reduces the element\'s size.',
size,
- self._large_buffer_warn_threshold_mib)
+ self._large_buffer_warn_threshold_bytes)
self._flush_callback(self.get())
self._clear()
(beam) branch master updated (3cadc83d348 -> c22678b4efb)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 3cadc83d348 flink portable client configurations (#31188) add c22678b4efb Emit a warning when large elements are detected. (#31363) No new revisions were added by this update. Summary of changes: .../apache_beam/runners/worker/data_plane.py | 42 -- 1 file changed, 32 insertions(+), 10 deletions(-)
(beam) branch master updated: Update bigquery.py documentation (#31443)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 16d62827551 Update bigquery.py documentation (#31443) 16d62827551 is described below commit 16d62827551170c7af104995327ba88ddc1fcb88 Author: liferoad AuthorDate: Wed May 29 20:03:39 2024 -0400 Update bigquery.py documentation (#31443) * Update bigquery.py fix #31372 * fix lint --- sdks/python/apache_beam/io/gcp/bigquery.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py b/sdks/python/apache_beam/io/gcp/bigquery.py index a4d710b1288..caeed6b7b9b 100644 --- a/sdks/python/apache_beam/io/gcp/bigquery.py +++ b/sdks/python/apache_beam/io/gcp/bigquery.py @@ -283,7 +283,8 @@ method) could look like:: def chain_after(result): try: # This works for FILE_LOADS, where we run load and possibly copy jobs. - return (result.load_jobid_pairs, result.copy_jobid_pairs) | beam.Flatten() + return (result.destination_load_jobid_pairs, + result.destination_copy_jobid_pairs) | beam.Flatten() except AttributeError: # Works for STREAMING_INSERTS, where we return the rows BigQuery rejected return result.failed_rows
(beam) branch master updated (80d85aa38ff -> 90f020921c1)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 80d85aa38ff Add docs for YAML AssertThat. (#31448) add 90f020921c1 Update bigquery_tools.py (#31444) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/gcp/bigquery_tools.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated: pin the lower bound for typing-extensions (#31425)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new df067dba5dc pin the lower bound for typing-extensions (#31425) df067dba5dc is described below commit df067dba5dc509135e857596224759e485d47a0b Author: liferoad AuthorDate: Tue May 28 19:16:34 2024 -0400 pin the lower bound for typing-extensions (#31425) Co-authored-by: XQ Hu --- sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt b/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt index aebddf61226..e0a5c704de4 100644 --- a/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt +++ b/sdks/python/apache_beam/ml/inference/tensorflow_tests_requirements.txt @@ -19,4 +19,5 @@ keras<3.0.0 tensorflow>=2.12.0 tensorflow_hub>=0.10.0 Pillow>=9.0.0 +typing-extensions>=4.8.0
(beam) branch tvalentyn-gha updated (7e52aee8974 -> 919bbea3bbd)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 7e52aee8974 Explicitly install now-optional setuptools package as some tests depend on it. discard f9fc794196b drop: restrict tests to py312 only. discard 7bcbc139239 Support Python 3.12 in Beam. discard 290d0dcf4ed Support Python 3.12 in Dataflow. discard 679072e9b78 Build Python 3.12 wheels. discard ec34ca4b526 Switch remaining GHA workflows to test Python 3.12. discard c3a03fbd926 Switch Py3.11 Gradle suites to use Python 3.12. discard b8abd0ffa72 Gradle definitions for Python 3.12 discard 2e15be0d18a Add a tox suite for Py312. discard a905963e2d4 Define Python 3.12 test environments in tox. add 33fccac00e0 Updates the Java Expansion Service container to support gRPC ALTS authentication (#31352) add bedba65a1f7 Update README.md (fix typo) (#31380) add 2ee266bec03 Fixed the extension name in Java SDK Ordered Processing extension (#31382) add 5c8338a38c1 Reapply "[#30789] Add support for Flink 1.18 (#31062)" add a37df32264b [runners-flink] use mvn exec:exec for runQuickstartJavaFlinkLocal add 736ed4a0a31 Merge pull request #31328: Reapply "[#30789] Add support for Flink 1.18 (#31062)" add 81d58d83d87 Fix beam quest link add ec9e963735f Merge pull request #31383 from apache/users/damccorm/bql add 1f6319624a4 move playground ci nightly to highmem due to mem exaustion (#31384) add 0e075b69f07 Define Python 3.12 test environments in tox. add a0e950b454e Add a tox suite for Py312. add 7e102700b52 Gradle definitions for Python 3.12 add ddb4e7f29ff Switch Py3.11 Gradle suites to use Python 3.12. add dac22e3d074 Switch remaining GHA workflows to test Python 3.12. add d12c628a388 Explicitly install now-optional setuptools package as some tests depend on it. add 6486591594a Build Python 3.12 wheels. add 42c84822d42 Support Python 3.12 in Dataflow. add ab460c1f5ba Support Python 3.12 in Beam. add dc6b72caf73 Merge pull request #30828 - Add Python 3.12 support add 6876734da12 Don't install TF on Python 3.12. add 697c18fd6a5 Update setup.py add 1ed29e1f94a Merge pull request #31386 from tvalentyn/py312_tf add ca002724341 Update beam-master version to 20240524. (#31393) add 3b9b6ed0038 added pytest.mark.flaky for test_big_query_write_temp_table_append_sc… (#31364) add cf3592bbfa0 Fix an incompatibility with hamcrest 2.2 (#31395) add c10f5d40456 [runners-flink] Drop Flnk 1.14 and clean up code (#31112) add 0790d698251 Merge pull request #31394: #31112 drop flink 1.14 add 5a822adc269 [flink] #31390 emit watermark with empty source add 836e77e755b Merge pull request #31391: [flink] #31390 emit watermark with empty source add 5c30b1d6db2 Clean up stale code in BigtableService (#30172) add 2719ca7f37a Changed the retry order for test_big_query_write_temp_table_append_schema_update (#31407) add e764fc9c17d remove unused field (#31293) add 919bbea3bbd pin the lower bound for typing-extensions This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (7e52aee8974) \ N -- N -- N refs/heads/tvalentyn-gha (919bbea3bbd) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../test-properties.json | 2 +- .github/workflows/beam_Playground_CI_Nightly.yml | 2 +- .../workflows/beam_PostCommit_Java_Tpcds_Flink.yml | 2 +- .../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 2 +- .github/workflows/beam_PostCommit_Python.yml | 2 +- ..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +- CHANGES.md | 4 +- contributor-docs/release-guide.md | 2 +- examples/multi-language/README.md | 2 +- gradle.properties | 2 +- .../runner-concepts/description.md | 2 +- release/build.gradle.kts | 2 +- .../main/groovy/quickstart-java-flinklocal.groovy | 24 +++--- .../streaming/AbstractStreamOperatorCompat.java| 91 -- .../streaming/ProcessingTimeCallbackCompat.java| 22 -- .../io/source/compat/FlinkSourceCompat.j
(beam) branch master updated: Memoize some dataframes analysis operations. (#31377)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e488f41b9bd Memoize some dataframes analysis operations. (#31377)
e488f41b9bd is described below
commit e488f41b9bde0c16f396eefe0a9ab6e1adcfe5eb
Author: Robert Bradshaw
AuthorDate: Tue May 28 12:07:12 2024 -0700
Memoize some dataframes analysis operations. (#31377)
ReadFromCsv with an explicit dtype produced graphs that had quadratic
traversal (though the computed results, sets, were always correct).
This fixes https://github.com/apache/beam/issues/31152 and should help
other deep expressions with common references as well.
---
sdks/python/apache_beam/dataframe/expressions.py | 6 --
sdks/python/apache_beam/dataframe/io_test.py | 9 +
sdks/python/apache_beam/dataframe/transforms.py | 1 +
3 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/sdks/python/apache_beam/dataframe/expressions.py
b/sdks/python/apache_beam/dataframe/expressions.py
index ae08cdaf54c..91d237c7de9 100644
--- a/sdks/python/apache_beam/dataframe/expressions.py
+++ b/sdks/python/apache_beam/dataframe/expressions.py
@@ -365,8 +365,10 @@ class ComputedExpression(Expression):
self._preserves_partition_by = preserves_partition_by
def placeholders(self):
-return frozenset.union(
-frozenset(), *[arg.placeholders() for arg in self.args()])
+if not hasattr(self, '_placeholders'):
+ self._placeholders = frozenset.union(
+ frozenset(), *[arg.placeholders() for arg in self.args()])
+return self._placeholders
def args(self):
return self._args
diff --git a/sdks/python/apache_beam/dataframe/io_test.py
b/sdks/python/apache_beam/dataframe/io_test.py
index 782dac53e2c..92bb10225c7 100644
--- a/sdks/python/apache_beam/dataframe/io_test.py
+++ b/sdks/python/apache_beam/dataframe/io_test.py
@@ -117,6 +117,15 @@ A B
self.assertCountEqual(['a,b,c', '1,2,3', '3,4,7'],
set(self.read_all_lines(output + 'out.csv*')))
+ def test_wide_csv_with_dtypes(self):
+# Verify https://github.com/apache/beam/issues/31152 is resolved.
+cols = ','.join(f'col{ix}' for ix in range(123))
+data = ','.join(str(ix) for ix in range(123))
+input = self.temp_dir({'tmp.csv': f'{cols}\n{data}'})
+with beam.Pipeline() as p:
+ pcoll = p | beam.io.ReadFromCsv(f'{input}tmp.csv', dtype=str)
+ assert_that(pcoll | beam.Map(max), equal_to(['99']))
+
def test_sharding_parameters(self):
data = pd.DataFrame({'label': ['11a', '37a', '389a'], 'rank': [0, 1, 2]})
output = self.temp_dir()
diff --git a/sdks/python/apache_beam/dataframe/transforms.py
b/sdks/python/apache_beam/dataframe/transforms.py
index 2e815408314..852b49c4e2e 100644
--- a/sdks/python/apache_beam/dataframe/transforms.py
+++ b/sdks/python/apache_beam/dataframe/transforms.py
@@ -302,6 +302,7 @@ class _DataframeExpressionsTransform(transforms.PTransform):
self.outputs))
# First define some helper functions.
+@_memoize
def output_partitioning_in_stage(expr, stage):
"""Return the output partitioning of expr when computed in stage,
or returns None if the expression cannot be computed in this stage.
(beam) 01/01: Revert "Clean up stale code in BigtableService (#30172)"
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch revert-30172-old_code
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 2005a53766a1adbda75c6d03c08461391f05bbbc
Author: tvalentyn
AuthorDate: Tue May 28 10:51:44 2024 -0700
Revert "Clean up stale code in BigtableService (#30172)"
This reverts commit 5c30b1d6db27fff299fc28249f9f195d54651abd.
---
.../beam/sdk/io/gcp/bigtable/BigtableService.java | 7 ++
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 78 --
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 10 +++
.../io/gcp/bigtable/BigtableServiceImplTest.java | 43 +++-
4 files changed, 131 insertions(+), 7 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableService.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableService.java
index 1e3839b5df4..3a3de5622cd 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableService.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableService.java
@@ -30,6 +30,7 @@ import java.util.NoSuchElementException;
import java.util.concurrent.CompletionStage;
import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.BigtableSource;
import org.apache.beam.sdk.values.KV;
+import org.joda.time.Duration;
/** An interface for real or fake implementations of Cloud Bigtable. */
interface BigtableService extends Serializable {
@@ -75,6 +76,12 @@ interface BigtableService extends Serializable {
* current row because the last such call was unsuccessful.
*/
Row getCurrentRow() throws NoSuchElementException;
+
+// Workaround for ReadRows requests which requires to pass the timeouts in
+// ApiContext. Can be removed later once it's fixed in Veneer.
+Duration getAttemptTimeout();
+
+Duration getOperationTimeout();
}
/** Returns a {@link Reader} that will read from the specified source. */
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
index dad3370dae6..d6208be1bf9 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableServiceImpl.java
@@ -23,6 +23,7 @@ import com.google.api.core.ApiFuture;
import com.google.api.gax.batching.Batcher;
import com.google.api.gax.batching.BatchingException;
import com.google.api.gax.grpc.GrpcCallContext;
+import com.google.api.gax.retrying.RetrySettings;
import com.google.api.gax.rpc.ApiException;
import com.google.api.gax.rpc.ResponseObserver;
import com.google.api.gax.rpc.StreamController;
@@ -47,6 +48,8 @@ import com.google.cloud.bigtable.data.v2.models.RowAdapter;
import com.google.cloud.bigtable.data.v2.models.RowMutation;
import com.google.cloud.bigtable.data.v2.models.RowMutationEntry;
import com.google.protobuf.ByteString;
+import io.grpc.CallOptions;
+import io.grpc.Deadline;
import io.grpc.StatusRuntimeException;
import java.io.IOException;
import java.util.ArrayDeque;
@@ -105,6 +108,9 @@ class BigtableServiceImpl implements BigtableService {
BigtableServiceImpl(BigtableDataSettings settings) throws IOException {
this.projectId = settings.getProjectId();
this.instanceId = settings.getInstanceId();
+RetrySettings retry =
settings.getStubSettings().readRowsSettings().getRetrySettings();
+this.readAttemptTimeout =
Duration.millis(retry.getInitialRpcTimeout().toMillis());
+this.readOperationTimeout =
Duration.millis(retry.getTotalTimeout().toMillis());
this.client = BigtableDataClient.create(settings);
LOG.info("Started Bigtable service with settings {}", settings);
}
@@ -113,6 +119,10 @@ class BigtableServiceImpl implements BigtableService {
private final String projectId;
private final String instanceId;
+ private final Duration readAttemptTimeout;
+
+ private final Duration readOperationTimeout;
+
@Override
public BigtableWriterImpl openForWriting(BigtableWriteOptions writeOptions) {
return new BigtableWriterImpl(
@@ -135,6 +145,9 @@ class BigtableServiceImpl implements BigtableService {
private final RowFilter rowFilter;
private Iterator results;
+private final Duration attemptTimeout;
+private final Duration operationTimeout;
+
private Row currentRow;
@VisibleForTesting
@@ -144,13 +157,18 @@ class BigtableServiceImpl implements BigtableService {
String instanceId,
String tableId,
List ranges,
-@Nullable RowFilter rowFilter) {
+@Nul
(beam) branch revert-30172-old_code created (now 2005a53766a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch revert-30172-old_code in repository https://gitbox.apache.org/repos/asf/beam.git at 2005a53766a Revert "Clean up stale code in BigtableService (#30172)" This branch includes the following new commits: new 2005a53766a Revert "Clean up stale code in BigtableService (#30172)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: Include Python 3.12 in release validation steps. (#31415)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new bb4c1e6ded5 Include Python 3.12 in release validation steps. (#31415)
bb4c1e6ded5 is described below
commit bb4c1e6ded54550f9d9e87ad3083ef341bfdac39
Author: tvalentyn
AuthorDate: Tue May 28 10:25:16 2024 -0700
Include Python 3.12 in release validation steps. (#31415)
---
release/src/main/Dockerfile | 7 ---
release/src/main/python-release/python_release_automation.sh | 2 +-
release/src/main/scripts/build_release_candidate.sh | 2 +-
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/release/src/main/Dockerfile b/release/src/main/Dockerfile
index 62d389889ea..8f981fddfcd 100644
--- a/release/src/main/Dockerfile
+++ b/release/src/main/Dockerfile
@@ -33,9 +33,9 @@ RUN apt-get update && \
RUN apt-get install -y build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev llvm \
libncurses5-dev libncursesw5-dev xz-utils tk-dev \
-libffi-dev liblzma-dev python3-openssl
+libffi-dev liblzma-dev python3-openssl
-# Install pyenv and install all of the Python versions
+# Install pyenv and install all of the Python versions
# needed to build containers
RUN curl https://pyenv.run | bash && \
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> /root/.bashrc && \
@@ -46,7 +46,8 @@ RUN curl https://pyenv.run | bash && \
pyenv install 3.9.4 && \
pyenv install 3.10.7 && \
pyenv install 3.11.3 && \
-pyenv global 3.8.9 3.9.4 3.10.7 3.11.3
+pyenv install 3.12.3 && \
+pyenv global 3.8.9 3.9.4 3.10.7 3.11.3 3.12.3
# Install a Go version >= 1.16 so we can bootstrap higher
# Go versions
diff --git a/release/src/main/python-release/python_release_automation.sh
b/release/src/main/python-release/python_release_automation.sh
index e245406b57d..2f6986885a9 100755
--- a/release/src/main/python-release/python_release_automation.sh
+++ b/release/src/main/python-release/python_release_automation.sh
@@ -19,7 +19,7 @@
source
release/src/main/python-release/run_release_candidate_python_quickstart.sh
source
release/src/main/python-release/run_release_candidate_python_mobile_gaming.sh
-for version in 3.8 3.9 3.10 3.11
+for version in 3.8 3.9 3.10 3.11 3.12
do
run_release_candidate_python_quickstart"tar" "python${version}"
run_release_candidate_python_mobile_gaming "tar" "python${version}"
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index 81425f46ed5..8d37b53d372 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -348,7 +348,7 @@ if [[ $confirmation = "y" ]]; then
cd ${BEAM_ROOT_DIR}
RELEASE_COMMIT=$(git rev-list -n 1 "tags/${RC_TAG}")
# TODO(https://github.com/apache/beam/issues/20209): Don't hardcode py
version in this file.
- cd sdks/python && tox -e py38-docs
+ cd sdks/python && tox -e docs
GENERATED_PYDOC=~/${LOCAL_WEBSITE_UPDATE_DIR}/${LOCAL_PYTHON_DOC}/${BEAM_ROOT_DIR}/sdks/python/target/docs/_build
rm -rf ${GENERATED_PYDOC}/.doctrees
(beam) branch master updated: Changed the retry order for test_big_query_write_temp_table_append_schema_update (#31407)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 2719ca7f37a Changed the retry order for
test_big_query_write_temp_table_append_schema_update (#31407)
2719ca7f37a is described below
commit 2719ca7f37abaa490d3e5861ae80c12d978c2c37
Author: liferoad
AuthorDate: Mon May 27 13:38:07 2024 -0400
Changed the retry order for
test_big_query_write_temp_table_append_schema_update (#31407)
* changed the retry order
* updated shape
* fixed save_model
* added load_model_args
---
sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 2 +-
.../apache_beam/ml/inference/tensorflow_inference_test.py | 14 +-
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
index 4c8e8ebf489..b0140793cf7 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
@@ -506,7 +506,6 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
equal_to(bq_result_errors))
@pytest.mark.it_postcommit
- @retry(reraise=True, stop=stop_after_attempt(3))
@parameterized.expand([
param(file_format=FileFormat.AVRO),
param(file_format=FileFormat.JSON),
@@ -514,6 +513,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase):
])
@mock.patch(
"apache_beam.io.gcp.bigquery_file_loads._MAXIMUM_SOURCE_URIS", new=1)
+ @retry(reraise=True, stop=stop_after_attempt(3))
def test_big_query_write_temp_table_append_schema_update(self, file_format):
"""
Test that nested schema update options and schema relaxation
diff --git a/sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py
b/sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py
index 52c525cc0ea..52123516de1 100644
--- a/sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py
+++ b/sdks/python/apache_beam/ml/inference/tensorflow_inference_test.py
@@ -65,7 +65,7 @@ class FakeTFTensorModel:
def _create_mult2_model():
- inputs = tf.keras.Input(shape=(3))
+ inputs = tf.keras.Input(shape=(3, ))
outputs = tf.keras.layers.Lambda(lambda x: x * 2, dtype='float32')(inputs)
return tf.keras.Model(inputs=inputs, outputs=outputs)
@@ -127,7 +127,7 @@ class TFRunInferenceTest(unittest.TestCase):
def test_predict_tensor_with_batch_size(self):
model = _create_mult2_model()
-model_path = os.path.join(self.tmpdir, 'mult2')
+model_path = os.path.join(self.tmpdir, 'mult2.keras')
tf.keras.models.save_model(model, model_path)
with TestPipeline() as pipeline:
@@ -146,6 +146,7 @@ class TFRunInferenceTest(unittest.TestCase):
model_handler = TFModelHandlerTensor(
model_uri=model_path,
inference_fn=fake_batching_inference_fn,
+ load_model_args={'safe_mode': False},
min_batch_size=2,
max_batch_size=2)
examples = [
@@ -172,7 +173,7 @@ class TFRunInferenceTest(unittest.TestCase):
def test_predict_tensor_with_large_model(self):
model = _create_mult2_model()
-model_path = os.path.join(self.tmpdir, 'mult2')
+model_path = os.path.join(self.tmpdir, 'mult2.keras')
tf.keras.models.save_model(model, model_path)
with TestPipeline() as pipeline:
@@ -193,6 +194,7 @@ class TFRunInferenceTest(unittest.TestCase):
model_handler = TFModelHandlerTensor(
model_uri=model_path,
inference_fn=fake_batching_inference_fn,
+ load_model_args={'safe_mode': False},
large_model=True)
examples = [
tf.convert_to_tensor(numpy.array([1.1, 2.2, 3.3], dtype='float32')),
@@ -218,7 +220,7 @@ class TFRunInferenceTest(unittest.TestCase):
def test_predict_numpy_with_batch_size(self):
model = _create_mult2_model()
-model_path = os.path.join(self.tmpdir, 'mult2_numpy')
+model_path = os.path.join(self.tmpdir, 'mult2_numpy.keras')
tf.keras.models.save_model(model, model_path)
with TestPipeline() as pipeline:
@@ -237,6 +239,7 @@ class TFRunInferenceTest(unittest.TestCase):
model_handler = TFModelHandlerNumpy(
model_uri=model_path,
inference_fn=fake_batching_inference_fn,
+ load_model_args={'safe_mode': False},
min_batch_size=2,
max_batch_size=2)
examples = [
@@ -260,7 +263,7 @@ class TFRunInferenceTest(unittest.TestCase):
def test_predict_numpy_with_large_model(self):
model = _create_mult2_model()
-model_path = os.path.join(self.tmpdir, 'mult2_numpy')
+model_path = os
(beam) branch master updated: added pytest.mark.flaky for test_big_query_write_temp_table_append_sc… (#31364)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 3b9b6ed0038 added pytest.mark.flaky for test_big_query_write_temp_table_append_sc… (#31364) 3b9b6ed0038 is described below commit 3b9b6ed00386fa43cc6577745a61bba62d69622e Author: liferoad AuthorDate: Fri May 24 18:33:02 2024 -0400 added pytest.mark.flaky for test_big_query_write_temp_table_append_sc… (#31364) * added pytest.mark.flaky for test_big_query_write_temp_table_append_schema_update * updated the version * only run flaky when it is available * switched to tenacity * fixed the isort --- sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py index 4b728fe7ec1..4c8e8ebf489 100644 --- a/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py +++ b/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py @@ -34,6 +34,8 @@ import pytest import pytz from parameterized import param from parameterized import parameterized +from tenacity import retry +from tenacity import stop_after_attempt import apache_beam as beam from apache_beam.io.gcp.bigquery import BigQueryWriteFn @@ -457,7 +459,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase): @pytest.mark.it_postcommit def test_big_query_write_insert_non_transient_api_call_error(self): """ -Test that non-transient GoogleAPICallError errors returned +Test that non-transient GoogleAPICallError errors returned by beam.io.WriteToBigQuery are not retried and result in FAILED_ROWS containing both the failed rows and the reason for failure. @@ -504,6 +506,7 @@ class BigQueryWriteIntegrationTests(unittest.TestCase): equal_to(bq_result_errors)) @pytest.mark.it_postcommit + @retry(reraise=True, stop=stop_after_attempt(3)) @parameterized.expand([ param(file_format=FileFormat.AVRO), param(file_format=FileFormat.JSON),
(beam) branch revert-31386-py312_tf created (now 733ca11341a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch revert-31386-py312_tf in repository https://gitbox.apache.org/repos/asf/beam.git at 733ca11341a Revert "Don't install TF on Python 3.12, since there is no compatible version atm." This branch includes the following new commits: new 733ca11341a Revert "Don't install TF on Python 3.12, since there is no compatible version atm." The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Revert "Don't install TF on Python 3.12, since there is no compatible version atm."
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch revert-31386-py312_tf in repository https://gitbox.apache.org/repos/asf/beam.git commit 733ca11341a34baaa1cc09386ccc6edb50bd4c8b Author: tvalentyn AuthorDate: Fri May 24 08:52:00 2024 -0700 Revert "Don't install TF on Python 3.12, since there is no compatible version atm." --- sdks/python/setup.py | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sdks/python/setup.py b/sdks/python/setup.py index 3a885658315..4a7b57db917 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -488,9 +488,8 @@ if __name__ == '__main__': 'onnxruntime', 'sentence-transformers', 'skl2onnx', - # Support TF 2.16.0: https://github.com/apache/beam/issues/31294 - # Once TF version is unpinned, also don't restrict Python version. - 'tensorflow<2.16.0;python_version<"3.12"', + # https://github.com/apache/beam/issues/31294 + 'tensorflow<2.16.0', 'tensorflow-hub', # https://github.com/tensorflow/transform/issues/313 'tensorflow-transform;python_version<"3.11"',
(beam) 01/01: Merge pull request #31386 from tvalentyn/py312_tf
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 1ed29e1f94af9d6c3ae562d14e59c565cb2c731c Merge: dc6b72caf73 697c18fd6a5 Author: tvalentyn AuthorDate: Fri May 24 08:51:38 2024 -0700 Merge pull request #31386 from tvalentyn/py312_tf Don't install TF on Python 3.12, since there is no compatible version atm. sdks/python/setup.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam) branch master updated (dc6b72caf73 -> 1ed29e1f94a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from dc6b72caf73 Merge pull request #30828 - Add Python 3.12 support add 6876734da12 Don't install TF on Python 3.12. add 697c18fd6a5 Update setup.py new 1ed29e1f94a Merge pull request #31386 from tvalentyn/py312_tf The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/setup.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam) branch master updated (1f6319624a4 -> dc6b72caf73)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 1f6319624a4 move playground ci nightly to highmem due to mem exaustion
(#31384)
add 0e075b69f07 Define Python 3.12 test environments in tox.
add a0e950b454e Add a tox suite for Py312.
add 7e102700b52 Gradle definitions for Python 3.12
add ddb4e7f29ff Switch Py3.11 Gradle suites to use Python 3.12.
add dac22e3d074 Switch remaining GHA workflows to test Python 3.12.
add d12c628a388 Explicitly install now-optional setuptools package as some
tests depend on it.
add 6486591594a Build Python 3.12 wheels.
add 42c84822d42 Support Python 3.12 in Dataflow.
add ab460c1f5ba Support Python 3.12 in Beam.
add dc6b72caf73 Merge pull request #30828 - Add Python 3.12 support
No new revisions were added by this update.
Summary of changes:
.github/workflows/beam_PostCommit_Python.yml | 2 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 4 +--
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 2 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml| 6 ++---
...stCommit_Python_ValidatesContainer_Dataflow.yml | 2 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 2 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml | 2 +-
.../beam_PostCommit_Python_Xlang_Gcp_Direct.yml| 2 +-
.../beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +-
.../workflows/beam_PostCommit_Sickbay_Python.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml| 4 +--
.github/workflows/beam_PostCommit_XVR_Direct.yml | 4 +--
.github/workflows/beam_PostCommit_XVR_Flink.yml| 4 +--
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 4 +--
..._PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml | 4 +--
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 4 +--
.github/workflows/beam_PostCommit_XVR_Samza.yml| 4 +--
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 4 +--
.../workflows/beam_PreCommit_Portable_Python.yml | 2 +-
.github/workflows/beam_PreCommit_Python.yml| 2 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 4 +--
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 2 +-
.../beam_PreCommit_Python_Integration.yml | 2 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 2 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 10
.../workflows/beam_PreCommit_Python_Runners.yml| 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 4 +--
.github/workflows/build_wheels.yml | 2 +-
.github/workflows/python_dependency_tests.yml | 1 +
.github/workflows/python_tests.yml | 5 ++--
.github/workflows/update_python_dependencies.yml | 9 ---
CHANGES.md | 5 ++--
build.gradle.kts | 13 --
gradle.properties | 2 +-
sdks/python/apache_beam/__init__.py| 2 +-
.../runners/dataflow/internal/apiclient.py | 2 +-
.../runners/dataflow/internal/apiclient_test.py| 2 +-
sdks/python/setup.py | 2 ++
.../dataflow/{py310 => py312}/build.gradle | 2 +-
.../direct/{py38 => py312}/build.gradle| 2 +-
sdks/python/test-suites/gradle.properties | 22
.../portable/{py310 => py312}/build.gradle | 2 +-
.../test-suites/tox/{py310 => py312}/build.gradle | 4 +--
sdks/python/tox.ini| 30 +++---
settings.gradle.kts| 4 +++
54 files changed, 117 insertions(+), 100 deletions(-)
copy sdks/python/test-suites/dataflow/{py310 => py312}/build.gradle (97%)
copy sdks/python/test-suites/direct/{py38 => py312}/build.gradle (97%)
copy sdks/python/test-suites/portable/{py310 => py312}/build.gradle (97%)
copy sdks/python/test-suites/tox/{py310 => py312}/build.gradle (94%)
(beam) branch tvalentyn-gha updated (567720b3cb5 -> 7e52aee8974)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git omit 567720b3cb5 Install setuptools when running a juliaset IT. omit eafc2293946 Don't set SETUPTOOLS_USE_DISTUTILS=stdlib since it doesn't work on Py3.12 omit 8953bdcca5b drop: restrict tests to py312 only. omit c6b394cafd0 Support Python 3.12 in Beam. omit 065022612a2 Support Python 3.12 in Dataflow. omit 1eb98339af0 Build Python 3.12 wheels. omit b0d6b2e1c0e Switch remaining GHA workflows to test Python 3.12. omit 2f967944ff8 Switch Py3.11 Gradle suites to use Python 3.12. omit 1f3d7311fec Gradle definitions for Python 3.12 omit 89d5ad18cf1 Add a tox suite for Py312. omit 68d19da82e1 Define Python 3.12 test environments in tox. add 649e33c214d Don't set SETUPTOOLS_USE_DISTUTILS=stdlib (#31370) add a905963e2d4 Define Python 3.12 test environments in tox. add 2e15be0d18a Add a tox suite for Py312. add b8abd0ffa72 Gradle definitions for Python 3.12 add c3a03fbd926 Switch Py3.11 Gradle suites to use Python 3.12. add ec34ca4b526 Switch remaining GHA workflows to test Python 3.12. add 679072e9b78 Build Python 3.12 wheels. add 290d0dcf4ed Support Python 3.12 in Dataflow. add 7bcbc139239 Support Python 3.12 in Beam. add f9fc794196b drop: restrict tests to py312 only. add 7e52aee8974 Explicitly install now-optional setuptools package as some tests depend on it. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (567720b3cb5) \ N -- N -- N refs/heads/tvalentyn-gha (7e52aee8974) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: sdks/python/setup.py | 5 - sdks/python/test-suites/portable/common.gradle | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-)
(beam) branch master updated (611676d108b -> 649e33c214d)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 611676d108b Fix DebeziumIO testWrongHost trying to connect real external address (#31365) add 649e33c214d Don't set SETUPTOOLS_USE_DISTUTILS=stdlib (#31370) No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 -- .test-infra/jenkins/CommonJobProperties.groovy | 3 --- sdks/python/build.gradle | 1 - sdks/python/setup.py | 4 +++- 4 files changed, 3 insertions(+), 7 deletions(-)
(beam) branch tvalentyn-gha updated (9019925e3e3 -> 567720b3cb5)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 9019925e3e3 drop: maybe reword: don't set CIBW variable to use stdlib. discard 21312037d42 drop: restrict tests to py312 only. discard b89ba7776dd Upgrade cibuildwheel and also install setuptools. discard 5a743b49b93 Also install setuptools. discard 8d95889f9a4 Support Python 3.12 in Beam. discard a098bbb88d5 Support Python 3.12 in Dataflow. discard f99b3bc2210 Build Python 3.12 wheels. discard e76cd89eac0 Switch remaining GHA workflows to test Python 3.12. discard 6971be752bc Switch Py3.11 Gradle suites to use Python 3.12. discard 73624667677 Gradle definitions for Python 3.12 discard 347e082c7c6 Add a tox suite for Py312. discard 3bcd1539bc5 Define Python 3.12 test environments in tox. add a435f452211 [Java] Disable soft delete policy when creating a default bucket for a project. (#31324) add f851cbb317e Update hamcrest dependency documentation version (#31309) add 71f2278c79f Update beam_Inference_Python_Benchmarks_Dataflow_Pytorch_Imagenet_Classification_Resnet_152_Tesla_T4_GPU.txt (#31341) add 092f769a4b4 add core transforms category to approximatequantiles.py and approximateunique.py to pass playground validation (#31331) add 2e4a152cecd start the connection before calling receive in JmsIOTest (#31334) add fed64891240 Implement CoderNullable pullDecoderNoAlloc case (#31340) add 89795c0e7c7 Make GrpcCommitWorkStream thread-safe as documented by moving batcher out of it. (#31304) add 88af35e6df4 PubsubIO: Add readMessagesWithAttributesWithCoderAndParseFn (#31206) add 0e734fe1c38 [Go SDK] Make it clearer that timers and data don't interact negatively in element batches. (#31319) add f4119f564fb Upgrade jetty to latest 9.x for dataflow legacy worker (#31351) add 675dab26ec0 Enable BigQueryIO write throttling detection (#31253) add 1a52285109c Logging loading filesystems failures. (#31219) add f508b033c2b Bump com.gradle.enterprise from 3.17.2 to 3.17.4 (#31345) add cfe206bb352 skip requests version 2.32.0,2.32.1 (#31359) add c5b6475db2a [Python] Disable soft delete policy when creating new default bucket. (#31344) add a998107a1f5 [Java] Add warning to gcpTempLocation when its bucket has soft delete enabled (#31358) add 98b5d5b93d3 Upgrade cibuildwheel in another place as well. (#31337) add 611676d108b Fix DebeziumIO testWrongHost trying to connect real external address (#31365) add 68d19da82e1 Define Python 3.12 test environments in tox. add 89d5ad18cf1 Add a tox suite for Py312. add 1f3d7311fec Gradle definitions for Python 3.12 add 2f967944ff8 Switch Py3.11 Gradle suites to use Python 3.12. add b0d6b2e1c0e Switch remaining GHA workflows to test Python 3.12. add 1eb98339af0 Build Python 3.12 wheels. add 065022612a2 Support Python 3.12 in Dataflow. add c6b394cafd0 Support Python 3.12 in Beam. add 8953bdcca5b drop: restrict tests to py312 only. add eafc2293946 Don't set SETUPTOOLS_USE_DISTUTILS=stdlib since it doesn't work on Py3.12 add 567720b3cb5 Install setuptools when running a juliaset IT. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9019925e3e3) \ N -- N -- N refs/heads/tvalentyn-gha (567720b3cb5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 1 - ...enet_Classification_Resnet_152_Tesla_T4_GPU.txt | 4 +- .test-infra/jenkins/CommonJobProperties.groovy | 3 - CHANGES.md | 1 + .../google-cloud-dataflow-java/worker/build.gradle | 6 +- .../dataflow/worker/StreamingDataflowWorker.java | 3 +- .../worker/windmill/client/WindmillStream.java | 39 +-- .../commits/StreamingApplianceWorkCommitter.java | 3 +- .../commits/StreamingEngineWorkCommitter.java | 43 .../windmill/client/grpc/GrpcCommitWorkStream.java | 72 ++-- .../dataflow/worker/FakeWindmillServer.java| 104 +++--- .../commits/StreamingEngineWorkCommitterTest.java | 57 -- .../client/grpc/GrpcWindmillServerTest.java| 121 +++-- sdks/go/pkg/beam
(beam) branch master updated: Upgrade cibuildwheel in another place as well. (#31337)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 98b5d5b93d3 Upgrade cibuildwheel in another place as well. (#31337)
98b5d5b93d3 is described below
commit 98b5d5b93d3d4b1e1fe4e5ac50a01325a143d632
Author: tvalentyn
AuthorDate: Tue May 21 16:38:45 2024 -0700
Upgrade cibuildwheel in another place as well. (#31337)
---
.github/workflows/build_wheels.yml | 2 +-
sdks/python/build.gradle | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build_wheels.yml
b/.github/workflows/build_wheels.yml
index f7410d6536f..a1e3d7dee8a 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -267,7 +267,7 @@ jobs:
name: Set up QEMU
- name: Install cibuildwheel
# note: sync cibuildwheel version with gradle task sdks:python:bdistPy*
steps
- run: pip install cibuildwheel==2.9.0
+ run: pip install cibuildwheel==2.17.0 setuptools
- name: Build wheel
working-directory: apache-beam-source
env:
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index e10b2a90dfb..72ea587b88f 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -166,7 +166,7 @@ platform_identifiers_map.each { platform, idsuffix ->
args '-c', ". ${envdir}/bin/activate && " +
// note: sync cibuildwheel version with GitHub Action
// .github/workflows/build_wheel.yml:build_wheels "Install
cibuildwheel" step
- "pip install cibuildwheel==2.17.0 && " +
+ "pip install cibuildwheel==2.17.0 setuptools && " +
"cibuildwheel --print-build-identifiers --platform
${platform} --archs ${archs} && " +
"cibuildwheel --output-dir ${buildDir} --platform ${platform}
--archs ${archs} "
}
(beam) branch tvalentyn-patch-5 deleted (was c885fd41377)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-5 in repository https://gitbox.apache.org/repos/asf/beam.git was c885fd41377 Update version tag for dataflow dev containres. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch tvalentyn-patch-3 deleted (was d7cfc7825d4)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-3 in repository https://gitbox.apache.org/repos/asf/beam.git was d7cfc7825d4 remove --pre for testing purposes The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch tvalentyn-patch-1 deleted (was 32f8fe73189)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git was 32f8fe73189 Also allow links to Drive materials. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch tvalentyn-patch-2 deleted (was e17c19f403d)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-2 in repository https://gitbox.apache.org/repos/asf/beam.git was e17c19f403d Don't pin an older commit in the documentation. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch tvalentyn-gha updated (9d64f3c6a2a -> 9019925e3e3)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 9d64f3c6a2a drop: restrict tests to py312 only. discard 82e4a6bb0c0 Upgrade cibuildwheel and also install setuptools. discard 97991328554 Also install setuptools. discard e1ce6602619 Upgrade cibuildwheel. discard 485913128fe Support Python 3.12 in Beam. discard f4417e1395f Support Python 3.12 in Dataflow. discard 98010e16018 Build Python 3.12 wheels. discard ca99deabe27 Switch remaining GHA workflows to test Python 3.12. discard 02fe949438f Switch Py3.11 Gradle suites to use Python 3.12. discard 232633fae6d Gradle definitions for Python 3.12 discard 69abdd16576 Add a tox suite for Py312. discard b30085d10de Define Python 3.12 test environments in tox. add 93a5bc76577 [Python] Use retry decorator for retrying (#31273) add 28a7199e80b [Java] Add job name to GCS custom audit info (#31316) add 4f0c5fb Done with python scripts/tools/bomupgrader.py 26.39.0 based on https://github.com/apache/beam/blob/master/contributor-docs/java-dependency-upgrades.md (#31318) add d15d98c0026 Update code-change-guide.md (#31333) add d7165f940cc Upgrade cibuildwheel version. (#31321) add 44177d1a9f3 [Python] BigQuery handler for enrichment transform (#31295) add 3bcd1539bc5 Define Python 3.12 test environments in tox. add 347e082c7c6 Add a tox suite for Py312. add 73624667677 Gradle definitions for Python 3.12 add 6971be752bc Switch Py3.11 Gradle suites to use Python 3.12. add e76cd89eac0 Switch remaining GHA workflows to test Python 3.12. add f99b3bc2210 Build Python 3.12 wheels. add a098bbb88d5 Support Python 3.12 in Dataflow. add 8d95889f9a4 Support Python 3.12 in Beam. add 5a743b49b93 Also install setuptools. add b89ba7776dd Upgrade cibuildwheel and also install setuptools. add 21312037d42 drop: restrict tests to py312 only. add 9019925e3e3 drop: maybe reword: don't set CIBW variable to use stdlib. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (9d64f3c6a2a) \ N -- N -- N refs/heads/tvalentyn-gha (9019925e3e3) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/trigger_files/beam_PostCommit_Python.json | 1 + .github/workflows/build_wheels.yml | 1 - CHANGES.md | 1 + .../org/apache/beam/gradle/BeamModulePlugin.groovy | 8 +- contributor-docs/code-change-guide.md | 4 +- .../container/license_scripts/dep_urls_java.yaml | 2 +- .../gcp/util/RetryHttpRequestInitializer.java | 11 + .../beam/sdk/extensions/gcp/util/Transport.java| 32 ++- .../sdk/extensions/gcp/util/TransportTest.java | 35 ++- sdks/python/apache_beam/io/requestresponse.py | 121 ++--- sdks/python/apache_beam/io/requestresponse_test.py | 61 +++-- sdks/python/apache_beam/transforms/enrichment.py | 14 +- .../transforms/enrichment_handlers/bigquery.py | 256 ++ .../enrichment_handlers/bigquery_it_test.py| 289 + .../enrichment_handlers/bigquery_test.py | 70 + sdks/python/build.gradle | 1 - sdks/python/scripts/generate_pydoc.sh | 1 + 17 files changed, 806 insertions(+), 102 deletions(-) create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_it_test.py create mode 100644 sdks/python/apache_beam/transforms/enrichment_handlers/bigquery_test.py
(beam) branch tvalentyn-gha updated (97991328554 -> 9d64f3c6a2a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 97991328554 Also install setuptools. add 82e4a6bb0c0 Upgrade cibuildwheel and also install setuptools. add 9d64f3c6a2a drop: restrict tests to py312 only. No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python.yml | 2 +- .github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +- .github/workflows/build_wheels.yml| 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
(beam) branch master updated: Upgrade cibuildwheel version. (#31321)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new d7165f940cc Upgrade cibuildwheel version. (#31321)
d7165f940cc is described below
commit d7165f940cc700e369363cbd8ef3469f31448914
Author: tvalentyn
AuthorDate: Fri May 17 09:30:05 2024 -0700
Upgrade cibuildwheel version. (#31321)
---
sdks/python/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index e21f5db75ad..e10b2a90dfb 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -166,7 +166,7 @@ platform_identifiers_map.each { platform, idsuffix ->
args '-c', ". ${envdir}/bin/activate && " +
// note: sync cibuildwheel version with GitHub Action
// .github/workflows/build_wheel.yml:build_wheels "Install
cibuildwheel" step
- "pip install cibuildwheel==2.9.0 && " +
+ "pip install cibuildwheel==2.17.0 && " +
"cibuildwheel --print-build-identifiers --platform
${platform} --archs ${archs} && " +
"cibuildwheel --output-dir ${buildDir} --platform ${platform}
--archs ${archs} "
}
(beam) branch master updated: [Python] Use retry decorator for retrying (#31273)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 93a5bc76577 [Python] Use retry decorator for retrying (#31273)
93a5bc76577 is described below
commit 93a5bc765773c168ada907ba78cd0d04b1c5e390
Author: Ritesh Ghorse
AuthorDate: Thu May 16 20:33:17 2024 -0400
[Python] Use retry decorator for retrying (#31273)
---
sdks/python/apache_beam/io/requestresponse_test.py | 61 ++
1 file changed, 29 insertions(+), 32 deletions(-)
diff --git a/sdks/python/apache_beam/io/requestresponse_test.py
b/sdks/python/apache_beam/io/requestresponse_test.py
index a4427bcefa2..3bc85a5e103 100644
--- a/sdks/python/apache_beam/io/requestresponse_test.py
+++ b/sdks/python/apache_beam/io/requestresponse_test.py
@@ -18,6 +18,10 @@ import logging
import time
import unittest
+from tenacity import retry
+from tenacity import retry_if_exception_type
+from tenacity import stop_after_attempt
+
import apache_beam as beam
from apache_beam.testing.test_pipeline import TestPipeline
@@ -134,39 +138,32 @@ class TestCaller(unittest.TestCase):
| RequestResponseIO(caller=caller, repeater=None))
self.assertRegex(cm.exception.message, 'retries = 0')
+ @retry(
+ retry=retry_if_exception_type(IndexError),
+ reraise=True,
+ stop=stop_after_attempt(MAX_TEST_RETRIES))
def test_default_throttler(self):
-for i in range(MAX_TEST_RETRIES):
- try:
-caller = CallerWithTimeout()
-throttler = DefaultThrottler(
-window_ms=1, bucket_ms=5000, overload_ratio=1)
-# manually override the number of received requests for testing.
-throttler.throttler._all_requests.add(time.time() * 1000, 100)
-test_pipeline = TestPipeline()
-_ = (
-test_pipeline
-| beam.Create(['sample_request'])
-| RequestResponseIO(caller=caller, throttler=throttler))
-result = test_pipeline.run()
-result.wait_until_finish()
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name('throttled_requests'))
-self.assertEqual(metrics['counters'][0].committed, 1)
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name(
-'cumulativeThrottlingSeconds'))
-self.assertGreater(metrics['counters'][0].committed, 0)
-metrics = result.metrics().query(
-beam.metrics.MetricsFilter().with_name('responses'))
-self.assertEqual(metrics['counters'][0].committed, 1)
- except IndexError:
-if i != MAX_TEST_RETRIES - 1:
- _LOGGER.warning(
- 'Retry %s: Metrics were expected but not found' % i + 1)
- continue
-raise IndexError('List index out of range. Metrics not found.')
- else:
-break
+caller = CallerWithTimeout()
+throttler = DefaultThrottler(
+window_ms=1, bucket_ms=5000, overload_ratio=1)
+# manually override the number of received requests for testing.
+throttler.throttler._all_requests.add(time.time() * 1000, 100)
+test_pipeline = TestPipeline()
+_ = (
+test_pipeline
+| beam.Create(['sample_request'])
+| RequestResponseIO(caller=caller, throttler=throttler))
+result = test_pipeline.run()
+result.wait_until_finish()
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('throttled_requests'))
+self.assertEqual(metrics['counters'][0].committed, 1)
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('cumulativeThrottlingSeconds'))
+self.assertGreater(metrics['counters'][0].committed, 0)
+metrics = result.metrics().query(
+beam.metrics.MetricsFilter().with_name('responses'))
+self.assertEqual(metrics['counters'][0].committed, 1)
if __name__ == '__main__':
(beam) branch tvalentyn-gha updated (e1ce6602619 -> 97991328554)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from e1ce6602619 Upgrade cibuildwheel. add 97991328554 Also install setuptools. No new revisions were added by this update. Summary of changes: sdks/python/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch tvalentyn-gha updated (485913128fe -> e1ce6602619)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 485913128fe Support Python 3.12 in Beam. add e1ce6602619 Upgrade cibuildwheel. No new revisions were added by this update. Summary of changes: sdks/python/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch tvalentyn-gha updated (dbd91a046f8 -> 485913128fe)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard dbd91a046f8 Support Python 3.12 in Beam. discard 7a8a7a4a98a Support Python 3.12 in Dataflow. discard cabe9c8cd36 Build Python 3.12 wheels. discard ead13ba7ef8 Switch remaining GHA workflows to test Python 3.12. discard e878ca40b0a Switch Py3.11 Gradle suites to use Python 3.12. add 02fe949438f Switch Py3.11 Gradle suites to use Python 3.12. add ca99deabe27 Switch remaining GHA workflows to test Python 3.12. add 98010e16018 Build Python 3.12 wheels. add f4417e1395f Support Python 3.12 in Dataflow. add 485913128fe Support Python 3.12 in Beam. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (dbd91a046f8) \ N -- N -- N refs/heads/tvalentyn-gha (485913128fe) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python_Examples_Dataflow.yml | 4 ++-- .github/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml | 6 +++--- .../workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml| 2 +- .github/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml | 2 +- .github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml | 2 +- .../beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml| 4 ++-- .github/workflows/beam_PreCommit_Python_PVR_Flink.yml | 10 +- .github/workflows/python_dependency_tests.yml | 1 + .github/workflows/update_python_dependencies.yml | 9 + sdks/python/test-suites/gradle.properties | 2 +- 10 files changed, 22 insertions(+), 20 deletions(-)
(beam) branch tvalentyn-gha updated (2af26187399 -> dbd91a046f8)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 2af26187399 Support Python 3.12 in Beam. discard 376c0fafe6e Support Python 3.12 in Dataflow. discard a71b5108e79 Build Python 3.12 wheels. discard 885f65ecbf0 Switch remaining GHA workflows to test Python 3.12. discard b0d6d0169e0 Switch Py3.11 Gradle suites to use Python 3.12. discard 060b9b35d79 Gradle definitions for Python 3.12 discard 357796d4b26 Add a tox suite for Py312. discard 902bf98c6f2 Define Python 3.12 test environments in tox. add 2babd0aee5b Merge pull request #31120: Update Guava version to 33.1.0-jre to match the GCP Libraries BOM 26.38.0 add b30085d10de Define Python 3.12 test environments in tox. add 69abdd16576 Add a tox suite for Py312. add 232633fae6d Gradle definitions for Python 3.12 add e878ca40b0a Switch Py3.11 Gradle suites to use Python 3.12. add ead13ba7ef8 Switch remaining GHA workflows to test Python 3.12. add cabe9c8cd36 Build Python 3.12 wheels. add 7a8a7a4a98a Support Python 3.12 in Dataflow. add dbd91a046f8 Support Python 3.12 in Beam. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2af26187399) \ N -- N -- N refs/heads/tvalentyn-gha (dbd91a046f8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- gradle.properties | 2 +- settings.gradle.kts| 3 +++ 3 files changed, 5 insertions(+), 2 deletions(-)
(beam) branch tvalentyn-gha updated (c84e23a623c -> 2af26187399)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git omit c84e23a623c Support Python 3.12 in Beam. omit ce22bd1b778 Support Python 3.12 in Dataflow. omit fca3febef73 Build Python 3.12 wheels. omit 6b4848a1099 Switch remaining GHA workflows to test Python 3.12. add b0d6d0169e0 Switch Py3.11 Gradle suites to use Python 3.12. add 885f65ecbf0 Switch remaining GHA workflows to test Python 3.12. add a71b5108e79 Build Python 3.12 wheels. add 376c0fafe6e Support Python 3.12 in Dataflow. add 2af26187399 Support Python 3.12 in Beam. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c84e23a623c) \ N -- N -- N refs/heads/tvalentyn-gha (2af26187399) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: sdks/python/test-suites/gradle.properties | 20 ++-- 1 file changed, 10 insertions(+), 10 deletions(-)
(beam) branch tvalentyn-gha updated (57aa3e6f689 -> c84e23a623c)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 57aa3e6f689 Support Python 3.12 in Beam.
discard 8b6741ad1e3 Support Python 3.12 in Dataflow.
discard a2e489db53c Build Python 3.12 wheels.
discard f9f8b59b3d3 Switch remaining GHA workflows to test Python 3.12.
discard be08e19d246 Gradle definitions for Python 3.12
discard 1391921c7d5 Add a tox suite for Py312.
discard f5fe5a5fe0d Define Python 3.12 test environments in tox.
add 999f7deb94e [Python] Add job name to GCS custom audit info (#31300)
add 902bf98c6f2 Define Python 3.12 test environments in tox.
add 357796d4b26 Add a tox suite for Py312.
add 060b9b35d79 Gradle definitions for Python 3.12
add 6b4848a1099 Switch remaining GHA workflows to test Python 3.12.
add fca3febef73 Build Python 3.12 wheels.
add ce22bd1b778 Support Python 3.12 in Dataflow.
add c84e23a623c Support Python 3.12 in Beam.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (57aa3e6f689)
\
N -- N -- N refs/heads/tvalentyn-gha (c84e23a623c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
sdks/python/apache_beam/io/gcp/gcsio.py| 48 --
sdks/python/apache_beam/io/gcp/gcsio_test.py | 47 +
.../runners/dataflow/internal/apiclient.py | 20 ++---
.../apache_beam/runners/interactive/utils.py | 18 +---
.../runners/portability/sdk_container_builder.py | 14 ++-
.../dataflow/{py310 => py312}/build.gradle | 2 +-
.../direct/{py38 => py312}/build.gradle| 2 +-
.../portable/{py310 => py312}/build.gradle | 2 +-
8 files changed, 94 insertions(+), 59 deletions(-)
copy sdks/python/test-suites/dataflow/{py310 => py312}/build.gradle (97%)
copy sdks/python/test-suites/direct/{py38 => py312}/build.gradle (97%)
copy sdks/python/test-suites/portable/{py310 => py312}/build.gradle (97%)
(beam) branch tvalentyn-gha updated (82558028498 -> 57aa3e6f689)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 82558028498 Support Python 3.12 in Beam.
discard 9e0a5d49fce Support Python 3.12 in Dataflow.
discard 911fd417af8 Build Python 3.12 wheels.
discard 2194aa644d5 Switch remaining GHA workflows to test Python 3.12.
discard f88089de8a3 Add a tox suite for Py312.
discard 2e8c62d54d2 Define Python 3.12 test environments in tox.
discard ede1d831152 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
add 118c51404d9 Implement DeduplicateTensorPerRow in MLTransform (#31307)
add 084f23b2a8d Install Beam from wheels in Dependency Compat Test Suite.
(#31308)
add 743e34e0098 Fix iceberg unit tests (#31314)
add 126d92249f9 Support newer versions of Pyarrow in Beam. (#31305)
add 840faea2262 Create option to specify temp query project, and wire into
source tab… (#31128)
add f5fe5a5fe0d Define Python 3.12 test environments in tox.
add 1391921c7d5 Add a tox suite for Py312.
add be08e19d246 Gradle definitions for Python 3.12
add f9f8b59b3d3 Switch remaining GHA workflows to test Python 3.12.
add a2e489db53c Build Python 3.12 wheels.
add 8b6741ad1e3 Support Python 3.12 in Dataflow.
add 57aa3e6f689 Support Python 3.12 in Beam.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (82558028498)
\
N -- N -- N refs/heads/tvalentyn-gha (57aa3e6f689)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
..._Gcp_Direct.json => IO_Iceberg_Unit_Tests.json} | 0
...json => beam_PostCommit_Python_Dependency.json} | 0
.github/workflows/IO_Iceberg_Unit_Tests.yml| 4 +-
.github/workflows/beam_PostCommit_Python_Arm.yml | 2 +-
.../beam_PostCommit_Python_Dependency.yml | 1 +
.../beam_PostCommit_Python_Examples_Spark.yml | 2 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 2 +-
.../beam_PostCommit_TransformService_Direct.yml| 2 +-
.github/workflows/beam_PostCommit_XVR_Flink.yml| 4 +-
...eam_PostCommit_XVR_JavaUsingPython_Dataflow.yml | 4 +-
...eam_PostCommit_XVR_PythonUsingJava_Dataflow.yml | 4 +-
.github/workflows/beam_PostCommit_XVR_Samza.yml| 4 +-
.github/workflows/beam_PostCommit_XVR_Spark3.yml | 4 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 4 +-
.github/workflows/beam_PreCommit_Python_ML.yml | 2 +-
build.gradle.kts | 12 -
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 39 +++---
.../sdk/io/gcp/bigquery/BigQueryIOTranslation.java | 10 +++-
.../sdk/io/gcp/bigquery/BigQueryQueryHelper.java | 15 +++---
.../io/gcp/bigquery/BigQueryQuerySourceDef.java| 26 +++--
.../gcp/bigquery/BigQueryStorageQuerySource.java | 8 +++
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java| 26 -
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 6 +++
.../io/gcp/bigquery/BigQueryIOTranslationTest.java | 1 +
sdks/python/apache_beam/ml/transforms/tft.py | 22
sdks/python/apache_beam/ml/transforms/tft_test.py | 62 ++
26 files changed, 225 insertions(+), 41 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Python_Xlang_Gcp_Direct.json =>
IO_Iceberg_Unit_Tests.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json
=> beam_PostCommit_Python_Dependency.json} (100%)
(beam) branch master updated: Support newer versions of Pyarrow in Beam. (#31305)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 126d92249f9 Support newer versions of Pyarrow in Beam. (#31305)
126d92249f9 is described below
commit 126d92249f9babb1977087df277940f8646916d9
Author: tvalentyn
AuthorDate: Thu May 16 08:50:52 2024 -0700
Support newer versions of Pyarrow in Beam. (#31305)
* Increase pyarrow upper bound. Add compat tests, and remove some of the
compat suites for pyarrow to reduce test suite runtime.
---
.../beam_PostCommit_Python_Dependency.json | 0
sdks/python/setup.py | 14 +
sdks/python/test-suites/tox/py38/build.gradle | 63 +++---
sdks/python/tox.ini| 30 ---
4 files changed, 54 insertions(+), 53 deletions(-)
diff --git a/.github/trigger_files/beam_PostCommit_Python_Dependency.json
b/.github/trigger_files/beam_PostCommit_Python_Dependency.json
new file mode 100644
index 000..e69de29bb2d
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 3e1bfda0dd3..852f14117d8 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -143,21 +143,9 @@ except ImportError:
# [BEAM-8181] pyarrow cannot be installed on 32-bit Windows platforms.
if sys.platform == 'win32' and sys.maxsize <= 2**32:
pyarrow_dependency = ['']
-elif sys.platform == 'win32' or sys.platform == 'cygwin':
- # https://github.com/apache/beam/issues/28410 - pyarrow>=13 seeing issues
- # on windows with error
- # C:\arrow\cpp\src\arrow\filesystem\s3fs.cc:2904: arrow::fs::FinalizeS3 was
- # not called even though S3 was initialized. This could lead to a
- # segmentation fault at exit. Keep pyarrow<13 until this is resolved.
- pyarrow_dependency = [
- 'pyarrow>=3.0.0,<12.0.0',
- # NOTE: We can remove this once Beam increases the pyarrow lower bound
- # to a version that fixes CVE.
- 'pyarrow-hotfix<1'
- ]
else:
pyarrow_dependency = [
- 'pyarrow>=3.0.0,<15.0.0',
+ 'pyarrow>=3.0.0,<17.0.0',
# NOTE(https://github.com/apache/beam/issues/29392): We can remove this
# once Beam increases the pyarrow lower bound to a version that fixes
CVE.
'pyarrow-hotfix<1'
diff --git a/sdks/python/test-suites/tox/py38/build.gradle
b/sdks/python/test-suites/tox/py38/build.gradle
index 52299d8c31f..2ca82d3d926 100644
--- a/sdks/python/test-suites/tox/py38/build.gradle
+++ b/sdks/python/test-suites/tox/py38/build.gradle
@@ -44,36 +44,48 @@ project.tasks.register("preCommitPyCoverage") {
// e.g. pyarrow and pandas also run on PreCommit Dataframe and Coverage
project.tasks.register("postCommitPyDep") {}
-// Create a test task for each supported major version of pyarrow
+// Create a test task for supported major versions of pyarrow
+// We should have a test for the lowest supported version and
+// For versions that we would like to prioritize for testing,
+// for example versions released in a timeframe of last 1-2 years.
+
toxTask "testPy38pyarrow-3", "py38-pyarrow-3", "${posargs}"
test.dependsOn "testPy38pyarrow-3"
postCommitPyDep.dependsOn "testPy38pyarrow-3"
-toxTask "testPy38pyarrow-4", "py38-pyarrow-4", "${posargs}"
-test.dependsOn "testPy38pyarrow-4"
-postCommitPyDep.dependsOn "testPy38pyarrow-4"
+toxTask "testPy38pyarrow-9", "py38-pyarrow-9", "${posargs}"
+test.dependsOn "testPy38pyarrow-9"
+postCommitPyDep.dependsOn "testPy38pyarrow-9"
-toxTask "testPy38pyarrow-5", "py38-pyarrow-5", "${posargs}"
-test.dependsOn "testPy38pyarrow-5"
-postCommitPyDep.dependsOn "testPy38pyarrow-5"
+toxTask "testPy38pyarrow-10", "py38-pyarrow-10", "${posargs}"
+test.dependsOn "testPy38pyarrow-10"
+postCommitPyDep.dependsOn "testPy38pyarrow-10"
-toxTask "testPy38pyarrow-6", "py38-pyarrow-6", "${posargs}"
-test.dependsOn "testPy38pyarrow-6"
-postCommitPyDep.dependsOn "testPy38pyarrow-6"
+toxTask "testPy38pyarrow-11", "py38-pyarrow-11", "${posargs}"
+test.dependsOn "testPy38pyarrow-11"
+postCommitPyDep.dependsOn "testPy38pyarrow-11"
-toxTask "testPy38pyarrow-7", "py38-pyarrow-7", "${posargs}"
-test.dependsOn "testPy38pyarrow-7"
-postCommitPyDep.dependsOn "testPy38pyarrow-7"
+toxTask "testPy38pyarrow-12", "py38-pyarrow-12", "${posargs}"
+test.dependsOn "testPy38
(beam) branch tvalentyn-gha updated (2194aa644d5 -> 82558028498)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from 2194aa644d5 Switch remaining GHA workflows to test Python 3.12. add 911fd417af8 Build Python 3.12 wheels. add 9e0a5d49fce Support Python 3.12 in Dataflow. add 82558028498 Support Python 3.12 in Beam. No new revisions were added by this update. Summary of changes: .github/workflows/build_wheels.yml | 2 +- sdks/python/apache_beam/__init__.py | 2 +- sdks/python/apache_beam/runners/dataflow/internal/apiclient.py | 2 +- sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py | 2 +- sdks/python/setup.py| 1 + 5 files changed, 5 insertions(+), 4 deletions(-)
(beam) branch tvalentyn-gha updated (f88089de8a3 -> 2194aa644d5)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git from f88089de8a3 Add a tox suite for Py312. add 2194aa644d5 Switch remaining GHA workflows to test Python 3.12. No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python.yml | 2 +- .github/workflows/beam_PostCommit_Python_Examples_Flink.yml | 2 +- .../workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml | 2 +- .../beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml| 2 +- .github/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml| 2 +- .github/workflows/beam_PostCommit_Python_ValidatesRunner_Samza.yml| 2 +- .github/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml| 2 +- .github/workflows/beam_PostCommit_Sickbay_Python.yml | 2 +- .github/workflows/beam_PostCommit_TransformService_Direct.yml | 2 +- .github/workflows/beam_PostCommit_XVR_Direct.yml | 4 ++-- .github/workflows/beam_PreCommit_Portable_Python.yml | 2 +- .github/workflows/beam_PreCommit_Python_Examples.yml | 2 +- .github/workflows/beam_PreCommit_Python_Integration.yml | 2 +- .github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml | 4 ++-- 14 files changed, 16 insertions(+), 16 deletions(-)
(beam) branch tvalentyn-gha updated (ead701cf765 -> f88089de8a3)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
omit ead701cf765 Install Beam from wheels in Dependency Compat Test Suite.
omit 3dac627e918 trigger
omit 7593608e219 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
add 041a8dd1173 Touch Samza ValidatesRunner trigger files
add f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
add 2196758c20b Merge pull request #31270: Reapply "Add Redistribute
translation to Samza runner"
add 6cb30cc5c86 setup GCP auth before running tests (#31306)
add ede1d831152 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
add 2e8c62d54d2 Define Python 3.12 test environments in tox.
add f88089de8a3 Add a tox suite for Py312.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (ead701cf765)
\
N -- N -- N refs/heads/tvalentyn-gha (f88089de8a3)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../beam_PostCommit_Java_PVR_Samza.json| 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
.../beam_PostCommit_Python_Dependency.json | 0
.github/workflows/IO_Iceberg_Unit_Tests.yml| 6 ++
.../beam_PostCommit_Python_Dependency.yml | 1 -
.../beam_PostCommit_Python_Examples_Direct.yml | 2 +-
.github/workflows/beam_PreCommit_Python.yml| 2 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 2 +-
.../workflows/beam_PreCommit_Python_Runners.yml| 2 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 2 +-
.github/workflows/python_tests.yml | 5 +-
build.gradle.kts | 1 +
.../translation/RedistributeByKeyTranslator.java | 64 ++
.../samza/translation/ReshuffleTranslator.java | 14 -
.../samza/translation/SamzaPipelineTranslator.java | 1 +
.../test-suites/tox/{py311 => py312}/build.gradle | 2 +-
sdks/python/tox.ini| 30 +-
settings.gradle.kts| 1 +
18 files changed, 113 insertions(+), 28 deletions(-)
delete mode 100644 .github/trigger_files/beam_PostCommit_Python_Dependency.json
create mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
copy sdks/python/test-suites/tox/{py311 => py312}/build.gradle (97%)
(beam) branch tvalentyn-gha updated (db6a0ec8340 -> ead701cf765)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
discard db6a0ec8340 Define Python 3.12 test environments in tox.
discard a01bcfd53a8 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
omit 2196758c20b Merge pull request #31270: Reapply "Add Redistribute
translation to Samza runner"
omit f58a9d6680e Reapply "Add Redistribute translation to Samza runner"
omit 041a8dd1173 Touch Samza ValidatesRunner trigger files
add 7593608e219 Increase pyarrow upper bound. Add compat tests, and remove
some of the compat suites for pyarrow to reduce test suite runtime.
add 3dac627e918 trigger
add ead701cf765 Install Beam from wheels in Dependency Compat Test Suite.
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (db6a0ec8340)
\
N -- N -- N refs/heads/tvalentyn-gha (ead701cf765)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../beam_PostCommit_Java_PVR_Samza.json| 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
...json => beam_PostCommit_Python_Dependency.json} | 0
.../beam_PostCommit_Python_Dependency.yml | 1 +
.../translation/RedistributeByKeyTranslator.java | 64 --
.../samza/translation/ReshuffleTranslator.java | 14 +
.../samza/translation/SamzaPipelineTranslator.java | 1 -
sdks/python/tox.ini| 30 +-
8 files changed, 20 insertions(+), 96 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json
=> beam_PostCommit_Python_Dependency.json} (100%)
delete mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
(beam) branch tvalentyn-gha updated (b3bf42b65e6 -> db6a0ec8340)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard b3bf42b65e6 trigger discard 00ad85d284d Attmpt to update pyarrow and add compat test suite definitions. discard 1ba37cd6d69 Define Python 3.12 test environments in tox. add 5a42d54bfd1 Catch org.elasticsearch.client.ResponseException and return document with withHasError(true) add 29b47e69716 Remove status add 84b1d92a73e Fix Style add ff858b8b966 doc -> doc.withHasError(true) add 3cbf7de5796 Add tests add 1647872e054 Fix styles via spotlessApply add 41772b015c2 rename INJECT_SOME_INVALID_DOCS and insert numDoc add 00c59b251ec Catch with java.io.IOExecption add 591d2c75679 explicit import add 0f6a79cd2eb Removed unused import add a7380297011 java util explicit import add 035c88da6cf spotlessapply add c6eabd19faf Fix max batch size add 77da5add258 Rename message from org.elasticsearch.client.ResponseException to java.io.IOException add 1ee2f0ec5e6 Add test to verify response items message add e010a3ecc66 spotlessapply add 5ee569f930b fix comment add ed84ba0e0dc Fix naming add 1a399b96977 Verify the whole error message string add e97064aa14c Remove mapToInputDoc add 946f4959b7b retry 500-range http except 501 add 68790eb242f Add elasticResponseExceptionMessage predicate add 3d897b27635 Merge pull request #31151 from andyzhangdialpad/master add 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288) add 041a8dd1173 Touch Samza ValidatesRunner trigger files add f58a9d6680e Reapply "Add Redistribute translation to Samza runner" add 2196758c20b Merge pull request #31270: Reapply "Add Redistribute translation to Samza runner" add a01bcfd53a8 Increase pyarrow upper bound. Add compat tests, and remove some of the compat suites for pyarrow to reduce test suite runtime. add db6a0ec8340 Define Python 3.12 test environments in tox. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b3bf42b65e6) \ N -- N -- N refs/heads/tvalentyn-gha (db6a0ec8340) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../beam_PostCommit_Java_PVR_Samza.json| 3 +- ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +- .../beam_PostCommit_Python_Dependency.json | 0 .../translation/RedistributeByKeyTranslator.java | 64 ++ .../samza/translation/ReshuffleTranslator.java | 14 - .../samza/translation/SamzaPipelineTranslator.java | 1 + .../sdk/io/elasticsearch/ElasticsearchIOTest.java | 6 ++ .../elasticsearch/ElasticsearchIOTestCommon.java | 52 ++ .../io/elasticsearch/ElasticsearchIOTestUtils.java | 26 - .../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 54 ++ .../ml/transforms/embeddings/huggingface_test.py | 2 + sdks/python/setup.py | 20 ++- sdks/python/test-suites/tox/py38/build.gradle | 39 ++--- sdks/python/tox.ini| 45 +-- 14 files changed, 247 insertions(+), 82 deletions(-) delete mode 100644 .github/trigger_files/beam_PostCommit_Python_Dependency.json create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
(beam) branch master updated: Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288) 6de9a60dc9e is described below commit 6de9a60dc9e456d34b3fcd0583e29372656c995c Author: tvalentyn AuthorDate: Wed May 15 08:38:01 2024 -0700 Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288) * Don't install TFT on Python 3.11+ * Add an upper bound to TF due to #31294 . * Run huggingface tests without parallelism: see comments in #31287 --- .../apache_beam/ml/transforms/embeddings/huggingface_test.py | 2 ++ sdks/python/setup.py | 8 +--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py b/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py index f38fc8e8542..f94e747c5ed 100644 --- a/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py +++ b/sdks/python/apache_beam/ml/transforms/embeddings/huggingface_test.py @@ -21,6 +21,7 @@ import unittest import uuid import numpy as np +import pytest from parameterized import parameterized import apache_beam as beam @@ -80,6 +81,7 @@ _parameterized_inputs = [ ] +@pytest.mark.no_xdist @unittest.skipIf( SentenceTransformerEmbeddings is None, 'sentence-transformers is not installed.') diff --git a/sdks/python/setup.py b/sdks/python/setup.py index 181d810971a..3e1bfda0dd3 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -490,16 +490,18 @@ if __name__ == '__main__': # We don't expect users to install this extra. Users should install # necessary dependencies individually, or we should create targeted # extras. Keeping the bounds open as much as possible so that we - # can find out early when using Beam with new versions doesn't work. + # can find out early when Beam doesn't work with new versions. 'ml_test': [ 'datatable', 'embeddings', 'onnxruntime', 'sentence-transformers', 'skl2onnx', - 'tensorflow', + # https://github.com/apache/beam/issues/31294 + 'tensorflow<2.16.0', 'tensorflow-hub', - 'tensorflow_transform', + # https://github.com/tensorflow/transform/issues/313 + 'tensorflow-transform;python_version<"3.11"', 'tf2onnx', 'torch', 'transformers',
(beam) branch tvalentyn-gha updated (bd832576057 -> b3bf42b65e6)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
discard bd832576057 Trigger tests
discard 15c077fac15 Don't install TFT on Python 3.11+
add 785855e6c4c Exclude testRedistributePreservesMetadata from Spark PVR
batch (#31286)
add 54db453b032 Fix postrelease revert leftover (#31292)
add 1ba37cd6d69 Define Python 3.12 test environments in tox.
add 00ad85d284d Attmpt to update pyarrow and add compat test suite
definitions.
add b3bf42b65e6 trigger
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (bd832576057)
\
N -- N -- N refs/heads/tvalentyn-gha (b3bf42b65e6)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 ---
.../beam_PostCommit_Java_PVR_Spark_Batch.json | 3 +-
...json => beam_PostCommit_Python_Dependency.json} | 0
release/build.gradle.kts | 2 +-
runners/flink/flink_runner.gradle | 8 --
runners/spark/job-server/spark_job_server.gradle | 3 +-
sdks/python/setup.py | 7 ++---
sdks/python/test-suites/tox/py38/build.gradle | 28 ++
sdks/python/tox.ini| 33 +-
9 files changed, 61 insertions(+), 27 deletions(-)
delete mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json
rename .github/trigger_files/{beam_PostCommit_Python_ML.json =>
beam_PostCommit_Python_Dependency.json} (100%)
(beam) branch tvalentyn-gha updated (15c077fac15 -> bd832576057)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch tvalentyn-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
from 15c077fac15 Don't install TFT on Python 3.11+
add bd832576057 Trigger tests
No new revisions were added by this update.
Summary of changes:
...t_Python_ValidatesRunner_Flink.json => beam_PostCommit_Python_ML.json} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Python_ValidatesRunner_Flink.json
=> beam_PostCommit_Python_ML.json} (100%)
(beam) branch tvalentyn-gha updated (0a827f31b7a -> 15c077fac15)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git omit 0a827f31b7a Include ML deps for coverage analysis. omit 404b57a7fa1 Move ML compatibility tests to postcommits. omit 13c133176f1 Lint omit 05451b62a97 Add a subsuite of unit tests for ML package add fbb3ae1ebef add retries for flaky test (#31247) add 45c730ad0c7 Update version tag for dataflow dev containers. (#31248) add 2e5f69c4bb9 Update try-apache-beam-yaml.ipynb (#31257) add 66da760357c Update yaml to non-beta link (#31210) add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and PVR suites add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK" add 737be230347 Merge pull request #31156: Reapply "Add Redistribute transform to Java SDK" add 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265) add 749c128bef1 Disclaim direct runner result can be different with remote runners (#31250) add c230655a1e5 Implement the hash_words TFT operation (#31249) add 296c13a291d Avoid side input issue where Latest.globally produces 2 elements in a window (#31263) add c88a863870c Add a subsuite of unit tests for ML package and remove them from 'coverage' GHA workflow. (#31255) add 31e81ffe12a add windmill computation state cache, config loaders, and status pages (#31133) add 459c87af706 Revert "[flink] Add documentation for Flink runner versions updates" add 557b5bad1b0 Revert "[#30789] Add support for Flink 1.18 (#31062)" add 0542c83fc1a Un-revert doc fixes add bcac88b815e Merge pull request #31274 from apache/debugpostrelease add c7937001981 Beam YAML day zero notebook save file fix (#31266) add c1d1bd642d1 Fix a typo in skip condition. (#31267) add 4a15af019cc Touch trigger files for Dataflow ValidatesRunner tests add 238792d7c65 Reapply "Add direct Dataflow translation of Redistribute transform" add 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform" add 8869175a887 Fix nullness errors in TestPipeline add 7f9264ad91d Merge pull request #31230: Fix nullness errors in TestPipeline add ee170e142b9 Set backlog in gauge metric (#31137) add 5c5bcccf516 Copy edit the code contribution guide (#31279) add c061b2776b0 Revert "Don't specify an unnecessary config given that tox task and gradle config already defines the Python version." (#31283) add 15c077fac15 Don't install TFT on Python 3.11+ This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (0a827f31b7a) \ N -- N -- N refs/heads/tvalentyn-gha (15c077fac15) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../test-properties.json | 2 +- .../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 + .../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 + .../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 + .../beam_PostCommit_Java_PVR_Samza.json| 4 + .../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 + .../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +- ...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 4 +- ...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 + ...it_Java_ValidatesRunner_Dataflow_Streaming.json | 4 +- ...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 4 +- ...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 4 +- ...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 + ...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 + ...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +- ...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +- ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +- ...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +- ...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +- ...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 + ...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +- .../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +- .githu
(beam) branch master updated (5c5bcccf516 -> c061b2776b0)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 5c5bcccf516 Copy edit the code contribution guide (#31279) add c061b2776b0 Revert "Don't specify an unnecessary config given that tox task and gradle config already defines the Python version." (#31283) No new revisions were added by this update. Summary of changes: .github/workflows/beam_PreCommit_Python.yml| 3 ++- .github/workflows/beam_PreCommit_Python_Dataframes.yml | 3 ++- .github/workflows/beam_PreCommit_Python_Examples.yml | 3 ++- .github/workflows/beam_PreCommit_Python_ML.yml | 3 ++- .github/workflows/beam_PreCommit_Python_Runners.yml| 3 ++- .github/workflows/beam_PreCommit_Python_Transforms.yml | 3 ++- sdks/python/apache_beam/transforms/sql.py | 2 +- 7 files changed, 13 insertions(+), 7 deletions(-)
(beam) branch master updated: Fix a typo in skip condition. (#31267)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new c1d1bd642d1 Fix a typo in skip condition. (#31267) c1d1bd642d1 is described below commit c1d1bd642d152f05ff5bcf64b4630e059c681ba7 Author: tvalentyn AuthorDate: Tue May 14 08:04:27 2024 -0700 Fix a typo in skip condition. (#31267) --- sdks/python/apache_beam/dataframe/frames_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/dataframe/frames_test.py b/sdks/python/apache_beam/dataframe/frames_test.py index b3ddfee95fc..076ab504add 100644 --- a/sdks/python/apache_beam/dataframe/frames_test.py +++ b/sdks/python/apache_beam/dataframe/frames_test.py @@ -1449,7 +1449,7 @@ class DeferredFrameTest(_AbstractFrameTest): self._run_test(lambda s: s.unstack(level=0), s) @unittest.skipIf( - sys.version_info >= (2, 12) and PD_VERSION < (2, 3), + sys.version_info >= (3, 12) and PD_VERSION < (2, 3), 'https://github.com/pandas-dev/pandas/issues/58604') def test_unstack_pandas_example3(self): index = self._unstack_get_categorical_index()
(beam) branch master updated (296c13a291d -> c88a863870c)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 296c13a291d Avoid side input issue where Latest.globally produces 2
elements in a window (#31263)
add c88a863870c Add a subsuite of unit tests for ML package and remove
them from 'coverage' GHA workflow. (#31255)
No new revisions were added by this update.
Summary of changes:
.github/workflows/README.md | 1 +
.github/workflows/beam_PreCommit_Python.yml | 3 +--
.../workflows/beam_PreCommit_Python_Dataframes.yml| 3 +--
.github/workflows/beam_PreCommit_Python_Examples.yml | 3 +--
...ython_Runners.yml => beam_PreCommit_Python_ML.yml} | 19 +--
.github/workflows/beam_PreCommit_Python_Runners.yml | 3 +--
.../workflows/beam_PreCommit_Python_Transforms.yml| 3 +--
.../apache_beam/ml/inference/huggingface_inference.py | 2 +-
.../ml/inference/huggingface_inference_test.py| 2 +-
.../apache_beam/ml/inference/onnx_inference_test.py | 4
sdks/python/setup.py | 19 +++
sdks/python/test-suites/tox/common.gradle | 3 +++
sdks/python/test-suites/tox/py38/build.gradle | 18 ++
sdks/python/test-suites/tox/pycommon/build.gradle | 4 ++--
sdks/python/tox.ini | 8
15 files changed, 63 insertions(+), 32 deletions(-)
copy .github/workflows/{beam_PreCommit_Python_Runners.yml =>
beam_PreCommit_Python_ML.yml} (88%)
(beam) branch master updated (737be230347 -> 466516b016c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 737be230347 Merge pull request #31156: Reapply "Add Redistribute transform to Java SDK" add 466516b016c [Python] Use BufferedLogger instead of logger to log in boot.go (#31265) No new revisions were added by this update. Summary of changes: sdks/python/container/boot.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) branch tvalentyn-patch-6 created (now 9d655d9583c)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-patch-6 in repository https://gitbox.apache.org/repos/asf/beam.git at 9d655d9583c Fix a typo in skip condition. This branch includes the following new commits: new 9d655d9583c Fix a typo in skip condition. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) 01/01: Fix a typo in skip condition.
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch tvalentyn-patch-6 in repository https://gitbox.apache.org/repos/asf/beam.git commit 9d655d9583ccb948a219164e9760c74691425055 Author: tvalentyn AuthorDate: Mon May 13 08:44:59 2024 -0700 Fix a typo in skip condition. --- sdks/python/apache_beam/dataframe/frames_test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/dataframe/frames_test.py b/sdks/python/apache_beam/dataframe/frames_test.py index b3ddfee95fc..076ab504add 100644 --- a/sdks/python/apache_beam/dataframe/frames_test.py +++ b/sdks/python/apache_beam/dataframe/frames_test.py @@ -1449,7 +1449,7 @@ class DeferredFrameTest(_AbstractFrameTest): self._run_test(lambda s: s.unstack(level=0), s) @unittest.skipIf( - sys.version_info >= (2, 12) and PD_VERSION < (2, 3), + sys.version_info >= (3, 12) and PD_VERSION < (2, 3), 'https://github.com/pandas-dev/pandas/issues/58604') def test_unstack_pandas_example3(self): index = self._unstack_get_categorical_index()
(beam) branch tvalentyn-gha created (now 0a827f31b7a)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch tvalentyn-gha in repository https://gitbox.apache.org/repos/asf/beam.git at 0a827f31b7a Include ML deps for coverage analysis. No new revisions were added by this update.
(beam) branch master updated: Update version tag for dataflow dev containers. (#31248)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 45c730ad0c7 Update version tag for dataflow dev containers. (#31248) 45c730ad0c7 is described below commit 45c730ad0c706489ad435fc3210ae0fac606f07e Author: tvalentyn AuthorDate: Fri May 10 13:05:18 2024 -0700 Update version tag for dataflow dev containers. (#31248) --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index 4870bcef5e5..bc2d1f5b403 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -34,6 +34,6 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Unreleased sdks use container image tag specified below. # Update this tag whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240417' +BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240510' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
(beam) branch master updated: Fix Python 3.12 python hinting errors on MacOS (#31246)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 8b5d0009a1d Fix Python 3.12 python hinting errors on MacOS (#31246) 8b5d0009a1d is described below commit 8b5d0009a1d2561d4d04df7d952d5c33655f7a15 Author: Jack McCluskey <34928439+jrmcclus...@users.noreply.github.com> AuthorDate: Fri May 10 14:03:24 2024 -0400 Fix Python 3.12 python hinting errors on MacOS (#31246) * Fix typo in Intrinsic Operations test * Fix load_fast_and_clear --- sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py | 4 +--- sdks/python/apache_beam/typehints/opcodes.py| 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py index 8291ef2df5b..adffc945baa 100644 --- a/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py +++ b/sdks/python/apache_beam/typehints/intrinsic_one_ops_test.py @@ -30,9 +30,7 @@ class IntrinsicOneOpsTest(unittest.TestCase): def test_unary_intrinsic_ops_are_in_the_same_order_as_in_cpython(self): if sys.version_info >= (3, 12): dis_order = dis.__dict__['_intrinsic_1_descs'] - beam_ops = [fn.__name_upper() for fn in intrinsic_one_ops.INT_ONE_OPS] - for fn in intrinsic_one_ops.INT_ONE_OPS: -beam_ops.append(fn.__name__.upper()) + beam_ops = [fn.__name__.upper() for fn in intrinsic_one_ops.INT_ONE_OPS] self.assertListEqual(dis_order, beam_ops) diff --git a/sdks/python/apache_beam/typehints/opcodes.py b/sdks/python/apache_beam/typehints/opcodes.py index 11e607543e9..62c7a8fadc3 100644 --- a/sdks/python/apache_beam/typehints/opcodes.py +++ b/sdks/python/apache_beam/typehints/opcodes.py @@ -471,7 +471,7 @@ load_fast_check = load_fast def load_fast_and_clear(state, arg): state.stack.append(state.vars[arg]) - del state.vars[arg] + state.vars[arg] = None def store_fast(state, arg):
(beam) 01/01: Update version tag for dataflow dev containres.
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch tvalentyn-patch-5 in repository https://gitbox.apache.org/repos/asf/beam.git commit c885fd413777f151448372b3d01e59330dedaaa9 Author: tvalentyn AuthorDate: Fri May 10 10:59:23 2024 -0700 Update version tag for dataflow dev containres. --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index 4870bcef5e5..bc2d1f5b403 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -34,6 +34,6 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Unreleased sdks use container image tag specified below. # Update this tag whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240417' +BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20240510' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
