stankiewicz commented on issue #39667:
URL: https://github.com/apache/beam/issues/39667#issuecomment-5873664955
With help from AI:
The performance regression is caused by a pip dependency conflict during
Dataflow worker startup. Because the pipeline option specifies
--requirements_file=apache_beam/testing/benchmarks/cloudml/requirements.txt,
every Dataflow worker runs pip install -r requirements.txt at boot time. On
20260831, pip discovers an irreconcilable conflict between TensorFlow
Transform's dependency tree and Apache Beam's new Google Cloud Bigtable
requirement, causing pip to uninstall apache-beam 2.77.0.dev0 and downgrade it
to apache-beam==2.75.0
tensorflow-metadata 1.21.0 Caps Protobuf - tfx_bsl==1.21.0 and
tensorflow-transform==1.21.0 both depend on tensorflow-metadata==1.21.0. Its
distribution metadata specifies: protobuf<=6.32 and >=4.21.6; python_version <
"3.11"
Beam Bumped google-cloud-bigtable in Master: In
google-cloud-bigtable==2.42.0 protobuf<8.0.0, >=6.33.5
When pip installs requirements.txt on the worker:
1. tensorflow-metadata 1.21.0 is explicitly pinned in requirements.txt, so
pip cannot change it (protobuf <= 6.32 is mandatory).
2. The pre-installed apache-beam 2.77.0.dev0 requires bigtable >= 2.42.0,
which demands protobuf >= 6.33.5.
3. Because [6.0.0, 6.32] and [6.33.5, 8.0.0) do not overlap, pip concludes
that the pre-installed apache-beam 2.77.0.dev0 is incompatible.
4. tfx_bsl 1.21.0 only specifies apache-beam[gcp]<3, >=2.53. Therefore, pip
is allowed to replace Apache Beam with any older release from PyPI matching <3,
>=2.53.
5. Pip checks PyPI:
5.1 . 2.76.0: Also requires bigtable >= 2.42.0 (rejected).
5.2 . 2.75.0: Allowed bigtable >= 2.19.0 (selected, because it
allows bigtable==2.39.0 which supports protobuf <= 6.32).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
