shahar1 opened a new pull request, #39213:
URL: https://github.com/apache/beam/pull/39213
## What & why
`EnvoyRateLimiter` (`sdks/python/apache_beam/io/components/rate_limiter.py`)
depends on `envoy-data-plane` purely to obtain a handful of protobuf message
classes. That package pulls in `betterproto==2.0.0b6` — an outdated pre-release
of a protobuf *reimplementation* — plus `grpclib` and a transitive subtree
(`h2`, `hpack`, `hyperframe`, `multidict`).
This PR removes `envoy-data-plane` (and therefore `betterproto`) entirely,
replacing it with a minimal, self-contained vendored proto.
**Why this is worth doing, and why vendoring beats bumping the dependency:**
- **The dependency was already fought, not used.** `betterproto` emits
*async* `grpclib` stubs that are incompatible with Beam's synchronous `grpcio`.
The code already hand-writes the `RateLimitServiceStub` bridge to work around
that — so the package only ever contributed 6 message dataclasses, and the wire
is plain protobuf over `grpcio` regardless.
- **It resolves a real downstream block.** The `betterproto==2.0.0b6` pin is
the last blocker stopping Apache Airflow from un-suspending its Beam provider
(apache/airflow#66952). Because Airflow's constraint solver reads Beam's
`install_requires`, removing the dependency here fixes it for **all** Python
versions — no `betterproto` pin needed on Airflow's side.
- **Bumping to `envoy-data-plane>=2.1.0` would only move the problem.** That
route swaps in `betterproto2` (still pre-1.0, `0.9.x`), requires porting to the
betterproto2 API, and drops Python 3.10 support (2.x requires `>=3.11`).
Vendoring ends the version-chase permanently and lets us delete the existing
`python_version` split in `setup.py`.
- **This is an established Beam pattern, not a novelty.**
`apache_beam/coders/proto2_coder_test_messages_pb2.py` is already a checked-in,
hand-regenerated `_pb2.py` outside the `gen_protos.py` model pipeline. This
change follows it exactly (same lint-exclusion spots, same header style). The
vendored proto is a frozen external contract — Envoy RLS v3 field numbers are
GA and cannot change without breaking every client — so it is a write-once
artifact, not an ongoing maintenance burden.
## How it works
- New `sdks/python/apache_beam/io/components/rate_limit.proto`: a ~30-line
self-contained subset of the Envoy Rate Limit Service. **Field numbers match
Envoy's `rls.proto` / `ratelimit.proto`**, so the messages are wire-compatible
with a real RLS server (protobuf carries only field numbers and types on the
wire — not message or package names).
- New checked-in `rate_limit_pb2.py`, generated via `grpc_tools.protoc`. The
generated `runtime_version` guard is removed so the module stays compatible
across Beam's full `protobuf>=3.20.3,<7` runtime range (matching the existing
test pb2). The regeneration command is documented in both files' headers.
- `rate_limiter.py` / `rate_limiter_test.py`: swapped imports to
`rate_limit_pb2`; the one behavioral adjustment is `google.protobuf.Duration`
handling (`dur.ToTimedelta().total_seconds()` — `betterproto` used to auto-map
`Duration` → `timedelta`).
## Testing
- **8/8 unit tests pass.** The original 5 mock-based tests cover the
retry/throttle logic; **3 new wire-format tests** pin the exact serialized
bytes and enum values against Envoy's field numbers — these fail loudly if the
vendored proto ever drifts, which the mock tests cannot catch.
- Verified `rate_limiter` imports and constructs with `envoy_data_plane` and
`betterproto` blocked from `sys.modules`, confirming the dependency is
genuinely gone.
- `yapf`, `isort` (repo CI flags), and `ruff` all clean.
## Follow-up
The 12 container `*_requirements.txt` files have the two **direct** packages
(`envoy-data-plane`, `betterproto`) removed so they stay consistent with
`setup.py`. A full `./gradlew
:sdks:python:container:generatePythonRequirementsAll` regeneration should run
in CI/by a committer to also prune the now-orphaned transitives (`grpclib`,
`h2`, `multidict`, …) and refresh pins — that step requires interpreters for
all supported Python versions plus a full network resolution.
------------------------
- [x] Mention the appropriate issue in your description — Part of #37854
(removes the `envoy-data-plane` tech debt called out there; does **not** close
the issue, whose TFT/protobuf-5/6 conflict remains open). Unblocks
apache/airflow#66952.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache ICLA. (Not
large.)
------------------------
##### Was AI tooling used to author this PR?
Yes — authored with Claude Code (Opus 4.8, 1M context). All changes were
reviewed by the PR author. Generated following the project's gen-AI
contribution guidelines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
