sollhui opened a new pull request, #69391:
URL: https://github.com/apache/doris/pull/69391
### What problem does this PR solve?
Routine Load forwards custom Kafka properties to librdkafka without limiting
the configuration surface. Restrict the exposed options before FE metadata
requests and BE client configuration.
- Share explicit property allowlists through Thrift constants, with FE
validation for CREATE/ALTER and persisted jobs, and BE validation before any
custom `Conf::set()` call.
- Reject command/module/provider options and unknown properties. Certificate
paths must reference Doris-managed `FILE:` objects.
- Disable librdkafka's internal Kerberos ticket acquisition and renewal.
Only a zero refresh interval is accepted; FE also sends it explicitly for older
BEs during rolling upgrades.
- Publish converted properties only after successful file resolution, so a
failed conversion cannot leave a partial cache. Preserve supported consumer,
SASL, TLS, OAuth and AWS MSK IAM options.
### Release note
Routine Load Kafka options are now restricted to an explicit allowlist.
Unknown and executable options are rejected, and certificate paths must use
files registered with `CREATE FILE`. Kerberos jobs must migrate from per-job
keytabs/kinit commands to operator-managed ticket caches on eligible BEs.
Existing jobs with rejected properties must be recreated with supported
options. See `docs/routine-load-kafka-properties.md` for migration details.
### Check List (For Author)
- Test:
- [ ] Regression test: added broker-independent CREATE rejection cases;
execution blocked locally by missing JDK 17.
- [ ] Unit Test: added FE CREATE/ALTER, persisted-job and rolling-upgrade
cases, plus BE property validation, managed-file and default-Kerberos cases.
FE/BE test entry points were attempted but stopped at the JDK 17 environment
check.
- [x] Manual/static validation: FE Maven Checkstyle (zero violations),
clang-format 16 verification, BE build-hygiene gates, and `git diff --check`
passed. Checked all listed librdkafka options against pinned 2.11.0 source.
Checked the affected Thrift definition on branch-3.1/4.0/4.1/4.2; no field IDs
or wire formats change.
- Full compilation and clang-tidy remain unverified: this checkout lacks
third-party build dependencies and `compile_commands.json`.
- Behavior changed: Yes; property acceptance and Kerberos ticket management
are restricted as described above.
- Does this need documentation: Yes; migration and supported-option policy
are included in `docs/routine-load-kafka-properties.md`.
No new network request path is introduced. Existing Kafka/OAuth endpoints
remain configured by users holding the Routine Load LOAD privilege.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
