goutamadwant opened a new pull request, #19202:
URL: https://github.com/apache/pinot/pull/19202
## Summary
- preserve Kafka ConfigProvider references while Pinot resolves environment
variables and system properties in table configs
- pass `config.providers` and `config.providers.<alias>.*` through Kafka
consumer and AdminClient property filtering
- apply the behavior to Kafka 3.x and Kafka 4.x, including the production
shared AdminClient path
- cover legacy and multi-stream table configs, provider scoping, provider
parameters, and actual Kafka client construction
## Why
Two independent transformations currently prevent Kafka ConfigProviders from
working in realtime table configs:
1. Pinot interprets a value such as
`${file:/vault/secrets/kafka.properties:keystore.password}` as its own
`${name:default}` expression while reading the table config. Kafka never
receives the reference.
2. Pinot filters Kafka client properties using
`ConsumerConfig.configNames()` or `AdminClientConfig.configNames()`. Kafka's
dynamic `config.providers.*` namespace is not part of those static name sets,
so the provider configuration is removed before client construction.
This change preserves a `${provider:...}` reference only when that provider
alias is declared by `config.providers` in the containing config object. Normal
Pinot environment and system-property substitution remains unchanged, including
across separate stream config maps.
The Kafka client filter continues to reject unrelated Pinot stream settings
and admits only the target client's known properties plus Kafka's reserved
config-provider namespace. The shared AdminClient path now applies the same
filtering.
Provider values are resolved when a Kafka client is constructed. Recreating
a consumer therefore rereads the mounted provider file; hot reloading an
already-running Kafka client is outside this change.
Kafka client properties should use their unprefixed names in
`streamConfigs`, for example `ssl.keystore.password`. The
`stream.kafka.consumer.prop.*` prefix is not a generic Kafka property namespace.
## Testing
- `./mvnw -pl pinot-spi test` (787 tests passed)
- `./mvnw -pl pinot-spi -Dtest=ConfigUtilsTest test` (5 tests passed)
- `./mvnw -pl
pinot-plugins/pinot-stream-ingestion/pinot-kafka-3.0,pinot-plugins/pinot-stream-ingestion/pinot-kafka-4.0
-am -Dtest=KafkaPartitionLevelConnectionHandlerTest
-Dsurefire.failIfNoSpecifiedTests=false test` (3 tests passed for each
connector)
- `./mvnw -pl
pinot-spi,pinot-plugins/pinot-stream-ingestion/pinot-kafka-3.0,pinot-plugins/pinot-stream-ingestion/pinot-kafka-4.0
spotless:check checkstyle:check license:check`
The connector regression creates a temporary FileConfigProvider file,
carries the unresolved password reference through Pinot table-config resolution
and Kafka property filtering, forwards `allowed.paths`, verifies the resolved
password, and constructs both a KafkaConsumer and the shared AdminClient. It
runs against Kafka 3.9.2 and Kafka 4.2.0.
User documentation lives in `pinot-contrib/pinot-docs` and should be updated
through its separate documentation workflow after this source change.
Addresses #19184
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
