ashmeet13 commented on PR #12988:
URL: https://github.com/apache/kafka/pull/12988#issuecomment-2094869760
Hi @mjsax, apologies for the extremely absent behavior on this PR. I have
gone ahead and implemented the changes. The tests are pending and currently
working on them. Detailing the implementation down.
There are two pieces that KS controls -
1. Custom Default - Configs that have custom default values for KS compared
to the actual defaults. These values can also be overwritten by the user.
2. Controlled Configs - Configs that are controlled by KS and cannot be
overwritten by the user (We want to warn the user that this value is being
overwritten if set by the user)
### Previous Implementation -
1. We used to have the following data structures
```java
String[] NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS // Controlled KS Consumer
Configs
String[] NON_CONFIGURABLE_CONSUMER_EOS_CONFIG // Controlled KS Consumer
Configs when EoS enabled
String[] NON_CONFIGURABLE_PRODUCER_EOS_CONFIGS // Controlled KS Producer
Config when EoS enabled
Map<String, Object> PRODUCER_DEFAULT_OVERRIDES // Producer Custom Default +
Controlled Config Values
Map<String, Object> PRODUCER_EOS_OVERRIDES // Producer Custom Default +
Controlled Config Values with EoS
Map<String, Object> CONSUMER_DEFAULT_OVERRIDES // Consumer Custom Default +
Controlled Config Values
Map<String, Object> CONSUMER_EOS_OVERRIDES // Consumer Custom Default +
Controlled Config Values with EoS
```
2. The steps to return the required config broadly were:
1. **Get client configs**: Gather client configurations with prefixes
either `consumer.` or `producer.` and put them in `clientProvidedProps`.
2. **Clean `clientProvidedProps`**: Use the method
`checkIfUnexpectedUserSpecifiedConsumerConfig` to tidy up `clientProvidedProps`.
3. **Create `props`**: Generate `props` using either
`<>_DEFAULT_OVERRIDES` or `<>_EOS_OVERRIDES`.
4. **Overwrite `props`**: Replace `props` with the cleaned
`clientProvidedProps`.
5. **Fetch additional configs (only for consumer props)**: If it's
consumer props, fetch configurations set using `main.consumer.`,
`global.consumer.`, or `restore.consumer.` and add them to the `props` map.
3. After the initial setup, we make some tweaks based on whether it's for a
consumer or producer, and then we hand back the props.
### Current Implementation -
1. Give away with the old data structures and define the following new ones -
```java
Map<String, Object> KS_DEFAULT_PRODUCER_CONFIGS // KS Custom Defaults for
Producer
Map<String, Object> KS_DEFAULT_PRODUCER_CONFIGS_EOS_ENABLED // KS Custom
Defaults for Producer with EoS
Map<String, Object> KS_CONTROLLED_PRODUCER_CONFIGS // KS Controlled Configs
for Producer
Map<String, Object> KS_CONTROLLED_PRODUCER_CONFIGS_EOS_ENABLED // KS
Controlled Configs for Producer with EoS
Map<String, Object> KS_DEFAULT_CONSUMER_CONFIGS // KS Custom Defaults for
Consumer
Map<String, Object> KS_CONTROLLED_CONSUMER_CONFIGS // KS Controlled Configs
for Consumer
Map<String, Object> KS_CONTROLLED_CONSUMER_CONFIGS_EOS_ENABLED // KS
Controlled Configs for Consumer with EoS
```
2. The steps to return the required config now are:
1. **Get client configs**: Obtain client configurations with prefixes
either `consumer.` or `producer.` and place them in `clientProvidedProps`.
2. **Create `props`**: Generate `props` using either
`KS_DEFAULT_<>_CONFIGS` or `KS_DEFAULT_<>_CONFIGS_EOS_ENABLED`.
3. **Overwrite `props`**: Replace `props` with the cleaned
`clientProvidedProps`.
4. **Fetch additional configs (only for consumer props)**: If it's
consumer props, fetch configurations set using `main.consumer.`,
`global.consumer.`, or `restore.consumer.` and add them to the `props` map.
5. **Run validation check over `props`**: Perform a validation check on
`props`. This check will use `KS_CONTROLLED_<>_CONFIGS` or
`KS_CONTROLLED_<>_CONFIGS_EOS_ENABLED` maps to see if the values are already
set in `props`. If they are, log a warning and overwrite them. If not, add them
to `props`.
4. After the initial setup, we make some tweaks based on whether it's for a
consumer or producer, and then we hand back the props.
Below, I'll share the configurations organized into custom defaults and
controlled configs for both consumers and producers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
