[
https://issues.apache.org/jira/browse/KAFKA-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Chen updated KAFKA-15083:
------------------------------
Description:
Based on the
[KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Configs]:
|_{color:#000000}remote.log.metadata.*{color}_|{color:#000000}Default RLMM
implementation creates producer and consumer instances. Common client
propoerties can be configured with `remote.log.metadata.common.client.` prefix.
User can also pass properties specific to
{color}{color:#000000}producer/consumer with `remote.log.metadata.producer.`
and `remote.log.metadata.consumer.` prefixes. These will override properties
with `remote.log.metadata.common.client.` prefix.{color}
{color:#000000}Any other properties should be prefixed with
"remote.log.metadata." and these will be passed to
RemoteLogMetadataManager#configure(Map<String, ?> props).{color}
{color:#000000}For ex: Security configuration to connect to the local broker
for the listener name configured are passed with props.{color}|
This is missed from current implementation.
When configuring RLMM, the configs passed into {{configure}} method is the
{{{}RemoteLogManagerConfig{}}}. But in {{{}RemoteLogManagerConfig{}}}, there's
no configs related to {{{}remote.log.metadata.*{}}}, ex:
{{{}remote.log.metadata.topic.replication.factor{}}}. So, even if users have
set the config in broker, it'll never be applied.
This PR fixed the issue to allow users setting RLMM prefix:
{{remote.log.metadata.manager.impl.prefix}} (default is {{{}rlmm.config.{}}}),
and then, appending the desired {{remote.log.metadata.*}} configs, it'll pass
into RLMM, including
{{{}remote.log.metadata.common.client.{}}}/{{{}remote.log.metadata.producer.{}}}/
{{remote.log.metadata.consumer.}} prefixes.
Ex:
# default value
# remote.log.storage.manager.impl.prefix=rsm.config.
# remote.log.metadata.manager.impl.prefix=rlmm.config.
rlmm.config.remote.log.metadata.topic.num.partitions=50
rlmm.config.remote.log.metadata.topic.replication.factor=4
rsm.config.test=value
was:
Based on the
[KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Configs]:
|_{color:#000000}remote.log.metadata.*{color}_|{color:#000000}Default RLMM
implementation creates producer and consumer instances. Common client
propoerties can be configured with `remote.log.metadata.common.client.` prefix.
User can also pass properties specific to
{color}{color:#000000}producer/consumer with `remote.log.metadata.producer.`
and `remote.log.metadata.consumer.` prefixes. These will override properties
with `remote.log.metadata.common.client.` prefix.{color}
{color:#000000}Any other properties should be prefixed with
"remote.log.metadata." and these will be passed to
RemoteLogMetadataManager#configure(Map<String, ?> props).{color}
{color:#000000}For ex: Security configuration to connect to the local broker
for the listener name configured are passed with props.{color}|
This is missed from current implementation.
> Passing "remote.log.metadata.*" configs into RLMM
> -------------------------------------------------
>
> Key: KAFKA-15083
> URL: https://issues.apache.org/jira/browse/KAFKA-15083
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Luke Chen
> Assignee: Luke Chen
> Priority: Major
>
> Based on the
> [KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-Configs]:
> |_{color:#000000}remote.log.metadata.*{color}_|{color:#000000}Default RLMM
> implementation creates producer and consumer instances. Common client
> propoerties can be configured with `remote.log.metadata.common.client.`
> prefix. User can also pass properties specific to
> {color}{color:#000000}producer/consumer with `remote.log.metadata.producer.`
> and `remote.log.metadata.consumer.` prefixes. These will override properties
> with `remote.log.metadata.common.client.` prefix.{color}
> {color:#000000}Any other properties should be prefixed with
> "remote.log.metadata." and these will be passed to
> RemoteLogMetadataManager#configure(Map<String, ?> props).{color}
> {color:#000000}For ex: Security configuration to connect to the local broker
> for the listener name configured are passed with props.{color}|
>
> This is missed from current implementation.
>
> When configuring RLMM, the configs passed into {{configure}} method is the
> {{{}RemoteLogManagerConfig{}}}. But in {{{}RemoteLogManagerConfig{}}},
> there's no configs related to {{{}remote.log.metadata.*{}}}, ex:
> {{{}remote.log.metadata.topic.replication.factor{}}}. So, even if users have
> set the config in broker, it'll never be applied.
> This PR fixed the issue to allow users setting RLMM prefix:
> {{remote.log.metadata.manager.impl.prefix}} (default is
> {{{}rlmm.config.{}}}), and then, appending the desired
> {{remote.log.metadata.*}} configs, it'll pass into RLMM, including
> {{{}remote.log.metadata.common.client.{}}}/{{{}remote.log.metadata.producer.{}}}/
> {{remote.log.metadata.consumer.}} prefixes.
> Ex:
> # default value
> # remote.log.storage.manager.impl.prefix=rsm.config.
> # remote.log.metadata.manager.impl.prefix=rlmm.config.
> rlmm.config.remote.log.metadata.topic.num.partitions=50
> rlmm.config.remote.log.metadata.topic.replication.factor=4
> rsm.config.test=value
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
