[
https://issues.apache.org/jira/browse/KAFKA-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16316820#comment-16316820
]
Randall Hauch edited comment on KAFKA-6387 at 1/8/18 7:25 PM:
--------------------------------------------------------------
Created
https://cwiki.apache.org/confluence/display/KAFKA/KIP-246%3A+Connect+producers+and+consumers+should+inherit+worker+configs.
However, while writing up the KIP I discovered that there are a number of
cases where this behavior might actually change how the producers and consumers
used for connectors are configured. Consider a worker configuration includes
the following configuration properties:
{code}
retries=2
max.partition.fetch.bytes=262144
{code}
This configuration file does not define
"{{consumer.max.partition.fetch.bytes}}" or
"{{producer.max.partition.fetch.bytes}}", so currently these would default to
1048576. However, after this proposed change if implemented the
"{{consumer.max.partition.fetch.bytes}}" or
"{{producer.max.partition.fetch.bytes}}" values would be the inherited value of
262144, not the default 1048576.
In short, implementing this change would break backward compatibility. We could
implement a configuration switch that controls whether the configurations are
inherited, but this adds complexity to the already-complex configuration
mechanism.
was (Author: rhauch):
There are a number of cases where this behavior might actually change how the
producers and consumers used for connectors are configured. Consider a worker
configuration includes the following configuration properties:
{code}
retries=2
max.partition.fetch.bytes=262144
{code}
This configuration file does not define
"{{consumer.max.partition.fetch.bytes}}" or
"{{producer.max.partition.fetch.bytes}}", so currently these would default to
1048576. However, after this proposed change if implemented the
"{{consumer.max.partition.fetch.bytes}}" or
"{{producer.max.partition.fetch.bytes}}" values would be the inherited value of
262144, not the default 1048576.
In short, implementing this change would break backward compatibility. We could
implement a configuration switch that controls whether the configurations are
inherited, but this adds complexity to the already-complex configuration
mechanism.
> Worker's producer and consumer configs should inherit from worker configs
> -------------------------------------------------------------------------
>
> Key: KAFKA-6387
> URL: https://issues.apache.org/jira/browse/KAFKA-6387
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Affects Versions: 1.0.0
> Reporter: Randall Hauch
> Labels: needs-kip
>
> Currently, the worker configuration file defines the connection properties
> for the three separate types of connections being made to the Kafka cluster:
> # the worker group membership,
> # producers for source connectors,
> # the consumers for sink connectors.
> The configs are namespaced because to properly support things like
> interceptors where the configs for 2 and 3 would conflict (same config name,
> different value).
> However, it would be beneficial when such control is not required for the
> producers and consumers to inherit the top-level configurations yet be able
> to override them with the {{producer.}} and {{consumer.}} namespaced
> configurations. This way the producer- and consumer-specific configurations
> need only be specified if/when they need to override the top-level
> configurations. This may be necessary, for example, to have different ACLs
> than the connector tasks compared to the producers and consumers.
> This will require a minimal KIP to explain the new behavior.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
