[
https://issues.apache.org/jira/browse/KAFKA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507768#comment-17507768
]
Chris Egerton edited comment on KAFKA-7509 at 3/16/22, 5:41 PM:
----------------------------------------------------------------
Bumping this again as the problem continues to persist for Connect (and
possibly other projects).
While I agree that there's value in logging warnings about unrecognized values
in some contexts, that value is lost when the false positive rate for these
warnings becomes too high, which appears to be the case for Connect. As
[~rhauch] has noted with his exhaustive research into this problem space, it's
basically impossible at the moment for Connect to preemptively filter out
irrelevant values for some of the Kafka clients that it creates. And, as
[~mjsax] has noted, it's not an option to unconditionally demote these log
messages to {{DEBUG}} level for all Kafka clients, since there are plenty of
projects that do not generate these kinds of false positives, in which case
these messages do serve as effective warnings that there may be a typo in the
client's config.
One compromise could be to do the following:
# Add a new {{AbstractConfig::logUnused}} variant that operates at {{DEBUG}}
level instead of {{WARN}} (exact details negotiable; this could be an
overloading of the existing {{logUnused}} method that accepts a boolean or some
other parameter to toggle the log level used, or a separate {{debugUnused}}
method that's hardcoded to use {{{}DEBUG{}}}-level logging, or something else
entirely).
# Add a new {{warn.on.unused}} boolean property to the producer, consumer, and
admin client API, with a default of {{{}true{}}}. When enabled, the current
behavior of logging unrecognized values at {{WARN}} level will take place. When
disabled (i.e., set to {{{}false{}}}), unrecognized values will be instead
logged at {{DEBUG}} level, using the new {{AbstractConfig}} API from part 1.
# Modify Connect to set {{warn.on.unused}} to {{false}} by default for all
Kafka clients that it uses for its internal topics (and possibly in other
contexts where the rate of false positives is also high). Allow users to
override this default by explicitly specifying a value for the property in the
Connect worker config.
This would be completely backwards compatible outside of Connect but also allow
for other projects to opt in to a reduction in {{{}WARN{}}}-level log spam, and
within Connect, would greatly reduce the false positive rate for
{{{}WARN{}}}-level messages.
Obviously, it would require a KIP, but I'm happy to write one if this seems
like a reasonable high-level approach. Thoughts?
was (Author: chrisegerton):
Bumping this again as the problem continues to persist for Connect (and
possibly other projects).
While I agree that there's value in logging warnings about unrecognized values
in some contexts, that value is lost when the false positive rate for these
warnings becomes too high, which appears to be the case for Connect. As
[~rhauch] has noted with his exhaustive research into this problem space, it's
basically impossible at the moment for Connect to preemptively filter out
irrelevant values for some of the Kafka clients that it creates. And, as
[~mjsax] has noted, it's not an option to unconditionally demote these log
messages to {{DEBUG}} level for all Kafka clients, since there are plenty of
projects that do not generate these kinds of false positives, in which case
these messages do serve as effective warnings that there may be a typo in the
client's config.
One compromise could be to do the following:
# Add a new {{AbstractConfig::logUnused}} variant that operates at {{DEBUG}}
level instead of {{WARN}} (exact details negotiable; this could be an
overloading of the existing {{logUnused}} method that accepts a boolean or some
other parameter to toggle the log level used, or a separate {{debugUnused}}
method that's hardcoded to use {{{}DEBUG{}}}-level logging, or something else
entirely).
# Add a new {{warn.on.unused}} boolean property to the producer, consumer, and
admin client API, with a default of {{{}true{}}}. When enabled, the current
behavior of logging unrecognized values at {{WARN}} level will take place. When
disabled (i.e., set to {{{}false{}}}), unrecognized values will be instead
logged at {{DEBUG}} level, using the new {{AbstractConfig}} API from part 1.
# Modify Connect to set {{warn.on.unused}} to {{false}} by default for all
Kafka clients that it uses for its internal topics (and possibly in other
contexts where the rate of false positives is also high). Allow users to
override this default by explicitly specifying a value for the property in the
Connect worker config.
This would be completely backwards compatible outside of Connect but also allow
for other projects to opt in to a reduction in {{{}WARN{}}}-level log spam, and
within Connect, would greatly reduce the false positive rate for
{{{}WARN{}}}-level messages.
Obviously, it would require a KIP, but I'm happy to write one if this seems
like a reasonable high-level approach. Thoughts?
> Kafka Connect logs unnecessary warnings about unused configurations
> -------------------------------------------------------------------
>
> Key: KAFKA-7509
> URL: https://issues.apache.org/jira/browse/KAFKA-7509
> Project: Kafka
> Issue Type: Improvement
> Components: clients, KafkaConnect
> Affects Versions: 0.10.2.0
> Reporter: Randall Hauch
> Assignee: Randall Hauch
> Priority: Major
>
> When running Connect, the logs contain quite a few warnings about "The
> configuration '{}' was supplied but isn't a known config." This occurs when
> Connect creates producers, consumers, and admin clients, because the
> AbstractConfig is logging unused configuration properties upon construction.
> It's complicated by the fact that the Producer, Consumer, and AdminClient all
> create their own AbstractConfig instances within the constructor, so we can't
> even call its {{ignore(String key)}} method.
> See also KAFKA-6793 for a similar issue with Streams.
> There are no arguments in the Producer, Consumer, or AdminClient constructors
> to control whether the configs log these warnings, so a simpler workaround
> is to only pass those configuration properties to the Producer, Consumer, and
> AdminClient that the ProducerConfig, ConsumerConfig, and AdminClientConfig
> configdefs know about.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
