+1 to Matthias' proposal.
First reduce the false positives, then improve the various clients.
best regards
Patrik
On Thu, 12 Mar 2020 at 07:20, Matthias J. Sax wrote:
> Personally, I think that adding a specify prefix for user configs would
> be the simplest fix. Kafka Streams and Kafka Connect
Personally, I think that adding a specify prefix for user configs would
be the simplest fix. Kafka Streams and Kafka Connect could just use this
prefix to avoid the false positive WARN logs.
Hence, the KIP does not need to fix every false positive warning, it
just changes the AbstractConfig about
Long time no see, was on holiday .
Thread got an interesting twist into reporting and fixing every single
warning individually, which is outside of my capacity. As I mentioned
in the KIP and also as highlighted by Patric - you don't have to do
anything extraordinary to get these warnings, you
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
What is the status of this KIP?
- -Matthias
On 2/17/20 2:41 PM, John Roesler wrote:
> Thanks Matthias,
>
> I got the impression this was considered and rejected in
> KAFKA-7509, but I'm not sure why. Maybe it was never really
> considered at all,
Thanks Matthias,
I got the impression this was considered and rejected in KAFKA-7509,
but I'm not sure why. Maybe it was never really considered at all, just
proposed and not-noticed? Perhaps Randall or Sönke can comment.
See:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
I am just getting aware of this KIP (not sure why I missed it).
In Kafka Streams we have nested clients and need to "forward" configs
from outer layer to inner layers -- hence, we prefix some configs to
be able to know which inner nested clients
Thanks Patrik,
This seems to be a long and wandering issue. It seems that KAFKA-7509
has followed a similar trajectory to KAFKA-6793/KIP-552 , and 7509 is
just recently closed in favor of whatever we decide to do in KAFKA-6793.
Considering (what I hope is) the whole history of this issue, a few
Hi John
Regarding Kafka Streams this can probably be fixed easily, but it does not
handle the underlying issue that other custom prefixes are not supported.
Seems I even did a short analysis several months ago and forgot about it,
see
Ah... I've just looked at some integration tests in Streams, and see the same
thing.
I need to apologize to everyone in the thread for my lack of understanding, and
to
thank Gwen for her skepticism. Looking back at the KIP itself, I see that Artur
specifically
listed log messages caused by
Hi John
Starting an empty streams instance
final String bootstrapServers = "broker0:9092";
Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "configDemo");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
Looking at where the log message comes from:
org.apache.kafka.common.config.AbstractConfig#logUnused
it seems like maybe the warning just happens when you pass
extra configs to a client that it has no knowledge of (and therefore
doesn't "use").
I'm now suspicious if Streams is actually sending
Ah, got it! I am indeed curious why they do this :)
Maybe John can shed more light. But if we can't find a better fix, perhaps the
nice thing to do is really a separate logger, so users who are not worried
about shooting themselves in the foot can make those warnings go away.
Gwen Shapira
Hi Gwen
Kafka Streams is not a third party library and produces a lot of these
warnings, e.g.
*The configuration 'main.consumer.max.poll.records' was supplied but isn't
a known config.*
*The configuration 'admin.retries' was supplied but isn't a known config.*
and various others if you try to
INFO is the default log level, and while it looks less "alarming" than WARN,
users will still see it and in my experience, they will worry that something is
wrong anyway. Or if INFO isn't the default, users won't see it, so it is no
different from debug and we are left with no way of warning
Hi John,
In out case it wont help, since we are running instance per partition
and even with summary only we get 32 warnings per rollout.
Hi Gwen,
Thanks for you reply, I understand and share your concern, I also
mentioned it earlier in the thread. Do you think it will work if we
change DEBUG
Sorry for late response. The reason that unused configs is in WARN is that if
you misspell a config, it means that it will not apply. In some cases (default
retention) you want know until too late. We wanted to warn admins about
possible misconfigurations.
In the context of a company
Hi Artur,
Thanks for the update! This looks good to me.
What do you think about also adding a summary message
at WARN level, like:
> WARN 42 unknown configs were ignored and logged at DEBUG level.
That way, we can quit spamming the logs while still preserving the same
level of visibility in
Hi,
I updated KIP
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641934
and submitted PR accordingly to the discussion in this thread
https://github.com/apache/kafka/pull/8043
Please have a look,
Artur
On Mon, Jan 6, 2020 at 1:22 PM Stanislav Kozlovski
wrote:
>
> Hey
Hey Artur,
Perhaps changing the log level to DEBUG is the simplest approach.
I wonder if other people know what the motivation behind the WARN log was?
I'm struggling to think up of a scenario where I'd like to see unused
values printed in anything above DEBUG.
Best,
Stanislav
On Mon, Dec 30,
Hi,
Indeed changing the log level for the whole AbstractConfig is not an
option, because logAll is extremely useful.
Grouping warnings into 1 (with the count of unused only) will not be a
good option for us either. It will still be pretty noisy. Imagine we
have 32 partitions and scaled up the
Hi all,
Would printing all the unused configurations in one line, versus N lines,
be more helpful? I know that it would greatly reduce the verbosity in log
visualization tools like Kibana while still allowing us to see all the
relevant information without the need for an explicit action (e.g
Hi Artur,
That’s a good point.
One thing you can do is log a summary at WARN level, like “27 configurations
were ignored. Ignored configurations are logged at DEBUG level.”
I looked into the code a little, and these log messages are generated in
AbstractConfig (logAll and logUnused). They
Hi,
Indeed changing log level to debug would be the easiest and I think
that would be a good solution. When no one object I'm ready to move
forward with this approach and submit a MR.
The only minor thing I have – having it at debug log level might make
it a bit less friendly for developers,
Thanks for the KIP, Artur!
For reference, here is the kip:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-552%3A+Add+interface+to+handle+unused+config
I agree, these warnings are kind of a nuisance. Would it be feasible just to
leverage log4j in some way to make it easy to filter these
Hi,
This KIP provides a way to deal with a warning "The configuration {}
was supplied but isn't a known config." when it is not relevant.
Cheers,
Artur
25 matches
Mail list logo
