Sorry for late response. The reason that unused configs is in WARN is that if you misspell a config, it means that it will not apply. In some cases (default retention) you want know until too late. We wanted to warn admins about possible misconfigurations.
In the context of a company supporting Kafka - customers run logs at INFO level normally, so if we suspect a misconfiguration, we don't want to ask the customer to change level to DEBUG and bounce the broker. It is time consuming and can be risky. *Gwen Shapira* Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter ( https://twitter.com/ConfluentInc ) | blog ( http://www.confluent.io/blog ) Sent via Superhuman ( https://sprh.mn/?vip=g...@confluent.io ) On Mon, Jan 06, 2020 at 4:21 AM, Stanislav Kozlovski < stanis...@confluent.io > wrote: > > > > Hey Artur, > > > > Perhaps changing the log level to DEBUG is the simplest approach. > > > > I wonder if other people know what the motivation behind the WARN log was? > I'm struggling to think up of a scenario where I'd like to see unused > values printed in anything above DEBUG. > > > > Best, > Stanislav > > > > On Mon, Dec 30, 2019 at 12:52 PM Artur Burtsev < artjock@ gmail. com ( > artj...@gmail.com ) > wrote: > > >> >> >> Hi, >> >> >> >> Indeed changing the log level for the whole AbstractConfig is not an >> option, because logAll is extremely useful. >> >> >> >> Grouping warnings into 1 (with the count of unused only) will not be a >> good option for us either. It will still be pretty noisy. Imagine we have >> 32 partitions and scaled up the application to 32 instances then we still >> have 32 warnings per application (instead of 96 now) while we would like >> to have 0 warnings because we are perfectly aware of using >> schema.registry.url and its totally fine, and we don't have to be warned >> every time we start the application. Now imagine we use more than one >> consumer per application, then it will add another multiplication factor >> to these grouped warnings and we still have a lot of those. So I would say >> grouping doesn't help much. >> >> >> >> I think adding extra logger like >> "org.apache.kafka.clients.producer.ProducerConfig.unused" could be another >> good option. That would leave the existing interface untouched and give >> everyone an option to mute irrelevant warnings. >> >> >> >> To summarize, I still can see 3 options with its pros and cons discussed >> in the thread: >> 1) extra config with interface to handle unused >> 2) change unused warn to debug >> 3) add extra logger for unused >> >> >> >> Please let me know what do you think. >> >> >> >> Thanks, >> Artur >> >> >> >> On Mon, Dec 30, 2019 at 11:07 AM Stanislav Kozlovski >> < stanislav@ confluent. io ( stanis...@confluent.io ) > wrote: >> >> >>> >>> >>> Hi all, >>> >>> >>> >>> Would printing all the unused configurations in one line, versus N lines, >>> be more helpful? I know that it would greatly reduce the verbosity in log >>> visualization tools like Kibana while still allowing us to see all the >>> relevant information without the need for an explicit action (e.g changing >>> the log level) >>> >>> >>> >>> Best, >>> Stanislav >>> >>> >>> >>> On Sat, Dec 28, 2019 at 3:13 PM John Roesler < vvcephei@ apache. org ( >>> vvcep...@apache.org ) > >>> >>> >> >> >> >> wrote: >> >> >>> >>>> >>>> >>>> Hi Artur, >>>> >>>> >>>> >>>> That’s a good point. >>>> >>>> >>>> >>>> One thing you can do is log a summary at WARN level, like “27 >>>> configurations were ignored. Ignored configurations are logged at DEBUG >>>> level.” >>>> >>>> >>>> >>>> I looked into the code a little, and these log messages are generated >>>> >>>> >>> >>> >> >> >> >> in >> >> >>> >>>> >>>> >>>> AbstractConfig (logAll and logUnused). They both use the logger >>>> >>>> >>> >>> >> >> >> >> associated >> >> >>> >>>> >>>> >>>> with the relevant config class (StreamsConfig, ProducerConfig, etc.). >>>> >>>> >>> >>> >> >> >> >> The >> >> >>> >>>> >>>> >>>> list of all configs is logged at INFO level, and the list of unused >>>> >>>> >>> >>> >> >> >> >> configs >> >> >>> >>>> >>>> >>>> is logged at WARN level. This means that it's not possible to silence >>>> >>>> >>> >>> >> >> >> >> the >> >> >>> >>>> >>>> >>>> unused config messages while still logging the list of all configs. You >>>> could only silence both by setting (for example) ProducerConfig logger >>>> >>>> >>> >>> >> >> >> >> to >> >> >>> >>>> >>>> >>>> ERROR or OFF. >>>> >>>> >>>> >>>> If it's desirable to be able to toggle them independently, then you can >>>> create a separate logger for unused configs, named something like >>>> "org.apache.kafka.clients.producer.ProducerConfig.unused". Then, you >>>> >>>> >>> >>> >> >> >> >> can >> >> >>> >>>> >>>> >>>> leave the log at WARN, so it would continue to be printed by default, >>>> >>>> >>> >>> >> >> >> >> and >> >> >>> >>>> >>>> >>>> anyone could disable it by setting >>>> "org.apache.kafka.clients.producer.ProducerConfig.unused" to ERROR or >>>> >>>> >>> >>> >> >> >> >> OFF, >> >> >>> >>>> >>>> >>>> without disturbing the rest of the config log messages. >>>> >>>> >>>> >>>> It's simpler without the extra logger, but you also get less control. >>>> >>>> >>> >>> >> >> >> >> Do >> >> >>> >>>> >>>> >>>> you think the extra control is necessary, versus printing a summary at >>>> >>>> >>> >>> >> >> >> >> WARN >> >> >>> >>>> >>>> >>>> level? >>>> -John >>>> >>>> >>>> >>>> On Fri, Dec 27, 2019, at 04:26, Artur Burtsev wrote: >>>> >>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> Indeed changing log level to debug would be the easiest and I think that >>>>> would be a good solution. When no one object I'm ready to move forward >>>>> with this approach and submit a MR. >>>>> >>>>> >>>>> >>>>> The only minor thing I have – having it at debug log level might make it a >>>>> bit less friendly for developers, especially for those who just do the >>>>> first steps in Kafka. For example, if you misspelled the property name and >>>>> trying to understand why things don't do what you expect. Having a warning >>>>> might save some time in this case. Other >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> than >> >> >>> >>>> >>>>> >>>>> >>>>> that I cannot see any reasons to have warnings there. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> Artur >>>>> >>>>> >>>>> >>>>> On Thu, Dec 26, 2019 at 10:01 PM John Roesler < vvcephei@ apache. org ( >>>>> vvcep...@apache.org ) > >>>>> >>>>> >>>> >>>> >>>> >>>> wrote: >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> Thanks for the KIP, Artur! >>>>>> >>>>>> >>>>>> >>>>>> For reference, here is the kip: >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ >> KIP-552%3A+Add+interface+to+handle+unused+config >> ( >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-552%3A+Add+interface+to+handle+unused+config >> ) >> >> >>> >>>> >>>>> >>>>>> >>>>>> >>>>>> I agree, these warnings are kind of a nuisance. Would it be >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> feasible >> >> >>> >>>> >>>> >>>> just to leverage log4j in some way to make it easy to filter these >>>> messages? For example, we could move those warnings to debug level, or >>>> >>>> >>> >>> >> >> >> >> even >> >> >>> >>>> >>>> >>>> use a separate logger for them. >>>> >>>> >>>>> >>>>>> >>>>>> >>>>>> Thanks for starting the discussion. >>>>>> -John >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Dec 24, 2019, at 07:23, Artur Burtsev wrote: >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> This KIP provides a way to deal with a warning "The >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> >> >> configuration {} >> >> >>> >>>> >>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> was supplied but isn't a known config." when it is not relevant. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Cheers, >>>>>>> Artur >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Best, >>> Stanislav >>> >>> >> >> > > > > -- > Best, > Stanislav > > >
