Andrew, one question: you are relying on auto topic creation for the producer and that's why you can't just disable it?
On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma <ism...@juma.me.uk> wrote: > Thanks for sharing Andrew! > > Ismael > > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <o...@wikimedia.org> wrote: > >> We recently had a pretty serious Kafka outage >> < >> https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary >> > >> caused by a bug in one of our consumers that caused it to create new >> topics >> in an infinite loop AKA a topic bomb! Having consumers restricted from >> creating topics would have prevented this for us. >> >> On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ism...@juma.me.uk> wrote: >> >> > Generally, I think positive configs (`allow` instead of `suppress`) are >> > easier to understand. >> > >> > Ismael >> > >> > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <matth...@confluent.io >> > >> > wrote: >> > >> > > Thanks for the summary! >> > > >> > > We might want to add a diagram/table to the docs when we add this >> > > feature (with whatever config name we choose) to explain how broker >> > > config `auto.create.topics.enable` and the consumer config work >> together. >> > > >> > > I think both options are equally easy to understand. "allow" means >> > > follow the broker config, while "suppress" implies ignore the broker >> > > config and don't auto-create. >> > > >> > > >> > > -Matthias >> > > >> > > >> > > On 8/22/18 10:36 PM, Dhruvil Shah wrote: >> > > > *"suppress" is the opposite of "allow", so >> > > > setting suppress.auto.create.topics=false would mean that we do >> _not_ >> > > allow >> > > > auto topic creation; when set to true, the server configuration will >> > > > determine whether we allow automatic creation or not.* >> > > > >> > > > Sorry, I meant suppress.auto.create.topics=true above to disallow >> auto >> > > > topic creation. >> > > > >> > > > >> > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <dhru...@confluent.io >> > >> > > wrote: >> > > > >> > > >> To be clear, we will allow auto topic creation only when server >> config >> > > >> auto.create.topics.enable=true and consumer config >> > > >> allow.auto.create.topics=true; when either is false, we would not >> > create >> > > >> the topic if it does not exist. >> > > >> >> > > >> "suppress" is the opposite of "allow", so >> > > >> setting suppress.auto.create.topics=false would mean that we do >> _not_ >> > > allow >> > > >> auto topic creation; when set to true, the server configuration >> will >> > > >> determine whether we allow automatic creation or not. >> > > >> >> > > >> I think "allow" is easier to understand but I am open to >> suggestions. >> > > >> >> > > >> - Dhruvil >> > > >> >> > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner < >> > > >> brandon.kirch...@gmail.com> wrote: >> > > >> >> > > >>> “allow=false” seems a bit more intuitive to me than >> “suppress=false” >> > > >>> >> > > >>> Brandon >> > > >>> >> > > >>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > > >>>> >> > > >>>> We may also consider : >> > > >>>> >> > > >>>> "suppress.auto.topic.creation" >> > > >>>> >> > > >>>> or >> > > >>>> >> > > >>>> "allow.auto.topic.creation" >> > > >>>> >> > > >>>> w.r.t. suppress or allow, I don't have strong opinion either. >> It's >> > > just >> > > >>> a >> > > >>>> matter of choosing the proper default value. >> > > >>>> >> > > >>>> Cheers >> > > >>>> >> > > >>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah < >> dhru...@confluent.io >> > > >> > > >>> wrote: >> > > >>>>> >> > > >>>>> Hi Matthias, >> > > >>>>> >> > > >>>>> Do you mean something like "suppress.auto.create.topic"? I am >> > leaning >> > > >>> a bit >> > > >>>>> towards "allow.auto.create.topics" but I don't have a strong >> > > preference >> > > >>>>> either. Let's wait to hear if anyone else has an opinion on >> this. >> > > >>>>> >> > > >>>>> Thanks, >> > > >>>>> Dhruvil >> > > >>>>> >> > > >>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax < >> > > matth...@confluent.io >> > > >>>> >> > > >>>>> wrote: >> > > >>>>> >> > > >>>>>> Thanks for the KIP Dhruvil! >> > > >>>>>> >> > > >>>>>> I agree with Jason's comment. An alternative might be to use >> > > >>> "suppress" >> > > >>>>>> what would revert the logic of "allow". Not sure which one is >> more >> > > >>>>>> intuitive and I am fine with both (no personal preference). >> Just >> > > >>> wanted >> > > >>>>>> to mention it as an alternative. >> > > >>>>>> >> > > >>>>>> Don't have any further comments/question so far. >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> -Matthias >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote: >> > > >>>>>>> Hey Dhruvil, >> > > >>>>>>> >> > > >>>>>>> I would suggest using the verb "allow" rather than "enable. >> The >> > > >>>>> consumer >> > > >>>>>>> cannot enable auto topic creation because it is configured on >> the >> > > >>>>> broker. >> > > >>>>>>> All it can do is prevent it from happening if it is enabled. >> > > >>>>>>> >> > > >>>>>>> -Jason >> > > >>>>>>> >> > > >>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah < >> > > dhru...@confluent.io> >> > > >>>>>> wrote: >> > > >>>>>>> >> > > >>>>>>>> Hi, >> > > >>>>>>>> >> > > >>>>>>>> I would like to start discussion on KIP-361 that proposes we >> > add a >> > > >>>>>> consumer >> > > >>>>>>>> configuration to disable auto topic creation. >> > > >>>>>>>> >> > > >>>>>>>> Link to the KIP: >> > > >>>>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>> >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+ >> > > >>>>>>>> Configuration+to+Disable+Auto+Topic+Creation >> > > >>>>>>>> >> > > >>>>>>>> Suggestions and feedback are welcome! >> > > >>>>>>>> >> > > >>>>>>>> Thanks, >> > > >>>>>>>> Dhruvil >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >>>>> >> > > >>> >> > > >> >> > > > >> > > >> > > >> > >> >
