Thanks for sharing Andrew! Ismael
On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto <o...@wikimedia.org> wrote: > We recently had a pretty serious Kafka outage > < > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary > > > caused by a bug in one of our consumers that caused it to create new topics > in an infinite loop AKA a topic bomb! Having consumers restricted from > creating topics would have prevented this for us. > > On Thu, Aug 23, 2018 at 4:27 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > Generally, I think positive configs (`allow` instead of `suppress`) are > > easier to understand. > > > > Ismael > > > > On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax <matth...@confluent.io> > > wrote: > > > > > Thanks for the summary! > > > > > > We might want to add a diagram/table to the docs when we add this > > > feature (with whatever config name we choose) to explain how broker > > > config `auto.create.topics.enable` and the consumer config work > together. > > > > > > I think both options are equally easy to understand. "allow" means > > > follow the broker config, while "suppress" implies ignore the broker > > > config and don't auto-create. > > > > > > > > > -Matthias > > > > > > > > > On 8/22/18 10:36 PM, Dhruvil Shah wrote: > > > > *"suppress" is the opposite of "allow", so > > > > setting suppress.auto.create.topics=false would mean that we do _not_ > > > allow > > > > auto topic creation; when set to true, the server configuration will > > > > determine whether we allow automatic creation or not.* > > > > > > > > Sorry, I meant suppress.auto.create.topics=true above to disallow > auto > > > > topic creation. > > > > > > > > > > > > On Wed, Aug 22, 2018 at 10:34 PM Dhruvil Shah <dhru...@confluent.io> > > > wrote: > > > > > > > >> To be clear, we will allow auto topic creation only when server > config > > > >> auto.create.topics.enable=true and consumer config > > > >> allow.auto.create.topics=true; when either is false, we would not > > create > > > >> the topic if it does not exist. > > > >> > > > >> "suppress" is the opposite of "allow", so > > > >> setting suppress.auto.create.topics=false would mean that we do > _not_ > > > allow > > > >> auto topic creation; when set to true, the server configuration will > > > >> determine whether we allow automatic creation or not. > > > >> > > > >> I think "allow" is easier to understand but I am open to > suggestions. > > > >> > > > >> - Dhruvil > > > >> > > > >> On Wed, Aug 22, 2018 at 6:53 PM Brandon Kirchner < > > > >> brandon.kirch...@gmail.com> wrote: > > > >> > > > >>> “allow=false” seems a bit more intuitive to me than > “suppress=false” > > > >>> > > > >>> Brandon > > > >>> > > > >>>> On Aug 22, 2018, at 8:48 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > >>>> > > > >>>> We may also consider : > > > >>>> > > > >>>> "suppress.auto.topic.creation" > > > >>>> > > > >>>> or > > > >>>> > > > >>>> "allow.auto.topic.creation" > > > >>>> > > > >>>> w.r.t. suppress or allow, I don't have strong opinion either. It's > > > just > > > >>> a > > > >>>> matter of choosing the proper default value. > > > >>>> > > > >>>> Cheers > > > >>>> > > > >>>>> On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah < > dhru...@confluent.io > > > > > > >>> wrote: > > > >>>>> > > > >>>>> Hi Matthias, > > > >>>>> > > > >>>>> Do you mean something like "suppress.auto.create.topic"? I am > > leaning > > > >>> a bit > > > >>>>> towards "allow.auto.create.topics" but I don't have a strong > > > preference > > > >>>>> either. Let's wait to hear if anyone else has an opinion on this. > > > >>>>> > > > >>>>> Thanks, > > > >>>>> Dhruvil > > > >>>>> > > > >>>>> On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax < > > > matth...@confluent.io > > > >>>> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Thanks for the KIP Dhruvil! > > > >>>>>> > > > >>>>>> I agree with Jason's comment. An alternative might be to use > > > >>> "suppress" > > > >>>>>> what would revert the logic of "allow". Not sure which one is > more > > > >>>>>> intuitive and I am fine with both (no personal preference). Just > > > >>> wanted > > > >>>>>> to mention it as an alternative. > > > >>>>>> > > > >>>>>> Don't have any further comments/question so far. > > > >>>>>> > > > >>>>>> > > > >>>>>> -Matthias > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>>> On 8/21/18 4:42 PM, Jason Gustafson wrote: > > > >>>>>>> Hey Dhruvil, > > > >>>>>>> > > > >>>>>>> I would suggest using the verb "allow" rather than "enable. The > > > >>>>> consumer > > > >>>>>>> cannot enable auto topic creation because it is configured on > the > > > >>>>> broker. > > > >>>>>>> All it can do is prevent it from happening if it is enabled. > > > >>>>>>> > > > >>>>>>> -Jason > > > >>>>>>> > > > >>>>>>> On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah < > > > dhru...@confluent.io> > > > >>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> Hi, > > > >>>>>>>> > > > >>>>>>>> I would like to start discussion on KIP-361 that proposes we > > add a > > > >>>>>> consumer > > > >>>>>>>> configuration to disable auto topic creation. > > > >>>>>>>> > > > >>>>>>>> Link to the KIP: > > > >>>>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+ > > > >>>>>>>> Configuration+to+Disable+Auto+Topic+Creation > > > >>>>>>>> > > > >>>>>>>> Suggestions and feedback are welcome! > > > >>>>>>>> > > > >>>>>>>> Thanks, > > > >>>>>>>> Dhruvil > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>> > > > >> > > > > > > > > > > > > >
