On Thu, Aug 8, 2019, at 11:51, Jun Rao wrote: > Hi, Justine, > > Thanks for the KIP. Overall, it seems to be a good improvement. > > However, I think Harsha's point seems reasonable. We had > auto.create.topics.enable config on the broker to allow admins to disable > topic creation from the producer/consumer clients before we had the > security feature. The need for that config is reduced with the security > feature, but may still be present since not all places have security > enabled. It's true that a non-secured environment is vulnerable to some > additional attacks, but producer/consumer are the most common way for a > user to interact with the broker. So, keeping that config for backward > compatibility could still be useful if it's not introducing too much effort > or extra confusion. > > Here is a one potential alternative way that I was thinking. We add a new > field in the CreateTopicRequest to indicate whether it's from the producer > or not. If auto.create.topics.enable is false, CreateTopicRequest from the > producer will be rejected.
Hi Jun, Thanks for taking a look. One major goal of this KIP is to allow sysadmins to disable server-side topic auto-creation through the metadata request, while still enabling client-side auto-creation for producers. So perhaps we could have three values for "auto.create.topics.enable" on the broker: true, false, and client-side. "client-side" would indicate that auto topic creation through the metadata request would be blocked, but through create topics request would be enabled. > We probably don't need to introduce the new > config (which seems a bit hard to explain) in the producer. Instead, the > new producer always uses MetadataRequest with AllowAutoTopicCreation set to > false to get the metadata and if the metadata is not present, send the > new CreateTopicRequest > (assuming the broker supports it) to try to create the topic automatically. > Whether the creation is allowed or not will be determined by the broker. > This will make the behavior backward compatible and we can still achieve > the main goal of the KIP, which is not relying on MetadataRequest for topic > creation. What do you think? I agree that it may not be necessary to introduce a new configuration for the producer, if we add the server-side access control you suggested. However, new clients will still need to support auto-creation on older brokers, so that producers that rely on it keep working across a client upgrade. That means attempting to do it through MetadataRequest, if the new mechanism is not supported. >From a compatibility point of view, if we rely on a new field in >CreateTopicsRequest, we definitely won't be able to use the >CreateTopicsRequest mechanism against any broker but a 2.4+ one. That's >probably fine (and consistent with some of the other discussion we've been >having here) but it worth noting. best, Colin > > Thanks, > > Jun > > On Thu, Aug 8, 2019 at 1:34 AM M. Manna <manme...@gmail.com> wrote: > > > Hi, > > > > If I may, perhaps you could simplify everything by using only > > 'auto.create.topics.enable' as a value along with true. In other words, the > > public interfaces section should only have [true,auto.create.topics.enable, > > false]. > > > > The reason for this is that auto.create.topics.enable is already known to > > users as a "Server-SIde" config. So all you are saying is > > > > a) To avoid day 1 impact, it will follow whatever auto.create.topics.enable > > value is set. > > b) False means - no client side topic creation > > c) True means client side topic creation. > > > > It saves creating 2 more new strings :). But not too expensive anyway. > > > > Also, when you deprecate auto.create.topics.enable - you must provide > > sufficient logic to ensure that things like rolling upgrade doesn't > > temporarily break anything. I apologise if you have already accounted for > > this, but wanted to mention since I didn't notice this on the KIP. > > > > Let me know how this sounds. > > > > Regards, > > > > On Wed, 7 Aug 2019 at 19:10, Justine Olshan <jols...@confluent.io> wrote: > > > > > Hi Harsha, > > > > > > I think my message may have gotten lost in all the others. > > > > > > Two of the goals of this KIP are to 1) allow auto-creation on specific > > > clients when the broker default is false and 2) eventually replace the > > > broker config. > > > > > > In order to accomplish these two goals, we need the producer to be able > > to > > > create topics despite the broker config. (How can we replace a function > > > when we rely on it?) > > > I think at this point we have a fundamental disagreement in what we > > should > > > allow the producer to do. > > > In my previous message I mentioned a config that would allow for the > > broker > > > to prevent producer auto-creation. (It would be disabled by default.) It > > > would fix your issue for now, but could lead to more complications later. > > > > > > Thank you, > > > Justine > > > > > > > > > On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani <ka...@harsha.io> > > > wrote: > > > > > > > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe <co...@cmccabe.xyz> > > wrote: > > > > > > > > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote: > > > > > > > > > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > > > > > > > > wrote: > > > > > > > > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > > > > > > > > > > Hi Colin, > > > > > "Hmm... I'm not sure I follow. Users don't have to build their own > > > > > tooling, right? They can use any of the shell scripts that we've > > > shipped > > > > in > > > > > the last few releases. For example, if any of your users run it, this > > > > shell > > > > > script will delete all of the topics from your non-security-enabled > > > > > cluster: > > > > > > > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > > > > > --bootstrap-server localhost:9092 --list 2>/dev/null > > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > > > > > --bootstrap-server localhost:9092 --delete > > > > > --topic > > > > > > > > > > They will need to fill in the correct bootstrap servers list, of > > > course, > > > > > not localhost. This deletion script will work on some pretty old > > > brokers, > > > > > even back to the 0.10 releases. It seems a little odd to trust your > > > users > > > > > with this power, but not trust them to avoid changing a particular > > > > > configuration key." > > > > > > > > > > The above will blocked by the server if we set delete.topic.enable to > > > > > false and thats exactly what I am asking for. > > > > > > > > > > Hi Harsha, > > > > > > > > > > I was wondering if someone was going to bring up that configuration > > :) > > > > > > > > > > it's an interesting complication, but globally disabling topic > > deletion > > > > is > > > > > not very practical for most use-cases. > > > > > > > > > > In any case, there are plenty of other bad things that users with > > full > > > > > permissions can do that aren't blocked by any server configuration. > > For > > > > > example, they can delete every record in every topic. I can write a > > > > script > > > > > for that too, and there's no server configuration you can set to > > > disable > > > > > it. Or I could simply create hundreds of thousands of topics, until > > > > cluster > > > > > performance becomes unacceptable (this will be even more of a problem > > > if > > > > > someone configured delete.topic.enable as false). Or publish bad data > > > to > > > > > every topic, etc. etc. > > > > > > > > > > The point I'm trying to make here is that you can't rely on these > > kind > > > of > > > > > server-side configurations for security. At most, they're a way to > > set > > > up > > > > > certain very simple policies. But the policies are so simple that > > > they're > > > > > hardly ever useful any more. > > > > > > > > > > For example, if the problem you want to solve is that you want a user > > > to > > > > > only be able to create 50 topics and not delete anyone else's topics, > > > you > > > > > can solve that with a CreateTopicsPolicy that limits the number of > > > > topics, > > > > > and some ACLs. There's no combination of auto.create.topics.enable > > and > > > > > delete.topic.enable that will help here. > > > > > > > > > > Hi Colin, > > > > > > > > > > Well you gave the example that a user can delete the > > topics > > > > > just by running that script :). > > > > > > > > > > I understand there are open APIs in Kafka and can lead to rogue > > clients > > > > > taking advantage of it without proper security in place. > > > > > > > > > > What I am asking so far in this thread is , this KIP is changing the > > > > > producer behavior and its not backward compatible. > > > > > > > > > > The change is backwards compatible. The default will still be > > > server-side > > > > > topic auto-creation, just like now. > > > > > > > > > You will have to specifically change the producer config to get the new > > > > > behavior. > > > > > > > > > > > > > > > > > I disagree. Today server can turn off the topic creation neither > > > producer > > > > or consumer can create a topic. With this KIP , producer can create a > > > topic > > > > by turning on client side config when server side config is turned off. > > > > > > > > > > > > We can still achieve > > > > > the main goal of the KIP which is to change MetadataRequest creating > > > > > topics and send a CreateTopicRequest from Producer and also keep the > > > > server > > > > > side config to have precedence. This KIP originally written to have > > > > server > > > > > side preference and there is not much explanation why it changed to > > > have > > > > > Producer side preference. > > > > > > > > > > Arguing that AdminClient can do that and so we are going to make > > > Producer > > > > > do the same doesn't make sense. > > > > > > > > > > "The downside is that if we wanted to check a server side > > configuration > > > > > before sending the create topics request, the code would be more > > > complex. > > > > > The behavior would also not be consistent with how topic > > auto-creation > > > is > > > > > handled in Kafka Streams." > > > > > > > > > > I am not sure why you need to check server side configuration before > > > > > sending create topics request. A user enables producer side config to > > > > > create topics. > > > > > Producer sends a request to the broker and if the broker has > > > > > auto.topic.create.enable to true (default) it will allow creation of > > > > > topics. If it set to false it returns error back to the client. > > > > > > > > > > auto.topic.create.enable has never affected CreateTopicsRequest. If > > you > > > > > submit a CreateTopicsRequest and you are authorized, a topic will be > > > > > created, regardless of what the value of auto.topic.create.enable is. > > > > This > > > > > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?) > > > > > > > > > > I don't see how this behavior will be different in Kafka streams. By > > > > > default server allows the topic creation and with this KIP, It will > > > only > > > > > allow creation of topic when both producer and server side are turned > > > on. > > > > > Its exactly the same behavior in KIP-361. > > > > > > > > > > I suggest running a streams job as a test. It creates the topics it > > > needs > > > > > using AdminClient. The value of auto.topic.create.enable is not > > > relevant. > > > > > Whether it is true or false, the topics will still be created. > > > > > > > > > > "In general, it would be nice if we could keep the security and > > access > > > > > control model simple and not introduce a lot of special cases and > > > > > exceptions. Kafka has basically converged on using ACLs and > > > > > CreateTopicPolicy classes to control who can create topics and where. > > > > > Adding more knobs seems like a step backwards, especially when the > > > > proposed > > > > > knobs don't work consistently across components, and don't provide > > true > > > > > security." This is not about access control at all. Shipping sane > > > > defaults > > > > > should be prioritized. > > > > > > > > > > I don't think the default is really the question here. I think > > everyone > > > > > agrees that the default for client-side auto-topic creation should be > > > > off. > > > > > > > > > > My point goes back to KIP-361 why we kept the server side config to > > > have > > > > > preference. We can keep bringing back the discussion of security > > > policies > > > > > but a producer client overiding server side setting is not just > > > security > > > > > policy issue . > > > > > > > > > > "Producer client can override server side setting and not consumer > > and > > > > > also producer can only override creation of topic but no the > > > replication > > > > > and partitions. " This doesn't make sense to me and why we want to > > > > > introduce such a confusing behavior. > > > > > > > > > > The scenarios that you're describing (such as dealing with a poorly > > > > > configured client that tries to create topics it should not) do seem > > to > > > > be > > > > > about access control. > > > > > > > > > > We keep talking about CreateTopicPolicy and yet we don't have default > > > one > > > > > and asking every user of Kafka implement their own doesn't make sense > > > > here. > > > > > > > > > > The point of CreateTopicPolicy is exactly that you can implement your > > > > own, > > > > > though. It's a way of customizing what the policy is in your specific > > > > > cluster. > > > > > > > > > > I agree that most people don't want to write a custom policy. But > > most > > > of > > > > > those people are satisfied with just setting up ACLs to set up simple > > > > > policies like this user can create topics, that user can't, etc. > > That's > > > > why > > > > > I suggested adding support for ACLs to insecure clusters might be > > > > helpful. > > > > > > > > > > Also, just as a side note, wouldn't it would be impossible for us to > > > > > specify a default CreateTopicsPolicy that did anything at this point > > > > anyway > > > > > without breaking backwards compatibility? Maybe I'm misunderstanding > > > the > > > > > proposal. > > > > > > > > > > I am asking about exact behavior that we shipped for consumer side > > > > https:/ > > > > > / cwiki. apache. org/ confluence/ display/ KAFKA/ > > > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation > > > > > ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation > > > > > ) > > > > > > > > > > I still didn't get any response why this behavior shouldn't be > > exactly > > > > > like Kafka consumer and why do we want to have producer to overrider > > > > server > > > > > side config and while not allowing consumer to do so. We are not even > > > > > allowing the same contract and this will cause more confusion from > > the > > > > > users standpoint. > > > > > > > > > > Hmm. The consumer already has a way to override the server side > > > > > configuration. See KIP-361: Add Consumer Configuration to Disable > > Auto > > > > > Topic Creation. This lets the consumer skip auto-creating topics, > > even > > > if > > > > > auto-creation is enabled on the broker. > > > > > > > > > > Yes that's what I am saying and it doesn't allow topic creation if > > the > > > > > server side auto-creation is disabled and in consumer its enabled. > > I > > > > > would like to see the exact behavior for producer as stated in > > KIP-361. > > > > > > > > > > To be fair, the KIP should probably discuss why we don't want to > > > > implement > > > > > client-side topic creation in the consumer in "rejected > > alternatives." > > > > > Maybe Justine can add more context here in the KIP. > > > > > > > > > > The last time we talked about this, the reasoning was that we wanted > > to > > > > > eventually get rid of consumer-side auto-topic creation entirely, and > > > so > > > > it > > > > > wasn't worth implementing an improved way of doing it. Producer side > > > auto > > > > > topic-creation, in contrast, will probably stick around, although > > we'd > > > > like > > > > > to deprecate and remove the problematic server-side mechanism over > > > time. > > > > > > > > > > We should implement the producer side topic creation with proper > > create > > > > > topic request and I am in favor of this KIP. > > > > > > > > > > All I am asking is don't make the producer to override server side > > > config > > > > > and keep it similar to KIP-361 just like consumer, it honors what set > > > on > > > > > server side which takes the precedence. Changing this behavior in > > > > > producer is backward incompatible and will catch users by surprise. > > > > > > > > > > All arguments for enforcing producer side overriding config can still > > > be > > > > > achieved. Server side auto creation turned on by default and any new > > > > > producer client setting their auto creation config to true will > > create > > > > > topics in default behavior and any user who set server side to false > > > and > > > > > upgrades to latest kafka with this KIP will not see any unwanted > > > behavior > > > > > from clients. IMO this is not a unreasonable request on this KIP and > > > > this > > > > > requested behavior is exactly what the KIP initially proposed. > > > > > > > > > > Thanks, > > > > > > > > > > Harsha > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe < cmccabe@ apache. org ( > > > > > cmcc...@apache.org ) > wrote: > > > > > > > > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote: > > > > > > > > > > Not sure how the AdminClient applies here, It is an external API and > > is > > > > > not part of KafkaProducer so any user who updates to latest version > > of > > > > > Kafka with this feature get to create the topics. > > > > > They have to build a tooling around AdminClient allow themselves to > > > > > > > > > > create > > > > > > > > > > topics. > > > > > > > > > > Hi Harsha, > > > > > > > > > > Hmm... I'm not sure I follow. Users don't have to build their own > > > > tooling, > > > > > right? They can use any of the shell scripts that we've shipped in > > the > > > > last > > > > > few releases. For example, if any of your users run it, this shell > > > script > > > > > will delete all of the topics from your non-security-enabled cluster: > > > > > > > > > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > > > > > --bootstrap-server localhost:9092 --list 2>/dev/null > > > > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh ) > > > > > --bootstrap-server localhost:9092 --delete > > > > > --topic > > > > > > > > > > They will need to fill in the correct bootstrap servers list, of > > > course, > > > > > not localhost. This deletion script will work on some pretty old > > > brokers, > > > > > even back to the 0.10 releases. It seems a little odd to trust your > > > users > > > > > with this power, but not trust them to avoid changing a particular > > > > > configuration key. > > > > > > > > > > There is no behavior in Kafka producer that allowed users to delete > > the > > > > > topics or delete the records. So citing them as an example doesn't > > > makes > > > > > sense in this context. > > > > > > > > > > I think Kafka Streams is relevant here. After all, it's software that > > > we > > > > > ship as part of the official Kafka release. And it auto-creates > > topics > > > > even > > > > > when auto.create.topics.enable is set to false on the broker. > > > > > > > > > > I think that auto.create.topics.enable was never intended as a > > security > > > > > setting to control access. It was intended as a way to disable the > > > > > broker-side auto-create feature specifically, because there were some > > > > > problems with that specific feature. Broker-side auto-creation is > > > > > frustrating because it's on by default, and because it can > > auto-create > > > > > topics even if the producer or consumer didn't explicitly ask for > > that. > > > > > Neither of those problems applies to this KIP: producers have to > > > > > specifically opt in, and it won't be on by default. Basically, we > > think > > > > > that client-side auto-creation is actually a lot better-- hence this > > > KIP > > > > > :) > > > > > > > > > > But there is > > > > > a functionality which allowed creation of topics if they don't exist > > in > > > > > > > > > > the > > > > > > > > > > cluster and this behavior could be controlled by a config on the > > server > > > > > side. Now with this KIP we are allowing producer to make that > > decision > > > > > without any gateway on the server via configs. Any user who is not > > > aware > > > > > > > > > > of > > > > > > > > > > such changes > > > > > can accidentally create these topics and we are essentially removing > > a > > > > > config that exists in brokers today to block this accidental creation > > > and > > > > > allowing clients to take control. > > > > > > > > > > Again, I hope I'm not misinterpreting, but I don't see how this can > > be > > > > > turned on accidentally. The user would have to specifically turn this > > > on > > > > in > > > > > the producer by setting the configuration key. > > > > > > > > > > I still didn't get any positives of not having server side configs? > > if > > > > you > > > > > want to turn them on and allow any client to create topics set the > > > > default > > > > > to true > > > > > and allow users who doesn't want to have this feature let them turn > > > them > > > > > off. It will be the exact behavior as it is today , as far as > > producer > > > is > > > > > concerned. I am not > > > > > understanding why not having server side configs to gateways such a > > > hard > > > > > requirement and this behavior exists today. As far I am concerned > > this > > > > > breaks the backward compatibility. > > > > > > > > > > The downside is that if we wanted to check a server side > > configuration > > > > > before sending the create topics request, the code would be more > > > complex. > > > > > The behavior would also not be consistent with how topic > > auto-creation > > > is > > > > > handled in Kafka Streams. > > > > > > > > > > In general, it would be nice if we could keep the security and access > > > > > control model simple and not introduce a lot of special cases and > > > > > exceptions. Kafka has basically converged on using ACLs and > > > > > CreateTopicPolicy classes to control who can create topics and where. > > > > > Adding more knobs seems like a step backwards, especially when the > > > > proposed > > > > > knobs don't work consistently across components, and don't provide > > true > > > > > security. > > > > > > > > > > Maybe we should support an insecure mode where there are still users > > > and > > > > > ACLs. That would help people who don't want to set up Kerberos, etc. > > > but > > > > > still want to protect against misconfigured clients or accidents. > > > Hadoop > > > > > has something like this, and I think it was useful. NFS also > > supported > > > > > (supports?) a mode where you just pass whatever user ID you want and > > > the > > > > > system believes you. These things clearly don't protect against > > > malicious > > > > > users, but they can help set up policies when needed. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe < cmccabe@ apache. org ( > > > > > cmcc...@apache.org ) > wrote: > > > > > > > > > > Hi Harsha, > > > > > > > > > > Thanks for taking a look. > > > > > > > > > > I'm not sure I follow the discussion about AdminClient. > > > > > > > > > > KafkaAdminClient > > > > > > > > > > has been around for about 2 years at this point as a public class. > > > > > > > > > > There > > > > > > > > > > are many programs that use it to automatically create topics. Kafka > > > > > Streams does this, for example. If any of your users run Kafka > > > > > > > > > > Streams, > > > > > > > > > > they will be auto-creating topics, regardless of what setting you use > > > > > > > > > > for > > > > > > > > > > auto.create.topics.enable. > > > > > > > > > > A big part of the annoyance of auto-topic creation right now is that > > > > > > > > > > it is > > > > > > > > > > on by default. The new configuration proposed by KIP-487 wouldn't be. > > > > > Users would have to explicitly opt in to the new behavior of > > > > > > > > > > client-side > > > > > > > > > > topic creation. If you run without security, you're already putting a > > > > > > > > > > huge > > > > > > > > > > amount of trust in your users. For example, you trust them not to > > > > > > > > > > delete > > > > > > > > > > records with the kafka-delete-records. sh ( > > > http://kafka-delete-records. > > > > > sh/ > > > > > ) command, or delete topics with kafka-topics. sh ( > > > http://kafka-topics. > > > > > sh/ > > > > > ). Trusting them not to set a certain config value seems minor in > > > > > comparison, right? > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote: > > > > > > > > > > Hi, > > > > > Even with policies one needs to implement that, so for every > > > > > > > > > > user who > > > > > > > > > > doesn't want a producer to create topics or have limits around > > > > > > > > > > partitions & > > > > > > > > > > replication factor they have to implement a policy. The KIP is > > changing > > > > > the behavior , it might not be introducing > > > > > > > > > > the > > > > > > > > > > new functionality but it will enable producers to override the create > > > > > > > > > > topic > > > > > > > > > > config settings on the broker. What I am asking for to provide a > > > > > > > > > > config > > > > > > > > > > that will disable auto creation of topics and if its enabled set some > > > > > > > > > > sane > > > > > > > > > > defaults so that clients can create a topic with in those limits. I > > > > > > > > > > don't > > > > > > > > > > see how this not related to this KIP. > > > > > If the server config options as I suggested doesn't interest > > > > > > > > > > you at > > > > > > > > > > least have a default CreateTopicPolices in place. > > > > > To give an example, In our environment we disable the > > > > > auto.create.topic.enable and force users to go through a centralized > > > > > service as we want capture more details about the topic creation and > > > > > requirements. With this KIP, a producer can create a topic with no > > > > > > > > > > bounds. > > > > > > > > > > Another example max.message.size we define that at cluster level > > > > > > > > > > and one > > > > > > > > > > can override max.messsage.size at topic level, any misconfiguration > > > > > > > > > > at > > > > > > > > > > this > > > > > > > > > > will cause service degradation. Its not always about the rogue > > > > > > > > > > clients, > > > > > > > > > > users can easily misconfigure and can cause an outage. Again we can > > > talk > > > > > about CreateTopicPolicy but without having a > > > > > > > > > > default > > > > > > > > > > implementation and asking everyone to implement their own while > > > > > > > > > > changing > > > > > > > > > > the behavior in producer doesn't make sense to me. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Tue, Aug 06 , 2019 at 7:41 AM, Ismael Juma < ismael@ juma. me. > > uk ( > > > > > ism...@juma.me.uk ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Harsha, > > > > > > > > > > I mentioned policies and the authorizer. For example, with > > > > > CreateTopicPolicy, you can implement the limits you describe. If > > > > > > > > > > you > > > > > > > > > > have > > > > > > > > > > ideas of how that should be improved, please submit a KIP. My > > > > > > > > > > point is > > > > > > > > > > that > > > > > > > > > > this KIP is not introducing any new functionality with regards to > > > > > > > > > > what > > > > > > > > > > rogue clients can do. It's using the existing protocol that is > > > > > > > > > > already > > > > > > > > > > exposed via the AdminClient. So, I don't think we need to address > > > > > > > > > > it in > > > > > > > > > > this KIP. Does that make sense? > > > > > > > > > > Ismael > > > > > > > > > > On Tue, Aug 6 , 2019 at 7:13 AM Harsha Chintalapani < > > > > > > > > > > kafka@ harsha. io ( ka...@harsha.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Ismael, > > > > > Sure AdminClient can do that and we should've shipped a config or > > > > > > > > > > use > > > > > > > > > > the > > > > > > > > > > existing one to block that. Not all users are yet to upgrade to > > > > > > > > > > AdminClient > > > > > > > > > > and start using that to cause issues yet. In shared environment we > > > > > > > > > > should > > > > > > > > > > allow server to set sane defaults and not allow every client to go > > > > > > > > > > ahead > > > > > > > > > > create random no.of topic/partitions and replication factor. Even > > > > > > > > > > if > > > > > > > > > > the > > > > > > > > > > users want to allow topic creation proposed in the KIP , it makes > > > > > > > > > > sense to > > > > > > > > > > have some guards against the no.of partitions and replication > > > > > > > > > > factor. > > > > > > > > > > Authorizer is not always an answer to block requests and having > > > > > > > > > > users > > > > > > > > > > set > > > > > > > > > > server side configs to protect a multi-tenant environment is > > > > > > > > > > required. > > > > > > > > > > In a > > > > > > > > > > non-secure environment Authorizer is a blunt instrument either you > > > > > > > > > > end > > > > > > > > > > up > > > > > > > > > > blocking everyone or allowing everyone. > > > > > I am asking to have server side that allow clients to create > > > > > > > > > > topics or > > > > > > > > > > not > > > > > > > > > > , if they are allowed set a ceiling on max no.of partitions and > > > > > replication-factor. > > > > > > > > > > -Harsha > > > > > > > > > > On Mon, Aug 5 2019 at 8:58 PM, < ismaelj@ gmail. com ( > > > isma...@gmail.com > > > > > ) > > > > > > > > > > wrote: > > > > > > > > > > Harsha, > > > > > > > > > > Rogue clients can use the admin client to create topics and > > > > > > > > > > partitions. > > > > > > > > > > ACLs and policies can help in that case as well as this one. > > > > > > > > > > Ismael > > > > > > > > > > On Mon, Aug 5 , 2019, 7:48 PM Harsha Chintalapani < kafka@ harsha. > > io > > > ( > > > > > ka...@harsha.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > Thanks for the KIP. > > > > > "When server-side auto-creation is disabled, client-side > > > > > > > > > > auto-creation > > > > > > > > > > will try to use client-side configurations" > > > > > If I understand correctly, this KIP is removing any server-side > > > > > > > > > > blocking > > > > > > > > > > client auto creation of topic? > > > > > if so this will present potential issue of rogue client creating > > > > > > > > > > ton of > > > > > > > > > > topic-partitions and potentially bringing down the service for > > > > > > > > > > everyone > > > > > > > > > > or > > > > > > > > > > degrade the service itself. > > > > > By reading the KIP its not clear to me that there is a clear way to > > > > > > > > > > block > > > > > > > > > > auto creation topics of all together from clients by server side > > > > > > > > > > config. > > > > > > > > > > Server side configs of default topic, partitions should take higher > > > > > precedence and client shouldn't be able to create a topic with > > > > > > > > > > higher > > > > > > > > > > no.of > > > > > > > > > > partitions, replication than what server config specifies. > > > > > > > > > > Thanks, > > > > > Harsha > > > > > > > > > > On Mon, Aug 05 , 2019 at 5:24 PM, Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi all, > > > > > I made some changes to the KIP. Hopefully this configuration change > > > > > > > > > > will > > > > > > > > > > make things a little clearer. > > > > > > > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( > > > > https://cwiki. > > > > > apache.org/confluence/display/KAFKA/ ) > > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > > > > > > > > > Please let me know if you have any feedback or questions! > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Wed, Jul 31 , 2019 at 1:44 PM Colin McCabe < cmccabe@ apache. > > org ( > > > > > cmcc...@apache.org ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Mickael, > > > > > > > > > > I think you bring up a good point. It would be better if we didn't > > > > > > > > > > ever > > > > > > > > > > have to set up client-side configuration for this feature, and > > > > > > > > > > KIP-464 > > > > > > > > > > would let us skip this entirely. > > > > > > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > > > > > > > > > Hi Mickael, > > > > > I agree that KIP-464 works on newer brokers, but I was a bit > > > > > > > > > > worried > > > > > > > > > > how > > > > > > > > > > things would play out on older brokers that* do not *have KIP 464 > > > > > > > > > > included. > > > > > > > > > > Is it enough to throw an error in this case when producer configs > > > > > > > > > > are > > > > > > > > > > not > > > > > > > > > > specified? > > > > > > > > > > I think the right thing to do would be to log an error message in > > > > > > > > > > the > > > > > > > > > > client. We will need to have that capability in any case, to cover > > > > > scenarios like the client trying to auto-create a topic that they > > > > > > > > > > don't > > > > > > > > > > have permission to create. Or a client trying to create a topic on > > > > > > > > > > a > > > > > > > > > > broker > > > > > > > > > > so old that CreateTopicsRequest is not supported. > > > > > > > > > > The big downside to relying on KIP-464 is that it is a very recent > > > > > > > > > > feature > > > > > > > > > > -- so recent that it hasn't even made its way to any official > > > > > > > > > > Apache > > > > > > > > > > release. It's scheduled for the upcoming 2.4 release in a few > > > > > > > > > > months. > > > > > > > > > > So if you view this KIP as a step towards removing broker-side > > > > > auto-create, you might want to support older brokers just to > > > > > > > > > > accelerate > > > > > > > > > > adoption, and hasten the day when we can finally flip broker-side > > > > > auto-create to off (or even remove it entirely). > > > > > > > > > > I have to agree, though, that having client-side configurations for > > > > > > > > > > number > > > > > > > > > > of partitions and replication factor is messy. Maybe it would be > > > > > > > > > > worth > > > > > > > > > > it > > > > > > > > > > to restrict support to post-KIP-464 brokers, if we could avoid > > > > > > > > > > creating > > > > > > > > > > more configs. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Wed, Jul 31 , 2019 at 9:10 AM Mickael Maison < > > > > > > > > > > mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > We can rely on KIP-464 which allows to omit the partition count or > > > > > replication factor when creating a topic. In that case, the broker > > > > > > > > > > defaults > > > > > > > > > > are used. > > > > > > > > > > On Wed, Jul 31 , 2019 at 4:55 PM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Michael, > > > > > That makes sense to me! > > > > > To clarify, in the current state of the KIP, the producer does not > > > > > > > > > > rely > > > > > > > > > > on > > > > > > > > > > the broker to autocreate--if the broker's config is disabled, then > > > > > > > > > > the > > > > > > > > > > producer can autocreate on its own with a create topic request (the > > > > > > > > > > same > > > > > > > > > > type of request the admin client uses). > > > > > However, if both configs are enabled, the broker will autocreate > > > > > > > > > > through > > > > > > > > > > a > > > > > > > > > > metadata request before the producer gets a chance. Of course, the > > > > > > > > > > way > > > > > > > > > > to > > > > > > > > > > avoid this, is to do as you suggested, and set > > > > > > > > > > the > > > > > > > > > > "allow_auto_topic_creation" field to false. > > > > > > > > > > I think the only thing we need to be careful with in this setup is > > > > > > > > > > without > > > > > > > > > > KIP 464, we can not use broker defaults for this topic. A user > > > > > > > > > > needs > > > > > > > > > > to > > > > > > > > > > specify the number of partition and replication factor in the > > > > > > > > > > config. > > > > > > > > > > An > > > > > > > > > > alternative to this is to have coded defaults for when these > > > > > > > > > > configs > > > > > > > > > > are > > > > > > > > > > unspecified, but it is not immediately apparent what these defaults > > > > > > > > > > should > > > > > > > > > > be. > > > > > > > > > > Thanks again for reading my KIP, > > > > > Justine > > > > > > > > > > On Wed, Jul 31 , 2019 at 4:19 AM Mickael Maison < > > > > > > > > > > mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the response! > > > > > In my opinion, it would be better if the producer did not rely at > > > > > > > > > > all > > > > > > > > > > on the broker auto create feature as this is what we're aiming to > > > > > deprecate. When requesting metadata we can set the > > > > > "allow_auto_topic_creation" field to false to avoid the broker auto > > > > > creation. Then if the topic is not existing, send a > > > > > > > > > > CreateTopicRequest. > > > > > > > > > > What do you think? > > > > > > > > > > On Mon, Jul 29 , 2019 at 6:34 PM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Currently the way it is implemented, the broker auto-creation > > > > > > > > > > configuration > > > > > > > > > > takes precedence. The producer will not use the CreateTopics > > > > > > > > > > request. > > > > > > > > > > (Technically it can--but the topic will already be created > > > > > > > > > > through > > > > > > > > > > the > > > > > > > > > > broker, so it will never try to create the topic.) It is possible > > > > > > > > > > to > > > > > > > > > > change this however, and I'd be happy to > > > > > > > > > > discuss > > > > > > > > > > the > > > > > > > > > > benefits of this alternative. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Mon, Jul 29 , 2019 at 10:26 AM Mickael Maison < > > > > > > > > > > mickael. maison@ gmail. com ( mickael.mai...@gmail.com ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > In case auto creation is enabled on both the client and server, > > > > > > > > > > will > > > > > > > > > > the producer still use the AdminClient (CreateTopics request) > > > > > > > > > > to > > > > > > > > > > create topics? and not rely on the broker auto create. I'm > > > > > > > > > > guessing the > > > > > > > > > > answer is yes but can you make it explicit. > > > > > > > > > > Thank you > > > > > > > > > > On Wed, Jul 24 , 2019 at 6:23 PM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi, > > > > > Just a friendly reminder to take a look at this KIP if you > > > > > > > > > > have > > > > > > > > > > the > > > > > > > > > > time. > > > > > > > > > > I was thinking about broker vs. client default precedence, > > > > > > > > > > and I > > > > > > > > > > think it > > > > > > > > > > makes sense to keep the broker as the default used when both > > > > > > > > > > client-side > > > > > > > > > > and broker-side defaults are configured. The idea is that > > > > > > > > > > there > > > > > > > > > > would be > > > > > > > > > > pretty clear documentation, and that many systems with > > > > > > > > > > configurations > > > > > > > > > > that > > > > > > > > > > the client could not change would likely have the auto-create > > > > > > > > > > default > > > > > > > > > > off. > > > > > > > > > > (In cloud for example). > > > > > > > > > > It also seems like in most cases, the consumer config > > > > > ' allow. auto. create. topics ( http://allow.auto.create.topics/ ) ' > > > was > > > > > created to actually prevent > > > > > > > > > > the > > > > > > > > > > creation > > > > > > > > > > of > > > > > > > > > > topics, so the loss of creation functionality will not be a > > > > > > > > > > big > > > > > > > > > > problem. > > > > > > > > > > I'm happy to discuss any other compatibility problems or > > > > > > > > > > components > > > > > > > > > > of > > > > > > > > > > this KIP. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Wed, Jul 17 , 2019 at 9:11 AM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > wrote: > > > > > > > > > > Hello all, > > > > > > > > > > I was looking at this KIP again, and there is a decision I > > > > > > > > > > made > > > > > > > > > > that I > > > > > > > > > > think is worth discussing. > > > > > > > > > > In the case where both the broker and producer's > > > > > 'auto.create.topics.enable' are set to true, we have to > > > > > > > > > > choose > > > > > > > > > > either > > > > > > > > > > the > > > > > > > > > > broker configs or the producer configs for the replication > > > > > factor/partitions. > > > > > > > > > > Currently, the decision is to have the broker defaults take > > > > > > > > > > precedence. > > > > > > > > > > (It is easier to do this in the implementation.) It also > > > > > > > > > > makes > > > > > > > > > > some > > > > > > > > > > sense > > > > > > > > > > for this behavior to take precedence since this behavior > > > > > > > > > > already > > > > > > > > > > occurs as > > > > > > > > > > the default. > > > > > > > > > > However, I was wondering if it would be odd for those who > > > > > > > > > > can > > > > > > > > > > only > > > > > > > > > > see > > > > > > > > > > the > > > > > > > > > > producer side to set configs for replication > > > > > > > > > > factor/partitions > > > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > different results. Currently the documentation for the > > > > > > > > > > config > > > > > > > > > > states > > > > > > > > > > that > > > > > > > > > > the config values are only used when the broker config is > > > > > > > > > > not > > > > > > > > > > enabled, > > > > > > > > > > but > > > > > > > > > > this might not always be clear to the user. Changing the > > > > > > > > > > code > > > > > > > > > > to > > > > > > > > > > have > > > > > > > > > > the > > > > > > > > > > producer's configurations take precedence is possible, but > > > > > > > > > > I > > > > > > > > > > was > > > > > > > > > > wondering > > > > > > > > > > what everyone thought. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Fri, Jul 12 , 2019 at 2:49 PM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Just a quick update-- > > > > > > > > > > It seems that enabling both the broker and producer > > > > > > > > > > configs > > > > > > > > > > works > > > > > > > > > > fine, > > > > > > > > > > except that the broker configurations for partitions, > > > > > > > > > > replication > > > > > > > > > > factor > > > > > > > > > > take precedence. > > > > > I don't know if that is something we would want to > > > > > > > > > > change, but > > > > > > > > > > I'll be > > > > > > > > > > updating the KIP for now to reflect this. Perhaps we would > > > > > > > > > > want to > > > > > > > > > > add more > > > > > > > > > > to the documentation of the the producer configs to > > > > > > > > > > clarify. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Fri, Jul 12 , 2019 at 9:28 AM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Colin, > > > > > > > > > > Thanks for looking at the KIP. I can definitely add to > > > > > > > > > > the > > > > > > > > > > title > > > > > > > > > > to > > > > > > > > > > make > > > > > > > > > > it more clear. > > > > > > > > > > It makes sense that both configurations could be turned > > > > > > > > > > on > > > > > > > > > > since > > > > > > > > > > there > > > > > > > > > > are many cases where the user can not control the > > > > > > > > > > server-side > > > > > > > > > > configurations. I was a little concerned about how both > > > > > > > > > > interacting > > > > > > > > > > would > > > > > > > > > > work out -- if there would be to many requests for new > > > > > > > > > > topics, > > > > > > > > > > for > > > > > > > > > > example. > > > > > > > > > > But it since it does make sense to allow both > > > > > > > > > > configurations > > > > > > > > > > enabled, I > > > > > > > > > > will test out some scenarios and I'll change the KIP to > > > > > > > > > > support > > > > > > > > > > this. > > > > > > > > > > I also agree with documentation about distinguishing the > > > > > > > > > > differences. I > > > > > > > > > > was having some trouble with the wording but I like the > > > > > > > > > > phrases > > > > > > > > > > "server-side" and "client-side." That's a good > > > > > > > > > > distinction I > > > > > > > > > > can > > > > > > > > > > use > > > > > > > > > > when > > > > > > > > > > describing. > > > > > > > > > > I'll try to update the KIP soon keeping everyone's input > > > > > > > > > > in > > > > > > > > > > mind. > > > > > > > > > > Thanks, > > > > > Justine > > > > > > > > > > On Thu, Jul 11 , 2019 at 5:39 PM Colin McCabe < > > > > > > > > > > cmccabe@ apache. org ( cmcc...@apache.org ) > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP. This seems like a good step towards > > > > > > > > > > removing > > > > > > > > > > server-side topic auto-creation. > > > > > > > > > > We should add included "client-side" to the title of > > > > > > > > > > the KIP > > > > > > > > > > somewhere, > > > > > > > > > > to make it clear that we're talking about client-side > > > > > > > > > > auto > > > > > > > > > > creation. > > > > > > > > > > The KIP says: > > > > > > > > > > In order to automatically create topics with the > > > > > > > > > > producer, the > > > > > > > > > > producer's > > > > > > > > > > auto.create.topics.enable config must be set to true > > > > > > > > > > and > > > > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > config should be set to false > > > > > > > > > > From a user's point of view, this seems > > > > > > > > > > counter-intuitive. > > > > > > > > > > In > > > > > > > > > > order to > > > > > > > > > > auto-create topics the broker's > > > > > > > > > > auto.create.topics.enable > > > > > > > > > > config > > > > > > > > > > should be > > > > > > > > > > set to false? It seems like the server-side > > > > > > > > > > auto-create is > > > > > > > > > > unrelated to > > > > > > > > > > the client-side auto-create. We could have both turned > > > > > > > > > > on > > > > > > > > > > (and > > > > > > > > > > I'm > > > > > > > > > > sure > > > > > > > > > > that in the real world, people will try this > > > > > > > > > > configuration...) > > > > > > > > > > There's no > > > > > > > > > > reason not to support this, I think. > > > > > > > > > > We should add some documentation explaining the > > > > > > > > > > difference > > > > > > > > > > between > > > > > > > > > > server-side and client-side auto-creation. Without > > > > > > > > > > documentation, > > > > > > > > > > an admin > > > > > > > > > > might think that they had disabled all forms of > > > > > > > > > > auto-creation by > > > > > > > > > > setting > > > > > > > > > > the -side setting to false-- but this is not the case, > > > > > > > > > > of > > > > > > > > > > course. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > > > > > > > Hi Dhruvil, > > > > > > > > > > Thanks for reading the KIP! > > > > > That was the general idea for deprecation. We would > > > > > > > > > > log a > > > > > > > > > > warning > > > > > > > > > > when the > > > > > > > > > > config is enabled on the broker. > > > > > I also believe that there would be a change to > > > > > > > > > > documentation. > > > > > > > > > > If there is anything else that should be done, please > > > > > > > > > > let > > > > > > > > > > me > > > > > > > > > > know! > > > > > > > > > > Justine > > > > > > > > > > On Thu, Jul 11 , 2019 at 4:17 PM Dhruvil Shah < > > > > > > > > > > dhruvil@ confluent. io ( dhru...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hi Justine, > > > > > > > > > > Thanks for the KIP, this is great! > > > > > > > > > > Could you add some more information about what > > > > > > > > > > deprecating > > > > > > > > > > the > > > > > > > > > > broker > > > > > > > > > > configuration means? Would we log a warning in the > > > > > > > > > > logs > > > > > > > > > > when > > > > > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > creation is enabled on the broker, for example? > > > > > > > > > > Thanks, > > > > > Dhruvil > > > > > > > > > > On Thu, Jul 11 , 2019 at 10:28 AM Justine Olshan < > > > > > > > > > > jolshan@ confluent. io ( jols...@confluent.io ) > > > > > > > > > > > wrote: > > > > > > > > > > Hello all, > > > > > > > > > > I'd like to start a discussion thread for KIP-487. This KIP plans > > > > > > > > > > to > > > > > > > > > > deprecate the current system of > > > > > > > > > > auto-creating > > > > > > > > > > topics > > > > > > > > > > through requests to the metadata and give the > > > > > > > > > > producer the > > > > > > > > > > ability to > > > > > > > > > > automatically create topics instead. > > > > > > > > > > More information can be found here: > > > > > > > > > > https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ ( > > > > https://cwiki. > > > > > apache.org/confluence/display/KAFKA/ ) > > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > > > > > > > Thank you, > > > > > Justine Olshan > > > > > > > > > > > > > > > > > > > >
