cmccabe commented on PR #13742: URL: https://github.com/apache/kafka/pull/13742#issuecomment-1564712814
Hi all, Thanks for the reviews and comments. > @divijvaidya : But I was wondering if an additional guard could be to have a default TopicCreationPolicy with a MaxValue of X number of topics per requests (X is probably <= 10K). On every topic creation request, we will apply the policy and ensure that the request gets rejected upfront before entering the system. We could extend this pattern for other use cases where we would like to restrict range of certain parameters in the requests. What do you think? I like this idea, but it would require a KIP to implement. > @divijvaidya : I am concerned about the user facing aspect of this change. If I am a user and get this exception, what am I expected to do to resolve it? The documentation does not call out any limitation on max topics that I can create in one API call. How do I know what the limit is? The alternative approach I proposed above (topic policy limitation) solves this. We can document policies with the constraints and follow a similar pattern for other out of bound configuration/request params. I think the thing to keep in mind is that this PR doesn't make any request fail that wouldn't have already failed. > @mumrah : Colin can correct me if I'm mistaken, but I believe this patch is mainly about closing an existing edge case until we implement KIP-868 (metadata transactions). Once we have transactions in the controller, we can allow arbitrarily large batches of records. Yes, that's correct. > @mumrah : I also wondered if we could solve this in ControllerResult rather than in each control manager separately. I think the issue is that people can request truly gigantic, garbage-collector killing lists of records to be constructed. You want to cut them off before they build the giant list, not after. > @mumrah : Will we remove this logic once transactions are implemented? I think we'll need some kind of limit even with metadata transactions in place. But it will be a limit not set by the implementation, but by our policy. > @edoardocomar : I had just done an alternative smaller PR for the same issue Thank you, Edoardo, I guess we were thinking along the same lines. One thing to keep in mind is that the problem is bigger than just CreateTopics. Any operation can be too big and cause implementation problems. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
