I updated the KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+IncrementalAlterConfigs+API
Updates: * Use "incrementalAlterConfigs" rather than "modifyConfigs," for consistency with the other "alter" APIs. * Implement Magnus' idea of supporting "append" and "subtract" on configuration keys that contain lists. best, Colin On Mon, Jul 16, 2018, at 14:12, Colin McCabe wrote: > Hi Magnus, > > Thanks for taking a look. > > On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote: > > Thanks for driving this KIP, Colin. > > > > I agree with Dong that a new similar modifyConfigs API (and protocol API) > > is confusing and that > > we should try to extend the current alterConfigs interface to support the > > incremental mode instead, > > deprecating the non-incremental mode in the process. > > In the longer term, I think that the non-incremental mode should > definitely go away, and not be an option at all. That's why I don't > think of this KIP as "adding more options to AlterConfigs" but as > getting rid of a broken API. I've described a lot of reasons why non- > incremental mode is broken. I've also described why the brokenness is > subtle and an easy trap for newbies to fall into. Hopefully everyone > agrees that getting rid of non-incremental mode completely should be the > eventual goal. > > I do not think that having a different name for modifyConfigs is > confusing. "Deleting all the configs and then setting some designated > ones" is a very different operation from "modifying some > configurations". Giving the two operations different names expresses > the fact that they really are very different. Would it be less > confusing if the new function were called alterConfigsIncremental rather > than modifyConfigs? > > I think it's important to have a new name for the new function. If the > names are the same, how can we even explain to users which API they > should or should not use? "Use the three argument overload, or the two > argument overload where the Options class is not the final parameter" > That is not user-friendly. > > You could say that some of the overloads would be deprecated. However, > my experience as a Hadoop developer is that most users simply don't care > about deprecation warnings. They will use autocomplete in their IDEs > and use whatever function seems to have the parameters they need. > Hadoop and Kafka themselves use plenty of deprecated APIs. But somehow > we expect that our users have much more respect for @deprecated than we > ourselves do. > > I would further argue that function overloads in Java are intended to > provide additional parameters, not to fundamentally change the semantics > of a function. If you have two functions int addTwoNumbers(int a, int > b) and int addTwoNumbers(int a, int b, boolean verbose), they should > both add together two numbers. And a user should be able to expect that > the plain old addTwoNumbers is equivalent to either > addTwoNumbers(verbose=true) or addTwoNumbers(verbose=false), not a > totally different operation. > > Every time programmers violate this contract, it inevitably leads to > misunderstanding. One example is how in HDFS there are multiple > function overloads for renaming a file. Depending on which one you > call, you will get either RENAME or RENAME2, which have different > semantics. I think RENAME2 has a different set of return codes > surrounding "destination exists" conditions, among other things. Of > course users have no idea of whether they're getting RENAME or RENAME2 > unless they're developers. It's not obvious from the function call, > which is named "rename" in both cases, just with different function > parameters. So the whole thing is just a huge source of confusion and > user pain. > > Another thing to consider is that since function overloads are also not > an option in C or Go, we need a different solution for those languages > anyway. A separate function name works well for this. > > > > > Another thing that I believe is missing is a per-config-entry operation > > type, namely SET (ovewrite), APPEND or DELETE. > > The current proposal only has SET (changes) and DELETE (removals) > > semantics, but I understand there are configuration properties (such as > > SASL auth) where > > it should be possible to APPEND to a configuration property, otherwise > > we'll have the same > > non-atomic read-modify-write cycle problems as with the current API. > > Instead of providing two sets of config: changes and removals, I think > > it might be better to just have one set where each Config entry has > > the operation type set, this also voids any confusion on what happens > > if a property is included in both changes,removals sets. > > That's a very good point. I guess the idea is that APPEND would add a > new entry in the comma-separated (or other delimiter-separated) list of > the config key? That would require per-key support, since not all > configuration keys have the same delimiter. That's probably not too > difficult, though. > > There are probably also lots of keys where APPEND makes no sense and > should be rejected. For example, APPENDing to a configuration > controlling the number of active threads for a subsystem does not make > sense. Also, if we have APPEND, we probably also want SUBTRACT, right? > > best, > Colin > > > > > Regards, > > Magnus > > > > 2018-07-16 20:23 GMT+02:00 Dong Lin <lindon...@gmail.com>: > > > > > Hey Colin, > > > > > > Thanks much for the explanation. Yeah it makes sense to deprecate the > > > existing non-incremental mode completely. LGTM. I just have two minor > > > comments. > > > > > > I am not too strong with the following argument but it may be useful to > > > just put it here for discussion. I am still wondering whether we can just > > > overload alterConfigs(...) instead of using modifyConfigs(...). In the > > > "Rejected Alternatives" section it is said that this approach does not > > > allow us to deprecate the non-incremental mode. Not sure why it is the > > > case. Can you explain a bit more? > > > > > > Regarding whether this approach is surprising and baffling to users, > > > personally I feel that with the existing approach in the KIP, a new name > > > such as modifyConfigs(...) does make it very explicit that this API is > > > different from the existing alterConfigs(...). But it does not really tell > > > user how they are different and users will have to read the Java doc to > > > figure this out. On the other hand, if we just overload the > > > alterConfigs(...) and deprecate the existing non-incremental > > > alterConfigs(...), it would also make it reasonable clear to user that the > > > two methods are different. Commonly-used IDE such as IntellIj would show > > > that one API is deprecated and the other is not. And user would then read > > > the Java doc to understand the difference. So the difference between these > > > two approaches in the short term is probably not much. And in the long > > > term > > > it might be preferred to use "alter" instead of "modify". > > > > > > Another minor comment: should we include specify in the "Compatibility, > > > Deprecation, and Migration Plan" section that we intend to deprecate the > > > existing API? And do we plan to deprecate the > > > AlterConfigsRequest/AlterConfigsResponse > > > as well? The latter may be important to non-Kafka projects that have > > > implemented AdminClient interface. > > > > > > > > > Thanks, > > > Dong > > > > > > > > > On Mon, Jul 16, 2018 at 9:43 AM, Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > On Fri, Jul 13, 2018, at 23:20, Dong Lin wrote: > > > > > Hey Colin, > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > It seems that the AlterConfigsResult is pretty much the same as > > > > > ModifyConfigsResult. Instead of adding ModifyConfigs API and > > > deprecating > > > > > AlterConfigs API, would it be simpler to just add API alterConfigs( > > > > > Map<ConfigResource, Config> changes, Set<ConfigResource> removals, > > > final > > > > > ModifyConfigsOptions options)? > > > > > > > > > > Currently we use the word "alter" in method names such as > > > > > "alterReplicaLogDirs" and "alterCheckpointDir". So it is probably more > > > > > preferred to keep using the word "alter" instead of "modify" if > > > > posssible. > > > > > And if we can overload the alterConfigs(...) API to allow incremental > > > > > change, it might make sense to keep the existing > > > > > alterConfigs(Map<ConfigResource, > > > > > Config> configs) for users who simply want to overwrite the entire > > > > configs. > > > > > > > > If we have two functions with these type signatures: > > > > > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource, Config> changes); > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource, Config> changes, > > > > Set<ConfigResource> removals); > > > > > > > > It will be extremely surprising, even baffling, to users that the > > > > second > > > > function overload makes incremental changes, whereas the first function > > > > overload clears the entire configuration before applying changes. Just > > > > looking at the type signatures (which is all most developers will look > > > at, > > > > especially if they're using IDE autocomplete), you would not expect such > > > a > > > > radical difference between them. You would expect the second one to > > > > work > > > > just like the first, except maybe it would also perform some removals. > > > > > > > > Calling the two functions different names is good because it reflects > > > > the > > > > fact that they are very different. > > > > > > > > > And those user would not have to make code change due to API > > > deprecation. > > > > > What do you think? > > > > > > > > alterConfigs needs to be deprecated, though. Code using alterConfigs is > > > > almost certainly buggy because of the possibility of simultaneous > > > > read-modify-write cycles, and the fact that some configs can't be read. > > > > > > > > best, > > > > Colin > > > > > > > > > > > > > > Thanks, > > > > > Dong > > > > > > > > > > On Wed, Jul 11, 2018 at 10:54 AM, Colin McCabe <cmcc...@apache.org> > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > Previously, we discussed some issues with alterConfigs here on the > > > > mailing > > > > > > list, and eventually came to the conclusion that the RPC as > > > implemented > > > > > > can't be used for a shell command modifying configurations. > > > > > > > > > > > > I wrote up a small KIP to fix the issues with the RP Please take a > > > > look > > > > > > at https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > 339%3A+Create+a+new+ModifyConfigs+API > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > >
