Hi, To weigh-in, I agree with Colin on the API naming, overloads shouldn't change behavior. I think all of the Java APIs I've used so far followed this principle and I think we shouldn't diverge.
Also I think I have an entry about this incremental thing in KIP-248. It died off a bit at voting (I guess 2.0 came quick) but I was about to revive and restructure it a bit. If you remember it would have done something similar. Back then we discussed an "incremental_update" flag would have been sufficient to keep backward compatibility with the protocol. Since here you designed a new protocol I think I'll remove this bit from my KIP and also align the other parts/namings to yours so we'll have a more unified interface on this front. At last, one minor comment: is throttling a part of your protocol similarly to alterConfigs? Viktor On Fri, Jul 20, 2018 at 8:05 PM Colin McCabe <cmcc...@apache.org> wrote: > I updated the KIP. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+IncrementalAlterConfigs+API > > Updates: > * Use "incrementalAlterConfigs" rather than "modifyConfigs," for > consistency with the other "alter" APIs. > * Implement Magnus' idea of supporting "append" and "subtract" on > configuration keys that contain lists. > > best, > Colin > > > On Mon, Jul 16, 2018, at 14:12, Colin McCabe wrote: > > Hi Magnus, > > > > Thanks for taking a look. > > > > On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote: > > > Thanks for driving this KIP, Colin. > > > > > > I agree with Dong that a new similar modifyConfigs API (and protocol > API) > > > is confusing and that > > > we should try to extend the current alterConfigs interface to support > the > > > incremental mode instead, > > > deprecating the non-incremental mode in the process. > > > > In the longer term, I think that the non-incremental mode should > > definitely go away, and not be an option at all. That's why I don't > > think of this KIP as "adding more options to AlterConfigs" but as > > getting rid of a broken API. I've described a lot of reasons why non- > > incremental mode is broken. I've also described why the brokenness is > > subtle and an easy trap for newbies to fall into. Hopefully everyone > > agrees that getting rid of non-incremental mode completely should be the > > eventual goal. > > > > I do not think that having a different name for modifyConfigs is > > confusing. "Deleting all the configs and then setting some designated > > ones" is a very different operation from "modifying some > > configurations". Giving the two operations different names expresses > > the fact that they really are very different. Would it be less > > confusing if the new function were called alterConfigsIncremental rather > > than modifyConfigs? > > > > I think it's important to have a new name for the new function. If the > > names are the same, how can we even explain to users which API they > > should or should not use? "Use the three argument overload, or the two > > argument overload where the Options class is not the final parameter" > > That is not user-friendly. > > > > You could say that some of the overloads would be deprecated. However, > > my experience as a Hadoop developer is that most users simply don't care > > about deprecation warnings. They will use autocomplete in their IDEs > > and use whatever function seems to have the parameters they need. > > Hadoop and Kafka themselves use plenty of deprecated APIs. But somehow > > we expect that our users have much more respect for @deprecated than we > > ourselves do. > > > > I would further argue that function overloads in Java are intended to > > provide additional parameters, not to fundamentally change the semantics > > of a function. If you have two functions int addTwoNumbers(int a, int > > b) and int addTwoNumbers(int a, int b, boolean verbose), they should > > both add together two numbers. And a user should be able to expect that > > the plain old addTwoNumbers is equivalent to either > > addTwoNumbers(verbose=true) or addTwoNumbers(verbose=false), not a > > totally different operation. > > > > Every time programmers violate this contract, it inevitably leads to > > misunderstanding. One example is how in HDFS there are multiple > > function overloads for renaming a file. Depending on which one you > > call, you will get either RENAME or RENAME2, which have different > > semantics. I think RENAME2 has a different set of return codes > > surrounding "destination exists" conditions, among other things. Of > > course users have no idea of whether they're getting RENAME or RENAME2 > > unless they're developers. It's not obvious from the function call, > > which is named "rename" in both cases, just with different function > > parameters. So the whole thing is just a huge source of confusion and > > user pain. > > > > Another thing to consider is that since function overloads are also not > > an option in C or Go, we need a different solution for those languages > > anyway. A separate function name works well for this. > > > > > > > > Another thing that I believe is missing is a per-config-entry operation > > > type, namely SET (ovewrite), APPEND or DELETE. > > > The current proposal only has SET (changes) and DELETE (removals) > > > semantics, but I understand there are configuration properties (such > as SASL auth) where > > > it should be possible to APPEND to a configuration property, otherwise > we'll have the same > > > non-atomic read-modify-write cycle problems as with the current API. > > > Instead of providing two sets of config: changes and removals, I think > > > it might be better to just have one set where each Config entry has > > > the operation type set, this also voids any confusion on what happens > > > if a property is included in both changes,removals sets. > > > > That's a very good point. I guess the idea is that APPEND would add a > > new entry in the comma-separated (or other delimiter-separated) list of > > the config key? That would require per-key support, since not all > > configuration keys have the same delimiter. That's probably not too > > difficult, though. > > > > There are probably also lots of keys where APPEND makes no sense and > > should be rejected. For example, APPENDing to a configuration > > controlling the number of active threads for a subsystem does not make > > sense. Also, if we have APPEND, we probably also want SUBTRACT, right? > > > > best, > > Colin > > > > > > > > Regards, > > > Magnus > > > > > > 2018-07-16 20:23 GMT+02:00 Dong Lin <lindon...@gmail.com>: > > > > > > > Hey Colin, > > > > > > > > Thanks much for the explanation. Yeah it makes sense to deprecate the > > > > existing non-incremental mode completely. LGTM. I just have two minor > > > > comments. > > > > > > > > I am not too strong with the following argument but it may be useful > to > > > > just put it here for discussion. I am still wondering whether we can > just > > > > overload alterConfigs(...) instead of using modifyConfigs(...). In > the > > > > "Rejected Alternatives" section it is said that this approach does > not > > > > allow us to deprecate the non-incremental mode. Not sure why it is > the > > > > case. Can you explain a bit more? > > > > > > > > Regarding whether this approach is surprising and baffling to users, > > > > personally I feel that with the existing approach in the KIP, a new > name > > > > such as modifyConfigs(...) does make it very explicit that this API > is > > > > different from the existing alterConfigs(...). But it does not > really tell > > > > user how they are different and users will have to read the Java doc > to > > > > figure this out. On the other hand, if we just overload the > > > > alterConfigs(...) and deprecate the existing non-incremental > > > > alterConfigs(...), it would also make it reasonable clear to user > that the > > > > two methods are different. Commonly-used IDE such as IntellIj would > show > > > > that one API is deprecated and the other is not. And user would then > read > > > > the Java doc to understand the difference. So the difference between > these > > > > two approaches in the short term is probably not much. And in the > long term > > > > it might be preferred to use "alter" instead of "modify". > > > > > > > > Another minor comment: should we include specify in the > "Compatibility, > > > > Deprecation, and Migration Plan" section that we intend to deprecate > the > > > > existing API? And do we plan to deprecate the > > > > AlterConfigsRequest/AlterConfigsResponse > > > > as well? The latter may be important to non-Kafka projects that have > > > > implemented AdminClient interface. > > > > > > > > > > > > Thanks, > > > > Dong > > > > > > > > > > > > On Mon, Jul 16, 2018 at 9:43 AM, Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > > > On Fri, Jul 13, 2018, at 23:20, Dong Lin wrote: > > > > > > Hey Colin, > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > It seems that the AlterConfigsResult is pretty much the same as > > > > > > ModifyConfigsResult. Instead of adding ModifyConfigs API and > > > > deprecating > > > > > > AlterConfigs API, would it be simpler to just add API > alterConfigs( > > > > > > Map<ConfigResource, Config> changes, Set<ConfigResource> > removals, > > > > final > > > > > > ModifyConfigsOptions options)? > > > > > > > > > > > > Currently we use the word "alter" in method names such as > > > > > > "alterReplicaLogDirs" and "alterCheckpointDir". So it is > probably more > > > > > > preferred to keep using the word "alter" instead of "modify" if > > > > > posssible. > > > > > > And if we can overload the alterConfigs(...) API to allow > incremental > > > > > > change, it might make sense to keep the existing > > > > > > alterConfigs(Map<ConfigResource, > > > > > > Config> configs) for users who simply want to overwrite the > entire > > > > > configs. > > > > > > > > > > If we have two functions with these type signatures: > > > > > > > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource, Config> > changes); > > > > > > AlterConfigsResult alterConfigs(Map<ConfigResource, Config> > changes, > > > > > Set<ConfigResource> removals); > > > > > > > > > > It will be extremely surprising, even baffling, to users that the > second > > > > > function overload makes incremental changes, whereas the first > function > > > > > overload clears the entire configuration before applying changes. > Just > > > > > looking at the type signatures (which is all most developers will > look > > > > at, > > > > > especially if they're using IDE autocomplete), you would not > expect such > > > > a > > > > > radical difference between them. You would expect the second one > to work > > > > > just like the first, except maybe it would also perform some > removals. > > > > > > > > > > Calling the two functions different names is good because it > reflects the > > > > > fact that they are very different. > > > > > > > > > > > And those user would not have to make code change due to API > > > > deprecation. > > > > > > What do you think? > > > > > > > > > > alterConfigs needs to be deprecated, though. Code using > alterConfigs is > > > > > almost certainly buggy because of the possibility of simultaneous > > > > > read-modify-write cycles, and the fact that some configs can't be > read. > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > > > Thanks, > > > > > > Dong > > > > > > > > > > > > On Wed, Jul 11, 2018 at 10:54 AM, Colin McCabe < > cmcc...@apache.org> > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > Previously, we discussed some issues with alterConfigs here on > the > > > > > mailing > > > > > > > list, and eventually came to the conclusion that the RPC as > > > > implemented > > > > > > > can't be used for a shell command modifying configurations. > > > > > > > > > > > > > > I wrote up a small KIP to fix the issues with the RP Please > take a > > > > > look > > > > > > > at https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > 339%3A+Create+a+new+ModifyConfigs+API > > > > > > > > > > > > > > best, > > > > > > > Colin > > > > > > > > > > > > > > > > >
