[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981225#comment-16981225 ] Matthias J. Sax commented on KAFKA-3729: Unassigned this ticket due to inactivity. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, needs-dicussion, needs-kip, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825574#comment-16825574 ] ASF GitHub Bot commented on KAFKA-3729: --- mjsax commented on pull request #6630: KAFKA-3729: Revert adding Serde auto-config URL: https://github.com/apache/kafka/pull/6630 Merged PRs were not backward compatible. Need to revert for 2.3 release. Ticket reopened with more details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." > After the PR was merged, we realized that the current approach to implement > this features is actually not backward compatible. Thus, we need to revert > the commit for now to not break backward compatibility in 2.3 release. After > some more thinking, it seems that this feature is actually more complicated > to get right as it seem on the surface and hence it would required a proper > KIP. > The following issues are identified: > * in the new code, configure() would be called twice, one in user code (if > people don't rewrite existing applications) and later via Kafka Streams – the > second call could "reconfigure" the Serde and overwrite the correct > configuration from the first call done by the user > * if there are multiple Serdes using the same configuration parameters > names, it's only possible to specify this parameter name once in the global > StreamsConfig; hence, it's not possible for users to configure both Serdes > differently > * basically, the global StreamsConfig needs to contain all configuration > parameters over all used Serdes to make a potential second call to > `configure()` idempotant > To address the issues, some ideas would be: > * pass in the configuration via the constructor and deprecate `configure()` > method > * add a new method `isConfigured()` that would allow to skip the second > configuration call within Kafka Streams runtime > There might be other ways to address this, and the different options should > be discussed on the KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822540#comment-16822540 ] ASF GitHub Bot commented on KAFKA-3729: --- mjsax commented on pull request #6461: [KAFKA-3729] Auto-configure non-default SerDes passed alongside the topology builder URL: https://github.com/apache/kafka/pull/6461 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794668#comment-16794668 ] ASF GitHub Bot commented on KAFKA-3729: --- tedyu commented on pull request #6399: [KAFKA-3729] Auto-configure non-default SerDes passed alongside the topology builder URL: https://github.com/apache/kafka/pull/6399 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794667#comment-16794667 ] ASF GitHub Bot commented on KAFKA-3729: --- tedyu commented on pull request #6461: [KAFKA-3729] Auto-configure non-default SerDes passed alongside the topology builder URL: https://github.com/apache/kafka/pull/6461 Only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed alongside the topology builder as well. Added new tests. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794549#comment-16794549 ] Ted Yu commented on KAFKA-3729: --- 3729.v6.txt shows the progress toward passing StreamsConfig to Store#init(). > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Assignee: Ted Yu >Priority: Major > Labels: api, newbie > Attachments: 3729.txt, 3729.v6.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787312#comment-16787312 ] Ted Yu commented on KAFKA-3729: --- https://github.com/apache/kafka/pull/6399 > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > Attachments: 3729.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787294#comment-16787294 ] Guozhang Wang commented on KAFKA-3729: -- I read the diff file as well and that looks promising (btw I think you do not need to send a diff before sending a PR, a PR would make the reviews easier and we can always iterate on that even if it is not on the right direction :) > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > Attachments: 3729.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787275#comment-16787275
]
Matthias J. Sax commented on KAFKA-3729:
`configure()` expects `Map configs` but returns `Map`, so the types don't match and you get "unchecked" warning (and warning
fail the build). You would need to add `@SuppressWarning("unckecked")` to the
method to suppress the warning ant to make the build pass.
> Auto-configure non-default SerDes passed alongside the topology builder
>
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Fred Patton
>Priority: Major
> Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But
> we could auto-configure other serdes passed alongside the topology builder as
> well."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787272#comment-16787272
]
Ted Yu commented on KAFKA-3729:
---
The following seems to result in compilation error:
{code}
/Users/yute/kafka/streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:667:
warning: [unchecked] unchecked call to configure(Map,boolean) as a
member of the raw type Serializer
sn.getKeySer().configure(config.originals(), true);
^
/Users/yute/kafka/streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:668:
warning: [unchecked] unchecked call to configure(Map,boolean) as a
member of the raw type Serializer
sn.getValueSer().configure(config.originals(), false);
^
{code}
I wonder how the warnings can be suppressed.
I checked existing calls to configure() which doesn't give me much clue.
> Auto-configure non-default SerDes passed alongside the topology builder
>
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Reporter: Fred Patton
>Priority: Major
> Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But
> we could auto-configure other serdes passed alongside the topology builder as
> well."
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787241#comment-16787241 ] Matthias J. Sax commented on KAFKA-3729: [~yuzhih...@gmail.com] The patch seems to go the right way. Note that global topology does not have sink node (we can simplify there). Additionally, we need to take care of stores and changelog topics, that have Serdes, as well. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > Attachments: 3729.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785119#comment-16785119 ] Ted Yu commented on KAFKA-3729: --- Attached tentative patch. If it is on right track, I can send out a PR. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > Attachments: 3729.txt > > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784828#comment-16784828 ] Guozhang Wang commented on KAFKA-3729: -- [~mjsax] Yes `configure()` is tended to be idempotent, and if users impl is not to be idempotent they should use a boolean flag, e.g. to make sure the logic is executed only once. [~yuzhih...@gmail.com] It is from the interface `ProcessorContext#appConfigs`, and yes they can be used. Note one tricky thing is that the configs are only passed in at the construction time of the KafkaStreams, whereas the topology is built beforehand, so we can only do the auto-configure at KafkaStreams construction time, not at the topology building time since users are passing in the serde objects. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784755#comment-16784755 ] Ted Yu commented on KAFKA-3729: --- Can AbstractProcessorContext#appConfigs() be used to obtain the Map which configure() uses ? > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784075#comment-16784075 ] Matthias J. Sax commented on KAFKA-3729: The scope did no change. The "problem" is the following: As you might know, `Serializer` and `Deserializer` have a `configure(Map configs, boolean isKey)` method. Also `Serde` has this method and forwards the corresponding call to the wrapped instances. Furthermore, there are two ways Kafka Streams can use Serdes: they are passed as parameters into operators directly or Kafka Streams creates instances internally based on `StreamsConfig`. For the later case, Kafka Streams calls configure() and passes in a `Map` base on StreamsConfig (cf. `StreamsConfig#defaultKeySerde()` and `StreamsConfig#defaultValueSerde`). However, for the former case the user needs to call configure() in their own code (if it's required to configure a Serde) before they are passed into an operator. Thus, the idea of this ticket is, to lift this requirement. Kafka Streams should call configure() for all Serde objects that are passed into any operator, too, use the same `Map` it passed into `Serde` if it creates them itself. The only thing I am wondering atm is, if `configure()` is supposed to be idempotent? If not, this would be a semantic change and might require a KIP to document the change. Or we might need a config to enable the configuration but don't do it on default to preserve backward compatibility. \cc [~guozhang] > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784019#comment-16784019 ] Ted Yu commented on KAFKA-3729: --- Joined class was added Sep 6 10:55:43 2017 Guozhang's comment above was made in 2016. I wonder what needs to be done for this JIRA based on the current codebase. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783818#comment-16783818 ] Matthias J. Sax commented on KAFKA-3729: [~yuzhih...@gmail.com] The ticket is unassigned. Feel free to pick it up. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder
[ https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783778#comment-16783778 ] Ted Yu commented on KAFKA-3729: --- I wonder if a third person can work on this - considering the lack of response for two years. > Auto-configure non-default SerDes passed alongside the topology builder > > > Key: KAFKA-3729 > URL: https://issues.apache.org/jira/browse/KAFKA-3729 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Fred Patton >Priority: Major > Labels: api, newbie > > From Guozhang Wang: > "Only default serdes provided through configs are auto-configured today. But > we could auto-configure other serdes passed alongside the topology builder as > well." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
