[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-11-24 Thread Matthias J. Sax (Jira)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981225#comment-16981225
 ] 

Matthias J. Sax commented on KAFKA-3729:


Unassigned this ticket due to inactivity.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, needs-dicussion, needs-kip, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
>  "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."
> After the PR was merged, we realized that the current approach to implement 
> this features is actually not backward compatible. Thus, we need to revert 
> the commit for now to not break backward compatibility in 2.3 release. After 
> some more thinking, it seems that this feature is actually more complicated 
> to get right as it seem on the surface and hence it would required a proper 
> KIP.
> The following issues are identified:
>  * in the new code, configure() would be called twice, one in user code (if 
> people don't rewrite existing applications) and later via Kafka Streams – the 
> second call could "reconfigure" the Serde and overwrite the correct 
> configuration from the first call done by the user
>  * if there are multiple Serdes using the same configuration parameters 
> names, it's only possible to specify this parameter name once in the global 
> StreamsConfig; hence, it's not possible for users to configure both Serdes 
> differently
>  * basically, the global StreamsConfig needs to contain all configuration 
> parameters over all used Serdes to make a potential second call to 
> `configure()` idempotant
> To address the issues, some ideas would be:
>  * pass in the configuration via the constructor and deprecate `configure()` 
> method
>  * add a new method `isConfigured()` that would allow to skip the second 
> configuration call within Kafka Streams runtime
> There might be other ways to address this, and the different options should 
> be discussed on the KIP.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-05-01 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830898#comment-16830898
 ] 

ASF GitHub Bot commented on KAFKA-3729:
---

mjsax commented on pull request #6630: KAFKA-3729: Revert adding Serde 
auto-config
URL: https://github.com/apache/kafka/pull/6630
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, needs-dicussion, needs-kip, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
>  "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."
> After the PR was merged, we realized that the current approach to implement 
> this features is actually not backward compatible. Thus, we need to revert 
> the commit for now to not break backward compatibility in 2.3 release. After 
> some more thinking, it seems that this feature is actually more complicated 
> to get right as it seem on the surface and hence it would required a proper 
> KIP.
> The following issues are identified:
>  * in the new code, configure() would be called twice, one in user code (if 
> people don't rewrite existing applications) and later via Kafka Streams – the 
> second call could "reconfigure" the Serde and overwrite the correct 
> configuration from the first call done by the user
>  * if there are multiple Serdes using the same configuration parameters 
> names, it's only possible to specify this parameter name once in the global 
> StreamsConfig; hence, it's not possible for users to configure both Serdes 
> differently
>  * basically, the global StreamsConfig needs to contain all configuration 
> parameters over all used Serdes to make a potential second call to 
> `configure()` idempotant
> To address the issues, some ideas would be:
>  * pass in the configuration via the constructor and deprecate `configure()` 
> method
>  * add a new method `isConfigured()` that would allow to skip the second 
> configuration call within Kafka Streams runtime
> There might be other ways to address this, and the different options should 
> be discussed on the KIP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-04-24 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825574#comment-16825574
 ] 

ASF GitHub Bot commented on KAFKA-3729:
---

mjsax commented on pull request #6630: KAFKA-3729: Revert adding Serde 
auto-config
URL: https://github.com/apache/kafka/pull/6630
 
 
   Merged PRs were not backward compatible. Need to revert for 2.3 release. 
Ticket reopened with more details.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
>  "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."
> After the PR was merged, we realized that the current approach to implement 
> this features is actually not backward compatible. Thus, we need to revert 
> the commit for now to not break backward compatibility in 2.3 release. After 
> some more thinking, it seems that this feature is actually more complicated 
> to get right as it seem on the surface and hence it would required a proper 
> KIP.
> The following issues are identified:
>  * in the new code, configure() would be called twice, one in user code (if 
> people don't rewrite existing applications) and later via Kafka Streams – the 
> second call could "reconfigure" the Serde and overwrite the correct 
> configuration from the first call done by the user
>  * if there are multiple Serdes using the same configuration parameters 
> names, it's only possible to specify this parameter name once in the global 
> StreamsConfig; hence, it's not possible for users to configure both Serdes 
> differently
>  * basically, the global StreamsConfig needs to contain all configuration 
> parameters over all used Serdes to make a potential second call to 
> `configure()` idempotant
> To address the issues, some ideas would be:
>  * pass in the configuration via the constructor and deprecate `configure()` 
> method
>  * add a new method `isConfigured()` that would allow to skip the second 
> configuration call within Kafka Streams runtime
> There might be other ways to address this, and the different options should 
> be discussed on the KIP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-04-20 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822540#comment-16822540
 ] 

ASF GitHub Bot commented on KAFKA-3729:
---

mjsax commented on pull request #6461: [KAFKA-3729] Auto-configure non-default 
SerDes passed alongside the topology builder
URL: https://github.com/apache/kafka/pull/6461
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794668#comment-16794668
 ] 

ASF GitHub Bot commented on KAFKA-3729:
---

tedyu commented on pull request #6399: [KAFKA-3729] Auto-configure non-default 
SerDes passed alongside the topology builder
URL: https://github.com/apache/kafka/pull/6399
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-17 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794667#comment-16794667
 ] 

ASF GitHub Bot commented on KAFKA-3729:
---

tedyu commented on pull request #6461: [KAFKA-3729] Auto-configure non-default 
SerDes passed alongside the topology builder
URL: https://github.com/apache/kafka/pull/6461
 
 
   Only default serdes provided through configs are auto-configured today. But 
we could auto-configure other serdes passed alongside the topology builder as 
well.
   
   Added new tests.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-17 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794549#comment-16794549
 ] 

Ted Yu commented on KAFKA-3729:
---

3729.v6.txt shows the progress toward passing StreamsConfig to Store#init().

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Assignee: Ted Yu
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt, 3729.v6.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-07 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787312#comment-16787312
 ] 

Ted Yu commented on KAFKA-3729:
---

https://github.com/apache/kafka/pull/6399

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-07 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787294#comment-16787294
 ] 

Guozhang Wang commented on KAFKA-3729:
--

I read the diff file as well and that looks promising (btw I think you do not 
need to send a diff before sending a PR, a PR would make the reviews easier and 
we can always iterate on that even if it is not on the right direction :)

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-07 Thread Matthias J. Sax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787275#comment-16787275
 ] 

Matthias J. Sax commented on KAFKA-3729:


`configure()` expects `Map configs` but returns `Map`, so the types don't match and you get "unchecked" warning (and warning 
fail the build). You would need to add `@SuppressWarning("unckecked")` to the 
method to suppress the warning ant to make the build pass.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-07 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787272#comment-16787272
 ] 

Ted Yu commented on KAFKA-3729:
---

The following seems to result in compilation error:
{code}
/Users/yute/kafka/streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:667:
 warning: [unchecked] unchecked call to configure(Map,boolean) as a 
member of the raw type Serializer
sn.getKeySer().configure(config.originals(), true);
^
/Users/yute/kafka/streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:668:
 warning: [unchecked] unchecked call to configure(Map,boolean) as a 
member of the raw type Serializer
sn.getValueSer().configure(config.originals(), false);
  ^
{code}
I wonder how the warnings can be suppressed.
I checked existing calls to configure() which doesn't give me much clue.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-07 Thread Matthias J. Sax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787241#comment-16787241
 ] 

Matthias J. Sax commented on KAFKA-3729:


[~yuzhih...@gmail.com] The patch seems to go the right way. Note that global 
topology does not have sink node (we can simplify there). Additionally, we need 
to take care of stores and changelog topics, that have Serdes, as well.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-05 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785119#comment-16785119
 ] 

Ted Yu commented on KAFKA-3729:
---

Attached tentative patch.
If it is on right track, I can send out a PR.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
> Attachments: 3729.txt
>
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-05 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784828#comment-16784828
 ] 

Guozhang Wang commented on KAFKA-3729:
--

[~mjsax] Yes `configure()` is tended to be idempotent, and if users impl is not 
to be idempotent they should use a boolean flag, e.g. to make sure the logic is 
executed only once.

[~yuzhih...@gmail.com] It is from the interface `ProcessorContext#appConfigs`, 
and yes they can be used. Note one tricky thing is that the configs are only 
passed in at the construction time of the KafkaStreams, whereas the topology is 
built beforehand, so we can only do the auto-configure at KafkaStreams 
construction time, not at the topology building time since users are passing in 
the serde objects.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-05 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784755#comment-16784755
 ] 

Ted Yu commented on KAFKA-3729:
---

Can AbstractProcessorContext#appConfigs() be used to obtain the Map which 
configure() uses ?

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-04 Thread Matthias J. Sax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784075#comment-16784075
 ] 

Matthias J. Sax commented on KAFKA-3729:


The scope did no change. The "problem" is the following: As you might know, 
`Serializer` and `Deserializer` have a `configure(Map configs, 
boolean isKey)` method. Also `Serde` has this method and forwards the 
corresponding call to the wrapped instances.

Furthermore, there are two ways Kafka Streams can use Serdes: they are passed 
as parameters into operators directly or Kafka Streams creates instances 
internally based on `StreamsConfig`. For the later case, Kafka Streams calls 
configure() and passes in a `Map` base on StreamsConfig (cf. 
`StreamsConfig#defaultKeySerde()` and `StreamsConfig#defaultValueSerde`). 
However, for the former case the user needs to call configure() in their own 
code (if it's required to configure a Serde) before they are passed into an 
operator.

Thus, the idea of this ticket is, to lift this requirement. Kafka Streams 
should call configure() for all Serde objects that are passed into any 
operator, too, use the same `Map` it passed into `Serde` if it creates them 
itself.

The only thing I am wondering atm is, if `configure()` is supposed to be 
idempotent? If not, this would be a semantic change and might require a KIP to 
document the change. Or we might need a config to enable the configuration but 
don't do it on default to preserve backward compatibility. \cc [~guozhang]

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-04 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784019#comment-16784019
 ] 

Ted Yu commented on KAFKA-3729:
---

Joined class was added Sep 6 10:55:43 2017

Guozhang's comment above was made in 2016.

I wonder what needs to be done for this JIRA based on the current codebase.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-04 Thread Matthias J. Sax (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783818#comment-16783818
 ] 

Matthias J. Sax commented on KAFKA-3729:


[~yuzhih...@gmail.com] The ticket is unassigned. Feel free to pick it up.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KAFKA-3729) Auto-configure non-default SerDes passed alongside the topology builder

2019-03-04 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783778#comment-16783778
 ] 

Ted Yu commented on KAFKA-3729:
---

I wonder if a third person can work on this - considering the lack of response 
for two years.

>  Auto-configure non-default SerDes passed alongside the topology builder
> 
>
> Key: KAFKA-3729
> URL: https://issues.apache.org/jira/browse/KAFKA-3729
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Fred Patton
>Priority: Major
>  Labels: api, newbie
>
> From Guozhang Wang:
> "Only default serdes provided through configs are auto-configured today. But 
> we could auto-configure other serdes passed alongside the topology builder as 
> well."



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)