[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288056#comment-15288056 ] Christian Posta commented on KAFKA-3055: the correct config for the worker to change the cache size is value.converter.schemas.cache.size Need to have this fixed before you can use it though: https://issues.apache.org/jira/browse/KAFKA-3723 > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.1, 0.10.0.0 > > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139051#comment-15139051 ] Kishore Senji commented on KAFKA-3055: -- Please look here: https://github.com/ksenji/kafka-connect-test/blob/master/src/test/java/org/apache/kafka/connect/json/JsonConverterWithNoCacheTest.java > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.1, 0.9.1.0 > > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15138155#comment-15138155 ] Ewen Cheslack-Postava commented on KAFKA-3055: -- There's a config schemas.cache.size. Since it's an LRU cache, setting the size to 1 would presumably work. > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.1, 0.9.1.0 > > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128891#comment-15128891 ] Saravanan Tirugnanum commented on KAFKA-3055: - Hi Kishore I am facing this same problem and trying to see if we can handle without updating the patch fix done already. Can you please let me know how to disable cache as you mentioned this works fine when cache is disabled. regards Saravanan > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.1, 0.9.1.0 > > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081345#comment-15081345 ] ASF GitHub Bot commented on KAFKA-3055: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/722 > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.1, 0.9.1.0 > > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15076221#comment-15076221 ] ASF GitHub Bot commented on KAFKA-3055: --- GitHub user ksenji opened a pull request: https://github.com/apache/kafka/pull/722 Fix for KAFKA-3055 You can merge this pull request into a Git repository by running: $ git pull https://github.com/ksenji/kafka trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/722.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #722 commit 60a5b5cecd39ccdd4ff2f977a6bfdef123cadb44 Author: ksenji Date: 2016-01-01T02:01:44Z Fix for KAFKA-3055 > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[ https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15076222#comment-15076222 ] Kishore Senji commented on KAFKA-3055: -- PR: https://github.com/apache/kafka/pull/722 > JsonConverter mangles schema during serialization (fromConnectData) > --- > > Key: KAFKA-3055 > URL: https://issues.apache.org/jira/browse/KAFKA-3055 > Project: Kafka > Issue Type: Bug > Components: copycat >Affects Versions: 0.9.0.0 >Reporter: Kishore Senji >Assignee: Ewen Cheslack-Postava > > Test case is here: > https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json > If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest > runs successfully. Otherwise the test JsonConverterTest fails. > The reason is that the JsonConverter has a bug where it mangles the schema as > it assigns all String fields with the same name (and similar for all Int32 > fields) > This is how the schema & payload gets serialized for the Person Struct (with > caching disabled): > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > where as when caching is enabled the same Struct gets serialized as (with > caching enabled) : > {code} > {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}} > {code} > As we can see all String fields became "email" and all int32 fields became > "weightInKgs". -- This message was sent by Atlassian JIRA (v6.3.4#6332)