[jira] [Issue Comment Deleted] (KAFKA-3723) Cannot change size of schema cache for JSON converter
[ https://issues.apache.org/jira/browse/KAFKA-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Posta updated KAFKA-3723: --- Comment: was deleted (was: Pull request on github: https://github.com/apache/kafka/pull/1401) > Cannot change size of schema cache for JSON converter > - > > Key: KAFKA-3723 > URL: https://issues.apache.org/jira/browse/KAFKA-3723 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Christian Posta >Assignee: Ewen Cheslack-Postava > > using this worker config, value.converter.schemas.cache.size, we should be > able to change the size of the cache. however, because of an incorrect > integer cast, we cannot change it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3723) Cannot change size of schema cache for JSON converter
[ https://issues.apache.org/jira/browse/KAFKA-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288059#comment-15288059 ] Christian Posta commented on KAFKA-3723: Pull request on github: https://github.com/apache/kafka/pull/1401 > Cannot change size of schema cache for JSON converter > - > > Key: KAFKA-3723 > URL: https://issues.apache.org/jira/browse/KAFKA-3723 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Christian Posta >Assignee: Ewen Cheslack-Postava > > using this worker config, value.converter.schemas.cache.size, we should be > able to change the size of the cache. however, because of an incorrect > integer cast, we cannot change it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)
[
https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288056#comment-15288056
]
Christian Posta commented on KAFKA-3055:
the correct config for the worker to change the cache size is
value.converter.schemas.cache.size
Need to have this fixed before you can use it though:
https://issues.apache.org/jira/browse/KAFKA-3723
> JsonConverter mangles schema during serialization (fromConnectData)
> ---
>
> Key: KAFKA-3055
> URL: https://issues.apache.org/jira/browse/KAFKA-3055
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Kishore Senji
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.1, 0.10.0.0
>
>
> Test case is here:
> https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json
> If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest
> runs successfully. Otherwise the test JsonConverterTest fails.
> The reason is that the JsonConverter has a bug where it mangles the schema as
> it assigns all String fields with the same name (and similar for all Int32
> fields)
> This is how the schema & payload gets serialized for the Person Struct (with
> caching disabled):
> {code}
> {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}}
> {code}
> where as when caching is enabled the same Struct gets serialized as (with
> caching enabled) :
> {code}
> {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}}
> {code}
> As we can see all String fields became "email" and all int32 fields became
> "weightInKgs".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-3723) Cannot change size of schema cache for JSON converter
Christian Posta created KAFKA-3723: -- Summary: Cannot change size of schema cache for JSON converter Key: KAFKA-3723 URL: https://issues.apache.org/jira/browse/KAFKA-3723 Project: Kafka Issue Type: Bug Components: KafkaConnect Reporter: Christian Posta Assignee: Ewen Cheslack-Postava using this worker config, value.converter.schemas.cache.size, we should be able to change the size of the cache. however, because of an incorrect integer cast, we cannot change it -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3376) have sensible defaults for the command-line tools to facilitate local development
[ https://issues.apache.org/jira/browse/KAFKA-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Posta updated KAFKA-3376: --- Component/s: packaging admin > have sensible defaults for the command-line tools to facilitate local > development > - > > Key: KAFKA-3376 > URL: https://issues.apache.org/jira/browse/KAFKA-3376 > Project: Kafka > Issue Type: Improvement > Components: admin, packaging >Affects Versions: 0.9.0.1 >Reporter: Christian Posta > > for the command-line tools, it's great we can send in params to connect to > brokers/zk clusters, etc. > would be great if some of those params would come with sensible defaults so > we don't have to add them for each command we run. for example, for > --zookeeper we could have it default to "localhost:2181" or default to > checking an environment variable "KAFKA_TOOLS_ZOOKEEPER" or something. > or, even better, maybe we could do something like we had in apache > activemq-apollo, where we had a command shell that implements all of these > commands and can hold "context" information about what brokers and zk > clusters exist > https://github.com/christian-posta/activemq-apollo/blob/trunk/apollo-cli/src/main/scala/org/apache/activemq/apollo/cli/Apollo.scala > so we don't have 10+ shell scripts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3376) have sensible defaults for the command-line tools to facilitate local development
[ https://issues.apache.org/jira/browse/KAFKA-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christian Posta updated KAFKA-3376: --- Affects Version/s: 0.9.0.1 > have sensible defaults for the command-line tools to facilitate local > development > - > > Key: KAFKA-3376 > URL: https://issues.apache.org/jira/browse/KAFKA-3376 > Project: Kafka > Issue Type: Improvement > Components: admin, packaging >Affects Versions: 0.9.0.1 >Reporter: Christian Posta > > for the command-line tools, it's great we can send in params to connect to > brokers/zk clusters, etc. > would be great if some of those params would come with sensible defaults so > we don't have to add them for each command we run. for example, for > --zookeeper we could have it default to "localhost:2181" or default to > checking an environment variable "KAFKA_TOOLS_ZOOKEEPER" or something. > or, even better, maybe we could do something like we had in apache > activemq-apollo, where we had a command shell that implements all of these > commands and can hold "context" information about what brokers and zk > clusters exist > https://github.com/christian-posta/activemq-apollo/blob/trunk/apollo-cli/src/main/scala/org/apache/activemq/apollo/cli/Apollo.scala > so we don't have 10+ shell scripts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2967) Move Kafka documentation to ReStructuredText
[ https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184283#comment-15184283 ] Christian Posta commented on KAFKA-2967: Many thanks for the review [~gwenshap]!! Let me take a look at the styling and see what can be done to unify the UX between the main apache site and the doco. BTW, what's used as the release manager? is it the apache jenkins (builds.apache.org)? > Move Kafka documentation to ReStructuredText > > > Key: KAFKA-2967 > URL: https://issues.apache.org/jira/browse/KAFKA-2967 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Assignee: Gwen Shapira > > Storing documentation as HTML is kind of BS :) > * Formatting is a pain, and making it look good is even worse > * Its just HTML, can't generate PDFs > * Reading and editting is painful > * Validating changes is hard because our formatting relies on all kinds of > Apache Server features. > I suggest: > * Move to RST > * Generate HTML and PDF during build using Sphinx plugin for Gradle. > Lots of Apache projects are doing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2967) Move Kafka documentation to ReStructuredText
[ https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183815#comment-15183815 ] Christian Posta edited comment on KAFKA-2967 at 3/7/16 10:05 PM: - All, I've got a quick spike of converting docs to gitbook https://www.gitbook.com which is basically markdown with some styling and can build websites, ebooks, pdf, etc. http://christianposta.com/kafka-docs/_book/index.html Point of this spike was to 1) get some feedback 2) understand how the current docs are folded into the website, and 3) figure out where some of the configs (e.g., broker configs) are in the website at apache, but not in the src :) For example, not all of these in the table here http://kafka.apache.org/documentation.html#brokerconfigs are in here https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147 This branch here as all of the code/integrated with gradle builds, etc https://github.com/christian-posta/kafka/tree/ceposta-doco again, this is currently WIP for discussion purposes, not complete by any means. was (Author: ceposta): All, I've got a quick spike of converting docs to gitbook https://www.gitbook.com which is basically markdown with some styling and can build websites, ebooks, pdf, etc. Point of this spike was to 1) get some feedback 2) understand how the current docs are folded into the website, and 3) figure out where some of the configs (e.g., broker configs) are in the website at apache, but not in the src :) For example, not all of these in the table here http://kafka.apache.org/documentation.html#brokerconfigs are in here https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147 This branch here as all of the code/integrated with gradle builds, etc https://github.com/christian-posta/kafka/tree/ceposta-doco again, this is currently WIP for discussion purposes, not complete by any means. > Move Kafka documentation to ReStructuredText > > > Key: KAFKA-2967 > URL: https://issues.apache.org/jira/browse/KAFKA-2967 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Assignee: Gwen Shapira > > Storing documentation as HTML is kind of BS :) > * Formatting is a pain, and making it look good is even worse > * Its just HTML, can't generate PDFs > * Reading and editting is painful > * Validating changes is hard because our formatting relies on all kinds of > Apache Server features. > I suggest: > * Move to RST > * Generate HTML and PDF during build using Sphinx plugin for Gradle. > Lots of Apache projects are doing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2967) Move Kafka documentation to ReStructuredText
[ https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183815#comment-15183815 ] Christian Posta commented on KAFKA-2967: All, I've got a quick spike of converting docs to gitbook https://www.gitbook.com which is basically markdown with some styling and can build websites, ebooks, pdf, etc. Point of this spike was to 1) get some feedback 2) understand how the current docs are folded into the website, and 3) figure out where some of the configs (e.g., broker configs) are in the website at apache, but not in the src :) For example, not all of these in the table here http://kafka.apache.org/documentation.html#brokerconfigs are in here https://github.com/apache/kafka/blob/trunk/docs/configuration.html#L147 This branch here as all of the code/integrated with gradle builds, etc https://github.com/christian-posta/kafka/tree/ceposta-doco again, this is currently WIP for discussion purposes, not complete by any means. > Move Kafka documentation to ReStructuredText > > > Key: KAFKA-2967 > URL: https://issues.apache.org/jira/browse/KAFKA-2967 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Assignee: Gwen Shapira > > Storing documentation as HTML is kind of BS :) > * Formatting is a pain, and making it look good is even worse > * Its just HTML, can't generate PDFs > * Reading and editting is painful > * Validating changes is hard because our formatting relies on all kinds of > Apache Server features. > I suggest: > * Move to RST > * Generate HTML and PDF during build using Sphinx plugin for Gradle. > Lots of Apache projects are doing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3274) Document command line tools
[ https://issues.apache.org/jira/browse/KAFKA-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159966#comment-15159966 ] Christian Posta commented on KAFKA-3274: Where's the best place to put this? In the docs collocated with the src (i.e., https://github.com/christian-posta/kafka/tree/trunk/docs) or in the confluence wiki? > Document command line tools > --- > > Key: KAFKA-3274 > URL: https://issues.apache.org/jira/browse/KAFKA-3274 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > While the command line tools have "help" option, it is fairly brief (by > design) and also lack documentation outside the command line options (things > from environment variables to design patters). > Will be nice to add this to the docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3272) Add debugging options to kafka-run-class.sh so we can easily run remote debugging
Christian Posta created KAFKA-3272: -- Summary: Add debugging options to kafka-run-class.sh so we can easily run remote debugging Key: KAFKA-3272 URL: https://issues.apache.org/jira/browse/KAFKA-3272 Project: Kafka Issue Type: Improvement Components: tools Affects Versions: 0.9.0.1 Reporter: Christian Posta Priority: Minor Add a KAFKA_DEBUG environment variable to easily enable remote debugging -- This message was sent by Atlassian JIRA (v6.3.4#6332)
