[jira] [Commented] (KAFKA-2633) Default logging from tools to Stderr
[ https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955241#comment-14955241 ] Gwen Shapira commented on KAFKA-2633: - [~brocknoland] - technically it was fixed, the change was merged yesterday and was not rolled back. It will be in the next release. > Default logging from tools to Stderr > > > Key: KAFKA-2633 > URL: https://issues.apache.org/jira/browse/KAFKA-2633 > Project: Kafka > Issue Type: Bug >Reporter: Grant Henke >Assignee: Grant Henke > Fix For: 0.9.0.0 > > > Currently the default logging for tool is stdout, this can make parsing the > output difficult in cases where exceptions are thrown, or even when expected > logging messages are output. The most affected tool is the console-consumer > but others will have this issue as well. > Changing the output to stderr by default allows the user to redirect the > output to a log file without effecting the tools output. > Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to > use the configuration of their choice. This Jira is to change the default to > be more user friendly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2633) Default logging from tools to Stderr
[ https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955194#comment-14955194 ] Gwen Shapira commented on KAFKA-2633: - [~ewencp], thanks for highlighting a possible issue. I'm not clear if the goal of the reopen is to discuss whether the change is a good idea (which [~granthenke] followed up on in the PR), or to request documentation for the change (which can be done in a separate JIRA). > Default logging from tools to Stderr > > > Key: KAFKA-2633 > URL: https://issues.apache.org/jira/browse/KAFKA-2633 > Project: Kafka > Issue Type: Bug >Reporter: Grant Henke >Assignee: Grant Henke > Fix For: 0.9.0.0 > > > Currently the default logging for tool is stdout, this can make parsing the > output difficult in cases where exceptions are thrown, or even when expected > logging messages are output. The most affected tool is the console-consumer > but others will have this issue as well. > Changing the output to stderr by default allows the user to redirect the > output to a log file without effecting the tools output. > Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to > use the configuration of their choice. This Jira is to change the default to > be more user friendly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds
[
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2613:
Resolution: Fixed
Fix Version/s: 0.9.0.0
Status: Resolved (was: Patch Available)
> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
> Issue Type: Sub-task
> Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> We currently set `maxParallelForks` to the number returned by
> `Runtime.availableProcessors`.
> {code}
> tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
> }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle
> property and use that property to set it to an appropriate value when the
> build is run from Jenkins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds
[
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954250#comment-14954250
]
Gwen Shapira commented on KAFKA-2613:
-
Well, I'm not sure we are any stabler now, but I didn't break Jenkins so I'm
closing this one.
> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
> Issue Type: Sub-task
> Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.0
>
>
> We currently set `maxParallelForks` to the number returned by
> `Runtime.availableProcessors`.
> {code}
> tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
> }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle
> property and use that property to set it to an appropriate value when the
> build is run from Jenkins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds
[
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954215#comment-14954215
]
Gwen Shapira commented on KAFKA-2613:
-
1 :)
> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
> Issue Type: Sub-task
> Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by
> `Runtime.availableProcessors`.
> {code}
> tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
> }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle
> property and use that property to set it to an appropriate value when the
> build is run from Jenkins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2613) Consider capping `maxParallelForks` for Jenkins builds
[
https://issues.apache.org/jira/browse/KAFKA-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954205#comment-14954205
]
Gwen Shapira commented on KAFKA-2613:
-
Committed and updated the Jenkins builds.
Waiting to make sure tests pass before closing this one.
> Consider capping `maxParallelForks` for Jenkins builds
> --
>
> Key: KAFKA-2613
> URL: https://issues.apache.org/jira/browse/KAFKA-2613
> Project: Kafka
> Issue Type: Sub-task
> Components: build
>Reporter: Ismael Juma
>Assignee: Ewen Cheslack-Postava
>
> We currently set `maxParallelForks` to the number returned by
> `Runtime.availableProcessors`.
> {code}
> tasks.withType(Test) {
> maxParallelForks = Runtime.runtime.availableProcessors()
> }
> {code}
> This returns the number of logical cores (including hyperthreaded cores) in
> the machine.
> This is usually OK when running the tests locally, but the Apache Jenkins
> slaves run 2 to 3 jobs simultaneously causing a higher number of timing
> related failures.
> A potential solution is to allow `maxParallelForks` to be set via a Gradle
> property and use that property to set it to an appropriate value when the
> build is run from Jenkins.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2633) Default logging from tools to Stderr
[ https://issues.apache.org/jira/browse/KAFKA-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2633: Resolution: Fixed Fix Version/s: 0.9.0.0 Status: Resolved (was: Patch Available) Issue resolved by pull request 296 [https://github.com/apache/kafka/pull/296] > Default logging from tools to Stderr > > > Key: KAFKA-2633 > URL: https://issues.apache.org/jira/browse/KAFKA-2633 > Project: Kafka > Issue Type: Bug >Reporter: Grant Henke >Assignee: Grant Henke > Fix For: 0.9.0.0 > > > Currently the default logging for tool is stdout, this can make parsing the > output difficult in cases where exceptions are thrown, or even when expected > logging messages are output. The most affected tool is the console-consumer > but others will have this issue as well. > Changing the output to stderr by default allows the user to redirect the > output to a log file without effecting the tools output. > Note: Users can change the logging settings by setting $KAFKA_LOG4J_OPTS to > use the configuration of their choice. This Jira is to change the default to > be more user friendly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1682) Security for Kafka
[ https://issues.apache.org/jira/browse/KAFKA-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952648#comment-14952648 ] Gwen Shapira commented on KAFKA-1682: - Note that each subtask has "fix version" in its details. This indicates which version includes the fix. In this case all subtasks are marked with "0.9.0.0", which is a future release. So, no existing release includes the security patches, those will be part of the next release. > Security for Kafka > -- > > Key: KAFKA-1682 > URL: https://issues.apache.org/jira/browse/KAFKA-1682 > Project: Kafka > Issue Type: New Feature > Components: security >Affects Versions: 0.10.0.0 >Reporter: Jay Kreps > > Parent ticket for security. Wiki and discussion is here: > https://cwiki.apache.org/confluence/display/KAFKA/Security -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2611) Document MirrorMaker
Gwen Shapira created KAFKA-2611: --- Summary: Document MirrorMaker Key: KAFKA-2611 URL: https://issues.apache.org/jira/browse/KAFKA-2611 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Its been part of our platform for a while, will be nice to add some documentation around how to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2425) Migrate website from SVN to Git
[ https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2425. - Resolution: Fixed Fix Version/s: 0.9.0.0 Issue resolved by pull request 171 [https://github.com/apache/kafka/pull/171] > Migrate website from SVN to Git > > > Key: KAFKA-2425 > URL: https://issues.apache.org/jira/browse/KAFKA-2425 > Project: Kafka > Issue Type: Improvement >Reporter: Ismael Juma >Assignee: Manikumar Reddy > Fix For: 0.9.0.0 > > > The preference is to share the same Git repo for the code and website as per > discussion in the mailing list: > http://search-hadoop.com/m/uyzND1Dux842dm7vg2 > Useful reference: > https://blogs.apache.org/infra/entry/git_based_websites_available -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2247) Merge kafka.utils.Time and kafka.common.utils.Time
[ https://issues.apache.org/jira/browse/KAFKA-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14942301#comment-14942301 ] Gwen Shapira commented on KAFKA-2247: - I have absolutely no idea what I was trying to say back in June, sorry :) > Merge kafka.utils.Time and kafka.common.utils.Time > -- > > Key: KAFKA-2247 > URL: https://issues.apache.org/jira/browse/KAFKA-2247 > Project: Kafka > Issue Type: Improvement >Reporter: Aditya Auradkar >Assignee: Aditya Auradkar >Priority: Minor > > We currently have 2 different versions of Time in clients and core. These > need to be merged. > It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is > used by some tests while `o.a.kafka.common.utils.Time` does not. We either > need to add this functionality or change the tests not to need it anymore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2585) ConsoleConsumer should not hang infinitely upon exception
[ https://issues.apache.org/jira/browse/KAFKA-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2585. - Resolution: Fixed Fix Version/s: 0.9.0.0 Issue resolved by pull request 247 [https://github.com/apache/kafka/pull/247] > ConsoleConsumer should not hang infinitely upon exception > - > > Key: KAFKA-2585 > URL: https://issues.apache.org/jira/browse/KAFKA-2585 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 0.9.0.0 > > > Due to imcompatibility beween Java 1.7 and Java 1.8, if Kafka ConsoleConsumer > was compiled by Java 1.8 but run by Java 1.7, and if ConsoleConsumer has > consumer.timeout.ms in the consumer.properties, then when it timesout, it > will throw the following exception and hang infinitely. > This will cause problem for e.g. Ducktape system test, which currently runs > consumer with Java 1.7 and waits for consumer to timeout. This bug causes the > Ducktape to hang for a long time. > To prevent ConsoleConsumer to hang in case of such exception, we can wrap > try/catch around run() and do System.exit(0) at the end of main() in > ConsoleConsumer. > [2015-09-28 05:41:31,499] ERROR Error processing message, stopping consumer: > (kafka.tools.ConsoleConsumer$) > kafka.consumer.ConsumerTimeoutException > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69) > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33) > at > kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66) > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58) > at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38) > at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:46) > at kafka.consumer.OldConsumer.receive(BaseConsumer.scala:70) > at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:94) > at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:57) > at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:41) > at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) > Processed a total of 78 messages > Exception in thread "main" java.lang.NoSuchMethodError: > java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView; > at kafka.utils.Pool.keys(Pool.scala:77) > at > kafka.consumer.FetchRequestAndResponseStatsRegistry$.removeConsumerFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:69) > at > kafka.metrics.KafkaMetricsGroup$.removeAllConsumerMetrics(KafkaMetricsGroup.scala:189) > at > kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:202) > at kafka.consumer.OldConsumer.stop(BaseConsumer.scala:75) > at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:98) > at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:57) > at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:41) > at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2368) Add Copycat standalone CLI
[ https://issues.apache.org/jira/browse/KAFKA-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14909909#comment-14909909 ] Gwen Shapira commented on KAFKA-2368: - While closing JIRA without doing any work is a form of cheating, I can't think of any other features we need for the stand-alone CLI. > Add Copycat standalone CLI > -- > > Key: KAFKA-2368 > URL: https://issues.apache.org/jira/browse/KAFKA-2368 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > > Define and add the CLI for running Copycat in standalone mode. KAFKA-2366 may > add a simple version, but this JIRA should address agreeing on the final > interface for the functionality defined in KIP-26, which proposed a basic API > but may need further extension. > Since the standalone mode is not meant to be persistent, this may end up > being a pretty straightforward API. However, it should cleanly handle both > worker-level configs and multiple connector configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2531) Add Ducktape based tests for KafkaLog4jAppender
[ https://issues.apache.org/jira/browse/KAFKA-2531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2531. - Resolution: Fixed Fix Version/s: 0.9.0.0 Issue resolved by pull request 235 [https://github.com/apache/kafka/pull/235] > Add Ducktape based tests for KafkaLog4jAppender > --- > > Key: KAFKA-2531 > URL: https://issues.apache.org/jira/browse/KAFKA-2531 > Project: Kafka > Issue Type: Test >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Fix For: 0.9.0.0 > > > Add ducktape based tests for KafkaLog4jAppender. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2374) Implement Copycat log/file connector
[ https://issues.apache.org/jira/browse/KAFKA-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908897#comment-14908897 ] Gwen Shapira commented on KAFKA-2374: - Good question. I am in favor of keeping Kafka itself simple (and avoid bringing extra dependencies) and publishing a full-featured Log Source for Copycat elsewhere. Does that make sense? > Implement Copycat log/file connector > > > Key: KAFKA-2374 > URL: https://issues.apache.org/jira/browse/KAFKA-2374 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > > This is a good baseline connector that has zero dependencies and works well > as both a demonstration and a practical use case for standalone mode. > Two key features it should ideally support: support multiple files and > rolling log files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2579) Unauthorized clients should not be able to join groups
[ https://issues.apache.org/jira/browse/KAFKA-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2579. - Resolution: Fixed Fix Version/s: 0.9.0.0 Issue resolved by pull request 240 [https://github.com/apache/kafka/pull/240] > Unauthorized clients should not be able to join groups > --- > > Key: KAFKA-2579 > URL: https://issues.apache.org/jira/browse/KAFKA-2579 > Project: Kafka > Issue Type: Sub-task > Components: security >Affects Versions: 0.9.0.0 >Reporter: Jason Gustafson >Assignee: Jason Gustafson > Fix For: 0.9.0.0 > > > The JoinGroup authorization is only checked in the response callback which is > invoked after the request has been forwarded to the ConsumerCoordinator and > the client has joined the group. This allows unauthorized members to impact > the rest of the group since the coordinator will assign partitions to them. > It would be better to check permission and return immediately if the client > is unauthorized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2373) Copycat distributed offset storage
[ https://issues.apache.org/jira/browse/KAFKA-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907422#comment-14907422 ] Gwen Shapira commented on KAFKA-2373: - Thanks for the contribution [~ewencp] and for the review [~wushujames]! > Copycat distributed offset storage > -- > > Key: KAFKA-2373 > URL: https://issues.apache.org/jira/browse/KAFKA-2373 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > > Add offset storage for Copycat that works in distributed mode, which likely > means storing the data in a Kafka topic. Copycat workers will use this by > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2571) KafkaLog4jAppender dies while specifying "acks" config
[
https://issues.apache.org/jira/browse/KAFKA-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2571:
Resolution: Fixed
Fix Version/s: 0.9.0.0
Status: Resolved (was: Patch Available)
Issue resolved by pull request 231
[https://github.com/apache/kafka/pull/231]
> KafkaLog4jAppender dies while specifying "acks" config
> --
>
> Key: KAFKA-2571
> URL: https://issues.apache.org/jira/browse/KAFKA-2571
> Project: Kafka
> Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.9.0.0
>
>
> KafkaLog4jAppender specifies "acks" config's value as int, however
> KafkaProducer expects it as a String.
> Below is the exception that gets thrown.
> {code}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException:
> Invalid value -1 for configuration acks: Expected value to be a string, but
> it was a java.lang.Integer
> at
> org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:219)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:172)
> at
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
> at
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
> at
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:274)
> at
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:179)
> at
> org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:132)
> at
> org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:126)
> at
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
> at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
> at
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.(VerifiableLog4jAppender.java:141)
> at
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.createFromArgs(VerifiableLog4jAppender.java:124)
> at
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.main(VerifiableLog4jAppender.java:158)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2373) Copycat distributed offset storage
[ https://issues.apache.org/jira/browse/KAFKA-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2373: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 202 [https://github.com/apache/kafka/pull/202] > Copycat distributed offset storage > -- > > Key: KAFKA-2373 > URL: https://issues.apache.org/jira/browse/KAFKA-2373 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.9.0.0 > > > Add offset storage for Copycat that works in distributed mode, which likely > means storing the data in a Kafka topic. Copycat workers will use this by > default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2554) change 0.8.3 to 0.9.0 in ApiVersion
[ https://issues.apache.org/jira/browse/KAFKA-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2554. - Resolution: Fixed Thanks for contributing! > change 0.8.3 to 0.9.0 in ApiVersion > --- > > Key: KAFKA-2554 > URL: https://issues.apache.org/jira/browse/KAFKA-2554 > Project: Kafka > Issue Type: Bug > Components: core >Reporter: Jun Rao >Assignee: Manikumar Reddy >Priority: Blocker > Fix For: 0.9.0.0 > > > Since we are renaming 0.8.3 to 0.9.0, we need to change the version in > ApiVersion accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2559) Add ducktape task to gradle
[ https://issues.apache.org/jira/browse/KAFKA-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901487#comment-14901487 ] Gwen Shapira commented on KAFKA-2559: - Your scenario is ideal. I'll be fine with a task that assumes I already setup the environment and started vagrants, because these failure scenarios are easier to figure out than a bad build. I'll leave the naming up to you :) > Add ducktape task to gradle > --- > > Key: KAFKA-2559 > URL: https://issues.apache.org/jira/browse/KAFKA-2559 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Will be awesome if we could just run: > "./gradlew clean ducktape" and have it build the jar and run all the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2559) Add ducktape task to gradle
Gwen Shapira created KAFKA-2559: --- Summary: Add ducktape task to gradle Key: KAFKA-2559 URL: https://issues.apache.org/jira/browse/KAFKA-2559 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Will be awesome if we could just run: "./gradlew clean ducktape" and have it build the jar and run all the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2549) Checkstyle reporting failure in trunk due to unused imports in Selector.java
[ https://issues.apache.org/jira/browse/KAFKA-2549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2549. - Resolution: Fixed Reviewer: Gwen Shapira Thanks for the quick fix! > Checkstyle reporting failure in trunk due to unused imports in Selector.java > > > Key: KAFKA-2549 > URL: https://issues.apache.org/jira/browse/KAFKA-2549 > Project: Kafka > Issue Type: Bug >Reporter: Parth Brahmbhatt >Assignee: Parth Brahmbhatt >Priority: Blocker > > Introduced in > https://github.com/apache/kafka/commit/d02ca36ca1cccdb6962191b97f54ce96b9d75abc#diff-db8f8be6ef2f1c81515d1ed83b3ab107 > in which the Selector.java was modified with some unused imports so the > trunk can not execute test targets as it fails in client section during > checkstyle stage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2548) kafka-merge-pr tool fails to update JIRA with fix version 0.9.0.0
Gwen Shapira created KAFKA-2548:
---
Summary: kafka-merge-pr tool fails to update JIRA with fix version
0.9.0.0
Key: KAFKA-2548
URL: https://issues.apache.org/jira/browse/KAFKA-2548
Project: Kafka
Issue Type: Bug
Reporter: Gwen Shapira
Trying to update JIRA where the fix version is set to '0.9.0.0', I get the
following error:
{code}
Traceback (most recent call last):
File "kafka-merge-pr.py", line 474, in
main()
File "kafka-merge-pr.py", line 460, in main
resolve_jira_issues(commit_title, merged_refs, jira_comment)
File "kafka-merge-pr.py", line 317, in resolve_jira_issues
resolve_jira_issue(merge_branches, comment, jira_id)
File "kafka-merge-pr.py", line 285, in resolve_jira_issue
(major, minor, patch) = v.split(".")
ValueError: too many values to unpack
{code}
We need to handle 3 and 4 part versions (x.y.z and x.y.z.w)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2548) kafka-merge-pr tool fails to update JIRA with fix version 0.9.0.0
[
https://issues.apache.org/jira/browse/KAFKA-2548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2548:
Labels: newbie (was: )
> kafka-merge-pr tool fails to update JIRA with fix version 0.9.0.0
> -
>
> Key: KAFKA-2548
> URL: https://issues.apache.org/jira/browse/KAFKA-2548
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
> Labels: newbie
>
> Trying to update JIRA where the fix version is set to '0.9.0.0', I get the
> following error:
> {code}
> Traceback (most recent call last):
> File "kafka-merge-pr.py", line 474, in
> main()
> File "kafka-merge-pr.py", line 460, in main
> resolve_jira_issues(commit_title, merged_refs, jira_comment)
> File "kafka-merge-pr.py", line 317, in resolve_jira_issues
> resolve_jira_issue(merge_branches, comment, jira_id)
> File "kafka-merge-pr.py", line 285, in resolve_jira_issue
> (major, minor, patch) = v.split(".")
> ValueError: too many values to unpack
> {code}
> We need to handle 3 and 4 part versions (x.y.z and x.y.z.w)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2504) Stop logging WARN when client disconnects
[
https://issues.apache.org/jira/browse/KAFKA-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2504:
Resolution: Fixed
Reviewer: Gwen Shapira
Status: Resolved (was: Patch Available)
Pushed to trunk. Thanks for fixing, [~jholoman].
> Stop logging WARN when client disconnects
> -
>
> Key: KAFKA-2504
> URL: https://issues.apache.org/jira/browse/KAFKA-2504
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Jeff Holoman
> Fix For: 0.9.0.0
>
>
> I thought we fixed this one, but it came back. This can be fill logs and is
> fairly useless. Should be logged at DEBUG level:
> {code}
> [2015-09-02 12:05:59,743] WARN Error in I/O with connection to /10.191.0.36
> (org.apache.kafka.common.network.Selector)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at
> org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:111)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:81)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:296)
> at kafka.network.Processor.run(SocketServer.scala:405)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-2536) topics tool should allow users to alter topic configuration
Gwen Shapira created KAFKA-2536: --- Summary: topics tool should allow users to alter topic configuration Key: KAFKA-2536 URL: https://issues.apache.org/jira/browse/KAFKA-2536 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira When we added dynamic config, we added a kafka-config tool (which can be used to maintain configs for non-topic entities), and remove the capability from kafka-topic tool. Removing the capability from kafka-topic is: 1. Breaks backward compatibility in our most essential tools. This has significant impact on usability. 2. Kinda confusing that --create --config works but --alter --config does not. I suggest fixing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2529) Brokers should write current version to log when they first start
[ https://issues.apache.org/jira/browse/KAFKA-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2529. - Resolution: Fixed Sorry, missed the jira that fixed this. > Brokers should write current version to log when they first start > - > > Key: KAFKA-2529 > URL: https://issues.apache.org/jira/browse/KAFKA-2529 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > It is currently non-trivial to tell, by looking at log files, which version > of Kafka is the log from. > Having this information can be useful in some troubleshooting scenarios. We > are exposing this via JMX, but since troubleshooting usually involves asking > for logs, it will be nice if this information will be included there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2529) Brokers should write current version to log when they first start
Gwen Shapira created KAFKA-2529: --- Summary: Brokers should write current version to log when they first start Key: KAFKA-2529 URL: https://issues.apache.org/jira/browse/KAFKA-2529 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira It is currently non-trivial to tell, by looking at log files, which version of Kafka is the log from. Having this information can be useful in some troubleshooting scenarios. We are exposing this via JMX, but since troubleshooting usually involves asking for logs, it will be nice if this information will be included there. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2447) Add capability to KafkaLog4jAppender to be able to use SSL
[ https://issues.apache.org/jira/browse/KAFKA-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737793#comment-14737793 ] Gwen Shapira commented on KAFKA-2447: - I'm a bit uncomfortable having a completely untested component (this applied to entire KafkaLog4jAppender, not just the SSL bit). Does it make sense to add Ducktape-based tests for the appender? it will save you the manual testing for next time :) (This can be a followup patch, since the appender was never covered by tests) > Add capability to KafkaLog4jAppender to be able to use SSL > -- > > Key: KAFKA-2447 > URL: https://issues.apache.org/jira/browse/KAFKA-2447 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > With Kafka supporting SSL, it makes sense to augment KafkaLog4jAppender to be > able to use SSL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2489) System tests: update benchmark tests to run with new and old consumer
[ https://issues.apache.org/jira/browse/KAFKA-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2489: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 179 [https://github.com/apache/kafka/pull/179] > System tests: update benchmark tests to run with new and old consumer > - > > Key: KAFKA-2489 > URL: https://issues.apache.org/jira/browse/KAFKA-2489 > Project: Kafka > Issue Type: Bug >Reporter: Geoff Anderson >Assignee: Geoff Anderson >Priority: Blocker > Fix For: 0.8.3 > > > Update benchmark tests to run w/new consumer to help catch performance > regressions > For context: > https://www.mail-archive.com/dev@kafka.apache.org/msg33633.html > The new consumer was previously reaching getting good performance. However, a > recent report on the mailing list indicates it's dropped significantly. After > evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, > compared to 600+MB/s previously. Before release, we should get the > performance > back on par. > Some details about where the regression occurred from the mailing list > http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E > : > bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good > performance -> an error due to broker apparently not accepting the partition > assignment strategy. Since this commit seems to add heartbeats and the server > side code for partition assignment strategies, I assume we were missing > something on the client side and by filling in the server side, things > stopped > working. > bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or > a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the > perf > test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, but > that's still far less than the 600MB/s I saw on the earlier commits. > Ideally we would also at least have a system test in place for the new > consumer, even if regressions weren't automatically detected. It would at > least > allow for manually checking for regressions. This should not be difficult > since > there are already old consumer performance tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2453) enable new consumer in EndToEndLatency
[ https://issues.apache.org/jira/browse/KAFKA-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2453: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 158 [https://github.com/apache/kafka/pull/158] > enable new consumer in EndToEndLatency > -- > > Key: KAFKA-2453 > URL: https://issues.apache.org/jira/browse/KAFKA-2453 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jun Rao >Assignee: Ben Stopford >Priority: Blocker > Fix For: 0.8.3 > > > We need to add an option to enable the new consumer in EndToEndLatency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733244#comment-14733244 ] Gwen Shapira commented on KAFKA-2510: - No, the issue I'm trying to prevent is definitely not resolved by controlled shutdown. Here's the scenario (I realize it sounds a bit contrived, but I've seen it happen twice): * Shut down an entire Kafka cluster for maintenance (Kafka upgrade, OS upgrade, hardware upgrade, whatever) * Sysadmin deploys a configuration change via automated tool. The tool replaces all the configuration on the machine, including server.properties. * Unfortunately, the new server.properties has a typo in the logs.dir parameter, pointing to the wrong location. * Bring up the cluster. Everything looks normal for a while, but all historical data is gone. By the time you realize what went wrong, you face the choice of either getting your old data back and losing the last few hours / days of new data, or saying goodbye to your history. It is obviously the sysadmin fault for misconfiguring, but most other datastores would refuse to start under similar scenarios (i.e they have multiple sources of truth regarding the existing data and will not start under mismatches). It looks like we have the ability to make Kafka safer for our users, and I don't see a reason not to do so. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731629#comment-14731629 ] Gwen Shapira commented on KAFKA-2510: - [~jkreps] I agree that this capability is important to some uses cases. I also think that preventing sysadmins from accidentally losing data on an entire cluster can be an important thing to support. Which is why I think it should be configurable (like delete.topic.enable and unsafe.leader.election) - there are legit tradeoffs. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731403#comment-14731403 ] Gwen Shapira commented on KAFKA-2510: - the old data directory will be untouched, but because there are no real errors and everything will appear to be working. New data coming in starting with offset 0, consumers happily consuming this data, and a new __offsets topic. Few hours or days later, when you figured what happened - you are kind of stuck - what are you going to do with the old data directory? > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2516) Rename o.a.k.client.tools to o.a.k.tools
Gwen Shapira created KAFKA-2516: --- Summary: Rename o.a.k.client.tools to o.a.k.tools Key: KAFKA-2516 URL: https://issues.apache.org/jira/browse/KAFKA-2516 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Priority: Blocker Fix For: 0.8.3 Currently our new performance tools are in o.a.k.client.tools but packaged in kafka-tools not kafka-clients. This is a bit confusing. Since they deserve their own jar (you don't want our client tools packaged in your app), lets give them a separate package and call it o.a.k.tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2461) request logger no longer logs extra information in debug mode
[
https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729576#comment-14729576
]
Gwen Shapira commented on KAFKA-2461:
-
Well, the reason we have two different log statements (one for trace and one
for debug) is that the partition information can be HUGE on large clusters, and
not always needed when diagnosing performance issues (the main use of this
log). So we need one level with details (TRACE) and one level without (DEBUG).
> request logger no longer logs extra information in debug mode
> -
>
> Key: KAFKA-2461
> URL: https://issues.apache.org/jira/browse/KAFKA-2461
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
>Priority: Blocker
> Fix For: 0.8.3
>
>
> Currently request logging calls are identical for trace and debug:
> {code}
> if(requestLogger.isTraceEnabled)
> requestLogger.trace("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime,
> requestQueueTime, apiLocalTime, apiRemoteTime, responseQueueTime,
> responseSendTime))
> else if(requestLogger.isDebugEnabled)
> requestLogger.debug("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime, requestQueueTime,
> apiLocalTime, apiRemoteTime, responseQueueTime, responseSendTime))
> {code}
> I think in the past (3 refactoring steps ago), we used to print more
> information about specific topics and partitions in debug mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2461) request logger no longer logs extra information in debug mode
[
https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729540#comment-14729540
]
Gwen Shapira commented on KAFKA-2461:
-
TRIVIAL pull request that doesn't require a JIRA makes sense.
> request logger no longer logs extra information in debug mode
> -
>
> Key: KAFKA-2461
> URL: https://issues.apache.org/jira/browse/KAFKA-2461
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
>Priority: Blocker
> Fix For: 0.8.3
>
>
> Currently request logging calls are identical for trace and debug:
> {code}
> if(requestLogger.isTraceEnabled)
> requestLogger.trace("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime,
> requestQueueTime, apiLocalTime, apiRemoteTime, responseQueueTime,
> responseSendTime))
> else if(requestLogger.isDebugEnabled)
> requestLogger.debug("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime, requestQueueTime,
> apiLocalTime, apiRemoteTime, responseQueueTime, responseSendTime))
> {code}
> I think in the past (3 refactoring steps ago), we used to print more
> information about specific topics and partitions in debug mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
[ https://issues.apache.org/jira/browse/KAFKA-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729413#comment-14729413 ] Gwen Shapira commented on KAFKA-2510: - Note that if you accidentally manage to set an entire cluster to the wrong directory (easy when Chef or similar manages your configuration), you also lose the consumer offsets - so only clients that use external offset store will even notice that the data is gone. Losing ALL data in the cluster without any errors is a huge problem. > Prevent broker from re-replicating / losing data due to disk misconfiguration > - > > Key: KAFKA-2510 > URL: https://issues.apache.org/jira/browse/KAFKA-2510 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira > > Currently Kafka assumes that whatever it sees in the data directory is the > correct state of the data. > This means that if an admin mistakenly configures Chef to use wrong data > directory, one of the following can happen: > 1. The broker will replicate a bunch of partitions and take over the network > 2. If you did this to enough brokers, you can lose entire topics and > partitions. > We have information about existing topics, partitions and their ISR in > zookeeper. > We need a mode in which if a broker starts, is in ISR for a partition and > doesn't have any data or directory for the partition, the broker will issue a > huge ERROR in the log and refuse to do anything for the partition. > [~fpj] worked on the problem for ZK and had some ideas on what is required > here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2510) Prevent broker from re-replicating / losing data due to disk misconfiguration
Gwen Shapira created KAFKA-2510: --- Summary: Prevent broker from re-replicating / losing data due to disk misconfiguration Key: KAFKA-2510 URL: https://issues.apache.org/jira/browse/KAFKA-2510 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Currently Kafka assumes that whatever it sees in the data directory is the correct state of the data. This means that if an admin mistakenly configures Chef to use wrong data directory, one of the following can happen: 1. The broker will replicate a bunch of partitions and take over the network 2. If you did this to enough brokers, you can lose entire topics and partitions. We have information about existing topics, partitions and their ISR in zookeeper. We need a mode in which if a broker starts, is in ISR for a partition and doesn't have any data or directory for the partition, the broker will issue a huge ERROR in the log and refuse to do anything for the partition. [~fpj] worked on the problem for ZK and had some ideas on what is required here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2461) request logger no longer logs extra information in debug mode
[
https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2461:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Issue resolved by pull request 169
[https://github.com/apache/kafka/pull/169]
> request logger no longer logs extra information in debug mode
> -
>
> Key: KAFKA-2461
> URL: https://issues.apache.org/jira/browse/KAFKA-2461
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
>Priority: Blocker
> Fix For: 0.8.3
>
>
> Currently request logging calls are identical for trace and debug:
> {code}
> if(requestLogger.isTraceEnabled)
> requestLogger.trace("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime,
> requestQueueTime, apiLocalTime, apiRemoteTime, responseQueueTime,
> responseSendTime))
> else if(requestLogger.isDebugEnabled)
> requestLogger.debug("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime, requestQueueTime,
> apiLocalTime, apiRemoteTime, responseQueueTime, responseSendTime))
> {code}
> I think in the past (3 refactoring steps ago), we used to print more
> information about specific topics and partitions in debug mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-2447) Add capability to KafkaLog4jAppender to be able to use SSL
[ https://issues.apache.org/jira/browse/KAFKA-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728507#comment-14728507 ] Gwen Shapira commented on KAFKA-2447: - Agree. But as nice-to-have, not a blocker on release, IMO. I'm still horrified that people who care about security enough to want to encrypt Log4J are perfectly happy to put their keystore password in a log4j configuration file. WTF. It does seem to be fairly standard though, so I guess we can do it :) There are no unit tests here. Did you validate that this works? > Add capability to KafkaLog4jAppender to be able to use SSL > -- > > Key: KAFKA-2447 > URL: https://issues.apache.org/jira/browse/KAFKA-2447 > Project: Kafka > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > > With Kafka supporting SSL, it makes sense to augment KafkaLog4jAppender to be > able to use SSL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time
[ https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1387: Priority: Critical (was: Blocker) > Kafka getting stuck creating ephemeral node it has already created when two > zookeeper sessions are established in a very short period of time > - > > Key: KAFKA-1387 > URL: https://issues.apache.org/jira/browse/KAFKA-1387 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.1.1 >Reporter: Fedor Korotkiy >Assignee: Flavio Junqueira >Priority: Critical > Labels: newbie, patch, zkclient-problems > Fix For: 0.8.3 > > Attachments: KAFKA-1387.patch, kafka-1387.patch > > > Kafka broker re-registers itself in zookeeper every time handleNewSession() > callback is invoked. > https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala > > Now imagine the following sequence of events. > 1) Zookeeper session reestablishes. handleNewSession() callback is queued by > the zkClient, but not invoked yet. > 2) Zookeeper session reestablishes again, queueing callback second time. > 3) First callback is invoked, creating /broker/[id] ephemeral path. > 4) Second callback is invoked and it tries to create /broker/[id] path using > createEphemeralPathExpectConflictHandleZKBug() function. But the path is > already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting > stuck in the infinite loop. > Seems like controller election code have the same issue. > I'am able to reproduce this issue on the 0.8.1 branch from github using the > following configs. > # zookeeper > tickTime=10 > dataDir=/tmp/zk/ > clientPort=2101 > maxClientCnxns=0 > # kafka > broker.id=1 > log.dir=/tmp/kafka > zookeeper.connect=localhost:2101 > zookeeper.connection.timeout.ms=100 > zookeeper.sessiontimeout.ms=100 > Just start kafka and zookeeper and then pause zookeeper several times using > Ctrl-Z. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2258) Port mirrormaker_testsuite
[ https://issues.apache.org/jira/browse/KAFKA-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2258: Priority: Blocker (was: Major) > Port mirrormaker_testsuite > -- > > Key: KAFKA-2258 > URL: https://issues.apache.org/jira/browse/KAFKA-2258 > Project: Kafka > Issue Type: Sub-task > Components: system tests >Reporter: Geoff Anderson >Assignee: Geoff Anderson >Priority: Blocker > Fix For: 0.8.3 > > > Port mirrormaker_testsuite to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2257) port replication_testsuite
[ https://issues.apache.org/jira/browse/KAFKA-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2257: Priority: Blocker (was: Major) > port replication_testsuite > -- > > Key: KAFKA-2257 > URL: https://issues.apache.org/jira/browse/KAFKA-2257 > Project: Kafka > Issue Type: Sub-task > Components: system tests >Reporter: Geoff Anderson >Assignee: Geoff Anderson >Priority: Blocker > Fix For: 0.8.3 > > > Port subset of replication_testsuite to run on ducktape. Details to follow -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2259) port offset_management_testsuite
[ https://issues.apache.org/jira/browse/KAFKA-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2259: Priority: Blocker (was: Major) > port offset_management_testsuite > > > Key: KAFKA-2259 > URL: https://issues.apache.org/jira/browse/KAFKA-2259 > Project: Kafka > Issue Type: Sub-task > Components: system tests >Reporter: Geoff Anderson >Assignee: Geoff Anderson >Priority: Blocker > Fix For: 0.8.3 > > > Port to run on ducktape -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe
[ https://issues.apache.org/jira/browse/KAFKA-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2311: Reviewer: Jason Gustafson (was: Neha Narkhede) > Consumer's ensureNotClosed method not thread safe > - > > Key: KAFKA-2311 > URL: https://issues.apache.org/jira/browse/KAFKA-2311 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Tim Brooks >Assignee: Tim Brooks >Priority: Blocker > Fix For: 0.8.3 > > Attachments: KAFKA-2311.patch, KAFKA-2311.patch > > > When a call is to the consumer is made, the first check is to see that the > consumer is not closed. This variable is not volatile so there is no > guarantee previous stores will be visible before a read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2311) Consumer's ensureNotClosed method not thread safe
[ https://issues.apache.org/jira/browse/KAFKA-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2311: Priority: Major (was: Blocker) > Consumer's ensureNotClosed method not thread safe > - > > Key: KAFKA-2311 > URL: https://issues.apache.org/jira/browse/KAFKA-2311 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Tim Brooks >Assignee: Tim Brooks > Fix For: 0.8.3 > > Attachments: KAFKA-2311.patch, KAFKA-2311.patch > > > When a call is to the consumer is made, the first check is to see that the > consumer is not closed. This variable is not volatile so there is no > guarantee previous stores will be visible before a read. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2338) Warn users if they change max.message.bytes that they also need to update broker and consumer settings
[ https://issues.apache.org/jira/browse/KAFKA-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2338: Priority: Critical (was: Blocker) > Warn users if they change max.message.bytes that they also need to update > broker and consumer settings > -- > > Key: KAFKA-2338 > URL: https://issues.apache.org/jira/browse/KAFKA-2338 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.1 >Reporter: Ewen Cheslack-Postava >Assignee: Edward Ribeiro >Priority: Critical > Fix For: 0.8.3 > > Attachments: KAFKA-2338.patch, KAFKA-2338_2015-07-18_00:37:31.patch, > KAFKA-2338_2015-07-21_13:21:19.patch, KAFKA-2338_2015-08-24_14:32:38.patch > > > We already have KAFKA-1756 filed to more completely address this issue, but > it is waiting for some other major changes to configs to completely protect > users from this problem. > This JIRA should address the low hanging fruit to at least warn users of the > potential problems. Currently the only warning is in our documentation. > 1. Generate a warning in the kafka-topics.sh tool when they change this > setting on a topic to be larger than the default. This needs to be very > obvious in the output. > 2. Currently, the broker's replica fetcher isn't logging any useful error > messages when replication can't succeed because a message size is too large. > Logging an error here would allow users that get into a bad state to find out > why it is happening more easily. (Consumers should already be logging a > useful error message.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2454) Dead lock between delete log segment and shutting down.
[
https://issues.apache.org/jira/browse/KAFKA-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2454:
Priority: Major (was: Blocker)
> Dead lock between delete log segment and shutting down.
> ---
>
> Key: KAFKA-2454
> URL: https://issues.apache.org/jira/browse/KAFKA-2454
> Project: Kafka
> Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> When the broker shutdown, it will shutdown scheduler which grabs the
> scheduler lock then wait for all the threads in scheduler to shutdown.
> The dead lock will happen when the scheduled task try to delete old log
> segment, it will schedule a log delete task which also needs to acquire the
> scheduler lock. In this case the shutdown thread will hold scheduler lock and
> wait for the the log deletion thread to finish, but the log deletion thread
> will block on waiting for the scheduler lock.
> Related stack trace:
> {noformat}
> "Thread-1" #21 prio=5 os_prio=0 tid=0x7fe7601a7000 nid=0x1a4e waiting on
> condition [0x7fe7cf698000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000640d53540> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
> at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:94)
> - locked <0x000640b6d480> (a kafka.utils.KafkaScheduler)
> at
> kafka.server.KafkaServer$$anonfun$shutdown$4.apply$mcV$sp(KafkaServer.scala:352)
> at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:79)
> at kafka.utils.Logging$class.swallowWarn(Logging.scala:92)
> at kafka.utils.CoreUtils$.swallowWarn(CoreUtils.scala:51)
> at kafka.utils.Logging$class.swallow(Logging.scala:94)
> at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:51)
> at kafka.server.KafkaServer.shutdown(KafkaServer.scala:352)
> at
> kafka.server.KafkaServerStartable.shutdown(KafkaServerStartable.scala:42)
> at com.linkedin.kafka.KafkaServer.notifyShutdown(KafkaServer.java:99)
> at
> com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyShutdownListener(LifeCycleMgr.java:123)
> at
> com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyListeners(LifeCycleMgr.java:102)
> at
> com.linkedin.util.factory.lifecycle.LifeCycleMgr.notifyStop(LifeCycleMgr.java:82)
> - locked <0x000640b77bb0> (a java.util.ArrayDeque)
> at com.linkedin.util.factory.Generator.stop(Generator.java:177)
> - locked <0x000640b77bc8> (a java.lang.Object)
> at
> com.linkedin.offspring.servlet.OffspringServletRuntime.destroy(OffspringServletRuntime.java:82)
> at
> com.linkedin.offspring.servlet.OffspringServletContextListener.contextDestroyed(OffspringServletContextListener.java:51)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:813)
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:160)
> at
> org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:516)
> at com.linkedin.emweb.WebappContext.doStop(WebappContext.java:35)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
> - locked <0x0006400018b8> (a java.lang.Object)
> at
> com.linkedin.emweb.ContextBasedHandlerImpl.doStop(ContextBasedHandlerImpl.java:112)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
> - locked <0x000640001900> (a java.lang.Object)
> at
> com.linkedin.emweb.WebappDeployerImpl.stop(WebappDeployerImpl.java:349)
> at
> com.linkedin.emweb.WebappDeployerImpl.doStop(WebappDeployerImpl.java:414)
> - locked <0x0006400019c0> (a
> com.linkedin.emweb.MapBasedHandlerImpl)
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
> - locked <0x0006404ee8e8> (a java.lang.Object)
> at
> org.eclipse.jetty.util.component.AggregateLifeCycle.doStop(AggregateLifeCycle.java:107)
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:69)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.doStop(HandlerWrapper.java:108)
> at org.eclipse.jetty.server.Server.doStop(Server.java:338)
> at
> org.ecli
[jira] [Updated] (KAFKA-2504) Stop logging WARN when client disconnects
[
https://issues.apache.org/jira/browse/KAFKA-2504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2504:
Fix Version/s: 0.8.3
> Stop logging WARN when client disconnects
> -
>
> Key: KAFKA-2504
> URL: https://issues.apache.org/jira/browse/KAFKA-2504
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
> Fix For: 0.8.3
>
>
> I thought we fixed this one, but it came back. This can be fill logs and is
> fairly useless. Should be logged at DEBUG level:
> {code}
> [2015-09-02 12:05:59,743] WARN Error in I/O with connection to /10.191.0.36
> (org.apache.kafka.common.network.Selector)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at
> org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:111)
> at
> org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:81)
> at
> org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
> at
> org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
> at
> org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:296)
> at kafka.network.Processor.run(SocketServer.scala:405)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-2504) Stop logging WARN when client disconnects
Gwen Shapira created KAFKA-2504:
---
Summary: Stop logging WARN when client disconnects
Key: KAFKA-2504
URL: https://issues.apache.org/jira/browse/KAFKA-2504
Project: Kafka
Issue Type: Bug
Reporter: Gwen Shapira
I thought we fixed this one, but it came back. This can be fill logs and is
fairly useless. Should be logged at DEBUG level:
{code}
[2015-09-02 12:05:59,743] WARN Error in I/O with connection to /10.191.0.36
(org.apache.kafka.common.network.Selector)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at
org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:111)
at
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:81)
at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
at
org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
at
org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
at org.apache.kafka.common.network.Selector.poll(Selector.java:296)
at kafka.network.Processor.run(SocketServer.scala:405)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KAFKA-2503) Metrics should be better documented
Gwen Shapira created KAFKA-2503: --- Summary: Metrics should be better documented Key: KAFKA-2503 URL: https://issues.apache.org/jira/browse/KAFKA-2503 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Priority: Minor metric.reporters configuration is missing from our docs. In addition some explanation about the metric reporters and a pointer to list of available metrics will be helpful. Once we move away from Yammer, we will also need to document how to write a reporter for KafkaMetrics. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2492) Upgrade zkclient dependency to 0.6
[ https://issues.apache.org/jira/browse/KAFKA-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2492: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 184 [https://github.com/apache/kafka/pull/184] > Upgrade zkclient dependency to 0.6 > -- > > Key: KAFKA-2492 > URL: https://issues.apache.org/jira/browse/KAFKA-2492 > Project: Kafka > Issue Type: Task >Affects Versions: 0.8.2.1 >Reporter: Stevo Slavic >Assignee: Stevo Slavic >Priority: Trivial > Fix For: 0.8.3 > > > If zkclient does not get replaced with curator (via KAFKA-873) sooner please > consider upgrading zkclient dependency to recently released 0.6. > zkclient 0.6 has few important changes included like: > - > [fix|https://github.com/sgroschupf/zkclient/commit/0630c9c6e67ab49a51e80bfd939e4a0d01a69dfe] > to fail retryUntilConnected actions with clear exception in case client gets > closed > - [upgraded zookeeper dependency from 3.4.6 to > 3.4.3|https://github.com/sgroschupf/zkclient/commit/8975c1790f7f36cc5d4feea077df337fb1ddabdb] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2120: Fix Version/s: 0.8.3 > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Fix For: 0.8.3 > > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1927) Replace requests in kafka.api with requests in org.apache.kafka.common.requests
[ https://issues.apache.org/jira/browse/KAFKA-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1927: Fix Version/s: (was: 0.8.3) > Replace requests in kafka.api with requests in > org.apache.kafka.common.requests > --- > > Key: KAFKA-1927 > URL: https://issues.apache.org/jira/browse/KAFKA-1927 > Project: Kafka > Issue Type: Improvement >Reporter: Jay Kreps >Assignee: Gwen Shapira > Attachments: KAFKA-1927.patch > > > The common package introduced a better way of defining requests using a new > protocol definition DSL and also includes wrapper objects for these. > We should switch KafkaApis over to use these request definitions and consider > the scala classes deprecated (we probably need to retain some of them for a > while for the scala clients). > This will be a big improvement because > 1. We will have each request now defined in only one place (Protocol.java) > 2. We will have built-in support for multi-version requests > 3. We will have much better error messages (no more cryptic underflow errors) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2500) Make logEndOffset available in the 0.8.3 Consumer
[ https://issues.apache.org/jira/browse/KAFKA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726238#comment-14726238 ] Gwen Shapira commented on KAFKA-2500: - [~hachikuji], do you want to take a look at this one? > Make logEndOffset available in the 0.8.3 Consumer > - > > Key: KAFKA-2500 > URL: https://issues.apache.org/jira/browse/KAFKA-2500 > Project: Kafka > Issue Type: Improvement > Components: consumer >Affects Versions: 0.8.3 >Reporter: Will Funnell >Assignee: Neha Narkhede >Priority: Critical > Fix For: 0.8.3 > > > Originally created in the old consumer here: > https://issues.apache.org/jira/browse/KAFKA-1977 > The requirement is to create a snapshot from the Kafka topic but NOT do > continual reads after that point. For example you might be creating a backup > of the data to a file. > This ticket covers the addition of the functionality to the new consumer. > In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps > was to expose the high watermark, as maxEndOffset, from the FetchResponse > object through to each MessageAndMetadata object in order to be aware when > the consumer has reached the end of each partition. > The submitted patch achieves this by adding the maxEndOffset to the > PartitionTopicInfo, which is updated when a new message arrives in the > ConsumerFetcherThread and then exposed in MessageAndMetadata. > See here for discussion: > http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)
[
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-1566:
Fix Version/s: (was: 0.8.3)
> Kafka environment configuration (kafka-env.sh)
> --
>
> Key: KAFKA-1566
> URL: https://issues.apache.org/jira/browse/KAFKA-1566
> Project: Kafka
> Issue Type: Improvement
> Components: tools
>Reporter: Cosmin Lehene
>Assignee: Sriharsha Chintalapani
> Labels: newbie
> Attachments: KAFKA-1566.patch, KAFKA-1566_2015-02-21_21:57:02.patch,
> KAFKA-1566_2015-03-17_17:01:38.patch, KAFKA-1566_2015-03-17_17:19:23.patch
>
>
> It would be useful (especially for automated deployments) to have an
> environment configuration file that could be sourced from the launcher files
> (e.g. kafka-run-server.sh).
> This is how this could look like kafka-env.sh
> {code}
> export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseCompressedOops
> -XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC
> -XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35' %>"
> export KAFKA_HEAP_OPTS="'-Xmx1G -Xms1G' %>"
> export KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=/var/log/kafka"
> {code}
> kafka-server-start.sh
> {code}
> ...
> source $base_dir/config/kafka-env.sh
> ...
> {code}
> This approach is consistent with Hadoop and HBase. However the idea here is
> to be able to set these values in a single place without having to edit
> startup scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2143) Replicas get ahead of leader and fail
[
https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2143:
Fix Version/s: (was: 0.8.3)
> Replicas get ahead of leader and fail
> -
>
> Key: KAFKA-2143
> URL: https://issues.apache.org/jira/browse/KAFKA-2143
> Project: Kafka
> Issue Type: Bug
> Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Evan Huus
>Assignee: Jiangjie Qin
>
> On a cluster of 6 nodes, we recently saw a case where a single
> under-replicated partition suddenly appeared, replication lag spiked, and
> network IO spiked. The cluster appeared to recover eventually on its own,
> Looking at the logs, the thing which failed was partition 7 of the topic
> {{background_queue}}. It had an ISR of 1,4,3 and its leader at the time was
> 3. Here are the interesting log lines:
> On node 3 (the leader):
> {noformat}
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when
> processing fetch request for partition [background_queue,7] offset 3722949957
> from follower with correlation id 148185816. Possible cause: Request for
> offset 3722949957 but we only have log segments in the range 3648049863 to
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when
> processing fetch request for partition [background_queue,7] offset 3722949957
> from follower with correlation id 156007054. Possible cause: Request for
> offset 3722949957 but we only have log segments in the range 3648049863 to
> 3722949955. (kafka.server.ReplicaManager)
> [2015-04-23 16:50:13,960] INFO Partition [background_queue,7] on broker 3:
> Shrinking ISR for partition [background_queue,7] from 1,4,3 to 3
> (kafka.cluster.Partition)
> {noformat}
> Note that both replicas suddenly asked for an offset *ahead* of the available
> offsets.
> And on nodes 1 and 4 (the replicas) many occurrences of the following:
> {noformat}
> [2015-04-23 16:50:05,935] INFO Scheduling log segment 3648049863 for log
> background_queue-7 for deletion. (kafka.log.Log) (edited)
> {noformat}
> Based on my reading, this looks like the replicas somehow got *ahead* of the
> leader, asked for an invalid offset, got confused, and re-replicated the
> entire topic from scratch to recover (this matches our network graphs, which
> show 3 sending a bunch of data to 1 and 4).
> Taking a stab in the dark at the cause, there appears to be a race condition
> where replicas can receive a new offset before the leader has committed it
> and is ready to replicate?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2303) Fix for KAFKA-2235 LogCleaner offset map overflow causes another compaction failures
[
https://issues.apache.org/jira/browse/KAFKA-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2303:
Fix Version/s: (was: 0.8.3)
> Fix for KAFKA-2235 LogCleaner offset map overflow causes another compaction
> failures
>
>
> Key: KAFKA-2303
> URL: https://issues.apache.org/jira/browse/KAFKA-2303
> Project: Kafka
> Issue Type: Bug
> Components: core, log
>Affects Versions: 0.8.2.1
>Reporter: Alexander Demidko
>Assignee: Guozhang Wang
>
> We have rolled out the patch for KAFKA-2235 to our kafka cluster, and
> recently instead of
> {code}
> "kafka.log.LogCleaner - [kafka-log-cleaner-thread-0], Error due to
> java.lang.IllegalArgumentException: requirement failed: Attempt to add a new
> entry to a full offset map."
> {code}
> we started to see
> {code}
> kafka.log.LogCleaner - [kafka-log-cleaner-thread-0], Error due to
> java.lang.IllegalArgumentException: requirement failed: 131390902 messages in
> segment -cgstate-8/79840768.log but offset map can
> fit only 80530612. You can increase log.cleaner.dedupe.buffer.size or
> decrease log.cleaner.threads
> {code}
> So, we had to roll it back to avoid disk depletion although I'm not sure if
> it needs to be rolled back in trunk. This patch applies more strict checks
> than were in place before: even if there is only one unique key for a
> segment, cleanup will fail if this segment is too big.
> Does it make sense to eliminate a limit for the offset map slots count, for
> example to use an offset map backed by a memory mapped file?
> The limit of 80530612 slots comes from memory / bytesPerEntry, where memory
> is Int.MaxValue (we use only one cleaner thread) and bytesPerEntry is 8 +
> digest hash size. Might be wrong, but it seems if the overall number of
> unique keys per partition is more than 80M slots in an OffsetMap, compaction
> will always fail and cleaner thread will die.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2338) Warn users if they change max.message.bytes that they also need to update broker and consumer settings
[ https://issues.apache.org/jira/browse/KAFKA-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2338: Fix Version/s: (was: 0.8.3) > Warn users if they change max.message.bytes that they also need to update > broker and consumer settings > -- > > Key: KAFKA-2338 > URL: https://issues.apache.org/jira/browse/KAFKA-2338 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.2.1 >Reporter: Ewen Cheslack-Postava >Assignee: Edward Ribeiro > Attachments: KAFKA-2338.patch, KAFKA-2338_2015-07-18_00:37:31.patch, > KAFKA-2338_2015-07-21_13:21:19.patch, KAFKA-2338_2015-08-24_14:32:38.patch > > > We already have KAFKA-1756 filed to more completely address this issue, but > it is waiting for some other major changes to configs to completely protect > users from this problem. > This JIRA should address the low hanging fruit to at least warn users of the > potential problems. Currently the only warning is in our documentation. > 1. Generate a warning in the kafka-topics.sh tool when they change this > setting on a topic to be larger than the default. This needs to be very > obvious in the output. > 2. Currently, the broker's replica fetcher isn't logging any useful error > messages when replication can't succeed because a message size is too large. > Logging an error here would allow users that get into a bad state to find out > why it is happening more easily. (Consumers should already be logging a > useful error message.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2229) Phase 1: Requests and KafkaApis
[ https://issues.apache.org/jira/browse/KAFKA-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2229: Fix Version/s: (was: 0.8.3) > Phase 1: Requests and KafkaApis > --- > > Key: KAFKA-2229 > URL: https://issues.apache.org/jira/browse/KAFKA-2229 > Project: Kafka > Issue Type: Sub-task >Reporter: Andrii Biletskyi >Assignee: Andrii Biletskyi > Attachments: KAFKA-2229.patch, KAFKA-2229_2015-06-30_16:59:17.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2073) Replace TopicMetadata request/response with o.a.k.requests.metadata
[ https://issues.apache.org/jira/browse/KAFKA-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2073: Fix Version/s: (was: 0.8.3) > Replace TopicMetadata request/response with o.a.k.requests.metadata > --- > > Key: KAFKA-2073 > URL: https://issues.apache.org/jira/browse/KAFKA-2073 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira >Assignee: Andrii Biletskyi > > Replace TopicMetadata request/response with o.a.k.requests.metadata. > Note, this is more challenging that it appears because while the wire > protocol is identical, the objects are completely different. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2120: Fix Version/s: (was: 0.8.3) > Add a request timeout to NetworkClient > -- > > Key: KAFKA-2120 > URL: https://issues.apache.org/jira/browse/KAFKA-2120 > Project: Kafka > Issue Type: New Feature >Reporter: Jiangjie Qin >Assignee: Mayuresh Gharat > Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, > KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, > KAFKA-2120_2015-08-12_10:59:09.patch > > > Currently NetworkClient does not have a timeout setting for requests. So if > no response is received for a request due to reasons such as broker is down, > the request will never be completed. > Request timeout will also be used as implicit timeout for some methods such > as KafkaProducer.flush() and kafkaProducer.close(). > KIP-19 is created for this public interface change. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2442) QuotasTest should not fail when cpu is busy
[ https://issues.apache.org/jira/browse/KAFKA-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2442: Fix Version/s: (was: 0.8.3) > QuotasTest should not fail when cpu is busy > --- > > Key: KAFKA-2442 > URL: https://issues.apache.org/jira/browse/KAFKA-2442 > Project: Kafka > Issue Type: Bug >Reporter: Dong Lin >Assignee: Aditya Auradkar > > We observed that testThrottledProducerConsumer in QuotasTest may fail or > succeed randomly. It appears that the test may fail when the system is slow. > We can add timer in the integration test to avoid random failure. > See an example failure at > https://builds.apache.org/job/kafka-trunk-git-pr/166/console for patch > https://github.com/apache/kafka/pull/142. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1853) Unsuccessful suffix rename of expired LogSegment can leak open files and also leave the LogSegment in an invalid state
[ https://issues.apache.org/jira/browse/KAFKA-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1853: Fix Version/s: (was: 0.8.3) > Unsuccessful suffix rename of expired LogSegment can leak open files and also > leave the LogSegment in an invalid state > -- > > Key: KAFKA-1853 > URL: https://issues.apache.org/jira/browse/KAFKA-1853 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.1.1 >Reporter: jaikiran pai >Assignee: jaikiran pai > Attachments: KAFKA-1853_2015-01-20_22:04:29.patch, > KAFKA-1853_2015-01-24_10:48:08.patch, KAFKA-1853_2015-01-24_11:21:07.patch, > KAFKA-1853_2015-01-26_19:41:33.patch > > > As noted in this discussion in the user mailing list > http://mail-archives.apache.org/mod_mbox/kafka-users/201501.mbox/%3C54AE3661.8080007%40gmail.com%3E > an unsuccessful attempt at renaming the underlying files of a LogSegment can > lead to file leaks and also leave the LogSegment in an invalid state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1763) validate_index_log in system tests runs remotely but uses local paths
[
https://issues.apache.org/jira/browse/KAFKA-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-1763:
Fix Version/s: (was: 0.8.3)
> validate_index_log in system tests runs remotely but uses local paths
> -
>
> Key: KAFKA-1763
> URL: https://issues.apache.org/jira/browse/KAFKA-1763
> Project: Kafka
> Issue Type: Bug
> Components: system tests
>Affects Versions: 0.8.1.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1763.patch
>
>
> validate_index_log is the only validation step in the system tests that needs
> to execute a Kafka binary and it's currently doing so remotely, like the rest
> of the test binaries. However, this is probably incorrect since it looks like
> logs are synced back to the driver host and in other cases are operated on
> locally. It looks like validate_index_log mixes up local/remote paths,
> causing an exception in DumpLogSegments:
> {quote}
> 2014-11-10 12:09:57,665 - DEBUG - executing command [ssh vagrant@worker1 -o
> 'HostName 127.0.0.1' -o 'Port ' -o 'UserKnownHostsFile /dev/null' -o
> 'StrictHostKeyChecking no' -o 'PasswordAuthentication no' -o 'IdentityFile
> /Users/ewencp/.vagrant.d/insecure_private_key' -o 'IdentitiesOnly yes' -o
> 'LogLevel FATAL' '/opt/kafka/bin/kafka-run-class.sh
> kafka.tools.DumpLogSegments --file
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> --verify-index-only 2>&1'] (system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Dumping
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.index
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - Exception in thread "main"
> java.io.FileNotFoundException:
> /Users/ewencp/kafka.git/system_test/replication_testsuite/testcase_0008/logs/broker-3/kafka_server_3_logs/test_1-2/1294.log
> (No such file or directory) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at java.io.FileInputStream.open(Native
> Method) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> java.io.FileInputStream.(FileInputStream.java:146)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.utils.Utils$.openChannel(Utils.scala:162) (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.log.FileMessageSet.(FileMessageSet.scala:74)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$.kafka$tools$DumpLogSegments$$dumpIndex(DumpLogSegments.scala:108)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,673 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:80)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$$anonfun$main$1.apply(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments$.main(DumpLogSegments.scala:73)
> (kafka_system_test_utils)
> 2014-11-10 12:09:58,674 - DEBUG - at
> kafka.tools.DumpLogSegments.main(DumpLogSegments.scala)
> (kafka_system_test_utils)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1755) Improve error handling in log cleaner
[ https://issues.apache.org/jira/browse/KAFKA-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1755: Fix Version/s: (was: 0.8.3) > Improve error handling in log cleaner > - > > Key: KAFKA-1755 > URL: https://issues.apache.org/jira/browse/KAFKA-1755 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Labels: newbie++ > Attachments: KAFKA-1755.patch, KAFKA-1755_2015-02-23_14:29:54.patch, > KAFKA-1755_2015-02-26_10:54:50.patch > > > The log cleaner is a critical process when using compacted topics. > However, if there is any error in any topic (notably if a key is missing) > then the cleaner exits and all other compacted topics will also be adversely > affected - i.e., compaction stops across the board. > This can be improved by just aborting compaction for a topic on any error and > keep the thread from exiting. > Another improvement would be to reject messages without keys that are sent to > compacted topics although this is not enough by itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-438) Code cleanup in MessageTest
[ https://issues.apache.org/jira/browse/KAFKA-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-438: --- Fix Version/s: (was: 0.8.3) > Code cleanup in MessageTest > --- > > Key: KAFKA-438 > URL: https://issues.apache.org/jira/browse/KAFKA-438 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 0.7.1 >Reporter: Jim Plush >Priority: Trivial > Attachments: KAFKA-438 > > > While exploring the Unit Tests this class had an unused import statement, > some ambiguity on which HashMap implementation was being used and assignments > of function returns when not required. > Trivial stuff -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1967) Support more flexible serialization in Log4jAppender
[ https://issues.apache.org/jira/browse/KAFKA-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1967: Fix Version/s: (was: 0.8.3) > Support more flexible serialization in Log4jAppender > > > Key: KAFKA-1967 > URL: https://issues.apache.org/jira/browse/KAFKA-1967 > Project: Kafka > Issue Type: Improvement >Reporter: Jesse Yates >Priority: Minor > Attachments: kafka-1967-trunk.patch > > > It would be nice to allow subclasses of the standard KafkfaLog4jAppender to > be able to serialize the LoggingEvent however they chose, rather than always > having to write out a string. > A possible use case - the one I'm interested in - allows implementors to > convert the event to any sort of bytes. This means downstream consumers don't > lose data based on the logging format, but instead can get the entire event > to do with as they please -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2179) no graceful nor fast way to shutdown every broker without killing them
[ https://issues.apache.org/jira/browse/KAFKA-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2179: Fix Version/s: (was: 0.8.3) > no graceful nor fast way to shutdown every broker without killing them > -- > > Key: KAFKA-2179 > URL: https://issues.apache.org/jira/browse/KAFKA-2179 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.1.2 >Reporter: Joe Stein >Priority: Minor > > if you do a controlled shutdown of every broker at the same time the > controlled shutdown process spins out of control. Every leader can't go > anywhere because every broker is trying to controlled shutdown itself. The > result is the brokers take a long (variable) time before it eventually does > actually shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2470) kafka-producer-perf-test.sh can't configure all to request-num-acks
[ https://issues.apache.org/jira/browse/KAFKA-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2470: Fix Version/s: (was: 0.8.3) > kafka-producer-perf-test.sh can't configure all to request-num-acks > --- > > Key: KAFKA-2470 > URL: https://issues.apache.org/jira/browse/KAFKA-2470 > Project: Kafka > Issue Type: Bug > Components: clients, tools >Affects Versions: 0.8.2.1 > Environment: Linux >Reporter: Bo Wang > Original Estimate: 24h > Remaining Estimate: 24h > > For New Producer API, kafka-producer-perf-test.sh can't configure all to > request-num-acks : > bin]# ./kafka-producer-perf-test.sh --topic test --broker-list host:port > --messages 100 --message-size 200 --new-producer --sync --batch-size 1 > --request-num-acks all > Exception in thread "main" joptsimple.OptionArgumentConversionException: > Cannot convert argument 'all' of option ['request-num-acks'] to class > java.lang.Integer > at > joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:237) > at joptsimple.OptionSet.valuesOf(OptionSet.java:226) > at joptsimple.OptionSet.valueOf(OptionSet.java:170) > at > kafka.tools.ProducerPerformance$ProducerPerfConfig.(ProducerPerformance.scala:146) > at kafka.tools.ProducerPerformance$.main(ProducerPerformance.scala:42) > at kafka.tools.ProducerPerformance.main(ProducerPerformance.scala) > Caused by: joptsimple.internal.ReflectionException: > java.lang.NumberFormatException: For input string: "all" > at > joptsimple.internal.Reflection.reflectionException(Reflection.java:136) > at joptsimple.internal.Reflection.invoke(Reflection.java:123) > at > joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48) > at > joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:234) > ... 5 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2491) Update ErrorMapping with New Consumer Errors
[ https://issues.apache.org/jira/browse/KAFKA-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2491: Assignee: Jason Gustafson > Update ErrorMapping with New Consumer Errors > > > Key: KAFKA-2491 > URL: https://issues.apache.org/jira/browse/KAFKA-2491 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Minor > Fix For: 0.8.3 > > > Some errors used by the new consumer have not been added to ErrorMapping. > Until this class is removed, it should probably be kept consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2146) adding partition did not find the correct startIndex
[ https://issues.apache.org/jira/browse/KAFKA-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2146: Fix Version/s: (was: 0.8.3) > adding partition did not find the correct startIndex > - > > Key: KAFKA-2146 > URL: https://issues.apache.org/jira/browse/KAFKA-2146 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 0.8.2.0 >Reporter: chenshangan >Priority: Minor > Attachments: KAFKA-2146.patch > > > TopicCommand provide a tool to add partitions for existing topics. It try to > find the startIndex from existing partitions. There's a minor flaw in this > process, it try to use the first partition fetched from zookeeper as the > start partition, and use the first replica id in this partition as the > startIndex. > One thing, the first partition fetched from zookeeper is not necessary to be > the start partition. As partition id begin from zero, we should use partition > with id zero as the start partition. > The other, broker id does not necessary begin from 0, so the startIndex is > not necessary to be the first replica id in the start partition. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2378) Add Copycat embedded API
[ https://issues.apache.org/jira/browse/KAFKA-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2378: Assignee: Ewen Cheslack-Postava > Add Copycat embedded API > > > Key: KAFKA-2378 > URL: https://issues.apache.org/jira/browse/KAFKA-2378 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Much of the required Copycat API will exist from previous patches since any > main() method will need to do very similar operations. However, integrating > with any other Java code may require additional API support. > For example, one of the use cases when integrating with any stream processing > application will require knowing which topics will be written to. We will > need to add APIs to expose the topics a registered connector is writing to so > they can be consumed by a stream processing task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2417) Ducktape tests for SSL/TLS
[ https://issues.apache.org/jira/browse/KAFKA-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2417: Assignee: Geoff Anderson > Ducktape tests for SSL/TLS > -- > > Key: KAFKA-2417 > URL: https://issues.apache.org/jira/browse/KAFKA-2417 > Project: Kafka > Issue Type: Sub-task > Components: security >Reporter: Ismael Juma >Assignee: Geoff Anderson > Fix For: 0.8.3 > > > The tests should be complementary to the unit/integration tests written as > part of KAFKA-1685. > Things to consider: > * Upgrade/downgrade to turning on/off SSL > * Failure testing > * Expired/revoked certificates > * Renegotiation > Some changes to ducktape may be required for upgrade scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2376) Add Copycat metrics
[ https://issues.apache.org/jira/browse/KAFKA-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2376: Assignee: Ewen Cheslack-Postava > Add Copycat metrics > --- > > Key: KAFKA-2376 > URL: https://issues.apache.org/jira/browse/KAFKA-2376 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Copycat needs good metrics for monitoring since that will be the primary > insight into the health of connectors as they copy data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2449) Update mirror maker (MirrorMaker) docs
[ https://issues.apache.org/jira/browse/KAFKA-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2449: Assignee: Geoff Anderson > Update mirror maker (MirrorMaker) docs > -- > > Key: KAFKA-2449 > URL: https://issues.apache.org/jira/browse/KAFKA-2449 > Project: Kafka > Issue Type: Bug >Reporter: Geoff Anderson >Assignee: Geoff Anderson > Fix For: 0.8.3 > > > The Kafka docs on Mirror Maker state that it mirrors from N source clusters > to 1 destination, but this is no longer the case. Docs should be updated to > reflect that it mirrors from single source cluster to single target cluster. > Docs I've found where this should be updated: > http://kafka.apache.org/documentation.html#basic_ops_mirror_maker > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.
[ https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2254: Fix Version/s: (was: 0.8.3) > The shell script should be optimized , even kafka-run-class.sh has a syntax > error. > -- > > Key: KAFKA-2254 > URL: https://issues.apache.org/jira/browse/KAFKA-2254 > Project: Kafka > Issue Type: Bug > Components: build >Affects Versions: 0.8.2.1 > Environment: linux >Reporter: Bo Wang > Labels: client-script, kafka-run-class.sh, shell-script > Attachments: kafka-shell-script.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > kafka-run-class.sh 128 line has a syntax error(missing a space): > 127-loggc) > 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then > 129GC_LOG_ENABLED="true" > 130 fi > And use the ShellCheck to check the shell scripts, the results shows some > errors 、 warnings and notes： > https://github.com/koalaman/shellcheck/wiki/SC2068 > https://github.com/koalaman/shellcheck/wiki/Sc2046 > https://github.com/koalaman/shellcheck/wiki/Sc2086 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2075) Validate that all kafka.api requests has been removed and clean up compatibility code
[ https://issues.apache.org/jira/browse/KAFKA-2075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2075: Fix Version/s: (was: 0.8.3) > Validate that all kafka.api requests has been removed and clean up > compatibility code > - > > Key: KAFKA-2075 > URL: https://issues.apache.org/jira/browse/KAFKA-2075 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira > > Once we finished all other subtasks - the old kafka.api requests/responses > shouldn't be used anywhere. > We need to validate that the classes are indeed gone, remove the unittests > for serializing/deserializing them and clean up the compatibility code added > in KAFKA-2044. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2070) Replace OffsetRequest/response with ListOffsetRequest/response from org.apache.kafka.common.requests
[ https://issues.apache.org/jira/browse/KAFKA-2070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2070: Fix Version/s: (was: 0.8.3) > Replace OffsetRequest/response with ListOffsetRequest/response from > org.apache.kafka.common.requests > > > Key: KAFKA-2070 > URL: https://issues.apache.org/jira/browse/KAFKA-2070 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira > > Replace OffsetRequest/response with ListOffsetRequest/response from > org.apache.kafka.common.requests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2207) The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest fails consistently with the following exception:
[ https://issues.apache.org/jira/browse/KAFKA-2207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2207: Resolution: Cannot Reproduce Status: Resolved (was: Patch Available) I think this is no longer failing. > The testCannotSendToInternalTopic test method in ProducerFailureHandlingTest > fails consistently with the following exception: > - > > Key: KAFKA-2207 > URL: https://issues.apache.org/jira/browse/KAFKA-2207 > Project: Kafka > Issue Type: Bug >Reporter: Deepthi > Fix For: 0.8.3 > > Attachments: KAFKA-2207.patch > > > kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic FAILED > java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.TimeoutException: Failed to update metadata > after 3000 ms. > at > org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:437) > at > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:352) > at > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248) > at > kafka.api.ProducerFailureHandlingTest.testCannotSendToInternalTopic(ProducerFailureHandlingTest.scala:309) > Caused by: > org.apache.kafka.common.errors.TimeoutException: Failed to update > metadata after 3000 ms. > The following attached patch has resolved the issue -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1853) Unsuccessful suffix rename of expired LogSegment can leak open files and also leave the LogSegment in an invalid state
[ https://issues.apache.org/jira/browse/KAFKA-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1853: Assignee: jaikiran pai > Unsuccessful suffix rename of expired LogSegment can leak open files and also > leave the LogSegment in an invalid state > -- > > Key: KAFKA-1853 > URL: https://issues.apache.org/jira/browse/KAFKA-1853 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8.1.1 >Reporter: jaikiran pai >Assignee: jaikiran pai > Fix For: 0.8.3 > > Attachments: KAFKA-1853_2015-01-20_22:04:29.patch, > KAFKA-1853_2015-01-24_10:48:08.patch, KAFKA-1853_2015-01-24_11:21:07.patch, > KAFKA-1853_2015-01-26_19:41:33.patch > > > As noted in this discussion in the user mailing list > http://mail-archives.apache.org/mod_mbox/kafka-users/201501.mbox/%3C54AE3661.8080007%40gmail.com%3E > an unsuccessful attempt at renaming the underlying files of a LogSegment can > lead to file leaks and also leave the LogSegment in an invalid state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2064) Replace ConsumerMetadataRequest and Response with org.apache.kafka.common.requests objects
[ https://issues.apache.org/jira/browse/KAFKA-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2064: Fix Version/s: (was: 0.8.3) > Replace ConsumerMetadataRequest and Response with > org.apache.kafka.common.requests objects > --- > > Key: KAFKA-2064 > URL: https://issues.apache.org/jira/browse/KAFKA-2064 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira > > Replace ConsumerMetadataRequest and response with > org.apache.kafka.common.requests objects -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2066) Replace FetchRequest / FetchResponse with their org.apache.kafka.common.requests equivalents
[ https://issues.apache.org/jira/browse/KAFKA-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2066: Fix Version/s: (was: 0.8.3) > Replace FetchRequest / FetchResponse with their > org.apache.kafka.common.requests equivalents > > > Key: KAFKA-2066 > URL: https://issues.apache.org/jira/browse/KAFKA-2066 > Project: Kafka > Issue Type: Sub-task >Reporter: Gwen Shapira > > Replace FetchRequest / FetchResponse with their > org.apache.kafka.common.requests equivalents. > Note that they can't be completely removed until we deprecate the > SimpleConsumer API (and it will require very careful patchwork for the places > where core modules actually use the SimpleConsumer API). > This also requires a solution on how to stream from memory-mapped files > (similar to what existing code does with FileMessageSet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2375) Implement elasticsearch Copycat sink connector
[ https://issues.apache.org/jira/browse/KAFKA-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2375: Fix Version/s: (was: 0.8.3) > Implement elasticsearch Copycat sink connector > -- > > Key: KAFKA-2375 > URL: https://issues.apache.org/jira/browse/KAFKA-2375 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava > > Implement an elasticsearch sink connector for Copycat. This should send > records to elasticsearch with unique document IDs, given appropriate configs > to extract IDs from input records. > The motivation here is to provide a good end-to-end example with built-in > connectors that require minimal dependencies. Because Elasticsearch has a > very simple REST API, an elasticsearch connector shouldn't require any extra > dependencies and logs -> Elasticsearch (in combination with KAFKA-2374) > provides a compelling out-of-the-box Copycat use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1898) compatibility testing framework
[
https://issues.apache.org/jira/browse/KAFKA-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-1898:
Fix Version/s: (was: 0.8.3)
> compatibility testing framework
>
>
> Key: KAFKA-1898
> URL: https://issues.apache.org/jira/browse/KAFKA-1898
> Project: Kafka
> Issue Type: Bug
>Reporter: Joe Stein
> Attachments: cctk.png
>
>
> There are a few different scenarios where you want/need to know the
> status/state of a client library that works with Kafka. Client library
> development is not just about supporting the wire protocol but also the
> implementations around specific interactions of the API. The API has
> blossomed into a robust set of producer, consumer, broker and administrative
> calls all of which have layers of logic above them. A Client Library may
> choose to deviate from the path the project sets out and that is ok. The goal
> of this ticket is to have a system for Kafka that can help to explain what
> the library is or isn't doing (regardless of what it claims).
> The idea behind this stems in being able to quickly/easily/succinctly analyze
> the topic message data. Once you can analyze the topic(s) message you can
> gather lots of information about what the client library is doing, is not
> doing and such. There are a few components to this.
> 1) dataset-generator
> Test Kafka dataset generation tool. Generates a random text file with given
> params:
> --filename, -f - output file name.
> --filesize, -s - desired size of output file. The actual size will always be
> a bit larger (with a maximum size of $filesize + $max.length - 1)
> --min.length, -l - minimum generated entry length.
> --max.length, -h - maximum generated entry length.
> Usage:
> ./gradlew build
> java -jar dataset-generator/build/libs/dataset-generator-*.jar -s 10 -l 2
> -h 20
> 2) dataset-producer
> Test Kafka dataset producer tool. Able to produce the given dataset to Kafka
> or Syslog server. The idea here is you already have lots of data sets that
> you want to test different things for. You might have different sized
> messages, formats, etc and want a repeatable benchmark to run and re-run the
> testing on. You could just have a days worth of data and just choose to
> replay it. The CCTK idea is that you are always starting from CONSUME in
> your state of library. If your library is only producing then you will fail a
> bunch of tests and that might be ok for people.
> Accepts following params:
> {code}
> --filename, -f - input file name.
> --kafka, -k - Kafka broker address in host:port format. If this parameter is
> set, --producer.config and --topic must be set too (otherwise they're
> ignored).
> --producer.config, -p - Kafka producer properties file location.
> --topic, -t - Kafka topic to produce to.
> --syslog, -s - Syslog server address. Format: protocol://host:port
> (tcp://0.0.0.0:5140 or udp://0.0.0.0:5141 for example)
> --loop, -l - flag to loop through file until shut off manually. False by
> default.
> Usage:
> ./gradlew build
> java -jar dataset-producer/build/libs/dataset-producer-*.jar --filename
> dataset --syslog tcp://0.0.0.0:5140 --loop true
> {code}
> 3) extract
> This step is good so you can save data and compare tests. It could also be
> removed if folks are just looking for a real live test (and we could support
> that too). Here we are taking data out of Kafka and putting it into
> Cassandra (but other data stores can be used too and we should come up with a
> way to abstract this out completely so folks could implement whatever they
> wanted.
> {code}
> package ly.stealth.shaihulud.reader
> import java.util.UUID
> import com.datastax.spark.connector._
> import com.datastax.spark.connector.cql.CassandraConnector
> import consumer.kafka.MessageAndMetadata
> import consumer.kafka.client.KafkaReceiver
> import org.apache.spark._
> import org.apache.spark.storage.StorageLevel
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.dstream.DStream
> object Main extends App with Logging {
> val parser = new scopt.OptionParser[ReaderConfiguration]("spark-reader") {
> head("Spark Reader for Kafka client applications", "1.0")
> opt[String]("testId") unbounded() optional() action { (x, c) =>
> c.copy(testId = x)
> } text ("Source topic with initial set of data")
> opt[String]("source") unbounded() required() action { (x, c) =>
> c.copy(sourceTopic = x)
> } text ("Source topic with initial set of data")
> opt[String]("destination") unbounded() required() action { (x, c) =>
> c.copy(destinationTopic = x)
> } text ("Destination topic with processed set of data")
> opt[Int]("partitions") unbounded() optional() action { (x, c) =>
> c.copy(partitions = x)
> } text ("Partitions in topic")
[jira] [Resolved] (KAFKA-2486) New consumer performance
[ https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2486. - Resolution: Fixed Issue resolved by pull request 180 [https://github.com/apache/kafka/pull/180] > New consumer performance > > > Key: KAFKA-2486 > URL: https://issues.apache.org/jira/browse/KAFKA-2486 > Project: Kafka > Issue Type: Sub-task > Components: consumer >Reporter: Ewen Cheslack-Postava >Assignee: Jason Gustafson > Fix For: 0.8.3 > > > The new consumer was previously reaching getting good performance. However, a > recent report on the mailing list indicates it's dropped significantly. After > evaluation, even with a local broker it seems to only be reaching a 2-10MB/s, > compared to 600+MB/s previously. Before release, we should get the > performance back on par. > Some details about where the regression occurred from the mailing list > http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E > : > bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good > performance -> an error due to broker apparently not accepting the partition > assignment strategy. Since this commit seems to add heartbeats and the server > side code for partition assignment strategies, I assume we were missing > something on the client side and by filling in the server side, things > stopped working. > bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or > a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the > perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing, > but that's still far less than the 600MB/s I saw on the earlier commits. > Ideally we would also at least have a system test in place for the new > consumer, even if regressions weren't automatically detected. It would at > least allow for manually checking for regressions. This should not be > difficult since there are already old consumer performance tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2485) Allow producer performance to take properties from a file via --producer.config command line parameter
[ https://issues.apache.org/jira/browse/KAFKA-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2485: Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Issue resolved by pull request 174 [https://github.com/apache/kafka/pull/174] > Allow producer performance to take properties from a file via > --producer.config command line parameter > -- > > Key: KAFKA-2485 > URL: https://issues.apache.org/jira/browse/KAFKA-2485 > Project: Kafka > Issue Type: Improvement >Reporter: Dong Lin >Assignee: Dong Lin > Fix For: 0.8.3 > > > In order to allow ProducerPerformance to produce via ssl connection, we need > the following (example) configuration: > security.protocol=SSL > ssl.protocol=TLS > ssl.truststore.type=JKS > ssl.truststore.location=serverkeystore.jks > ssl.truststore.password=password > ssl.keymanager.algorithm=SunX509 > ssl.trustmanager.algorithm=SunX509 > There are two ways to achieve it: 1) extend ProducerPerformance to explicitly > accept all 7 ssl-related parameters; 2) allow ProducerPerformance to take > properties from a file via --consumer.config command line parameter. > It seems option 2) is better, because it requires less code, allows new > options to be easily added in the future, and doesn't require user to specify > password in the command line. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2475) Reduce copycat configs to only specify a converter or serializer, not both
[ https://issues.apache.org/jira/browse/KAFKA-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2475: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 172 [https://github.com/apache/kafka/pull/172] > Reduce copycat configs to only specify a converter or serializer, not both > -- > > Key: KAFKA-2475 > URL: https://issues.apache.org/jira/browse/KAFKA-2475 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Ultimately, all we care about is getting from byte[] -> Copycat data API. The > current split of the interfaces makes it easy to reuse existing serializers, > but you still have to implement a new class. > It will be simpler, both conceptually and by requiring fewer configs, to > combine both these steps into a single API. This also allows certain formats > to preserve more information across these (e.g. for primitive types in > schema, which otherwise could lose certain schema information). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2377) Add copycat system tests
[ https://issues.apache.org/jira/browse/KAFKA-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2377: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 150 [https://github.com/apache/kafka/pull/150] > Add copycat system tests > > > Key: KAFKA-2377 > URL: https://issues.apache.org/jira/browse/KAFKA-2377 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Add baseline system tests for Copycat, covering both standalone and > distributed mode. > This should cover basic failure modes and verify at-least-one delivery of > data, both from source system -> Kafka and Kafka -> sink system. This, of > course, requires testing the core, built-in connectors provided with Copycat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2367: Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request 163 [https://github.com/apache/kafka/pull/163] > Add Copycat runtime data API > > > Key: KAFKA-2367 > URL: https://issues.apache.org/jira/browse/KAFKA-2367 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Design the API used for runtime data in Copycat. This API is used to > construct schemas and records that Copycat processes. This needs to be a > fairly general data model (think Avro, JSON, Protobufs, Thrift) in order to > support complex, varied data types that may be input from/output to many data > systems. > This should issue should also address the serialization interfaces used > within Copycat, which translate the runtime data into serialized byte[] form. > It is important that these be considered together because the data format can > be used in multiple ways (records, partition IDs, partition offsets), so it > and the corresponding serializers must be sufficient for all these use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-1683) Implement a "session" concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1683. - Resolution: Fixed Issue resolved by pull request 155 [https://github.com/apache/kafka/pull/155] > Implement a "session" concept in the socket server > -- > > Key: KAFKA-1683 > URL: https://issues.apache.org/jira/browse/KAFKA-1683 > Project: Kafka > Issue Type: Sub-task > Components: security >Affects Versions: 0.8.2.1 >Reporter: Jay Kreps >Assignee: Gwen Shapira > Fix For: 0.8.3 > > Attachments: KAFKA-1683.patch, KAFKA-1683.patch > > > To implement authentication we need a way to keep track of some things > between requests. The initial use for this would be remembering the > authenticated user/principle info, but likely more uses would come up (for > example we will also need to remember whether and which encryption or > integrity measures are in place on the socket so we can wrap and unwrap > writes and reads). > I was thinking we could just add a Session object that might have a user > field. The session object would need to get added to RequestChannel.Request > so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715409#comment-14715409 ] Gwen Shapira commented on KAFKA-1977: - I lost track of the new consumer APIs, so I'm ok with whatever [~hachikuji] thinks :) > Make logEndOffset available in the Zookeeper consumer > - > > Key: KAFKA-1977 > URL: https://issues.apache.org/jira/browse/KAFKA-1977 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Will Funnell >Priority: Minor > Attachments: > Make_logEndOffset_available_in_the_Zookeeper_consumer.patch > > > The requirement is to create a snapshot from the Kafka topic but NOT do > continual reads after that point. For example you might be creating a backup > of the data to a file. > In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps > was to expose the high watermark, as maxEndOffset, from the FetchResponse > object through to each MessageAndMetadata object in order to be aware when > the consumer has reached the end of each partition. > The submitted patch achieves this by adding the maxEndOffset to the > PartitionTopicInfo, which is updated when a new message arrives in the > ConsumerFetcherThread and then exposed in MessageAndMetadata. > See here for discussion: > http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712401#comment-14712401 ] Gwen Shapira commented on KAFKA-2367: - My opinions / comments on the above: * Logical types are important (timestamp, decimals) - but can be done separately * I don't see much value in non-ordered versions (it doesn't give you any information you don't get from the schema itself), so an int makes more sense. We also need to support lack of versions. While there, we should also support lack of schema (since some connectors won't know the schema or won't care) * If we recommend ByteBuffer, I don't know if we should offer byte[]. I don't feel strongly either way though. * I like the new explicit schemas! Much less error-prone. * Follow up jira for caching, its important. * Lack of unions is fine, IMO Avro unions are somewhat misused and we have better support for nulls ;) * Lack of IndexedRecord is fine given the current API. * I agree with how you defined the default+nullable behavior and will watch out during the review. > Add Copycat runtime data API > > > Key: KAFKA-2367 > URL: https://issues.apache.org/jira/browse/KAFKA-2367 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Design the API used for runtime data in Copycat. This API is used to > construct schemas and records that Copycat processes. This needs to be a > fairly general data model (think Avro, JSON, Protobufs, Thrift) in order to > support complex, varied data types that may be input from/output to many data > systems. > This should issue should also address the serialization interfaces used > within Copycat, which translate the runtime data into serialized byte[] form. > It is important that these be considered together because the data format can > be used in multiple ways (records, partition IDs, partition offsets), so it > and the corresponding serializers must be sufficient for all these use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2461) request logger no longer logs extra information in debug mode
[
https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711994#comment-14711994
]
Gwen Shapira commented on KAFKA-2461:
-
We'll need it in 0.8.3, since we are losing diagnosis capabilities without this
(the ability to attribute specific request messages to topics)
> request logger no longer logs extra information in debug mode
> -
>
> Key: KAFKA-2461
> URL: https://issues.apache.org/jira/browse/KAFKA-2461
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
> Fix For: 0.8.3
>
>
> Currently request logging calls are identical for trace and debug:
> {code}
> if(requestLogger.isTraceEnabled)
> requestLogger.trace("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime,
> requestQueueTime, apiLocalTime, apiRemoteTime, responseQueueTime,
> responseSendTime))
> else if(requestLogger.isDebugEnabled)
> requestLogger.debug("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime, requestQueueTime,
> apiLocalTime, apiRemoteTime, responseQueueTime, responseSendTime))
> {code}
> I think in the past (3 refactoring steps ago), we used to print more
> information about specific topics and partitions in debug mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2461) request logger no longer logs extra information in debug mode
[
https://issues.apache.org/jira/browse/KAFKA-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gwen Shapira updated KAFKA-2461:
Fix Version/s: 0.8.3
> request logger no longer logs extra information in debug mode
> -
>
> Key: KAFKA-2461
> URL: https://issues.apache.org/jira/browse/KAFKA-2461
> Project: Kafka
> Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Ashish K Singh
> Fix For: 0.8.3
>
>
> Currently request logging calls are identical for trace and debug:
> {code}
> if(requestLogger.isTraceEnabled)
> requestLogger.trace("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime,
> requestQueueTime, apiLocalTime, apiRemoteTime, responseQueueTime,
> responseSendTime))
> else if(requestLogger.isDebugEnabled)
> requestLogger.debug("Completed request:%s from connection
> %s;totalTime:%d,requestQueueTime:%d,localTime:%d,remoteTime:%d,responseQueueTime:%d,sendTime:%d"
> .format(requestDesc, connectionId, totalTime, requestQueueTime,
> apiLocalTime, apiRemoteTime, responseQueueTime, responseSendTime))
> {code}
> I think in the past (3 refactoring steps ago), we used to print more
> information about specific topics and partitions in debug mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-2367) Add Copycat runtime data API
[ https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2367: Status: Patch Available (was: Open) > Add Copycat runtime data API > > > Key: KAFKA-2367 > URL: https://issues.apache.org/jira/browse/KAFKA-2367 > Project: Kafka > Issue Type: Sub-task > Components: copycat >Reporter: Ewen Cheslack-Postava >Assignee: Ewen Cheslack-Postava > Fix For: 0.8.3 > > > Design the API used for runtime data in Copycat. This API is used to > construct schemas and records that Copycat processes. This needs to be a > fairly general data model (think Avro, JSON, Protobufs, Thrift) in order to > support complex, varied data types that may be input from/output to many data > systems. > This should issue should also address the serialization interfaces used > within Copycat, which translate the runtime data into serialized byte[] form. > It is important that these be considered together because the data format can > be used in multiple ways (records, partition IDs, partition offsets), so it > and the corresponding serializers must be sufficient for all these use cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2122) Remove controller.message.queue.size Config
[ https://issues.apache.org/jira/browse/KAFKA-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2122: Fix Version/s: 0.8.3 > Remove controller.message.queue.size Config > --- > > Key: KAFKA-2122 > URL: https://issues.apache.org/jira/browse/KAFKA-2122 > Project: Kafka > Issue Type: Bug >Reporter: Onur Karaman >Assignee: Sriharsha Chintalapani > Fix For: 0.8.3 > > Attachments: KAFKA-2122.patch, KAFKA-2122_2015-04-19_12:44:41.patch > > > A deadlock can happen during a delete topic if controller.message.queue.size > is overridden to a custom value. Details are here: > https://issues.apache.org/jira/browse/KAFKA-2046?focusedCommentId=14380776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14380776 > Given that KAFKA-1993 is enabling delete topic by default, it would be unsafe > to simultaneously allow a configurable controller.message.queue.size -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2472) Fix kafka ssl configs to not throw warnings
[ https://issues.apache.org/jira/browse/KAFKA-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711500#comment-14711500 ] Gwen Shapira commented on KAFKA-2472: - I'm wondering if we still need the unknown config warnings at all - the entire system was designed to pass arguments to SSL, serializers, reporters, etc. Unknown configs are expected at this point. > Fix kafka ssl configs to not throw warnings > --- > > Key: KAFKA-2472 > URL: https://issues.apache.org/jira/browse/KAFKA-2472 > Project: Kafka > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 0.8.3 > > > This is a follow-up fix on kafka-1690. > [2015-08-25 18:20:48,236] WARN The configuration ssl.truststore.password = > striker was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig) > [2015-08-25 18:20:48,236] WARN The configuration security.protocol = SSL was > supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
