[jira] [Created] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

Ciprian Pascu created KAFKA-4788: Summary: Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic. Key: KAFKA-4788 URL: https://issues.apache.org/jira/browse/KAFKA-47

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880190#comment-15880190 ] Ismael Juma commented on KAFKA-4788: Thanks for the report. What was the most recent v

[jira] [Created] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

Hamidreza Afzali created KAFKA-4789: --- Summary: ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics Key: KAFKA-4789 URL: https://issues.apache.org/jira/browse/KAFKA-4789

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880268#comment-15880268 ] Ciprian Pascu commented on KAFKA-4788: -- I am using command like this to create the to

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880272#comment-15880272 ] Ciprian Pascu commented on KAFKA-4788: -- One addition: I have log.segment.bytes=209715

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

@Matthias about the point 9: What about keeping only the --topic option, and support this format: `--topic t1:0,1,2 --topic t2 --topic t3:2` In this case topics t1, t2, and t3 will be selected: topic t1 with partitions 0,1 and 2; topic t2 with all its partitions; and topic t3, with only partitio

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880281#comment-15880281 ] Manikumar commented on KAFKA-4788: -- looks like KAFKA-4092 change introduced this issue.

[GitHub] kafka pull request #2587: KAFKA-4461: Added support to ProcessorTopologyTest...

GitHub user hrafzali opened a pull request: https://github.com/apache/kafka/pull/2587 KAFKA-4461: Added support to ProcessorTopologyTestDriver to forward timestamps to internal topics This resolves the issue in the ProcessorTopologyTestDriver that the extracted timestamp is not for

[jira] [Commented] (KAFKA-4461) When using ProcessorTopologyTestDriver, the combination of map and .groupByKey does not produce any result

[ https://issues.apache.org/jira/browse/KAFKA-4461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880293#comment-15880293 ] ASF GitHub Bot commented on KAFKA-4461: --- GitHub user hrafzali opened a pull request:

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880300#comment-15880300 ] Ismael Juma commented on KAFKA-4788: [~omkreddy], would you like to provide a PR? I'll

[jira] [Updated] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4788: --- Labels: regression (was: ) > Broker level configuration 'log.segment.bytes' not used when 'segment.by

Re: [DISCUSS] KIP-124: Request rate quotas

Guozhang/Dong, Thank you for the feedback. Guozhang : I have updated the section on co-existence of byte rate and request time quotas. Dong: I hadn't added much detail to the metrics and sensors since they are going to be very similar to the existing metrics and sensors. To avoid confusion, I ha

[jira] [Updated] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4788: --- Fix Version/s: 0.10.2.1 0.10.3.0 > Broker level configuration 'log.segment.bytes' n

Re: [DISCUSS] KIP-82 - Add Record Headers

append-only would mean that if (for whatever reason) i want to replace a header or strip it out i'd need to copy the whole record. On Wed, Feb 22, 2017 at 5:09 PM, Michael Pearce wrote: > Im happy to compromise to keep it mutable but move to an append style api. > (as in guava interables conca

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[ https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880367#comment-15880367 ] Armin Braun commented on KAFKA-1895: I think the way the threading works is somewhat s

[jira] [Created] (KAFKA-4790) Kafka can't not recover after a disk full

Pengwei created KAFKA-4790: -- Summary: Kafka can't not recover after a disk full Key: KAFKA-4790 URL: https://issues.apache.org/jira/browse/KAFKA-4790 Project: Kafka Issue Type: Bug Affects Versi

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880385#comment-15880385 ] Manikumar commented on KAFKA-4788: -- [~ijuma] While using kafka-topics.sh, we are doing t

[jira] [Comment Edited] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880385#comment-15880385 ] Manikumar edited comment on KAFKA-4788 at 2/23/17 1:03 PM: --- [~ij

[jira] [Updated] (KAFKA-4790) Kafka can't not recover after a disk full

[ https://issues.apache.org/jira/browse/KAFKA-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4790: --- Labels: reliability (was: ) > Kafka can't not recover after a disk full > ---

[jira] [Reopened] (KAFKA-4092) retention.bytes should not be allowed to be less than segment.bytes

[ https://issues.apache.org/jira/browse/KAFKA-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4092: > retention.bytes should not be allowed to be less than segment.bytes >

[GitHub] kafka pull request #2588: KAFKA-4788: Revert "KAFKA-4092: retention.bytes sh...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/2588 KAFKA-4788: Revert "KAFKA-4092: retention.bytes should not be allowed to be less than segment.bytes" The intent is good, but it needs to take into account broker configs as well. See KAFKA-4788 f

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880409#comment-15880409 ] ASF GitHub Bot commented on KAFKA-4788: --- GitHub user ijuma opened a pull request:

[jira] [Assigned] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram reassigned KAFKA-4779: - Assignee: Rajini Sivaram > Failure in kafka/tests/kafkatest/tests/core/security_rolling_u

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880453#comment-15880453 ] Ismael Juma commented on KAFKA-4788: [~omkreddy], I think KAFKA-4092 is valuable, but

[jira] [Commented] (KAFKA-4790) Kafka can't not recover after a disk full

[ https://issues.apache.org/jira/browse/KAFKA-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880567#comment-15880567 ] Pengwei commented on KAFKA-4790: The index max file set to 1MB log.index.size.max.bytes=10

[GitHub] kafka pull request #2587: KAFKA-4789: Added support to ProcessorTopologyTest...

Github user hrafzali closed the pull request at: https://github.com/apache/kafka/pull/2587 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is en

[jira] [Commented] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[ https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880637#comment-15880637 ] ASF GitHub Bot commented on KAFKA-4789: --- Github user hrafzali closed the pull reques

[GitHub] kafka pull request #2590: KAFKA-4789: Added support to ProcessorTopologyTest...

GitHub user hrafzali opened a pull request: https://github.com/apache/kafka/pull/2590 KAFKA-4789: Added support to ProcessorTopologyTestDriver to forward timestamps to internal topics This resolves the issue in the ProcessorTopologyTestDriver that the extracted timestamp is not for

[GitHub] kafka pull request #2589: KAFKA-4779: Fix security upgrade system test to be...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/2589 KAFKA-4779: Fix security upgrade system test to be non-disruptive The phase_two security upgrade test verifies upgrading inter-broker and client protocols to the same value as well as differen

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880642#comment-15880642 ] ASF GitHub Bot commented on KAFKA-4779: --- GitHub user rajinisivaram opened a pull req

[jira] [Commented] (KAFKA-4789) ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics

[ https://issues.apache.org/jira/browse/KAFKA-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880641#comment-15880641 ] ASF GitHub Bot commented on KAFKA-4789: --- GitHub user hrafzali opened a pull request:

[jira] [Commented] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

[ https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880684#comment-15880684 ] Rajini Sivaram commented on KAFKA-4779: --- I couldn't recreate the failure, but the co

[jira] [Created] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

Bart Vercammen created KAFKA-4791: - Summary: Kafka Streams - unable to add state stores when using wildcard topics on the source Key: KAFKA-4791 URL: https://issues.apache.org/jira/browse/KAFKA-4791 P

Re: [DISCUSS] KIP-124: Request rate quotas

Thanks for the KIP, Rajini. This is a welcome improvement and the KIP page covers it well. A few comments: 1. Can you expand a bit on the motivation for throttling requests that fail authorization for ClusterAction? Under what scenarios would this help? 2. I think we should rename `throttle_time_

Re: [DISCUSS] KIP-124: Request rate quotas

i dont think time/cpu% are easy to reason about. most user-facing quota systems i know (especially the commercial ones) focus on things users understand better - iops and bytes. as for quotas and "overhead" requests like heartbeats - on the one hand subjecting them to the quota may cause clients t

[jira] [Assigned] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reassigned KAFKA-4791: -- Assignee: Bill Bejeck > Kafka Streams - unable to add state stores when using wildcard topics o

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880782#comment-15880782 ] Bill Bejeck commented on KAFKA-4791: picking this one up. > Kafka Streams - unable to

[GitHub] kafka pull request #2588: KAFKA-4788: Revert "KAFKA-4092: retention.bytes sh...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2588 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880785#comment-15880785 ] ASF GitHub Bot commented on KAFKA-4788: --- Github user asfgit closed the pull request

[jira] [Resolved] (KAFKA-4788) Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic.

[ https://issues.apache.org/jira/browse/KAFKA-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4788. Resolution: Fixed > Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' >

Re: [DISCUSS] KIP-124: Request rate quotas

A few minor comments: 1. Isn't it the case that the throttling time response field should have the total time your request was throttled irrespective of the quotas that caused that. Limiting it to byte rate quota doesn't make sense, but I also I don't think we want to end up adding new

[jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880817#comment-15880817 ] Jun Rao commented on KAFKA-2729: [~prasincs], if the controller is partitioned off other b

Re: [DISCUSS] KIP-124: Request rate quotas

Hi Jay, Regarding 1, I definitely like the simplicity of keeping a single throttle time field in the response. The downside is that the client metrics will be more coarse grained. Regarding 3, we have `leader.imbalance.per.broker.percentage` and `log.cleaner.min.cleanable.ratio`. Ismael On Thu,

Re: [DISCUSS] KIP-124: Request rate quotas

Hi, Ismael, For #3, typically, an admin won't configure more io threads than CPU cores, but it's possible for an admin to start with fewer io threads than cores and grow that later on. Hi, Dong, I think the throttleTime sensor on the broker tells the admin whether a user/clentId is throttled or

[GitHub] kafka pull request #2583: MINOR: Reduce metrics overheads

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2583 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

So you suggest to merge "scope options" --topics, --topic, and --partitions into a single option? Sound good to me. I like the compact way to express it, ie, topicname:list-of-partitions with "all partitions" if not partitions are specified. It's quite intuitive to use. Just wondering, if we coul

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880966#comment-15880966 ] Matthias J. Sax commented on KAFKA-4791: You should not call {{connectStateStoreNa

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880971#comment-15880971 ] Matthias J. Sax commented on KAFKA-4791: \cc [~damianguy] > Kafka Streams - unabl

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Yeap, `--topic t1=1,2`LGTM Don't have idea neither about getting rid of repeated --topic, but --group is also repeated in the case of deletion, so it could be ok to have repeated --topic arguments. El jue., 23 feb. 2017 a las 19:14, Matthias J. Sax () escribió: > So you suggest to merge "scope o

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Jun, Yeah you are right. I thought it wasn't because at LinkedIn it will be too much pressure on inGraph to expose those per-clientId metrics so we ended up printing them periodically to local log. Never mind if it is not a general problem. Hey Rajini, - I agree with Jay that we probably don

[jira] [Comment Edited] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881018#comment-15881018 ] Bill Bejeck edited comment on KAFKA-4791 at 2/23/17 6:56 PM: -

[GitHub] kafka pull request #2591: HOTFIX: use explicit version in upgrade.html

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/2591 HOTFIX: use explicit version in upgrade.html You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka KHotfix-explicit-version A

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881018#comment-15881018 ] Bill Bejeck commented on KAFKA-4791: Fair enough. I did not look into the issue at al

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

+1 (non-binding) On Wed, Feb 22, 2017 at 10:52 PM, Manikumar wrote: > +1 (non-binding) > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Jun, > > > > Thanks a lot for the comments and reviews. > > I agree we should log the username. > > What

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881043#comment-15881043 ] Bart Vercammen commented on KAFKA-4791: --- This then means that {{addStateStore}} shou

[jira] [Commented] (KAFKA-4601) Avoid duplicated repartitioning in KStream DSL

[ https://issues.apache.org/jira/browse/KAFKA-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881030#comment-15881030 ] Michal Borowiecki commented on KAFKA-4601: -- Don't know if this belongs in this ti

[jira] [Created] (KAFKA-4792) Kafka Connect: Add ByteArray Converter

Gwen Shapira created KAFKA-4792: --- Summary: Kafka Connect: Add ByteArray Converter Key: KAFKA-4792 URL: https://issues.apache.org/jira/browse/KAFKA-4792 Project: Kafka Issue Type: Improvement

[jira] [Comment Edited] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881043#comment-15881043 ] Bart Vercammen edited comment on KAFKA-4791 at 2/23/17 7:18 PM:

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Regarding (2) - yes that's a good point. @Onur - I think the KIP should explicitly call this out. It is something that we did consider and decided against optimizing for. i.e., we just wrote that off as a minor caveat of the upgrade path in that there will be a few duplicates, but not too many give

[GitHub] kafka pull request #2592: MINOR: fixed javadoc typo in KafkaProducer::partit...

GitHub user jpdaigle opened a pull request: https://github.com/apache/kafka/pull/2592 MINOR: fixed javadoc typo in KafkaProducer::partitionsFor You can merge this pull request into a Git repository by running: $ git pull https://github.com/tripadvisor/kafka minor_typo_fix Alt

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

@Dong, Thanks for the comments. The default behavior of the producer won't change. If the users want to use the uncompressed message size, they probably will also bump up the batch size to somewhere close to the max message size. This would be in the document. BTW the default batch size is 16K whi

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Rajini, The current KIP says that the maximum delay will be reduced to window size if it is larger than the window size. I have a concern with this: 1) This essentially means that the user is allowed to exceed their quota over a long period of time. Can you provide an upper bound on this devi

Re: [DISCUSS] KIP-124: Request rate quotas

@jun: i wasnt concerned about tying up a request processing thread, but IIUC the code does still read the entire request out, which might add-up to a non-negligible amount of memory. On Thu, Feb 23, 2017 at 11:55 AM, Dong Lin wrote: > Hey Rajini, > > The current KIP says that the maximum delay w

[jira] [Created] (KAFKA-4793) Kafka Connect: POST /connectors/(string: name)/restart doesn't start failed tasks

Gwen Shapira created KAFKA-4793: --- Summary: Kafka Connect: POST /connectors/(string: name)/restart doesn't start failed tasks Key: KAFKA-4793 URL: https://issues.apache.org/jira/browse/KAFKA-4793 Project

Re: [DISCUSS] KIP-82 - Add Record Headers

So currently KIP has: append filter (get by key, but not inferring perf as noted expectation on O(1) perf with a get method) I agree that supporting some form of remove replace is standard in this kind of API and for the benefit of making a usable API, we should explore adding these. If we’r

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881280#comment-15881280 ] Damian Guy commented on KAFKA-4791: --- [~mjsax] this is a bug. Bart is calling {{addStateS

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Yeah, I agree it is a bit complex to do that approach for a one-time migration. Probably not worth it. Here is another idea to reduce, but not eliminate, the amount of message duplication during migration. I am fine with not doing it. Just want to see the opinion from open source community. The pr

[GitHub] kafka pull request #2591: HOTFIX: use explicit version in upgrade.html

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2591 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hey Becket, Yeah that makes sense. I agree that you'd really have to both fix the estimation (i.e. make it per topic or make it better estimate the high percentiles) AND have the recovery mechanism. If you are underestimating often and then paying a high recovery price that won't fly. I think yo

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Sounds reasonable. If we have multiple --topic arguments, it does also not matter if we use t1:1,2 or t2=1,2 I just suggested '=' because I wanted use ':' to chain multiple topics. -Matthias On 2/23/17 10:49 AM, Jorge Esteban Quilcate Otoya wrote: > Yeap, `--topic t1=1,2`LGTM > > Don't have i

[jira] [Commented] (KAFKA-4601) Avoid duplicated repartitioning in KStream DSL

[ https://issues.apache.org/jira/browse/KAFKA-4601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881339#comment-15881339 ] Matthias J. Sax commented on KAFKA-4601: [~mihbor] Thanks for your comment. I gues

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Oh ok :) So, we can keep `--topic t1:1,2,3` I think with this one we have most of the feedback applied. I will update the KIP with this change. El jue., 23 feb. 2017 a las 22:38, Matthias J. Sax () escribió: > Sounds reasonable. > > If we have multiple --topic arguments, it does also not matter

[jira] [Created] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

Florian Hussonnois created KAFKA-4794: - Summary: Add access to OffsetStorageReader from SourceConnector Key: KAFKA-4794 URL: https://issues.apache.org/jira/browse/KAFKA-4794 Project: Kafka

[jira] [Updated] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

[ https://issues.apache.org/jira/browse/KAFKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florian Hussonnois updated KAFKA-4794: -- Affects Version/s: 0.10.2.0 > Add access to OffsetStorageReader from SourceConnector > -

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Dong, Just so that we are on the same page. Let me spec out the alternative design a bit more and then compare. Let's call the current design A and the alternative design B. Design B: New ZK path failed log directory path (persistent): This is created by a broker when a log directory fails a

[jira] [Commented] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

[ https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881456#comment-15881456 ] Matthias J. Sax commented on KAFKA-4791: Ack. My bad. I thought [~clouTrix] calls

Re: [DISCUSS] KIP-124: Request rate quotas

Thank you all for the feedback. Ismael #1. It makes sense not to throttle inter-broker requests like LeaderAndIsr etc. The simplest way to ensure that clients cannot use these requests to bypass quotas for DoS attacks is to ensure that ACLs prevent clients from using these requests and unauthorize

[jira] [Created] (KAFKA-4795) Confusion around topic deletion

Vahid Hashemian created KAFKA-4795: -- Summary: Confusion around topic deletion Key: KAFKA-4795 URL: https://issues.apache.org/jira/browse/KAFKA-4795 Project: Kafka Issue Type: Bug Affects

Build failed in Jenkins: kafka-trunk-jdk8 #1296

See Changes: [wangguoz] MINOR: Reduce stream thread metrics overhead -- [...truncated 1.42 MB...] org.apache.kafka.streams.state.internals.RocksDBKeyValueStoreTest > tes

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi All, If there are no more concerns, I'd like to start vote for this KIP. Thanks! Jorge. El jue., 23 feb. 2017 a las 22:50, Jorge Esteban Quilcate Otoya (< quilcate.jo...@gmail.com>) escribió: > Oh ok :) > > So, we can keep `--topic t1:1,2,3` > > I think with this one we have most of the feed

[VOTE] KIP-122: Add Reset Consumer Group Offsets tooling

Hi All, It seems that there is no further concern with the KIP-122. At this point we would like to start the voting process. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling Thanks! Jorge.

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, Thanks for you reply! Let me first comment on the things that you listed as advantage of B over A. 1) No change in LeaderAndIsrRequest protocol. I agree with this. 2) Step 1. One less round of LeaderAndIsrRequest and no additional ZK writes to record the created flag. I don't think th

[jira] [Created] (KAFKA-4796) Fix some findbugs warnings in Kafka Java client

Colin P. McCabe created KAFKA-4796: -- Summary: Fix some findbugs warnings in Kafka Java client Key: KAFKA-4796 URL: https://issues.apache.org/jira/browse/KAFKA-4796 Project: Kafka Issue Type:

[GitHub] kafka pull request #2593: KAFKA-4796: Fix some findbugs warnings in Kafka Ja...

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/2593 KAFKA-4796: Fix some findbugs warnings in Kafka Java client You can merge this pull request into a Git repository by running: $ git pull https://github.com/cmccabe/kafka KAFKA-4796 Alternativ

[jira] [Commented] (KAFKA-4796) Fix some findbugs warnings in Kafka Java client

[ https://issues.apache.org/jira/browse/KAFKA-4796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881667#comment-15881667 ] ASF GitHub Bot commented on KAFKA-4796: --- GitHub user cmccabe opened a pull request:

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Dong, My replies are inlined below. On Thu, Feb 23, 2017 at 4:47 PM, Dong Lin wrote: > Hey Jun, > > Thanks for you reply! Let me first comment on the things that you listed as > advantage of B over A. > > 1) No change in LeaderAndIsrRequest protocol. > > I agree with this. > > 2) Step 1. On

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, Sure, here is my explanation. Design B would not work if it doesn't store created replicas in the ZK. For example, say broker B is health when it is shutdown. At this moment no offline replica is written in ZK for this broker. Suppose log directory is damaged when broker is offline, then

[jira] [Commented] (KAFKA-4766) Document lz4 and lz4hc in confluence

[ https://issues.apache.org/jira/browse/KAFKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881869#comment-15881869 ] Lee Dongjin commented on KAFKA-4766: [~ijuma] Could you review the wiki update I did f

[jira] [Assigned] (KAFKA-4766) Document lz4 and lz4hc in confluence

[ https://issues.apache.org/jira/browse/KAFKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lee Dongjin reassigned KAFKA-4766: -- Assignee: Lee Dongjin > Document lz4 and lz4hc in confluence >

[jira] [Created] (KAFKA-4797) Add lag in seconds JMX metrics for fether threads

Jun Ma created KAFKA-4797: - Summary: Add lag in seconds JMX metrics for fether threads Key: KAFKA-4797 URL: https://issues.apache.org/jira/browse/KAFKA-4797 Project: Kafka Issue Type: Bug C

[jira] [Updated] (KAFKA-4797) Add lag in time JMX metrics for fether threads

[ https://issues.apache.org/jira/browse/KAFKA-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Ma updated KAFKA-4797: -- Summary: Add lag in time JMX metrics for fether threads (was: Add lag in seconds JMX metrics for fether threads

[jira] [Updated] (KAFKA-4797) Add lag in time JMX metrics for fether threads

[ https://issues.apache.org/jira/browse/KAFKA-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Ma updated KAFKA-4797: -- Attachment: maxLag.jpg > Add lag in time JMX metrics for fether threads > --

[jira] [Updated] (KAFKA-4797) Add lag in time JMX metrics for fether threads

[ https://issues.apache.org/jira/browse/KAFKA-4797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Ma updated KAFKA-4797: -- Attachment: (was: maxLag.png) > Add lag in time JMX metrics for fether threads > ---

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

I just read the update KIP once more. I would suggest to rename --to-duration to --by-duration Or as a second idea, rename --to-duration to --shift-by-duration and at the same time rename --shift-offset-by to --shift-by-offset Not sure what the best option is, but naming would be more consistent

Consumption on a explicitly (dynamically) created topic has a 5 minute delay

(Re)posting this from the user mailing list to dev mailing list, hoping for some inputs from the Kafka dev team: We are on Kafka 0.10.0.1 (server and client) and use Java consumer/producer APIs. We have an application where we create Kafka topics dynamically (using the AdminUtils Java API) and

[jira] [Created] (KAFKA-4798) Javadocs for 0.10.2 are missing from kafka.apache.org

James Cheng created KAFKA-4798: -- Summary: Javadocs for 0.10.2 are missing from kafka.apache.org Key: KAFKA-4798 URL: https://issues.apache.org/jira/browse/KAFKA-4798 Project: Kafka Issue Type: B

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

> On Feb 23, 2017, at 10:03 PM, Jaikiran Pai wrote: > > (Re)posting this from the user mailing list to dev mailing list, hoping for > some inputs from the Kafka dev team: > > We are on Kafka 0.10.0.1 (server and client) and use Java consumer/producer > APIs. We have an application where we cr

Re: Consumption on a explicitly (dynamically) created topic has a 5 minute delay

James, thank you very much for this explanation and I now understand the situation much more clearly. I wasn't aware that the consumer's metadata.max.age.ms could play a role in this. I was under the impression that the 5 minute timeout is some broker level config which was triggering this cons

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hey Jun, I think there is one simpler design that doesn't need to add "create" flag in LeaderAndIsrRequest and also remove the need for controller to track/update which replicas are created. The idea is for each broker to persist the created replicas in per-broker-per-topic znode. When a replica i