test of producer's delay and consumer's delay

hello，I have done a series of tests on kafka 0.9.0，and one of the results confused me. test enviroment: kafka cluster: 3 brokers，8core cpu ／ 8g mem ／1g netcard client：4core cpu／4g mem topic：6 partitions，2 replica total messages：1 singal message size：1024byte

Re: FYI - Apache ZooKeeper Backup, a Treatise

See HBASE-14379. The points on configuration, state management, and security apply. > On Jun 17, 2016, at 7:25 PM, Martin Serrano wrote: > > Why are you seeking to undo it? > >> On 06/17/2016 09:34 PM, Andrew Purtell wrote: >> HBase stores replication peering

Re: FYI - Apache ZooKeeper Backup, a Treatise

HBase stores replication peering configuration in ZK. We're working on undoing that, but for now that information exists nowhere else. On Thu, Jun 16, 2016 at 2:47 PM, Ismael Juma wrote: > Hi Jordan, > > Kafka stores ACLs as well as client and topic configs in ZooKeeper so

[jira] [Updated] (KAFKA-1429) Yet another deadlock in controller shutdown

[ https://issues.apache.org/jira/browse/KAFKA-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengwei updated KAFKA-1429: --- Attachment: kafka_0.9.0.0_controller_dead_lock.patch upload file is the patch to fix this bug, can somebody

[GitHub] kafka pull request #1521: MINOR: Check null keys in KTableSource

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/1521 MINOR: Check null keys in KTableSource You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka

[GitHub] kafka pull request #1520: HOTFIX: Check hasNext in KStreamWindowReduce

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/1520 HOTFIX: Check hasNext in KStreamWindowReduce You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka

[jira] [Commented] (KAFKA-3862) Kafka streams documentation: partition<->thread<->task assignment unclear

[ https://issues.apache.org/jira/browse/KAFKA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337202#comment-15337202 ] Guozhang Wang commented on KAFKA-3862: -- Good to know [~dminkovsky]! Please do let us know if you

[jira] [Comment Edited] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337172#comment-15337172 ] Guozhang Wang edited comment on KAFKA-3839 at 6/17/16 11:00 PM: Hi

[jira] [Updated] (KAFKA-3836) KStreamReduce and KTableReduce should not pass nulls to Deserializers

[ https://issues.apache.org/jira/browse/KAFKA-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3836: - Labels: architecture (was: ) > KStreamReduce and KTableReduce should not pass nulls to

[jira] [Resolved] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3839. -- Resolution: Duplicate > Handling null keys in KTable.groupBy >

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337172#comment-15337172 ] Guozhang Wang commented on KAFKA-3839: -- Hi [~jeyhunkarimov], It seems I created this ticket as a

[jira] [Commented] (KAFKA-3861) Shrunk ISR before leader crash makes the partition unavailable

[ https://issues.apache.org/jira/browse/KAFKA-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337080#comment-15337080 ] James Cheng commented on KAFKA-3861: Semi-related to https://issues.apache.org/jira/browse/KAFKA-3410

[jira] [Commented] (KAFKA-3863) Add system test for connector failure/restart

[ https://issues.apache.org/jira/browse/KAFKA-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337034#comment-15337034 ] ASF GitHub Bot commented on KAFKA-3863: --- GitHub user hachikuji opened a pull request:

[GitHub] kafka pull request #1519: KAFKA-3863: System tests covering connector/task f...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/1519 KAFKA-3863: System tests covering connector/task failure and restart You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

[jira] [Created] (KAFKA-3863) Add system test for connector failure/restart

Jason Gustafson created KAFKA-3863: -- Summary: Add system test for connector failure/restart Key: KAFKA-3863 URL: https://issues.apache.org/jira/browse/KAFKA-3863 Project: Kafka Issue Type:

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

Jason, Thanks for the KIP. +1 Just one clarification. The KIP adds a rebalance timeout in the protocol, but didn't say what value will be used. I guess we will use max.poll.interval.ms? Jun On Thu, Jun 16, 2016 at 11:44 AM, Jason Gustafson wrote: > Hi All, > > I'd like to

Re: [DISCUSS] KIP-4 Create Topic Schema

Grant, I think Dana has a valid point. Currently, we throw an InvalidRequestException and close the connection only when the broker can't deserialize the bytes into a request. In this case, the deserialization is fine. It just that there are some additional constraints that can't be specified at

[jira] [Commented] (KAFKA-3632) ConsumerLag metrics persist after partition migration

[ https://issues.apache.org/jira/browse/KAFKA-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336852#comment-15336852 ] ASF GitHub Bot commented on KAFKA-3632: --- GitHub user hachikuji opened a pull request:

[GitHub] kafka pull request #1518: KAFKA-3632; remove fetcher metrics on shutdown and...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/1518 KAFKA-3632; remove fetcher metrics on shutdown and leader migration You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka

[jira] [Commented] (KAFKA-3862) Kafka streams documentation: partition<->thread<->task assignment unclear

[ https://issues.apache.org/jira/browse/KAFKA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336800#comment-15336800 ] Dmitry Minkovsky commented on KAFKA-3862: - Oh hey, I somehow missed 0.10 and Confluent 3.0.

[jira] [Resolved] (KAFKA-3862) Kafka streams documentation: partition<->thread<->task assignment unclear

[ https://issues.apache.org/jira/browse/KAFKA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Minkovsky resolved KAFKA-3862. - Resolution: Fixed Fix Version/s: 0.10.0.0 > Kafka streams documentation:

[jira] [Commented] (KAFKA-3837) Report the name of the blocking thread when throwing ConcurrentModificationException

[ https://issues.apache.org/jira/browse/KAFKA-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336761#comment-15336761 ] Bharat Viswanadham commented on KAFKA-3837: --- Hi Simon, I have updated the code. You can have a

[jira] [Assigned] (KAFKA-3836) KStreamReduce and KTableReduce should not pass nulls to Deserializers

[ https://issues.apache.org/jira/browse/KAFKA-3836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeyhun Karimov reassigned KAFKA-3836: - Assignee: Jeyhun Karimov > KStreamReduce and KTableReduce should not pass nulls to

[jira] [Resolved] (KAFKA-3838) Bump zkclient and Zookeeper versions

[ https://issues.apache.org/jira/browse/KAFKA-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] FIlipe Azevedo resolved KAFKA-3838. --- Resolution: Fixed > Bump zkclient and Zookeeper versions >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Thank you, Jun. I have removed user_principal from the KIP. On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao wrote: > Rajini, > > 10. Yes, then we can probably leave out the user_principal field and keep > the version to be 1. > > Other than that, the KIP looks good to me. > >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Rajini, 10. Yes, then we can probably leave out the user_principal field and keep the version to be 1. Other than that, the KIP looks good to me. Thanks, Jun On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > Jun, > > 10. Since entity_type "users" is

[jira] [Updated] (KAFKA-3862) Kafka streams documentation: partition<->thread<->task assignment unclear

[ https://issues.apache.org/jira/browse/KAFKA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Minkovsky updated KAFKA-3862: Description: Hi. Not sure if this is the right place to post about documentation (this

[jira] [Updated] (KAFKA-3862) Kafka streams documentation: partition<->thread<->task assignment unclear

[ https://issues.apache.org/jira/browse/KAFKA-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Minkovsky updated KAFKA-3862: Description: Hi. Not sure if this is the right place to post about documentation (this

[jira] [Comment Edited] (KAFKA-2359) New consumer - partitions auto assigned only on poll

[ https://issues.apache.org/jira/browse/KAFKA-2359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336331#comment-15336331 ] Mirza Gaush Beg edited comment on KAFKA-2359 at 6/17/16 4:41 PM: - Another

[jira] [Updated] (KAFKA-3861) Shrunk ISR before leader crash makes the partition unavailable

[ https://issues.apache.org/jira/browse/KAFKA-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated KAFKA-3861: --- Description: We observed a case that the leader experienced a crash and lost its in-memory

[jira] [Updated] (KAFKA-3861) Shrunk ISR before leader crash makes the partition unavailable

[ https://issues.apache.org/jira/browse/KAFKA-3861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maysam Yabandeh updated KAFKA-3861: --- Description: We observed a case that the leader experienced a crash and lost its in-memory

[jira] [Created] (KAFKA-3861) Shrunk ISR before leader crash makes the partition unavailable

Maysam Yabandeh created KAFKA-3861: -- Summary: Shrunk ISR before leader crash makes the partition unavailable Key: KAFKA-3861 URL: https://issues.apache.org/jira/browse/KAFKA-3861 Project: Kafka

Re: [DISCUSS] KIP-4 Create Topic Schema

I'm unconvinced (crazy, right?). Comments below: On Fri, Jun 17, 2016 at 7:27 AM, Grant Henke wrote: > Hi Dana, > > You mentioned one of the reasons I error and disconnect. Because I can't > return an error for every request so the cardinality between request and > response

[jira] [Commented] (KAFKA-3839) Handling null keys in KTable.groupBy

[ https://issues.apache.org/jira/browse/KAFKA-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336310#comment-15336310 ] Jeyhun Karimov commented on KAFKA-3839: --- It seems that groupby operator is gone from KTable.

Re: [DISCUSS] Java 8 as a minimum requirement

-1 Hi Ismael, Although I really like the Java 8 features and understand the advantages you mentioned about Java 8 migration, I would suggest to stay with Java 7 as a minimum requirement for a while. I think there are two aspects to consider - Kafka Server and Kafka clients. On the server part

[jira] [Commented] (KAFKA-3487) Support per-connector/per-task classloaders in Connect

[ https://issues.apache.org/jira/browse/KAFKA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336254#comment-15336254 ] Randall Hauch commented on KAFKA-3487: -- I hope this can move toward an approach that can eventually

[jira] [Commented] (KAFKA-3693) Race condition between highwatermark-checkpoint thread and handleLeaderAndIsrRequest at broker start-up

[ https://issues.apache.org/jira/browse/KAFKA-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336241#comment-15336241 ] Maysam Yabandeh commented on KAFKA-3693: [~junrao] I am thinking of a panic reaction to the first,

Re: [DISCUSS] KIP-4 Create Topic Schema

Hi Dana, You mentioned one of the reasons I error and disconnect. Because I can't return an error for every request so the cardinality between request and response would be different. Beyond that though, I am handling this protocol rule/parsing error the same way all other messages do. Parsing

Re: Embedding zookeeper and kafka in java process.

If you are using it for tests, this works with Kafka 10 (tune broker configs as per your req) public class TestKafkaCluster { final KafkaServer kafkaServer; final ZkClient zkClient; private String zkConnectionString; public TestKafkaCluster(String zkConnectionString, int

Re: [DISCUSS] Java 8 as a minimum requirement

Hi Marcus, Thanks for your feedback. With regards to IBM WebSphere, the latest stable release (8.5.5) supports Java 8 according to the documentation: http://www-01.ibm.com/support/docview.wss?uid=swg27005002 Having said that, it is fair to discuss servers and clients separately. In Kafka, you

Re: [DISCUSS] Java 8 as a minimum requirement

On Thu, Jun 16, 2016 at 5:20 PM, Ismael Juma wrote: > On Thu, Jun 16, 2016 at 11:13 PM, Stephen Boesch > wrote: > > > @Jeff Klukas What is the concern about scala 2.11 vs 2.12? 2.11 runs on > > both java7 and java8 > > > > Scala 2.10.5 and 2.10.6 also

[jira] [Assigned] (KAFKA-3185) Allow users to cleanup internal data

[ https://issues.apache.org/jira/browse/KAFKA-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-3185: -- Assignee: Matthias J. Sax > Allow users to cleanup internal data >

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

+1 (non-binding) On Fri, Jun 17, 2016 at 3:37 PM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > +1 (non-binding) > > On Fri, Jun 17, 2016 at 4:45 AM, Grant Henke wrote: > > > +1 > > > > On Thu, Jun 16, 2016 at 8:50 PM, tao xiao wrote: > > >

Re: [VOTE] KIP-55: Secure quotas for authenticated users

Jun, 10. Since entity_type "users" is new, shouldn't the JSON for these entities have version 1? I have moved "user_principal" out of the config in the samples and added to the entries as well. But actually, do we need to store the non-encoded principal at all? The node name is

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

+1 (non-binding) On Fri, Jun 17, 2016 at 4:45 AM, Grant Henke wrote: > +1 > > On Thu, Jun 16, 2016 at 8:50 PM, tao xiao wrote: > > > +1 > > > > On Fri, 17 Jun 2016 at 09:03 Harsha wrote: > > > > > +1 (binding) > > > Thanks, > > >

Build failed in Jenkins: kafka-trunk-jdk8 #705

See Changes: [ismael] KAFKA-3838; Update zkClient to 0.9 and Zookeeper to 3.4.8 -- [...truncated 13570 lines...] org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest >

[GitHub] kafka pull request #1517: [Kafka Streams] Clean-up script [WIP]

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1517 [Kafka Streams] Clean-up script [WIP] Anyof: @guozhangwang @enothereska @dguy @miguno Two variants: 1. python client + bash script (seems not to be the right way, due to external

[jira] [Commented] (KAFKA-3838) Bump zkclient and Zookeeper versions

[ https://issues.apache.org/jira/browse/KAFKA-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335584#comment-15335584 ] ASF GitHub Bot commented on KAFKA-3838: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #1512: KAFKA-3838 zkClient and Zookeeper version bump

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1512 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-3576) Unify KStream and KTable API

[ https://issues.apache.org/jira/browse/KAFKA-3576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-3576. --- Resolution: Resolved Resolved by https://issues.apache.org/jira/browse/KAFKA-3561 > Unify KStream

[jira] [Updated] (KAFKA-3860) No broker partitions consumed by consumer thread

[ https://issues.apache.org/jira/browse/KAFKA-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kundan updated KAFKA-3860: -- Description: I'm using kakfa-clients-0.8.2.1 to consume messages as Java KafkaConsumer. Problem: I have two

[jira] [Created] (KAFKA-3860) No broker partitions consumed by consumer thread

Kundan created KAFKA-3860: - Summary: No broker partitions consumed by consumer thread Key: KAFKA-3860 URL: https://issues.apache.org/jira/browse/KAFKA-3860 Project: Kafka Issue Type: Bug