[jira] [Created] (KAFKA-4782) change chroot for a kafka instance

polarbear created KAFKA-4782: Summary: change chroot for a kafka instance Key: KAFKA-4782 URL: https://issues.apache.org/jira/browse/KAFKA-4782 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-124: Request rate quotas

Dong, Onur & Becket, Thank you all for the very useful feedback. The choice of request handling time as opposed to request rate was based on the observation in KAFKA-4195 that request rates may be less intuitive to configure than percentage utili

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

Hi Jason, Yes, this is the idea. The connector assigns a subset of files to each task. A task stores the size of file, the bytes offset and the bytes size of the last sent record as a source offsets. A file is finished when recordBytesOffsets + recordBytesSize = fileBytesSize. The connector shou

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Rajini, Thanks for the explanation. I have some follow up questions regarding the types of requests that will be covered by this quota. Since this KIP focus only on throttling the traffic between client and broker and client never sends LeaderAndIsrRequest to broker, should we exclude LeaderAn

[jira] [Commented] (KAFKA-4686) Null Message payload is shutting down broker

[ https://issues.apache.org/jira/browse/KAFKA-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874455#comment-15874455 ] Hannu Valtonen commented on KAFKA-4686: --- Hit this again in a test enviroment of ours

[jira] [Commented] (KAFKA-4686) Null Message payload is shutting down broker

[ https://issues.apache.org/jira/browse/KAFKA-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874555#comment-15874555 ] Hannu Valtonen commented on KAFKA-4686: --- Oh yeah, thought to mention that just befor

Re: [DISCUSS] KIP-124: Request rate quotas

I have updated the KIP to use request rates instead of request processing time, I have removed all requests that require ClusterAction permission (LeaderAndIsr and UpdateMetdata as well in addition to stop/shutdown). But I have left Metadata request in. Quota windows which limit the maximum delay

[GitHub] kafka pull request #2577: MINOR: add 2 failure test methods

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/2577 MINOR: add 2 failure test methods You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka minor-benchmark-args Alternatively you

[jira] [Commented] (KAFKA-4782) change chroot for a kafka instance

[ https://issues.apache.org/jira/browse/KAFKA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874650#comment-15874650 ] Jayesh Thakrar commented on KAFKA-4782: --- A good question and not an uncommon situati

[jira] [Work started] (KAFKA-4703) Test with two SASL_SSL listeners with different JAAS contexts

[ https://issues.apache.org/jira/browse/KAFKA-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4703 started by Balint Molnar. > Test with two SASL_SSL listeners with different JAAS contexts > ---

Re: Kafka exception

Hi Liwu, Correct me if I am wrong. When calling the method ConsumerConnector.shutdown(), it will send "ZookeeperConsumerConnector.shutdownCommand" to the queue, not set ConsumerIterator's state is NOT_READY directly. So the consumer will continue consuming until get the shutdownCommand in the q

[jira] [Commented] (KAFKA-2096) Enable keepalive socket option for broker to prevent socket leak

[ https://issues.apache.org/jira/browse/KAFKA-2096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874848#comment-15874848 ] Faisal commented on KAFKA-2096: --- Does this solution also resolve following error in spark st

[jira] [Commented] (KAFKA-1895) Investigate moving deserialization and decompression out of KafkaConsumer

[ https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874894#comment-15874894 ] Armin Braun commented on KAFKA-1895: [~jkreps] Maybe let me give the perspective of a

[jira] [Comment Edited] (KAFKA-2045) Memory Management on the consumer

[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874896#comment-15874896 ] Armin Braun edited comment on KAFKA-2045 at 2/20/17 6:10 PM: -

[jira] [Commented] (KAFKA-2045) Memory Management on the consumer

[ https://issues.apache.org/jira/browse/KAFKA-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874896#comment-15874896 ] Armin Braun commented on KAFKA-2045: add a suggestion on how to concretely tackly this

[DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey everyone. I made a KIP that provides a mechanism for migrating from ZookeeperConsumerConnector to KafkaConsumer as well as a mechanism for rolling back from KafkaConsumer to ZookeeperConsumerConnector: https://cwiki.apache.org/confluence/display/KAFKA/KIP-125%3A+ZookeeperConsumerConnector+to+K

Re: [DISCUSS] KIP-124: Request rate quotas

I think this proposal makes a lot of sense (especially now that it is oriented around request rate) and fills the biggest remaining gap in the multi-tenancy story. I think for intra-cluster communication (StopReplica, etc) we could avoid throttling entirely. You can secure or otherwise lock-down t

Re: [DISCUSS] KIP-124: Request rate quotas

If the purpose of the KIP is only to protect the cluster from being overwhelmed by crazy clients and is not intended to address resource allocation problem among the clients, I am wondering if using request handling time quota (CPU time quota) is a better option. Here are the reasons: 1. request h

[jira] [Commented] (KAFKA-4782) change chroot for a kafka instance

[ https://issues.apache.org/jira/browse/KAFKA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15875133#comment-15875133 ] Stephane Maarek commented on KAFKA-4782: Use https://github.com/kshchepanovskyi/zk

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

Thanks for the explanation, Guozhang. That makes sense. On Sun, Feb 19, 2017 at 7:28 PM, Guozhang Wang wrote: > Thanks Becket. > > Actually sequence is associated with a message, not a message set. For > example if a message set sent by producer contains 100 messages, and the > first message's s

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Becket/Rajini, When I thought about it more deeply I came around to the "percent of processing time" metric too. It seems a lot closer to the thing we actually care about and need to protect. I also think this would be a very useful metric even in the absence of throttling just to debug whose

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Hey Onur, Thanks for the well-written KIP! I have two questions below. 1) In the process of migrating from OZKCCs and MDZKCCs to MEZKCCs, we will may a mix of OZKCCs, MDZKCCs and MEZKCCs. OZKCC and MDZKCC will only commit to zookeeper and MDZKCC will use kafka-based offset storage. Would we lose

Re: [DISCUSS] KIP-125: ZookeeperConsumerConnector to KafkaConsumer Migration and Rollback

Regarding 1: We won't lose the offset from zookeeper upon partition transfer from OZKCC/MDZKCC to MEZKCC because MEZKCC has "dual.commit.enabled" set to true as well as "offsets.storage" set to kafka. The combination of these configs results in the consumer fetching offsets from both kafka and zook

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

Hi, thanks for updating the KIP. Couple of follow up comments: * Nit: Why is "Reset to Earliest" and "Reset to Latest" a "reset by time" option -- IMHO it belongs to "reset by position"? * Nit: Description of "Reset to Earliest" > using Kafka Consumer's `auto.offset.reset` to `earliest` I thi

Re: [DISCUSS] KIP-124: Request rate quotas

This is a great proposal, glad to see it happening. I am inclined to the CPU throttling, or more specifically processing time ratio instead of the request rate throttling as well. Becket has very well summed my rationales above, and one thing to add here is that the former has a good support for b

Re: [DISCUSS] KIP-124: Request rate quotas

Hey Jay, Yeah, I agree that enforcing the CPU time is a little tricky. I am thinking that maybe we can use the existing request statistics. They are already very detailed so we can probably see the approximate CPU time from it, e.g. something like (total_time - request/response_queue_time - remote

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

Hi, Dong, Sorry for the delay. A few more comments. 20. One complexity that I found in the current KIP is that the way the broker communicates failed replicas to the controller is inefficient. When a log directory fails, the broker only sends an indication through ZK to the controller and the con

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi, Mayuresh, Thanks for the updated KIP. A couple of more comments. 1. Do we convert java.security.Principal to KafkaPrincipal for authorization check in SimpleAclAuthorizer? If so, it would be useful to mention that in the wiki so that people can understand how this change doesn't affect the de