Re: Any idea why acceptor is volatile in SocketServer?

Thanks! Yeah, the shutdown hook will run on a separate thread. In that case, I don't need (or want) to keep acceptors in ConcurrentHashMap - we are either starting up or shutting down a broker, but don't want to do both at same time. I'll just synchronize the acceptors startup and shutdown in So

Re: Any idea why acceptor is volatile in SocketServer?

I think this is so that the thread that calls shutdown() is guaranteed to see the latest value of acceptor. The thread that calls shutdown() is typically different from the thread that calls startup() where acceptor is initialized. Thanks, Jun On Tue, Mar 31, 2015 at 4:54 PM, Gwen Shapira wrote

Re: Does system_test work on MacOs?

It's been awhile and I mostly ran them in VMs, but I don't recall having any issues running them on OS X. -Ewen On Tue, Mar 31, 2015 at 6:00 PM, Gwen Shapira wrote: > Hi, > > The first line of the system_test README says: > This test framework currently doesn't support MacOS due to different >

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389839#comment-14389839 ] Jiangjie Qin commented on KAFKA-2076: - Return a map of time->offset would be useful. I

Re: Review Request 28769: Patch for KAFKA-1809

> On March 29, 2015, 7:33 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/protocol/ApiVersion.java, > > line 28 > > > > > > Since this is for intra-broker communication, should we move this clas

Does system_test work on MacOs?

Hi, The first line of the system_test README says: This test framework currently doesn't support MacOS due to different "ps" argument options from Linux. The correct ps execution is required to terminate the background running processes properly. I use system_test on MacOS and never had any issue

Re: KIP Hangout - March 31, 2015 - TL;DR;

Yes - we will be using the quota manager integration with KM. The main reason for using KM is that the end-state we agreed on is for all metrics to move over to KM and remove our dependency on YM completely. Thanks, Joel On Tue, Mar 31, 2015 at 09:51:56PM +, Jiangjie Qin wrote: > Thanks for

Re: Review Request 31893: Patch for KAFKA-2013

> On March 31, 2015, 11:47 p.m., Jun Rao wrote: > > core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala, line 261 > > > > > > Could we just put FakeOperation to the delayQueue directly instead of > > wrappi

[jira] [Updated] (KAFKA-2013) benchmark test for the purgatory

[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-2013: Attachment: KAFKA-2013_2015-03-31_17:30:56.patch > benchmark test for the purgatory > --

[jira] [Commented] (KAFKA-2013) benchmark test for the purgatory

[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389764#comment-14389764 ] Yasuhiro Matsuda commented on KAFKA-2013: - Updated reviewboard https://reviews.apa

Re: Review Request 31893: Patch for KAFKA-2013

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/ --- (Updated April 1, 2015, 12:31 a.m.) Review request for kafka. Bugs: KAFKA-201

Any idea why acceptor is volatile in SocketServer?

Hi, In SocketServer.scala, acceptor is defined as a volatile var: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L54 My understanding is that @volatile is used to protect variables that are accessed and modified by multiple threads. However, it loo

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389700#comment-14389700 ] Jay Kreps commented on KAFKA-2076: -- I guess the question we need to answer is whether we

Re: Review Request 31893: Patch for KAFKA-2013

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/#review76975 --- Thanks for the new patch. I really like the desgin of this test! A f

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389652#comment-14389652 ] Jiangjie Qin commented on KAFKA-2076: - Got it. Thanks for the explanation, [~jkreps].

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389553#comment-14389553 ] Jay Kreps commented on KAFKA-2076: -- In terms of mechanics we did a discussion on the cons

Re: KIP Hangout - March 31, 2015 - TL;DR;

Thanks for clarification Joel. Just wondering if we are going to depend on any KM specific features? Asking this because KM metric config has quota in side it. On 3/31/15, 1:53 PM, "Joel Koshy" wrote: >We will be using KM for quota'ing on the new client-id-specific >metrics. > >On Tue, Mar 31, 2

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389483#comment-14389483 ] Jay Kreps commented on KAFKA-2076: -- Actually rather than trying to ensure we have everyth

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389471#comment-14389471 ] Parth Brahmbhatt commented on KAFKA-2035: - As stated in description, the current T

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389468#comment-14389468 ] Jay Kreps commented on KAFKA-2035: -- Currently this is the log itself--Log.config always g

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389466#comment-14389466 ] Jiangjie Qin commented on KAFKA-2076: - [~jkreps] Totally agree that we need to think a

[jira] [Commented] (KAFKA-2016) RollingBounceTest takes long

[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389452#comment-14389452 ] Sriharsha Chintalapani commented on KAFKA-2016: --- [~malaskat] Sorry for the d

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389441#comment-14389441 ] Aditya Auradkar commented on KAFKA-1546: Good point Jun. I've added those metrics

[jira] [Updated] (KAFKA-1546) Automate replica lag tuning

[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-1546: --- Attachment: documentation.diff > Automate replica lag tuning > --- > >

Re: Review Request 30809: Patch for KAFKA-1888

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/#review78270 --- bin/kafka-run-class.sh

[jira] [Commented] (KAFKA-2016) RollingBounceTest takes long

[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389421#comment-14389421 ] Gwen Shapira commented on KAFKA-2016: - I believe Ted addressed your comments, [~harsha

[jira] [Commented] (KAFKA-1961) Looks like its possible to delete _consumer_offsets topic

[ https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389409#comment-14389409 ] Gwen Shapira commented on KAFKA-1961: - pinging for review again [~nehanarkhede]... >

[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException

[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389407#comment-14389407 ] Aravind commented on KAFKA-2078: Also continuously getting this INFO logged: [PDT] 2015-0

Re: KIP Hangout - March 31, 2015 - TL;DR;

We will be using KM for quota'ing on the new client-id-specific metrics. On Tue, Mar 31, 2015 at 08:44:44PM +, Jiangjie Qin wrote: > Thanks a lot for the summary, Gwen! > About the Quota, does that mean the first quota implementation will be > based on YM? I¹m thinking can we pursue a quota so

Re: KIP Hangout - March 31, 2015 - TL;DR;

Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which

[jira] [Created] (KAFKA-2080) quick cleanup of producer performance scripts

Gwen Shapira created KAFKA-2080: --- Summary: quick cleanup of producer performance scripts Key: KAFKA-2080 URL: https://issues.apache.org/jira/browse/KAFKA-2080 Project: Kafka Issue Type: Bug

Re: Can I be added as a contributor?

Done for both. Thanks, Jun On Mon, Mar 30, 2015 at 9:23 AM, Brock Noland wrote: > Hi, > > Could I be added as a contributor and to confluence? I am brocknoland > on JIRA and brockn at gmail on confluence. > > Cheers! > Brock >

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389322#comment-14389322 ] Jun Rao commented on KAFKA-1546: For the doc change, do we need to make the following chan

Re: [DISCUSS] KIP-11- Authorization design for kafka security

21. What you suggested makes sense. Could you include the categorization of each request in the wiki? 24. We have a jira (KAFKA-1595) to look for a better json parser. However, it depends on dropping the scala 2.9.x support, which is being discussed. Thanks, Jun On Tue, Mar 31, 2015 at 10:56 AM

[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException

[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389231#comment-14389231 ] Aravind commented on KAFKA-2078: Sorry, above error is for some other test. For this test

KIP Hangout - March 31, 2015 - TL;DR;

Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all A

Re: [DISCUSS] KIP-11- Authorization design for kafka security

From the design doc , one of the added config: * kafka.superusers: list of users that will be given superuser access. These users will have access to everything. Users should set this to the user kafka broker processes are running as to avoid duplicate configuration for every single topic like ALL

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389020#comment-14389020 ] Jay Kreps commented on KAFKA-2076: -- Hey [~granthenke] I don't really get the relationship

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Thanks for reviewing, comments inline: On 3/31/15, 9:21 AM, "Jun Rao" mailto:j...@confluent.io>> wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to

[jira] [Updated] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2035: Status: Patch Available (was: Open) > Add a topic config cache. > -

[jira] [Updated] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2035: Attachment: KAFKA-2035_2015-03-31_10:52:12.patch > Add a topic config cache. > -

[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException

[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388944#comment-14388944 ] Aravind commented on KAFKA-2078: Harsha, I see quite a few of these below errors at that

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388945#comment-14388945 ] Parth Brahmbhatt commented on KAFKA-2035: - Updated reviewboard https://reviews.apa

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388947#comment-14388947 ] Grant Henke commented on KAFKA-2076: This functionality and the edge cases that [~jkre

Re: Review Request 32460: Patch for KAFKA-2035

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32460/ --- (Updated March 31, 2015, 5:52 p.m.) Review request for kafka. Summary (update

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Yes in case of kerberos we will use superACL and this will be equivalent to kafka broker’s principal name. But in SSL as two-way auth is not mandatory the only option if we want enforce authorizer in case of ssl is to force two-way auth. Again this can be an issue on client side , lets say if a p

[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException

[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388898#comment-14388898 ] Sriharsha Chintalapani commented on KAFKA-2078: --- [~aravind2015] do you see a

Re: [DISCUSS] KIP-11- Authorization design for kafka security

>Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? I feel, in Kerberos environment, “kafka” keytab wo

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1433#comment-1433 ] Joel Koshy commented on KAFKA-1910: --- BTW, just to be clear, what I would like to discuss

[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.

[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388879#comment-14388879 ] Jay Kreps commented on KAFKA-2076: -- This makes sense, and would be useful. The question

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai

[jira] [Comment Edited] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException

[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14387612#comment-14387612 ] Aravind edited comment on KAFKA-2078 at 3/31/15 5:18 PM: - Hi Harsh

Re: [DISCUSS] KIP-11- Authorization design for kafka security

>21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERAT

Re: [DISCUSSION] Keep docs updated per jira

+1 On Tue, Mar 31, 2015 at 9:23 AM, Jay Kreps wrote: > Yeah the protocol should probably move off the wiki and into the > release-versioned docs. > > -Jay > > On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy wrote: > >> Also for the wikis - those should probably correspond to the latest >> released v

Re: Metrics package discussion

> My main concern is that we don't do the migration in 0.8.3, we will be left with some metrics in YM format and some others in KM format (as we start sharing client code on the broker). This is probably a worse situation to be in. +1. I am not sure how our servo adaptor will work if there are two

Re: Metrics package discussion

(2) The metrics are clearly part of the client API and we are not changing that (at least for the new client). Arguably, the metrics are also part of the broker side API. However, since they affect fewer parties (mostly just the Kafka admins), it may be easier to make those changes. My main concer

Re: Metrics package discussion

(2) I believe we agreed that our metrics are a public API. I believe we also agree we don't break API in minor releases. So, it seems obvious to me that we can't make breaking changes to metrics in minor releases. I'm not convinced "we did it in the past" is a good reason to do it again. Is there

Re: [DISCUSSION] Keep docs updated per jira

Yeah the protocol should probably move off the wiki and into the release-versioned docs. -Jay On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy wrote: > Also for the wikis - those should probably correspond to the latest > released version right? So for e.g., if we add or modify the protocol > on tru

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What abo

Re: KIP discussion Mar 24 at 11am PST

Jun, I would like to participate but have not received the invite. Can you please send to me? Thanks. Tong Li OpenStack & Kafka Community Development Building 501/B205 liton...@us.ibm.com From: Jun Rao To: "dev@kafka.apache.org" Date: 03/27/2015 10:35 PM Subject:Re: KIP d

Re: [DISCUSSION] Keep docs updated per jira

For the wire protocol in the wiki, we need to tag each new request version with the first release when it's supported. So, currently, any new request version introduced in trunk will be tagged with 0.8.3. Thanks, Jun On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy wrote: > Also for the wikis - tho

Re: Metrics package discussion

(2) Not sure why we can't do this in 0.8.3. We changed the metrics names in 0.8.2 already. Given that we need to share code btw the client and the core, and we need to keep the metrics consistent on the broker, it seems that we have no choice but to migrate to KM. If so, it seems that the sooner th

[jira] [Comment Edited] (KAFKA-1932) kafka topic (creation) templates

[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL edited comment on KAFKA-1932 at 3/31/15 8:41 AM: -

[jira] [Comment Edited] (KAFKA-1932) kafka topic (creation) templates

[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL edited comment on KAFKA-1932 at 3/31/15 8:32 AM: -

[jira] [Commented] (KAFKA-1932) kafka topic (creation) templates

[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL commented on KAFKA-1932: [~becket_qin] , thanks, I wasn't aware of KIP's.