Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/ --- (Updated May 18, 2015, 2:58 p.m.) Review request for kafka. Bugs: 1928 and

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

2015-05-18 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-18_17:57:39.patch Move kafka.network over to using the network

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-18 Thread Sriharsha Chintalapani
Gianmarco,             I’ll send the patch soon. Thanks, Harsha On May 18, 2015 at 1:34:50 AM, Gianmarco De Francisci Morales (g...@apache.org) wrote: Hi, If everything is in order, can we proceed to implement it? Cheers, -- Gianmarco On 13 May 2015 at 03:06, Jiangjie Qin

[jira] [Commented] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

2015-05-18 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548121#comment-14548121 ] Gwen Shapira commented on KAFKA-1928: - Updated reviewboard

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Gwen Shapira
On May 15, 2015, 10:12 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/network/MultiSend.java, line 74 https://reviews.apache.org/r/33065/diff/11/?file=961086#file961086line74 Actually, with MultiSend, we will be sending a 4-byte size plus the payload, the

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Gwen Shapira
On May 15, 2015, 10:12 p.m., Jun Rao wrote: core/src/main/scala/kafka/network/SocketServer.scala, line 354 https://reviews.apache.org/r/33065/diff/11/?file=961124#file961124line354 I guess we will make use of this in a followup patch? Actually, I think its no longer needed and can

Re: Review Request 33620: Patch for KAFKA-1690

2015-05-18 Thread Rajini Sivaram
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review84144 ---

[jira] [Commented] (KAFKA-2194) Produce request failure after Kafka + Zookeeper restart

2015-05-18 Thread Ian Morgan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547704#comment-14547704 ] Ian Morgan commented on KAFKA-2194: --- Hi Jiangjie, thanks for the reply. I've tried both

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

2015-05-18 Thread Gianmarco De Francisci Morales
Hi, If everything is in order, can we proceed to implement it? Cheers, -- Gianmarco On 13 May 2015 at 03:06, Jiangjie Qin j...@linkedin.com.invalid wrote: Hi Harsha, If you open this link https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposa ls All the KIPs are

[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-05-18 Thread Will Funnell (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547733#comment-14547733 ] Will Funnell commented on KAFKA-1977: - [~vamsi360] I've tested this with 0.8.1.2 and

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/ --- (Updated May 18, 2015, 3:56 p.m.) Review request for kafka. Bugs: 1928 and

[jira] [Commented] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

2015-05-18 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548197#comment-14548197 ] Gwen Shapira commented on KAFKA-1928: - Updated reviewboard

[jira] [Updated] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics

2015-05-18 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1374: --- Attachment: KAFKA-1374_2015-05-18_22:55:48.patch LogCleaner (compaction) does not support

Re: Review Request 24214: Patch for KAFKA-1374

2015-05-18 Thread Manikumar Reddy O
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24214/ --- (Updated May 18, 2015, 5:29 p.m.) Review request for kafka. Bugs: KAFKA-1374

Re: Review Request 24214: Patch for KAFKA-1374

2015-05-18 Thread Manikumar Reddy O
On May 12, 2015, 2:01 p.m., Joel Koshy wrote: core/src/main/scala/kafka/log/LogCleaner.scala, line 409 https://reviews.apache.org/r/24214/diff/9/?file=824405#file824405line409 I would suggest one of two options over this (i.e., instead of two helper methods) - Inline both

Re: [Discussion] Using Client Requests and Responses in Server

2015-05-18 Thread Jun Rao
Andri, Let me clarify a bit how things work now. You can see if this fits your need or if it can be improved. If you look at OffsetCommitRequest, our convention is the following. 1. The request object can be constructed from a set of required fields. The client typically constructs a request

[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics

2015-05-18 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548354#comment-14548354 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard

Re: Writing a client: Connection pooling

2015-05-18 Thread Guozhang Wang
Hi Warren, Maybe I was a little misleading in my previous email: 1. The statement that The server guarantees that on a single TCP connection, requests will be processed in the order is still valid. On a single connection, all types of requests (produce/fetch/metadata) are handled in order. 2.

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

2015-05-18 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548343#comment-14548343 ] Jiangjie Qin commented on KAFKA-1778: - Hey [~joestein], I'm a little bit confused

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

2015-05-18 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-18_18:55:38.patch Move kafka.network over to using the network

[VOTE] KIP-21 Dynamic Configuration

2015-05-18 Thread Aditya Auradkar
https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration Aditya

Re: Next is 0.8.3 or is it 0.9?

2015-05-18 Thread Neha Narkhede
The next version will be 0.8.3. On Sun, May 17, 2015 at 9:55 PM, Jaikiran Pai jai.forums2...@gmail.com wrote: Hello everyone, What is the next planned version of Kafka going to be? I vaguely remember reading in some mail that it will be 0.9 and trunk is where it would be released from. I

Re: [Discussion] Using Client Requests and Responses in Server

2015-05-18 Thread Andrii Biletskyi
Hi all, I started working on it and it seems we are going the wrong way. So it appears we need to distinguish constructors by versions in request/response (so we can set correct schema). Request/Response classes will look like: class SomeRequest extends AbstractRequest {

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/#review84158 --- Thanks for the latest patch. Looks good. Just a few minor comments

Re: [VOTE] KIP-21 Dynamic Configuration

2015-05-18 Thread Jun Rao
Aditya, Another thing to consider. In KIP-4, we are adding a new RPC request to change and retrieve topic configs. Do we want to add a similar RPC request to change configs per client id? If so, do we want to introduce a separate new request or have a combined new request for both topic and

Re: Review Request 33552: Patch for KAFKA-2142

2015-05-18 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33552/#review84191 ---

Re: [Discussion] Using Client Requests and Responses in Server

2015-05-18 Thread Jun Rao
Hi, Andrii, Good point on the constructor for MetadataResponse. Perhaps in V2 of MetadataResponse, we can add a new constructor with both cluster and a topicConfig map. Currently, cluster doesn't really need topic configs and we can leave it as it is now. Thanks, Jun On Mon, May 18, 2015 at

[jira] [Updated] (KAFKA-2169) Upgrade to zkclient-0.5

2015-05-18 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2169: --- Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Thanks for

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

2015-05-18 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549117#comment-14549117 ] Jun Rao commented on KAFKA-1778: [~anigam], could you explain a bit the use case of

[jira] [Assigned] (KAFKA-2182) zkClient dies if there is any exception while reconnecting

2015-05-18 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt reassigned KAFKA-2182: --- Assignee: Parth Brahmbhatt zkClient dies if there is any exception while

Re: jira backlogs

2015-05-18 Thread Joel Koshy
I forgot to add - I believe you need to be logged into jira at the time you view the page. The search basically filters by patch-available, non-empty reviewer and ordered by updated date. Thanks, Joel On Mon, May 18, 2015 at 03:10:03PM -0700, Joel Koshy wrote: Here is (roughly) what I had in

[jira] [Commented] (KAFKA-2199) Make signing artifacts optional, setting maven repository possible from command line

2015-05-18 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549151#comment-14549151 ] Ewen Cheslack-Postava commented on KAFKA-2199: -- Created reviewboard

Re: [Discussion] Using Client Requests and Responses in Server

2015-05-18 Thread Gwen Shapira
Sorry for the confusion regarding errorResponses... On Mon, May 18, 2015 at 9:10 PM, Andrii Biletskyi andrii.bilets...@stealth.ly wrote: Jun, Thanks for the explanation. I believe my understanding is close to what you have written. I see, I still think that this approach is somewhat

Re: Review Request 34273: remove roundrobin identical topic constraint in consumer coordinator

2015-05-18 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34273/#review84223 --- core/src/main/scala/kafka/coordinator/PartitionAssignor.scala

[jira] [Created] (KAFKA-2199) Make signing artifacts optional, setting maven repository possible from command line

2015-05-18 Thread Ewen Cheslack-Postava (JIRA)
Ewen Cheslack-Postava created KAFKA-2199: Summary: Make signing artifacts optional, setting maven repository possible from command line Key: KAFKA-2199 URL:

[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-05-18 Thread Vamsi Subhash Achanta (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549252#comment-14549252 ] Vamsi Subhash Achanta commented on KAFKA-1977: -- Hi Will, Even I have applied

Re: jira backlogs

2015-05-18 Thread Joel Koshy
Here is (roughly) what I had in mind for this: https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review#Patchsubmissionandreview-Reviewqueue Earlier versions of wiki/jira integration were much better in that the inline table had pagination within the wiki page itself.

[jira] [Commented] (KAFKA-2160) DelayedOperationPurgatory should remove the pair in watchersForKey with empty watchers list

2015-05-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549234#comment-14549234 ] Guozhang Wang commented on KAFKA-2160: -- Updated reviewboard

[jira] [Updated] (KAFKA-2160) DelayedOperationPurgatory should remove the pair in watchersForKey with empty watchers list

2015-05-18 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2160: - Attachment: KAFKA-2160_2015-05-18_14:07:48.patch DelayedOperationPurgatory should remove the

Re: Review Request 34283: Third Attempt: use a global read-write lock in purgatory; remove join-group purgatory from coordinator

2015-05-18 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34283/ --- (Updated May 18, 2015, 9:08 p.m.) Review request for kafka. Bugs: KAFKA-2160

[DISCUSSION] Two behaviors in new producer

2015-05-18 Thread Jiangjie Qin
Hi, I have two questions when writing patch for KAFKA-2142 and needs some help: 1. In send(), we actually might call callback.onCompletion(), this might break the guarantee that callbacks are fired in sending order of a particular partition. 2. In RecordAccumulator, current logic is that

[jira] [Resolved] (KAFKA-1082) zkclient dies after UnknownHostException in zk reconnect

2015-05-18 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1082. Resolution: Duplicate Fix Version/s: 0.8.3 This is already resolved in KAFKA-2169. zkclient dies

RE: [VOTE] KIP-21 Dynamic Configuration

2015-05-18 Thread Aditya Auradkar
Yeah, that was just a typo. I've fixed it. Thanks for calling it out. In KIP-4, I believe we have 3 types of requests: CreateTopic, AlterTopic and DeleteTopic. The topic configs are a sub-type of the Create and Alter commands. I think it would be nice to simply have a AlterConfig command that

[jira] [Comment Edited] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-05-18 Thread Vamsi Subhash Achanta (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549252#comment-14549252 ] Vamsi Subhash Achanta edited comment on KAFKA-1977 at 5/18/15 9:21 PM:

Re: [Discussion] Using Client Requests and Responses in Server

2015-05-18 Thread Andrii Biletskyi
No worries, I think it's hard to foresee all nuances before actually implementing/writing code. I also have a feeling there will be some other issues with requests versioning so I plan to finish end-to-end MetadataRequest_V1 to ensure we don't miss anything in terms of AbstractRequest/Response

Re: Review Request 34283: Third Attempt: use a global read-write lock in purgatory; remove join-group purgatory from coordinator

2015-05-18 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34283/ --- (Updated May 18, 2015, 9:07 p.m.) Review request for kafka. Summary

Re: [DISCUSSION] Two behaviors in new producer

2015-05-18 Thread Jay Kreps
Hey Becket, 1. I agree that this is a problem. I think this ended up this way because there are two ways a TimeoutException can be thrown, the server can timeout or we can timeout waiting for memory. I think the complaint at the time was that it was annoying that you needed to write the same

Re: Review Request 34394: Patch for KAFKA-1907

2015-05-18 Thread Jaikiran Pai
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34394/ --- (Updated May 19, 2015, 5:20 a.m.) Review request for kafka. Bugs: KAFKA-1907

Review Request 34394: Patch for KAFKA-1907

2015-05-18 Thread Jaikiran Pai
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34394/ --- Review request for kafka. Bugs: KAFKA-1907

[jira] [Updated] (KAFKA-1907) ZkClient can block controlled shutdown indefinitely

2015-05-18 Thread jaikiran pai (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1907: Attachment: KAFKA-1907.patch ZkClient can block controlled shutdown indefinitely

[jira] [Updated] (KAFKA-1907) ZkClient can block controlled shutdown indefinitely

2015-05-18 Thread jaikiran pai (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1907: Status: Patch Available (was: In Progress) ZkClient can block controlled shutdown indefinitely

[jira] [Commented] (KAFKA-1907) ZkClient can block controlled shutdown indefinitely

2015-05-18 Thread jaikiran pai (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14549796#comment-14549796 ] jaikiran pai commented on KAFKA-1907: - Created reviewboard

Re: Writing a client: Connection pooling

2015-05-18 Thread Warren Falk
Thanks Guozhang, Actually your previous email was clear and I understood it. Current broker design means that parallel requests require parallel connections. But I think you misunderstood me. I am not asking if that is how the broker works, now, I am proposing that the broker should not work

[jira] [Created] (KAFKA-2200) kafkaProducer.send() should not call callback.onCompletion()

2015-05-18 Thread Jiangjie Qin (JIRA)
Jiangjie Qin created KAFKA-2200: --- Summary: kafkaProducer.send() should not call callback.onCompletion() Key: KAFKA-2200 URL: https://issues.apache.org/jira/browse/KAFKA-2200 Project: Kafka

Re: [DISCUSSION] Two behaviors in new producer

2015-05-18 Thread Jiangjie Qin
Hi Jay, Thanks for the explanations, Jay. That makes sense. I¹ll create a ticket to for (1). For (2), as you said it depends on traffic pattern a lot. I will try to run some benchmark to see if we want to change (2), but we might still want to make the java doc clearer if possible. Thanks,

[jira] [Updated] (KAFKA-2199) Make signing artifacts optional, setting maven repository possible from command line

2015-05-18 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2199: - Status: Patch Available (was: Open) Make signing artifacts optional, setting

[jira] [Updated] (KAFKA-2199) Make signing artifacts optional, setting maven repository possible from command line

2015-05-18 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2199: - Attachment: KAFKA-2199.patch Make signing artifacts optional, setting maven

Review Request 34369: Patch for KAFKA-2199

2015-05-18 Thread Ewen Cheslack-Postava
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34369/ --- Review request for kafka. Bugs: KAFKA-2199

Re: Writing a client: Connection pooling

2015-05-18 Thread Dana Powers
cc: kafka-clients mailing list On May 18, 2015 4:24 PM, Warren Falk war...@warrenfalk.com wrote: Thanks Guozhang, Actually your previous email was clear and I understood it. Current broker design means that parallel requests require parallel connections. But I think you misunderstood me.

[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-05-18 Thread Vamsi Subhash Achanta (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547606#comment-14547606 ] Vamsi Subhash Achanta commented on KAFKA-1977: -- can this be merged in the old

Re: [Vote] KIP-11 Authorization design for kafka security

2015-05-18 Thread Joe Stein
+1 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, May 15, 2015 at 7:35 PM, Jun Rao j...@confluent.io wrote: +1 Thanks, Jun On Fri, May 15, 2015 at 9:18 AM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi,

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

2015-05-18 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14547838#comment-14547838 ] Joe Stein commented on KAFKA-1778: -- I was thinking that the broker when starting up would

Re: Review Request 33065: Patch for KAFKA-1928

2015-05-18 Thread Gwen Shapira
On May 15, 2015, 10:12 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java, lines 159-162 https://reviews.apache.org/r/33065/diff/11/?file=961076#file961076line159 We don't need this since the idle conenctions only need to be closed on