Re: Review Request 32199: Patch for KAFKA-1858

2015-03-19 Thread Adrian Preston
> On March 18, 2015, 3:17 p.m., Jiangjie Qin wrote: > > This will definitely make the test more stable. I am just thinking does it > > make sense to add a non deamon thread check in each test? So we can make > > sure there is no deamon thread left from some tests. Good question. I think this d

[jira] [Updated] (KAFKA-2016) RollingBounceTest takes long

2015-03-19 Thread Ted Malaska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated KAFKA-2016: --- Attachment: KAFKA-2016-2.patch Made adjustment based on Sriharsha Chintalapani review comments > Roll

[jira] [Commented] (KAFKA-1715) better advertising of the bound and working interface

2015-03-19 Thread Adrian Preston (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369322#comment-14369322 ] Adrian Preston commented on KAFKA-1715: --- Hi [~joestein], I found this bug via the 'n

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-19 Thread Guozhang Wang
+1 on broker writing to ZK for async handling. I was thinking that in the end state the admin requests would be eventually sent to controller either through re-routing or clients discovering them, instead of letting controller listen on ZK admin path. But thinking about it a second time, I think it

[jira] [Created] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Jason Rosenberg (JIRA)
Jason Rosenberg created KAFKA-2032: -- Summary: ConsumerConfig doesn't validate partition.assignment.strategy values Key: KAFKA-2032 URL: https://issues.apache.org/jira/browse/KAFKA-2032 Project: Kafka

[jira] [Commented] (KAFKA-1930) Move server over to new metrics library

2015-03-19 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369623#comment-14369623 ] Jun Rao commented on KAFKA-1930: Another thing that we need to think through is the histog

[jira] [Commented] (KAFKA-1912) Create a simple request re-routing facility

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369627#comment-14369627 ] Guozhang Wang commented on KAFKA-1912: -- I was thinking that in the end state the admi

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Jun Rao
1. Delay + no error seems reasonable to me. However, I do feel that we need to give the client an indicator that it's being throttled, instead of doing this silently. For that, we probably need to evolve the produce/fetch protocol to include an extra status field in the response. We probably need t

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

2015-03-19 Thread Andrii Biletskyi
Guys, I like this idea too. Let's stick with that. I'll update KIP accordingly. I was also thinking we can avoid adding dedicated status check requests for topic commands. - We have everything in DescribeTopic for that! E.g.: User issued CreateTopic - to check the status client sends DescribeTopic

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Jay Kreps
Yeah originally our plan was errors when the quota was exceeded, enforced at a per-topic level. But after some discussion this doesn't work that well. The problem is the consumer. Popular topics have many consumers, any one of which can either misbehave or do some kind of "catch-up" or full reload.

Re: Review Request 31568: Patch for KAFKA-1989

2015-03-19 Thread Yasuhiro Matsuda
> On March 17, 2015, 8:56 p.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/utils/timer/TimerTaskList.scala, line 72 > > > > > > It seems the task entry of the task will only be set once throughout > > its life

Re: Review Request 32199: Patch for KAFKA-1858

2015-03-19 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32199/#review77060 --- LGTM - Mayuresh Gharat On March 18, 2015, 11:12 a.m., Adrian Pres

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Neha Narkhede
I'm in favor of (1) for the sake of simplicity and as Jay mentions to reduce the number of different APIs. Can you explain when (1) does not work? On Wed, Mar 18, 2015 at 2:52 PM, Jay Kreps wrote: > Personally I'm in favor of (1) just to reduce the number of different APIs. > People will find th

Re: Review Request 31742: Patch for KAFKA-527

2015-03-19 Thread Yasuhiro Matsuda
> On March 19, 2015, 3:58 a.m., Jun Rao wrote: > > core/src/test/scala/unit/kafka/message/MessageWriterTest.scala, lines 31-33 > > > > > > Could we just use Random.nextBytes()? My intention was to have some control of

mirror maker and failover to another data center

2015-03-19 Thread Kane Kim
Hello, What's the best strategy for failover when using mirror-maker to replicate across datacenters? As I understand offsets in both datacenters will be different, how consumers should be reconfigured to continue reading from the same point where they stopped without data loss and/or duplication?

Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/ --- Review request for kafka. Bugs: KAFKA-2032 https://issues.apache.org/jira/b

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369863#comment-14369863 ] Parth Brahmbhatt commented on KAFKA-2032: - Created reviewboard https://reviews.apa

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Assignee: Parth Brahmbhatt Status: Patch Available (was: Open) > ConsumerConfig doesn

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Attachment: KAFKA-2032.patch > ConsumerConfig doesn't validate partition.assignment.strategy

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369871#comment-14369871 ] Jason Rosenberg commented on KAFKA-2032: Currently, I am using the scala ConsumerC

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/ --- (Updated March 19, 2015, 6:42 p.m.) Review request for kafka. Bugs: KAFKA-203

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369877#comment-14369877 ] Parth Brahmbhatt commented on KAFKA-2032: - Updated reviewboard https://reviews.apa

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Attachment: KAFKA-2032_2015-03-19_11:42:07.patch > ConsumerConfig doesn't validate partition

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/ --- (Updated March 19, 2015, 6:44 p.m.) Review request for kafka. Bugs: KAFKA-203

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2015-03-19 Thread Steven Zhen Wu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369878#comment-14369878 ] Steven Zhen Wu commented on KAFKA-1660: --- when this patch is merged, would it be poss

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369881#comment-14369881 ] Jason Rosenberg commented on KAFKA-2032: What does "blah" signify in the java clie

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369880#comment-14369880 ] Parth Brahmbhatt commented on KAFKA-2032: - Updated reviewboard https://reviews.apa

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Attachment: KAFKA-2032_2015-03-19_11:44:48.patch > ConsumerConfig doesn't validate partition

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/ --- (Updated March 19, 2015, 6:47 p.m.) Review request for kafka. Bugs: KAFKA-203

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369884#comment-14369884 ] Parth Brahmbhatt commented on KAFKA-2032: - Updated reviewboard https://reviews.apa

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Attachment: KAFKA-2032_2015-03-19_11:47:24.patch > ConsumerConfig doesn't validate partition

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Jason Rosenberg
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/#review77081 --- core/src/main/scala/kafka/consumer/ConsumerConfig.scala

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369898#comment-14369898 ] Parth Brahmbhatt commented on KAFKA-2032: - I keep forgetting the kafka post-review

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
> On March 19, 2015, 6:52 p.m., Jason Rosenberg wrote: > > core/src/main/scala/kafka/consumer/ConsumerConfig.scala, line 90 > > > > > > Should this be called 'validatePartitionAssignmentStrategy'? > > > > At an

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Parth Brahmbhatt
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/ --- (Updated March 19, 2015, 7:19 p.m.) Review request for kafka. Bugs: KAFKA-203

[jira] [Updated] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2032: Attachment: KAFKA-2032_2015-03-19_12:19:45.patch > ConsumerConfig doesn't validate partition

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Parth Brahmbhatt (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369948#comment-14369948 ] Parth Brahmbhatt commented on KAFKA-2032: - Updated reviewboard https://reviews.apa

[jira] [Commented] (KAFKA-2032) ConsumerConfig doesn't validate partition.assignment.strategy values

2015-03-19 Thread Jason Rosenberg (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369952#comment-14369952 ] Jason Rosenberg commented on KAFKA-2032: LGTM Thanks! > ConsumerConfig doesn't va

Re: Review Request 32251: Patch for KAFKA-2032

2015-03-19 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32251/#review77090 --- Ship it! Ship It! - Sriharsha Chintalapani On March 19, 2015, 7:

RE: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Aditya Auradkar
Some thoughts about metrics in quotas. I took a look at the new metric package in clients/common and thought it was very nicely done. I especially liked using data points from current and previous windows. The Quota implementation is pretty general and I think we should use it in this quota pr

Review Request 32258: Follow-up KAFKA-1910

2015-03-19 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32258/ --- Review request for kafka. Bugs: KAFKA-1910 https://issues.apache.org/jira/b

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370061#comment-14370061 ] Guozhang Wang commented on KAFKA-1910: -- Created reviewboard https://reviews.apache.or

[jira] [Updated] (KAFKA-1910) Refactor KafkaConsumer

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1910: - Attachment: KAFKA-1910.patch > Refactor KafkaConsumer > -- > >

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Joel Koshy
(1) should work, but as Jun suggested earlier in the thread it is slightly misleading. The (intuitive) post-condition of "close" is that the producer has shutdown - i.e., its sender thread, closed its metrics, serializer/deserializer, etc. That is not necessarily a post-condition of "close(0)" alth

[jira] [Commented] (KAFKA-1546) Automate replica lag tuning

2015-03-19 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370193#comment-14370193 ] Aditya Auradkar commented on KAFKA-1546: [~nehanarkhede][~junrao] I've incorporate

Re: Review Request 32258: Follow-up KAFKA-1910

2015-03-19 Thread Onur Karaman
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32258/#review77114 --- Ship it! clients/src/main/java/org/apache/kafka/common/record/Memo

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Jiangjie Qin
Hi Neha, As Joel said, (1) and (2) provides almost exact same function as far as I can see. I actually have no strong preference. The only difference is that (2) provides the close() semantic meaning of making sure all the resources have gone away, at the cost of adding another abort() interface.

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Jay Kreps
Yeah guys, I think this is one where the cure is worse than the disease. Let's just have the two close methods, that is confusing enough. Adding a third won't help. I understand that having a few microseconds before the thread shuts down could be unexpected but I think there is nothing inherently w

offset returned for produce request is different btw 0.8.1 and 0.8.2

2015-03-19 Thread Jun Rao
Hi, Everyone, Our new java producer in 0.8.2 now exposes message offset to the client. If you are utilizing the returned offset, you will need to make sure that your broker is on 0.8.2 too. This is because in 0.8.1, we had a bug in the broker that returns inconsistent offset in the response to a p

[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370281#comment-14370281 ] Guozhang Wang commented on KAFKA-1910: -- Pushed two fixes to the new consumer to trunk

Build failed in Jenkins: Kafka-trunk #428

2015-03-19 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-1910; Fix two bugs on MemoryRecords and KafkaConsumer; reviewed by Onur Karaman -- Started by an SCM change Building remotely on H11 (Ubuntu ubuntu) in workspace

Re: Review Request 31893: Patch for KAFKA-2013

2015-03-19 Thread Yasuhiro Matsuda
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/ --- (Updated March 19, 2015, 11:31 p.m.) Review request for kafka. Bugs: KAFKA-20

[jira] [Updated] (KAFKA-2013) benchmark test for the purgatory

2015-03-19 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-2013: Attachment: KAFKA-2013_2015-03-19_16:30:52.patch > benchmark test for the purgatory > --

[jira] [Commented] (KAFKA-2013) benchmark test for the purgatory

2015-03-19 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370341#comment-14370341 ] Yasuhiro Matsuda commented on KAFKA-2013: - Updated reviewboard https://reviews.apa

Review Request 32265: WIP KAFKA-1892

2015-03-19 Thread Guozhang Wang
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32265/ --- Review request for kafka. Bugs: KAFKA-1892 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1892) System tests for the new consumer and co-ordinator

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1892: - Status: Patch Available (was: Open) > System tests for the new consumer and co-ordinator > --

[jira] [Updated] (KAFKA-1892) System tests for the new consumer and co-ordinator

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1892: - Attachment: KAFKA-1892.patch > System tests for the new consumer and co-ordinator > --

[jira] [Commented] (KAFKA-1892) System tests for the new consumer and co-ordinator

2015-03-19 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370362#comment-14370362 ] Guozhang Wang commented on KAFKA-1892: -- Created reviewboard https://reviews.apache.or

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

2015-03-19 Thread Abhishek Nigam (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370385#comment-14370385 ] Abhishek Nigam commented on KAFKA-1778: --- I have a design for pinning the controller

[jira] [Work started] (KAFKA-1778) Create new re-elect controller admin function

2015-03-19 Thread Abhishek Nigam (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-1778 started by Abhishek Nigam. - > Create new re-elect controller admin function > -

Build failed in Jenkins: KafkaPreCommit #41

2015-03-19 Thread Apache Jenkins Server
See Changes: [wangguoz] KAFKA-1910; Fix two bugs on MemoryRecords and KafkaConsumer; reviewed by Onur Karaman [wangguoz] trivial fix on coding style -- [...truncated 2588 lines...] kafka.log.File

Re: mirror maker and failover to another data center

2015-03-19 Thread Guozhang Wang
Hi Kane, Unfortunately we do not have a perfect solution for this problem yet, since as you said the offsets on different DCs are different. At LinkedIn we bite the bullet on data duplication, i.e. when failover happens, the consumer queries the other DC for some offsets appended some time before

[jira] [Commented] (KAFKA-1778) Create new re-elect controller admin function

2015-03-19 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370426#comment-14370426 ] Mayuresh Gharat commented on KAFKA-1778: I did not get : c) Broker x will rightfu

Build failed in Jenkins: Kafka-trunk #429

2015-03-19 Thread Apache Jenkins Server
See Changes: [wangguoz] trivial fix on coding style -- [...truncated 1194 lines...] kafka.log.LogTest > testAppendMessageWithNullPayload PASSED kafka.log.LogTest > testCorruptLog PASSED kafka.log.Lo

Re: Review Request 31742: Patch for KAFKA-527

2015-03-19 Thread Yasuhiro Matsuda
> On March 19, 2015, 3:58 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/message/ByteBufferMessageSet.scala, lines 43-55 > > > > > > Instead of passing in a function, I think it will be easier to > > understand the

Re: Review Request 31742: Patch for KAFKA-527

2015-03-19 Thread Yasuhiro Matsuda
> On March 19, 2015, 3:58 a.m., Jun Rao wrote: > > core/src/main/scala/kafka/message/MessageWriter.scala, line 144 > > > > > > Shouldn't we test off < b.length and off + len < b.length? This is actually fine. OutputSt

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Joel Koshy
> in KAFKA-1930). I agree that this KIP doesn't need to block on the > migration of the metrics package. Can you clarify the above? i.e., if we are going to quota on something then we would want to have migrated that metric over right? Or do you mean we don't need to complete the migration of all

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Jay Kreps
Yeah I was saying was that we are blocked on picking an approach for metrics but not necessarily the full conversion. Clearly if we pick the new metrics package we would need to implement the two metrics we want to quota on. But the conversion of the remaining metrics can be done asynchronously. -

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Jiangjie Qin
OK, we are going to follow (1) then. It looks the KIP does not need further updates. Should we resume the voting process? On 3/19/15, 3:28 PM, "Jay Kreps" wrote: >Yeah guys, I think this is one where the cure is worse than the disease. >Let's just have the two close methods, that is confusing en

Re: [VOTE] KIP-15 add a close method with timeout to KafkaProducer

2015-03-19 Thread Jiangjie Qin
We had some additional discussions on the discussion thread. Pushing up this thread to resume voting. On 3/11/15, 8:47 PM, "Jay Kreps" wrote: >Yeah guys, I'd like to second that. I'd really really love to get the >quality of these to the point where we could broadly solicit user input >and >use

[jira] [Commented] (KAFKA-527) Compression support does numerous byte copies

2015-03-19 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14370703#comment-14370703 ] Yasuhiro Matsuda commented on KAFKA-527: Updated reviewboard https://reviews.apache

Re: Review Request 31742: Patch for KAFKA-527

2015-03-19 Thread Yasuhiro Matsuda
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31742/ --- (Updated March 20, 2015, 4:33 a.m.) Review request for kafka. Bugs: KAFKA-527

[jira] [Updated] (KAFKA-527) Compression support does numerous byte copies

2015-03-19 Thread Yasuhiro Matsuda (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-527: --- Attachment: KAFKA-527_2015-03-19_21:32:24.patch > Compression support does numerous byte copies

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Jun Rao
Hmm, I was thinking a bit differently on the metrics stuff. I think it would be confusing to have some metrics defined in the new metrics package while some others defined in Coda Hale. Those metrics will look different (e.g., rates in Coda Hale will have special attributes such as 1-min-average).

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Jun Rao
So in (1), if a close() or close(timeout) is called from a callback, we will just turn that into a close(0)? Implementation wise, how do we know whether a close() call is made from the sender thread or not? Thanks, Jun On Wed, Mar 18, 2015 at 2:13 PM, Jiangjie Qin wrote: > It looks we have ano

Re: [DISCUSSION] KIP-15 close(timeout) for producer

2015-03-19 Thread Joel Koshy
close can probably just getCurrentThread and check if == senderThread. I'm actually not sure from this thread if there was clear agreement on whether it should change close(timeout)/close() to close(0) or if it should log an error and block up to the timeout. On Thursday, March 19, 2015, Jun Rao

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Joel Koshy
That is a valid concern but in that case I think it would be better to just migrate completely to the new metrics package first. On Thursday, March 19, 2015, Jun Rao wrote: > Hmm, I was thinking a bit differently on the metrics stuff. I think it > would be confusing to have some metrics defined

Re: [KIP-DISCUSSION] KIP-13 Quotas

2015-03-19 Thread Joel Koshy
Actually thinking again - since these will be a few new metrics at the client id level (bytes in and bytes out to start with) maybe it is fine to have the two type of metrics coexist and we can migrate the existing metrics in parallel. On Thursday, March 19, 2015, Joel Koshy wrote: > That is a v