Re: [KIP-DISCUSSION] Mirror Maker Enhancement

2015-02-10 Thread Guozhang Wang
Being late on this thread... I think people are on the same page that: 1. We need the zero data-loss MM for sure. 2. The handler function itself is not adding much complexity to MM. 3. The current zero data-loss approach based on the data channel seems too complex. For 3), I agree with Joel that

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

2015-02-10 Thread Honghai Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315710#comment-14315710 ] Honghai Chen commented on KAFKA-1646: - How about add one more column to recovery-point

[jira] [Commented] (KAFKA-1944) Rename LogCleaner and related classes to LogCompactor

2015-02-10 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315708#comment-14315708 ] Manikumar Reddy commented on KAFKA-1944: Can we wait till we check-in KAFKA-1374?

[jira] [Commented] (KAFKA-1866) LogStartOffset gauge throws exceptions after log.delete()

2015-02-10 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315682#comment-14315682 ] Sriharsha Chintalapani commented on KAFKA-1866: --- Updated reviewboard https:/

[jira] [Updated] (KAFKA-1866) LogStartOffset gauge throws exceptions after log.delete()

2015-02-10 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1866: -- Attachment: KAFKA-1866_2015-02-10_22:50:09.patch > LogStartOffset gauge throws e

Re: Review Request 30084: Patch for KAFKA-1866

2015-02-10 Thread Sriharsha Chintalapani
> On Jan. 26, 2015, 2:35 a.m., Neha Narkhede wrote: > > Can we add a test for this? If we add more metrics in the future, hopefully > > that test would fail if we forget to add the new metrics to this > > removeMetrics() API Added a test please take a look. - Sriharsha ---

Re: Review Request 30084: Patch for KAFKA-1866

2015-02-10 Thread Sriharsha Chintalapani
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30084/ --- (Updated Feb. 11, 2015, 6:50 a.m.) Review request for kafka. Bugs: KAFKA-1866

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

2015-02-10 Thread Jay Kreps
Hey Joel, The data channels are actually a big part of the complexity of the zero data loss design, though, right? Because then you need some reverse channel to flow the acks back to the consumer based on where you are versus just acking what you have read and written (as in the code snippet I put

Re: Review Request 29468: Patch for KAFKA-1805

2015-02-10 Thread Eric Olander
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29468/#review71928 --- clients/src/main/java/org/apache/kafka/clients/producer/ProducerRec

Re: Review Request 29831: Patch for KAFKA-1476

2015-02-10 Thread Eric Olander
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29831/#review71925 --- core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

Re: [DISCUSSION] KAFKA-1697 - make NotEnoughReplicasAfterAppend a non-retriable exception

2015-02-10 Thread Jay Kreps
Yeah there are really two concepts here as I think you noted: 1. Retry safe: we know that the write did not occur 2. Retry fixable: if you send that again it could work (probably there are better names for these). Some things we know did not do a write and may be fixed by retrying (no leader). So

[jira] [Commented] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315418#comment-14315418 ] Aditya Auradkar commented on KAFKA-1943: I'll follow Gwen's suggestion. > Produce

Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints

2015-02-10 Thread Gwen Shapira
On Mon, Feb 9, 2015 at 5:47 PM, Joel Koshy wrote: > For (1) - +1 especially since the existing clients will keep working. > For (2) - I'm less clear on the proposal. Can you incorporate it into > the KIP and/or linked wiki? > Added detail on wire.protocol.version to the KIP (under upgrade plan).

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/ --- (Updated Feb. 11, 2015, 1:06 a.m.) Review request for kafka. Bugs: KAFKA-1697

[jira] [Commented] (KAFKA-1697) remove code related to ack>1 on the broker

2015-02-10 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315329#comment-14315329 ] Gwen Shapira commented on KAFKA-1697: - Updated reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-1697) remove code related to ack>1 on the broker

2015-02-10 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1697: Attachment: KAFKA-1697_2015-02-10_17:06:51.patch > remove code related to ack>1 on the broker >

[jira] [Assigned] (KAFKA-1944) Rename LogCleaner and related classes to LogCompactor

2015-02-10 Thread Ashish Kumar Singh (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh reassigned KAFKA-1944: - Assignee: Ashish Kumar Singh > Rename LogCleaner and related classes to LogCompac

[jira] [Created] (KAFKA-1944) Rename LogCleaner and related classes to LogCompactor

2015-02-10 Thread Gwen Shapira (JIRA)
Gwen Shapira created KAFKA-1944: --- Summary: Rename LogCleaner and related classes to LogCompactor Key: KAFKA-1944 URL: https://issues.apache.org/jira/browse/KAFKA-1944 Project: Kafka Issue Type:

[DISCUSSION] KAFKA-1697 - make NotEnoughReplicasAfterAppend a non-retriable exception

2015-02-10 Thread Gwen Shapira
Hi Kafka Devs, Need your thoughts on retriable exceptions: If a user configures Kafka with min.isr > 1 and there are not enough replicas to safely store the data, there are two possibilities: 1. The lack of replicas was discovered before the message was written. We throw NotEnoughReplicas. 2. Th

[jira] [Commented] (KAFKA-1920) Add a metric to count client side errors in BrokerTopicMetrics

2015-02-10 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315243#comment-14315243 ] Aditya Auradkar commented on KAFKA-1920: [~nehanarkhede] I was referring to bad re

[jira] [Commented] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315241#comment-14315241 ] Gwen Shapira commented on KAFKA-1943: - How about an umbrella ticket and add those as s

[jira] [Commented] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315239#comment-14315239 ] Aditya Auradkar commented on KAFKA-1943: I've filed KAFKA-1936 and 1920 in additio

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 11:34 p.m., Joel Koshy wrote: > > clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java, > > line 24 > > > > > > Actually this is exactly what I meant. i.e.

[jira] [Commented] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315194#comment-14315194 ] Jiangjie Qin commented on KAFKA-1943: - It seems recently we have several tickets creat

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/#review71874 --- clients/src/main/java/org/apache/kafka/common/errors/NotEnoughRepli

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 10:58 p.m., Joel Koshy wrote: > > clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java, > > line 24 > > > > > > I think our interpretation of retriable is

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/#review71869 --- clients/src/main/java/org/apache/kafka/common/errors/NotEnoughRepli

Build failed in Jenkins: KafkaPreCommit #2

2015-02-10 Thread Apache Jenkins Server
See Changes: [neha.narkhede] KAFKA-1476 Added a ConsumerCommand tool that will replace other consumer related tools in the future; reviewed by Neha Narkhede -- [...truncated 1821 lines...] kafka.pro

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 8:12 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/server/KafkaApis.scala, line 166 > > > > > > Rather than do this here in kafka-apis, how about moving this to > > ReplicaManager.appendMes

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 8:12 p.m., Joel Koshy wrote: > > core/src/main/scala/kafka/server/KafkaApis.scala, line 170 > > > > > > Should we just do Errors.INVALID_REQUIRED_ACKS? lol. yeah, I've no clue what I was trying to

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 10:10 p.m., Joel Koshy wrote: > > clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java, > > line 24 > > > > > > Understood, but if someone uses required.ac

Re: Review Request 29831: Patch for KAFKA-1476

2015-02-10 Thread Onur Karaman
> On Feb. 2, 2015, 5:42 p.m., Neha Narkhede wrote: > > Onur, we should be able to check in after these review comments are > > addressed. Also, how would deleting offsets for a group work when the > > offset storage is Kafka? It's fine to not address it in this patch. Can you > > please create

[jira] [Updated] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya A Auradkar updated KAFKA-1943: - Attachment: KAFKA-1943.patch > Producer request failure rate should not include MessageSet

[jira] [Created] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya A Auradkar (JIRA)
Aditya A Auradkar created KAFKA-1943: Summary: Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException Key: KAFKA-1943 URL: https://issues.apache.org/jira/brows

[jira] [Assigned] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar reassigned KAFKA-1943: -- Assignee: Aditya Auradkar > Producer request failure rate should not include MessageSet

[jira] [Commented] (KAFKA-1943) Producer request failure rate should not include MessageSetSizeTooLarge and MessageSizeTooLargeException

2015-02-10 Thread Aditya A Auradkar (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315068#comment-14315068 ] Aditya A Auradkar commented on KAFKA-1943: -- Created reviewboard https://reviews.a

Review Request 30848: Patch for KAFKA-1943

2015-02-10 Thread Aditya Auradkar
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30848/ --- Review request for kafka. Bugs: KAFKA-1943 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1942) ConsumerGroupCommand does not show offset information in ZK for deleted topic

2015-02-10 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-1942: Attachment: delete-topic-describe-group-2-10-2015.txt Attached a sample run showing the bug. > Cons

[jira] [Created] (KAFKA-1942) ConsumerGroupCommand does not show offset information in ZK for deleted topic

2015-02-10 Thread Onur Karaman (JIRA)
Onur Karaman created KAFKA-1942: --- Summary: ConsumerGroupCommand does not show offset information in ZK for deleted topic Key: KAFKA-1942 URL: https://issues.apache.org/jira/browse/KAFKA-1942 Project: Ka

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/#review71857 --- clients/src/main/java/org/apache/kafka/common/errors/NotEnoughRepli

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Gwen Shapira
> On Feb. 10, 2015, 8:12 p.m., Joel Koshy wrote: > > clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java, > > line 24 > > > > > > Can you clarify why this should no longer be

Re: New consumer client

2015-02-10 Thread Jay Kreps
What this means is that the CRC that was stored with the data no longer matches. There could be two reasons for this: 1. We have some rare bug in the CRC calculation or check logic or are otherwise corrupting data in flight 2. You have disk or network corruption which changed the bits. To diagnose

Jenkins build is back to normal : Kafka-trunk #390

2015-02-10 Thread Apache Jenkins Server
See

Re: Kafka New(Java) Producer Connection reset by peer error and LB

2015-02-10 Thread Jay Kreps
I don't think this is a bug. Currently we don't support timing out connections in the clients, which would be a good feature to add. As a result the connection remains until the LB kills it. When that happens you get a message logged that the connection was unexpectedly closed, which I think is wha

[jira] [Created] (KAFKA-1941) Timeout connections in the clients

2015-02-10 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-1941: Summary: Timeout connections in the clients Key: KAFKA-1941 URL: https://issues.apache.org/jira/browse/KAFKA-1941 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer

2015-02-10 Thread Jay Kreps
Yeah we could do that, I guess I just feel like it adds confusion because then you have to think about which timeout you want, when likely you don't want a timeout at all. I guess the pattern I was thinking of was fflush or the java equivalent, which don't have timeouts: http://docs.oracle.com/jav

Re: New consumer client

2015-02-10 Thread Steven Wu
Jay, we have observed CRC corruption too occasionally. I reported in an thread and asked how should we handle some error conditions from old high-level consumer. On Mon, Feb 9, 2015 at 11:36 PM, Bhavesh Mistry wrote: > Hi Jay, > > 1) Sorry to get back to you so late. It is CRC check error on an

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314962#comment-14314962 ] Tong Li commented on KAFKA-1926: [~harsha_ch], yes, new bee mistake. used the tool submitt

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: (was: KAFKA-1926-p1.patch) > Replace kafka.utils.Utils with o.a.k.common.utils.Utils > ---

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926.patch > Replace kafka.utils.Utils with o.a.k.common.utils.Utils > -

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314948#comment-14314948 ] Tong Li commented on KAFKA-1926: Created reviewboard https://reviews.apache.org/r/30845/di

Review Request 30845: Patch for KAFKA-1926

2015-02-10 Thread Tong Li
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30845/ --- Review request for kafka. Bugs: KAFKA-1926 https://issues.apache.org/jira/b

Review Request 30844: Patch for KAFKA-1926

2015-02-10 Thread Tong Li
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30844/ --- Review request for kafka. Bugs: KAFKA-1926 https://issues.apache.org/jira/b

[jira] [Updated] (KAFKA-1826) add command to delete all consumer group information for a topic in zookeeper

2015-02-10 Thread Onur Karaman (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-1826: Resolution: Done Status: Resolved (was: Patch Available) Implemented this as part of KAFKA-

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314920#comment-14314920 ] Sriharsha Chintalapani commented on KAFKA-1926: --- [~tongli] you might want t

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Attachment: KAFKA-1926-p1.patch This is the first attempt to fix the issue. Would like to get some reviews and

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

2015-02-10 Thread Tong Li (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tong Li updated KAFKA-1926: --- Reviewer: Neha Narkhede Labels: newbie patch (was: newbie) Affects Version/s: 0.8.

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

2015-02-10 Thread Joel Koshy
On Tue, Feb 10, 2015 at 12:13:46PM -0800, Neha Narkhede wrote: > I think all of us agree that we want to design MirrorMaker for 0 data loss. > With the absence of the data channel, 0 data loss will be much simpler to > implement. The data channel is irrelevant to the implementation of zero data

Re: Cannot stop Kafka server if zookeeper is shutdown first

2015-02-10 Thread Neha Narkhede
Since we have heard back from Stefan on this issue, we should just ask him to release another version of ZkClient with the fixes we want. It may be better to take the discussion of whether or not we should replace ZkClient with Curator to its own KIP thread. In general, there are several issues we

Re: [DISCUSS] KIPs

2015-02-10 Thread Neha Narkhede
Thanks Joel. The status table is useful. On Mon, Feb 9, 2015 at 8:25 PM, Joe Stein wrote: > I did https://cwiki.apache.org/confluence/display/KAFKA/drafts > > ~ Joestein > > On Mon, Feb 9, 2015 at 11:01 PM, Jay Kreps wrote: > > > Yeah no pressure. I think you added a holding area for incomplete

[jira] [Created] (KAFKA-1940) Initial checkout and build failing

2015-02-10 Thread Martin Lemanski (JIRA)
Martin Lemanski created KAFKA-1940: -- Summary: Initial checkout and build failing Key: KAFKA-1940 URL: https://issues.apache.org/jira/browse/KAFKA-1940 Project: Kafka Issue Type: Bug

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

2015-02-10 Thread Neha Narkhede
I think all of us agree that we want to design MirrorMaker for 0 data loss. With the absence of the data channel, 0 data loss will be much simpler to implement. My arguments for adding a message handler are that: > 1. It is more efficient to do something in common for all the clients in > pipeline

Re: Review Request 29647: Patch for KAFKA-1697

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/#review71842 --- clients/src/main/java/org/apache/kafka/common/errors/NotEnoughRepli

[jira] [Updated] (KAFKA-1476) Get a list of consumer groups

2015-02-10 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1476: - Resolution: Fixed Fix Version/s: (was: 0.9.0) 0.8.3 Assignee

Re: Review Request 29831: Patch for KAFKA-1476

2015-02-10 Thread Neha Narkhede
> On Feb. 2, 2015, 5:42 p.m., Neha Narkhede wrote: > > Onur, we should be able to check in after these review comments are > > addressed. Also, how would deleting offsets for a group work when the > > offset storage is Kafka? It's fine to not address it in this patch. Can you > > please create

Re: Kafka New(Java) Producer Connection reset by peer error and LB

2015-02-10 Thread Bhavesh Mistry
HI Ewen, The root of the problem is leak of TCP connection which idle for while. It is just a log message as you mentioned, but suppose you have 50 or more producer instances created by application and everyone of then will print above log that becomes little concern. We configured producer wit

Re: Review Request 30810: Patch for KAFKA-1937

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30810/#review71839 --- core/src/main/scala/kafka/tools/MirrorMaker.scala

Re: Review Request 30810: Patch for KAFKA-1937

2015-02-10 Thread Joel Koshy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30810/#review71837 --- core/src/main/scala/kafka/tools/MirrorMaker.scala

Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer

2015-02-10 Thread Joel Koshy
I think tryFlush with a timeout sounds good to me. This is really more for consistency than anything else. I cannot think of any standard blocking calls off the top of my head that don't have a timed variant. E.g., Thread.join, Object.wait, Future.get Either that, or they provide an entirely non-bl

Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer

2015-02-10 Thread Joel Koshy
> silly. But in the absense of flush there is no way to say that. As you say > you only may that penalty on one of the get() calls, but if the linger.ms > is high (say 60 seconds) that will be a huge penalty. That makes sense - thanks for clarifying. On Mon, Feb 09, 2015 at 08:11:46PM -0800, Jay

Re: [KIP-DISCUSSION] Mirror Maker Enhancement

2015-02-10 Thread Joel Koshy
I think the message handler adds little to no complexity to the mirror maker. Jay/Neha, the MM became scary due to the rearchitecture we did for 0.8 due to performance issues compared with 0.7 - we should remove the data channel if it can match the current throughput. I agree it is worth prototypin

[jira] [Commented] (KAFKA-1884) New Producer blocks forever for Invalid topic names

2015-02-10 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314518#comment-14314518 ] Manikumar Reddy commented on KAFKA-1884: looks like we are already returning error

Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer

2015-02-10 Thread Joe Stein
Jay, The .flush() call seems like it would be the best way if you wanted to-do a clean shutdown of the new producer? So, you could in your code "stop all incoming requests && producer.flush() && system.exit(value)" and know pretty much you won't drop anything on the floor. This can be done with

Re: [DISCUSS] KIP-8 Add a flush method to the new Java producer

2015-02-10 Thread Jay Kreps
Hey Bhavesh, If a broker is not available a new one should be elected to take over, so although the flush might take longer it should still be quick. Even if not this should result in an error not a hang. The cases you enumerated are all covered already--if the user wants to retry that is covered

[jira] [Updated] (KAFKA-1933) Fine-grained locking in log append

2015-02-10 Thread Maxim Ivanov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Ivanov updated KAFKA-1933: Fix Version/s: (was: 0.8.2.0) 0.8.3 > Fine-grained locking in log append > --

Re: Kafka New(Java) Producer Connection reset by peer error and LB

2015-02-10 Thread Ewen Cheslack-Postava
Bhavesh, I'm unclear what the impact is here. The line numbers don't match up exactly with trunk or 0.8.2.0, but it looks like this exception is just caught and logged. As far as I can tell the producer would continue to function normally. Does this have any impact on the producer or is the concer