[jira] [Resolved] (KAFKA-5792) Transient failure in KafkaAdminClientTest.testHandleTimeout

2017-09-09 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5792. Resolution: Fixed > Transient failure in KafkaAdminClientTest.testHandleTime

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

2017-09-08 Thread Ismael Juma
f there's no meaningful replication > factor I > > can't apply the policy. > > > > Am I missing something? > > > > Thanks, > > > > Tom > > > > On 8 September 2017 at 13:50, Ted Yu wrote: > > > >> I think increasePartiti

[jira] [Resolved] (KAFKA-5656) Support bulk attributes request on KafkaMbean where some attributes do not exist

2017-09-08 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5656. Resolution: Fixed Fix Version/s: 1.0.0 > Support bulk attributes request on KafkaMbean wh

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

2017-09-08 Thread Ismael Juma
Hmm, maybe it should be createPartitions for symmetry with createTopics? Ismael On Fri, Sep 8, 2017 at 12:33 PM, Tom Bentley wrote: > Hi Paolo, > > Thanks for commenting. > > The main reason I suggested `NumPartitionsIncrease` rather than just > `NumPartitions` was in case we ever implement a d

Re: [VOTE] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-09-08 Thread Ismael Juma
Thanks for the KIP. +1 (binding) from me. Just a minor suggestion, I would mention the following under "Public Interfaces": Default value of delivery.timeout.ms = 120 seconds Default value of retries will be changed to MAX_INT request.timeout.ms – current meaning, but messages are not expired afte

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-09-08 Thread Ismael Juma
Thanks! On Fri, Sep 8, 2017 at 6:09 AM, Sumant Tambe wrote: > Just did :) > > On 7 September 2017 at 17:52, Ismael Juma wrote: > > > Can we please start the vote on this KIP? The KIP must be accepted by > next > > Wednesday in order to make the cut for 1.0.0.

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

2017-09-08 Thread Ismael Juma
rote: > Hi Ted and Ismael, > > Thanks for the comments. > > Ted, I've fixed the KIP for your comments. > > Ismael, see responses inline: > > On 8 September 2017 at 02:00, Ismael Juma wrote: > > > Thanks Tom. Thanks for the KIP. A few comments: > > >

[jira] [Created] (KAFKA-5859) Avoid retaining AbstractRequest in RequestChannel.Request

2017-09-07 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5859: -- Summary: Avoid retaining AbstractRequest in RequestChannel.Request Key: KAFKA-5859 URL: https://issues.apache.org/jira/browse/KAFKA-5859 Project: Kafka Issue

[jira] [Resolved] (KAFKA-5858) consumer.poll() shouldn't throw exception due to deserialization error

2017-09-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5858. Resolution: Duplicate Duplicate of KAFKA-5470, > consumer.poll() shouldn't throw exceptio

Re: [VOTE] KIP-192 : Provide cleaner semantics when idempotence is enabled

2017-09-07 Thread Ismael Juma
Thanks for the KIP. +1 (binding) from me. A few minor comments: 1. We should add a note to the backwards compatibility section explaining the impact of throwing DuplicateSequenceException (a new exception) from `send`. As I understand it, it's not an issue, but good to include it in the KIP. 2. F

Re: [DISCUSS] KIP-195: AdminClient.increasePartitions

2017-09-07 Thread Ismael Juma
Thanks Tom. Thanks for the KIP. A few comments: 1. Does the `PartitionCount` class still make sense given that the method can only increase the number of partitions now? 2,. In `NewTopic`, we have `numPartitions`. Should we keep using that variant and call the method `increaseNumPartitions`? 3. Si

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-09-07 Thread Ismael Juma
delivery.timeout.ms if retries have been exhausted. > > Thanks, > Apurva > > > On Thu, Sep 7, 2017 at 6:07 AM, Ismael Juma wrote: > > > Good question regarding retries Sumant. A few comments: > > > > 1. Defaulting to MAX_INT makes sense in the context of >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-09-07 Thread Ismael Juma
e: I wonder if it would be a good idea for the KIP template to have a > "Depends on" field so people can more easily keep track of how multiple > in-flight KIPs depend on one another? > > Cheers, > > Tom > > On 7 September 2017 at 16:42, Ismael Juma wrote: > >

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

2017-09-07 Thread Ismael Juma
name doesn't reflect that > its heritage in the original CreateTopicPolicy. > > Cheers, > > Tom > > On 5 September 2017 at 18:48, Edoardo Comar wrote: > > > Hi Ismael, > > I was on leave for a long while. I will update the KIP. > > > > Edo > > >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-09-07 Thread Ismael Juma
Hi Tom, What do you think of moving `increasePartitionsCount` (or `increaseNumPartitions`) to a separate KIP? That is simple enough that we could potentially include it in 1.0.0. I'd be happy to review it. ReassignPartitions is more complex and we can probably aim to include that in the January re

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-09-07 Thread Ismael Juma
t? > > > > So do we ignore retries config or throw a ConfigException if weirdness > like > > above is detected? > > > > -Sumant > > > > > > On 5 September 2017 at 17:34, Ismael Juma wrote: > > > > > Thanks for updating the KIP, Su

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

2017-09-07 Thread Ismael Juma
t > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > >

Re: KAFKA-5723: Refactor BrokerApiVersionsCommand to use the new AdminClient

2017-09-07 Thread Ismael Juma
I would also add that it would be easier to review if there were smaller PRs than one big PR. So, it may be worth thinking how progress could be made more incrementally. Ismael On Thu, Sep 7, 2017 at 11:17 AM, Tom Bentley wrote: > I can't speak for the committers, but there's nothing to stop yo

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-09-06 Thread Ismael Juma
;>> > > >> > > >>>> Thanks, > > >> > > >>>> Bill > > >> > > >>>> > > >> > > >>>> On Thu, Aug 31, 2017 at 10:31 AM, Vahid S Hashemian < > > >> &

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

2017-09-06 Thread Ismael Juma
Thanks for the KIP. +1 (binding). Please make it clear in the KIP that removal will happen in 2.0.0. Ismael On Tue, Aug 8, 2017 at 11:53 AM, Paolo Patierno wrote: > Hi devs, > > > I didn't see any more comments about this KIP. The JIRAs related to the > first step (so making --new-consumer as d

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-09-05 Thread Ismael Juma
t; > > > > > > > > > > > You are right. Currently we are reset the PID and > > > > resend > > > > > > the > > > > > > > > > > batches > > > > > > > > > > > to > > >

Re: [VOTE] KIP-188 - Add new metrics to support health checks

2017-09-05 Thread Ismael Juma
The other > > aspect > > > is that for produce requests, we can up convert as well, so it seemed > > > better to keep it generic. > > > > > > Yeah, so I thought maybe we could bypass the question and drop `Message` > > from the names if they were already

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

2017-09-05 Thread Ismael Juma
; when+idempotence+is+enabled > > On Tue, Sep 5, 2017 at 2:03 PM, Ismael Juma wrote: > > > Hi Ted, > > > > The current proposal has 3 options: requested, required, off ("safe" was > in > > an earlier proposal). I think these convey the meaning more clearly

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

2017-09-05 Thread Ismael Juma
t > idempotence > would be safe. > > However, since it really means best effort, the 'safety' is debatable. > > Why not just call the new mode besteffort ? > > Cheers > > On Tue, Sep 5, 2017 at 11:24 AM, Ismael Juma wrote: > > > If we add the message

Re: [DISCUSS] KIP-192 - Provide cleaner semantics when idempotence is enabled

2017-09-05 Thread Ismael Juma
If we add the message format version (a topic config) in the response of TopicMetadata, we should consider adding the max message bytes as well. That would allow us to later improve the implementation of KIP-126 to split the batch _before_ sending. Ismael On Tue, Sep 5, 2017 at 7:17 PM, Jason Gus

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

2017-09-05 Thread Ismael Juma
Thanks for the KIP, +1 (binding). Ismael On Wed, Aug 30, 2017 at 4:51 PM, Jason Gustafson wrote: > I'd like to open the vote for KIP-189: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL. > Thanks to everyone who help

Re: [VOTE] KIP-188 - Add new metrics to support health checks

2017-09-05 Thread Ismael Juma
cide if it should be 'Record' instead. > > On Tue, Sep 5, 2017 at 10:20 AM, Ismael Juma wrote: > > > Thanks Rajini. > > > > 1. I meant a topic metric, but we could have one for fetch and one for > > produce differentiated by the additional tag. The advanta

Re: [VOTE] KIP-188 - Add new metrics to support health checks

2017-09-05 Thread Ismael Juma
ce > allocated for conversions. For other requests, this metric will not be > created (unless we find a request where the size is large and this > information is useful). > > Thank you, > > Rajini > > > On Tue, Sep 5, 2017 at 4:55 PM, Ismael Juma wrote:

Re: [VOTE] KIP-188 - Add new metrics to support health checks

2017-09-05 Thread Ismael Juma
Thanks Rajini, +1 (binding) from me. Just a few minor comments: 1. FetchDownConversionsPerSec should probably be MessageConversionsPerSec with a request tag for consistency with MessageConversionsTimeMs. The text in that paragraph should also be updated to talk about message conversions instead of

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

2017-09-05 Thread Ismael Juma
t > a > > string to the validate method so that we have options if we need to > extend > > it. > > 3. Agree, we should have the DeletePolicy define its RequestMetadata > class, too > > > > 4. Do we want to enhance the AlterConfigs policy as well? > > 4. I don't

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

2017-09-04 Thread Ismael Juma
. Very little hit at all. > > -Todd > > On Sep 4, 2017 12:45 AM, "Ismael Juma" wrote: > > > By the way, in-kernel TLS has now landed in the Linux kernel: > > > > https://github.com/torvalds/linux/blob/master/ > > Documentation/networking/tls.t

[jira] [Resolved] (KAFKA-5831) MemoryRecordsBuilder uses small compression buffer leading to snappy low compression ratio

2017-09-04 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5831. Resolution: Duplicate Thanks for filing the report. This is a duplicate of KAFKA-5236

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-09-04 Thread Ismael Juma
Hi Tom, You can update the KIP for minor things like that. Worth updating the thread if it's something that is done during the PR review. With regards to exceptions, yes, that's definitely desired. I filed a JIRA a while back for this: https://issues.apache.org/jira/browse/KAFKA-5445 Ideally, n

Re: Plans to improve SSL performance in Kafka, for 0.10.x?

2017-09-04 Thread Ismael Juma
;t see how this is going to help us at the app > layer. > > -Todd > > On Tuesday, September 6, 2016, Ismael Juma wrote: > > > Hi Todd, > > > > Thanks for sharing your experience enabling TLS in your clusters. Very > > helpful. One comment below. > > &

[jira] [Resolved] (KAFKA-5785) Always close connection if KafkaChannel.setSend throws exception

2017-09-01 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5785. Resolution: Duplicate KAFKA-5607 includes a fix for this. > Always close connection

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-08-31 Thread Ismael Juma
Thanks for the KIP, +1 (binding) from me. Ismael On Tue, Aug 29, 2017 at 4:56 PM, Tom Bentley wrote: > Hi all, > > I would like to start the vote on KIP-183 which will provide an AdminClient > interface for electing the preferred replica, and refactor the > kafka-preferred-replica-election.sh t

Re: [VOTE] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-08-31 Thread Ismael Juma
Thanks for the KIP, +1 (binding). Ismael On 31 Aug 2017 8:38 am, "Attila Kreiner" wrote: Hi All, Thx for the comments, I pretty much see a consensus here. So I'd like to start the vote for: https://cwiki.apache.org/confluence/display/KAFKA/KIP-191%3A+KafkaConsumer. subscribe%28%29+overload+tha

[jira] [Reopened] (KAFKA-5763) Refactor NetworkClient to use LogContext

2017-08-30 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5763: NetworkClient is a different class and it doesn't use LogContext yet. > Refactor NetworkClien

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-08-30 Thread Ismael Juma
; Should I rename it in the KIP now, even though I initiated a VOTE thread > yesterday? > > Cheers, > > Tom > > On 30 August 2017 at 16:01, Ismael Juma wrote: > > > Hi Tom, > > > > Thanks for the KIP, it's a useful one. I find the proposed method name >

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-30 Thread Ismael Juma
t; per-connection > > >> informational messages at info-level, but if you think it is useful, > we > > >> could do this in a separate JIRA. Let me know what you think. > > >> > > >> On Tue, Aug 29, 2017 at 1:09 PM, Roger Hoover > > > >

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-08-30 Thread Ismael Juma
Hi Tom, Thanks for the KIP, it's a useful one. I find the proposed method name `electPreferredReplicaLeader` a little hard to read. It seems that a small change would make it clearer: `electPreferredReplicaAsLeader`. The next point is that this is a batch API, so it should ideally be plural like t

[jira] [Resolved] (KAFKA-5762) Refactor AdminClient to use LogContext

2017-08-30 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5762. Resolution: Fixed > Refactor AdminClient to use LogCont

Re: Permission to create KIP for discussions and the max.in.flight.requests.per.connection config doc

2017-08-29 Thread Ismael Juma
Hi Bhaskar, Comment inline. On Sun, Aug 27, 2017 at 12:37 PM, Bhaskar Gollapudi < bhaskargollap...@gmail.com> wrote: > max.in.flight.requests.per.connection The maximum number of unacknowledged > requests the client will send on a single connection before blocking. Note > that if this setting is

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-29 Thread Ismael Juma
Thanks for the proposals. I think they make sense and I also agree with Jason's suggestions. Also, it would be good to include the updated ProduceRequest/Response schema in the KIP. Ismael On Tue, Aug 22, 2017 at 11:42 PM, Jason Gustafson wrote: > Thanks Apurva, > > On compatibility: I think th

Re: [DISCUSS] KIP-191: KafkaConsumer.subscribe() overload that takes just Pattern

2017-08-29 Thread Ismael Juma
Sounds good to me too. Since this is a non controversial change, I suggest starting the vote in 1-2 days if no-one else comments. Ismael On Thu, Aug 24, 2017 at 7:32 PM, Jason Gustafson wrote: > Seems reasonable. I don't recall any specific reason for not providing this > method initially. > >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-29 Thread Ismael Juma
is high, we can add the code to expire them later. Is > that > > ok? > > > > Regards, > > > > Rajini > > > > > > > > On Fri, Aug 18, 2017 at 9:41 AM, Ismael Juma wrote: > > > > > Can we quantify the cost of having these metrics a

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-29 Thread Ismael Juma
Thanks for the KIP, +1 (binding) from me. Ismael On Thu, Aug 24, 2017 at 6:29 PM, Rajini Sivaram wrote: > Hi all, > > I would like to start vote on KIP-152 to improve diagnostics of > authentication failures and to update clients to treat authentication > failures as fatal exceptions rather tha

Re: [VOTE] KIP-187 - Add cumulative count metric for all Kafka rate metrics

2017-08-29 Thread Ismael Juma
Thanks for the KIP, +1 (binding) from me. Ismael On Thu, Aug 24, 2017 at 6:48 PM, Rajini Sivaram wrote: > Hi all, > > I would like to start the vote on KIP-187 that adds a cumulative count > metric associated with each Kafka rate metric to improve downstream > processing of rate metrics. Detail

[jira] [Resolved] (KAFKA-4380) Update the usage description of clean shutdown file.

2017-08-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4380. Resolution: Fixed Fix Version/s: 1.0.0 > Update the usage description of clean shutdown f

[jira] [Created] (KAFKA-5790) SocketServer.processNewResponses should not skip a response if exception is thrown

2017-08-25 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5790: -- Summary: SocketServer.processNewResponses should not skip a response if exception is thrown Key: KAFKA-5790 URL: https://issues.apache.org/jira/browse/KAFKA-5790 Project

[jira] [Created] (KAFKA-5785) Always close connection if KafkaChannel.setSend throws exception

2017-08-25 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5785: -- Summary: Always close connection if KafkaChannel.setSend throws exception Key: KAFKA-5785 URL: https://issues.apache.org/jira/browse/KAFKA-5785 Project: Kafka

[jira] [Resolved] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

2017-08-25 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5595. Resolution: Fixed I'm going to close this. If we see the issue again, we can reopen it. >

[jira] [Reopened] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

2017-08-25 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5595: > Illegal state in SocketServer; attempt to send with another send in progr

[jira] [Resolved] (KAFKA-5595) Illegal state in SocketServer; attempt to send with another send in progress

2017-08-25 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5595. Resolution: Fixed Assignee: Rajini Sivaram Fix Version/s: 1.0.0 > Illegal state

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

2017-08-24 Thread Ismael Juma
Congratulations Becket! On 24 Aug 2017 6:20 am, "Joel Koshy" wrote: Hi everyone, Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has contributed significantly to several major patches, reviews and discussions since. I am glad to announce that Becket is now a member of the A

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-08-23 Thread Ismael Juma
xtracting an expired batch from a > request also introduces some complexity. Again, personally I think it is > fine to expire a little bit late. So maybe we don't need to expire a batch > that is already in flight. In the worst case we will expire it with delay > of request.tim

[jira] [Resolved] (KAFKA-4856) Calling KafkaProducer.close() from multiple threads may cause spurious error

2017-08-22 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4856. Resolution: Fixed Fix Version/s: 1.0.0 0.11.0.1 > Call

[jira] [Created] (KAFKA-5763) Refactor NetworkClient to use LogContext

2017-08-22 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5763: -- Summary: Refactor NetworkClient to use LogContext Key: KAFKA-5763 URL: https://issues.apache.org/jira/browse/KAFKA-5763 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-5762) Refactor AdminClient to use LogContext

2017-08-22 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5762: -- Summary: Refactor AdminClient to use LogContext Key: KAFKA-5762 URL: https://issues.apache.org/jira/browse/KAFKA-5762 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

2017-08-22 Thread Ismael Juma
Hi all, The discussion has been going on for a while, would it help to have a call to discuss this? I'd like to start a vote soonish so that we can include this in 1.0.0. I personally prefer max.message.delivery.wait.ms. It seems like Jun, Apurva and Jason also prefer that. Sumant, it seems like y

[jira] [Resolved] (KAFKA-5753) ShellTest.testRunProgramWithErrorReturn fails on macOS

2017-08-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5753. Resolution: Fixed > ShellTest.testRunProgramWithErrorReturn fails on ma

[jira] [Resolved] (KAFKA-5744) ShellTest: add tests for attempting to run nonexistent program, error return

2017-08-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5744. Resolution: Fixed > ShellTest: add tests for attempting to run nonexistent program, error ret

[jira] [Reopened] (KAFKA-5753) ShellTest.testRunProgramWithErrorReturn fails locally

2017-08-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5753: Assignee: Ismael Juma > ShellTest.testRunProgramWithErrorReturn fails loca

Jenkins updates: Java 9 and less noise in PR builds

2017-08-20 Thread Ismael Juma
Hi all, A couple of Jenkins updates: 1. We now have Java 9 builds for PRs and trunk. The Java 9 build doesn't currently run the tests or findBugs because EasyMock, PowerMock and findBugs don't support Java 9 yet. The Java 9 builds will ensure that we don't reintroduce code that doesn't compile wi

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

2017-08-19 Thread Ismael Juma
Thanks for the KIP Jason. It seems reasonable and cleans up some inconsistencies in that area. It would be great to get some feedback from Mayuresh and others who worked on KIP-111. Ismael On Thu, Aug 17, 2017 at 1:21 AM, Jason Gustafson wrote: > Hi All, > > I've added a new KIP to improve and

Re: [DISCUSS] KIP-152 - Improve diagnostics for SASL authentication failures

2017-08-18 Thread Ismael Juma
that this can be implemented for 1.0.0. > > > Thanks, > > Rajini > > > On Thu, May 4, 2017 at 10:22 PM, Ismael Juma wrote: > > > Hi Rajini, > > > > I think we were talking about slightly different things. I was just > > referring to the fact that t

[jira] [Resolved] (KAFKA-5745) Partition.makeLeader() should convert HW to OffsetMetadata before becoming the leader

2017-08-18 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5745. Resolution: Fixed Fix Version/s: 1.0.0 0.11.0.1 > Partition.makeLea

[jira] [Resolved] (KAFKA-5744) ShellTest: add tests for attempting to run nonexistent program, error return

2017-08-18 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5744. Resolution: Fixed Fix Version/s: 1.0.0 > ShellTest: add tests for attempting to

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-18 Thread Ismael Juma
17 at 7:42 AM, Manikumar > wrote: > > > I agree it will be good if we can add "commit id/version" as an > > attribute value. > > It will be easy to parse. But as of now, KafkaMetric supports only > > numerical values. > > > > On Fri, Aug 18, 2017

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-18 Thread Ismael Juma
Hi Apurva and Jason, A few thoughts: 1. We should only delay it if there's a concrete benefit (e.g. we agree that we need to improve OutOfOrderSequence and we can't do it in time). I doubt that we will get much additional testing from users if we keep it off by default for another release cycle (

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-17 Thread Ismael Juma
t). I also added the client-id tag which is used in > all metrics from clients. > > Regards, > > Rajini > > > On Thu, Aug 17, 2017 at 10:55 AM, Ismael Juma wrote: > > > Thanks for the KIP, Rajini. I think this is helpful too. A few minor > > comments. > >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

2017-08-17 Thread Ismael Juma
Thanks for the KIP, Rajini. I think this is helpful too. A few minor comments. 1. About the number of metrics and expiration, if we dynamically register metrics for the error codes, the number is likely to be much lower than 30*30, probably less than 100. If we were using Kafka Metrics for this, w

[jira] [Resolved] (KAFKA-2206) Add AlterConfig and DescribeConfig requests to Kafka

2017-08-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2206. Resolution: Duplicate Thanks [~omkreddy], closing as duplicate. > Add AlterConfig

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

2017-08-16 Thread Ismael Juma
: paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: isma...@gmail.com on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Wednesday, August 2, 2017 8:51 AM >

[jira] [Resolved] (KAFKA-5611) One or more consumers in a consumer-group stop consuming after rebalancing

2017-08-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5611. Resolution: Fixed I'm going to mark this as closed. If [~pskianis] or someone else can stil

Re: [DISCUSS] 0.11.0.1 bug fix release

2017-08-16 Thread Ismael Juma
Thanks for driving the release Damian. Sounds good to me. Ismael On Wed, Aug 16, 2017 at 1:29 PM, Damian Guy wrote: > Hi, > > It seems like it must be time for 0.11.0.1 bug fix release! > > Since the 0.11.0.0 release we've fixed 30 JIRAs that > are targeted for 0.11.0.1: > > https://issues.apac

Re: Kafka 0.11.0.0

2017-08-14 Thread Ismael Juma
Hi Srikanth, 0.11.0.0 is looking pretty good so far. One concern is: https://issues.apache.org/jira/browse/KAFKA-5600 There is no concrete plan for 0.11.0.1, but I'd expect a RC within a few weeks (2 to 4, probably). Ismael On Mon, Aug 14, 2017 at 5:15 AM, Srikanth Sampath wrote: > Hi, > We

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-12 Thread Ismael Juma
quence, but that should be rare and we probably should > > not optimize for that. > > > The problem is described in https://issues.apache.org/ > jira/browse/KAFKA-5494 > . > > Specifically, when you have max.in.flight=N, you need to retain the > offset/sequence/timest

Re: [VOTE] KIP-180: Adding a new metric for brokers specifying the number of consumer group rebalances in progress

2017-08-12 Thread Ismael Juma
Thanks for the KIP, +1 (binding). Seems like the KIP title has to be updated still. Ismael On Fri, Aug 11, 2017 at 11:00 PM, Colin McCabe wrote: > Hi all, > > I think it's a good time to vote on KIP-180. It adds a helpful new > metric that shows consumer group states. > > The full proposal, an

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-12 Thread Ismael Juma
Hi all, I will send a more detail email later, some quick comments: 1. It's unlikely that defaults will suit everyone. I think the question is: what is the most likely configuration for a typical Kafka user _today_? Kafka's usage is growing well beyond its original use cases and correctness is of

[jira] [Resolved] (KAFKA-5077) Make server start script work against Java 9

2017-08-09 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5077. Resolution: Fixed > Make server start script work against Jav

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-08-09 Thread Ismael Juma
> > > > > >> > specifically, this this the workflow from the > coordinator's > > > point > > > > > of > > > > > > > >> > view: > > > > > > > >> > > > > > > > > >> > 1. de

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-09 Thread Ismael Juma
Hi Jason, Thanks for the correction. See inline. On Wed, Aug 9, 2017 at 5:13 PM, Jason Gustafson wrote: > Minor correction: the OutOfOrderSequenceException is not fatal for the > idempotent producer and it is not necessarily tied to the acks setting > (though it is more likely to be thrown with

[jira] [Resolved] (KAFKA-2360) The kafka-consumer-perf-test.sh script help information print useless parameters.

2017-08-09 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2360. Resolution: Fixed Fix Version/s: 1.0.0 > The kafka-consumer-perf-test.sh script h

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-09 Thread Ismael Juma
Thanks for the KIP, Apurva. In general, I think it's a good idea to strengthen the guarantees we provide by default. And people who are willing to trade correctness for performance can then change the configs to suit them. I will comment on the KIP specifics in more detail later, but one additional

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-08-09 Thread Ismael Juma
On Wed, Aug 9, 2017 at 11:40 AM, Tom Bentley wrote: > > There are responses with detailed error messages as well as the codes, > CreateTopicsResponse, {Describe|Alter}ConfigsResponse, and the responses > for managing ACLs for instance. To be honest, I assumed including a message > was the norm. In

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

2017-08-09 Thread Ismael Juma
Hi Pranav, In the JIRA and the previous mailing list thread, some of us wondered if the benefit is worth the transition pain. To quote Jason: "The "log cleaner" naming may not be ideal, but it is not incorrect and some of the terminology used elsewhere makes more sense given this name (e.g. clean

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

2017-08-09 Thread Ismael Juma
Thanks for the KIP, +1 from me. Ismael On Wed, Aug 9, 2017 at 1:24 AM, Ewen Cheslack-Postava wrote: > Hi all, > > I posted a simple new KIP for a problem we see with a lot of users: > KIP-186: Increase offsets retention default to 7 days > > https://cwiki.apache.org/confluence/display/KAFKA/KIP

[jira] [Resolved] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

2017-08-08 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5710. Resolution: Duplicate Duplicate of KAFKA-5658. > KafkaAdminClient should remove inflight c

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-07 Thread Ismael Juma
Hi Dong, Thanks for the explanation. Comments inline. On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin wrote: > 1. Yes it has been considered. Here are the reasons why we don't do it > through controller. > > - There can be use-cases where we only want to rebalance the load of log > directories on a gi

[jira] [Resolved] (KAFKA-5602) Add --custom-ducktape flag to ducker-ak

2017-08-05 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5602. Resolution: Fixed Fix Version/s: 1.0.0 > Add --custom-ducktape flag to ducker

Re: [VOTE] KIP-177: Consumer perf tool should count rebalance time

2017-08-04 Thread Ismael Juma
Thanks for the KIP, +1 (binding) Ismael On Fri, Aug 4, 2017 at 3:13 AM, Hu Xi wrote: > Hi all, > > > I'd like to kick off the vote for KIP-177: https://cwiki.apache.org/ > confluence/display/ARIES/KIP-177%3A+Consumer+perf+tool+ > should+count+rebalance+time. > > > A PR for this KIP can be found

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-04 Thread Ismael Juma
Thanks Dong. I have a few initial questions, sorry if I it has been discussed and I missed it. 1. The KIP suggests that the reassignment tool is responsible for sending the ChangeReplicaDirRequests to the relevant brokers. I had imagined that this would be done by the Controller, like the rest of

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-08-04 Thread Ismael Juma
sponses for > whoever > > > >> > requested join > > > >> > \|/ > > > >> > | > > > >> > | waiting for the sync group request from the > > > leader > > > >> >

Re: [DISCUSS] KIP-178: Size-based log directory selection strategy

2017-08-04 Thread Ismael Juma
Hi, Thanks for the KIP. As Dong said, this is not straightforward. One option would be to keep the existing system, which is very simple and rely on KIP-113 to keep the data balanced across disks. If we don't think that's good enough, we should explain why it's not in the KIP. Thanks, Ismael On

[jira] [Resolved] (KAFKA-2507) Replace ControlledShutdown{Request,Response} with org.apache.kafka.common.requests equivalent

2017-08-03 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2507. Resolution: Fixed Assignee: Jason Gustafson (was: Grant Henke) Fix Version/s

[jira] [Resolved] (KAFKA-2959) Remove temporary mapping to deserialize functions in RequestChannel

2017-08-03 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2959. Resolution: Fixed Assignee: Jason Gustafson Fix Version/s: 1.0.0 > Remove tempor

[jira] [Resolved] (KAFKA-5663) LogDirFailureTest system test fails

2017-08-02 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5663. Resolution: Fixed Fix Version/s: 1.0.0 > LogDirFailureTest system test fa

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

2017-08-02 Thread Ismael Juma
Hi Arseniy, At some point (before I joined), it was decided that clients would be written in Java for a few reasons: 1. It's easier to maintain binary compatibility 2. Calling Java from Scala is easier than calling Scala from Java (without extra effort) 3. A single artifact instead of one artifac

Re: Handler Threads in SocketServer

2017-08-01 Thread Ismael Juma
Hi Haseeb, Yes, KafkaRequestHandler is a handler thread indeed. The `Handler` singleton object you refer to is unused. I submitted a PR to remove it. Thanks, Ismael On Tue, Aug 1, 2017 at 10:10 PM, Javed, Haseeb wrote: > Hello, > > > I have recently started looking into the Kafka Server code t

<    8   9   10   11   12   13   14   15   16   17   >