Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-08-01 Thread Ismael Juma
Hi both, Thanks for the replies. A few points: 1. It's worth emphasising that reassigning partitions is a different process than what happens when a topic is created, so not sure trying to make it symmetric is beneficial. In addition to what was already discussed, one should also enable replicati

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

2017-08-01 Thread Ismael Juma
ter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: isma...@gmail.com on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Tuesday, Aug

Re: Command tools : from Scala to Java, from Zookeeper utils to Admin Client API

2017-08-01 Thread Ismael Juma
Hi Paolo, Another option is to write the new tool in Java without support for `--zookeeper` and include some logic in the shell script to pick the implementation based on the presence of `--bootstrap-server` or `--zookeeper`. This would mean that we can deprecate the Scala tool while still support

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-08-01 Thread Ismael Juma
Hi Tom, A high-level point for discussion before going into the details. The proposed protocol API `alterTopics` has 2 types of operations: 1. Operations that cause data movement (or deletion): increase/decrease of replication factor and partition reassignment. These are currently done by `kafka-

[jira] [Created] (KAFKA-5647) Use async ZookeeperClient for Admin operations

2017-07-26 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5647: -- Summary: Use async ZookeeperClient for Admin operations Key: KAFKA-5647 URL: https://issues.apache.org/jira/browse/KAFKA-5647 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-5646) Use async ZookeeperClient for Config and ISR management

2017-07-26 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5646: -- Summary: Use async ZookeeperClient for Config and ISR management Key: KAFKA-5646 URL: https://issues.apache.org/jira/browse/KAFKA-5646 Project: Kafka Issue Type

[jira] [Created] (KAFKA-5645) Use async ZookeeperClient in SimpleAclAuthorizer

2017-07-26 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5645: -- Summary: Use async ZookeeperClient in SimpleAclAuthorizer Key: KAFKA-5645 URL: https://issues.apache.org/jira/browse/KAFKA-5645 Project: Kafka Issue Type: Sub

[jira] [Resolved] (KAFKA-5328) consider switching json parser from scala to jackson

2017-07-26 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5328. Resolution: Duplicate Duplicate of KAFKA-1595. > consider switching json parser from scala

[jira] [Resolved] (KAFKA-3210) Using asynchronous calls through the raw ZK API in ZkUtils

2017-07-26 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3210. Resolution: Won't Fix We are following a slightly different approach, see KAFKA-5501. &g

[jira] [Reopened] (KAFKA-5501) use async zookeeper apis everywhere

2017-07-25 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5501: > use async zookeeper apis everywhere > --- > >

Re: [DISCUSS] 2017 October release planning and release version

2017-07-25 Thread Ismael Juma
ort older > releases > > > > longer and do bug fix release for older releases, too. > > > > > > > > Other projects (like Flink) do a fantastic job with this regard and > we > > > > should learn from them. > > > > > > > > -Matthias > > >

[jira] [Created] (KAFKA-5637) Document compatibility and release policies

2017-07-25 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5637: -- Summary: Document compatibility and release policies Key: KAFKA-5637 URL: https://issues.apache.org/jira/browse/KAFKA-5637 Project: Kafka Issue Type

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-25 Thread Ismael Juma
> My two cents: > > > > > > > > > > > > "Dead" and "Empty" states are transient: groups usually only > > leaves in > > > > > this > > > > > > state for a short while and then being deleted or transited to

Re: [DISCUSS] 2017 October release planning and release version

2017-07-25 Thread Ismael Juma
On Tue, Jul 18, 2017 at 4:04 PM, Guozhang Wang wrote: > I was actually thinking about using dot as well for the rc as well moving > forward, but I can be convinced if we have some reason to keep it as dash > as well. It seems reasonable to use a dash for the RC part as it's a qualifier that is

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-07-25 Thread Ismael Juma
dy exists: alterConfigs(). > > This API is better than having separate methods to set the number of > partitions/replicas and assigning them to brokers, since sometimes people > will want to set the assignment at the same time as changing the > partitions/replicas. > > An API like t

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-24 Thread Ismael Juma
> > "Dead" and "Empty" states are transient: groups usually only > leaves in > > > > this > > > > > state for a short while and then being deleted or transited to > other > > > > > states. > > > > > >

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-07-24 Thread Ismael Juma
Hi Tom, I think it makes sense to keep progress reporting simple in the initial version. As you say, time to completion is tricky to compute and it seems like it should be tackled as its own KIP. Regarding waiting for reassignment completion, it's easy enough for people to do that via a script, s

[jira] [Created] (KAFKA-5631) Use Jackson for serialising to JSON

2017-07-24 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5631: -- Summary: Use Jackson for serialising to JSON Key: KAFKA-5631 URL: https://issues.apache.org/jira/browse/KAFKA-5631 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] 2017 October release planning and release version

2017-07-21 Thread Ismael Juma
ciding what's the version number of the next major release does not > necessarily mean we need now to change any of these or to set the hard > rules along with it; I'd like to keep them as two separate discussions as > they seem semi-orthogonal to me. > > > Guozhang >

Re: [DISCUSS] 2017 October release planning and release version

2017-07-21 Thread Ismael Juma
d for. Current clients support 0.10.x and newer brokers. 7. How long are message formats supported for. We never discussed this. I think 5 years would probably be the minimum. Ismael P.S. I'll file a JIRA to capture this information. On Wed, Jul 19, 2017 at 10:12 AM, Ismael Juma wrote:

Re: [DISCUSS] KIP-180: Add a broker metric specifying the number of consumer group rebalances in progress

2017-07-21 Thread Ismael Juma
Thanks for the KIP, Colin. This will definitely be useful. One question: would it be useful to have a metric for for the number of groups in each possible state? The KIP suggests "PreparingRebalance" and "AwaitingSync". That leaves "Stable", "Dead" and "Empty". Are those not useful? Ismael On Thu

Re: Consumer throughput drop

2017-07-21 Thread Ismael Juma
Thanks for reporting the results. Maybe you could submit a PR that updates the ops section? https://github.com/apache/kafka/blob/trunk/docs/ops.html Ismael On Fri, Jul 21, 2017 at 2:49 PM, Ovidiu-Cristian MARCU < ovidiu-cristian.ma...@inria.fr> wrote: > After some tuning, I got better results.

Re: [DISCUSS] 2017 October release planning and release version

2017-07-20 Thread Ismael Juma
gt; > It should be also clear how long is each version supported - e.g. if > > > > minor.patch had meaning that there are no backward incompatible > > changes, > > > > it's OK to file a bug only for current major.minor.patch; previous > > major > >

Re: Reason why TopicFilter and related classes are deprecated

2017-07-20 Thread Ismael Juma
Hi Paolo, That's right, we deprecated those classes as they are mainly used by the old consumer. Sounds good to avoid using them in the TopicCommand written in Java. Ismael On Thu, Jul 20, 2017 at 3:33 AM, Paolo Patierno wrote: > I found that the deprecation was added speaking about the KIP-10

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

2017-07-20 Thread Ismael Juma
x27;t check the tables. So this is now KIP-179. The old > link > > will point you to the right place. > > > > On 19 July 2017 at 15:55, Ismael Juma wrote: > > > >> One more thing, it looks like there is already a KIP-178: > >> > >> KIP-178: Size-

Re: [VOTE] KIP-164 Add unavailablePartitionCount and per-partition Unavailable metrics

2017-07-20 Thread Ismael Juma
Thanks for the KIP, +1 (binding). On Thu, Jun 1, 2017 at 9:44 AM, Dong Lin wrote: > Hi all, > > Can you please vote for KIP-164? The KIP can be found at > https://cwiki.apache.org/confluence/display/KAFKA/KIP-164-+Add+ > UnderMinIsrPartitionCount+and+per-partition+UnderMinIsr+metrics > . > > Tha

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-07-20 Thread Ismael Juma
> > > Per 1. I suggested exposing the constant since we are doing so for > consumer > > and producer configs prefix as well (CONSUMER_PREFIX, etc). > > > > Guozhang > > > > On Wed, Jul 19, 2017 at 6:01 AM, Ismael Juma wrote: > > > > > Thank

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

2017-07-20 Thread Ismael Juma
ourselves compatible changes > given the output format that we have chosen for a tool. > > -Jason > > On Wed, Jul 19, 2017 at 7:54 AM, Ismael Juma wrote: > > > I think this is a good chance although it's unfortunate that it's likely > to > > break code that is

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

2017-07-19 Thread Ismael Juma
; On 19 July 2017 at 14:59, Ismael Juma wrote: > > > Hi Tom, > > > > It's fine for the tool not to have this functionality from the start. > > However, since we're adding new Kafka protocol APIs, we need to consider > > some of these details to ensure

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

2017-07-19 Thread Ismael Juma
I think this is a good chance although it's unfortunate that it's likely to break code that is parsing the output of the performance tool. Would it make sense to only enable this if an option is provided? Ismael On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson wrote: > +Users > > Thanks for the

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

2017-07-19 Thread Ismael Juma
Hi Tom, It's fine for the tool not to have this functionality from the start. However, since we're adding new Kafka protocol APIs, we need to consider some of these details to ensure we're building towards the end state, if that makes sense. Protocol APIs are used by multiple clients, so there is

Re: [VOTE] KIP-173: Add prefix to StreamsConfig to enable setting default internal topic configs

2017-07-19 Thread Ismael Juma
Thanks for the KIP, Damian. +1 (binding). A couple of minor comments: 1. Do we need to expose the TOPIC_PREFIX constant? 2. The vote thread ended up inside the discuss thread in Gmail. It may be worth sending another email to make it clear that the vote is ongoing. You can link back to this thread

Re: [DISCUSS] KIP-178: Change ReassignPartitionsCommand to use AdminClient

2017-07-19 Thread Ismael Juma
Hi Tom, Thanks for the KIP. Two quick comments: 1. Have you considered how progress would be reported? Partition reassignment can take a long time and it would be good to have a mechanism for progress reporting. 2. Removals can only happen in major releases. In your example, the removal could onl

Re: [VOTE] KIP-168: Add TotalTopicCount metric per cluster

2017-07-19 Thread Ismael Juma
Thanks for the KIP, +1 (binding) from me. Ismael On Wed, Jun 21, 2017 at 1:17 PM, Abhishek Mendhekar < abhishek.mendhe...@gmail.com> wrote: > Hi Kafka Dev, > > I did like to start the voting on - > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 168%3A+Add+TotalTopicCount+metric+per+clu

Re: [DISCUSS] 2017 October release planning and release version

2017-07-19 Thread Ismael Juma
Hi Stevo, Thanks for your feedback. We should definitely do a better job of documenting things. We basically follow semantic versioning, but it's currently a bit confusing because: 1. There are 4 segments in the version. The "0." part should be ignored when deciding what is major, minor and patch

Re: [DISCUSS] 2017 October release planning and release version

2017-07-18 Thread Ismael Juma
of all of them. > > Guozhang > > On Tue, Jul 18, 2017 at 3:49 PM, Gwen Shapira wrote: > > > Also fine with the change in general. > > > > As you mentioned, 1.x indicates mature APIs, compatibility and stability. > > Are we going to remove the @unstable annot

Re: [DISCUSS] 2017 October release planning and release version

2017-07-18 Thread Ismael Juma
Hi Guozhang, Thanks for volunteering to be the release manager for the next release! I am +1 on naming the next release 1.0.0. As you said, Kafka is mature enough and this will make it easier for others to understand our versioning policy. A couple of minor questions inline. On Tue, Jul 18, 201

Re: [DISCUSS] KIP-176: Remove deprecated new-consumer option in ConsoleConsumer tool

2017-07-18 Thread Ismael Juma
On Mon, Jul 17, 2017 at 10:11 AM, Matthias J. Sax wrote: > I thinks, for backward compatibility, this check should be there. The > only change should be, that if `--bootstrap-servers` is used, one can > omit `--new-consumer` (and if used, as warning is printed). > You can omit "--new-consumer" s

[jira] [Resolved] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

2017-07-18 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4669. Resolution: Fixed Assignee: Rajini Sivaram Closing this for now. If it reoccurs, let's r

[jira] [Created] (KAFKA-5606) Review consumer's RequestFuture usage pattern

2017-07-18 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5606: -- Summary: Review consumer's RequestFuture usage pattern Key: KAFKA-5606 URL: https://issues.apache.org/jira/browse/KAFKA-5606 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-5587) Processor got uncaught exception: NullPointerException

2017-07-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5587. Resolution: Fixed Fix Version/s: 0.11.1.0 0.11.0.1 > Processor

Re: Comments on JIRAs

2017-07-14 Thread Ismael Juma
Hi Tom, All active committers should be subscribed to jira@ so I don't think this changes much from that perspective. If you want to get wider input on a particular topic, then you should send an email to the mailing list, however. This was the case before the creation of jira@ as mailing list fil

[jira] [Resolved] (KAFKA-5584) Incorrect log size for topics larger than 2 GB

2017-07-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5584. Resolution: Fixed > Incorrect log size for topics larger than 2

Re: [ANNOUNCE] New Kafka PMC member Jason Gustafson

2017-07-12 Thread Ismael Juma
Congratulations Jason! Ismael On Tue, Jul 11, 2017 at 10:32 PM, Guozhang Wang wrote: > Hi Everyone, > > Jason Gustafson has been very active in contributing to the Kafka community > since he became a Kafka committer last September and has done lots of > significant work including the most recen

Re: Release date for 0.11.0.1

2017-07-07 Thread Ismael Juma
Hi Nishkam, Not yet. Is there a particular fix that you are interested in? Ismael On Fri, Jul 7, 2017 at 6:53 PM, Nishkam Ravi wrote: > Hi All, > Do we have an expected release date for 0.11.0.1 ? > > Thanks, > Nishkam >

[jira] [Created] (KAFKA-5570) Join request's timeout should be slightly higher than the rebalance timeout

2017-07-07 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5570: -- Summary: Join request's timeout should be slightly higher than the rebalance timeout Key: KAFKA-5570 URL: https://issues.apache.org/jira/browse/KAFKA-5570 Project:

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

2017-07-06 Thread Ismael Juma
Thanks everyone! Ismael On Wed, Jul 5, 2017 at 9:55 PM, Jun Rao wrote: > Hi, Everyone, > > Ismael Juma has been active in the Kafka community since he became > a Kafka committer about a year ago. I am glad to announce that Ismael is > now a member of Kafka PMC. > > Co

[jira] [Resolved] (KAFKA-5544) The LastStableOffsetLag metric should be removed when partition is deleted

2017-06-29 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5544. Resolution: Fixed > The LastStableOffsetLag metric should be removed when partition is dele

[ANNOUNCE] Apache Kafka 0.11.0.0 Released

2017-06-28 Thread Ismael Juma
lack-Postava, gosubpl, Grant Henke, Guozhang Wang, Gwen Shapira, Hamidreza Afzali, Hao Chen, hejiefang, Hojjat Jafarpour, huxi, Ismael Juma, Ivan A. Melnikov, Jaikiran Pai, James Cheng, James Chien, Jan Lukavsky, Jason Gustafson, Jean-Philippe Daigle, Jeff Chao, Jeff Widman, Jeyhun Karimov, Jiangjie Qin,

Re: 0.11.0.0 RC2 Vote

2017-06-28 Thread Ismael Juma
> +1 > > > > Verified 0110 web docs and java docs; verified quick start with 2.11 / > > 2.12 scala versions. > > > > One minor observation: on the web docs we show the cmd for 2.11 scala > > version; we'd better make it templated with the version num

Re: [VOTE] 0.11.0.0 RC2

2017-06-27 Thread Ismael Juma
h the release process and the release announcement will follow in the next few days. Ismael P.S. Guozhang's vote was in a separate thread due to some delivery issues when replying to this thread. On Fri, Jun 23, 2017 at 2:16 AM, Ismael Juma wrote: > Hello Kafka users, develope

[jira] [Resolved] (KAFKA-4815) Idempotent/transactional Producer (KIP-98)

2017-06-27 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4815. Resolution: Fixed > Idempotent/transactional Producer (KIP

[jira] [Created] (KAFKA-5527) Idempotent/transactional Producer part 2 (KIP-98)

2017-06-27 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5527: -- Summary: Idempotent/transactional Producer part 2 (KIP-98) Key: KAFKA-5527 URL: https://issues.apache.org/jira/browse/KAFKA-5527 Project: Kafka Issue Type: New

Re: [VOTE] 0.11.0.0 RC2

2017-06-27 Thread Ismael Juma
ead, Test > > worker-SendThread(127.0.0.1:59720), /127.0.0.1:5494 > > 2 to /127.0.0.1:54926 workers Thread 3, > > ZkClient-EventThread-22-127.0.0.1:54976, ProcessThread(sid:0 > > cport:54976):, Test worker-SendThread(127.0.0.1:54976), Fin > > alizer, metrics-meter-tick-threa

Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Ismael Juma
gCommandTest > testQuotaDescribeEntities PASSED > > kafka.admin.AdminTest > testGetBrokerMetadatas PASSED > > kafka.admin.ConfigCommandTest > shouldParseArgumentsForClientsEntityType > PASSED > > kafka.admin.AclCommandTest > testAclCli PASSED > > kafka.admin.ReassignParti

Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Ismael Juma
:54976):, Test worker-SendThread(127.0.0.1:54976), Fin > alizer, metrics-meter-tick-thread-1) > > I tested on a VM and a physical machine, and both give me a lot of errors > like this. > > Thanks. > --Vahid > > > > > From: Ismael Juma > To: Vahid S

Re: Open PRs

2017-06-26 Thread Ismael Juma
Hey Tom, Thanks for the PRs you have submitted. As you suggested, most of us are busy testing the upcoming release. It's a major release with many exciting features and we want to make sure it's a high quality release. I understand that it can be frustrating to wait. At the same time, there are n

Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Ismael Juma
checkstyle] [ERROR] C:\Users\User\Downloads\kafka- > 0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\ > requests\AbstractResponse.java:26:1: Class Fan-Out Complexity is 42 (max > allowed is 40). [ClassFanOutComplexity] > :clients:checkstyleMain FAILED > > FAILURE: Build

[jira] [Reopened] (KAFKA-5506) bin/kafka-consumer-groups.sh failing to query offsets

2017-06-23 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5506: Assignee: Ismael Juma Nice catch, there are 2 bugs here. See the PR for the details. > bin/ka

Re: [VOTE] 0.11.0.0 RC2

2017-06-22 Thread Ismael Juma
: https://github.com/apache/kafka/commit/186e3d5efc79ed803f0915d472ace77cbec88694 Full diff: https://github.com/apache/kafka/compare/5b351216621f52a471c21826d0dec3ce3187e697...0.11.0.0-rc2 Ismael On Fri, Jun 23, 2017 at 2:16 AM, Ismael Juma wrote: > Hello Kafka users, developers and cli

[VOTE] 0.11.0.0 RC2

2017-06-22 Thread Ismael Juma
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 0.11.0.0. This is a major version release of Apache Kafka. It includes 32 new KIPs. See the release notes and release plan ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+

Re: CreateTopicResponse Error Code 41

2017-06-22 Thread Ismael Juma
Yes, it needs to be sent to the Controller. Metadata response has the controller id. Ismael On Fri, Jun 23, 2017 at 12:45 AM, Vineet Goel wrote: > Hi, > > I get an Error (Code 41) when sending a CreateTopicRequest to any of the > brokers except 1. Why might this be? Does this request need to be

Re: [VOTE] 0.11.0.0 RC1

2017-06-22 Thread Ismael Juma
> > [ant:checkstyle] [ERROR] >> > > C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\ >> > > java\org\apache\kafka\common\protocol\Errors.java:89:1: >> > > Class Fan-Out Complexity is 60 (max allowed is 40). >> > > [ClassFa

Re: [VOTE] 0.11.0.0 RC0

2017-06-22 Thread Ismael Juma
rformance tests run across cloud instances where there might > be contention and variability, we typically do a multiple runs on different > clusters with each setup before reporting results. > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, Jun 22, 2017 at 4:08 PM, Ismae

Re: [VOTE] 0.11.0.0 RC0

2017-06-22 Thread Ismael Juma
k though. > > Given that the blockers fixed between RC0 and RC1 haven't changed much in > the areas we tested, I think the positive results here still apply. > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, Jun 8, 2017 at 2:55 PM, Ismael Juma wrote

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

2017-06-22 Thread Ismael Juma
Thanks for the KIP, Edoardo. A few comments: 1. Have you considered extending RequestMetadata with the additional information you need? We could add Cluster to it, which has topic assignment information, for example. This way, there would be no need for a V2 interface. 2. Something else that coul

[jira] [Resolved] (KAFKA-4059) Documentation still refers to AsyncProducer and SyncProducer

2017-06-22 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4059. Resolution: Fixed Fix Version/s: 0.11.0.0 > Documentation still refers to AsyncProducer

Re: [DISCUSS] Streams DSL/StateStore Refactoring

2017-06-22 Thread Ismael Juma
Thanks Damian. I think both options have pros and cons. And both are better than overload abuse. The fluent API approach reads better, no mention of builder or build anywhere. The main downside is that the method signatures are a little less clear. By reading the method signature, one doesn't nece

[jira] [Created] (KAFKA-5486) org.apache.kafka logging should go to server.log

2017-06-21 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5486: -- Summary: org.apache.kafka logging should go to server.log Key: KAFKA-5486 URL: https://issues.apache.org/jira/browse/KAFKA-5486 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-2465) Need to document replica.fetcher.backoff.ms

2017-06-20 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2465. Resolution: Fixed > Need to document replica.fetcher.backoff

Re: Pull-Requests not worked on

2017-06-19 Thread Ismael Juma
don't remember the details). Ismael On Tue, Jun 20, 2017 at 12:52 AM, Jeff Widman wrote: > On Thu, Jun 8, 2017 at 5:00 PM, Ismael Juma wrote: > > > One of the reasons why some PRs remain open even if they should be > > closed is that we have no way to close PRs oursel

[jira] [Created] (KAFKA-5470) Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class

2017-06-19 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5470: -- Summary: Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class Key: KAFKA-5470 URL: https://issues.apache.org/jira/browse/KAFKA-5470

Re: [VOTE] 0.11.0.0 RC0

2017-06-19 Thread Ismael Juma
ested the exactly once features, because various core > committers said that they didn't expect this feature to be perfect in this > release. We expect to test this this week though. > > Given that the blockers fixed between RC0 and RC1 haven't changed much in > the areas we t

Re: [VOTE] 0.11.0.0 RC1

2017-06-19 Thread Ismael Juma
> > > > > > +1 (non-binding) > > > > > > Passes librdkafka integration tests (v0.9.5 and master) > > > > > > > > > 2017-06-19 0:32 GMT+02:00 Ismael Juma : > > > > > >> Hello Kafka users, developers and client-develo

[VOTE] 0.11.0.0 RC1

2017-06-18 Thread Ismael Juma
Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 0.11.0.0. This is a major version release of Apache Kafka. It includes 32 new KIPs. See the release notes and release plan (https://cwiki.apache.org/conf luence/display/KAFKA/Release+Plan

[jira] [Resolved] (KAFKA-5336) ListGroup requires Describe on Cluster, but the command-line AclCommand tool does not allow this to be set

2017-06-18 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5336. Resolution: Fixed Even though this was fixed by KAFKA-5292, it's a worth tracking th

[jira] [Reopened] (KAFKA-5336) ListGroup requires Describe on Cluster, but the command-line AclCommand tool does not allow this to be set

2017-06-18 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-5336: > ListGroup requires Describe on Cluster, but the command-line AclCommand tool > does not allo

[jira] [Resolved] (KAFKA-5463) Controller incorrectly logs rack information when new brokers are added

2017-06-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5463. Resolution: Fixed Fix Version/s: 0.11.0.0 Issue resolved by pull request 3358 [https

[jira] [Resolved] (KAFKA-5457) MemoryRecordsBuilder.hasRoomfor doesn't take into account the headers while computing available space

2017-06-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5457. Resolution: Fixed Issue resolved by pull request 3355 [https://github.com/apache/kafka/pull/3355

[jira] [Resolved] (KAFKA-5449) Flaky test TransactionsTest.testReadCommittedConsumerShouldNotSeeUndecidedData

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5449. Resolution: Fixed Fix Version/s: 0.11.0.0 Issue resolved by pull request 3348 [https

[jira] [Updated] (KAFKA-5020) Update protocol documentation to mention message format v2

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5020: --- Fix Version/s: (was: 0.11.0.0) 0.11.0.1 > Update protocol documentation

[jira] [Updated] (KAFKA-5032) Think through implications of max.message.size affecting record batches in message format V2

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5032: --- Fix Version/s: (was: 0.11.0.0) 0.11.0.1 > Think through implications

[jira] [Updated] (KAFKA-5417) Clients get inconsistent connection states when SASL/SSL connection is marked CONECTED and DISCONNECTED at the same time

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5417: --- Fix Version/s: (was: 0.11.0.0) 0.11.0.1 > Clients get inconsistent connect

[jira] [Updated] (KAFKA-4879) KafkaConsumer.position may hang forever when deleting a topic

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4879: --- Fix Version/s: 0.11.1.0 > KafkaConsumer.position may hang forever when deleting a to

[jira] [Updated] (KAFKA-5272) Improve validation for Alter Configs (KIP-133)

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5272: --- Summary: Improve validation for Alter Configs (KIP-133) (was: Improve validation for Describe/Alter

[jira] [Assigned] (KAFKA-4680) min.insync.replicas can be set higher than replication factor

2017-06-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-4680: -- Assignee: Ismael Juma > min.insync.replicas can be set higher than replication fac

[jira] [Commented] (KAFKA-5453) Controller may miss requests sent to the broker when zk session timeout happens.

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050067#comment-16050067 ] Ismael Juma commented on KAFKA-5453: Is this a regression in 0.11.0.0? > Con

[jira] [Updated] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5275: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Commented] (KAFKA-5445) Document exceptions thrown by AdminClient methods

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049406#comment-16049406 ] Ismael Juma commented on KAFKA-5445: Sounds good. > Document exceptions th

[jira] [Updated] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5443: --- Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

[jira] [Updated] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5443: --- Status: Patch Available (was: Open) > Consumer should use last offset from batch to set next fe

[jira] [Updated] (KAFKA-5272) Improve validation for Describe/Alter Configs (KIP-133)

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5272: --- Fix Version/s: (was: 0.11.0.0) 0.11.0.1 > Improve validation for Descr

[jira] [Updated] (KAFKA-5276) Support derived and prefixed configs in DescribeConfigs (KIP-133)

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5276: --- Fix Version/s: (was: 0.11.0.0) 0.11.0.1 > Support derived and prefixed conf

[jira] [Updated] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5275: --- Status: Patch Available (was: Open) > Review and potentially tweak AdminClient API for the init

[jira] [Updated] (KAFKA-5431) LogCleaner stopped due to org.apache.kafka.common.errors.CorruptRecordException

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5431: --- Priority: Major (was: Critical) > LogCleaner stopped due

[jira] [Updated] (KAFKA-5413) Log cleaner fails due to large offset in segment file

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5413: --- Priority: Critical (was: Major) > Log cleaner fails due to large offset in segment f

[jira] [Updated] (KAFKA-5447) AdminClient metrics

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5447: --- Labels: needs-kip (was: ) > AdminClient metrics > --- > >

[jira] [Commented] (KAFKA-5447) AdminClient metrics

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16049216#comment-16049216 ] Ismael Juma commented on KAFKA-5447: cc [~cmccabe] > AdminClient

[jira] [Created] (KAFKA-5447) AdminClient metrics

2017-06-14 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-5447: -- Summary: AdminClient metrics Key: KAFKA-5447 URL: https://issues.apache.org/jira/browse/KAFKA-5447 Project: Kafka Issue Type: Improvement Reporter

[jira] [Updated] (KAFKA-5431) LogCleaner stopped due to org.apache.kafka.common.errors.CorruptRecordException

2017-06-14 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-5431: --- Priority: Critical (was: Major) > LogCleaner stopped due

<    9   10   11   12   13   14   15   16   17   18   >