Re: Do the Jackson security vulnerabilities affect Kafka at all?

2018-02-20 Thread Ismael Juma
Hi Jeff, Have you checked trunk and 1.1? They should be using the latest version. Ismael On Tue, Feb 20, 2018 at 10:38 PM, Jeff Widman wrote: > The Jackson JSON parser library had a couple of CVE's announced: > 1. CVE-2017-7525 > 2. CVE 2017-15095 > > Here's a skimmable summary: > https://adam

[jira] [Resolved] (KAFKA-6430) Improve Kafka GZip compression performance

2018-02-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6430. Resolution: Fixed Fix Version/s: 1.1.0 > Improve Kafka GZip compression performa

Re: 1.1 release progress

2018-02-15 Thread Ismael Juma
Sounds good. Thanks Becket. Ismael On 15 Feb 2018 5:30 pm, "Becket Qin" wrote: > Hi Ismael, > > Yes, I am working on the fix. Will submit patch today. > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Feb 15, 2018 at 2:53 PM, Ismael Juma wrote: > >

Re: 1.1 release progress

2018-02-15 Thread Ismael Juma
Hi Becket, Thanks for filing that. Are you working on a fix? Ismael On Thu, Feb 15, 2018 at 2:51 PM, Becket Qin wrote: > Hi Damian, > > I just created another ticket KAFKA-6568, which I believe should also be a > blocker unless people disagree. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed,

Re: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-31 Thread Ismael Juma
u need > any additional information. I'm new to this project so hopefully the > format is what you were looking for. > > - Kyle > > -Original Message- > From: Ismael Juma [mailto:isma...@gmail.com] > Sent: Tuesday, January 30, 2018 9:01 PM > To: dev > Subject: Re:

Re: Excessive Memory Usage with Compression enabled and possible resolutions

2018-01-30 Thread Ismael Juma
Thanks for the report. I haven't looked at the code, but it seems like we would want to do both 1 and 2. Can you please file a JIRA with 1.1.0 as the target version? Ismael On 30 Jan 2018 5:46 pm, "Kyle Tinker" wrote: > I'm using Kafka 1.0.0 and the Java producer. > > I've noticed high memory u

[jira] [Resolved] (KAFKA-6464) Base64URL encoding under JRE 1.7 is broken due to incorrect padding assumption

2018-01-30 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6464. Resolution: Fixed Reviewer: Rajini Sivaram (was: Ismael Juma) > Base64URL encoding under

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

2018-01-24 Thread Ismael Juma
gt; > > On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote: > > > Colin, > > > > > > You get a cumulative count for rates since we added > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 187+-+Add+cumulative+count+metric+for+all+Kafka+rate

Re: [VOTE] KIP-86: Configurable SASL callback handlers

2018-01-24 Thread Ismael Juma
Thanks for the KIP, Rajini. This is a useful improvement, so +1 (binding) from me. I really don't like how the Java Security classes work, so I would have preferred to avoid emulating them, but the KIP is consistent with previous related KIPs and that's the direction we chose previously. Also, I t

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-23 Thread Ismael Juma
1,k2=v2* > > If anyone has any concerns about these options being added to > kafka-configs.sh, please let me know. Otherwise, I will update KIP-226 and > add the options to one of the KIP-226 PRs. > > Thank you, > > Rajini > > On Wed, Jan 10, 2018 at 5:14 AM, Ismael Juma

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

2018-01-23 Thread Ismael Juma
Colin, You get a cumulative count for rates since we added https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics Ismael On Tue, Jan 23, 2018 at 4:21 PM, Colin McCabe wrote: > On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote: > > Hi, Col

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

2018-01-23 Thread Ismael Juma
Thanks for the KIP, +1 (binding). On Tue, Dec 19, 2017 at 7:28 PM, Colin McCabe wrote: > Hi all, > > I'd like to start the vote on KIP-227: Incremental Fetch Requests. > > The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 227%3A+Introduce+Incremental+FetchRequests+to+Incr

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

2018-01-17 Thread Ismael Juma
Congratulations Rajini! On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Since then, she remained active in the community and contributed major patches, reviews and KIP discussions. I am glad to announce that

Re: [ANNOUNCE] New committer: Matthias J. Sax

2018-01-12 Thread Ismael Juma
Congratulations Matthias! On 12 Jan 2018 10:59 pm, "Guozhang Wang" wrote: > Hello everyone, > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our > newest Kafka committer. > > Matthias has made tremendous contributions to Kafka Streams API since early > 2016. His footprint ha

Re: [DISCUSS] February Release Plan

2018-01-12 Thread Ismael Juma
Thanks for volunteering Damian! On Fri, Jan 12, 2018 at 11:04 AM, Damian Guy wrote: > Hi all, > > I would like to volunteer to be the release manager > for our next time-based feature release (v1.1.0). See > https://cwiki.apache.org/ > confluence/display/KAFKA/Time+Based+Release+Plan if you miss

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-10 Thread Ismael Juma
Thanks Rajini. Sounds good. Ismael On Wed, Jan 10, 2018 at 11:41 AM, Rajini Sivaram wrote: > Hi Ismael, > > I have updated the KIP to use AES-256 if available and AES-128 otherwise > for password encryption. Looking at GCM, it looks like GCM is typically > used with a variable initialization ve

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-09 Thread Ismael Juma
Hi Rajini, Quick question (sorry if this was already discussed). How were the following chosen? Name: password.encoder.keyfactory.algorithm Type: String Default: PBKDF2WithHmacSHA512 if available, otherwise PBKDF2WithHmacSHA1 (e.g. Java7) Name: password.encoder.cipher.algorithm Type: String De

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

2018-01-08 Thread Ismael Juma
On Fri, Jan 5, 2018 at 7:15 PM, Jason Gustafson wrote: > > class OffsetAndMetadata { > long offset; > byte[] offsetMetadata; > String metadata; > } > Admittedly, the naming is a bit annoying, but we can probably come up with > something better. Internally the byte array would have a versio

Re: [DISCUSS] KIP-237: More Controller Health Metrics

2018-01-04 Thread Ismael Juma
the name of the Java class, e.g. the > TotalQueueSize in ControllerChannelManager. > > But I am open to change the type to ControllerStats if we decide that all > future metrics in Controller will have type ControllerStats. Do you think > this would be better? > > Thanks, > Dong &

[jira] [Resolved] (KAFKA-6307) mBeanName should be removed before returning from JmxReporter#removeAttribute()

2018-01-01 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6307. Resolution: Fixed Fix Version/s: 1.1.0 > mBeanName should be removed before returning f

[jira] [Resolved] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.

2017-12-22 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2729. Resolution: Fixed Assignee: Onur Karaman Fix Version/s: 1.1.0 We believe this is

[jira] [Resolved] (KAFKA-6320) move ZK metrics in KafkaHealthCheck to ZookeeperClient

2017-12-22 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6320. Resolution: Fixed > move ZK metrics in KafkaHealthCheck to ZookeeperCli

[jira] [Resolved] (KAFKA-3496) Add reconnect attemps policies for client

2017-12-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3496. Resolution: Won't Fix We decided to add configs for enabling exponential backoff instead.

[jira] [Resolved] (KAFKA-5895) Gradle 3.0+ is needed on the build

2017-12-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5895. Resolution: Fixed Fix Version/s: 1.1.0 > Gradle 3.0+ is needed on the bu

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-21 Thread Ismael Juma
that: https://github.com/apache/kafka/pull/4352 Ismael On Fri, Dec 22, 2017 at 1:02 AM, Ismael Juma wrote: > It's done! Committers, please set up your account: > > https://gitbox.apache.org/setup/ > > On Wed, Dec 20, 2017 at 12:30 AM, Ismael Juma wrote: > >> Forgot t

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-21 Thread Ismael Juma
It's done! Committers, please set up your account: https://gitbox.apache.org/setup/ On Wed, Dec 20, 2017 at 12:30 AM, Ismael Juma wrote: > Forgot the link to the relevant Infra JIRA: https://issues.apache. > org/jira/browse/INFRA-15676 > > On Tue, Dec 19, 2017 at 11:59 PM, I

[jira] [Resolved] (KAFKA-6331) Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpointkafka.api.AdminClientIntegrationTest.testAlterReplicaLogDirs

2017-12-20 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6331. Resolution: Fixed Fix Version/s: 1.1.0 > Transient failure

[jira] [Created] (KAFKA-6390) Update ZooKeeper to 3.4.11 and other minor updates

2017-12-20 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6390: -- Summary: Update ZooKeeper to 3.4.11 and other minor updates Key: KAFKA-6390 URL: https://issues.apache.org/jira/browse/KAFKA-6390 Project: Kafka Issue Type: Bug

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-19 Thread Ismael Juma
Forgot the link to the relevant Infra JIRA: https://issues.apache.org/jira/browse/INFRA-15676 On Tue, Dec 19, 2017 at 11:59 PM, Ismael Juma wrote: > GitBox migration will happen today. Committers, please make sure to > associate your github ID with your apache.org account via id.apac

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-19 Thread Ismael Juma
GitBox migration will happen today. Committers, please make sure to associate your github ID with your apache.org account via id.apache.org, and make sure to enable 2 factor authentication in GitHub. Ismael On Fri, Dec 15, 2017 at 3:40 PM, Ismael Juma wrote: > Thanks to everyone who voted

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

2017-12-19 Thread Ismael Juma
Hi Steven, As a general rule, we don't freeze KIPs after the vote passes. It's reasonably common for things to come up during code review, for example. If we think of improvements, we shouldn't refrain from doing them because of of the vote. If we do minor changes after the KIP passes, we usually

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

2017-12-19 Thread Ismael Juma
t; > > > > > > > > now. > > > > > > > > > > > > > > > > > > > > > > > > For the purposes of KIP-218, how about adding a new > > > > interface > > > > > > > > > > > &

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-15 Thread Ismael Juma
, Dec 12, 2017 at 9:49 PM, Rajini Sivaram < > rajinisiva...@gmail.com > > > > > > wrote: > > > > > > > +1 > > > > > > > > Thanks, Ismael! > > > > > > > > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy > &g

[jira] [Resolved] (KAFKA-5631) Use Jackson for serialising to JSON

2017-12-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5631. Resolution: Fixed > Use Jackson for serialising to J

[VOTE] Allowing write access to GitHub repositories (aka GitBox)

2017-12-12 Thread Ismael Juma
Hi all, The Apache Infra team has started a new project earlier this year called GitBox that supports two-way synchronization between GitHub and git-wip-us.apache.org and, most importantly, provides GitHub write access to committers. GitBox is not generally available yet, but individual projects c

Re: [DISCUSS] Allowing write access to GitHub repositories

2017-12-12 Thread Ismael Juma
; > Thank you for looking into this. Sounds like a good improvement. > > > > Regards, > > > > Rajini > > > > On Mon, Dec 11, 2017 at 6:35 AM, Guozhang Wang > wrote: > > > > > Sounds good to me. > > > > > > On Sun, Dec 10, 2017

[jira] [Resolved] (KAFKA-6319) kafka-acls regression for comma characters (and maybe other characters as well)

2017-12-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6319. Resolution: Fixed > kafka-acls regression for comma characters (and maybe other characters

Re: [DISCUSS] KIP-237: More Controller Health Metrics

2017-12-12 Thread Ismael Juma
Thanks for the KIP, Dong. The general idea is good. In fact, two of the three metrics had been listed under future work for KIP-143: "KAFKA-5028 introduced a queue for Controller events. It would be useful to have a gauge for the queue size and a histogram for how long an event waits in the queue

[jira] [Resolved] (KAFKA-6194) Server crash while deleting segments

2017-12-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6194. Resolution: Fixed Assignee: Ismael Juma Fix Version/s: 1.1.0 I believe this was

[jira] [Resolved] (KAFKA-6075) Kafka cannot recover after an unclean shutdown on Windows

2017-12-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6075. Resolution: Fixed Assignee: Ismael Juma Fix Version/s: 1.1.0 I believe this was

[jira] [Resolved] (KAFKA-6322) Error deleting log for topic, all log dirs failed.

2017-12-12 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6322. Resolution: Fixed Assignee: Ismael Juma I believe this was fixed by KAFKA-6324, please reopen

Re: [DISCUSS] Allowing write access to GitHub repositories

2017-12-10 Thread Ismael Juma
ing, but we could also consider allow both > upstream and downstream cherry-picking) than completely discard it. > > > Guozhang > > On Sat, Dec 9, 2017 at 11:39 PM, Ismael Juma wrote: > > > One thing I forgot to mention, many projects have requested and been > using &g

Re: [DISCUSS] Allowing write access to GitHub repositories

2017-12-09 Thread Ismael Juma
%20DESC%2C%20created%20ASC On Sun, Dec 10, 2017 at 9:35 AM, Ismael Juma wrote: > Hi all, > > The Apache Infra team has started a new project earlier this year called > GitBox that supports two-way synchronization between GitHub and > git-wip-us.apache.org and, most importantly, provid

[DISCUSS] Allowing write access to GitHub repositories

2017-12-09 Thread Ismael Juma
Hi all, The Apache Infra team has started a new project earlier this year called GitBox that supports two-way synchronization between GitHub and git-wip-us.apache.org and, most importantly, provides GitHub write access to committers. GitBox is not generally available yet, but individual projects c

[jira] [Created] (KAFKA-6332) Kafka system tests should use nc instead of log grep to detect start-up

2017-12-08 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6332: -- Summary: Kafka system tests should use nc instead of log grep to detect start-up Key: KAFKA-6332 URL: https://issues.apache.org/jira/browse/KAFKA-6332 Project: Kafka

[jira] [Created] (KAFKA-6330) KafkaZkClient request queue time metric

2017-12-08 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6330: -- Summary: KafkaZkClient request queue time metric Key: KAFKA-6330 URL: https://issues.apache.org/jira/browse/KAFKA-6330 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-12-08 Thread Ismael Juma
One correction below. On Fri, Dec 8, 2017 at 11:16 AM, Jan Filipiak wrote: > We only check max.message.bytes to late to guard against consumer stalling. > we dont have a notion of max.networkpacket.size before we allocate the > bytebuffer to read it into. We do: socket.request.max.bytes. Isma

[jira] [Created] (KAFKA-6324) Change LogSegment.delete to deleteIfExists and harden log recovery

2017-12-07 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6324: -- Summary: Change LogSegment.delete to deleteIfExists and harden log recovery Key: KAFKA-6324 URL: https://issues.apache.org/jira/browse/KAFKA-6324 Project: Kafka

[jira] [Resolved] (KAFKA-6313) Kafka Core should have explicit SLF4J API dependency

2017-12-07 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6313. Resolution: Fixed > Kafka Core should have explicit SLF4J API depende

[jira] [Resolved] (KAFKA-6065) Add ZooKeeperRequestLatencyMs to KafkaZkClient

2017-12-06 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6065. Resolution: Fixed > Add ZooKeeperRequestLatencyMs to KafkaZkCli

[jira] [Created] (KAFKA-6317) Maven artifact for kafka should not depend on log4j

2017-12-06 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6317: -- Summary: Maven artifact for kafka should not depend on log4j Key: KAFKA-6317 URL: https://issues.apache.org/jira/browse/KAFKA-6317 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-6174) Add methods in Options classes to keep binary compatibility with 0.11

2017-12-06 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6174. Resolution: Fixed Fix Version/s: 1.1.0 > Add methods in Options classes to keep bin

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

2017-12-05 Thread Ismael Juma
One comment below. On Mon, Dec 4, 2017 at 11:40 PM, Dong Lin wrote: > In my opinion this changes the semantics of ListGroupsResponse in a > counter-intuitive way. Usually we use the ACL to determine whether the > operation on the specified object can be performed or not. The response > should pr

[jira] [Resolved] (KAFKA-2188) JBOD Support

2017-12-04 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2188. Resolution: Duplicate KIP-112 and KIP-113 replaced this one. > JBOD Supp

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-11-23 Thread Ismael Juma
Hi James, There are 2 options being discussed. Option A is similar to the existing approach where the follower informs the leader of offsets it has seen by asking for the next ones. We just skip the partitions where the offset hasn't changed. Option B involves the leader keeping track of the off

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-11-23 Thread Ismael Juma
Hi Becket, Relying on the cluster metadata doesn't seem like it would work if there are multiple fetcher threads, right? It also doesn't work for the consumer case, which Jay suggested would be good to handle. Ismael On Thu, Nov 23, 2017 at 2:21 AM, Becket Qin wrote: > Thanks for the KIP, Coli

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-11-22 Thread Ismael Juma
Hi Jay, On Thu, Nov 23, 2017 at 12:01 AM, Jay Kreps wrote: > I was also thinking there could be mechanical improvements that would help > efficiency such as sharing topic name or TopicPartition objects to reduce > the footprint in a flyweight style. Coincidentally, I was thinking of this earli

[jira] [Created] (KAFKA-6261) Request logging throws exception if acks=0

2017-11-22 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6261: -- Summary: Request logging throws exception if acks=0 Key: KAFKA-6261 URL: https://issues.apache.org/jira/browse/KAFKA-6261 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-1044) change log4j to slf4j

2017-11-22 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-1044. Resolution: Fixed Fix Version/s: 1.1.0 > change log4j to sl

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-11-22 Thread Ismael Juma
Thanks for the KIP, Colin. As others have said, I think something along these lines is definitely useful, we just need to work out the various details. Jun and Jay, I think you are discussing 2 orthogonal things: 1. Should we have a separate IncrementalFetchRequest or do we reuse FetchRequest? Li

Re: Adding log4j-extras to Apache Kafka?

2017-11-21 Thread Ismael Juma
-api.jar to avoid having to migrate everything > at once, but still benefit from appenders already included with log4j2 > (e.g. RollingFileAppender is already built into log4j2 core) > > On Tue, Nov 21, 2017 at 4:16 PM Ismael Juma wrote: > > > Hi Colin, > > > > I thin

Re: Adding log4j-extras to Apache Kafka?

2017-11-21 Thread Ismael Juma
Hi Colin, I think it's reasonable to include log4j-extras with the broker in the next release (1.1.0). In the 2.0.0 timeframe, we may decide to move to log4j2 (or logback), but people can benefit from log4j-extras in the meantime. Ismael On Tue, Nov 21, 2017 at 11:50 PM, Colin McCabe wrote: >

[jira] [Resolved] (KAFKA-4871) Kafka doesn't respect TTL on Zookeeper hostname - crash if zookeeper IP changes

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4871. Resolution: Duplicate Duplicate of KAFKA-5473. > Kafka doesn't respect TTL on Zookeeper

[jira] [Resolved] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4041. Resolution: Fixed Marking as a duplicate of KAFKA-5473. In that JIRA, we will recreate the

[jira] [Reopened] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4041: > kafka unable to reconnect to zookeeper behind an

[jira] [Resolved] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4041. Resolution: Duplicate > kafka unable to reconnect to zookeeper behind an

[jira] [Resolved] (KAFKA-2193) Intermittent network + DNS issues can cause brokers to permanently drop out of a cluster

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2193. Resolution: Duplicate Duplicate of KAFKA-5473. > Intermittent network + DNS issues can ca

[jira] [Resolved] (KAFKA-2864) Bad zookeeper host causes broker to shutdown uncleanly and stall producers

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2864. Resolution: Duplicate Duplicate of KAFKA-5473. > Bad zookeeper host causes broker to shutd

[jira] [Resolved] (KAFKA-6247) Fix system test dependency issues

2017-11-21 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6247. Resolution: Fixed Assignee: Colin P. McCabe Fix Version/s: 1.1.0 > Fix system t

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-20 Thread Ismael Juma
I figured we'll need a clear > message around what the removal means and what users need to do. > > Gwen > > > On Mon, Nov 20, 2017 at 8:21 AM Ismael Juma wrote: > > > It's worth emphasizing that the impact to such users is independent of > > whether we remove

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-20 Thread Ismael Juma
ally > non-trivial). > > We can say "not our problem", but as we know, lack of good migration path > really slows down adoption (Python 3.0, for instance). > > I'd love to at least get a feel of how many in the community will be > impacted. > > Gwen > >

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-20 Thread Ismael Juma
short period of downtime while doing a migration > > probably have in-house experts on Kafka who are familiar with the issues > > and willing to figure out a solution. The rest of the world can generally > > handle a short maintenance window. > > >

[jira] [Resolved] (KAFKA-6223) Please delete old releases from mirroring system

2017-11-19 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6223. Resolution: Fixed > Please delete old releases from mirroring sys

Re: [ANNOUNCE] Apache Kafka 0.11.0.2 Released

2017-11-19 Thread Ismael Juma
Thanks for running the release, Rajini! And with a single RC. :) Ismael On Fri, Nov 17, 2017 at 7:15 PM, Rajini Sivaram wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka > 0.11.0.2. > > > This is a bug fix release and it includes fixes and improvements fr

[jira] [Resolved] (KAFKA-6103) one broker appear to dead lock after running serval hours with a fresh cluster

2017-11-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6103. Resolution: Duplicate > one broker appear to dead lock after running serval hours with a fr

[jira] [Resolved] (KAFKA-6046) DeleteRecordsRequest to a non-leader

2017-11-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6046. Resolution: Fixed > DeleteRecordsRequest to a non-lea

[jira] [Resolved] (KAFKA-679) Phabricator for code review

2017-11-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-679. --- Resolution: Won't Do Yes, this can be safely closed. > Phabricator for cod

[jira] [Resolved] (KAFKA-1408) Kafk broker can not stop itself normaly after problems with connection to ZK

2017-11-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-1408. Resolution: Duplicate Closing as duplicate of KAFKA-1317 based on other comments and the fact that

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

2017-11-16 Thread Ismael Juma
Thanks Sebb. For reference, Sebb filed KAFKA-6222 and KAFKA-6223, so let's keep the discussion in the relevant JIRAs. Ismael On Thu, Nov 16, 2017 at 1:49 PM, sebb wrote: > On 1 November 2017 at 15:38, Guozhang Wang wrote: > > The Apache Kafka community is pleased to announce the release for Ap

[jira] [Reopened] (KAFKA-1993) Enable topic deletion as default

2017-11-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-1993: > Enable topic deletion as default > > >

[jira] [Resolved] (KAFKA-1993) Enable topic deletion as default

2017-11-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-1993. Resolution: Duplicate > Enable topic deletion as defa

[jira] [Resolved] (KAFKA-4675) Subsequent CreateTopic command could be lost after a DeleteTopic command

2017-11-16 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4675. Resolution: Duplicate KAFKA-6098 is the same issue and has more information. > Subsequ

Re: [VOTE] 0.11.0.2 RC0

2017-11-15 Thread Ismael Juma
+1 (binding). Tested the quickstart with the source and binary (Scala 2.12) artifacts, ran the tests on the source artifact and verified some signatures and hashes on source and binary (Scala 2.12) artifacts. Thanks for managing this release Rajini! On Sat, Nov 11, 2017 at 12:37 AM, Rajini Sivara

[jira] [Resolved] (KAFKA-6210) IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version

2017-11-15 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6210. Resolution: Fixed > IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.vers

[jira] [Created] (KAFKA-6210) IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version

2017-11-15 Thread Ismael Juma (JIRA)
Ismael Juma created KAFKA-6210: -- Summary: IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version Key: KAFKA-6210 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

2017-11-13 Thread Ismael Juma
most of > > > them are> > Void or complex result represented by classes. > > > > > > In order to support future extensibility I like your idea. > > > > > > Let's see what's the others opinions otherwise I'll start to > > > implement

Re: kafka-pr-jdk9-scala2.12 keeps failing

2017-11-11 Thread Ismael Juma
2498/console > > > > Does anything get reverted? > > > > > > Guozhang > > > > > > On Tue, Nov 7, 2017 at 8:33 AM, Ted Yu wrote: > > > > > https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2470/ is green. > > > > >

Re: Experimenting with Kafka and OpenSSL

2017-11-11 Thread Ismael Juma
017/10/kafka-with-openssl.html -Jaikiran On 30/10/17 5:33 PM, Ismael Juma wrote: > If Java 9 is used by both clients and brokers, AES GCM is used by default. > I did a quick test a while back and there was a significant improvement: > > https://twitter.com/ijuma/status/905847523897724929 &g

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-10 Thread Ismael Juma
uld we resolve this ticket sooner than later to make clear about the > code > > deprecation and support duration when moving from 1.0.x to 2.0.x? > > > > > > Guozhang > > > > > > On Fri, Nov 10, 2017 at 3:44 AM, Ismael Juma wrote: > > > > >

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

2017-11-10 Thread Ismael Juma
xperience<http://paolopatierno.wordpress.com/> > > > > From: isma...@gmail.com on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Friday, November 10, 2017 12:33 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-204 : ad

Re: 0.11 git tags

2017-11-10 Thread Ismael Juma
They seem to be there: https://github.com/apache/kafka/tree/0.11.0.0 https://github.com/apache/kafka/tree/0.11.0.1 Ismael On Fri, Nov 10, 2017 at 12:41 PM, Tom Bentley wrote: > I just noticed that although every other release is tagged in git there are > no tags for 0.11.0.0 and 0.11.0.1. Did

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

2017-11-10 Thread Ismael Juma
Hi Paolo, The KIP looks good, I have a couple of comments: 1. partitionsAndOffsets could perhaps be `recordsToDelete`. 2. It seems a bit inconsistent that the argument is `RecordsToDelete`, but the result is just a `Long`. Should the result be `DeleteRecords` or something like that? It could then

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-10 Thread Ismael Juma
s see continued bug fixes and releases in the community or is the plan > to have one single main version that gets continuous updates and releases? > > By the way, why June 2018? :) > > -Jaikiran > > > > On 09/11/17 3:14 PM, Ismael Juma wrote: > >> Hi all, >

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-09 Thread Ismael Juma
idempotent producer the default mode. We missed the window on 1.0.0 because we hadn't completed necessary work. But we should have it done by June 2018. Thanks, Apurva On Thu, Nov 9, 2017 at 4:10 AM, Ismael Juma wrote: > That's correct, Tom. We can only remove deprecated APIs in major

[jira] [Resolved] (KAFKA-6185) Selector memory leak with high likelihood of OOM in case of down conversion

2017-11-09 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6185. Resolution: Fixed > Selector memory leak with high likelihood of OOM in case of down convers

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-09 Thread Ismael Juma
oss, but that's my two cents > > On 9 Nov. 2017 8:44 pm, "Ismael Juma" wrote: > > > Hi all, > > > > I'm starting this discussion early because of the potential impact. > > > > Kafka 1.0.0 was just released and the focus was on achieving t

[DISCUSS] Kafka 2.0.0 in June 2018

2017-11-09 Thread Ismael Juma
Hi all, I'm starting this discussion early because of the potential impact. Kafka 1.0.0 was just released and the focus was on achieving the original project vision in terms of features provided while maintaining compatibility for the most part (i.e. we did not remove deprecated components like t

[jira] [Resolved] (KAFKA-6146) minimize the number of triggers enqueuing PreferredReplicaLeaderElection events

2017-11-08 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6146. Resolution: Fixed > minimize the number of triggers enqueuing PreferredReplicaLeaderElect

Re: kafka-pr-jdk9-scala2.12 keeps failing

2017-11-07 Thread Ismael Juma
I changed the Jenkins jobs to use Oracle JDK 9 instead of 9.0.1 until INFRA-15448 is fixed. Ismael On Mon, Nov 6, 2017 at 6:25 PM, Ismael Juma wrote: > Thanks! > > Ismael > > On Mon, Nov 6, 2017 at 3:48 AM, Ted Yu wrote: > >> Logged https://issues.apache.org/jira/bro

Re: [VOTE] KIP-201: Rationalising policy interfaces

2017-11-07 Thread Ismael Juma
#x27;s a "client " class although should only reside within a > broker > > On 7 Nov. 2017 9:04 pm, "Ismael Juma" wrote: > > The location of the policies is fine. Note that the package _does not_ > include clients in the name. If we ever have enough server side on

<    6   7   8   9   10   11   12   13   14   15   >