Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-09-03 Thread Colin McCabe
sues, I think we should come up > > with a minimal validation and ensure it won't become more restrictive in > > the future. > > > > 4. I don't have strong opinion regarding this one but as the focus of the > > KIP is to gather the client's information, I

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-09-03 Thread Colin McCabe
> retain the > > > old > > > authorizer as-is. SimpleAclAuthorizer will continue to > > implement the > > > old > > > API, but will be deprecated. A new authorizer implementation > > > kafka.security.authoriz

[VOTE] KIP-482: The Kafka Protocol should Support Optional Tagged Fields

2019-09-03 Thread Colin McCabe
Hi all, I'd like to start the vote for KIP-482: The Kafka Protocol should Support Optional Tagged Fields. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields Discussion thread here: https://lists.apache.org/thread.html/cdc

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-09-03 Thread Colin McCabe
take up that work for making it configurable. > > > > > > Thanks > > > Maulin > > > > > > > > > > > > > > > > > > On Fri, Aug 30, 2019 at 10:34 AM Maulin Vasavada < > > > maulin.vasav...@gmail.com> > &

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-09-03 Thread Colin McCabe
es. This is something that we'll discuss when people propose release schedules. In general, this isn't fundamentally different than someone wanting a new release of 1.x because they don't want to upgrade to 2.x. If there's enough interest, we'll do it. best, Colin &g

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-08-30 Thread Colin McCabe
faces will be "file based" loading vs > > if somebody wants to customize any of it they can. > > > > Would that make sense? > > > > On Fri, Aug 30, 2019 at 10:03 AM Colin McCabe wrote: > > > >> +1 for making SslEngineBuilder configurable. This would

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-08-30 Thread Colin McCabe
gt;> > > > > > >> > > > wrote: > > >> > > > > > >> > > > Hi Colin > > >> > > > > > >> > > > When I refer to "standard" or "custom" algorithms I am following > >

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-08-30 Thread Colin McCabe
Hi Pere, It will depend on how quickly the KIP can be voted in, and then implemented. Manikumar Reddy is going to be the release manager for Apache Kafka 2.4.0. Right now, the KIP freeze is scheduled for Sep 25, 2019. A KIP must be accepted by this date in order to be part of 2.4.0. (Being a

Re: [ DISCUSS ] KIP-512:Adding headers to RecordMetaData

2019-08-29 Thread Colin McCabe
As Gwen commented earlier, the client already has the record that it sent, including all the headers. > > Future future = producer.send(myRecord, null); > future.get(); > System.out.println("I sent myRecord with headers " + myRecord.headers()); > best, Colin On Tue, Aug 27, 2019, at 17:06, Ren

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-29 Thread Colin McCabe
On Mon, Aug 26, 2019, at 14:03, Jason Gustafson wrote: > Hi Arjun, > > From a high level, I feel like we are making light of the JMX api because > it's convenient and the broker already has it. Personally I would take the > broker out of the picture. The JMX endpoint is not something we were happy

Re: Permissions to create a KIP for KAFKA-8843

2019-08-29 Thread Colin McCabe
On Thu, Aug 29, 2019, at 09:27, Pere Urbón Bayes wrote: > Thanks, > yes I know the KIP-500, how realistic is to have it landing for 2.4? if > not really, end of the day we will need something like KAFKA-8843 for this > version. > > Looking forward to help out. > Hi Pere, Thanks for contributi

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-08-29 Thread Colin McCabe
On Fri, Aug 23, 2019, at 00:07, Magnus Edenhill wrote: > Great proposal, this feature is well overdue! > > 1) > From an operator's perspective I don't think the kafka client > implementation name and version are sufficient, > I also believe the application name and version are of interest. > You c

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-28 Thread Colin McCabe
Hi all, After some discussion with Jun and Stan, we decided that we should bump the version of the topics znode from 1 to 2. The bump is backwards compatible (older brokers can read the v2 znode). I have updated the KIP. best, Colin On Thu, Aug 8, 2019, at 11:09, Colin McCabe wrote: >

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-27 Thread Colin McCabe
t is important enough to be in the spec > file, it should be in > a proper JSON field (e.g., about), which makes sure it ends up in the > generated protocol docs instead of being filtered out. > Yeah, we could consider reorganizing this a bit. This seems like a separate issue fr

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-26 Thread Colin McCabe
o nail down what we are voting for. > > Ryanne > > > On Fri, Aug 23, 2019, 12:58 PM Colin McCabe wrote: > > > On Fri, Aug 23, 2019, at 06:24, Ryanne Dolan wrote: > > > Colin, can you outline what specifically would be in scope for this KIP > > vs > &g

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

2019-08-23 Thread Colin McCabe
URPs based on that. +1 for this. (I assume Jason will update the KIP...) best, Colin > > Taking that into account, +1 from me! (non-binding) > > On Fri, Aug 23, 2019 at 7:00 PM Colin McCabe wrote: > > > +1 (binding). > > > > cheers, > > Colin > >

Re: [VOTE] KIP-352: Distinguish URPs caused by reassignment

2019-08-23 Thread Colin McCabe
+1 (binding). cheers, Colin On Tue, Aug 20, 2019, at 10:55, Jason Gustafson wrote: > Hi All, > > I'd like to start a vote on KIP-352, which is a follow-up to KIP-455 to > fix > a long-known shortcoming of URP reporting and to improve reassignment > monitoring: > https://cwiki.apache.org/conflue

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-23 Thread Colin McCabe
out the specific details. best, Colin > > On Thu, Aug 22, 2019, 11:58 AM Colin McCabe wrote: > > > On Wed, Aug 21, 2019, at 19:48, Ron Dagostino wrote: > > > Thanks, Colin. The changes you made to the KIP related to the bridge > > > release help make it clear

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-22 Thread Colin McCabe
s release. I added some more descriptive text to the bridge release paragraph-- hopefully this makes it clearer. best, Colin > > Do I understand it correctly, and might some change in phrasing or > additional clarification help others avoid the same confusion I had? > > Ron >

Re: ACL for group creation?

2019-08-21 Thread Colin McCabe
I think it's worth considering separating out the permissions needed to create a consumer group from the permissions needed to join one. We distinguish these permissions for topics, and people generally find it useful. We could start checking CREATE on GROUP, perhaps? It might be hard to do

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-21 Thread Colin McCabe
g)? > > Ryanne > > On Wed, Aug 21, 2019, 1:28 PM Colin McCabe wrote: > > > On Wed, Aug 21, 2019, at 06:38, Eno Thereska wrote: > > > Hi Colin, > > > > > > Nice KIP! For such a big change it would be good to add a pointer or > > > two to

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-21 Thread Colin McCabe
K world, the leader will make an RPC to the active > controller instead > > << controller. > >>>For example, the brokers may need to forward their requests to the > active controller. > > << registrations > >>>The new (active) controller will m

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-21 Thread Colin McCabe
rectly to > > the active controller > > > > << > instead > > >>>In the post-ZK world, the leader will make an RPC to the active > > controller instead > > > > << > controller. > > >>>For example, the brokers may ne

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-21 Thread Colin McCabe
gt; > > is partitioned from the controller > > > > > > << > > >>>Eventually, the active controller will ask the broker to finally go > > > offline > > > > > > << > > the controller > > > >>>New versi

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-19 Thread Colin McCabe
Hi all, The KIP has been out for a while, so I'm thinking about calling a vote some time this week. best, Colin On Mon, Aug 19, 2019, at 15:52, Colin McCabe wrote: > On Mon, Aug 19, 2019, at 12:52, David Arthur wrote: > > Thanks for the KIP, Colin. This looks great! > > &

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-19 Thread Colin McCabe
etch. This is similar to how we have checksums on regular data. Or if the question is about catching logic errors in the metadata handling code, that sounds more like something that should be caught by test cases. best, Colin > Thanks! > > On Fri, Aug 9, 2019 at 1:17 PM Colin McCabe

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-19 Thread Colin McCabe
t; > > > Regards, > > > > David > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 10:00 AM Jason Gustafson > > > > wrote: > > > > > > > > > > Right, I was planning on doing exactl

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-19 Thread Colin McCabe
gt; across all the requests or responses (e.g. tracing id). > > > > > > > > > > Regards, > > > > > David > > > > > > > > > > > > > > > > > > > > On Tue, Aug 13, 2019 at 10:00 AM Jason Gusta

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-08-19 Thread Colin McCabe
Hi Maulin, A lot of JSSE providers don't implement custom algorithms. Spire is a good example of a JSSE provider that doesn't, and yet is still useful to many people. Your JSSE provider can work fine even if it doesn't implement a custom algorithm. Maybe I'm missing something, but I don't un

Re: all tests passed but still got ‘Some checks were not successful’

2019-08-19 Thread Colin McCabe
I think we've been having some test runs hit the 4-hour time limit recently. I saw a few of those as well. This is just a guess, but I bet that there is a test that is hanging, and has no time limit configured. We should probably have time limits on every test. The person writing the test sh

Re: [VOTE] KIP-504 - Add new Java Authorizer Interface

2019-08-16 Thread Colin McCabe
+1 (binding) Thanks, Rajini! best, Colin On Fri, Aug 16, 2019, at 09:52, Rajini Sivaram wrote: > Hi all, > > I would like to start the vote for KIP-504: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface > > This KIP replaces the Scala Authorizer AP

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-16 Thread Colin McCabe
> > -Jason > > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe wrote: > > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote: > > > Hi Colin, > > > > > > Thanks for the KIP! This is a significant improvement. One of my personal > > >

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-16 Thread Colin McCabe
clude the version bump for all RPCs > > in this KIP, but we can implement it lazily as the protocols are converted. > > > > -Jason > > > > On Mon, Aug 12, 2019 at 7:16 PM Colin McCabe wrote: > > > > > On Mon, Aug 12, 2019, at 11:22, Jason Gustafson wrote: &g

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-08-15 Thread Colin McCabe
your help! > > Regards, > > Rajini > > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe wrote: > > > Hi Rajini, > > > > I think it would be good to rename KafkaRequestContext to something like > > AuthorizableRequestContext, and put it in the > >

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

2019-08-15 Thread Colin McCabe
; and Colin voting. > > -Jason > > On Tue, Aug 13, 2019 at 9:21 AM Colin McCabe wrote: > > > Hi Jason, > > > > Thanks for the KIP. > > > > Is there ever a desire to delete all the offsets for a given group? > > Should the protocol and tools sup

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

2019-08-14 Thread Colin McCabe
quot; > > > to > > > > > > > > > > *KafkaFuture>>* > > > > > > > > > > This is because we will have a hierarchy of two-layers of errors > > since > > > we > > > > > need to find out

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-08-14 Thread Colin McCabe
Hi Rajini, I think it would be good to rename KafkaRequestContext to something like AuthorizableRequestContext, and put it in the org.apache.kafka.server.authorizer namespace. If we put it in the org.apache.kafka.common namespace, then it's not really clear that it's part of the Authorizer AP

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

2019-08-13 Thread Colin McCabe
Hi Mickael, Considering that KIP-496, which adds a way of deleting consumer offsets from AdminClient, looks like it is going to get in, this seems like functionality we should definitely have. For alterConsumerGroupOffsets, is the intention to ignore partitions that are not specified in the ma

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

2019-08-13 Thread Colin McCabe
Hi Jason, Thanks for the KIP. Is there ever a desire to delete all the offsets for a given group? Should the protocol and tools support this? +1 (binding) best, Colin On Mon, Aug 12, 2019, at 10:57, Guozhang Wang wrote: > +1 (binding). > > Thanks Jason! > > On Wed, Aug 7, 2019 at 11:18 AM

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

2019-08-13 Thread Colin McCabe
this as it is until KIP-396 passes the vote (the vote > >> for KIP-396 opened at January and it still doesn't pass - 7 months - which > >> worries me a bit if it's going to pass the vote or not), but I also > >> respect > >> the lifecycle of KIP in Ka

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-12 Thread Colin McCabe
ir point. It would be shorter on average, but worse for some exceptional cases. Also, the decoding would be more complex, which might be a good reason to go for just having two varints. Yeah, let’s simplify. Regards, Colin > > Thanks, > Jason > > > On Fri, Aug 9, 2019 at 4:31 PM

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

2019-08-12 Thread Colin McCabe
rn against the list > > of available topics in the driver. > > > > As you use `assignment()` and store offsets in the Spark checkpoint, it > > seems that using consumer group management is not a good fit for the use > > case. > > > > > > Thoughts? > >

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

2019-08-12 Thread Colin McCabe
+1 for better access control here. In general, impersonating another user seems like it’s equivalent to super user access. Colin On Mon, Aug 12, 2019, at 05:43, Manikumar wrote: > Hi Viktor, > > As per the KIP, It's not only superuser, any user with required permissions > (CreateTokens on Clust

Re: [DISCUSS] Apache Kafka 2.4.0 release

2019-08-12 Thread Colin McCabe
+1. Thanks, Manikumar. Colin On Mon, Aug 12, 2019, at 08:25, Matthias J. Sax wrote: > Thanks Manikumar! SGTM. > > > On 8/12/19 7:54 AM, Manikumar wrote: > > Hi all, > > > > I would like to volunteer to be the release manager for our next time-based > > feature release (v2.4.0). > > > > If tha

Re: Dynamic configuration of interbroker SSL certificates

2019-08-12 Thread Colin McCabe
Hi Michael, The NetworkClient periodically fetches metadata so that it always knows what the cluster topology is. This also helps it to have some open connections when needed to reduce the latency of operations. To be fair, we haven’t thought very much about optimizing this since the overhead

Re: [DISCUSS] KIP-505 : Add new public method to only update assignment metadata in consumer

2019-08-12 Thread Colin McCabe
Hi, If there’s no need to consume records in the Spark driver, then the Consumer is probably the wrong thing to use. Instead, Spark should use AdminClient to find out what partitions exist and where, manage their offsets, and so on. There are some KIPs under discussion now that would add the ne

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-08-09 Thread Colin McCabe
Hi all, I've made some updates to this KIP. Specifically, I wanted to avoid including escape bytes in the serialization format, since it was too complex. Also, I think this is a good opportunity to slim down our variable length fields. best, Colin On Thu, Jul 11, 2019, at 20:52,

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

2019-08-09 Thread Colin McCabe
+1 (binding) cheers, Colin On Fri, Aug 9, 2019, at 09:56, Ron Dagostino wrote: > +1 (non-binding) > > The simplest of KIPs, with perhaps the biggest impact. Like removing > the thorn from the soles of my feet. > > Thanks for doing it. > > > On Aug 9, 2019, at 12:50 PM, Dongjin Lee wrote: >

Re: Alternative of poll(0) without pulling records

2019-08-09 Thread Colin McCabe
Hi Gabor, What is it that you want to do here? If you just want to check that the partitions exist, but not fetch any data, you could use AdminClient#describeTopics for that. If you want to create the topics, you could use AdminClient#createTopics. best, Colin On Fri, Aug 9, 2019, at 11:23

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-08-09 Thread Colin McCabe
ndamental disagreement in what we > > should > > > allow the producer to do. > > > In my previous message I mentioned a config that would allow for the > > broker > > > to prevent producer auto-creation. (It would be disabled by default.) It > > &g

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-09 Thread Colin McCabe
this > something we should keep? > > Thanks > > On Mon, Aug 5, 2019 at 7:44 PM Colin McCabe wrote: > > > > On Mon, Aug 5, 2019, at 10:02, Tom Bentley wrote: > > > Hi Colin, > > > > > > Thanks for the KIP. > > > > > > Curren

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

2019-08-08 Thread Colin McCabe
Harsha made a good point that you can achieve your goals through KIP-492. Security configuration is starting to get pretty complex-- is there a reason not to use the existing configurations? Also, it seems like most people who want a custom truststore / keystore will also want a custom SSL pro

Re: [DISCUSS] KIP-499 - Unify connection name flag for command line tool

2019-08-08 Thread Colin McCabe
I agree that limiting the scope of the KIP would be good. The configuration is actually bootstrap.servers with an S, though. I actually like --bootstrap-servers slightly better than --bootstrap-server, although I don't feel that strongly about either. ;) best, Colin On Thu, Aug 8, 2019, at 14

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

2019-08-08 Thread Colin McCabe
Hi Rajini, Thanks for the KIP. This will be a great improvement. Why not just pass the cluster ID directly to Authorizer#start, rather than dealing with the ClusterResourceListener interface? That seems like it would be simpler. If authorizers don't need that information, they can ignore tha

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-08 Thread Colin McCabe
> -Original Message----- > From: Colin McCabe > Sent: Wednesday, August 7, 2019 5:17 PM > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-455: Create an Administrative API for Replica > Reassignment > > On Wed, Aug 7, 2019, at 15:41, George Li wrote: > > T

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

2019-08-07 Thread Colin McCabe
oducing?  I was thinking about a PlacementPolicy filling the role of preventing people from creating single-replica partitions on a node that we didn't want to ever be the leader. I thought that it could also prevent people from designating those nodes as preferred leaders during topic creat

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-07 Thread Colin McCabe
started after some time. > This has definitely been an issue in the past, I agree. Thankfully, we recently did improve the robustness of the ReplicaFetcher. Check out "KIP-461: Improve Replica Fetcher behavior at handling partition failure." cheers, Colin > > > Than

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

2019-08-07 Thread Colin McCabe
ics that get put on the "bad" replica. Perhaps we should reopen the discussion about https://cwiki.apache.org/confluence/display/KAFKA/KIP-201%3A+Rationalising+Policy+interfaces regards, Colin > > Please let me know there are more question.  > > > Thanks, > George

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-08-07 Thread Colin McCabe
On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote: > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org > wrote: > > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > >> > >> Hi Colin, > >> "Hmm... I'm not sure I follow.

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-08-06 Thread Colin McCabe
't want to implement client-side topic creation in the consumer in "rejected alternatives." Maybe Justine can add more context here in the KIP. The last time we talked about this, the reasoning was that we wanted to eventually get rid of consumer-side auto-topic creation ent

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-08-06 Thread Colin McCabe
his, and I think it was useful. NFS also supported (supports?) a mode where you just pass whatever user ID you want and the system believes you. These things clearly don't protect against malicious users, but they can help set up policies when needed. best, Colin > > Thanks, > Harsha

Re: [ANNOUNCE] Apache Kafka 2.3.0

2019-08-06 Thread Colin McCabe
a.apache.org/protocol > > Thanks > > On Tue, Jul 2, 2019 at 2:05 AM Colin McCabe wrote: > > > > Hi Mickael, > > > > Thanks for pointing this out. It should be fixed now. > > > > best, > > Colin > > > > On Mon, Jul 1, 2019, at 09:14

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-06 Thread Colin McCabe
the new binary. Otherwise, the > reassignment task may not be completed if the controller changes to a > broker still on the old binary. > IBP is one way to achieve that. The main thing is that we need some way > for the controller to deal with the new ZK fields. Dealing with the &g

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-08-06 Thread Colin McCabe
her from clients by server side config. > > Server side configs of default topic, partitions should take higher > > precedence and client shouldn't be able to create a topic with higher > > > > no.of > > > > partitions, replication than what server config specifie

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

2019-08-05 Thread Colin McCabe
I think it would be useful to have this in AdminClient. Especially if we implement KIP-496: Administrative API to delete consumer offsets. It would be odd to have a way to delete consumer offsets in AdminClient, but not to create them. What do you think? best, Colin On Sun, Aug 4, 2019, at

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-05 Thread Colin McCabe
> what > > > > > we did in KIP-98 > > > > > < > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging > > > >

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-05 Thread Colin McCabe
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging > > > > >, > > > > including the new request protocols and how they are interacting in the > > > new > > > > cluster. For a complicated change lik

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-02 Thread Colin McCabe
On Fri, Aug 2, 2019, at 16:33, Jose Armando Garcia Sancio wrote: > Thanks Colin for the detail KIP. I have a few comments and questions. > > In the KIP's Motivation and Overview you mentioned the LeaderAndIsr and > UpdateMetadata RPC. For example, "updates which the controller pushes, such > as Le

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-02 Thread Colin McCabe
019 at 12:23 PM Colin McCabe wrote: > > > On Fri, Aug 2, 2019, at 07:50, Ryanne Dolan wrote: > > > Thanks Colin, interesting KIP. > > > > > > I'm concerned that the KIP does not actually address its stated > > > motivations. In particular, "S

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-02 Thread Colin McCabe
e less bandwidth to do so. It will even be possible for the brokers to cache this state locally in a file on disk, so that broker startup can be much faster. All of these are important to scaling Kafka in the future. Treating metadata as a log avoids a lot of the complex failure corner cases

Re: [DISCUSS] KIP-499 - Unify connection name flag for command line tool

2019-08-01 Thread Colin McCabe
On Wed, Jul 31, 2019, at 05:26, Mitchell wrote: > Hi Jason, > Thanks for looking at this! > > I wasn't exactly sure what to put in the compatibility section. I wrote > the KIP thinking that we should probably mark the old arguments for > deprecation for a release or two before actually removing t

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-08-01 Thread Colin McCabe
> > those > > > > > unneeded replicas. > > > > Right. Let's add this. > > > > > > > 13. Since we changed the format of the topics/[topic] zNode, should > > we bump > > > > > up the version number in the json value? >

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-01 Thread Colin McCabe
Hi Harsha, Thanks for the heads up. This should be fixed-- give it another try. best, Colin On Thu, Aug 1, 2019, at 14:15, Harsha wrote: > Hi Colin, > Looks like KIP is missing the images , links are broken. > Thanks, > Harsha > > On Thu, Aug 1, 2019, at 2:0

[DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-01 Thread Colin McCabe
Hi all, I've written a KIP about removing ZooKeeper from Kafka. Please take a look and let me know what you think: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum cheers, Colin

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-07-31 Thread Colin McCabe
s be clear to the user. Changing the code > > to > > > > have > > > > > > the > > > > > > > > producer's configurations take precedence is possible, but I > > was > > > > > > wondering > > > > > >

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-31 Thread Colin McCabe
Of course downgrade isn't officially supported, but it would be nice not to break it if we don't need to...) Changing the version number would also create problems during a rolling upgrade. best, Colin > > > > > > Jun > > > > > > On Mon, Jul 22, 2019

Re: [DISCUSS] KIP-497: Add inter-broker API to alter ISR

2019-07-28 Thread Colin McCabe
Hi Jason, This looks good. If the AlterIsr request returns a higher ZK version than the one the broker currently has, will the broker use that as its new ZK version? I suppose this could happen if some of the updates the controller pushed out were not received or not received yet by the broke

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

2019-07-25 Thread Colin McCabe
We still want to give the "blacklisted" broker the leadership if nobody else is available. Therefore, isn't putting a broker on the blacklist pretty much the same as moving it to the last entry in the replicas list and then triggering a preferred leader election? If we want this to be undone a

Re: Check the version of kafka information for MDT Mirror information

2019-07-25 Thread Colin McCabe
What is an MDT mirror server? best, Colin On Thu, Jul 18, 2019, at 18:29, Harry k wrote: > Hi, > How to check version information for Kafka that is being used on the MDT > Mirror servers? Is their any command to check that.I have any only access > to Kafka and zookeeper servers through putty.Any

Re: [DISCUSS] KIP-490: log when consumer groups lose a message because offset has been deleted

2019-07-22 Thread Colin McCabe
Hi Jose, One issue that I see here is that the number of log messages could be huge. I've seen people create tens of thousands of consumer groups. People can also have settings that create pretty small log files. A message per log file per group could be quite a lot of messages. A log messa

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-22 Thread Colin McCabe
son Gustafson > > wrote: > > > > > > +1 Thanks for the KIP. Really looking forward to this! > > > > > > -Jason > > > > > > On Wed, Jul 17, 2019 at 1:41 PM Colin McCabe wrote: > > > > > > > Thanks, Stanislav. Let's r

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

2019-07-17 Thread Colin McCabe
> > As always, you can re-read the KIP here > https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment > > Best, > Stanislav > > On Wed, May 22, 2019 at 6:12 PM Colin McCabe wrote: > > > Hi George, > >

Re: [VOTE] KIP-484: Expose metrics for group and transaction metadata loading duration

2019-07-15 Thread Colin McCabe
+1 (binding). Thanks for the KIP, Anastasia. best, Colin On Fri, Jul 12, 2019, at 10:09, Jason Gustafson wrote: > +1 Thanks for the KIP! > > -Jason > > On Fri, Jul 12, 2019 at 9:59 AM Gwen Shapira wrote: > > > +1 (binding) > > > > Looks great to me. Thank you for the KIP. > > > > On Mon, Ju

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-15 Thread Colin McCabe
On Sat, Jul 13, 2019, at 17:54, George Li wrote: > I just take a look at the updated KIP-455 again. I noticed > this "targetReplicas" is removed and instead, put "addingReplicas" & > "removingReplicas".  So where does the new reassigned "targetReplicas" > is stored? When all replicas in ISR, and

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-15 Thread Colin McCabe
On Mon, Jul 15, 2019, at 11:51, Jason Gustafson wrote: > Hi Colin, > > A few more questions below: > > 1. The KIP says that the --zookeeper option will be removed from > kafka-reassign.sh. Do you mean that it will be deprecated and eventually > removed? Keeping the --zookeeper flag here would be

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-15 Thread Colin McCabe
On Mon, Jul 15, 2019, at 14:31, Stanislav Kozlovski wrote: > Hey George, > > > Different replica threads for throttling > The reason we can't support throttling for reassigning partitions right now > is because we have no good way of telling whether a replica is part of an > ongoing reassignment o

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-15 Thread Colin McCabe
On Sat, Jul 13, 2019, at 17:54, George Li wrote: > Hi Stanislav, > > sorry for the late reply. comments below:  > > > Thanks for the reminder. A lot of your suggestions are outlined in the > > "Future Work" section of KIP-455. The pointer towards different > > ReplicaFetcher thread pools is inte

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-12 Thread Colin McCabe
the first partition in time, the > > producer will send a single record batch. In the worse case, it can be that > > every other batch has only a single record. Is this correct? If so, could > > we avoid that? > > > > Jun > > > > On Thu, Jul 11, 2019 at 5:23

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-07-11 Thread Colin McCabe
On Tue, Jul 9, 2019, at 15:29, Jose Armando Garcia Sancio wrote: > Thanks Colin for the KIP. For my own edification why are we doing this > "Optional fields can have any type, except for an array of structures."? > Why can't we have an array of structures? Optional fields are serialized starting w

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

2019-07-11 Thread Colin McCabe
Hi Justine, Thanks for the KIP. This seems like a good step towards removing server-side topic auto-creation. We should add included "client-side" to the title of the KIP somewhere, to make it clear that we're talking about client-side auto creation. The KIP says: > In order to automatically

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-11 Thread Colin McCabe
+1 (binding). Thanks, Justine! ComputedPartition#get probably should be ComputedPartition#partition or something. We typically name accessors the same as the variables that are being accessed. As we discussed in the other thread, one minor addition that might make this KIP even better is a S

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-11 Thread Colin McCabe
good feature and having > > that extended to RoundRobinPartitioner means 1 less KIP in the future. > > > > Would it be appropriate to extend the support to RoundRobinPartitioner too? > > > > Thanks, > > > > On Tue, 9 Jul 2019 at 17:24, Colin McCabe w

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-09 Thread Colin McCabe
;> I made some changes to the KIP. > > >> The idea is to clean up the code, make behavior more explicit, provide > > >> more flexibility, and to keep default behavior the same. > > >> > > >> Now we will change the partition in onNewBatch, and spec

Re: PR review

2019-07-08 Thread Colin McCabe
Hi M. Manna, I left a review. Take a look. Sorry for the delays. best, Colin On Mon, Jul 8, 2019, at 14:38, M. Manna wrote: > Hello, > > A few requests have been sent already. Could this please be reviewed ? Our > business implementation is holding due to this change. > > > > On Thu, 4 Ju

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-03 Thread Colin McCabe
we wait until the partitionB > > > move finishes as well? > > > We wait for the partitionB move to finish. The rationale is that we don't > > really ever know what is in ZK (it could change at any time, and our writes > > to ZK could race with someone

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

2019-07-02 Thread Colin McCabe
On Tue, Jul 2, 2019, at 10:47, Stanislav Kozlovski wrote: > Hey there, I need to start a new thread on KIP-455. I think there might be > an issue with the mailing server. For some reason, my replies to the > previous discussion thread could not be seen by others. After numerous > attempts, Colin su

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-07-02 Thread Colin McCabe
le message. I will try to clarify the text here. best, Colin > > Thanks, > > Tom > > On Wed, Jun 26, 2019 at 10:01 PM Colin McCabe wrote: > > > Hi all, > > > > I would like to start a discussion for KIP-482: > > > > https://cwiki.a

Re: [VOTE] KIP-476: Add Java AdminClient interface

2019-07-02 Thread Colin McCabe
On Tue, Jul 2, 2019, at 09:14, Colin McCabe wrote: > On Mon, Jul 1, 2019, at 23:30, Matthias J. Sax wrote: > > Not sure, if I understand the argument? > > > > Why would anyone need to support multiple client side versions? > > Clients/brokers are forward/backward c

Re: [VOTE] KIP-476: Add Java AdminClient interface

2019-07-02 Thread Colin McCabe
On Mon, Jul 1, 2019, at 23:30, Matthias J. Sax wrote: > Not sure, if I understand the argument? > > Why would anyone need to support multiple client side versions? > Clients/brokers are forward/backward compatible anyway. When you're using many different libraries, it is helpful if they don't imp

Re: [ANNOUNCE] Apache Kafka 2.3.0

2019-07-01 Thread Colin McCabe
e in a row this happens (2.1 and 2.2 had the same > issue at release). Last time, Guozhang confirmed this step is in the > release process but maybe this needs to be highlighted > > On Tue, Jun 25, 2019 at 8:22 PM Colin McCabe wrote: > > > > Thanks to everyone who revi

<    7   8   9   10   11   12   13   14   15   16   >