Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Thanks for KIP-455!  yes. KIP-236, etc. will depend on it.  It is the good direction to go for the RPC.  Regarding storing the new reassignments & original replicas at the topic/partition level.  I have some concerns when controller is failing over, and the scalability of scanning t

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi Jason, Sorry for the late response.  been busy.  I have taken a brief look at Colin's KIP-455. It's a good direction for Reassignments. I think KIP-236, KIP-435,  KIP-455 need some consensus / coordination for where the new reassignment data AND the original replicas is stored.   The curre

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks Bill! Get Outlook for iOS From: Bill Bejeck Sent: Tuesday, April 30, 2019 6:44:17 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams Thanks for the KIP Boyang. I have no additional comments

Re: [DISCUSS] KIP-401 TransformerSupplier/ProcessorSupplier enhancements

I definitely don't mind anyone jumping, Bruno, thanks for the comments! 1) I'm not totally sure I'm clear on your point, but I think we're on the same page - if we're adding a method to the XSupplier interfaces (by making them inherit from a super interface StateStoreConnector) then we definitely

[jira] [Created] (KAFKA-8308) Update jetty for security vulnerability CVE-2019-10241

Di Shang created KAFKA-8308: --- Summary: Update jetty for security vulnerability CVE-2019-10241 Key: KAFKA-8308 URL: https://issues.apache.org/jira/browse/KAFKA-8308 Project: Kafka Issue Type: Task

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

+1 (non-binding) Really looking forward to this. Thanks, Randall! On Tue, Apr 30, 2019, 20:47 Magesh Nandakumar wrote: > This will make connect debugging so much easier. Thanks a lot for driving > this Randall. > > +1 (non-binding) > > Thanks, > Magesh > > On Tue, Apr 30, 2019 at 7:19 PM Jeremy

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

This will make connect debugging so much easier. Thanks a lot for driving this Randall. +1 (non-binding) Thanks, Magesh On Tue, Apr 30, 2019 at 7:19 PM Jeremy Custenborder wrote: > +1 non binding > > On Mon, Apr 29, 2019 at 5:34 PM Randall Hauch wrote: > > > > I would like to start the vote f

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

Randall, The approach to return the to override configs could possibly make it cumbersome to implement a custom policy. This is a new configuration and if you don't explicitly set it the existing behavior remains as-is. Like Chris, I also preferred this approach for the sake of simplicity. If not

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

Also +1 on the issues/goals as Michael outlined them, I think that sets a great framework for the discussion. Regarding the SortedMap solution, my understanding is that the current proposal in the KIP is what is in my PR which (pending naming decisions) is roughly this: stream.split() .branch

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

+1 non binding On Mon, Apr 29, 2019 at 5:34 PM Randall Hauch wrote: > > I would like to start the vote for KIP-258: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-449%3A+Add+connector+contexts+to+Connect+worker+logs > > The KIP uses the Mapped Diagnostic Contexts (MDC) feature of SLF4J

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

Thanks, Chris. I still think it's strange to have a non-policy, since there's now special behavior for when the policy is not specified. Perhaps the inability for a policy implementation to represent the existing behavior suggests that the policy interface isn't quite right. Could the policy's "va

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

I'd like to +1 what Michael said about the issues with the existing branch method, I agree with what he's outlined and I think we should proceed by trying to alleviate these problems. Specifically it seems important to be able to cleanly access the individual branches (eg by mapping name->stream),

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the KIP Boyang. I have no additional comments from the ones already presented. +1(binding) -Bill On Tue, Apr 30, 2019 at 4:35 PM Boyang Chen wrote: > Thank you Guozhang! > > > From: Guozhang Wang > Sent: Wednesday, May 1, 2019 3:54 AM > To: dev@kaf

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

Randall, The special behavior for null was my suggestion. There is no implementation of the proposed interface that causes client overrides to be ignored, so the original idea was to have a special implementation that would be checked for by the Connect framework (probably via the instanceof opera

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

On Tue, Apr 30, 2019 at 4:20 PM Magesh Nandakumar wrote: > Randall, > > Thanks a lot for your feedback. > > You bring up an interesting point regarding the overrides being available > to the connectors. Today everything that is specified in the config while > creating is available for the connect

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

Nice simple improvement. Thanks, Paul! +1 (binding) Randall On Mon, Apr 29, 2019 at 5:06 PM Magesh Nandakumar wrote: > Looks good to me and a very useful feature. > > +1 ( non-binding) > > On Mon, Apr 29, 2019, 4:05 PM Arjun Satish wrote: > > > Thanks, Paul! This is very useful. > > > > +1 (n

Re: [DISCUSS] KIP-460: Admin Leader Election RPC

On Tue, Apr 30, 2019 at 11:39 AM Jason Gustafson wrote: > Thanks for the updates, Jose. The proposal looks good to me. Just one minor > question I had is whether we should even have a default --election-type in > kafka-leader-election.sh. I am wondering if it is reasonable to make the > user be e

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

Randall, Thanks a lot for your feedback. You bring up an interesting point regarding the overrides being available to the connectors. Today everything that is specified in the config while creating is available for the connector. But this is a specific case and we could do either of the following

[jira] [Created] (KAFKA-8307) Kafka Streams should provide some mechanism to determine topology equality and compatibility

John Roesler created KAFKA-8307: --- Summary: Kafka Streams should provide some mechanism to determine topology equality and compatibility Key: KAFKA-8307 URL: https://issues.apache.org/jira/browse/KAFKA-8307

Jenkins build is back to normal : kafka-2.1-jdk8 #176

See

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thank you Guozhang! From: Guozhang Wang Sent: Wednesday, May 1, 2019 3:54 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams +1 (binding) Guozhang On 2019/04/26 07:42:12, "Matthias J. Sax" wrote: > Thanks for the KIP!

Build failed in Jenkins: kafka-2.2-jdk8 #96

See Changes: [cmccabe] KAFKA-8134: `linger.ms` must be a long -- [...truncated 2.50 MB...] kafka.controller.PartitionLeaderElectionAlgorithmsTest > testReassignPartitionLeade

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

I’d like to propose a different way of thinking about this. To me, there are three problems with the existing branch signature: 1. If you use it the way most people do, Java raises unsafe type warnings. 2. The way in which you use the stream branches is positionally coupled to the ordering of the

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

+1 (binding) Guozhang On 2019/04/26 07:42:12, "Matthias J. Sax" wrote: > Thanks for the KIP! > > I agree that the change makes sense, and not only for the static group > membership case. > > For example, if a user `closes()` a `KafkaStreams` client and creates a > new one (for example to reco

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thank you Sophie! Added the case Matthias described in the Compatibility session. From: Sophie Blee-Goldman Sent: Wednesday, May 1, 2019 1:30 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams Hey Boyang, I think this

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

Hi Konstantine, I've updated the KIP to add default method implementations to the list of rejected alternatives. Technically this makes the changes in the KIP backwards incompatible, but I think I agree that for the majority of cases where it would even be an issue a compile-time error is likely t

Re: [DISCUSS] KIP-464 Default Replication Factor for AdminClient#createTopic

"lack of default doesn't force people to think - it gets them to pick a random number" - I suppose if I'm being pragmatic this rings very true to my experience as well... better a cluster administrator sets a sane default than having users choose random numbers! Consider me convinced - I'll update

Re: [DISCUSS] KIP-460: Admin Leader Election RPC

Thanks for the updates, Jose. The proposal looks good to me. Just one minor question I had is whether we should even have a default --election-type in kafka-leader-election.sh. I am wondering if it is reasonable to make the user be explicit about what they are trying to do? -Jason On Fri, Apr 26,

Re: [DISCUSS] KIP-464 Default Replication Factor for AdminClient#createTopic

Changing number of partitions is complicated in some use-cases and easy in other cases (when you use Kafka as a big pipe of events on the way to another system). I like making easy things easy and complex things complicated. Having defaults for both will allow the easy cases to be even easier. In

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hey Boyang, I think this sounds great but one thing you might want to update is the "Compatibility, Deprecation, and Migration Plan" -- I agree having two instances in the same JVM is probably a rare occurrence but the (presumably less rare) situation Matthias described would also be affected in c

Re: [DISCUSS] KIP-464 Default Replication Factor for AdminClient#createTopic

I have a preference toward requiring specifying partitions per topic, but I'm happy to be convinced otherwise. Changing replication factor after the fact is easy, but changing partitions is complicated since historical state gets messed up, so it could be beneficial to force clients to think about

[jira] [Resolved] (KAFKA-8134) ProducerConfig.LINGER_MS_CONFIG undocumented breaking change in kafka-clients 2.1

[ https://issues.apache.org/jira/browse/KAFKA-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe resolved KAFKA-8134. Resolution: Fixed > ProducerConfig.LINGER_MS_CONFIG undocumented breaking change in kafka-c

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

Hi Ivan, Thanks for the update. FWIW, I agree with Matthias that the current "start branching" operator is confusing when named the same way as the actual branches. "Split" seems like a good name. Alternatively, we can do without a "start branching" operator at all, and just do: stream .br

Re: [DISCUSS] KIP-464 Default Replication Factor for AdminClient#createTopic

Thanks for the KIP, Almog. This is a good change. I think we should also allow the partition count broker default to be used (the one used for auto topic creation). Ismael On Tue, Apr 30, 2019, 8:39 AM Almog Gavra wrote: > Hello Everyone, > > I'd like to start a discussion on KIP-464: > > https

[DISCUSS] KIP-464 Default Replication Factor for AdminClient#createTopic

Hello Everyone, I'd like to start a discussion on KIP-464: https://cwiki.apache.org/confluence/display/KAFKA/KIP-464%3A+Default+Replication+Factor+for+AdminClient%23createTopic It's about allowing users of the AdminClient to supply only a partition count and to use the default replication factor

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hey Bruno, "throttling purpose" means that we use `client.id` to track the request quota and do the throttling based on that. So in that context, it is expected to use same `client.id` for a certain set of consumers. Also thank you Guozhang for the comment! Merged with the KIP. Boyang

Re: Guava version upgrade

On 2019/03/15 16:44:32, "Colin McCabe" wrote: > Hi JIAHAO, > > Kafka does not use Guava. > > Some of the packages Kafka Connect depend on use Guava. Perhaps the right > thing to do is track down those projects and see how they are using Guava (if > they are vulnerable to the CVE). > > be

Build failed in Jenkins: kafka-2.1-jdk8 #175

See Changes: [matthias] MINOR: improve Session expiration notice (#6618) -- [...truncated 919.52 KB...] kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > log

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

I just re-read the discussion on the original Jira. It's still a little unclear to me, how this should work end-to-end? It would be good, to describe some test patterns that we want to support first. Maybe using some examples, that show how a test would be written? I don't think that we should bu

Re: [DISCUSS] KIP-456: Helper classes to make it simpler to write test logic with TopologyTestDriver

KIP-451 was discarded in favor this this KIP. So it seems we are all on the same page. >> Relating to KIP-448. >> What kind of alignment did you think about? Nothing in particular. Was more or less a random though. Maybe there is nothing to be aligned. Just wanted to bring it up. :) >> Some th

Re: [DISCUSS] KIP-315: Stream Join Sticky Assignor

Mike, I am still not sure, why we need to add this assignor to the project. Even after you pointed out that you cannot use Kafka Streams, the idea of the consumer to make the `PartitionAssignor` interface public and plugable is, that the project does not need to add strategies for all kind of use

Re: [DISCUSS] KIP-401 TransformerSupplier/ProcessorSupplier enhancements

Hi, @Paul: Thank you for the KIP! I hope you do not mind that I jump in. I have the following comments: 1) `null` vs empty list in the default implementation IIUC, returning `null` in the default implementation should basically signal that the method `stateStores` was not overridden. Why then p

Re: Fw: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hi Guozhang, What do you mean exactly with "throttling purposes"? @Boyang: Thank you for the KIP! Best, Bruno On Tue, Apr 30, 2019 at 1:15 AM Guozhang Wang wrote: > Hi Boyang, > > Thanks for the KIP. I think it makes sense. > > Just following up on the documentation part: since we are effecti

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

Sounds good! Let me work on this more and add some more information to this KIP before we continue. On Tue, Apr 30, 2019, 00:45 Bruno Cadonna wrote: > Hi Yishun, > > Thank you for continuing with this KIP. IMO, this KIP is very important to > develop robust code. > > I think, a good approach is

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

Hi Yishun, Thank you for continuing with this KIP. IMO, this KIP is very important to develop robust code. I think, a good approach is to do some research on mock development on the internet and in the literatures and then try to prototype the mocks. These activities should yield you a list of pr