Re: [KAFKA-7382] Guarantee atleast one replica of partition to be alive during topic creation

Team, I have addressed review comments from Mani Kumar Reddy. Please review and let me know if any changes are required. -Suman On Thu, Nov 22, 2018 at 12:39 PM Suman B N wrote: > Committers, > Please check this. > > On Thu, Nov 15, 2018 at 1:01 PM Suman B N wrote: > >> Any update on this? >>

Re: [DISCUSS] KIP-395: Encypt-then-MAC Delegation token metadata

Agree with Manikumar on having pluggable mechanism for entities required/created for delegation token mechanism. I will cover that as part of KAFKA-7694. Thanks, Satish. On Tue, Dec 11, 2018 at 12:35 PM Manikumar wrote: > > Hi, > > Thanks for the KIP. > > Currently, master/secret key is stored as

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> One based on hops using headers, and another based on topic naming. Michael, this was also suggested by Alex Mironov. Other replication engines use headers as you describe, but there are several issues with this approach: - The Connect framework provides Transformations that could be used for t

[jira] [Created] (KAFKA-7723) Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api

laomei created KAFKA-7723: - Summary: Kafka Connect support override worker kafka api configuration with connector configuration that post by rest api Key: KAFKA-7723 URL: https://issues.apache.org/jira/browse/KAFKA-7723

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Another benefit of using hops vs topic naming (also how we currently do master master in my org) You have a transactional processing app that's multi regioned. So for sake of discussion all A and B accounts process normally in ireland region all C and D in germany region and all E in Uk region.

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Hi ChienHsing, The other way I was thinking, this can be done outside of KafkaConsumer is by pausing and resuming TopicPartitions (may be in round robin fashion). There is some gotcha there as in you might not know if the consumer has already fetched data for the remaining partitions. Also I am no

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Thank you very much for your feedbacks. Currently, there is still lot of discussions regarding the Named interface. On the one hand we should provided consistency over the stream API and on the other hand we should not break the semantic as John point it up. Guozhang, I'm sorry, but I'm little bi

[jira] [Created] (KAFKA-7722) Increase ProducerPerformance precision by using nanoTime

Kevin Lu created KAFKA-7722: --- Summary: Increase ProducerPerformance precision by using nanoTime Key: KAFKA-7722 URL: https://issues.apache.org/jira/browse/KAFKA-7722 Project: Kafka Issue Type: New

Reviewers for PR: Automatic unique client id option for Kafka Connect

Hi everyone. I'm looking for a reviewer for a simple Kafka Connect PR. In short, this will enable full JMX metrics collection for Kafka Connect producers/consumers. We rely on this for monitoring our Mirus clusters in production, and I think other Kafka Connect users would benefit. The context i

Re: [Discuss] KIP-389: Enforce group.max.size to cap member metadata growth

Hey Jason, I think this is the correct understanding. One more question is whether you feel we should enforce group size cap statically or on runtime? Boyang From: Jason Gustafson Sent: Tuesday, December 11, 2018 3:24 AM To: dev Subject: Re: [Discuss] KIP-389: En

RE: [VOTE] KIP-383 Pluggable interface for SSL Factory

Since there was no objections, I'm calling a vote on KIP-383.

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

Hi Eno, Thanks for the clarification. From a high level, the main thing to keep in mind is that this is an opt-in feature. It is a bit like using acks=1 in the sense that a user is accepting slightly weaker guarantees in order to optimize for some metric (in this case, read locality). The default

RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Hi Mayuresh, To serve one poll call the logic greedily gets records from one completed fetch before including records from the next completed fetch from the queue, as you described. The algorithm remembers the current completed fetch as starting one when serving the next poll call. The net ef

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Hi Ryanne, We had an IM exchange about KIP-382 and Mirus a few weeks back, but I also want to post here to publicly express my support. I'm the primary developer of Mirus, which is a Kafka Connect based replication tool we wrote at Salesforce to replace Mirror Maker internally. We open-sourced Mir

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

Hi Harsha, Thanks for reviewing the KIP. 1) Yes, agree that we also need a max.connections configuration per-broker. I was thinking of doing that in a separate KIP, but I could add that here as well. 2) The number of connections processed in each iteration doesn't feel like an externalizable conf

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Hi ChienHsing, Thanks for the KIP. It would be great if you can explain with an example, what you mean by " Currently the implementation will return available records starting from the last partition the last poll call retrieves records from. This leads to unfair patterns of record consumption fro

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

Hi Rajini, Overall KIP looks good to me. Is it possible to use max.connections config that we already have, althought its per IP. But broker level max.connections would also be good have to guard against DOS'ing a broker. Eitherway having constant like 20 without a configurable

RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Jason, KIP 41 was initiated by you and this KIP is to change the logic discussed in the Ensure Fair Consumption. Your input on KIP-387

Re: [DISCUSS] KIP-396: Add Commit/List Offsets Operations to AdminClient

Thanks for the feedback Edo. 1) While the initial name was matching the Protocol, I agree it was not great for users. I've updated commitOffsets() to commitConsumerGroupOffsets() as suggested. 2) I thought passing the timestamp for each partition would make it harder to use in the most common cas

Jenkins build is back to normal : kafka-trunk-jdk8 #3255

See

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

Sorry for the delay in reply here. My personal life is a bit interesting at the moment so keeping up with email seems more and more impossible. :) That all makes sense to me! Thanks for entertaining the questions! On Thu, Dec 6, 2018 at 1:01 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.co

[jira] [Created] (KAFKA-7721) Connection to zookeeper refused

Mohammad Etemad created KAFKA-7721: -- Summary: Connection to zookeeper refused Key: KAFKA-7721 URL: https://issues.apache.org/jira/browse/KAFKA-7721 Project: Kafka Issue Type: Bug C

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Wikimedia currently implements 'master <-> master' replication by manually prefixing topics with datacenter names, and then configuring MirrorMaker to only replicate topics that begin with a DC name to another. While having topics named with topological details is manageable, I wouldn't say it is

Build failed in Jenkins: kafka-2.1-jdk8 #80

See Changes: [matthias] KAFKA-7443: OffsetOutOfRangeException in restoring state store from -- [...truncated 906.80 KB...] kafka.coordinator.transaction.TransactionCoordinator

Jenkins build is back to normal : kafka-trunk-jdk11 #151

See

Re: [DISCUSS] KIP-392: Allow consumers to fetch from the closest replica

Hi Jason, My question was on producer + consumer semantics, not just the producer semantics. I'll rephrase it slightly and split into two questions: - scenario 1: an application that both produces and consumes (e.g., like Kafka streams) produces synchronously a single record to a topic and then at

Build failed in Jenkins: kafka-trunk-jdk8 #3254

See Changes: [mjsax] KAFKA-7443: OffsetOutOfRangeException in restoring state store from -- [...truncated 2.24 MB...] org.apache.kafka.trogdor.workload.TopicsSpecTest > t

Build failed in Jenkins: kafka-2.0-jdk8 #201

See Changes: [matthias] KAFKA-7443: OffsetOutOfRangeException in restoring state store from -- [...truncated 435.43 KB...] kafka.controller.PartitionStateMachineTest > test

[DISCUSS] KIP-402: Improve fairness in SocketServer processors

Hi all, I have submitted a KIP to improve fairness in channel processing in SocketServer to protect brokers from connection storms: - https://cwiki.apache.org/confluence/display/KAFKA/KIP-402%3A+Improve+fairness+in+SocketServer+processors Feedback and suggestions welcome. Thank you, Raj

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

Matthias, For changelog topics, I think it does not make sense to allow skipping records if serialization fails? For internal repartitions topics, I am not sure if we should allow it or not. Would you agree with this? We should discuss the implication to derive a sound design. Can you explain the

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

So this is indeed what using headers with hops avoids is creating lots and lots of topics __, so you can have more complex topology setups. I ask why not support having two ways of setting up and closing the door? One based on hops using headers, and another based on topic naming. After all fle

[jira] [Created] (KAFKA-7720) kafka-configs script should also describe default broker entries

Edoardo Comar created KAFKA-7720: Summary: kafka-configs script should also describe default broker entries Key: KAFKA-7720 URL: https://issues.apache.org/jira/browse/KAFKA-7720 Project: Kafka

[jira] [Created] (KAFKA-7719) Improve fairness in SocketServer processors

Rajini Sivaram created KAFKA-7719: - Summary: Improve fairness in SocketServer processors Key: KAFKA-7719 URL: https://issues.apache.org/jira/browse/KAFKA-7719 Project: Kafka Issue Type: Impro