Re: [ANNOUNCE] New committer: Luke Chen

Congratulations Luke! Thanks, Mayuresh On Wed, Feb 9, 2022, 5:24 PM Ismael Juma wrote: > Congratulations Luke! > > On Wed, Feb 9, 2022 at 3:22 PM Guozhang Wang wrote: > > > The PMC for Apache Kafka has invited Luke Chen (showuon) as a committer > and > > we are pleased to announce that he

Re: Confluent Schema Registry Compatibility config

Hi Folks, I was reading docs on Confluent Schema Registry about Compatibility : https://docs.confluent.io/platform/current/schema-registry/avro.html#compatibility-types I was confused with "BACKWARDS" vs "BACKWARDS_TRANSITIVE". If we have 3 schemas X, X-1, X-2 and configure a schema registry

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John! Thanks, Mayuresh On Tue, Nov 12, 2019 at 4:54 PM Vahid Hashemian wrote: > Congratulations John! > > --Vahid > > On Tue, Nov 12, 2019 at 4:38 PM Adam Bellemare > wrote: > > > Congratulations John, and thanks for all your help on KIP-213! > > > > > On Nov 12, 2019, at

Re: [ANNOUNCE] New Committer: Vahid Hashemian

congrats !! On Tue, Jan 15, 2019 at 3:42 PM Matthias J. Sax wrote: > Congrats! > > On 1/15/19 3:34 PM, Boyang Chen wrote: > > This is exciting moment! Congrats Vahid! > > > > Boyang > > > > > > From: Rajini Sivaram > > Sent: Wednesday, January 16, 2019 6:50 AM

Re: [DISCUSS] : KIP-410: Add metric for request handler thread pool utilization by request type

uot; > > > connection-close-rate > Connections closed per second in the window. > > kafka.[producer|consumer|connect]:type=[producer|consumer|connect]-metrics,client-id=([-.\w]+) > > > > A separate nit - you may want to update the discussion thread link in the > KIP

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

s while group is in stable, the static map > will be cleaned up and rebalance will be triggered as expected (we are > already in dynamic mode now). > > I hope this answers your question, thanks! > > Boyang > > > From: Mayuresh Gharat &

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Regarding "However, we shall still attempt to remove the member static info if the given `member.id` points to an existing `group.instance.id` upon LeaveGroupRequest, because I could think of the possibility that in long term we could want to add static membership leave group logic for

[DISCUSS] : KIP-410: Add metric for request handler thread pool utilization by request type

I would like to get feedback on the proposal to add a metric for request handler thread pool utilization by request type. Please find the KIP here : KIP-410: Add metric for request handler thread pool utilization by request type

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

hanges does not add any additional calls to the > broker nor does it introduce significant processing logic; it just parses > the completed fetch earlier and have a list to manage them. > > > CH > > -Original Message- > From: Mayuresh Gharat > Sent: Tuesday, Dec

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

uld alternate between A and B. I was proposing to move to > the next one fetch for the next poll call based on the order in the > completed fetch queue, so the order becomes A, B, A, B, A, B. The > implementation parses the completed fetch only once. > > Thanks, CH > > -Original Messa

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

Hi ChienHsing, Thanks for the KIP. It would be great if you can explain with an example, what you mean by " Currently the implementation will return available records starting from the last partition the last poll call retrieves records from. This leads to unfair patterns of record consumption

Re: Kafka client Metadata update on demand?

Hi Ming, Kafka clients do update there metadata on NotLeaderForPartitionException. The metadata update happens asynchronously. Also if you are getting this exception for a longer time, it might mean that your client is fetching metadata from a broker whose metadata cache is not updated with the

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

+1 (non-binding) Thanks, Mayuresh On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger wrote: > +1 (non binding) > > On 12/4/18, 9:43 AM, "Patrick Williams" > wrote: > > Pls take me off this VOTE list > > Best, > > Patrick Williams > > Sales Manager, UK & Ireland, Nordics & Israel

Re: [VOTE] KIP-394: Require member.id for initial join group request

+1 (non-binding) Thanks, Mayuresh On Wed, Dec 5, 2018 at 3:59 AM Boyang Chen wrote: > Hey friends, > > I would like to start a vote for KIP-394< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-394%3A+Require+member.id+for+initial+join+group+request>. > The goal of this KIP is to

Re: [DISCUSS] KIP-394: Require member.id for initial join group request

Thanks for the KIP Boyang and great to see the progress on solving the rebalance issues with this and KIP-345. Thanks, Mayuresh On Mon, Dec 3, 2018 at 4:57 AM Stanislav Kozlovski wrote: > Everything sounds good to me. > > On Sun, Dec 2, 2018 at 1:24 PM Boyang Chen wrote: > > > In fact, it's

Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances

Hi Folks, Would it be good to move this to the DISCUSS thread and keep this thread only for voting purposes, else it will be hard to coordinate responses between 2 threads. Thanks, Mayuresh On Mon, Dec 3, 2018 at 5:43 PM Boyang Chen wrote: > Thanks Guozhang for the reply! > > 1.

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

the producer's txn ids if EOS is turned on, and > 3) the consumer's instance ids. > > So that Streams operators can read those values from KafkaStreams directly > before shutting it down and use the list in the LeaveGroupRequest API. How > about that? > > > Guozhang >

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

gt; > To: dev > > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by > > specifying member id > > > > Regarding Jason's question and Boyang's responses: > > > > 2) I once have a discussion about the LeaveGroupRequest for static > members, > > and the

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

of the new instance hasn't changed, then we can skip the > rebalance and return the current assignment without forcing a rebalance. > > The trick to making this work is in the error handling of the zombie > consumer. If the zombie simply resets its member.id and rejoins to get a > new one

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

Hi Edoardo, Thanks a lot for the KIP. I have a few questions/suggestions in addition to what Radai has mentioned above : 1. Is this meant only for 1:1 replication, for example one Kafka cluster replicating to other, instead of having multiple Kafka clusters mirroring into one Kafka

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

t? If yes, should we specify it in the KIP ? 7. Are there any specific failure scenarios when you say "other potential failure cases."? It would be good to mention them explicitly, if you think there are any. 8. It would be good to have a rollback plan as you have for roll

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

tps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Feur04.safelinks.protection.outlook.com%2F%3Furl%3Dhttp%253A%252F%252Fmax.poll.interval.ms%26data%3D02%257C01%257C%257Cb48d52bf63324bd91a5208d64f43247d%257C84df9e7fe9f640afb435%257C1%257C0%257C636783547118328245%26sdata%3DJWiS

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for updating the KIP. This is a step good direction for stateful applications and also mirroring applications whose latency is affected due to the rebalance issues that we have today. I had a few questions on the current version of the KIP : For the effectiveness of the KIP,

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

ore analysis and make a trade-off > comparison. Nice catch! > > > I hope the explanations make sense to you. I will keep polishing on the > edge cases and details. > > > Best, > > Boyang > > > From: Mayuresh Gharat > Sent: Sat

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hi Boyang, Thanks for the KIP and sorry for being late to the party. This KIP is really useful for us at Linkedin. I had a few questions : The idea of having static member name seems nice, but instead of a config, would it be possible for it to be passed in to the consumer at runtime? This is

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

Hi Lincong, Thanks for the KIP. As Colin pointed out, would it better to expose certain specific pieces of information from the request/response like api key, request headers, record counts, client ID instead of the entire request/response objects ? This enables us to change the request response

Re: Apache Kafka blog on more partitions support

Thanks Jun for sharing this. Looks nice ! Do we intend to shed light on how much time is required, on an average, for new Leader election. Also would it be good to add "if the controller waits for the LeaderAndIsrResponses before sending shutDown_OK to the shutting down broker". Thanks,

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

act > that > > as > > > already discussed - we do not really need any kind of configuration as > > this > > > data should not be thrown away at all. Submitting a PR sounds great, > > > although I feel a bit jealous you (LinkedIn) beat me to my first kafka >

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Zahari, Created the patch here : https://github.com/apache/kafka/pull/5844 Thanks, Mayuresh On Thu, Oct 25, 2018 at 4:42 PM Mayuresh Gharat wrote: > Hi Zahari, > > Oops. We had planned to put this patch upstream but somehow slipped my > mind. We were recently going over hotf

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

f configuration as this > data should not be thrown away at all. Submitting a PR sounds great, > although I feel a bit jealous you (LinkedIn) beat me to my first kafka > commit ;) Not sure how things stand with the voting process ? > > Zahari > > > > On Th

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

Hi Colin/Zahari, I have created a ticket for the similar/same feature : https://issues.apache.org/jira/browse/KAFKA-7548 We (Linkedin) had a use case in Samza at Linkedin when they moved from the SimpleConsumer to KafkaConsumer and they wanted to do this pause and resume pattern. They realized

[jira] [Created] (KAFKA-7548) KafkaConsumer should not throw away already fetched data for paused partitions.

Mayuresh Gharat created KAFKA-7548: -- Summary: KafkaConsumer should not throw away already fetched data for paused partitions. Key: KAFKA-7548 URL: https://issues.apache.org/jira/browse/KAFKA-7548

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Hi Wesley, Thanks for the KIP and sorry for being late to the party. I wanted to understand, the scenario you mentioned in Proposed changes : - > > Estimate the earliest message timestamp of an un-compacted log segment. we > only need to estimate earliest message timestamp for un-compacted log

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong !!! Thanks, Mayuresh On Mon, Aug 20, 2018 at 1:36 PM Gwen Shapira wrote: > Congrats Dong Lin! Well deserved! > > On Mon, Aug 20, 2018, 3:55 AM Ismael Juma wrote: > > > Hi everyone, > > > > Dong Lin became a committer in March 2018. Since then, he has remained > > active in the

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

hat's the case, I think this probably deserves a separate doc and > > > > design independent of this KIP. > > > > > > > > Lucas > > > > > > > > > > > > > > > > On Mon, Jul 23, 2018 at 12:39 PM, Dong Lin > > wrote: >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

>> I agree that there is no strong ordering when there are more than one > >>> socket connections. Currently, we rely on controllerEpoch and > leaderEpoch > >>> to ensure that the receiving broker picks up the latest state for each > >>> partition. >

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

ing and never reset, > hence a broker can leverage that to properly reject obsolete requests. > Thoughts? > > Thanks, > Lucas > > On Thu, Jul 19, 2018 at 12:11 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Actually nvm, correlationId is

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Actually nvm, correlationId is reset in case of connection loss, I think. Thanks, Mayuresh On Thu, Jul 19, 2018 at 11:11 AM Mayuresh Gharat wrote: > I agree with Dong that out-of-order processing can happen with having 2 > separate queues as well and it can even happen today. > C

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

at this design is tied to the coincidence > that > > > > > > we have two request priorities and there are two ends to a deque. > > > > > > Hence by using the proposed design, it seems the network layer is > > > > > > more tightly coupled with upper layer logic, e.g. if we were to > add > > > > > > an extra p

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

> >> config with a default value, say 20, guards us from issues in those > >> troublesome times, and IMO there isn't much downside of adding the extra > >> config. > >> > >> @Mayuresh > >> Good catch, this sentence is an obsolete statement based on a

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Lucas, Thanks for the KIP. I am trying to understand why you think "The memory consumption can rise given the total number of queued requests can go up to 2x" in the impact section. Normally the requests from controller to a Broker are not high volume, right ? Thanks, Mayuresh On Tue, Jul

[jira] [Created] (KAFKA-7096) Consumer should drop the data for unassigned topic partitions

Mayuresh Gharat created KAFKA-7096: -- Summary: Consumer should drop the data for unassigned topic partitions Key: KAFKA-7096 URL: https://issues.apache.org/jira/browse/KAFKA-7096 Project: Kafka

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

Sure . Thanks, Mayuresh On Wed, Sep 13, 2017 at 3:12 PM, Jun Rao wrote: > Hi, Mayuresh, > > Does this KIP obviate the need for KIP-111? If so, could you close that > one? > > Thanks, > > Jun > > On Wed, Sep 13, 2017 at 8:43 AM, Jason Gustafson > wrote:

Re: [VOTE] KIP-189 - Improve principal builder interface and add support for SASL

+1 (non-binding) Thanks, Mayuresh On Wed, Aug 30, 2017 at 8:51 AM, Jason Gustafson wrote: > I'd like to open the vote for KIP-189: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 189%3A+Improve+principal+builder+interface+and+add+support+for+SASL. > Thanks to

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

cipal` to the underlying > library through the `Extended_Plugged_In_Class` as you've suggested above. > Is that reasonable for this use case? > > Thanks, > Jason > > > On Fri, Aug 25, 2017 at 2:44 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > &g

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

t this use case. As long as the authorizer and the > principal builder both agree on how user principals are identified, then > they can still be used together. But I am explicitly leaving out support in > the ACL command tool for this use case in this KIP. This is mostly about > clarifying what is comp

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Principal would look like and its details, since it will just pass it to the third party library in Kafka Authorizer's authorize() call. Thanks, Mayuresh On Thu, Aug 24, 2017 at 10:21 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Sure. > > Thanks, > > Ma

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Sure. Thanks, Mayuresh On Wed, Aug 23, 2017 at 5:07 PM, Jun Rao wrote: > Hi, Mayuresh, > > Since this KIP covers the requirement in KIP-111, could you review it too? > > Thanks, > > Jun > > > On Tue, Aug 22, 2017 at 3:04 PM, Jason Gustafson > wrote: >

Re: Kafka Read Data from All Partition Using Key or Timestamp

Hi Senthil, Kafka does allow search message by timestamp after KIP-33 : https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index#KIP-33-Addatimebasedlogindex-Searchmessagebytimestamp The new consumer does provide you a way to get offsets by timestamp. You can use

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi Ismael, I went ahead and created a patch so that we get on same page regarding what we want to do. We can make changes if you have any comments. Thanks, Mayuresh On Mon, Apr 10, 2017 at 11:32 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Got it. > We can pro

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Ismael > > On Thu, Apr 6, 2017 at 5:50 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> > wrote: > > > Hi Ismael, > > > > Are you suggesting to use the InvalidRecordException when the key is > null? > > > > Thanks, > > > > M

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Ping Ismael. Thanks, Mayuresh On Thu, Apr 6, 2017 at 9:50 AM, Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > Hi Ismael, > > Are you suggesting to use the InvalidRecordException when the key is null? > > Thanks, > > Mayuresh > > On Thu, Apr 6,

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

uce a non retriable version of > this exception and use it in the various places where it makes sense. > > Ismael > > On Tue, Apr 4, 2017 at 12:22 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > It seems that there is n

Re: [VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Bumping up this thread. On Mon, Apr 3, 2017 at 4:22 PM, Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > Hi All, > > It seems that there is no further concern with the KIP-135. At this point > we would like to start the voting process. The KIP can be found at > htt

[VOTE] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi All, It seems that there is no further concern with the KIP-135. At this point we would like to start the voting process. The KIP can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-135+%3A+Send+of+null+key+to+a+compacted+topic+should+throw+non-retriable+error+back+to+user

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi Ismael, I have updated the KIP. Let me know if everything looks fine then I will begin voting. Thanks, Mayuresh On Wed, Mar 29, 2017 at 9:06 AM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi Ismael, > > I agree. I will change the compatibility para and start vot

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

since we want older clients to work with newer brokers. > That's why I recommended that we only throw this error if the > ProduceRequest is version 3 or higher. > > Ismael > > P.S. Note that we already send error messages back for the CreateTopics > protocol API (I adde

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

this idea in your previous reply. I > guess the question is, do we still need this KIP if we can have custom > error message specified in the exception via the other KIP? > > > On Mon, Mar 27, 2017 at 11:00 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > >

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

he client code, will increase overtime and become > harder to maintain. > > Thanks, > Dong > > > On Mon, Mar 27, 2017 at 7:20 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Dong, > > > > I had thought about this before an

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

istOffsetRequest. We don't return DuplicatedPartitionException in this > case. > > Thanks, > Dong > > > > On Wed, Mar 22, 2017 at 3:07 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > We have created KIP-135

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

tory is clearer, in my > opinion. > > Ismael > > On Wed, Mar 22, 2017 at 10:07 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Hi All, > > > > We have created KIP-135 to propose that Kafka should return a > non-retriable > > error wh

Re: [DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

u saying that when deploying this change, that you would > recommend upgrading clients before upgrading brokers? > > Thanks, > -James > > > > On Mar 22, 2017, at 3:07 PM, Mayuresh Gharat <gharatmayures...@gmail.com> > wrote: > > > > Hi All, > > >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

g to match the java.security.Principal generated from the > SSL or SASL library. How do we make sure that the same type of > java.security.Principal > can be created and will match? > > Thanks, > > Jun > > > On Wed, Mar 15, 2017 at 8:48 PM, Mayuresh Gharat < > gha

[DISCUSS] KIP-135 : Send of null key to a compacted topic should throw non-retriable error back to user

Hi All, We have created KIP-135 to propose that Kafka should return a non-retriable error when the producer produces a message with null key to a log compacted topic. Please find the KIP wiki in the link :

[jira] [Comment Edited] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935327#comment-15935327 ] Mayuresh Gharat edited comment on KAFKA-4808 at 3/21/17 9:00 PM: - [~ijuma

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935327#comment-15935327 ] Mayuresh Gharat commented on KAFKA-4808: [~ijuma] Please find the KIP here : https

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

to support other implementation of Java.security.Principal as well which seems necessary especially since Kafka provides pluggable Authorizer and PrincipalBuilder. Let me know your thoughts on this. Thanks, Mayuresh On Tue, Feb 28, 2017 at 2:33 PM, Mayuresh Gharat <gharatmayures...@gmail.

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

2 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > If we deprecate KafkaPrincipal, then the Authorizer interface will also > > need to change - i.e., deprecate the getAcls(KafkaPrincipal) method. > > > > On Tue, Feb 28, 2017 at 10:11 AM, Mayuresh Gharat < > &

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

aking clients without a deprecation period is something we only do as > a > > last resort. Is there strong justification for doing it here? > > > > Ismael > > > > On Mon, Feb 27, 2017 at 11:28 PM, Mayuresh Gharat < > > gharatmayures...@gmail.com

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

ld add a new field to `Session` with > the original principal. We can potentially deprecate the existing one, if > we're sure we don't need it (or we can leave it for now). > > Ismael > > On Mon, Feb 27, 2017 at 6:40 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote:

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886365#comment-15886365 ] Mayuresh Gharat commented on KAFKA-4808: [~ijuma] Sure we can provide a better error message if we

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

ved, we reset the ECR back to 1.0 and split > the batch. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon, Feb 27, 2017 at 10:30 AM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > > > Hi Becket, > > > > Seems like an interesting idea.

[jira] [Assigned] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat reassigned KAFKA-4808: -- Assignee: Mayuresh Gharat > send of null key to a compacted topic should throw er

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

[ https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886288#comment-15886288 ] Mayuresh Gharat commented on KAFKA-4808: [~ijuma] I was thinking if throwing "INVALID_RE

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

r <manikumar.re...@gmail.com> > wrote: > > > +1 (non-binding) > > > > On Thu, Feb 23, 2017 at 3:27 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com > > > wrote: > > > > > Hi Jun, > > > > > > Thanks a lot for th

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Hi Becket, Seems like an interesting idea. I had couple of questions : 1) How do we decide when the batch should be split? 2) What do you mean by slowly lowering the "actual" compression ratio? An example would really help here. Thanks, Mayuresh On Fri, Feb 24, 2017 at 3:17 PM, Becket Qin

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

expected user name is logged. Also, currently, we are already > creating a KafkaPrincipal on every request. +1 on the latest KIP. > > Thanks, > > Jun > > > On Tue, Feb 21, 2017 at 8:05 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

in the request log in > RequestChannel, which has the format of "principalType + SEPARATOR + > name;". > It would be good if we can keep the same convention after this KIP. One way > to do that is to convert java.security.Principal to KafkaPrincipal for > logging the requests. > &

Re: [DISCUSS] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

Apurva has a point that can be documented for this config. Overall, LGTM +1. Thanks, Mayuresh On Tue, Feb 21, 2017 at 6:41 PM, Becket Qin wrote: > Hi Apurva, > > Yes, it is true that the request size might be much smaller if the batching > is based on uncompressed size.

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi Jun, I have updated the KIP. Would you mind taking another look? Thanks, Mayuresh On Fri, Feb 17, 2017 at 4:42 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi Jun, > > Sure sounds good to me. > > Thanks, > > Mayuresh > > On Fri, Fe

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

kafka-acl.sh do authorization > based > > on > > > > that string name and not be aware of the principal builder? If you > > still > > > > think there is a need, perhaps you can add a more concrete use case > > that > > > > can't b

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

authorizer needs the full kerberos principal name, > > currently, the user can just customize "sasl.kerberos.principal.to. > > local.rules" > > to return the full principal name as the name for authorization, right? > > > > Thanks, > > > > Jun > > &

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

n Mon, Feb 13, 2017 at 1:39 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > Hi, Mayuresh, > > > > > > > > > > I seems to me that there are two common use cases of authorizer. > (1) > > > Use > > > > > the default SimpleAuthor

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Thanks, Mayuresh On Thu, Feb 9, 2017 at 11:29 AM, Jun Rao <j...@confluent.io> wrote: > Hi, Mayuresh, > > Thanks for the reply. A few more comments below. > > On Wed, Feb 8, 2017 at 9:14 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> > wrote: > >

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

s uses KafkaPrincipal." This is not ideal for existing users. > Could we avoid that? > > Thanks, > > Jun > > > On Fri, Feb 3, 2017 at 11:25 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > It seems tha

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Bumping up this thread. Thanks, Mayuresh On Fri, Feb 3, 2017 at 5:09 PM, radai <radai.rosenbl...@gmail.com> wrote: > +1 > > On Fri, Feb 3, 2017 at 11:25 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi All, > > > > It see

[VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi All, It seems that there is no further concern with the KIP-111. At this point we would like to start the voting process. The KIP can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67638388 Thanks, Mayuresh

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi All, If there are no more concerns, I will like to start vote for this KIP. Thanks, Mayuresh On Wed, Feb 1, 2017 at 8:38 PM, Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > Hi Dong, > > What I meant was "Right now Kafka just extracts the name out of the > Prin

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

lder instead of adding the method > *buildPrincipal(Map<String, > ?> principalConfigs)* in the PrincipalBuilder interface. But this is not a > blocking issue for me. > > > > > On Wed, Feb 1, 2017 at 2:54 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> > wrote: > > &g

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi All, I have updated the KIP as per our discussion here. It would be great if you can take another look and let me know if there are any concerns. Thanks, Mayuresh On Sat, Jan 28, 2017 at 6:10 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > I had offline discussions

[jira] (KAFKA-1610) Local modifications to collections generated from mapValues will be lost

[ https://issues.apache.org/jira/browse/KAFKA-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847355#comment-15847355 ] Mayuresh Gharat commented on KAFKA-1610: [~jozi-k] sure. The patch has been available but we

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

to use there own PrincipalBuilder in kafka-acls.sh. I will update the KIP accordingly. Thanks, Mayuresh On Thu, Jan 26, 2017 at 6:01 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi Dong, > > Thanks for the review. Please see the replies inline. > > > 1. I am

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

e > channelPrincipal in KafkaPrincipal class instead of the Session class? If > they work equally well to serve the use-case of this KIP, then it seems > better to put this field in the Session class to avoid changing interface > that needs to be implemented by custom principal. > > Dong >

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

> > > `This KIP is a pure addition to existing functionality and does not > > include any backward incompatible changes.` > > > > However, the KIP is proposing the addition of a method to the > > PrincipalBuilder pluggable interface, which is not a compatible change.

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi, Just wanted to see if anyone had any concerns with this KIP. I would like to put this to vote soon, if there are no concerns. Thanks, Mayuresh On Thu, Jan 12, 2017 at 11:21 AM, Mayuresh Gharat < gharatmayures...@gmail.com> wrote: > Hi Ismael, > > Fair point. I will update

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

for example. It would be > good to make this clear in the KIP. > > Ismael > > On Thu, Jan 12, 2017 at 5:44 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi all. > > > > We created KIP-111 to propose that Kafka should preserve th

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

+1 (non-binding). Thanks, Mayuresh On Wed, Jan 11, 2017 at 10:11 PM, radai wrote: > LGTM, +1 > > On Wed, Jan 11, 2017 at 1:01 PM, Dong Lin wrote: > > > Hi all, > > > > It seems that there is no further concern with the KIP-107. At this point >

[DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

Hi all. We created KIP-111 to propose that Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker. Please find the KIP wiki in the link https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67638388.

Re: [VOTE] KIP-107: Add purgeDataBefore() API in AdminClient

+1 (non-binding) Thanks, Mayuresh On Wed, Jan 11, 2017 at 1:03 PM, Dong Lin wrote: > Sorry for the duplicated email. It seems that gmail will put the voting > email in this thread if I simply replace DISCUSS with VOTE in the subject. > > On Wed, Jan 11, 2017 at 12:57 PM,

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

; Hey Mayuresh, > > > > Thanks for the comment. If the message's offset is below low_watermark, > > then it should have been deleted by log retention policy. Thus it is OK > not > > to expose this message to consumer. Does this answer your question? > > > > Thanks,

  1   2   3   4   5   6   >