答复: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Guozhang, Thanks for the vote and comments. I am not sure if I fully understand the parent metrics here. This KIP will introduce a client-level metric named 'records-lead-min' and three per-partition metrics tagged with topic Is it the child-parent relationship you mean?

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Thanks Hu Xi, I made a pass over the KIP and it lgtm. +1. Just a clarification question: for the cross-partition "records-lead-min" metric, would that be implemented as a parent metric of the per-partition metrics? Guozhang On Mon, Dec 4, 2017 at 3:07 PM, Dong Lin

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Mon, Dec 4, 2017, at 02:27, Jan Filipiak wrote: > > > On 03.12.2017 21:55, Colin McCabe wrote: > > On Sat, Dec 2, 2017, at 23:21, Becket Qin wrote: > >> Thanks for the explanation, Colin. A few more questions. > >> > >>> The session epoch is not complex. It's just a number which increments >

Re: Comparing Pulsar and Kafka: unified queuing and streaming

Thank you, Jason. Nice to see inputs from you about messaging retention. I have the same feeling that a centralized place for managing offsets (what Pulsar calls cursors) is actually much better than current kafka model. Would love to hear what is the plan for kafka community heading towards on

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

Hey Matthias, thanks for getting back to me. That's fine. But if we add it to `test` package, we don't need to talk about it in the KIP. `test` is not public API. Yes, that makes sense. It was in the KIP originally because I was, at one point, planning on including it. We can remove it now that

Re: Stream Processing Meetup@LinkedIn (Dec 4th)

Can some one post the link to the stream, please. Thanks. On Fri, Nov 17, 2017 at 3:49 PM, Becket Qin wrote: > Hi Paolo, > > Yes, we will stream the meetup. Usually the link will be posted to the > meetup website a couple of hours before the meetup. Feel free to ping me

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Mon, Dec 4, 2017, at 16:44, Jason Gustafson wrote: > Hi Colin, > > Just a few minor points/suggestions: > > 1. The last stable offset can only be advanced when the high watermark> > advances, so I think you can ignore it in your designation of a > "dirty"> partition. Hi Jason, Good catch. >

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hey Vahid, Thank you for the explanation. I am still wondering whether you have a specific use-case for user to be able to list the groups for which he/she has Describe access. I tried to think through this but couldn't find a specific use-case for this feature yet. I understand that admin of

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Thanks for the KIP, Charly. +1. The proposal looks good to me. I agree with Jun that it is better to make the metrics consistent with other metrics. That being said, arguably this is a backwards incompatible change. Since we are at 1.0, backwards incompatible changes are supposed to be in 2.0.

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Colin, Just a few minor points/suggestions: 1. The last stable offset can only be advanced when the high watermark advances, so I think you can ignore it in your designation of a "dirty" partition. 2. I think the fetch "epoch" is more properly a "sequence number" in its current usage. The

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Hi, Jiangjie, Since you proposed the original KIP-92, do you want to see if this KIP makes sense? Thanks, Jun On Wed, Nov 22, 2017 at 2:48 AM, charly molter wrote: > Hi, > > I would like to start the voting thread for KIP-225. > This KIP proposes to correct some lag

Re: [DISCUSS] KIP-222 - Add "describe consumer group" to KafkaAdminClient

Jorge, is there any update regarding this KIP? -Matthias On 11/17/17 9:14 AM, Guozhang Wang wrote: > Hello Jorge, > > I made a pass over the wiki, and here are a few comments: > > 1. First, regarding to Tom's comment #2 above, I think if we are only going > to include the String groupId.

Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

Jeyhun, is there any update regarding this KIP? -Matthias On 10/13/17 10:39 AM, Matthias J. Sax wrote: > I wanted to follow up with this KIP. > > One important thing that is missing in this KIP, is the question where > the TS is coming from? ATM, timestamps are not stored in the state >

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Vito, is there any update with regard to this KIP? -Matthias On 11/5/17 6:11 PM, vito jeng wrote: > Thanks, Guozhang and Matthias. Your comments very useful for me. > > I'll update KIP and keep going on. > > > > --- > Vito > > On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

I agree with Guozhang that just exposing meta data at the source level might not provide too much value. Furthermore, for timestamps we do already have a well defined contract and we should exploit it: timestamps can always be provided in a meaningful way. Also, for simple operations like

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

I saw your reply in the other email. I meant that permissions the person managing brokers gives the user should be known a priori. Otherwise there would be security hole somewhere. Cheers On Mon, Dec 4, 2017 at 3:47 PM, Vahid S Hashemian wrote: > Hi Ted, > > Thanks

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hi Ted, Thanks for the feedback. I tried to address Dong's comments on an earlier email. I hope that addresses your concern too. If not, please continue with the discussion until we hopefully agree on what (if anything) needs to be done. I'm also not sure if I understand your point about "The

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hi Dong, Thanks for reviewing the KIP and providing your feedback. As you mentioned, the KIP suggests a modification to the semantics of ListGroupResponse. The main reason for submitting the KIP and suggesting a modification is that there are issues with the current one (further explained in

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Jan, The KTableValueGetter thing is a valid point. I think we would need a backwards mapper (or merge both into one and sacrifices lambdas?). Another alternative would be, to drop the optimization and materialize the KTable.operator() result... (not a great solution either). I am personally fine

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

for point 1 i agree, its not too strong. only addition i could come up with is that it allows any utility to have better forwards compatability. a cli written that can inspect how a topic will be created would be able to give insight/expectations about configs that didn't exist at compilation

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

I don't have an interesting rejected alternative solution to put in the KIP. If there is good alternative solution from anyone in this thread, I am happy to discuss this and update the KIP accordingly. Thanks, Dong On Mon, Dec 4, 2017 at 1:12 PM, Ted Yu wrote: > It is

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

+1 (non-binding) On Wed, Nov 29, 2017 at 7:05 PM, Hu Xi wrote: > Hi all, > > As I didn't see any further discussion around this KIP, I'd like to start > voting. > > KIP documentation: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: [DISCUSS]: KIP-230: Name Windowing Joins

Matthias, thanks for the KIP. Can you please c the corresponding content instead of just putting links? A KIP should be a self-contained Wiki page. Also, if we add a optional config parameter, how would we specify it? Please list all changes to want to apply to `Joined` class. Furthermore,

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks again for the update:) I am not sure I fully understand the points (1) and (2) in the "Always Configure ALL Configs For a Topic". In my previous question, I don't mean that users should specify full list of topics configs. I mean that user can specify the full list of topic

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

Hey, About your questions: >>> Acknowledged, so is ProducerFencedException the only kind of exception I >>> need to change my behavior on? Or are there other types I need to check? Is >>> there a comprehensive list somewhere? I cannot think if any other atm. We should list all fatal exceptions

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Hi, Hu, Thanks for the KIP. +1 from me. Jun On Wed, Nov 29, 2017 at 7:05 PM, Hu Xi wrote: > Hi all, > > As I didn't see any further discussion around this KIP, I'd like to start > voting. > > KIP documentation: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Hi, Charly, Thanks for the proposal. Since the per partition level metric is likely not widely used, it's probably better to fix the metric name and make it more consistent. +1 from me. We probably want to document this change in the upgrade section. Jun On Wed, Nov 29, 2017 at 2:45 AM,

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

updated again :) by having users always set all configs you lose the ability to use the broker defaults as intended, since topic configs are overlaid. example in the kip doc. dan On Mon, Dec 4, 2017 at 11:47 AM, Dong Lin wrote: > Hey Dan, > > Thanks for the update. I just

Re: custom ProducerRecord timestamp and compact+delete cleanup

Both use the same timestamp that is record meta data timestamp. KIP-32 behavior was changed later on. -Matthias On 11/27/17 8:00 AM, Michal Hariš wrote: > Hi, > > I've read on the message timestamp(KIP-32) that for time-rolling deletes > use LogAppendTime, but this only applies to delete

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Jeyhun, thanks for updating the KIP. I am wondering if you intend to add a new class `Produced`? There is already `org.apache.kafka.streams.kstream.Produced`. So if we want to add a new class, it must have a different name -- or we might be able to merge both into one? Also, for the KStream

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

I agree with Dong on maintaining the semantics. The user should know which group(s) he / she is allowed to describe. Cheers On Mon, Dec 4, 2017 at 1:40 PM, Dong Lin wrote: > Hey Vahid, > > Thanks for the KIP. If I understand the you correctly, you want client to > be able

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Panuwat, Thanks a lot for the KIP! Just one nit: `does not follow provide a good` -> spelling: remove `follow` ? Otherwise, looks good to me. -Matthias On 12/4/17 10:49 AM, Ted Yu wrote: > Looks like you're implying logic similar to this: > > public synchronized GlobalKTable

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

I forgot another question. Can you provide a use-case where a user wants to list all groups for which he/she has the Describe access? Thanks, Dong On Mon, Dec 4, 2017 at 1:40 PM, Dong Lin wrote: > Hey Vahid, > > Thanks for the KIP. If I understand the you correctly, you

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hey Vahid, Thanks for the KIP. If I understand the you correctly, you want client to be able to list all the groups for which it currently has the describe access. As of now the ListGroupRequest does not allow user to specify the group. If user does not have the Describe Cluster access,

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

It is clearer now. I noticed that Rejected Alternatives section is empty. Have you considered any alternative ? Cheers On Mon, Dec 4, 2017 at 1:07 PM, Dong Lin wrote: > Ted, thanks for catching this. I have updated the sentence to make it > readable. > > Thanks, > Dong >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Ted, thanks for catching this. I have updated the sentence to make it readable. Thanks, Dong On Sat, Dec 2, 2017 at 3:05 PM, Ted Yu wrote: > bq. It the controller_epoch of the incoming MetadataResponse, or if the > controller_epoch is the same but the

Jenkins build is back to normal : kafka-trunk-jdk7 #3018

See

Jenkins build is back to normal : kafka-trunk-jdk8 #2257

See

Build failed in Jenkins: kafka-trunk-jdk9 #237

See Changes: [wangguoz] MINOR: increase request timeout for streams bounce test [wangguoz] KAFKA-6150: KIP-204 part III; Purge repartition topics with the admin --

Re: Comparing Pulsar and Kafka: unified queuing and streaming

Hi Khurrum, Thanks for sharing the article. I think one interesting aspect of Pulsar that stands out to me is its notion of a subscription and how it impacts message retention. In Kafka, consumers are more loosely coupled and retention is enforced independently of consumption. There are some

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks for the update. I just want to push the discussion a bit further. Another alternative, which currently is not described in the KIP, is for user to always create the topic with the full list of configs it may want to override. Can you help me understand what is the drawback of this

Build failed in Jenkins: kafka-trunk-jdk7 #3017

See Changes: [wangguoz] MINOR: increase request timeout for streams bounce test -- [...truncated 1.86 MB...]

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Dong, i added a section on current state and workarounds along with my arguments for why they are less than optimal to the wiki. but the jist of it is you can end up with messages in your topic in an incorrect/invalid state if you do this. thanks, dan On Mon, Dec 4, 2017 at 10:53 AM, Dong Lin

[jira] [Resolved] (KAFKA-2188) JBOD Support

[ https://issues.apache.org/jira/browse/KAFKA-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2188. Resolution: Duplicate KIP-112 and KIP-113 replaced this one. > JBOD Support > > >

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, Thanks for the KIP. Can you help me understand the motivation by providing a use-case that can not be easily completed without this KIP? It seems that most users will simply create the topic without worrying about the default configs. If a user has specific requirement for the default

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Looks like you're implying logic similar to this: public synchronized GlobalKTable globalTable(final String topic, final Consumed consumed) { StreamsBuilder is returned instead of GlobalKTable. Can you add

[GitHub] kafka pull request #4270: KAFKA-6150: Purge repartition topics

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4270 ---

[jira] [Resolved] (KAFKA-6150) Make Repartition Topics Transient

[ https://issues.apache.org/jira/browse/KAFKA-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6150. -- Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request 4270

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

updated to be more explicit On Mon, Dec 4, 2017 at 9:55 AM, Ted Yu wrote: > In the example, resources is derived from ConfigResource. > It would be clearer if you show the derivation. > > Cheers > > On Mon, Dec 4, 2017 at 9:51 AM, dan wrote: > > > I

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

What i am thinking right now is using the same approach as org.apache.kafka.streams.kstream.internals.InternalStreamsBuilder#globalTable On Mon, 4 Dec 2560 at 23:10 Ted Yu wrote: > Can you describe how sourceName is inferred based on the new API ? > > Please fill out JIRA

[GitHub] kafka pull request #4291: MINOR: increase request timeout for streams bounce...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4291 ---

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

In the example, resources is derived from ConfigResource. It would be clearer if you show the derivation. Cheers On Mon, Dec 4, 2017 at 9:51 AM, dan wrote: > I added sample usage as well as a WIP commit to the KIP. > > dan > > On Mon, Dec 4, 2017 at 9:36 AM, Ted Yu

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

I added sample usage as well as a WIP commit to the KIP. dan On Mon, Dec 4, 2017 at 9:36 AM, Ted Yu wrote: > Can you add some code snippet showing how ConfigResource delivers topic > defaults with the proposed change ? > > > Cheers > > On Mon, Dec 4, 2017 at 9:25 AM, dan

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Can you add some code snippet showing how ConfigResource delivers topic defaults with the proposed change ? Cheers On Mon, Dec 4, 2017 at 9:25 AM, dan wrote: > I would like to start a discussion about KIP-234 > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

[DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

I would like to start a discussion about KIP-234 https://cwiki.apache.org/confluence/display/KAFKA/KIP-234%3A+add+support+for+getting+topic+defaults+from+AdminClient thanks dan

[jira] [Created] (KAFKA-6309) add support for getting topic defaults from AdminClient

dan norwood created KAFKA-6309: -- Summary: add support for getting topic defaults from AdminClient Key: KAFKA-6309 URL: https://issues.apache.org/jira/browse/KAFKA-6309 Project: Kafka Issue

[jira] [Created] (KAFKA-6308) Connect: Struct#equals should use Arrays#deepEquals

Tobias Gies created KAFKA-6308: -- Summary: Connect: Struct#equals should use Arrays#deepEquals Key: KAFKA-6308 URL: https://issues.apache.org/jira/browse/KAFKA-6308 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Can you describe how sourceName is inferred based on the new API ? Please fill out JIRA number. BTW here is the URL for the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-233%3A+Simplify+StreamsBuilder%23addGlobalStore On Mon, Dec 4, 2017 at 7:39 AM, Panuwat Anawatmongkhon <

[DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Hi all, I created a KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP233%3A+Simplify+StreamsBuilder%23addGlobalStore Cheers, Benz

[jira] [Created] (KAFKA-6307) mBeanName should be removed before returning from JmxReporter#removeAttribute()

Ted Yu created KAFKA-6307: - Summary: mBeanName should be removed before returning from JmxReporter#removeAttribute() Key: KAFKA-6307 URL: https://issues.apache.org/jira/browse/KAFKA-6307 Project: Kafka

[GitHub] kafka pull request #4291: Minor increase request timeout for streams bounce ...

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/4291 Minor increase request timeout for streams bounce test Increase `REQUEST_TIMOUT_MS` to improve a flaky test until KIP-91 merged ### Committer Checklist (excluded from commit message)

Re: [VOTE] KIP-203: Add toLowerCase support to sasl.kerberos.principal.to.local rule

This vote has passed with 3 binding votes (Rajini, Jason, Guozhang) and one non-binding vote (Tom). Thanks you all for your participation. Thanks, Manikumar On Sat, Dec 2, 2017 at 4:07 AM, Guozhang Wang wrote: > Made a pass over the KIP. +1. > > Thanks! > > > Guozhang > >

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Tom, Thanks for the review. updated the motivation a little bit, it's better, but I have to admit can be improved. I made addWaiters public. Enjoy, Steven Op ma 4 dec. 2017 om 11:01 schreef Tom Bentley : > Hi Steven, > > Thanks for updating the KIP. I have a couple of

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On 03.12.2017 21:55, Colin McCabe wrote: On Sat, Dec 2, 2017, at 23:21, Becket Qin wrote: Thanks for the explanation, Colin. A few more questions. The session epoch is not complex. It's just a number which increments on each incremental fetch. The session epoch is also useful for

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steven, Thanks for updating the KIP. I have a couple of points: 1. Typo in the first sentence of the Motivation. Also what does "empty public abstract classes with one abstract method" mean -- if it's got one abstract method in what way is it empty? 2.From an entirely self-centred point of

[jira] [Created] (KAFKA-6306) Auto-commit of offsets fail, and not recover forever...

HongLiang created KAFKA-6306: Summary: Auto-commit of offsets fail, and not recover forever... Key: KAFKA-6306 URL: https://issues.apache.org/jira/browse/KAFKA-6306 Project: Kafka Issue Type:

[GitHub] kafka pull request #4290: Auto-commit of offsets fail, and not recover forev...

GitHub user leonhong opened a pull request: https://github.com/apache/kafka/pull/4290 Auto-commit of offsets fail, and not recover forever... * Auto-commit of offsets fail, and not recover **forever**. at sendOffsetCommitRequest, while "generation equal NULL", ConsumerCoordinator