Re: [DISCUSS] KIP-847: Add ProducerCount metrics

[Luke Chen]

Hi Artem,

Thanks for the update.
LGTM.

Luke

On Thu, Jun 30, 2022 at 6:51 AM Artem Livshits
 wrote:

> Thank you for your feedback. I've updated the KIP to elaborate on the
> motivation and provide some background on producer ids and how we measure
> them.
>
> Also, after some thinking and discussing it offline with some folks, I
> think that we don't really need partitioner level metrics.  We can use
> existing tools to do granular debugging.  I've moved partition level
> metrics to the rejected alternatives section.
>
> -Artem
>
> On Wed, Jun 29, 2022 at 1:57 AM Luke Chen  wrote:
>
> > Hi Artem,
> >
> > Could you elaborate more in the motivation section?
> > I'm interested to know what kind of scenarios this metric can benefit
> for.
> > What could it bring to us when a topic partition has 100 ProducerIdCount
> VS
> > another topic partition has 10 ProducerIdCount?
> >
> > Thank you.
> > Luke
> >
> > On Wed, Jun 29, 2022 at 6:30 AM Jun Rao 
> wrote:
> >
> > > Hi, Artem,
> > >
> > > Thanks for the KIP.
> > >
> > > Could you explain the partition level ProducerIdCount a bit more? Does
> > that
> > > reflect the number of PIDs ever produced to a partition since the
> broker
> > is
> > > started? Do we reduce the count after a PID expires?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jun 22, 2022 at 1:08 AM David Jacot
>  > >
> > > wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > The KIP LGTM.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits
> > > >  wrote:
> > > > >
> > > > > If there is no other feedback I'm going to start voting in a couple
> > > days.
> > > > >
> > > > > -Artem
> > > > >
> > > > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits <
> > alivsh...@confluent.io
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thank you for your feedback.  Updated the KIP and added the
> > Rejected
> > > > > > Alternatives section.
> > > > > >
> > > > > > -Artem
> > > > > >
> > > > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > >> If we don't track them separately, then it makes sense to keep
> it
> > as
> > > > one
> > > > > >> metric. I'd probably name it ProducerIdCount in that case.
> > > > > >>
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits
> > > > > >>  wrote:
> > > > > >>
> > > > > >> > Do you propose to have 2 metrics instead of one?  Right now we
> > > don't
> > > > > >> track
> > > > > >> > if the producer id was transactional or idempotent and for
> > metric
> > > > > >> > collection we'd either have to pay the cost of iterating over
> > > > producer
> > > > > >> ids
> > > > > >> > (which could be a lot) or split the producer map into 2 or
> cache
> > > the
> > > > > >> > counts, which complicates the code.
> > > > > >> >
> > > > > >> > From the monitoring perspective, I think one metric should be
> > > good,
> > > > but
> > > > > >> > maybe I'm missing some scenarios.
> > > > > >> >
> > > > > >> > -Artem
> > > > > >> >
> > > > > >> > On Fri, Jun 17, 2022 at 12:28 AM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > > wrote:
> > > > > >> >
> > > > > >> > > I like the suggestion to have IdempotentProducerCount and
> > > > > >> > > TransactionalProducerCount metrics.
> > > > > >> > >
> > > > > >> > > Ismael
> > > > > >> > >
> > > > > >> > > On Thu, Jun 16, 2022 at 2:27 PM Artem Livshits
> > > > > >> > >  wrote:
> > > > > >> > >
> > > > > >> > > > Hi Ismael,
> > > > > >> > > >
> > > > > >> > > > Thank you for your feedback.  Yes, this is counting the
> > number
> > > > of
> > > > > >> > > producer
> > > > > >> > > > ids tracked by the partition and broker.  Another options
> I
> > > was
> > > > > >> > thinking
> > > > > >> > > of
> > > > > >> > > > are the following:
> > > > > >> > > >
> > > > > >> > > > - IdempotentProducerCount
> > > > > >> > > > - TransactionalProducerCount
> > > > > >> > > > - ProducerIdCount
> > > > > >> > > >
> > > > > >> > > > Let me know if one of these seems better, or I'm open to
> > other
> > > > name
> > > > > >> > > > suggestions as well.
> > > > > >> > > >
> > > > > >> > > > -Artem
> > > > > >> > > >
> > > > > >> > > > On Wed, Jun 15, 2022 at 11:49 PM Ismael Juma <
> > > ism...@juma.me.uk
> > > > >
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > > Thanks for the KIP.
> > > > > >> > > > >
> > > > > >> > > > > ProducerCount seems like a misleading name since
> producers
> > > > > >> without a
> > > > > >> > > > > producer id are not counted. Is this meant to count the
> > > > number of
> > > > > >> > > > producer
> > > > > >> > > > > IDs tracked by the broker?
> > > > > >> > > > >
> > > > > >> > > > > Ismael
> > > > > >> > > > >
> > > > > >> > > > > On Wed, Jun 15, 2022, 3:12 PM Artem Livshits <
> > > > > >> alivsh...@confluent.io
> > > > > >> > > > > .invalid>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hello,
> > > > > >> > > > > >
> > > > > >> > > > > > I'd like to start a discussion on the 

Re: [DISCUSS] KIP-847: Add ProducerCount metrics

[Artem Livshits]

Thank you for your feedback. I've updated the KIP to elaborate on the
motivation and provide some background on producer ids and how we measure
them.

Also, after some thinking and discussing it offline with some folks, I
think that we don't really need partitioner level metrics.  We can use
existing tools to do granular debugging.  I've moved partition level
metrics to the rejected alternatives section.

-Artem

On Wed, Jun 29, 2022 at 1:57 AM Luke Chen  wrote:

> Hi Artem,
>
> Could you elaborate more in the motivation section?
> I'm interested to know what kind of scenarios this metric can benefit for.
> What could it bring to us when a topic partition has 100 ProducerIdCount VS
> another topic partition has 10 ProducerIdCount?
>
> Thank you.
> Luke
>
> On Wed, Jun 29, 2022 at 6:30 AM Jun Rao  wrote:
>
> > Hi, Artem,
> >
> > Thanks for the KIP.
> >
> > Could you explain the partition level ProducerIdCount a bit more? Does
> that
> > reflect the number of PIDs ever produced to a partition since the broker
> is
> > started? Do we reduce the count after a PID expires?
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jun 22, 2022 at 1:08 AM David Jacot  >
> > wrote:
> >
> > > Hi Artem,
> > >
> > > The KIP LGTM.
> > >
> > > Thanks,
> > > David
> > >
> > > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits
> > >  wrote:
> > > >
> > > > If there is no other feedback I'm going to start voting in a couple
> > days.
> > > >
> > > > -Artem
> > > >
> > > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits <
> alivsh...@confluent.io
> > >
> > > > wrote:
> > > >
> > > > > Thank you for your feedback.  Updated the KIP and added the
> Rejected
> > > > > Alternatives section.
> > > > >
> > > > > -Artem
> > > > >
> > > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma 
> > wrote:
> > > > >
> > > > >> If we don't track them separately, then it makes sense to keep it
> as
> > > one
> > > > >> metric. I'd probably name it ProducerIdCount in that case.
> > > > >>
> > > > >> Ismael
> > > > >>
> > > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits
> > > > >>  wrote:
> > > > >>
> > > > >> > Do you propose to have 2 metrics instead of one?  Right now we
> > don't
> > > > >> track
> > > > >> > if the producer id was transactional or idempotent and for
> metric
> > > > >> > collection we'd either have to pay the cost of iterating over
> > > producer
> > > > >> ids
> > > > >> > (which could be a lot) or split the producer map into 2 or cache
> > the
> > > > >> > counts, which complicates the code.
> > > > >> >
> > > > >> > From the monitoring perspective, I think one metric should be
> > good,
> > > but
> > > > >> > maybe I'm missing some scenarios.
> > > > >> >
> > > > >> > -Artem
> > > > >> >
> > > > >> > On Fri, Jun 17, 2022 at 12:28 AM Ismael Juma  >
> > > wrote:
> > > > >> >
> > > > >> > > I like the suggestion to have IdempotentProducerCount and
> > > > >> > > TransactionalProducerCount metrics.
> > > > >> > >
> > > > >> > > Ismael
> > > > >> > >
> > > > >> > > On Thu, Jun 16, 2022 at 2:27 PM Artem Livshits
> > > > >> > >  wrote:
> > > > >> > >
> > > > >> > > > Hi Ismael,
> > > > >> > > >
> > > > >> > > > Thank you for your feedback.  Yes, this is counting the
> number
> > > of
> > > > >> > > producer
> > > > >> > > > ids tracked by the partition and broker.  Another options I
> > was
> > > > >> > thinking
> > > > >> > > of
> > > > >> > > > are the following:
> > > > >> > > >
> > > > >> > > > - IdempotentProducerCount
> > > > >> > > > - TransactionalProducerCount
> > > > >> > > > - ProducerIdCount
> > > > >> > > >
> > > > >> > > > Let me know if one of these seems better, or I'm open to
> other
> > > name
> > > > >> > > > suggestions as well.
> > > > >> > > >
> > > > >> > > > -Artem
> > > > >> > > >
> > > > >> > > > On Wed, Jun 15, 2022 at 11:49 PM Ismael Juma <
> > ism...@juma.me.uk
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Thanks for the KIP.
> > > > >> > > > >
> > > > >> > > > > ProducerCount seems like a misleading name since producers
> > > > >> without a
> > > > >> > > > > producer id are not counted. Is this meant to count the
> > > number of
> > > > >> > > > producer
> > > > >> > > > > IDs tracked by the broker?
> > > > >> > > > >
> > > > >> > > > > Ismael
> > > > >> > > > >
> > > > >> > > > > On Wed, Jun 15, 2022, 3:12 PM Artem Livshits <
> > > > >> alivsh...@confluent.io
> > > > >> > > > > .invalid>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hello,
> > > > >> > > > > >
> > > > >> > > > > > I'd like to start a discussion on the KIP-847:
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-847%3A+Add+ProducerCount+metrics
> > > > >> > > > > > .
> > > > >> > > > > >
> > > > >> > > > > > -Artem
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > >
> >
>

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[Matthew Howlett]

My first reaction also is that the proposed configuration is surely too
complicated.

It seems like an ideal solution from a usability perspective (always a good
place to start) would be if the consumer just automatically behaved in this
way. To make that work:
1. auto.offset.reset=latest would need to behave like
auto.offset.reset=earliest in the case where a consumer is in a group, and
is assigned a newly created partition. This might seem a bit too "magic",
but from the perspective of the group, I think it makes conceptual sense
and people wouldn't find it surprising. Also, I don't think anyone would be
relying on the old behavior.
2. The group would need to detect the newly created partitions and
rebalance pretty quickly (this is not the case currently). The longer the
delay, the more tenuous the idea of changing the auto.offset.reset behavior
in this special circumstance.

I have a feeling this approach has implementation challenges (haven't
thought deeply), just throwing it out there.


On Wed, Jun 29, 2022 at 4:57 AM David Jacot  wrote:

> Thanks for the KIP.
>
> I read it and I am also worried by the complexity of the new
> configurations. They are not easy to grasp. I need to digest it a bit more,
> I think.
>
> Best,
> David
>
> Le mer. 29 juin 2022 à 02:25, Matthias J. Sax  a écrit :
>
> > Thanks for the KIP.
> >
> > I don't think I fully digested the proposal yet, but my first reaction
> > is: this is quite complicated. Frankly, I am worried about complexity
> > and usability.
> >
> > Especially the option `safe_latest` is a "weird" one IMHO, and `nearest`
> > is even more complex.
> >
> > The problem at hand (as I understand it from the Jira) is a real one,
> > but I am wondering if it would be something that should be addressed by
> > the application? If you pass in strategy `none`, and a new partition is
> > added, you can react to it by custom code. For regular startup you can
> > still go with "latest" to avoid reprocessing the history.
> >
> > Adding "latest/earliest_on_start" seems useful, as it seems to also
> > address https://issues.apache.org/jira/browse/KAFKA-3370
> >
> >
> > -Matthias
> >
> >
> > On 6/7/22 12:55 AM, hudeqi wrote:
> > > I think so too, what about Guozhang Wang and Luke Chen? Can I initiate
> a
> > voting process?
> > >
> > > Best,
> > > hudeqi
> > >
> > >  -原始邮件-
> > >  发件人: "邓子明" 
> > >  发送时间: 2022-06-07 10:23:37 (星期二)
> > >  收件人: dev@kafka.apache.org
> > >  抄送:
> > >  主题: Re: [DISCUSS] KIP-842: Add richer group offset reset
> mechanisms
> > > 
> > > 
> >
>

Re: [VOTE] KIP-825: introduce a new API to control when aggregated results are produced

[Hao Li]

Hi Dev Team,

Just want to inform you here that we made an update to this KIP. To
implement emit final in Session Window aggregation, we augment the
`SessionStore` by adding an
public API to range query from earliest session end time to latest session
end time. API is like this:

```
public interface SessionStore extends StateStore,
ReadOnlySessionStore {

/**
 * Return all the session window entries that ends between the
specified range (both ends are inclusive).
 * This function would be used to retrieve all closed and immutable
windows.
 *
 * @param earliestSessionEndTime earliest session end time to search
from, inclusive
 * @param latestSessionEndTime latest session end time to search to,
inclusive
 */
default KeyValueIterator, AGG> findSessions(final long
earliestSessionEndTime,
final long
latestSessionEndTime)
{
throw new UnsupportedOperationException(
"This API is not supported by this implementation of
SessionStore.");
}

// other existing functions
}
```

You can take a look at the KIP for more information. Thanks Guozhang for
adding it to the KIP.

Thanks,
Hao

On Thu, Mar 24, 2022 at 4:17 PM Hao Li  wrote:

> The vote passed with 5 binding votes from John, Guozhang, Bruno, Matthias
> and Bill.
>
> Thanks all for the feedback and vote.
>
> Hao
>
> On Thu, Mar 24, 2022 at 2:20 PM Bill Bejeck  wrote:
>
>> Thanks for KIP Hao!
>>
>> Glad to see we settled on option 1
>>
>> +1(binding)
>>
>> On Thu, Mar 24, 2022 at 5:13 PM Matthias J. Sax  wrote:
>>
>> > +1 (binding)
>> >
>> >
>> > On 3/24/22 1:52 PM, Hao Li wrote:
>> > > I hit reply on my phone in the mail app and changed the title and text
>> > > hoping it will start a new thread. Apparently it doesn't work...
>> > >
>> > > On Thu, Mar 24, 2022 at 12:36 PM Bruno Cadonna 
>> > wrote:
>> > >
>> > >> Hi Hao,
>> > >>
>> > >> Actually, this is the VOTE thread. Do you use GMail? Sometimes it is
>> a
>> > >> bit weird how it shows e-mails in threads.
>> > >>
>> > >> Anyways, thank you for the KIP and your patience!
>> > >>
>> > >> +1 (binding)
>> > >>
>> > >> Best,
>> > >> Bruno
>> > >>
>> > >> On 24.03.22 17:36, Hao Li wrote:
>> > >>> Thanks John and Guozhang. Didn't realize I used this discussion
>> thread
>> > >> for
>> > >>> voting. Let me start a new thread for voting. Will fix the KIP.
>> > >>>
>> > >>> On Thu, Mar 24, 2022 at 9:28 AM Guozhang Wang 
>> > >> wrote:
>> > >>>
>> >  +1 (binding).
>> > 
>> >  Thanks Hao!
>> > 
>> > 
>> >  Guozhang
>> > 
>> >  On Thu, Mar 24, 2022 at 9:20 AM John Roesler 
>> > >> wrote:
>> > 
>> > > Thanks, Hao!
>> > >
>> > > I'm +1 (binding)
>> > >
>> > > -John
>> > >
>> > > On Wed, 2022-03-23 at 22:25 -0700, Hao Li wrote:
>> > >> Hi all,
>> > >>
>> > >> I'd like to start a vote on Kafka Streams KIP-825:
>> > >>
>> > >
>> > 
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-825%3A+introduce+a+new+API+to+control+when+aggregated+results+are+produced
>> > >
>> > >
>> > 
>> >  --
>> >  -- Guozhang
>> > 
>> > >>>
>> > >>>
>> > >>
>> > >
>> > >
>> >
>>
>
>
> --
> Thanks,
> Hao
>


-- 
Thanks,
Hao

Potential bug for forwarded request time cost metrics

[Feiyan Yu]

Howdy!

It seems like `requestDequeueTimeNanos` is ignored when handling an
envelope request, which could lead to `requestDequeueTimeNanos` ending with
"-1", so that the metric,`LocalTimeMs` is not correct.

I created an issue related to this together with a tiny PR.
issue: https://issues.apache.org/jira/browse/KAFKA-14032
PR: https://github.com/apache/kafka/pull/12360


I'd appreciate it if someone is interested in it.

Feiyan Yu

[GitHub] [kafka-site] bbejeck commented on pull request #406: Add Qudosoft to powered-by

[GitBox]

bbejeck commented on PR #406:
URL: https://github.com/apache/kafka-site/pull/406#issuecomment-1170082018

   Hi @StephanZimmermann if you can rebase this PR we'll get this merged.  
Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #320: Update powered-by.html used by percentage

[GitBox]

bbejeck commented on PR #320:
URL: https://github.com/apache/kafka-site/pull/320#issuecomment-1170078043

   Thanks @JaviOverflow for the contribution!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #320: Update powered-by.html used by percentage

[GitBox]

bbejeck commented on PR #320:
URL: https://github.com/apache/kafka-site/pull/320#issuecomment-117005

   merged #320 into asf-site


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck merged pull request #320: Update powered-by.html used by percentage

[GitBox]

bbejeck merged PR #320:
URL: https://github.com/apache/kafka-site/pull/320


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck commented on pull request #413: Add Covage to powered-by

[GitBox]

bbejeck commented on PR #413:
URL: https://github.com/apache/kafka-site/pull/413#issuecomment-1170063176

   merged #413 into asf-site


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] bbejeck merged pull request #413: Add Covage to powered-by

[GitBox]

bbejeck merged PR #413:
URL: https://github.com/apache/kafka-site/pull/413


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Jorge Esteban Quilcate Otoya]

Thanks Chris! I have updated the KIP to include this fix.

I will keep the array as a potential improvement at the moment, and out of
the scope of this KIP.

Thanks,
Jorge.

On Tue, 28 Jun 2022 at 23:19, Chris Egerton  wrote:

> Hi Jorge,
>
> Apologies for the long delay, had my own KIP-related work to focus on.
>
> I think it's fine to include array accesses but it's not a blocker. I'm +1
> either way. On that front though, I think the MaskField section might need
> to be updated as it still mentions arrays and deep-scan?
>
> Cheers,
>
> Chris
>
> On Tue, Jun 28, 2022 at 4:38 PM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi there,
> >
> > I have update the KIP to the previous state voted, including the
> > configuration change from `field.style` to `field.syntax.version`.
> >
> > I'll bump the vote thread as well to see if there's agreement on adding
> > this feature to Connect.
> >
> > Cheers,
> > Jorge.
> >
> > On Wed, 15 Jun 2022 at 23:02, Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > Thanks, Chris. Your feedback is much appreciated!
> > >
> > > I see how the current proposal might be underestimating some edge
> cases.
> > > I'm happy to move the design for deep-scan and multi-values to future
> > > developments related with this KIP and reduce its scope, though open
> for
> > > more feedback.
> > >
> > > Also, just to be sure, are you proposing also to not include array
> access
> > > at this stage?
> > >
> > > Thanks,
> > > Jorge.
> > >
> > > On Tue, 14 Jun 2022 at 03:20, Chris Egerton 
> > > wrote:
> > >
> > >> Hi Jorge,
> > >>
> > >> I've done some more thinking and I hate to say it, but I think the
> > syntax
> > >> does need to be expanded. Right now it's clear what "a.b" refers to
> and
> > >> what "a..b" refers to, but what about "a...b"? Is that referring to
> > >> subfield ".b" of field "a", or subfield "b" of field "a."? This gets
> > even
> > >> more complicated when thinking about fields whose names are
> exclusively
> > >> made up of dots.
> > >>
> > >> I'm also a little hesitant to mix the cases of multi-value paths and
> > deep
> > >> scans. What if you only want to access one subfield deep for an SMT,
> > >> instead of recursing through all the children of a given field? It's
> > akin
> > >> to the distinction between * and ** with file globbing patterns, and
> > there
> > >> could be a substantial performance difference if you have
> heavily-nested
> > >> fields.
> > >>
> > >> Ultimately, I think that if the proposed "field.syntax.version"
> property
> > >> sits well with people, it might be better to reduce the scope of the
> KIP
> > >> back to the original proposal and just focus on adding support for
> > >> explicitly-specified nested values, with no multi-value paths
> > whatsoever,
> > >> knowing that we have an easy way to introduce new syntax and features
> in
> > >> the future. (We could probably leave the "a...b" case for that next
> > >> version
> > >> too.)
> > >>
> > >> I was a huge fan of this KIP before we started trying to address more
> > >> complex use cases, and although I don't want to write those off, I
> think
> > >> we
> > >> may have bitten off more than we can chew in time for the 3.3.0
> release
> > >> and
> > >> would hate to see this KIP get delayed as a result.
> > >>
> > >> I'd be really curious to hear from Joshua and Tom on this front,
> though.
> > >> Is
> > >> it acceptable to move more incrementally here and settle on the syntax
> > >> version property as our means of introducing new features, or is it
> > >> preferable to implement things monolithically and try to get
> everything
> > >> (or
> > >> at least, as much as possible) right the first time?
> > >>
> > >> Thanks again for your continued effort on this KIP!
> > >>
> > >> Cheers,
> > >>
> > >> Chris
> > >>
> > >> On Wed, Jun 8, 2022 at 5:41 PM Jorge Esteban Quilcate Otoya <
> > >> quilcate.jo...@gmail.com> wrote:
> > >>
> > >> > Thanks, Chris!
> > >> >
> > >> > Please, find my comments below:
> > >> >
> > >> > On Tue, 7 Jun 2022 at 04:39, Chris Egerton  >
> > >> > wrote:
> > >> >
> > >> > > Hi Jorge,
> > >> > >
> > >> > > Thanks! Sorry for the delay; here are my thoughts:
> > >> > >
> > >> > > 1. Under the "Accessing multiple values by deep-scan" header it's
> > >> stated
> > >> > > that "If deep-scan is used, it must have only one field after the
> > >> > asterisk
> > >> > > level.". However, in example 3 for the Cast SMT and other examples
> > for
> > >> > > other SMTs, the spec contains a field of "*.child.k2", which
> appears
> > >> to
> > >> > > have two fields after the asterisk level. I may be
> misunderstanding
> > >> the
> > >> > > proposal, but it seems like the two contradict each other.
> > >> > >
> > >> >
> > >> > Thanks for catching this. I have clarified it by removing this
> > >> restriction.
> > >> > Also, have extended the deep-scan scenarios.
> > >> >
> > >> >
> > >> > >
> > >> > > 2. I'm a little unclear on why 

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[David Jacot]

Thanks for the KIP.

I read it and I am also worried by the complexity of the new
configurations. They are not easy to grasp. I need to digest it a bit more,
I think.

Best,
David

Le mer. 29 juin 2022 à 02:25, Matthias J. Sax  a écrit :

> Thanks for the KIP.
>
> I don't think I fully digested the proposal yet, but my first reaction
> is: this is quite complicated. Frankly, I am worried about complexity
> and usability.
>
> Especially the option `safe_latest` is a "weird" one IMHO, and `nearest`
> is even more complex.
>
> The problem at hand (as I understand it from the Jira) is a real one,
> but I am wondering if it would be something that should be addressed by
> the application? If you pass in strategy `none`, and a new partition is
> added, you can react to it by custom code. For regular startup you can
> still go with "latest" to avoid reprocessing the history.
>
> Adding "latest/earliest_on_start" seems useful, as it seems to also
> address https://issues.apache.org/jira/browse/KAFKA-3370
>
>
> -Matthias
>
>
> On 6/7/22 12:55 AM, hudeqi wrote:
> > I think so too, what about Guozhang Wang and Luke Chen? Can I initiate a
> voting process?
> >
> > Best,
> > hudeqi
> >
> >  -原始邮件-
> >  发件人: "邓子明" 
> >  发送时间: 2022-06-07 10:23:37 (星期二)
> >  收件人: dev@kafka.apache.org
> >  抄送:
> >  主题: Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms
> > 
> > 
>

[GitHub] [kafka-site] ZachLC commented on pull request #413: Add Covage to powered-by

[GitBox]

ZachLC commented on PR #413:
URL: https://github.com/apache/kafka-site/pull/413#issuecomment-1169764733

   > HI @ZachLC can you rebase this PR and we can get this merged?
   
   Hi @bbejeck,
   Sure. thanks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-847: Add ProducerCount metrics

[Luke Chen]

Hi Artem,

Could you elaborate more in the motivation section?
I'm interested to know what kind of scenarios this metric can benefit for.
What could it bring to us when a topic partition has 100 ProducerIdCount VS
another topic partition has 10 ProducerIdCount?

Thank you.
Luke

On Wed, Jun 29, 2022 at 6:30 AM Jun Rao  wrote:

> Hi, Artem,
>
> Thanks for the KIP.
>
> Could you explain the partition level ProducerIdCount a bit more? Does that
> reflect the number of PIDs ever produced to a partition since the broker is
> started? Do we reduce the count after a PID expires?
>
> Thanks,
>
> Jun
>
> On Wed, Jun 22, 2022 at 1:08 AM David Jacot 
> wrote:
>
> > Hi Artem,
> >
> > The KIP LGTM.
> >
> > Thanks,
> > David
> >
> > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits
> >  wrote:
> > >
> > > If there is no other feedback I'm going to start voting in a couple
> days.
> > >
> > > -Artem
> > >
> > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits  >
> > > wrote:
> > >
> > > > Thank you for your feedback.  Updated the KIP and added the Rejected
> > > > Alternatives section.
> > > >
> > > > -Artem
> > > >
> > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma 
> wrote:
> > > >
> > > >> If we don't track them separately, then it makes sense to keep it as
> > one
> > > >> metric. I'd probably name it ProducerIdCount in that case.
> > > >>
> > > >> Ismael
> > > >>
> > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits
> > > >>  wrote:
> > > >>
> > > >> > Do you propose to have 2 metrics instead of one?  Right now we
> don't
> > > >> track
> > > >> > if the producer id was transactional or idempotent and for metric
> > > >> > collection we'd either have to pay the cost of iterating over
> > producer
> > > >> ids
> > > >> > (which could be a lot) or split the producer map into 2 or cache
> the
> > > >> > counts, which complicates the code.
> > > >> >
> > > >> > From the monitoring perspective, I think one metric should be
> good,
> > but
> > > >> > maybe I'm missing some scenarios.
> > > >> >
> > > >> > -Artem
> > > >> >
> > > >> > On Fri, Jun 17, 2022 at 12:28 AM Ismael Juma 
> > wrote:
> > > >> >
> > > >> > > I like the suggestion to have IdempotentProducerCount and
> > > >> > > TransactionalProducerCount metrics.
> > > >> > >
> > > >> > > Ismael
> > > >> > >
> > > >> > > On Thu, Jun 16, 2022 at 2:27 PM Artem Livshits
> > > >> > >  wrote:
> > > >> > >
> > > >> > > > Hi Ismael,
> > > >> > > >
> > > >> > > > Thank you for your feedback.  Yes, this is counting the number
> > of
> > > >> > > producer
> > > >> > > > ids tracked by the partition and broker.  Another options I
> was
> > > >> > thinking
> > > >> > > of
> > > >> > > > are the following:
> > > >> > > >
> > > >> > > > - IdempotentProducerCount
> > > >> > > > - TransactionalProducerCount
> > > >> > > > - ProducerIdCount
> > > >> > > >
> > > >> > > > Let me know if one of these seems better, or I'm open to other
> > name
> > > >> > > > suggestions as well.
> > > >> > > >
> > > >> > > > -Artem
> > > >> > > >
> > > >> > > > On Wed, Jun 15, 2022 at 11:49 PM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > Thanks for the KIP.
> > > >> > > > >
> > > >> > > > > ProducerCount seems like a misleading name since producers
> > > >> without a
> > > >> > > > > producer id are not counted. Is this meant to count the
> > number of
> > > >> > > > producer
> > > >> > > > > IDs tracked by the broker?
> > > >> > > > >
> > > >> > > > > Ismael
> > > >> > > > >
> > > >> > > > > On Wed, Jun 15, 2022, 3:12 PM Artem Livshits <
> > > >> alivsh...@confluent.io
> > > >> > > > > .invalid>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hello,
> > > >> > > > > >
> > > >> > > > > > I'd like to start a discussion on the KIP-847:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-847%3A+Add+ProducerCount+metrics
> > > >> > > > > > .
> > > >> > > > > >
> > > >> > > > > > -Artem
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> >
>

[GitHub] [kafka-site] d-t-w commented on pull request #372: Add kPow (https://www.kpow.io) to the powered-by page

[GitBox]

d-t-w commented on PR #372:
URL: https://github.com/apache/kafka-site/pull/372#issuecomment-1169631953

   Thanks @bbejeck much appreciated :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org