[jira] [Resolved] (KAFKA-14035) QuorumController handleRenounce throws NPE

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-14035.
-
Fix Version/s: 3.1.2
   3.2.1
   Resolution: Fixed

> QuorumController handleRenounce throws NPE
> --
>
> Key: KAFKA-14035
> URL: https://issues.apache.org/jira/browse/KAFKA-14035
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Niket Goel
>Assignee: Niket Goel
>Priority: Major
> Fix For: 3.3.0, 3.1.2, 3.2.1
>
>
> Sometimes when the controller is rolled you can encounter the following 
> exception, after which the controller in-memory state seems to become 
> inconsistent with the Metadata Log.
>  
> [Controller 1] handleRenounce[23]: failed with unknown server exception 
> NullPointerException at epoch -1 in  us. Reverting to last committed 
> offset .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14036) Kraft controller local time not computed correctly.

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-14036.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

> Kraft controller local time not computed correctly.
> ---
>
> Key: KAFKA-14036
> URL: https://issues.apache.org/jira/browse/KAFKA-14036
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Priority: Major
> Fix For: 3.3.0
>
>
> In `ControllerApis`, we are missing the logic to set the local processing end 
> time after `handle` returns. As a consequence of this, the remote time ends 
> up reported as the local time in the request level metrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions

[Luke Chen]

Hi Guozhang,

Thanks for the KIP.
Some comments:
1. I have the same question as Ziming, should we also add an option in
kafka-consumer-groups.sh in this KIP?
Or you'd like to keep the current scope, and other people can create a
follow-up KIP to address the kafka-consumer-groups.sh script?
2. The setter method name: `shouldRequireStable` might need to rename to
`requireStable` to be consistent with above `topicPartitions` getter/setter

Thank you.
Luke

On Fri, Jul 1, 2022 at 11:17 AM John Roesler  wrote:

> Thanks for the KIP, Guozhang!
>
> I’m +1 (binding)
>
> -John
>
> On Thu, Jun 30, 2022, at 21:17, deng ziming wrote:
> > Thanks for this KIP,
> > we have a kafka-consumer-groups.sh shell which is based on the API you
> > proposed to change, is it worth update it as well?
> >
> > --
> > Best,
> > Ziming
> >
> >> On Jul 1, 2022, at 9:04 AM, Guozhang Wang  wrote:
> >>
> >> Hello folks,
> >>
> >> I'd like to call out for a vote for the following KIP to expose the
> >> requireStable flag inside admin client's options as well:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions
> >>
> >> Any feedback as well as your votes are welcome.
> >>
> >> -- Guozhang
>

Re: [VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions

[John Roesler]

Thanks for the KIP, Guozhang!

I’m +1 (binding)

-John

On Thu, Jun 30, 2022, at 21:17, deng ziming wrote:
> Thanks for this KIP,
> we have a kafka-consumer-groups.sh shell which is based on the API you 
> proposed to change, is it worth update it as well?
>
> --
> Best,
> Ziming
>
>> On Jul 1, 2022, at 9:04 AM, Guozhang Wang  wrote:
>> 
>> Hello folks,
>> 
>> I'd like to call out for a vote for the following KIP to expose the
>> requireStable flag inside admin client's options as well:
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions
>> 
>> Any feedback as well as your votes are welcome.
>> 
>> -- Guozhang

Re: [VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions

[deng ziming]

Thanks for this KIP,
we have a kafka-consumer-groups.sh shell which is based on the API you proposed 
to change, is it worth update it as well?

--
Best,
Ziming

> On Jul 1, 2022, at 9:04 AM, Guozhang Wang  wrote:
> 
> Hello folks,
> 
> I'd like to call out for a vote for the following KIP to expose the
> requireStable flag inside admin client's options as well:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions
> 
> Any feedback as well as your votes are welcome.
> 
> -- Guozhang

[VOTE] KIP-851: : Add requireStable flag into ListConsumerGroupOffsetsOptions

[Guozhang Wang]

Hello folks,

I'd like to call out for a vote for the following KIP to expose the
requireStable flag inside admin client's options as well:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-851%3A+Add+requireStable+flag+into+ListConsumerGroupOffsetsOptions

Any feedback as well as your votes are welcome.

-- Guozhang

[jira] [Created] (KAFKA-14036) Kraft controller local time not computed correctly.

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-14036:
---

 Summary: Kraft controller local time not computed correctly.
 Key: KAFKA-14036
 URL: https://issues.apache.org/jira/browse/KAFKA-14036
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


In `ControllerApis`, we are missing the logic to set the local processing end 
time after `handle` returns. As a consequence of this, the remote time ends up 
reported as the local time in the request level metrics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14035) QuorumController handleRenounce throws NPE

[Niket Goel (Jira)]

Niket Goel created KAFKA-14035:
--

 Summary: QuorumController handleRenounce throws NPE
 Key: KAFKA-14035
 URL: https://issues.apache.org/jira/browse/KAFKA-14035
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Niket Goel
Assignee: Niket Goel
 Fix For: 3.3.0


Sometimes when the controller is rolled you can encounter the following 
exception, after which the controller in-memory state seems to become 
inconsistent with the Metadata Log.

 

[Controller 1] handleRenounce[23]: failed with unknown server exception 
NullPointerException at epoch -1 in  us. Reverting to last committed offset 
.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-847: Add ProducerCount metrics

[Jun Rao]

Hi, Artem,

Thanks for the reply.

The memory usage on the broker is proportional to the number of (partition,
pid) combinations. So, I am wondering if we could have a metric that
captures that. The proposed pid count metric doesn't fully capture that
since each pid could be associated with a different number of partitions.

Thanks,

Jun

On Thu, Jun 30, 2022 at 9:24 AM Justine Olshan 
wrote:

> Hi Artem,
> Thanks for the update to include motivation. Makes sense to me.
> Justine
>
> On Wed, Jun 29, 2022 at 6:51 PM Luke Chen  wrote:
>
> > Hi Artem,
> >
> > Thanks for the update.
> > LGTM.
> >
> > Luke
> >
> > On Thu, Jun 30, 2022 at 6:51 AM Artem Livshits
> >  wrote:
> >
> > > Thank you for your feedback. I've updated the KIP to elaborate on the
> > > motivation and provide some background on producer ids and how we
> measure
> > > them.
> > >
> > > Also, after some thinking and discussing it offline with some folks, I
> > > think that we don't really need partitioner level metrics.  We can use
> > > existing tools to do granular debugging.  I've moved partition level
> > > metrics to the rejected alternatives section.
> > >
> > > -Artem
> > >
> > > On Wed, Jun 29, 2022 at 1:57 AM Luke Chen  wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > Could you elaborate more in the motivation section?
> > > > I'm interested to know what kind of scenarios this metric can benefit
> > > for.
> > > > What could it bring to us when a topic partition has 100
> > ProducerIdCount
> > > VS
> > > > another topic partition has 10 ProducerIdCount?
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Wed, Jun 29, 2022 at 6:30 AM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Artem,
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > > Could you explain the partition level ProducerIdCount a bit more?
> > Does
> > > > that
> > > > > reflect the number of PIDs ever produced to a partition since the
> > > broker
> > > > is
> > > > > started? Do we reduce the count after a PID expires?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Wed, Jun 22, 2022 at 1:08 AM David Jacot
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Artem,
> > > > > >
> > > > > > The KIP LGTM.
> > > > > >
> > > > > > Thanks,
> > > > > > David
> > > > > >
> > > > > > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits
> > > > > >  wrote:
> > > > > > >
> > > > > > > If there is no other feedback I'm going to start voting in a
> > couple
> > > > > days.
> > > > > > >
> > > > > > > -Artem
> > > > > > >
> > > > > > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits <
> > > > alivsh...@confluent.io
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thank you for your feedback.  Updated the KIP and added the
> > > > Rejected
> > > > > > > > Alternatives section.
> > > > > > > >
> > > > > > > > -Artem
> > > > > > > >
> > > > > > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > >> If we don't track them separately, then it makes sense to
> keep
> > > it
> > > > as
> > > > > > one
> > > > > > > >> metric. I'd probably name it ProducerIdCount in that case.
> > > > > > > >>
> > > > > > > >> Ismael
> > > > > > > >>
> > > > > > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits
> > > > > > > >>  wrote:
> > > > > > > >>
> > > > > > > >> > Do you propose to have 2 metrics instead of one?  Right
> now
> > we
> > > > > don't
> > > > > > > >> track
> > > > > > > >> > if the producer id was transactional or idempotent and for
> > > > metric
> > > > > > > >> > collection we'd either have to pay the cost of iterating
> > over
> > > > > > producer
> > > > > > > >> ids
> > > > > > > >> > (which could be a lot) or split the producer map into 2 or
> > > cache
> > > > > the
> > > > > > > >> > counts, which complicates the code.
> > > > > > > >> >
> > > > > > > >> > From the monitoring perspective, I think one metric should
> > be
> > > > > good,
> > > > > > but
> > > > > > > >> > maybe I'm missing some scenarios.
> > > > > > > >> >
> > > > > > > >> > -Artem
> > > > > > > >> >
> > > > > > > >> > On Fri, Jun 17, 2022 at 12:28 AM Ismael Juma <
> > > ism...@juma.me.uk
> > > > >
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > I like the suggestion to have IdempotentProducerCount
> and
> > > > > > > >> > > TransactionalProducerCount metrics.
> > > > > > > >> > >
> > > > > > > >> > > Ismael
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Jun 16, 2022 at 2:27 PM Artem Livshits
> > > > > > > >> > >  wrote:
> > > > > > > >> > >
> > > > > > > >> > > > Hi Ismael,
> > > > > > > >> > > >
> > > > > > > >> > > > Thank you for your feedback.  Yes, this is counting
> the
> > > > number
> > > > > > of
> > > > > > > >> > > producer
> > > > > > > >> > > > ids tracked by the partition and broker.  Another
> > options
> > > I
> > > > > was
> > > > > > > >> > thinking
> > > > > > > >> > > of
> > > > > > > >> > > > are the following:
> > > > > > > >> > > >
> > > > > > > >> > > > 

[jira] [Created] (KAFKA-14034) Consistency violation: enabled idempotency doesn't prevent dublicates

[Denis Rystsov (Jira)]

Denis Rystsov created KAFKA-14034:
-

 Summary: Consistency violation: enabled idempotency doesn't 
prevent dublicates
 Key: KAFKA-14034
 URL: https://issues.apache.org/jira/browse/KAFKA-14034
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.1.1, 3.0.0
Reporter: Denis Rystsov


Hey folks, I've observed duplicated records in the log while idempotency was 
enabled and it looks like the kafka client is the culprit. I've tested on 3.0.0 
but the tip of kafka repo is also affected

Let a user sends two produce requests without async so there is two inflight 
requests
{code:java}
producer.send(A)
producer.send(B){code}
Let the first request results with a retry-able error after it was written to 
disk and let the second request results with UNKNOWN_SERVER_ERROR. Any 
unhandled exception on the broker side results in UNKNOWN_SERVER_ERROR so it 
may happen.

Since request A is retry-able it is put into the outbound queue there - 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L623]

Let B's UNKNOWN_SERVER_ERROR is received before A is retried. It is being 
processed in the following methods:
 * 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L642]
 * 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L742]
 * 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L761]
 * 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L624]
 * maybeTransitionToErrorState doesn't consider UNKNOWN_SERVER_ERROR fatal so 
it doesn't mark the request as such: 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L611]
 * as result handleFailedBatch requests epochBump 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L652]

When epoch is bumped it rewrites sequence numbers of the inflight requests: 
[https://github.com/apache/kafka/blob/1daa149730e3c56b7b6fe8f14369178273e8efc4/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L481]

In our case it rewrites A's sequence numbers and when the request is retried 
the broker can't dedupe it and writes it to the log thus violating the 
idempotency guarantees.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

PR #12368 review request

[Николай Ижиков]

Hello.

I found dead code.
Raised a PR [1] to remove it.

Can someone review the PR?


[1] https://github.com/apache/kafka/pull/12368

[jira] [Resolved] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-13936.
-
Resolution: Fixed

> Invalid consumer lag when monitoring from a kafka streams application
> -
>
> Key: KAFKA-13936
> URL: https://issues.apache.org/jira/browse/KAFKA-13936
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Prashanth Joseph Babu
>Assignee: Prashanth Joseph Babu
>Priority: Major
>
> I have a kafka streams application and I'm trying to monitor the consumer lag 
> via stream metrics.
> Here's some code snippet
> {code:java}
> metrics = streams.metrics();
> lag = 0;
> for (Metric m : metrics.values()) {
> tags = m.metricName().tags();
> if ( m.metricName().name().equals(MONITOR_CONSUMER_LAG) && 
> tags.containsKey(MONTOR_TAG_TOPIC) && 
> tags.get(MONTOR_TAG_TOPIC).equals(inputTopic) ) {
> partitionLag = 
> Float.valueOf(m.metricValue().toString()).floatValue();
> if ( !partitionLag.isNaN() ) {
> lag += partitionLag;
> }
> }
> }
> {code}
> Here MONITOR_CONSUMER_LAG is {{{}records-lag-max{}}}.
> However these numbers dont match with the consumer lag we see in the kafka UI 
> . is records-lag-max the right metric to track for a kafka streams 
> application when the objective is to get consumer lag?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka Log4J vulnerabilities - Urgent

[Dongjin Lee]

Hi,

Currently, Apache Kafka with log4j2 is planning to be 4.0, after the 3.5
release. If you are urgent, please have a look at the preview versions here
 - You
can find a custom preview based on 3.0.1, 3.1.1, and 3.2.0.

I wrote KIP-653 and KIP-719, and am maintaining the previews & bug fixes
.
I am also working on the docker image, but it is being delayed since I am
applying this feature into my employer
's internal Kafka
distribution.

If you experience problems trying the preview, don't hesitate to send me a
direct message.

Thanks,
Dongjin

On Tue, Jun 28, 2022 at 12:41 AM  wrote:

> Hi Team,
>
> Trust you are doing good and I hope I'm mailing the correct DL (if not
> kindly point me to one) !
>
> This mail is w.r.t Kafka Log4j vulnerabilities. PFB the description -
>
> Log4J 1.x vulnerability with Kafka is a known vulnerability. The published
> workaround is to remove the Appender Classes from the JAR artefact. This
> has already been implemented by DevOps team
>
> Kafka documentation referred from here -
> https://kafka.apache.org/cve-list
>
> However our Corporate Security Team wants Log4j 1.x versions to be
> completely removed and/or upgraded to log4j 2.x. We have not come across
> any published set up steps from Kafka documentation.
>
> There is one blog that talks about upgrade proposal but we are unsure
> whether it can be implemented(Blog link below) -
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender#KIP719:DeprecateLog4JAppender-1.Deprecatelog4j-appender
>
> Please advice the best way forward. This is a crucial issue and we are
> getting daily follow ups from the Security Teams .
>
> Thanks,
> Mayank
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [VOTE] KIP-825: introduce a new API to control when aggregated results are produced

[Matthias J. Sax]

Thanks for the update Hao.

Seems it's the best way forward. As we don't add it to 
`ReadOnlySessionStore` it's not exposed via IQ and thus the impact seems 
to be well contained.


No concerns from my side.

-Matthias

On 6/29/22 11:51 AM, Hao Li wrote:

Hi Dev Team,

Just want to inform you here that we made an update to this KIP. To
implement emit final in Session Window aggregation, we augment the
`SessionStore` by adding an
public API to range query from earliest session end time to latest session
end time. API is like this:

```
public interface SessionStore extends StateStore,
ReadOnlySessionStore {

 /**
  * Return all the session window entries that ends between the
specified range (both ends are inclusive).
  * This function would be used to retrieve all closed and immutable
windows.
  *
  * @param earliestSessionEndTime earliest session end time to search
from, inclusive
  * @param latestSessionEndTime latest session end time to search to,
inclusive
  */
 default KeyValueIterator, AGG> findSessions(final long
earliestSessionEndTime,
 final long
latestSessionEndTime)
{
 throw new UnsupportedOperationException(
 "This API is not supported by this implementation of
SessionStore.");
 }

 // other existing functions
}
```

You can take a look at the KIP for more information. Thanks Guozhang for
adding it to the KIP.

Thanks,
Hao

On Thu, Mar 24, 2022 at 4:17 PM Hao Li  wrote:


The vote passed with 5 binding votes from John, Guozhang, Bruno, Matthias
and Bill.

Thanks all for the feedback and vote.

Hao

On Thu, Mar 24, 2022 at 2:20 PM Bill Bejeck  wrote:


Thanks for KIP Hao!

Glad to see we settled on option 1

+1(binding)

On Thu, Mar 24, 2022 at 5:13 PM Matthias J. Sax  wrote:


+1 (binding)


On 3/24/22 1:52 PM, Hao Li wrote:

I hit reply on my phone in the mail app and changed the title and text
hoping it will start a new thread. Apparently it doesn't work...

On Thu, Mar 24, 2022 at 12:36 PM Bruno Cadonna 

wrote:



Hi Hao,

Actually, this is the VOTE thread. Do you use GMail? Sometimes it is

a

bit weird how it shows e-mails in threads.

Anyways, thank you for the KIP and your patience!

+1 (binding)

Best,
Bruno

On 24.03.22 17:36, Hao Li wrote:

Thanks John and Guozhang. Didn't realize I used this discussion

thread

for

voting. Let me start a new thread for voting. Will fix the KIP.

On Thu, Mar 24, 2022 at 9:28 AM Guozhang Wang 

wrote:



+1 (binding).

Thanks Hao!


Guozhang

On Thu, Mar 24, 2022 at 9:20 AM John Roesler 

wrote:



Thanks, Hao!

I'm +1 (binding)

-John

On Wed, 2022-03-23 at 22:25 -0700, Hao Li wrote:

Hi all,

I'd like to start a vote on Kafka Streams KIP-825:










https://cwiki.apache.org/confluence/display/KAFKA/KIP-825%3A+introduce+a+new+API+to+control+when+aggregated+results+are+produced





--
-- Guozhang
















--
Thanks,
Hao

Re: [DISCUSS] KIP-847: Add ProducerCount metrics

[Justine Olshan]

Hi Artem,
Thanks for the update to include motivation. Makes sense to me.
Justine

On Wed, Jun 29, 2022 at 6:51 PM Luke Chen  wrote:

> Hi Artem,
>
> Thanks for the update.
> LGTM.
>
> Luke
>
> On Thu, Jun 30, 2022 at 6:51 AM Artem Livshits
>  wrote:
>
> > Thank you for your feedback. I've updated the KIP to elaborate on the
> > motivation and provide some background on producer ids and how we measure
> > them.
> >
> > Also, after some thinking and discussing it offline with some folks, I
> > think that we don't really need partitioner level metrics.  We can use
> > existing tools to do granular debugging.  I've moved partition level
> > metrics to the rejected alternatives section.
> >
> > -Artem
> >
> > On Wed, Jun 29, 2022 at 1:57 AM Luke Chen  wrote:
> >
> > > Hi Artem,
> > >
> > > Could you elaborate more in the motivation section?
> > > I'm interested to know what kind of scenarios this metric can benefit
> > for.
> > > What could it bring to us when a topic partition has 100
> ProducerIdCount
> > VS
> > > another topic partition has 10 ProducerIdCount?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Jun 29, 2022 at 6:30 AM Jun Rao 
> > wrote:
> > >
> > > > Hi, Artem,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > Could you explain the partition level ProducerIdCount a bit more?
> Does
> > > that
> > > > reflect the number of PIDs ever produced to a partition since the
> > broker
> > > is
> > > > started? Do we reduce the count after a PID expires?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Wed, Jun 22, 2022 at 1:08 AM David Jacot
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi Artem,
> > > > >
> > > > > The KIP LGTM.
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Tue, Jun 21, 2022 at 9:32 PM Artem Livshits
> > > > >  wrote:
> > > > > >
> > > > > > If there is no other feedback I'm going to start voting in a
> couple
> > > > days.
> > > > > >
> > > > > > -Artem
> > > > > >
> > > > > > On Fri, Jun 17, 2022 at 3:50 PM Artem Livshits <
> > > alivsh...@confluent.io
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thank you for your feedback.  Updated the KIP and added the
> > > Rejected
> > > > > > > Alternatives section.
> > > > > > >
> > > > > > > -Artem
> > > > > > >
> > > > > > > On Fri, Jun 17, 2022 at 1:16 PM Ismael Juma  >
> > > > wrote:
> > > > > > >
> > > > > > >> If we don't track them separately, then it makes sense to keep
> > it
> > > as
> > > > > one
> > > > > > >> metric. I'd probably name it ProducerIdCount in that case.
> > > > > > >>
> > > > > > >> Ismael
> > > > > > >>
> > > > > > >> On Fri, Jun 17, 2022 at 12:04 PM Artem Livshits
> > > > > > >>  wrote:
> > > > > > >>
> > > > > > >> > Do you propose to have 2 metrics instead of one?  Right now
> we
> > > > don't
> > > > > > >> track
> > > > > > >> > if the producer id was transactional or idempotent and for
> > > metric
> > > > > > >> > collection we'd either have to pay the cost of iterating
> over
> > > > > producer
> > > > > > >> ids
> > > > > > >> > (which could be a lot) or split the producer map into 2 or
> > cache
> > > > the
> > > > > > >> > counts, which complicates the code.
> > > > > > >> >
> > > > > > >> > From the monitoring perspective, I think one metric should
> be
> > > > good,
> > > > > but
> > > > > > >> > maybe I'm missing some scenarios.
> > > > > > >> >
> > > > > > >> > -Artem
> > > > > > >> >
> > > > > > >> > On Fri, Jun 17, 2022 at 12:28 AM Ismael Juma <
> > ism...@juma.me.uk
> > > >
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > I like the suggestion to have IdempotentProducerCount and
> > > > > > >> > > TransactionalProducerCount metrics.
> > > > > > >> > >
> > > > > > >> > > Ismael
> > > > > > >> > >
> > > > > > >> > > On Thu, Jun 16, 2022 at 2:27 PM Artem Livshits
> > > > > > >> > >  wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Ismael,
> > > > > > >> > > >
> > > > > > >> > > > Thank you for your feedback.  Yes, this is counting the
> > > number
> > > > > of
> > > > > > >> > > producer
> > > > > > >> > > > ids tracked by the partition and broker.  Another
> options
> > I
> > > > was
> > > > > > >> > thinking
> > > > > > >> > > of
> > > > > > >> > > > are the following:
> > > > > > >> > > >
> > > > > > >> > > > - IdempotentProducerCount
> > > > > > >> > > > - TransactionalProducerCount
> > > > > > >> > > > - ProducerIdCount
> > > > > > >> > > >
> > > > > > >> > > > Let me know if one of these seems better, or I'm open to
> > > other
> > > > > name
> > > > > > >> > > > suggestions as well.
> > > > > > >> > > >
> > > > > > >> > > > -Artem
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Jun 15, 2022 at 11:49 PM Ismael Juma <
> > > > ism...@juma.me.uk
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Thanks for the KIP.
> > > > > > >> > > > >
> > > > > > >> > > > > ProducerCount seems like a misleading name since
> > producers
> > > > > > >> without a
> > > > > > >> > > > > producer id are not counted. Is this meant 

[GitHub] [kafka-site] JaviOverflow commented on pull request #320: Update powered-by.html used by percentage

[GitBox]

JaviOverflow commented on PR #320:
URL: https://github.com/apache/kafka-site/pull/320#issuecomment-1170851653

   After 1 year and 8 months, you may want to check if 80% is still the right 
number haha


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org