Starting out with Kafka

[vinay deshpande]

Hi All,
I have a basic question: I tried importing kafka source code into intellij
but there are bunch of imports that IDE cannot find like these:

import kafka.api.ApiVersion;
import kafka.log.CleanerConfig;
import kafka.log.LogConfig;
import kafka.log.LogManager;


TIA.

Thanks,
Vinay

[jira] [Created] (KAFKA-14382) StreamThreads can miss rebalance events when processing records during a rebalance

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-14382:
--

 Summary: StreamThreads can miss rebalance events when processing 
records during a rebalance
 Key: KAFKA-14382
 URL: https://issues.apache.org/jira/browse/KAFKA-14382
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: A. Sophie Blee-Goldman


One of the main improvements introduced by the cooperative protocol was the 
ability to continue processing records during a rebalance. In Streams, we take 
advantage of this by polling with a timeout of 0 when a rebalance is/has been 
in progress, so it can return immediately and continue on through the main loop 
to process new records. The main poll loop uses an algorithm based on the 
max.poll.interval.ms to ensure the StreamThread returns to call #poll in time 
to stay in the consumer group.

 

Generally speaking, it should exit the processing loop and invoke poll within a 
few minutes at most based on the poll interval, though typically it will break 
out much sooner once it's used up all the records from the last poll (based on 
the max.poll.records config which Streams sets to 1,000 by default). However, 
if doing heavy processing or setting a higher max.poll.records, the thread may 
continue processing for more than a few seconds. If it had sent out a JoinGroup 
request before going on to process and was waiting for its JoinGroup response, 
then once it does return to invoke #poll it will process this response and send 
out a SyncGroup – but if the processing took too long, this SyncGroup may 
immediately fail with the REBALANCE_IN_PROGRESS error.

 

Essentially, while the thread was processing the group leader will itself be 
processing the JoinGroup subscriptions of all members and generating an 
assignment, then sending this back in its SyncGroup. This may take only a few 
seconds or less, and the group coordinator will not yet have noticed (or care) 
that one of the consumers hasn't sent a SyncGroup – it will just return the 
assigned partitions in the SyncGroup request of the members who have responded 
in time, and "complete" the rebalance in their eyes. But if the assignment 
involved moving any partitions from one consumer to another, then it will need 
to trigger a followup rebalance right away to finish assigning those partitions 
which were revoked in the previous rebalance. This is what causes a new 
rebalance to be kicked off just seconds after the first one began.

 

If the consumer that was stuck processing was among those who needed to revoke 
partitions, this can lead to repeating rebalances – since it fails the 
SyncGroup of the 1st rebalance it never receives the assignment for it and 
never knows to revoke those partitions, meaning it will rejoin for the new 
rebalance still claiming them among its ownedPartitions. When the assignor 
generates the same assignment for the 2nd rebalance, it will again see that 
some partitions need to be revoked and will therefore trigger yet another new 
rebalance after finishing the 2nd. This can go on for as long as the 
StreamThreads are struggling to finish the JoinGroup phase in time due to 
processing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Jackson CVE's in Mirror Maker 2.13-2.8.2

[Luke Chen]

Hi Andrew,

Kafka community will only do bug fix release for last 3 releases based on
the wiki. So, there will be no newer 2.8 patch release.
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy
?

Thank you.
Luke

On Thu, Nov 10, 2022 at 11:30 PM Andrew Pomponio 
wrote:

> Hello Kafka Developers,
>
> I was wondering if there are any plans to back port fixes for certain
> CVE’s found in Mirror Maker 2.13-2.8.2. Scans of the code found the
> following unpatched CVE’s:
>
>
>   *   CVE-2022-42004
>   *   CVE-2022-42003
>   *   CVE-2020-36518
>
> It’s my understanding that there’s going to be a hot fix coming out for
> 2.13-2.8.2 and I was wondering if those CVE’s will be addressed. Any
> information is greatly appreciated. Thanks!
>
>
>
> Andrew Pomponio | Associate Enterprise Architect, OpenLogic<
> https://www.openlogic.com/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2019-common&utm_content=email-signature-link
> >
> Perforce Software<
> http://www.perforce.com/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link
> >
> P: +1 612.517.2100 
> Visit us on: LinkedIn<
> https://www.linkedin.com/company/perforce?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link>
> | Twitter<
> https://twitter.com/perforce?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link>
> | Facebook<
> https://www.facebook.com/perforce/?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link>
> | YouTube<
> https://www.youtube.com/user/perforcesoftware?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link
> >
>
> Use our new Community portal to submit/track support cases!<
> https://www.perforce.com/support/community-portal-faq?utm_source=sales-signature&utm_medium=email&utm_campaign=community-portal-faq&utm_content=resource?utm_leadsource=email-signature&utm_source=outlook-direct-email&utm_medium=email&utm_campaign=2021-common&utm_content=email-signature-link
> >
>
>
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and any
> attachments and notify us immediately.
>
>

[jira] [Resolved] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-14380.
-
Resolution: Duplicate

duplicated with KAFKA-14379

> consumer should refresh preferred read replica on metadata update
> -
>
> Key: KAFKA-14380
> URL: https://issues.apache.org/jira/browse/KAFKA-14380
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> The consumer (fetcher) clears the preferred read replica only on two 
> conditions:
>  # the consumer receives an OFFSET_OUT_OF_RANGE error
>  # the follower does not exist in the client's metadata (i.e., offline)
>  # preferred read replica value expires (5 minutes)
> For other errors, it will continue to reach to the possibly unavailable 
> follower and only after 5 minutes will it clear preferred read replica and go 
> back to the leader.
> A specific example is when a partition is reassigned. the consumer will get 
> NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred 
> read replica will not be refreshed as the follower is still online. it will 
> continue to reach out to the old follower until the preferred read replica 
> expires.
> the consumer can instead clear its preferred read replica whenever it updates 
> its metadata. so when the consumer receives NOT_LEADER_OR_FOLLOWER in the 
> scenario above it can find the new preferred read replica by fetching from 
> the leader without waiting for the old value to expire.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

[Deng Ziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deng Ziming resolved KAFKA-14378.
-
Resolution: Duplicate

duplicated with KAFKA-14379

> consumer should refresh preferred read replica on update metadata
> -
>
> Key: KAFKA-14378
> URL: https://issues.apache.org/jira/browse/KAFKA-14378
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> The consumer (fetcher) refreshes the preferred read replica only on three 
> conditions:
>  # the consumer receives an OFFSET_OUT_OF_RANGE error
>  # the follower does not exist in the client's metadata (i.e., offline)
>  # after metadata.max.age.ms (5 min default)
> For other errors, it will continue to reach to the possibly unavailable 
> follower and only after 5 minutes will it refresh the preferred read replica 
> and go back to the leader.
> A specific example is when a partition is reassigned. the consumer will get 
> NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred 
> read replica will not be refreshed as the follower is still online. it will 
> continue to reach out to the old follower until the preferred read replica 
> expires.
> the consumer can instead refresh its preferred read replica whenever it makes 
> a metadata update request. so when the consumer receives i.e. 
> NOT_LEADER_OR_FOLLOWER it can find the new preferred read replica without 
> waiting for the expiration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-886 Add Client Producer and Consumer Builders

[Knowles Atchison Jr]

This would be helpful. For our own client library wrappers we implemented
this functionality for any type with defaults for  and
 consumers/producers.

On Thu, Nov 10, 2022, 6:35 PM Dan S  wrote:

> Hello all,
>
> I think that adding builders for the producer and the consumer in kafka
> client would make it much easier for developers to instantiate new
> producers and consumers, especially if they are using an IDE with
> intellisense, and using the IDE to navigate to the documentation which
> could be added to the builder's withXYZ methods.
>
> Please let me know if you have any comments, questions, or suggestions!
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-886%3A+Add+Client+Producer+and+Consumer+Builders
>
> Thanks,
>
> Dan
>

[jira] [Created] (KAFKA-14381) Support listing all partitions being reassigned in a cluster

[Lucas Wang (Jira)]

Lucas Wang created KAFKA-14381:
--

 Summary: Support listing all partitions being reassigned in a 
cluster
 Key: KAFKA-14381
 URL: https://issues.apache.org/jira/browse/KAFKA-14381
 Project: Kafka
  Issue Type: Improvement
Reporter: Lucas Wang


The current implementation of kafka-topics.sh doesn't support listing all of 
the partitions that are being reassigned within a cluster.

Showing such info can be really useful during troubleshooting.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14380) consumer should refresh preferred read replica on metadata update

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-14380:


 Summary: consumer should refresh preferred read replica on 
metadata update
 Key: KAFKA-14380
 URL: https://issues.apache.org/jira/browse/KAFKA-14380
 Project: Kafka
  Issue Type: Bug
Reporter: Jeff Kim
Assignee: Jeff Kim


The consumer (fetcher) clears the preferred read replica only on two conditions:
 # the consumer receives an OFFSET_OUT_OF_RANGE error
 # the follower does not exist in the client's metadata (i.e., offline)
 # preferred read replica value expires (5 minutes)

For other errors, it will continue to reach to the possibly unavailable 
follower and only after 5 minutes will it clear preferred read replica and go 
back to the leader.

A specific example is when a partition is reassigned. the consumer will get 
NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred read 
replica will not be refreshed as the follower is still online. it will continue 
to reach out to the old follower until the preferred read replica expires.

the consumer can instead clear its preferred read replica whenever it updates 
its metadata. so when the consumer receives NOT_LEADER_OR_FOLLOWER in the 
scenario above it can find the new preferred read replica by fetching from the 
leader without waiting for the old value to expire.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14378) consumer should refresh preferred read replica on update metadata

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-14378:


 Summary: consumer should refresh preferred read replica on update 
metadata
 Key: KAFKA-14378
 URL: https://issues.apache.org/jira/browse/KAFKA-14378
 Project: Kafka
  Issue Type: Bug
Reporter: Jeff Kim
Assignee: Jeff Kim


The consumer (fetcher) refreshes the preferred read replica only on three 
conditions:
 # the consumer receives an OFFSET_OUT_OF_RANGE error
 # the follower does not exist in the client's metadata (i.e., offline)
 # after metadata.max.age.ms (5 min default)

For other errors, it will continue to reach to the possibly unavailable 
follower and only after 5 minutes will it refresh the preferred read replica 
and go back to the leader.

A specific example is when a partition is reassigned. the consumer will get 
NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred read 
replica will not be refreshed as the follower is still online. it will continue 
to reach out to the old follower until the preferred read replica expires.

the consumer can instead refresh its preferred read replica whenever it makes a 
metadata update request. so when the consumer receives i.e. 
NOT_LEADER_OR_FOLLOWER it can find the new preferred read replica without 
waiting for the expiration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14379) consumer should refresh preferred read replica on update metadata

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-14379:


 Summary: consumer should refresh preferred read replica on update 
metadata
 Key: KAFKA-14379
 URL: https://issues.apache.org/jira/browse/KAFKA-14379
 Project: Kafka
  Issue Type: Bug
Reporter: Jeff Kim
Assignee: Jeff Kim


The consumer (fetcher) refreshes the preferred read replica only on three 
conditions:
 # the consumer receives an OFFSET_OUT_OF_RANGE error
 # the follower does not exist in the client's metadata (i.e., offline)
 # after metadata.max.age.ms (5 min default)

For other errors, it will continue to reach to the possibly unavailable 
follower and only after 5 minutes will it refresh the preferred read replica 
and go back to the leader.

A specific example is when a partition is reassigned. the consumer will get 
NOT_LEADER_OR_FOLLOWER which triggers a metadata update but the preferred read 
replica will not be refreshed as the follower is still online. it will continue 
to reach out to the old follower until the preferred read replica expires.

the consumer can instead refresh its preferred read replica whenever it makes a 
metadata update request. so when the consumer receives i.e. 
NOT_LEADER_OR_FOLLOWER it can find the new preferred read replica without 
waiting for the expiration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-886 Add Client Producer and Consumer Builders

[Dan S]

Hello all,

I think that adding builders for the producer and the consumer in kafka
client would make it much easier for developers to instantiate new
producers and consumers, especially if they are using an IDE with
intellisense, and using the IDE to navigate to the documentation which
could be added to the builder's withXYZ methods.

Please let me know if you have any comments, questions, or suggestions!

https://cwiki.apache.org/confluence/display/KAFKA/KIP-886%3A+Add+Client+Producer+and+Consumer+Builders

Thanks,

Dan

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Jun Rao]

Hi, Divij,

Thanks for the reply.

It's true that the data in Kafka could be kept longer with KIP-405. How
much data do you envision to have per broker? For 100TB data per broker,
with 1GB segment and segment metadata of 100 bytes, it requires
100TB/1GB*100 = 10MB, which should fit in memory.

RemoteLogMetadataManager has two listRemoteLogSegments() methods. The one
you listed listRemoteLogSegments(TopicIdPartition topicIdPartition, int
leaderEpoch) does return data in offset order. However, the other
one listRemoteLogSegments(TopicIdPartition topicIdPartition) doesn't
specify the return order. I assume that you need the latter to calculate
the segment size?

Thanks,

Jun

On Thu, Nov 10, 2022 at 10:25 AM Divij Vaidya 
wrote:

> *Jun,*
>
> *"the default implementation of RLMM does local caching, right?"*
> Yes, Jun. The default implementation of RLMM does indeed cache the segment
> metadata today, hence, it won't work for use cases when the number of
> segments in remote storage is large enough to exceed the size of cache. As
> part of this KIP, I will implement the new proposed API in the default
> implementation of RLMM but the underlying implementation will still be a
> scan. I will pick up optimizing that in a separate PR.
>
> *"we also cache all segment metadata in the brokers without KIP-405. Do you
> see a need to change that?"*
> Please correct me if I am wrong here but we cache metadata for segments
> "residing in local storage". The size of the current cache works fine for
> the scale of the number of segments that we expect to store in local
> storage. After KIP-405, that cache will continue to store metadata for
> segments which are residing in local storage and hence, we don't need to
> change that. For segments which have been offloaded to remote storage, it
> would rely on RLMM. Note that the scale of data stored in RLMM is different
> from local cache because the number of segments is expected to be much
> larger than what current implementation stores in local storage.
>
> 2,3,4: RemoteLogMetadataManager.listRemoteLogSegments() does specify the
> order i.e. it returns the segments sorted by first offset in ascending
> order. I am copying the API docs for KIP-405 here for your reference
>
>
>
>
>
>
> *Returns iterator of remote log segment metadata, sorted by {@link
> RemoteLogSegmentMetadata#startOffset()} inascending order which contains
> the given leader epoch. This is used by remote log retention management
> subsystemto fetch the segment metadata for a given leader epoch.@param
> topicIdPartition topic partition@param leaderEpoch  leader
> epoch@return
> Iterator of remote segments, sorted by start offset in ascending order. *
>
> *Luke,*
>
> 5. Note that we are trying to optimize the efficiency of size based
> retention for remote storage. KIP-405 does not introduce a new config for
> periodically checking remote similar to log.retention.check.interval.ms
> which is applicable for remote storage. Hence, the metric will be updated
> at the time of invoking log retention check for remote tier which is
> pending implementation today. We can perhaps come back and update the
> metric description after the implementation of log retention check in
> RemoteLogManager.
>
> --
> Divij Vaidya
>
>
>
> On Thu, Nov 10, 2022 at 6:16 AM Luke Chen  wrote:
>
> > Hi Divij,
> >
> > One more question about the metric:
> > I think the metric will be updated when
> > (1) each time we run the log retention check (that is,
> > log.retention.check.interval.ms)
> > (2) When user explicitly call getRemoteLogSize
> >
> > Is that correct?
> > Maybe we should add a note in metric description, otherwise, when user
> got,
> > let's say 0 of RemoteLogSizeBytes, will be surprised.
> >
> > Otherwise, LGTM
> >
> > Thank you for the KIP
> > Luke
> >
> > On Thu, Nov 10, 2022 at 2:55 AM Jun Rao 
> wrote:
> >
> > > Hi, Divij,
> > >
> > > Thanks for the explanation.
> > >
> > > 1. Hmm, the default implementation of RLMM does local caching, right?
> > > Currently, we also cache all segment metadata in the brokers without
> > > KIP-405. Do you see a need to change that?
> > >
> > > 2,3,4: Yes, your explanation makes sense. However,
> > > currently, RemoteLogMetadataManager.listRemoteLogSegments() doesn't
> > specify
> > > a particular order of the iterator. Do you intend to change that?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 8, 2022 at 3:31 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Hey Jun
> > > >
> > > > Thank you for your comments.
> > > >
> > > > *1. "RLMM implementor could ensure that listRemoteLogSegments() is
> > fast"*
> > > > This would be ideal but pragmatically, it is difficult to ensure that
> > > > listRemoteLogSegments() is fast. This is because of the possibility
> of
> > a
> > > > large number of segments (much larger than what Kafka currently
> handles
> > > > with local storage today) would make it infeasible to adopt
> strategies
> > > such
> > > > as local caching to improve the pe

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Divij Vaidya]

*Jun,*

*"the default implementation of RLMM does local caching, right?"*
Yes, Jun. The default implementation of RLMM does indeed cache the segment
metadata today, hence, it won't work for use cases when the number of
segments in remote storage is large enough to exceed the size of cache. As
part of this KIP, I will implement the new proposed API in the default
implementation of RLMM but the underlying implementation will still be a
scan. I will pick up optimizing that in a separate PR.

*"we also cache all segment metadata in the brokers without KIP-405. Do you
see a need to change that?"*
Please correct me if I am wrong here but we cache metadata for segments
"residing in local storage". The size of the current cache works fine for
the scale of the number of segments that we expect to store in local
storage. After KIP-405, that cache will continue to store metadata for
segments which are residing in local storage and hence, we don't need to
change that. For segments which have been offloaded to remote storage, it
would rely on RLMM. Note that the scale of data stored in RLMM is different
from local cache because the number of segments is expected to be much
larger than what current implementation stores in local storage.

2,3,4: RemoteLogMetadataManager.listRemoteLogSegments() does specify the
order i.e. it returns the segments sorted by first offset in ascending
order. I am copying the API docs for KIP-405 here for your reference






*Returns iterator of remote log segment metadata, sorted by {@link
RemoteLogSegmentMetadata#startOffset()} inascending order which contains
the given leader epoch. This is used by remote log retention management
subsystemto fetch the segment metadata for a given leader epoch.@param
topicIdPartition topic partition@param leaderEpoch  leader epoch@return
Iterator of remote segments, sorted by start offset in ascending order. *

*Luke,*

5. Note that we are trying to optimize the efficiency of size based
retention for remote storage. KIP-405 does not introduce a new config for
periodically checking remote similar to log.retention.check.interval.ms
which is applicable for remote storage. Hence, the metric will be updated
at the time of invoking log retention check for remote tier which is
pending implementation today. We can perhaps come back and update the
metric description after the implementation of log retention check in
RemoteLogManager.

--
Divij Vaidya



On Thu, Nov 10, 2022 at 6:16 AM Luke Chen  wrote:

> Hi Divij,
>
> One more question about the metric:
> I think the metric will be updated when
> (1) each time we run the log retention check (that is,
> log.retention.check.interval.ms)
> (2) When user explicitly call getRemoteLogSize
>
> Is that correct?
> Maybe we should add a note in metric description, otherwise, when user got,
> let's say 0 of RemoteLogSizeBytes, will be surprised.
>
> Otherwise, LGTM
>
> Thank you for the KIP
> Luke
>
> On Thu, Nov 10, 2022 at 2:55 AM Jun Rao  wrote:
>
> > Hi, Divij,
> >
> > Thanks for the explanation.
> >
> > 1. Hmm, the default implementation of RLMM does local caching, right?
> > Currently, we also cache all segment metadata in the brokers without
> > KIP-405. Do you see a need to change that?
> >
> > 2,3,4: Yes, your explanation makes sense. However,
> > currently, RemoteLogMetadataManager.listRemoteLogSegments() doesn't
> specify
> > a particular order of the iterator. Do you intend to change that?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 8, 2022 at 3:31 AM Divij Vaidya 
> > wrote:
> >
> > > Hey Jun
> > >
> > > Thank you for your comments.
> > >
> > > *1. "RLMM implementor could ensure that listRemoteLogSegments() is
> fast"*
> > > This would be ideal but pragmatically, it is difficult to ensure that
> > > listRemoteLogSegments() is fast. This is because of the possibility of
> a
> > > large number of segments (much larger than what Kafka currently handles
> > > with local storage today) would make it infeasible to adopt strategies
> > such
> > > as local caching to improve the performance of listRemoteLogSegments.
> > Apart
> > > from caching (which won't work due to size limitations) I can't think
> of
> > > other strategies which may eliminate the need for IO
> > > operations proportional to the number of total segments. Please advise
> if
> > > you have something in mind.
> > >
> > > 2.  "*If the size exceeds the retention size, we need to determine the
> > > subset of segments to delete to bring the size within the retention
> > limit.
> > > Do we need to call RemoteLogMetadataManager.listRemoteLogSegments() to
> > > determine that?"*
> > > Yes, we need to call listRemoteLogSegments() to determine which
> segments
> > > should be deleted. But there is a difference with the use case we are
> > > trying to optimize with this KIP. To determine the subset of segments
> > which
> > > would be deleted, we only read metadata for segments which would be
> > deleted
> > > via the listRemoteLogSegments(). But to det

[DISCUSS] KIP-885: Expose Broker's Name and Version to Clients Skip to end of metadata

[Travis Bischel]

Hi all,

I've written a KIP to expose the BrokerSoftwareName and
BrokerSoftwareVersion to clients:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-885%3A+Expose+Broker%27s+Name+and+Version+to+Clients

If we agree this is useful, it would be great to have this in by 3.4.

Thank you,
- Travis

[jira] [Created] (KAFKA-14377) Expose Broker's Name and Version to Clients

[Travis Bischel (Jira)]

Travis Bischel created KAFKA-14377:
--

 Summary: Expose Broker's Name and Version to Clients
 Key: KAFKA-14377
 URL: https://issues.apache.org/jira/browse/KAFKA-14377
 Project: Kafka
  Issue Type: Improvement
Reporter: Travis Bischel


Implement KIP-885 as documented here: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-885%3A+Expose+Broker%27s+Name+and+Version+to+Clients



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14376) Add ConfigProvider to make use of environment variables

[Roman Schmitz (Jira)]

Roman Schmitz created KAFKA-14376:
-

 Summary: Add ConfigProvider to make use of environment variables
 Key: KAFKA-14376
 URL: https://issues.apache.org/jira/browse/KAFKA-14376
 Project: Kafka
  Issue Type: Improvement
  Components: config
Reporter: Roman Schmitz


So far it is not possible to inject additional configurations stored in 
environment variables. This topic came up in several projects and would be a 
useful feature to have as a Kafka config feature similar to file/directory 
providers, e.g.:

{{config.providers=env}}
{{{}config.providers.env.class=org.apache.kafka.common.config.provider.EnvVarConfigProvider{}}}{{{}ssl.key.password=${env:<...>:KEY_PASSPHRASE}{}}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka adopt two measurement systems

[Jun Rao]

Hi, Zhanchang,

We started with Yammer metrics on the broker side. When we developed java
clients, we wanted to reduce external dependencies. So, we added Kafka
metric and used it in the clients. Eventually, we could consider
consolidating the broker side metrics with Kafka metrics too. Currently,
the Histogram support in Kafka metric is not as accurate as the one in
Yammer. We probably need to evaluate whether it needs to be improved before
using it more widely.

Thanks,

Jun

On Wed, Nov 9, 2022 at 7:14 AM 张占昌  wrote:

> Hello!
> Why does Kafka adopt two measurement systems? One is yammer, and the other
> is its own metric.

PR 12343 review/merge

[Jonathan Albrecht]

Hi All,

I have a PR https://github.com/apache/kafka/pull/12343 that was approved in 
June but has not been merged yet. The PR fixes some issues with running on IBM 
Semeru JDKs.

I've rebased to trunk and the Java 17/Scala 2.13 and Java 8/Scala 2.12 tests 
have all passed. There were two test failures in the Java 11/Scala 2.13 build 
but they look unrelated to the changes and they pass in my local Java 11/Scala 
2.13 build. See: 
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-12343/5/pipeline

Could someone have a look when they have time?

Thanks,

Jonathan Albrecht
Advisory Software Developer
Linux on IBM Z Open Source Ecosystem
1 905 413 3577 Office
jonathan.albre...@ibm.com

IBM

Re: [VOTE] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[Greg Harris]

+1 (non-binding)

Thanks for the KIP, this is an important improvement.

Greg

On Thu, Nov 10, 2022 at 7:21 AM John Roesler  wrote:

> Thanks for the KIP, Daniel!
>
> I'm no MM expert, but I've read over the KIP and discussion, and it seems
> reasonable to me.
>
> I'm +1 (binding).
>
> Thanks,
> -John
>
> On 2022/10/22 07:38:38 Urbán Dániel wrote:
> > Hi everyone,
> >
> > I would like to start a vote on KIP-710 which aims to support running a
> > dedicated MM2 cluster in distributed mode:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> >
> > Regards,
> > Daniel
> >
> >
> > --
> > Ezt az e-mailt átvizsgálta az Avast AntiVirus szoftver.
> > www.avast.com
> >
>

[jira] [Created] (KAFKA-14375) Remove use of "authorizer-properties" n EndToEndAuthorizationTest.scala

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14375:
-

 Summary: Remove use of "authorizer-properties" n 
EndToEndAuthorizationTest.scala
 Key: KAFKA-14375
 URL: https://issues.apache.org/jira/browse/KAFKA-14375
 Project: Kafka
  Issue Type: Improvement
  Components: unit tests
Reporter: Proven Provenzano


The use of {{authorizer-properties}} in AclCommand is deprecated and 
EndToEndAuthroiztionTest.scala should be updated to not use it. 

I will instead set {{kafkaPrincipal}} as a super user and set up the brokers 
with AclAuthorzier. This will allow {{kafkaPrincipal}} to set ACLs and 
clientPrincipal to validate them as per the tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Sophie Blee-Goldman]

Hello again,

This is a reminder that the KIP freeze deadline is approaching, all KIPs
must be voted
and accepted by *next Wednesday* *(the 16th)*

Keep in mind that to allow for the full voting period, this means you must
kick off the
vote for your KIP no later than* next Monday* (*the 14th*).

The feature freeze deadline will be 2 weeks after this, so make sure to get
your KIPs in!

Best,
Sophie

On Tue, Oct 18, 2022 at 2:01 PM Sophie Blee-Goldman 
wrote:

> Hey all,
>
> I've created the release page for 3.4.0 with the current plan, which you
> can find here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.4.0
>
> The freeze deadlines for this release are as follows:
>
> 1. KIP Freeze November 16th, 2022
> 2. Feature Freeze November 30th, 2022
> 3. Code Freeze December 7th, 2022
>
> Please take a look at the list of planned KIPs for 3.4.0 and let me know
> if you have any
> others that you are targeting in the upcoming release.
>
> Cheers,
> Sophie
>
> On Mon, Oct 10, 2022 at 9:22 AM Matthew Benedict de Detrich
>  wrote:
>
>> Thanks for volunteering!
>>
>> On Mon, 10 Oct 2022, 10:56 Bruno Cadonna,  wrote:
>>
>> > +1
>> >
>> > Thanks Sophie!
>> >
>> > Best,
>> > Bruno
>> >
>> > On 06.10.22 00:01, Sophie Blee-Goldman wrote:
>> > > Hey all,
>> > >
>> > > I'd like to volunteer as release manager for the next feature release,
>> > > which will be Apache
>> > > Kafka 3.4.0. If that sounds good to everyone I'll update this thread
>> with
>> > > the release plan in the coming week.
>> > >
>> > > Cheers,
>> > > A. Sophie Blee-Goldman
>> > >
>> >
>>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1348

[Apache Jenkins Server]

See 

Jackson CVE's in Mirror Maker 2.13-2.8.2

[Andrew Pomponio]

Hello Kafka Developers,

I was wondering if there are any plans to back port fixes for certain CVE’s 
found in Mirror Maker 2.13-2.8.2. Scans of the code found the following 
unpatched CVE’s:


  *   CVE-2022-42004
  *   CVE-2022-42003
  *   CVE-2020-36518

It’s my understanding that there’s going to be a hot fix coming out for 
2.13-2.8.2 and I was wondering if those CVE’s will be addressed. Any 
information is greatly appreciated. Thanks!



Andrew Pomponio | Associate Enterprise Architect, 
OpenLogic
Perforce 
Software
P: +1 612.517.2100 
Visit us on: 
LinkedIn
 | 
Twitter
 | 
Facebook
 | 
YouTube

Use our new Community portal to submit/track support 
cases!



This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Re: [VOTE] KIP-710: Full support for distributed mode in dedicated MirrorMaker 2.0 clusters

[John Roesler]

Thanks for the KIP, Daniel!

I'm no MM expert, but I've read over the KIP and discussion, and it seems 
reasonable to me.

I'm +1 (binding).

Thanks,
-John

On 2022/10/22 07:38:38 Urbán Dániel wrote:
> Hi everyone,
> 
> I would like to start a vote on KIP-710 which aims to support running a 
> dedicated MM2 cluster in distributed mode:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> 
> Regards,
> Daniel
> 
> 
> -- 
> Ezt az e-mailt átvizsgálta az Avast AntiVirus szoftver.
> www.avast.com
> 

[jira] [Created] (KAFKA-14374) Kafka streams losing messages in State Store during first launch of app

[Youcef Sebiat (Jira)]

Youcef Sebiat created KAFKA-14374:
-

 Summary: Kafka streams losing messages in State Store during first 
launch of app
 Key: KAFKA-14374
 URL: https://issues.apache.org/jira/browse/KAFKA-14374
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.2.0, 2.8.0
Reporter: Youcef Sebiat
 Attachments: Screenshot 2022-11-09 at 14.56.00.png

We have been using Kafka Streams to implement a CDC based app. Attached is the 
sub topology of interest.

`table2` topic is created by Debezium who is connected to a SQL DB. It contains 
26K lines. We take `table2` and create a key which is only a conversion of the 
key from `string` to `int`. This means that we should expect that 
#table2=#repartition-topic=#state-store; which actually is not verified. What 
we end up with is the following #table2=#repartition-topic, but  
#repartition-topic>#state-store. We actually lose messages and thus corrupt the 
state store, which makes the app live in incorrect state. (Please note that 
there is no insertion in `table2` as we paused the connector to verify the 
cardinality.)

The above happens only during the first launch, i.e. the app has never been 
launched before, so internal topics do not exist yet. Restarts of pre-existing 
apps do not yield any problems.

We have:
1. Broker on Kafka 3.2.
2. Client app on 2.8|3.2 (we tried both and we faced the same issue).
2. All parameters are by default except `CACHE_MAX_BYTES_BUFFERING_CONFIG` set 
to `0` and `NUM_STREAM_THREADS_CONFIG` set to `>1`.

 

*What actually worked*
1. Use a monothread at first launch: using one thread solves the problem. The 
#table2=#repartition-topic=#state-store is verified.
2. Pre-creating kafka internal topics: we noticed that whenever there is 
rebalance during the first launch of Kafka Streams app, the state stores ended 
up missing values. This also happens when you launch multiple pods in K8s for 
example. When we read through the logs, we noticed that there is a rebalance 
that is triggered when we first launch the app. This comes from the fact that 
the internal topics get created and assigned, thus the rebalance. So by 
creating the internal topics before, we avoid the rebalance and we end up by 
#table2=#repartition-topic=#state-store.


*What we noticed from the logs*
On multi-thread mode, we noticed that it is the partitions that are assigned to 
the thread chosen by the Coordinator to be the Leader of consumers that suffer 
the data loss. What we think is happening is the following:
1. Consumers threads are launched and inform the coordinator.
2. Coordinator assign topics and choses the Leader among the threads.
3. App create internal topics.
4. Consumers/producers process data. Specifically the Consumer leader consumes 
from the repartition topic, which triggers the delete of those messages without 
flushing them to changelog topic.
5. Leader notified of new assignment with internal topics. Triggers rebalance.
6. Leader pauses partitions. 
7. Rebalance finished. The leader resumes partitions.
8. Leader fetches the oldest offset of repartition partitions he got assigned. 
He will not start from zero, but instead from where he got interrupted in 4. 
The chunk of early messages are thus lost.

Please note, that on mono-thread mode, there is no data loss which is weird 
since the leader is actually the unique thread. 

So my questions are:
1. Are we understanding wrongly what's happening?
2. What can be the origin of this problem?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Recall: Review request - PR#12753

[Sunnappugunta, Satheesh]

Sunnappugunta, Satheesh would like to recall the message, "[External] Review 
request - PR#12753".

RE: Review request - PR#12753

[Sunnappugunta, Satheesh]

Sure Sarbik

Thanks & Regards,
Satheesh S
LTS Integration - TIBCO
Email: ss...@allstate.com 
Ph.: +91-8105295295

-Original Message-
From: Dan S  
Sent: Thursday, November 10, 2022 1:17 PM
To: dev@kafka.apache.org
Subject: [External] Review request - PR#12753

Hello,

I would really appreciate another review on
https://urldefense.com/v3/__https://github.com/apache/kafka/pull/12753/files__;!!IIU9BLNPZ2ob!Lq4qqXr9vScv1zxA8V9x2lAUUg7QHTRpL3CD3Xl2RnOJFnAyeC63qJL-x2l94lt5TYutq4DCI4Y9BNBWbZY$
   as I think it would be
great to add a bit more documentation on the behaviour of seek, as well as some 
tests around invalid offsets (I found this very confusing when developing for 
it).

Thanks,

Dan