[jira] [Created] (KAFKA-14906) Extract the coordinator service log from server log

[hudeqi (Jira)]

hudeqi created KAFKA-14906:
--

 Summary: Extract the coordinator service log from server log
 Key: KAFKA-14906
 URL: https://issues.apache.org/jira/browse/KAFKA-14906
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.3.2
Reporter: hudeqi


Currently, the coordinator service log and server log are mixed together. When 
troubleshooting the coordinator problem, it is necessary to filter from the 
server log, which is not very convenient. Therefore, the coordinator log is 
separated like the controller log.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14139) Replaced disk can lead to loss of committed data even with non-empty ISR

[Calvin Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Calvin Liu resolved KAFKA-14139.

Resolution: Fixed

> Replaced disk can lead to loss of committed data even with non-empty ISR
> 
>
> Key: KAFKA-14139
> URL: https://issues.apache.org/jira/browse/KAFKA-14139
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Calvin Liu
>Priority: Major
> Fix For: 3.5.0
>
>
> We have been thinking about disk failure cases recently. Suppose that a disk 
> has failed and the user needs to restart the disk from an empty state. The 
> concern is whether this can lead to the unnecessary loss of committed data.
> For normal topic partitions, removal from the ISR during controlled shutdown 
> buys us some protection. After the replica is restarted, it must prove its 
> state to the leader before it can be added back to the ISR. And it cannot 
> become a leader until it does so.
> An obvious exception to this is when the replica is the last member in the 
> ISR. In this case, the disk failure itself has compromised the committed 
> data, so some amount of loss must be expected.
> We have been considering other scenarios in which the loss of one disk can 
> lead to data loss even when there are replicas remaining which have all of 
> the committed entries. One such scenario is this:
> Suppose we have a partition with two replicas: A and B. Initially A is the 
> leader and it is the only member of the ISR.
>  # Broker B catches up to A, so A attempts to send an AlterPartition request 
> to the controller to add B into the ISR.
>  # Before the AlterPartition request is received, replica B has a hard 
> failure.
>  # The current controller successfully fences broker B. It takes no action on 
> this partition since B is already out of the ISR.
>  # Before the controller receives the AlterPartition request to add B, it 
> also fails.
>  # While the new controller is initializing, suppose that replica B finishes 
> startup, but the disk has been replaced (all of the previous state has been 
> lost).
>  # The new controller sees the registration from broker B first.
>  # Finally, the AlterPartition from A arrives which adds B back into the ISR 
> even though it has an empty log.
> (Credit for coming up with this scenario goes to [~junrao] .)
> I tested this in KRaft and confirmed that this sequence is possible (even if 
> perhaps unlikely). There are a few ways we could have potentially detected 
> the issue. First, perhaps the leader should have bumped the leader epoch on 
> all partitions when B was fenced. Then the inflight AlterPartition would be 
> doomed no matter when it arrived.
> Alternatively, we could have relied on the broker epoch to distinguish the 
> dead broker's state from that of the restarted broker. This could be done by 
> including the broker epoch in both the `Fetch` request and in 
> `AlterPartition`.
> Finally, perhaps even normal kafka replication should be using a unique 
> identifier for each disk so that we can reliably detect when it has changed. 
> For example, something like what was proposed for the metadata quorum here: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1762

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14905) Failing tests in MM2 ForwardingAdmin test since KIP-894

[Greg Harris (Jira)]

Greg Harris created KAFKA-14905:
---

 Summary: Failing tests in MM2 ForwardingAdmin test since KIP-894
 Key: KAFKA-14905
 URL: https://issues.apache.org/jira/browse/KAFKA-14905
 Project: Kafka
  Issue Type: Test
  Components: mirrormaker
Reporter: Greg Harris
 Fix For: 3.5.0


There are three tests which are consistently failing in 
MirrorConnectorsWithCustomForwardingAdminIntegrationTest since the merge of 
KIP-894 in KAFKA-14420:

* testReplicationIsCreatingTopicsUsingProvidedForwardingAdmin()
* testCreatePartitionsUseProvidedForwardingAdmin()
* testSyncTopicConfigUseProvidedForwardingAdmin()



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1761

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 465544 lines...]
[2023-04-13T23:37:59.599Z] 
[2023-04-13T23:37:59.599Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() STARTED
[2023-04-13T23:38:06.444Z] 
[2023-04-13T23:38:06.444Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > GlobalThreadShutDownOrderTest > 
shouldFinishGlobalStoreOperationOnShutDown() PASSED
[2023-04-13T23:38:07.376Z] 
[2023-04-13T23:38:07.376Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailStopped() STARTED
[2023-04-13T23:38:07.376Z] 
[2023-04-13T23:38:07.376Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailStopped() PASSED
[2023-04-13T23:38:07.376Z] 
[2023-04-13T23:38:07.376Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) STARTED
[2023-04-13T23:38:09.415Z] 
[2023-04-13T23:38:09.415Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > 
shouldNotRequireQueryHandler(TestInfo) PASSED
[2023-04-13T23:38:09.415Z] 
[2023-04-13T23:38:09.415Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailNotStarted() STARTED
[2023-04-13T23:38:09.415Z] 
[2023-04-13T23:38:09.415Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailNotStarted() PASSED
[2023-04-13T23:38:09.415Z] 
[2023-04-13T23:38:09.415Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFetchFromPartition() STARTED
[2023-04-13T23:38:11.624Z] 
[2023-04-13T23:38:11.624Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFetchFromPartition() PASSED
[2023-04-13T23:38:11.624Z] 
[2023-04-13T23:38:11.624Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() STARTED
[2023-04-13T23:38:14.776Z] 
[2023-04-13T23:38:14.776Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > 
shouldFetchExplicitlyFromAllPartitions() PASSED
[2023-04-13T23:38:14.776Z] 
[2023-04-13T23:38:14.776Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailUnknownStore() STARTED
[2023-04-13T23:38:14.776Z] 
[2023-04-13T23:38:14.776Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldFailUnknownStore() PASSED
[2023-04-13T23:38:14.776Z] 
[2023-04-13T23:38:14.776Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldRejectNonRunningActive() STARTED
[2023-04-13T23:38:18.375Z] 
[2023-04-13T23:38:18.375Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > IQv2IntegrationTest > shouldRejectNonRunningActive() PASSED
[2023-04-13T23:38:20.416Z] 
[2023-04-13T23:38:20.416Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldCompactTopicsForKeyValueStoreChangelogs() STARTED
[2023-04-13T23:38:22.633Z] 
[2023-04-13T23:38:22.633Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldCompactTopicsForKeyValueStoreChangelogs() PASSED
[2023-04-13T23:38:22.633Z] 
[2023-04-13T23:38:22.633Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldGetToRunningWithWindowedTableInFKJ() STARTED
[2023-04-13T23:38:26.942Z] 
[2023-04-13T23:38:26.942Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldGetToRunningWithWindowedTableInFKJ() PASSED
[2023-04-13T23:38:26.942Z] 
[2023-04-13T23:38:26.942Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldCompactAndDeleteTopicsForWindowStoreChangelogs() STARTED
[2023-04-13T23:38:28.987Z] 
[2023-04-13T23:38:28.987Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > InternalTopicIntegrationTest > 
shouldCompactAndDeleteTopicsForWindowStoreChangelogs() PASSED
[2023-04-13T23:38:31.026Z] 
[2023-04-13T23:38:31.026Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationIntegrationTest > 
shouldAggregateSlidingWindows(TestInfo) STARTED
[2023-04-13T23:38:38.690Z] 
[2023-04-13T23:38:38.690Z] Gradle Test Run :streams:integrationTest > Gradle 
Test Executor 181 > KStreamAggregationIntegrationTest > 
shouldAggregateSlidingWindows(TestInfo) PASSED
[2023-04-13T23:38:38.690Z] 
[2023-04-13T23:38:38.690Z] Gradle Test Run :streams:integrationTest > 

Re: [DISCUSS] Re-visit end of life policy

[Ismael Juma]

Clarification below.

I did not understand your point about maintenance expense to ensure
> compatibility. I am confused because, IMO, irrespective of our bug fix
> support duration for minor versions, we should ensure that all prior minor
> versions are compatible. Hence, increasing the support duration to 24
> months will not add more expense than today to ensure compatibility.
>

No, I am not saying that. I am saying that there is no reason not to
upgrade from one minor release to another since we provide full
compatibility between minor releases. The expensive part is that we release
3 times a year, so you have to support 6 releases at any given point in
time. More importantly, you have to validate all these releases, handle any
additional bugs and so on. When it comes to the CVE stuff, you also have to
deal with cases where a project you depend on forces an upgrade to a
release with compatibility impact and so on. Having seen this first hand,
it's a significant amount of work.

Ismael

Re: Fwd: [VOTE] KIP-914 Join Processor Semantics for Versioned Stores

[Victoria Xia]

Hey everyone,

Two more minor updates to the KIP came out of wrapping up the
implementation and discussions with Guozhang and Matthias offline:

   - Versioned stores will be disabled for global tables and also the
   suppress operator, in order to limit the scope of these changes and to
   prevent unexpected semantic consequences.
   - The return type of `VersionedKeyValueStore#put(...)` has been updated
   from `void` to `long`, where the long represents the validTo timestamp of
   the newly put record, and may also be used to determine whether the call to
   put() was accepted or not (due to grace period having been expired). This
   is necessary in order to allow processors to determine whether records are
   out-of-order or not.

The KIP has been updated with more details about each of the two changes.
Barring objections, the current state of the KIP is what will be released
with 3.5 as the release deadline is fast approaching.

Thanks,
Victoria

On Tue, Apr 11, 2023 at 2:56 PM Victoria Xia 
wrote:

> Thanks for your comments and suggestions, Matthias, Lucas, and Guozhang!
>
> I was just in the process of responding when I saw Guozhang's message. I
> came up with a different approach to simplify my proposal with regards to
> the table aggregate processor, as part of mulling over comments from
> Matthias and Lucas: when aggregating a versioned table, instead of dropping
> out-of-order records at the aggregate processor (after the repartition), we
> can equivalently drop them before the repartition at the repartition map
> processor. With this new approach, no changes to the repartition topic
> format are necessary as part of this KIP.
>
> As an example, consider a table aggregation which counts the number of
> appearances of each value of the table. The repartition prior to the
> aggregation will map values (pre-repartition) to keys (post-repartition) as
> part of the groupBy(), in order for subsequent aggregation over the
> (original) values to occur.
>
> source record --> repartition record(s)
> (key=k, value=v1, ts=2) --> (key=v1, newValue=v1, oldValue=null, newTs=2,
> oldTs=-1)
> (key=k, value=v2, ts=1) --> (key=v2, newValue=v2, oldValue=null, newTs=1,
> oldTs=2), (key=v1, newValue=null, oldValue=v1, newTs=1, oldTs=2)
>
> Under the old proposal, the aggregate processor would see the last two
> repartition records and drop them as out-of-order (because newTs is older
> than oldTs). Under the new proposal, the repartition map processor would
> not send any repartition records upon seeing the second source record (with
> value=v2) because its timestamp is older than the latest timestamp seen for
> the key (k).
>
> This new approach is simpler than what's currently written in the KIP
> because no repartition topic format change is required, and also saves on
> unnecessary repartition topic records.
>
> In comparing this suggestion to Guozhang's suggestion:
>
>- A main difference between the two proposals is that under this
>proposal, the "old value" when an out-of-order record is forwarded
>downstream from a versioned table would be the current latest (by
>timestamp) value, rather than the same as the new value (as in Guozhang's
>proposal).
>- In both proposals, the various processors (table aggregate, joins,
>suppress) actually do not need to call get(key) on the materialization to
>determine whether the record is out-of-order or not. If the "old value" is
>the latest record, then a timestamp comparison would suffice (assuming that
>the old record timestamp is added into the `Change` object). If the "old
>value" is the same as the "new value", then an equality check on the two
>values is sufficient.
>- In both proposals, the repartition topic format does not need to be
>updated.
>
> I think regardless of which of the two implementations we go with, the net
> effect will be hidden from users, in which case it may be better to discuss
> which to pick as part of implementation rather than on this KIP itself. (I
> can tell I will need more time to process the tradeoffs :-) ) Regardless, I
> will update the KIP to reflect that no repartition topic format change is
> required, which is indeed a great simplification.
>
> > for the repartition topic format change, do we want to re-use flag=2, or
> should we introduce flag=3, and determine when compiling the DSL into the
> Topology if we want/need to include the timestamp, and if not, use format
> version=2 to avoid unnecessary overhead?
>
> I believe the new proposal above is even more efficient in terms of
> avoiding unnecessary overhead. LMK what you think.
>
> > for detecting out-of-order records, I need both new and old timestamp, I
> suppose I get it for the new record via timestamp extractor, can I not get
> it the same way from the old record that is passed down to the aggregation
> after KIP-904?
>
> Also subsumed by the updated proposal above, but I think the answer is not
> necessarily, both 

[jira] [Created] (KAFKA-14904) Flaky Test kafka.api.TransactionsBounceTest.testWithGroupId()

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-14904:
--

 Summary: Flaky Test  
kafka.api.TransactionsBounceTest.testWithGroupId()
 Key: KAFKA-14904
 URL: https://issues.apache.org/jira/browse/KAFKA-14904
 Project: Kafka
  Issue Type: Test
Reporter: Justine Olshan
Assignee: Justine Olshan


After merging KAFKA-14561 I noticed this test still occasionally failed via 

org.apache.kafka.common.errors.TimeoutException: Timeout expired after 6ms 
while awaiting EndTxn(true)

I will investigate the cause. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Max Multi Tenancy topic naming - Cloudflare

[Kenneth Eversole]

Hello,

I am on the Kafka team at Cloudflare, and I was hoping to find an answer on
what the max value is for the following subject,
https://kafka.apache.org/documentation/#multitenancy-topic-naming. We are
looking this for a workload that would bring us in the 100,000s of topics.


I look forward to hearing from you.

Best regards,

Kenneth

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1760

[Apache Jenkins Server]

See 

Re: [DISCUSS] Re-visit end of life policy

[Divij Vaidya]

Thank you for your comments, Ismael.

1.  About the last major version release: There is precedence in open
source projects such as Spark, Pulsar etc. that are already offering a
LTS/major version support with longer support duration than Kafka. If we
limit our support to include production impacting bugs with no workaround,
do you still think that we will have to invest

2. About the support of minor versions:
2.1 - Just to make sure that we are on the same page, the cost we are
talking about is the engineering effort required to backport changes and
perform a release. Is there anything else that I am missing? If yes, is
that cost acceptable in future as we increase our number of committers?
2.2 - We ensure compatibility (this is another thing that we should
formalize and document) amongst all minor versions within a major version.
I did not understand your point about maintenance expense to ensure
compatibility. I am confused because, IMO, irrespective of our bug fix
support duration for minor versions, we should ensure that all prior minor
versions are compatible. Hence, increasing the support duration to 24
months will not add more expense than today to ensure compatibility.

Apart from the above points, Ismael, what do you think about having
different durations of security support and bug fix support?

--
Divij Vaidya



On Thu, Apr 13, 2023 at 4:13 PM Ismael Juma  wrote:

> Hi,
>
> I think we're discussing two separate things and each has a very different
> cost and benefit:
>
> 1. Having a longer support period for the last minor release: major
> releases happen pretty rarely and have breaking changes. It seems
> reasonable to consider improving how this case is handled. And it will be
> especially important for the upcoming AK 4.0 where support for ZooKeeper
> will be removed. That said, 24 months is a very long time for an open
> source project - it would divert resources away from innovation.
>
> 2. Having a longer support period for any minor release: Apache Kafka
> maintains compatibility across minor releases and it would be very
> expensive to maintain each of them (we have one every 4 months) for 12
> months. The cost vs benefit for this one doesn't add up for me.
>
> Ismael
>
> On Thu, Apr 13, 2023 at 6:02 AM Divij Vaidya 
> wrote:
>
> > Hi folks
> >
> > The goal of this discussion is to re-visit the End Of Life (EOL) policy
> of
> > Kafka and introduce a few changes to it.
> >
> > *Current EOL policy*
> > Kafka currently follows a time based release plan with an EOL policy
> > documented here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy
> > ?
> >
> >
> > *Limitations of current policy*
> >
> > The current EOL policy suffers from limitations such as:
> > 1. It does not differentiate between support for bug fixes and support
> for
> > security vulnerability fixes. This leads to confusion for the users as
> some
> > CVEs  such as CVE-2023-25194 are only
> > fixed in latest versions and others such as CVE-2022-34917 are fixed even
> > in the previous major versions.
> > 2. Users find it difficult to upgrade major versions within a span of a
> > year. For example, 2.x to 3.x brought multiple changes. For such major
> > upgrades, users require more time than 12 months to schedule resources
> for
> > upgrade, change the configuration (since defaults have changed), ensure
> > compatibility with tools and test for any performance changes.
> > 3. Our current policy mentions both a time based and a version based
> > support period. We should clarify and confirm whether the support policy
> is
> > based on time or number of releases.
> >
> > *Kafka EOL policy*
> >
> > To overcome the above problems, I would like to propose the following
> > changes to end of life policy.
> >
> > "Minor versions within the latest major version will be maintained with
> bug
> > fix releases for a period of 12 months. For example, 3.3.x was first
> > released in Feb 2023 and will receive bug fixes until Feb 2024. The bug
> fix
> > release will be performed as-needed. Only security vulnerabilities and
> > production-impacting bugs without workarounds are backported to previous
> > minor versions. Missing features from latest releases are not backported
> by
> > default. Exceptions are approved by Lazy Majority."
> > Note that this is the same as current policy.
> >
> > "The last minor version within the previous major version will be
> > maintained with bug fix releases for a period of 24 months. For example,
> > 2.8.x was first released in Apr 2021 and will receive bug fixes until Apr
> > 2022. The bug fix release will be performed as-needed. Only security
> > vulnerabilities and production-impacting bugs without workarounds are
> added
> > to bug fix versions."
> > This is a new policy that provides users with 24 months (as compared to
> 12
> > months today) to upgrade their major version.
> >
> > "The 

Re: [VOTE] KIP-909: DNS Resolution Fallures Should Not Fail the Client

[Kirk True]

+1 (non-binding)

> On Apr 10, 2023, at 1:53 PM, Philip Nee  wrote:
> 
> Hey everyone!
> 
> I'm starting a vote for KIP-909: DNS Resolution Fallures Should Not Fail
> the Client 
> 
> Please refer to the discussion thread here:
> https://lists.apache.org/thread/st84zzwnq5m3pkzd1r7jk9lmqdt9m98s
> 
> Thanks!
> P

Re: [DISCUSS] Re-visit end of life policy

[Ismael Juma]

Hi,

I think we're discussing two separate things and each has a very different
cost and benefit:

1. Having a longer support period for the last minor release: major
releases happen pretty rarely and have breaking changes. It seems
reasonable to consider improving how this case is handled. And it will be
especially important for the upcoming AK 4.0 where support for ZooKeeper
will be removed. That said, 24 months is a very long time for an open
source project - it would divert resources away from innovation.

2. Having a longer support period for any minor release: Apache Kafka
maintains compatibility across minor releases and it would be very
expensive to maintain each of them (we have one every 4 months) for 12
months. The cost vs benefit for this one doesn't add up for me.

Ismael

On Thu, Apr 13, 2023 at 6:02 AM Divij Vaidya 
wrote:

> Hi folks
>
> The goal of this discussion is to re-visit the End Of Life (EOL) policy of
> Kafka and introduce a few changes to it.
>
> *Current EOL policy*
> Kafka currently follows a time based release plan with an EOL policy
> documented here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy
> ?
>
>
> *Limitations of current policy*
>
> The current EOL policy suffers from limitations such as:
> 1. It does not differentiate between support for bug fixes and support for
> security vulnerability fixes. This leads to confusion for the users as some
> CVEs  such as CVE-2023-25194 are only
> fixed in latest versions and others such as CVE-2022-34917 are fixed even
> in the previous major versions.
> 2. Users find it difficult to upgrade major versions within a span of a
> year. For example, 2.x to 3.x brought multiple changes. For such major
> upgrades, users require more time than 12 months to schedule resources for
> upgrade, change the configuration (since defaults have changed), ensure
> compatibility with tools and test for any performance changes.
> 3. Our current policy mentions both a time based and a version based
> support period. We should clarify and confirm whether the support policy is
> based on time or number of releases.
>
> *Kafka EOL policy*
>
> To overcome the above problems, I would like to propose the following
> changes to end of life policy.
>
> "Minor versions within the latest major version will be maintained with bug
> fix releases for a period of 12 months. For example, 3.3.x was first
> released in Feb 2023 and will receive bug fixes until Feb 2024. The bug fix
> release will be performed as-needed. Only security vulnerabilities and
> production-impacting bugs without workarounds are backported to previous
> minor versions. Missing features from latest releases are not backported by
> default. Exceptions are approved by Lazy Majority."
> Note that this is the same as current policy.
>
> "The last minor version within the previous major version will be
> maintained with bug fix releases for a period of 24 months. For example,
> 2.8.x was first released in Apr 2021 and will receive bug fixes until Apr
> 2022. The bug fix release will be performed as-needed. Only security
> vulnerabilities and production-impacting bugs without workarounds are added
> to bug fix versions."
> This is a new policy that provides users with 24 months (as compared to 12
> months today) to upgrade their major version.
>
> "The last minor version within the previous major version will be
> maintained with security fixes for a period of 3 years. For example, 2.8.x
> was first released in Apr 2021 and will receive security fixes until Apr
> 2024."
> This is a new policy inspired from the concept of LTS releases in open
> source projects such as Java, Ubuntu and Linux Kernel. Apache projects such
> as Spark [1] follow this policy.
>
> *EOL policy of other projects*
>
> [1] Spark [https://spark.apache.org/versioning-policy.html] - Feature
> branches / minor versions are maintained with bug fixes for 18 months and
> major versions are maintained for 31 months.
> [2] Airflow [
>
> https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html#version-life-cycle
> ]
> - 1 year support for each minor version.
> [3] Kubernetes [
> https://kubernetes.io/releases/patch-releases/#support-period] - Minor
> versions are maintained for 14 months.
> [4] Flink [
> https://flink.apache.org/downloads/#update-policy-for-old-releases] -
> Supports 2 minor versions with bug fixes. Mandates a patch release for
> minor versions going end of life.
> [5] Pulsar [https://pulsar.apache.org/contribute/release-policy/] -
> Different security and bug fix policy. Has a concept of LTS release. LTS
> receives bug fixes for 24 months and security fixes for 36 months.
>
>
> As our Bylaws
>  >
> don't
> mention the voting action required for changing EOL policy, I would propose
> a vote for this change 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1759

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 467254 lines...]
[2023-04-13T13:34:51.100Z] > Task :server-common:testClasses UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :raft:testClasses UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-04-13T13:34:51.100Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :connect:json:testJar
[2023-04-13T13:34:51.100Z] > Task :connect:json:testSrcJar
[2023-04-13T13:34:51.100Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task :metadata:testClasses UP-TO-DATE
[2023-04-13T13:34:51.100Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-04-13T13:34:54.489Z] 
[2023-04-13T13:34:54.489Z] > Task :connect:api:javadoc
[2023-04-13T13:34:54.489Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-04-13T13:34:56.309Z] 1 warning
[2023-04-13T13:34:56.309Z] 
[2023-04-13T13:34:56.309Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task :connect:api:jar UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-04-13T13:34:56.309Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task :connect:json:jar UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-04-13T13:34:56.309Z] > Task :connect:api:javadocJar
[2023-04-13T13:34:56.309Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-04-13T13:34:56.309Z] > Task :connect:json:publishToMavenLocal
[2023-04-13T13:34:56.309Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-04-13T13:34:56.309Z] > Task :connect:api:testJar
[2023-04-13T13:34:56.309Z] > Task :connect:api:testSrcJar
[2023-04-13T13:34:56.309Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-04-13T13:34:56.309Z] > Task :connect:api:publishToMavenLocal
[2023-04-13T13:34:59.606Z] > Task :streams:javadoc
[2023-04-13T13:34:59.606Z] > Task :streams:javadocJar
[2023-04-13T13:35:01.676Z] 
[2023-04-13T13:35:01.676Z] > Task :clients:javadoc
[2023-04-13T13:35:01.676Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API
[2023-04-13T13:35:01.676Z] 
[2023-04-13T13:35:01.676Z]  This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
[2023-04-13T13:35:01.676Z]  The type field in both files must match and must 
not change. The type field
[2023-04-13T13:35:01.676Z]  is used both for passing ScramCredentialUpsertion 
and for the internal
[2023-04-13T13:35:01.676Z]  UserScramCredentialRecord. Do not change the type 
field."
[2023-04-13T13:35:02.710Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-04-13T13:35:02.710Z] 2 warnings
[2023-04-13T13:35:03.745Z] 
[2023-04-13T13:35:03.745Z] > Task :clients:javadocJar
[2023-04-13T13:35:05.812Z] > Task :clients:srcJar
[2023-04-13T13:35:05.812Z] > Task :clients:testJar
[2023-04-13T13:35:06.846Z] > Task :clients:testSrcJar
[2023-04-13T13:35:06.846Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-04-13T13:35:06.846Z] > Task :clients:publishToMavenLocal
[2023-04-13T13:35:20.947Z] > Task :core:compileScala
[2023-04-13T13:36:59.568Z] > Task :core:classes
[2023-04-13T13:36:59.568Z] > Task :core:compileTestJava NO-SOURCE
[2023-04-13T13:37:30.475Z] > Task :core:compileTestScala
[2023-04-13T13:39:05.568Z] > Task :core:testClasses
[2023-04-13T13:39:05.568Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-04-13T13:39:05.568Z] > Task :streams:testClasses UP-TO-DATE
[2023-04-13T13:39:05.568Z] > Task :streams:testJar
[2023-04-13T13:39:05.568Z] > Task :streams:testSrcJar
[2023-04-13T13:39:05.568Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-04-13T13:39:05.568Z] > Task :streams:publishToMavenLocal
[2023-04-13T13:39:05.568Z] 
[2023-04-13T13:39:05.568Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.

[DISCUSS] KIP-918: MM2 Topic And Group Listener

[Dániel Urbán]

Hi everyone,

I would like to start a discussion on KIP-918: MM2 Topic And Group Listener
(
https://cwiki.apache.org/confluence/display/KAFKA/KIP-918%3A+MM2+Topic+And+Group+Listener
).
This new feature of MM2 would allow following the latest set of replicated
topics and groups, which is currently not possible in MM2. Additionally,
this would help IdentityReplicationPolicy users, as they could use this new
feature to track the replicated topics (which is not available through the
policy due to topics not being renamed during replication).

Thanks in advance,
Daniel

[DISCUSS] Re-visit end of life policy

[Divij Vaidya]

Hi folks

The goal of this discussion is to re-visit the End Of Life (EOL) policy of
Kafka and introduce a few changes to it.

*Current EOL policy*
Kafka currently follows a time based release plan with an EOL policy
documented here:
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy?


*Limitations of current policy*

The current EOL policy suffers from limitations such as:
1. It does not differentiate between support for bug fixes and support for
security vulnerability fixes. This leads to confusion for the users as some
CVEs  such as CVE-2023-25194 are only
fixed in latest versions and others such as CVE-2022-34917 are fixed even
in the previous major versions.
2. Users find it difficult to upgrade major versions within a span of a
year. For example, 2.x to 3.x brought multiple changes. For such major
upgrades, users require more time than 12 months to schedule resources for
upgrade, change the configuration (since defaults have changed), ensure
compatibility with tools and test for any performance changes.
3. Our current policy mentions both a time based and a version based
support period. We should clarify and confirm whether the support policy is
based on time or number of releases.

*Kafka EOL policy*

To overcome the above problems, I would like to propose the following
changes to end of life policy.

"Minor versions within the latest major version will be maintained with bug
fix releases for a period of 12 months. For example, 3.3.x was first
released in Feb 2023 and will receive bug fixes until Feb 2024. The bug fix
release will be performed as-needed. Only security vulnerabilities and
production-impacting bugs without workarounds are backported to previous
minor versions. Missing features from latest releases are not backported by
default. Exceptions are approved by Lazy Majority."
Note that this is the same as current policy.

"The last minor version within the previous major version will be
maintained with bug fix releases for a period of 24 months. For example,
2.8.x was first released in Apr 2021 and will receive bug fixes until Apr
2022. The bug fix release will be performed as-needed. Only security
vulnerabilities and production-impacting bugs without workarounds are added
to bug fix versions."
This is a new policy that provides users with 24 months (as compared to 12
months today) to upgrade their major version.

"The last minor version within the previous major version will be
maintained with security fixes for a period of 3 years. For example, 2.8.x
was first released in Apr 2021 and will receive security fixes until Apr
2024."
This is a new policy inspired from the concept of LTS releases in open
source projects such as Java, Ubuntu and Linux Kernel. Apache projects such
as Spark [1] follow this policy.

*EOL policy of other projects*

[1] Spark [https://spark.apache.org/versioning-policy.html] - Feature
branches / minor versions are maintained with bug fixes for 18 months and
major versions are maintained for 31 months.
[2] Airflow [
https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html#version-life-cycle]
- 1 year support for each minor version.
[3] Kubernetes [
https://kubernetes.io/releases/patch-releases/#support-period] - Minor
versions are maintained for 14 months.
[4] Flink [
https://flink.apache.org/downloads/#update-policy-for-old-releases] -
Supports 2 minor versions with bug fixes. Mandates a patch release for
minor versions going end of life.
[5] Pulsar [https://pulsar.apache.org/contribute/release-policy/] -
Different security and bug fix policy. Has a concept of LTS release. LTS
receives bug fixes for 24 months and security fixes for 36 months.


As our Bylaws

don't
mention the voting action required for changing EOL policy, I would propose
a vote for this change by Lazy Majority

.

Thoughts?

--
Divij Vaidya

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Dániel Urbán]

Hi everyone,

I would like to bump this thread. I think this would be very useful for any
MM2 users, as the current logs with certain architectures (e.g. fan-out)
are impossible to decipher.
I already submitted a PR to demonstrate the proposed solution:
https://github.com/apache/kafka/pull/13475

Thanks for your comments in advance,
Daniel

Dániel Urbán  ezt írta (időpont: 2023. márc. 30.,
Cs, 18:24):

> Hello everyone,
>
> I would like to kick off a discussion about KIP-916:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-916%3A+MM2+distributed+mode+flow+log+context
>
> The KIP aims to enhance the diagnostic information for MM2 distributed
> mode. MM2 relies on multiple Connect worker instances nested in a single
> process. In Connect, Connector names are guaranteed to be unique in a
> single process, but in MM2, this is not true. Because of this, the
> diagnostics provided by Connect (client.ids, log context) do not ensure
> that logs are distinguishable for different flows (Connect workers) inside
> an MM2 process.
>
> Thanks for all you input in advance,
> Daniel
>

[jira] [Created] (KAFKA-14903) MM2 Topic And Group Listener (KIP-918)

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-14903:


 Summary: MM2 Topic And Group Listener (KIP-918)
 Key: KAFKA-14903
 URL: https://issues.apache.org/jira/browse/KAFKA-14903
 Project: Kafka
  Issue Type: New Feature
Reporter: Daniel Urban
Assignee: Daniel Urban


MM2 has a dynamic topic and group filter mechanism, in which the replicated 
topics/groups can dynamically change, either due to changes in the available 
topics/groups, or changes in the filter settings.

In order to monitor the currently replicated topics/groups, MM2 should support 
a TopicListener and GroupListener plugin, which is triggered when MM2 changes 
the set of replicated topics/groups.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[David Jacot]

Hi Mickael,

Thanks for the heads up. As raised by Jeff earlier in this thread, we would
like to get the two small patches [1][2] for KIP-915 in 3.5. The PRs are in
review and I should be able to merge them in the next few days. I will
cherry-pick them to the release branch in your create it in the meantime.

[1] https://github.com/apache/kafka/pull/13511
[2] https://github.com/apache/kafka/pull/13526

Best,
David

On Thu, Apr 13, 2023 at 12:55 PM Mickael Maison 
wrote:

> Hi Luke,
>
> Thanks for the heads up. This would be great to get Tiered Storage in
> Early Access. Let me know if you can't get everything done this week.
>
> Mickael
>
> On Thu, Apr 13, 2023 at 12:54 PM Mickael Maison
>  wrote:
> >
> > Hi,
> >
> > We've now reached feature freeze for 3.5.0. From now on, only bug
> > fixes and changes related to stabilizing the release should be merged.
> >
> > I plan to create the release branch tomorrow (Friday 14). After this
> > point, you'll have to cherry pick changes to the release branch when
> > merging a PR. I'll send another message once the branch has been
> > created.
> >
> > I've updated the release plan and started moving KIPs that are not
> > complete to the postponed section. For now I've kept a few KIPs that
> > are still in progress. If they are not fully merged when I create a
> > branch, I'll mark them as postponed too.
> >
> > The next milestone is code freeze on April 26.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Apr 12, 2023 at 12:24 PM Luke Chen  wrote:
> > >
> > > Hi Mickael,
> > >
> > > I'd like to ask for some more days for KIP-405 tiered storage PRs to
> > > include in v3.5.
> > > Currently, we have 1 PR under reviewing (
> > > https://github.com/apache/kafka/pull/13535), and 1 PR soon will be
> opened
> > > for review.
> > > After these 2 PRs merged, we can have an "Early Access" for tiered
> storage
> > > feature, that allow users to use in non-production environments.
> > > Does that work for you?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Apr 6, 2023 at 2:49 AM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thank you.
> > > >
> > > > Best,
> > > > Jeff
> > > >
> > > > On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Ok, I've added KIP-915 to the release plan.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim
> 
> > > > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > I would like to bring up that KIP-915 proposes to patch 3.5
> > > > > > although it missed the KIP freeze date. If the patch is done
> before the
> > > > > > feature freeze date, 4/13, would this be acceptable? If so,
> should this
> > > > > > be added to the 3.5.0 Release Plan wiki?
> > > > > >
> > > > > > Best,
> > > > > > Jeff
> > > > > >
> > > > > > On Mon, Mar 27, 2023 at 1:02 PM Greg Harris
> > > >  > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Mickael,
> > > > > > >
> > > > > > > Just wanted to let you know that I will not be including
> KIP-898 in
> > > > the
> > > > > > > 3.5.0 release.
> > > > > > > I think the change needed is not reviewable before the feature
> freeze
> > > > > > > deadline, and would take resources away from other more
> necessary
> > > > > changes.
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Greg
> > > > > > >
> > > > > > > On Thu, Mar 23, 2023 at 9:01 AM Chia-Ping Tsai <
> chia7...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > > If you have a KIP that is accepted, make sure it is listed
> in
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > > > and that it's status is accurate.
> > > > > > > >
> > > > > > > > Thanks for the reminder. Have added KIP-641 to the list.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chia-Ping
> > > > > > > >
> > > > > > > > > Mickael Maison  於 2023年3月23日
> 下午11:51
> > > > 寫道：
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > KIP Freeze was yesterday. The next milestone is feature
> freeze on
> > > > > April
> > > > > > > > 12.
> > > > > > > > > If you have a KIP that is accepted, make sure it is listed
> in
> > > > > > > > >
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > > > and that it's status is accurate.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov <
> > > > > christolo...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Hello!
> > > > > > > > >>
> > > > > > > > >> What would you suggest as the best way to get more eyes on
> > > > > KIP-902 as
> > > > > > > I
> > > > > > > > would like it to be included it in 3.5.0?
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Christo
> > > > > > > > >>
> > > > > > > > >>> On 16 Mar 2023, at 10:33, 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi Luke,

Thanks for the heads up. This would be great to get Tiered Storage in
Early Access. Let me know if you can't get everything done this week.

Mickael

On Thu, Apr 13, 2023 at 12:54 PM Mickael Maison
 wrote:
>
> Hi,
>
> We've now reached feature freeze for 3.5.0. From now on, only bug
> fixes and changes related to stabilizing the release should be merged.
>
> I plan to create the release branch tomorrow (Friday 14). After this
> point, you'll have to cherry pick changes to the release branch when
> merging a PR. I'll send another message once the branch has been
> created.
>
> I've updated the release plan and started moving KIPs that are not
> complete to the postponed section. For now I've kept a few KIPs that
> are still in progress. If they are not fully merged when I create a
> branch, I'll mark them as postponed too.
>
> The next milestone is code freeze on April 26.
>
> Thanks,
> Mickael
>
> On Wed, Apr 12, 2023 at 12:24 PM Luke Chen  wrote:
> >
> > Hi Mickael,
> >
> > I'd like to ask for some more days for KIP-405 tiered storage PRs to
> > include in v3.5.
> > Currently, we have 1 PR under reviewing (
> > https://github.com/apache/kafka/pull/13535), and 1 PR soon will be opened
> > for review.
> > After these 2 PRs merged, we can have an "Early Access" for tiered storage
> > feature, that allow users to use in non-production environments.
> > Does that work for you?
> >
> > Thank you.
> > Luke
> >
> > On Thu, Apr 6, 2023 at 2:49 AM Jeff Kim 
> > wrote:
> >
> > > Hi Mickael,
> > >
> > > Thank you.
> > >
> > > Best,
> > > Jeff
> > >
> > > On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison 
> > > wrote:
> > >
> > > > Hi Jeff,
> > > >
> > > > Ok, I've added KIP-915 to the release plan.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim 
> > > > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > I would like to bring up that KIP-915 proposes to patch 3.5
> > > > > although it missed the KIP freeze date. If the patch is done before 
> > > > > the
> > > > > feature freeze date, 4/13, would this be acceptable? If so, should 
> > > > > this
> > > > > be added to the 3.5.0 Release Plan wiki?
> > > > >
> > > > > Best,
> > > > > Jeff
> > > > >
> > > > > On Mon, Mar 27, 2023 at 1:02 PM Greg Harris
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Mickael,
> > > > > >
> > > > > > Just wanted to let you know that I will not be including KIP-898 in
> > > the
> > > > > > 3.5.0 release.
> > > > > > I think the change needed is not reviewable before the feature 
> > > > > > freeze
> > > > > > deadline, and would take resources away from other more necessary
> > > > changes.
> > > > > >
> > > > > > Thanks!
> > > > > > Greg
> > > > > >
> > > > > > On Thu, Mar 23, 2023 at 9:01 AM Chia-Ping Tsai 
> > > > wrote:
> > > > > >
> > > > > > > > If you have a KIP that is accepted, make sure it is listed in
> > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > > and that it's status is accurate.
> > > > > > >
> > > > > > > Thanks for the reminder. Have added KIP-641 to the list.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > > Mickael Maison  於 2023年3月23日 下午11:51
> > > 寫道：
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > KIP Freeze was yesterday. The next milestone is feature freeze 
> > > > > > > > on
> > > > April
> > > > > > > 12.
> > > > > > > > If you have a KIP that is accepted, make sure it is listed in
> > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > > and that it's status is accurate.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov <
> > > > christolo...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hello!
> > > > > > > >>
> > > > > > > >> What would you suggest as the best way to get more eyes on
> > > > KIP-902 as
> > > > > > I
> > > > > > > would like it to be included it in 3.5.0?
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Christo
> > > > > > > >>
> > > > > > > >>> On 16 Mar 2023, at 10:33, Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> Hi,
> > > > > > > >>>
> > > > > > > >>> This is a reminder that KIP freeze is less than a week away 
> > > > > > > >>> (22
> > > > Mar).
> > > > > > > >>> For a KIP to be considered for this release, it must be voted
> > > and
> > > > > > > >>> accepted by that date.
> > > > > > > >>>
> > > > > > > >>> Feature freeze will be 3 weeks after this, so if you want KIPs
> > > or
> > > > > > > >>> other significant changes in the release, please get them 
> > > > > > > >>> ready
> > > > soon.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> > > > > > > >>> Mickael
> > > > > > > >>>
> > > > > > >  On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma <
> > > ism...@juma.me.uk
> > 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi,

We've now reached feature freeze for 3.5.0. From now on, only bug
fixes and changes related to stabilizing the release should be merged.

I plan to create the release branch tomorrow (Friday 14). After this
point, you'll have to cherry pick changes to the release branch when
merging a PR. I'll send another message once the branch has been
created.

I've updated the release plan and started moving KIPs that are not
complete to the postponed section. For now I've kept a few KIPs that
are still in progress. If they are not fully merged when I create a
branch, I'll mark them as postponed too.

The next milestone is code freeze on April 26.

Thanks,
Mickael

On Wed, Apr 12, 2023 at 12:24 PM Luke Chen  wrote:
>
> Hi Mickael,
>
> I'd like to ask for some more days for KIP-405 tiered storage PRs to
> include in v3.5.
> Currently, we have 1 PR under reviewing (
> https://github.com/apache/kafka/pull/13535), and 1 PR soon will be opened
> for review.
> After these 2 PRs merged, we can have an "Early Access" for tiered storage
> feature, that allow users to use in non-production environments.
> Does that work for you?
>
> Thank you.
> Luke
>
> On Thu, Apr 6, 2023 at 2:49 AM Jeff Kim 
> wrote:
>
> > Hi Mickael,
> >
> > Thank you.
> >
> > Best,
> > Jeff
> >
> > On Wed, Apr 5, 2023 at 1:28 PM Mickael Maison 
> > wrote:
> >
> > > Hi Jeff,
> > >
> > > Ok, I've added KIP-915 to the release plan.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Wed, Apr 5, 2023 at 6:48 PM Jeff Kim 
> > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I would like to bring up that KIP-915 proposes to patch 3.5
> > > > although it missed the KIP freeze date. If the patch is done before the
> > > > feature freeze date, 4/13, would this be acceptable? If so, should this
> > > > be added to the 3.5.0 Release Plan wiki?
> > > >
> > > > Best,
> > > > Jeff
> > > >
> > > > On Mon, Mar 27, 2023 at 1:02 PM Greg Harris
> >  > > >
> > > > wrote:
> > > >
> > > > > Mickael,
> > > > >
> > > > > Just wanted to let you know that I will not be including KIP-898 in
> > the
> > > > > 3.5.0 release.
> > > > > I think the change needed is not reviewable before the feature freeze
> > > > > deadline, and would take resources away from other more necessary
> > > changes.
> > > > >
> > > > > Thanks!
> > > > > Greg
> > > > >
> > > > > On Thu, Mar 23, 2023 at 9:01 AM Chia-Ping Tsai 
> > > wrote:
> > > > >
> > > > > > > If you have a KIP that is accepted, make sure it is listed in
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > and that it's status is accurate.
> > > > > >
> > > > > > Thanks for the reminder. Have added KIP-641 to the list.
> > > > > >
> > > > > > Thanks,
> > > > > > Chia-Ping
> > > > > >
> > > > > > > Mickael Maison  於 2023年3月23日 下午11:51
> > 寫道：
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > KIP Freeze was yesterday. The next milestone is feature freeze on
> > > April
> > > > > > 12.
> > > > > > > If you have a KIP that is accepted, make sure it is listed in
> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > > > and that it's status is accurate.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Fri, Mar 17, 2023 at 6:22 PM Christo Lolov <
> > > christolo...@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hello!
> > > > > > >>
> > > > > > >> What would you suggest as the best way to get more eyes on
> > > KIP-902 as
> > > > > I
> > > > > > would like it to be included it in 3.5.0?
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Christo
> > > > > > >>
> > > > > > >>> On 16 Mar 2023, at 10:33, Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>> Hi,
> > > > > > >>>
> > > > > > >>> This is a reminder that KIP freeze is less than a week away (22
> > > Mar).
> > > > > > >>> For a KIP to be considered for this release, it must be voted
> > and
> > > > > > >>> accepted by that date.
> > > > > > >>>
> > > > > > >>> Feature freeze will be 3 weeks after this, so if you want KIPs
> > or
> > > > > > >>> other significant changes in the release, please get them ready
> > > soon.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Mickael
> > > > > > >>>
> > > > > >  On Tue, Feb 14, 2023 at 10:44 PM Ismael Juma <
> > ism...@juma.me.uk
> > > >
> > > > > > wrote:
> > > > > > 
> > > > > >  Thanks!
> > > > > > 
> > > > > >  Ismael
> > > > > > 
> > > > > >  On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > >  wrote:
> > > > > > 
> > > > > > > Hi Ismael,
> > > > > > >
> > > > > > > Good call. I shifted all dates by 2 weeks and moved them to
> > > > > > Wednesdays.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma <
> > ism...@juma.me.uk
> > > >
> > > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1758

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14902) KafkaBasedLog infinite retries can lead to StackOverflowError

[Daniel Urban (Jira)]

Daniel Urban created KAFKA-14902:


 Summary: KafkaBasedLog infinite retries can lead to 
StackOverflowError
 Key: KAFKA-14902
 URL: https://issues.apache.org/jira/browse/KAFKA-14902
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Daniel Urban
Assignee: Daniel Urban


KafkaBasedLog subclasses use an infinite retry on producer sends, using a 
callback. Sometimes, when specific errors are encountered, the callback is 
invoked in the send call, on the calling thread. If this happens enough times, 
a stack overflow happens.

Example stacktrace from 2.5 (but the newest code can also encounter the same):
{code:java}
2023-01-14 12:48:23,487 ERROR org.apache.kafka.connect.runtime.WorkerTask: 
WorkerSourceTask{id=MirrorSourceConnector-1} Task threw an uncaught and 
unrecoverable exception java.lang.StackOverflowError: null at 
org.apache.kafka.common.metrics.stats.SampledStat.record(SampledStat.java:50) 
at org.apache.kafka.common.metrics.stats.Rate.record(Rate.java:60) at 
org.apache.kafka.common.metrics.stats.Meter.record(Meter.java:80) at 
org.apache.kafka.common.metrics.Sensor.record(Sensor.java:188) at 
org.apache.kafka.common.metrics.Sensor.record(Sensor.java:178) at 
org.apache.kafka.clients.producer.internals.BufferPool.recordWaitTime(BufferPool.java:202)
 at 
org.apache.kafka.clients.producer.internals.BufferPool.allocate(BufferPool.java:147)
 at 
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:221)
 at 
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:941) 
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862) 
at org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:238) at 
org.apache.kafka.connect.storage.KafkaStatusBackingStore$4.onCompletion(KafkaStatusBackingStore.java:298)
 ... at 
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:959) 
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862) 
at org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:238) at 
org.apache.kafka.connect.storage.KafkaStatusBackingStore$4.onCompletion(KafkaStatusBackingStore.java:298){code}
Note the repeated KafkaProducer.send -> KafkaProducer.doSend -> 
KafkaStatusBackingStore$4.onCompletion calls, causing the issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

A question about topic limit in produceBytesIn

[hudeqi]

Another question about using kafka recently:


Currently, Kafka's write throttle only supports the user dimension and clientId 
dimension of the request. In fact, such a situation is often encountered in 
actual use: a topic entry traffic suddenly increases, and the resource 
bottleneck is about to be reached. It needs to be restricted, but The user 
and/or clientId written to this topic also send data to other topics in this 
cluster, which may affect other topics, and it is impossible to make an 
effective restriction on a single topic. Do we have plans to add throttle in 
topic dimension?

A question about the reassign task in kafka

[hudeqi]

One question were found in the process of using kafka recently:

Why does Kafka start fetching from the leader's logStartOffset when Kafka is 
doing the topic reassign task (such as cluster expansion brokers)? If the 
amount of data stored locally by the partition leader is large, the disk IO 
will be full. Although there is a corresponding current throttle mechanism, it 
is difficult to grasp an appropriate limiting threshold. If the setting is not 
accurate, there will still be a instantaneous crit to the disk. 

I have an idea: when doing the reassign task, the new replica starts fetching 
data from logEndOffset/logStartOffset. According to some strategies (I haven’t 
thought about it yet, but it should not be very complicated), for example, when 
judging that the leader’s local data volume is not large, you can start 
directly from logStartOffset, so that the reassign task can be completed 
quickly. If the leader’s local data volume is huge, then the fetch can be 
started from logEndOffset, and only the amount of data accumulated by the new 
replica up to the data retention time can be added to the ISR to complete the 
reassign task.

What do you think about this issue?




best，

hudeqi

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1757

[Apache Jenkins Server]

See