[REVIEW REQUEST] LogCleaner, ZkConfig moved to java

2024-02-19 Thread Николай Ижиков 
Hello.

I prepared two PRs that moves parts of KafkaConfig to java.

[1] https://github.com/apache/kafka/pull/15387 - Log cleaner configuration 
moved to CleanerConfig
[2] https://github.com/apache/kafka/pull/15075 - ZooKeeper configuration moved 
to ZkConfig

Please, take a look.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2657

2024-02-19 Thread Apache Jenkins Server 
See

[VOTE] KIP-1019: Expose method to determine Metric Measurability

2024-02-19 Thread Apoorv Mittal 
Hi,
I’d like to start the voting for KIP-1019: Expose method to determine
Metric Measurability.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability

Regards,
Apoorv Mittal

[jira] [Created] (KAFKA-16280) Expose method to determine Metric Measurability

2024-02-19 Thread Apoorv Mittal (Jira) 
Apoorv Mittal created KAFKA-16280:
-

 Summary: Expose method to determine Metric Measurability
 Key: KAFKA-16280
 URL: https://issues.apache.org/jira/browse/KAFKA-16280
 Project: Kafka
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.8.0
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal
 Fix For: 3.8.0


The Jira is to track the development of KIP-1019: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16279) Avoid leaking abstractions of `StateStore`

2024-02-19 Thread Matthias J. Sax (Jira) 
Matthias J. Sax created KAFKA-16279:
---

 Summary: Avoid leaking abstractions of `StateStore`
 Key: KAFKA-16279
 URL: https://issues.apache.org/jira/browse/KAFKA-16279
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


The `StateStore` interface is user facing and contains a few life-cycle 
management methods (like `init()` and `close()`) – those methods are exposed 
for users to develop custom state stores.

However, we also use `StateStore` as base interface for store-handles in the 
PAPI, and thus life-cycle management methods are leaking into the PAPI (maybe 
also others – would need a dedicated discussion which one we consider useful 
for PAPI users and which not).

We should consider to change what we expose in the PAPI (atm, we only document 
via JavaDocs that eg. `close()` should never be called; but it's of course not 
ideal and would be better if `close()` et al. would not be expose for `PAPI` 
users to begin with.)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2024-02-19 Thread Rowland Smith 
Hi Artem,

I think that we both have the same understanding. An explicit prepare RPC
does not eliminate any conditions, it just reduces the window for possible
undesirable conditions like pending in-doubt transactions. So there is no
right or wrong answer, a prepare RPC will reduce the number of
occurrences of in-doubt transactions, but with a performance cost of an
extra RPC call on every transaction.

The Open Group DTP model and the XA interface requires that resource
managers be able to report prepared transactions only, so a prepare RPC
will be required. I will include it in my KIP for XA interface support, and
will propose an implementation where clients can choose whether they want a
prepare RPC when not using the XA interface. How does that sound?

- Rowland

On Fri, Feb 16, 2024 at 7:15 PM Artem Livshits
 wrote:

> Hi Rowland,
>
> > I am not sure what you mean by guarantee,
>
> A guarantee would be an elimination of complexity or a condition.  E.g. if
> adding an explicit prepare RPC eliminated in-doubt transactions, or
> eliminated a significant complexity in implementation.
>
> > 1. Transactions that haven’t reached “prepared” state can be aborted via
> timeout.
>
> The argument is that it doesn't eliminate any conditions, it merely reduces
> a subset of circumstances for the conditions to happen, but the conditions
> still happen and must be handled.  The operator still needs to set up
> monitoring for run-away transactions, there still needs to be an
> "out-of-band" channel to resolve run-away transactions (i.e. the operation
> would need a way that's not a part of the 2PC protocol to reconcile with
> the application owner), there still needs to be tooling for resolving
> run-away transactions.
>
> On the downside, an explicit prepare RPC would have a performance hit on
> the happy path in every single transaction.
>
> -Artem
>
> On Tue, Feb 6, 2024 at 7:35 PM Rowland Smith  wrote:
>
> > Hi Artem,
> >
> > I am not sure what you mean by guarantee, but I am referring to a better
> > operational experience. You mentioned this as the first benefit of an
> > explicit "prepare" RPC in the KIP.
> >
> >
> > 1. Transactions that haven’t reached “prepared” state can be aborted via
> > timeout.
> >
> > However, in explaining why an explicit "prepare" RPC was not included in
> > the design, you make no further mention of this benefit. So what I am
> > saying is this benefit is quite significant operationally. Many client
> > application failures may occur before the transaction reaches the
> prepared
> > state, and the ability to automatically abort those transactions and
> > unblock affected partitions without administrative intervention or fast
> > restart of the client would be a worthwhile benefit. An explicit
> "prepare"
> > RPC will also be needed by the XA implementation, so I would like to see
> it
> > implemented for that reason. Otherwise, I will need to add this work to
> my
> > KIP.
> >
> > - Rowland
> >
> > On Mon, Feb 5, 2024 at 9:35 PM Artem Livshits
> >  wrote:
> >
> > > Hi Rowland,
> > >
> > > Thank you for your reply.  I think I understand what you're saying and
> > just
> > > tried to provide a quick summary.  The
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC#KIP939:SupportParticipationin2PC-Explicit%E2%80%9Cprepare%E2%80%9DRPC
> > > actually goes into the details on what would be the benefits of adding
> an
> > > explicit prepare RPC and why those won't really add any advantages such
> > as
> > > elimination the needs for monitoring, tooling or providing additional
> > > guarantees.  Let me know if you think of a guarantee that prepare RPC
> > would
> > > provide.
> > >
> > > -Artem
> > >
> > > On Mon, Feb 5, 2024 at 6:22 PM Rowland Smith 
> wrote:
> > >
> > > > Hi Artem,
> > > >
> > > > I don't think that you understand what I am saying. In any
> transaction,
> > > > there is work done before the call to prepareTranscation() and work
> > done
> > > > afterwards. Any work performed before the call to
> prepareTransaction()
> > > can
> > > > be aborted after a relatively short timeout if the client fails. It
> is
> > > only
> > > > after the prepareTransaction() call that a transaction becomes
> in-doubt
> > > and
> > > > must be remembered for a much longer period of time to allow the
> client
> > > to
> > > > recover and make the decision to either commit or abort. A
> considerable
> > > > amount of time might be spent before prepareTransaction() is called,
> > and
> > > if
> > > > the client fails in this period, relatively quick transaction abort
> > would
> > > > unblock any partitions and make the system fully available. So a
> > prepare
> > > > RPC would reduce the window where a client failure results in
> > potentially
> > > > long-lived blocking transactions.
> > > >
> > > > Here is the proposed sequence from the KIP with 2 added steps (4 and
> > 5):
> > > >
> > > >
> > > >1. Begin database transaction
> > > >

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Divij Vaidya 
Great. In that case we can fix the license issue retrospectively. I have
created a JIRA for it https://issues.apache.org/jira/browse/KAFKA-16278 and
also updated the release process (which redirects to
https://issues.apache.org/jira/browse/KAFKA-12622) to check for the correct
license in both the kafka binaries.

I am +1 (binding) assuming Mickael's concerns about update notes to 3.7 are
addressed before release.

--
Divij Vaidya



On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison 
wrote:

> Hi,
>
> I agree with Josep, I don't think it's worth making a new RC just for this.
>
> Thanks Stanislav for sharing the test results. The last thing holding
> me from casting my vote is the missing upgrade notes for 3.7.0.
>
> Thanks,
> Mickael
>
>
>
> On Mon, Feb 19, 2024 at 4:28 PM Josep Prat 
> wrote:
> >
> > I think I remember finding a similar problem (NOTICE_binary) and it
> didn't
> > qualify for an extra RC
> >
> > Best,
> >
> > On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya 
> > wrote:
> >
> > > I have performed the following checks. The only thing I would like to
> call
> > > out is the missing licenses before providing a vote. How do we want
> > > to proceed on this? What have we done in the past? (Creating a new RC
> is
> > > overkill IMO for this license issue).
> > >
> > > ## License check
> > >
> > > Test: Validate license of dependencies for both 2.12 & 2.13 binary.
> > > Result: Missing license for some scala* libraries specifically for
> 2.12.
> > > Seems like we have been missing these licenses for quite some version
> now.
> > >
> > > ```
> > > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if ! grep
> -q
> > > ${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi;
> done
> > > scala-collection-compat_2.12-2.10.0 is missing in license file
> > > scala-java8-compat_2.12-1.0.2 is missing in license file
> > > scala-library-2.12.18 is missing in license file
> > > scala-logging_2.12-3.9.4 is missing in license file
> > > scala-reflect-2.12.18 is missing in license file
> > > ```
> > >
> > > ## Long running tests for memory leak (on ARM machine with zstd)
> > >
> > > Test: Run produce/consume for a few hours and verify no gradual
> increase in
> > > heap.
> > > Result: No heap increase observed. The overall CPU utilization is lower
> > > compared to 3.5.1.
> > >
> > > ## Verify system test results
> > >
> > > Test: Spot check the results of system tests.
> > > Result: I have verified that the system tests are passing across
> different
> > > runs.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
> > >  wrote:
> > >
> > > > The latest system test build completed successfully -
> > > >
> > > >
> > >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708250728--apache--3.7--02197edaaa/2024-02-18--001./2024-02-18--001./report.html
> > > >
> > > > *System tests are therefore all good*. We just have some flakes
> > > >
> > > > On Sun, Feb 18, 2024 at 10:45 AM Stanislav Kozlovski <
> > > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > > > The upgrade test passed ->
> > > > >
> > > >
> > >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708103771--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > > >
> > > > > The replica verification test succeeded in ZK mode, but failed in
> > > > > ISOLATED_KRAFT. It just seems to be very flaky.
> > > > >
> > > >
> > >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708100119--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > > >
> > > > > Scheduling another run in
> > > > >
> > >
> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6062/
> > > > >
> > > > > On Fri, Feb 16, 2024 at 6:39 PM Stanislav Kozlovski <
> > > > > stanis...@confluent.io> wrote:
> > > > >
> > > > >> Thanks all for the help in verifying.
> > > > >>
> > > > >> I have updated
> > > > >>
> > > >
> > >
> https://gist.github.com/stanislavkozlovski/820976fc7bfb5f4dcdf9742fd96a9982
> > > > >> with the system tests.
> > > > >> There were two builds ran, and across those - the following tests
> > > failed
> > > > >> two times in a row:
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> *kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest#test_replica_lagsArguments:{
> > > > >> "metadata_quorum": "ZK"}*Fails with the same error of
> > > > >> *`TimeoutError('Timed out waiting to reach non-zero number of
> replica
> > > > >> lags.')`*
> > > > >> I have scheduled a re-run of this specific test here ->
> > > > >>
> > > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6057
> > > > >>
> > > > >>
> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> > > > >> "compression_types": [

[jira] [Created] (KAFKA-16278) Missing license for scala related dependencies

2024-02-19 Thread Divij Vaidya (Jira) 
Divij Vaidya created KAFKA-16278:


 Summary: Missing license for  scala related dependencies
 Key: KAFKA-16278
 URL: https://issues.apache.org/jira/browse/KAFKA-16278
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.1, 3.7.0
Reporter: Divij Vaidya


We are missing the license for following dependency in 
[https://github.com/apache/kafka/blob/b71999be95325f6ea54e925cbe5b426425781014/LICENSE-binary#L261]
 

scala-collection-compat_2.12-2.10.0 is missing in license file
scala-java8-compat_2.12-1.0.2 is missing in license file
scala-library-2.12.18 is missing in license file
scala-logging_2.12-3.9.4 is missing in license file
scala-reflect-2.12.18 is missing in license file

The objective of this task is to add these dependencies in the LICENSE-binary 
file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2656

2024-02-19 Thread Apache Jenkins Server 
See

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Mickael Maison 
Hi,

I agree with Josep, I don't think it's worth making a new RC just for this.

Thanks Stanislav for sharing the test results. The last thing holding
me from casting my vote is the missing upgrade notes for 3.7.0.

Thanks,
Mickael



On Mon, Feb 19, 2024 at 4:28 PM Josep Prat  wrote:
>
> I think I remember finding a similar problem (NOTICE_binary) and it didn't
> qualify for an extra RC
>
> Best,
>
> On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya 
> wrote:
>
> > I have performed the following checks. The only thing I would like to call
> > out is the missing licenses before providing a vote. How do we want
> > to proceed on this? What have we done in the past? (Creating a new RC is
> > overkill IMO for this license issue).
> >
> > ## License check
> >
> > Test: Validate license of dependencies for both 2.12 & 2.13 binary.
> > Result: Missing license for some scala* libraries specifically for 2.12.
> > Seems like we have been missing these licenses for quite some version now.
> >
> > ```
> > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if ! grep -q
> > ${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi; done
> > scala-collection-compat_2.12-2.10.0 is missing in license file
> > scala-java8-compat_2.12-1.0.2 is missing in license file
> > scala-library-2.12.18 is missing in license file
> > scala-logging_2.12-3.9.4 is missing in license file
> > scala-reflect-2.12.18 is missing in license file
> > ```
> >
> > ## Long running tests for memory leak (on ARM machine with zstd)
> >
> > Test: Run produce/consume for a few hours and verify no gradual increase in
> > heap.
> > Result: No heap increase observed. The overall CPU utilization is lower
> > compared to 3.5.1.
> >
> > ## Verify system test results
> >
> > Test: Spot check the results of system tests.
> > Result: I have verified that the system tests are passing across different
> > runs.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
> >  wrote:
> >
> > > The latest system test build completed successfully -
> > >
> > >
> > https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708250728--apache--3.7--02197edaaa/2024-02-18--001./2024-02-18--001./report.html
> > >
> > > *System tests are therefore all good*. We just have some flakes
> > >
> > > On Sun, Feb 18, 2024 at 10:45 AM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > wrote:
> > >
> > > > The upgrade test passed ->
> > > >
> > >
> > https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708103771--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > >
> > > > The replica verification test succeeded in ZK mode, but failed in
> > > > ISOLATED_KRAFT. It just seems to be very flaky.
> > > >
> > >
> > https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708100119--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > >
> > > > Scheduling another run in
> > > >
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6062/
> > > >
> > > > On Fri, Feb 16, 2024 at 6:39 PM Stanislav Kozlovski <
> > > > stanis...@confluent.io> wrote:
> > > >
> > > >> Thanks all for the help in verifying.
> > > >>
> > > >> I have updated
> > > >>
> > >
> > https://gist.github.com/stanislavkozlovski/820976fc7bfb5f4dcdf9742fd96a9982
> > > >> with the system tests.
> > > >> There were two builds ran, and across those - the following tests
> > failed
> > > >> two times in a row:
> > > >>
> > > >>
> > > >>
> > >
> > *kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest#test_replica_lagsArguments:{
> > > >> "metadata_quorum": "ZK"}*Fails with the same error of
> > > >> *`TimeoutError('Timed out waiting to reach non-zero number of replica
> > > >> lags.')`*
> > > >> I have scheduled a re-run of this specific test here ->
> > > >>
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6057
> > > >>
> > > >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> > > >> "compression_types": [ "zstd" ], "from_kafka_version": "2.4.1",
> > > >> "to_message_format_version": null}*
> > > >> Fails with the same error of
> > > >> *`TimeoutError('Producer failed to produce messages for 20s.')`*
> > > >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> > > >> "compression_types": [ "lz4" ], "from_kafka_version": "3.0.2",
> > > >> "to_message_format_version": null}*
> > > >> Fails with the same error of *`TimeoutError('Producer failed to
> > produce
> > > >> messages for 20s.')`*
> > > >>
> > > >> I have scheduled a re-run of this test here ->
> > > >>
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6058/
> > > >>
> > > >> On Fri, Feb 16, 2024 at 12:15 PM Vedarth Sharma <
> > > vedarth.sha...@gmail.com>

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Josep Prat 
I think I remember finding a similar problem (NOTICE_binary) and it didn't
qualify for an extra RC

Best,

On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya 
wrote:

> I have performed the following checks. The only thing I would like to call
> out is the missing licenses before providing a vote. How do we want
> to proceed on this? What have we done in the past? (Creating a new RC is
> overkill IMO for this license issue).
>
> ## License check
>
> Test: Validate license of dependencies for both 2.12 & 2.13 binary.
> Result: Missing license for some scala* libraries specifically for 2.12.
> Seems like we have been missing these licenses for quite some version now.
>
> ```
> for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if ! grep -q
> ${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi; done
> scala-collection-compat_2.12-2.10.0 is missing in license file
> scala-java8-compat_2.12-1.0.2 is missing in license file
> scala-library-2.12.18 is missing in license file
> scala-logging_2.12-3.9.4 is missing in license file
> scala-reflect-2.12.18 is missing in license file
> ```
>
> ## Long running tests for memory leak (on ARM machine with zstd)
>
> Test: Run produce/consume for a few hours and verify no gradual increase in
> heap.
> Result: No heap increase observed. The overall CPU utilization is lower
> compared to 3.5.1.
>
> ## Verify system test results
>
> Test: Spot check the results of system tests.
> Result: I have verified that the system tests are passing across different
> runs.
>
> --
> Divij Vaidya
>
>
>
> On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
>  wrote:
>
> > The latest system test build completed successfully -
> >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708250728--apache--3.7--02197edaaa/2024-02-18--001./2024-02-18--001./report.html
> >
> > *System tests are therefore all good*. We just have some flakes
> >
> > On Sun, Feb 18, 2024 at 10:45 AM Stanislav Kozlovski <
> > stanis...@confluent.io>
> > wrote:
> >
> > > The upgrade test passed ->
> > >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708103771--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > >
> > > The replica verification test succeeded in ZK mode, but failed in
> > > ISOLATED_KRAFT. It just seems to be very flaky.
> > >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708100119--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > >
> > > Scheduling another run in
> > >
> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6062/
> > >
> > > On Fri, Feb 16, 2024 at 6:39 PM Stanislav Kozlovski <
> > > stanis...@confluent.io> wrote:
> > >
> > >> Thanks all for the help in verifying.
> > >>
> > >> I have updated
> > >>
> >
> https://gist.github.com/stanislavkozlovski/820976fc7bfb5f4dcdf9742fd96a9982
> > >> with the system tests.
> > >> There were two builds ran, and across those - the following tests
> failed
> > >> two times in a row:
> > >>
> > >>
> > >>
> >
> *kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest#test_replica_lagsArguments:{
> > >> "metadata_quorum": "ZK"}*Fails with the same error of
> > >> *`TimeoutError('Timed out waiting to reach non-zero number of replica
> > >> lags.')`*
> > >> I have scheduled a re-run of this specific test here ->
> > >>
> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6057
> > >>
> > >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> > >> "compression_types": [ "zstd" ], "from_kafka_version": "2.4.1",
> > >> "to_message_format_version": null}*
> > >> Fails with the same error of
> > >> *`TimeoutError('Producer failed to produce messages for 20s.')`*
> > >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> > >> "compression_types": [ "lz4" ], "from_kafka_version": "3.0.2",
> > >> "to_message_format_version": null}*
> > >> Fails with the same error of *`TimeoutError('Producer failed to
> produce
> > >> messages for 20s.')`*
> > >>
> > >> I have scheduled a re-run of this test here ->
> > >>
> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6058/
> > >>
> > >> On Fri, Feb 16, 2024 at 12:15 PM Vedarth Sharma <
> > vedarth.sha...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hey Stanislav,
> > >>>
> > >>> Thanks for the release candidate.
> > >>>
> > >>> +1 (non-binding)
> > >>>
> > >>> I tested and verified the docker image artifact
> > apache/kafka:3.7.0-rc4:-
> > >>> - verified create topic, produce messages and consume messages flow
> > when
> > >>> running the docker image with
> > >>> - default configs
> > >>> - configs provided via env variables
> > >>> - configs provided via file input
> > >>> - verified the html documentation for docker

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

2024-02-19 Thread via GitHub 


divijvaidya commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1494668977


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details

Review Comment:
   Just to make this more clear, could we add a sentence here: "Note that 
clients can continue to use JDK >= 8 to connect to Kafka brokers."



##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client

Re: [VOTE] 3.7.0 RC4

2024-02-19 Thread Divij Vaidya 
I have performed the following checks. The only thing I would like to call
out is the missing licenses before providing a vote. How do we want
to proceed on this? What have we done in the past? (Creating a new RC is
overkill IMO for this license issue).

## License check

Test: Validate license of dependencies for both 2.12 & 2.13 binary.
Result: Missing license for some scala* libraries specifically for 2.12.
Seems like we have been missing these licenses for quite some version now.

```
for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if ! grep -q
${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi; done
scala-collection-compat_2.12-2.10.0 is missing in license file
scala-java8-compat_2.12-1.0.2 is missing in license file
scala-library-2.12.18 is missing in license file
scala-logging_2.12-3.9.4 is missing in license file
scala-reflect-2.12.18 is missing in license file
```

## Long running tests for memory leak (on ARM machine with zstd)

Test: Run produce/consume for a few hours and verify no gradual increase in
heap.
Result: No heap increase observed. The overall CPU utilization is lower
compared to 3.5.1.

## Verify system test results

Test: Spot check the results of system tests.
Result: I have verified that the system tests are passing across different
runs.

--
Divij Vaidya



On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
 wrote:

> The latest system test build completed successfully -
>
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708250728--apache--3.7--02197edaaa/2024-02-18--001./2024-02-18--001./report.html
>
> *System tests are therefore all good*. We just have some flakes
>
> On Sun, Feb 18, 2024 at 10:45 AM Stanislav Kozlovski <
> stanis...@confluent.io>
> wrote:
>
> > The upgrade test passed ->
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708103771--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> >
> > The replica verification test succeeded in ZK mode, but failed in
> > ISOLATED_KRAFT. It just seems to be very flaky.
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708100119--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> >
> > Scheduling another run in
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6062/
> >
> > On Fri, Feb 16, 2024 at 6:39 PM Stanislav Kozlovski <
> > stanis...@confluent.io> wrote:
> >
> >> Thanks all for the help in verifying.
> >>
> >> I have updated
> >>
> https://gist.github.com/stanislavkozlovski/820976fc7bfb5f4dcdf9742fd96a9982
> >> with the system tests.
> >> There were two builds ran, and across those - the following tests failed
> >> two times in a row:
> >>
> >>
> >>
> *kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest#test_replica_lagsArguments:{
> >> "metadata_quorum": "ZK"}*Fails with the same error of
> >> *`TimeoutError('Timed out waiting to reach non-zero number of replica
> >> lags.')`*
> >> I have scheduled a re-run of this specific test here ->
> >> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6057
> >>
> >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> >> "compression_types": [ "zstd" ], "from_kafka_version": "2.4.1",
> >> "to_message_format_version": null}*
> >> Fails with the same error of
> >> *`TimeoutError('Producer failed to produce messages for 20s.')`*
> >> *kafkatest.tests.core.upgrade_test.TestUpgrade#test_upgradeArguments:{
> >> "compression_types": [ "lz4" ], "from_kafka_version": "3.0.2",
> >> "to_message_format_version": null}*
> >> Fails with the same error of *`TimeoutError('Producer failed to produce
> >> messages for 20s.')`*
> >>
> >> I have scheduled a re-run of this test here ->
> >> https://jenkins.confluent.io/job/system-test-kafka-branch-builder/6058/
> >>
> >> On Fri, Feb 16, 2024 at 12:15 PM Vedarth Sharma <
> vedarth.sha...@gmail.com>
> >> wrote:
> >>
> >>> Hey Stanislav,
> >>>
> >>> Thanks for the release candidate.
> >>>
> >>> +1 (non-binding)
> >>>
> >>> I tested and verified the docker image artifact
> apache/kafka:3.7.0-rc4:-
> >>> - verified create topic, produce messages and consume messages flow
> when
> >>> running the docker image with
> >>> - default configs
> >>> - configs provided via env variables
> >>> - configs provided via file input
> >>> - verified the html documentation for docker image.
> >>> - ran the example docker compose files successfully.
> >>>
> >>> All looks good for the docker image artifact!
> >>>
> >>> Thanks and regards,
> >>> Vedarth
> >>>
> >>>
> >>> On Thu, Feb 15, 2024 at 10:58 PM Mickael Maison <
> >>> mickael.mai...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi Stanislav,
> >>> >
> >>> > Thanks for running the release.
> >>> >
> >>> > I did the following testing:
>

[jira] [Reopened] (KAFKA-16167) Fix PlaintextConsumerTest.testAutoCommitOnCloseAfterWakeup

2024-02-19 Thread Lucas Brutschy (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-16167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy reopened KAFKA-16167:


> Fix PlaintextConsumerTest.testAutoCommitOnCloseAfterWakeup
> --
>
> Key: KAFKA-16167
> URL: https://issues.apache.org/jira/browse/KAFKA-16167
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Critical
>  Labels: consumer-threading-refactor, integration-tests, 
> kip-848-client-support
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

2024-02-19 Thread via GitHub 


mimaison commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1494562104


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.

Review Comment:
   For this, no need for another RC, just open a PR against kafka-site that 
adds this section. The section for 3.7.0 can probably be pretty much the same 
as for 3.6.0 as I don't think there's anything specific in 3.7.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-15140) Improve TopicCommandIntegrationTest to be less flaky

2024-02-19 Thread Luke Chen (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15140.
---
Fix Version/s: (was: 3.5.1)
   Resolution: Fixed

> Improve TopicCommandIntegrationTest to be less flaky
> 
>
> Key: KAFKA-15140
> URL: https://issues.apache.org/jira/browse/KAFKA-15140
> Project: Kafka
>  Issue Type: Test
>  Components: unit tests
>Reporter: Divij Vaidya
>Assignee: Lan Ding
>Priority: Minor
>  Labels: flaky-test, newbie
> Fix For: 3.8.0
>
>
> *This is a good Jira for folks who are new to contributing to Kafka.*
> Tests in TopicCommandIntegrationTest get flaky from time to time. The 
> objective of the task is to make them more robust by doing the following:
> 1. Replace the usage {-}createAndWaitTopic{-}() adminClient.createTopics() 
> method and other places where were are creating a topic (without waiting) 
> with 
> TestUtils.createTopicWithAdmin(). The latter method already contains the 
> functionality to create a topic and wait for metadata to sync up.
> 2. Replace the number 6 at places such as 
> "adminClient.createTopics(
> Collections.singletonList(new NewTopic("foo_bar", 1, 6.toShort)))" with a 
> meaningful constant.
> 3. Add logs if an assertion fails, for example, lines such as "
> assertTrue(rows(0).startsWith(s"\tTopic: $testTopicName"), output)" should 
> have a third argument which prints the actual output printed so that we can 
> observe in the test logs on what was the output when assertion failed.
> 4. Replace occurrences of "\n" with System.lineSeparator() which is platform 
> independent
> 5. We should wait for reassignment to complete whenever we are re-assigning 
> partitions using alterconfig before we call describe to validate it. We could 
> use 
> TestUtils.waitForAllReassignmentsToComplete()
> *Motivation of this task*
> Try to fix the flaky test behaviour such as observed in 
> [https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-13924/5/testReport/junit/kafka.admin/TopicCommandIntegrationTest/Build___JDK_11_and_Scala_2_13___testDescribeUnderMinIsrPartitionsMixed_String__quorum_zk/]
>  
> {noformat}
> org.opentest4j.AssertionFailedError: expected:  but was: 
>   at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>   at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>   at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
>   at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
>   at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:180)
>   at 
> app//kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitionsMixed(TopicCommandIntegrationTest.scala:794){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

2024-02-19 Thread via GitHub 


mimaison commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1494324650


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.

Review Comment:
   I think this link should be 
`https://kafka.apache.org/documentation.html#upgrade_3_7_0`. But as I said in 
the vote thread, this section is missing from the 3.7 docs.



##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability;>KIP-714
 Client metrics and observability:
+With KIP-714, operators get better visibility 
into the clients connecting to their cluster with broker-side support of 
client-level metrics via a standardized telemetry interface.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources;>KIP-1000
 List Client Metrics Configuration Resources:
+KIP-1000 supports KIP-714 by introducing a way 
to create, read, update, and delete the client metrics configuration resources 
using the existing RPCs and the kafka-configs.sh tool.
+
+

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

2024-02-19 Thread via GitHub 


stanislavkozlovski commented on PR #578:
URL: https://github.com/apache/kafka-site/pull/578#issuecomment-1952113268

   @mimaison thank you for the comments. Do you happen to be available for a 
second review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13566) producer exponential backoff implementation

2024-02-19 Thread Mickael Maison (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13566.

Resolution: Duplicate

> producer exponential backoff implementation
> ---
>
> Key: KAFKA-13566
> URL: https://issues.apache.org/jira/browse/KAFKA-13566
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13567) adminClient exponential backoff implementation

2024-02-19 Thread Mickael Maison (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13567.

Resolution: Duplicate

> adminClient exponential backoff implementation
> --
>
> Key: KAFKA-13567
> URL: https://issues.apache.org/jira/browse/KAFKA-13567
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13565) consumer exponential backoff implementation

2024-02-19 Thread Mickael Maison (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13565.

Fix Version/s: 3.7.0
   Resolution: Duplicate

> consumer exponential backoff implementation
> ---
>
> Key: KAFKA-13565
> URL: https://issues.apache.org/jira/browse/KAFKA-13565
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16253) kafka_server_alterPartition_metrics_network_io_total is not supported in kafka 3.5.1

2024-02-19 Thread Janardhana Gopalachar (Jira) 


 [ 
https://issues.apache.org/jira/browse/KAFKA-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janardhana Gopalachar resolved KAFKA-16253.
---
Resolution: Invalid

> kafka_server_alterPartition_metrics_network_io_total is not supported in 
> kafka 3.5.1
> 
>
> Key: KAFKA-16253
> URL: https://issues.apache.org/jira/browse/KAFKA-16253
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.5.1
>Reporter: Janardhana Gopalachar
>Priority: Major
>
> Hi 
> The metrics  below was supported in kafka 3.2.3 but in 3.5.1 it is not sent . 
> I tried to search the source code to see if this is present, but couldnt find 
> a reference. 
> Is the below metrics deprecated or not supported. ?
> kafka_server_alterPartition_metrics_network_io_total
>  
> in kafka3.2.3 we could get the metrics, but in 3.5.1 it is not available
> HELP kafka_server_alterPartition_metrics_network_io_total The total number of 
> network operations (reads or writes) on all connections 
> kafka.server:name=null,type=alterPartition-metrics,attribute=network-io-total
> :--:# TYPE kafka_server_alterPartition_metrics_network_io_total untyped
> -kafka_server_alterPartition_metrics_network_io_total{BrokerId="0",} 10.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Seeking Assistance with MM2 Rest API Configuration - Encountering 404 Error

2024-02-19 Thread aaron ai 
Hi Greg,

Thanks for the explanation. Really appreciate your help!

Best,
Aaron

On Mon, Feb 19, 2024 at 1:39 AM Greg Harris 
wrote:

> Hi Aaron,
>
> This is the expected behavior of KIP-710. The relevant section reproduced
> here:
>
> > Enabling a single Connect REST server in the MirrorMaker 2.0 node, only
> supporting the internal Connect endpoints.
>
> The endpoints that are typically exposed in a distributed Connect for
> creating connectors, getting status, etc are not enabled in dedicated mode,
> even with the new configuration. Only the internal endpoints that workers
> use to communicate with one another are enabled.
>
> If you want the other endpoints for getting the status, reconfiguring
> connectors, etc, I would recommend switching to the connect distributed
> mode and creating the MM2 connectors via the REST API.
>
> Hope this helps!
> Greg
>
> On Sun, Feb 18, 2024 at 2:00 AM aaron ai  wrote:
>
> > Kafka Development Team:
> >
> > I have enabled the internal REST API for the dedicated mode by setting
> > *dedicated.mode.enable.internal.rest* to true in my MM2 configuration
> > since KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > 2.0 clusters
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-710%3A+Full+support+for+distributed+mode+in+dedicated+MirrorMaker+2.0+clusters
> >.
> > However, when I attempt to make requests to the REST API on
> localhost:8083,
> > I consistently receive a 404 Not Found error. This behavior has left me
> > uncertain whether I'm facing a configuration issue or if this might be
> > indicative of a bug within the system.
> >
> > BTW, I found a similar question on Stack Overflow
> > <
> https://stackoverflow.com/questions/76780739/distributed-mode-in-dedicated-mirrormaker-2-cluster-not-balancing-tasks-evenly#:~:text=If%20I%20try%20running%20curl%20targeting%20the%20leaderUrl%20above%20(on%20the%20root%20or%20at%20/connectors)%20from%20any%20node%2C%20using%20the%20advertised%20IP%20(or%20localhost%20in%20the%20leader)%20I%20end%20up%20with%20%7B%22error_code%22%3A404%2C%22message%22%3A%22HTTP%20404%20Not%20Found%22%7D
> >.
> > My Kafka Version: 3.6.0
> >
> > Here is my configuration:
> > clusters = A, B
> > A.bootstrap.servers = 10.1.1.168:9092
> > B.bootstrap.servers = 10.1.1.168:9592
> > A->B.enabled = true
> > A->B.topics = .*
> > groups=.*
> > groups.exclude = test-.*
> > replication.factor=1
> > checkpoints.topic.replication.factor=1
> > heartbeats.topic.replication.factor=1
> > offset-syncs.topic.replication.factor=1
> > offset.storage.replication.factor=1
> > status.storage.replication.factor=1
> > config.storage.replication.factor=1
> > sync.topic.acls.enabled = true
> > emit.heartbeats.interval.seconds = 5
> > sync.group.offsets.enabled=true
> > sync.topic.configs.enabled=true
> > dedicated.mode.enable.internal.rest=true
> >
> > and the HTTP response.
> > [image: image.png]
> > [image: image.png]
> >
> > I eagerly await your response and appreciate any feedback you might have
> > on this matter.
> >
>