Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1441

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #11

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1440

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

[Jun Rao]

Hi, Divij,

Thanks for the explanation.

If there is in-memory cache, could we maintain the log size in the cache
with the existing API? For example, a replica could make a
listRemoteLogSegments(TopicIdPartition topicIdPartition) call on startup to
get the remote segment size before the current leaderEpoch. The leader
could then maintain the size incrementally afterwards. On leader change,
other replicas can make a listRemoteLogSegments(TopicIdPartition
topicIdPartition, int leaderEpoch) call to get the size of newly generated
segments.

Thanks,

Jun


On Wed, Dec 14, 2022 at 3:27 AM Divij Vaidya 
wrote:

> > Is the new method enough for doing size-based retention?
>
> Yes. You are right in assuming that this API only provides the Remote
> storage size (for current epoch chain). We would use this API for size
> based retention along with a value of localOnlyLogSegmentSize which is
> computed as Log.sizeInBytes(logSegments.filter(_.baseOffset >
> highestOffsetWithRemoteIndex)). Hence, (total_log_size =
> remoteLogSizeBytes + log.localOnlyLogSegmentSize). I have updated the KIP
> with this information. You can also check an example implementation at
>
> https://github.com/satishd/kafka/blob/2.8.x-tiered-storage/core/src/main/scala/kafka/log/Log.scala#L2077
>
>
> > Do you imagine all accesses to remote metadata will be across the network
> or will there be some local in-memory cache?
>
> I would expect a disk-less implementation to maintain a finite in-memory
> cache for segment metadata to optimize the number of network calls made to
> fetch the data. In future, we can think about bringing this finite size
> cache into RLM itself but that's probably a conversation for a different
> KIP. There are many other things we would like to do to optimize the Tiered
> storage interface such as introducing a circular buffer / streaming
> interface from RSM (so that we don't have to wait to fetch the entire
> segment before starting to send records to the consumer), caching the
> segments fetched from RSM locally (I would assume all RSM plugin
> implementations to do this, might as well add it to RLM) etc.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Dec 12, 2022 at 7:35 PM Jun Rao  wrote:
>
> > Hi, Divij,
> >
> > Thanks for the reply.
> >
> > Is the new method enough for doing size-based retention? It gives the
> total
> > size of the remote segments, but it seems that we still don't know the
> > exact total size for a log since there could be overlapping segments
> > between the remote and the local segments.
> >
> > You mentioned a disk-less implementation. Do you imagine all accesses to
> > remote metadata will be across the network or will there be some local
> > in-memory cache?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Wed, Dec 7, 2022 at 3:10 AM Divij Vaidya 
> > wrote:
> >
> > > The method is needed for RLMM implementations which fetch the
> information
> > > over the network and not for the disk based implementations (such as
> the
> > > default topic based RLMM).
> > >
> > > I would argue that adding this API makes the interface more generic
> than
> > > what it is today. This is because, with the current APIs an implementor
> > is
> > > restricted to use disk based RLMM solutions only (i.e. the default
> > > solution) whereas if we add this new API, we unblock usage of network
> > based
> > > RLMM implementations such as databases.
> > >
> > >
> > >
> > > On Wed 30. Nov 2022 at 20:40, Jun Rao 
> wrote:
> > >
> > > > Hi, Divij,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > Point#2. My high level question is that is the new method needed for
> > > every
> > > > implementation of remote storage or just for a specific
> implementation.
> > > The
> > > > issues that you pointed out exist for the default implementation of
> > RLMM
> > > as
> > > > well and so far, the default implementation hasn't found a need for a
> > > > similar new method. For public interface, ideally we want to make it
> > more
> > > > general.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Nov 21, 2022 at 7:11 AM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thank you Jun and Alex for your comments.
> > > > >
> > > > > Point#1: You are right Jun. As Alex mentioned, the "derived
> metadata"
> > > can
> > > > > increase the size of cached metadata by a factor of 10 but it
> should
> > be
> > > > ok
> > > > > to cache just the actual metadata. My point about size being a
> > > limitation
> > > > > for using cache is not valid anymore.
> > > > >
> > > > > Point#2: For a new replica, it would still have to fetch the
> metadata
> > > > over
> > > > > the network to initiate the warm up of the cache and hence,
> increase
> > > the
> > > > > start time of the archival process. Please also note the
> > repercussions
> > > of
> > > > > the warm up scan that Alex mentioned in this thread as part of
> > #102.2.
> > > > >
> > > > > 100#: Agreed Alex. Thanks for clarifying that. My point about size

Re: [VOTE] KIP-889 Versioned State Stores

[John Roesler]

Thanks for the thorough KIP, Victoria!

I'm +1 (binding)

-John

On 2022/12/15 19:56:21 Victoria Xia wrote:
> Hi all,
> 
> I'd like to start a vote on KIP-889 for introducing versioned key-value
> state stores to Kafka Streams:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores
> 
> The discussion thread has been open for a few weeks now and has converged
> among the current participants.
> 
> Thanks,
> Victoria
> 

Re: [DISCUSS] KIP-890 Server Side Defense

[Jun Rao]

Hi, Justine,

Thanks for the reply.

70. Assigning a new pid on int overflow seems a bit hacky. If we need a txn
level id, it will be better to model this explicitly. Adding a new field
would require a bit more work since it requires a new txn marker format in
the log. So, we probably need to guard it with an IBP or metadata version
and document the impact on downgrade once the new format is written to the
log.

71. Hmm, once the marker is written, the partition will expect the next
append to be on the next epoch. Does that cover the case you mentioned?

72. Also, just to be clear on the stucked message issue described in the
motivation. With EoS, we also validate the sequence id for idempotency. So,
with the current logic, if the producer epoch hasn't been bumped on the
broker, it seems that the stucked message will fail the sequence validation
and will be ignored. If the producer epoch has been bumped, we ignore the
sequence check and the stuck message could be appended to the log. So, is
the latter case that we want to guard?

Thanks,

Jun

On Wed, Dec 14, 2022 at 10:44 AM Justine Olshan
 wrote:

> Matthias — thanks again for taking time to look a this. You said:
>
> > My proposal was only focusing to avoid dangling
>
> transactions if records are added without registered partition. -- Maybe
>
> you can add a few more details to the KIP about this scenario for better
>
> documentation purpose?
>
>
> I'm not sure I understand what you mean here. The motivation section
> describes two scenarios about how the record can be added without a
> registered partition:
>
>
> > This can happen when a message gets stuck or delayed due to networking
> issues or a network partition, the transaction aborts, and then the delayed
> message finally comes in.
>
>
> > Another way hanging transactions can occur is that a client is buggy and
> may somehow try to write to a partition before it adds the partition to the
> transaction.
>
>
>
> For the first example of this would it be helpful to say that this message
> comes in after the abort, but before the partition is added to the next
> transaction so it becomes "hanging." Perhaps the next sentence describing
> the message becoming part of the next transaction (a different case) was
> not properly differentiated.
>
>
>
> Jun — thanks for reading the KIP.
>
> 70. The int typing was a concern. Currently we have a mechanism in place to
> fence the final epoch when the epoch is about to overflow and assign a new
> producer ID with epoch 0. Of course, this is a bit tricky when it comes to
> the response back to the client.
> Making this a long could be another option, but I wonder are there any
> implications on changing this field if the epoch is persisted to disk? I'd
> need to check the usages.
>
> 71.This was something Matthias asked about as well. I was considering a
> possible edge case where a produce request from a new transaction somehow
> gets sent right after the marker is written, but before the producer is
> alerted of the newly bumped epoch. In this case, we may include this record
> when we don't want to. I suppose we could try to do something client side
> to bump the epoch after sending an endTxn as well in this scenario — but I
> wonder how it would work when the server is aborting based on a server-side
> error. I could also be missing something and this scenario is actually not
> possible.
>
> Thanks again to everyone reading and commenting. Let me know about any
> further questions or comments.
>
> Justine
>
> On Wed, Dec 14, 2022 at 9:41 AM Jun Rao  wrote:
>
> > Hi, Justine,
> >
> > Thanks for the KIP. A couple of comments.
> >
> > 70. Currently, the producer epoch is an int. I am not sure if it's enough
> > to accommodate all transactions in the lifetime of a producer. Should we
> > change that to a long or add a new long field like txnId?
> >
> > 71. "it will write the prepare commit message with a bumped epoch and
> send
> > WriteTxnMarkerRequests with the bumped epoch." Hmm, the epoch is
> associated
> > with the current txn right? So, it seems weird to write a commit message
> > with a bumped epoch. Should we only bump up the epoch in EndTxnResponse
> and
> > rename the field to sth like nextProducerEpoch?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Mon, Dec 12, 2022 at 8:54 PM Matthias J. Sax 
> wrote:
> >
> > > Thanks for the background.
> > >
> > > 20/30: SGTM. My proposal was only focusing to avoid dangling
> > > transactions if records are added without registered partition. --
> Maybe
> > > you can add a few more details to the KIP about this scenario for
> better
> > > documentation purpose?
> > >
> > > 40: I think you hit a fair point about race conditions or client bugs
> > > (incorrectly not bumping the epoch). The complexity/confusion for using
> > > the bumped epoch I see, is mainly for internal debugging, ie,
> inspecting
> > > log segment dumps -- it seems harder to reason about the system for us
> > > humans. But if we get better 

[jira] [Created] (KAFKA-14497) LastStableOffset is advanced prematurely when a log is reopened.

[Vincent Jiang (Jira)]

Vincent Jiang created KAFKA-14497:
-

 Summary: LastStableOffset is advanced prematurely when a log is 
reopened.
 Key: KAFKA-14497
 URL: https://issues.apache.org/jira/browse/KAFKA-14497
 Project: Kafka
  Issue Type: Bug
Reporter: Vincent Jiang


In below test case, last stable offset of log is advanced prematurely after 
reopen:
 # producer #1 appends transaction records to leader. offsets = [0, 1, 2, 3]
 # producer #2 appends transactional records to leader. offsets =  [4, 5, 6, 7]
 # all records are replicated to followers and high watermark advanced to 8.
 # at this point, lastStableOffset = 0. (first offset of an open transaction)
 # producer #1 aborts the transaction by writing an abort marker at offset 8.  
ProducerStateManager.unreplicatedTxns contains the aborted transaction 
(firstOffset=0, lastOffset=8)
 # then the log is closed and reopened.
 # after reopen, log.lastStableOffset is initialized to 4.  This is because 
ProducerStateManager.unreplicatedTxns is empty after reopening log.

 

We should rebuild ProducerStateManager.unreplicatedTxns when reloading a log, 
so that lastStableOffset remains unchanged before and after reopen.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-889 Versioned State Stores

[Victoria Xia]

Thanks again for the great discussion, Sagar, Bruno, and Matthias. I've
just sent a message to start the vote on this KIP. Please have a look when
you get the chance.

Thanks,
Victoria

On Wed, Dec 14, 2022 at 12:28 PM Matthias J. Sax  wrote:

> Thanks for clarifying about the null-question. SGTM.
>
> On 12/13/22 3:06 PM, Victoria Xia wrote:
> > Hi Matthias,
> >
> > Thanks for chiming in! Barring objections from anyone on this thread, I
> > will start the vote for this KIP on Thursday. That should be enough time
> to
> > incorporate any lingering minor changes.
> >
> >> I slightly prefer to add `VersionedRecord` interface (also
> > like the name). I agree that it's low overhead and providing a clean
> > path forward for future changes seems worth it to me.
> >
> > OK, that makes two of us. I updated the KIP just now to formally include
> > VersionedRecord as the new return type from the various
> > VersionedKeyValueStore methods.
> >
> >> if we introduce `VersionedRecord`, I think we can keep the not-null
> > requirement for `ValueAndTimestamp`
> >
> > Not quite. VersionedRecord is only used as a return type from read
> methods,
> > which is why VersionedRecord is able to enforce that its value is never
> > null. If the value being returned would have been null, then we return a
> > null VersionedRecord instead, rather than non-null VersionedRecord with
> > null value. So, there's no use case for a VersionedRecord with null
> value.
> >
> > In contrast, even though ValueAndTimestamp is not anywhere in the public
> > VersionedKeyValueStore interface, ValueAndTimestamp still needs to be
> used
> > internally when representing a versioned key-value store as a
> > TimestampedKeyValueStore, since TimestampedKeyValueStore is used
> everywhere
> > throughout the internals of the codebase. In order to represent a
> versioned
> > key-value store as a TimestampedKeyValueStore, we have to support `put(K
> > key, ValueAndTimestamp value)`, which means ValueAndTimestamp needs to
> > support null value (with timestamp). Otherwise we cannot put a tombstone
> > into a versioned key-value store when using the internal
> > TimestampedKeyValueStore representation.
> >
> > It's very much an implementation detail that ValueAndTimestamp needs to
> be
> > relaxed to allow null values. I think this is a minor enough change that
> is
> > still preferable to the alternatives (refactoring the processors to not
> > require TimestampedKeyValueStore, or introducing a separate workaround
> > `put()` method on the TimestampedKeyValueStore representation of
> versioned
> > key-value stores), so I have left it in as part of the KIP.
> >
> > Best,
> > Victoria
> >
> > On Mon, Dec 12, 2022 at 8:42 PM Matthias J. Sax 
> wrote:
> >
> >> Thanks Victoria.
> >>
> >> I did not re-read the KIP in full on the wiki but only your email.
> >>
> >> Points (1)-(8) SGTM.
> >>
> >> About (9): I slightly prefer to add `VersionedRecord` interface (also
> >> like the name). I agree that it's low overhead and providing a clean
> >> path forward for future changes seems worth it to me. Btw: if we
> >> introduce `VersionedRecord`, I think we can keep the not-null
> >> requirement for `ValueAndTimestamp` what seems a small side benefit.
> >> (Btw: your code snippet in the KIP shows what `VersionedRecord` would
> >> have a non-null requirement for the value, but I think it would need to
> >> allow null as value?)
> >>
> >>
> >> -Matthias
> >>
> >> On 12/7/22 5:23 PM, Victoria Xia wrote:
> >>> Thanks for the discussion, Bruno, Sagar, and Matthias!
> >>>
> >>> It seems we've reached consensus on almost all of the discussion
> points.
> >>> I've updated the KIP with the following:
> >>> 1) renamed "timestampTo" in `get(key, timestampTo)` to "asOfTimestamp"
> to
> >>> clarify that this timestamp bound is inclusive, per the SQL guideline
> >> that
> >>> "AS OF " queries are inclusive. In the future, if we want to
> >>> introduce a timestamp range query, we can use `get(key, timestampFrom,
> >>> timestampTo)` and specify that timestampTo is exclusive in this method,
> >>> while avoiding confusing with the inclusive asOfTimestamp parameter in
> >> the
> >>> other method, given that the names are different.
> >>> 2) added a description of "history retention" semantics into the
> >>> VersionedKeyValueStore interface Javadoc, and updated the Javadoc for
> >>> `get(key, asOfTimestamp)` to mention explicitly that a null result is
> >>> returned if the provided timestamp bound is not within history
> retention.
> >>> 3) added a `delete(key, timestamp)` method (with return type
> >>> `ValueAndTimestamp`) to the VersionedKeyValueStore interface.
> >>> 4) updated the Javadoc for `segmentInterval` to clarify that the only
> >>> reason a user might be interested in this parameter is performance.
> >>>
> >>> Other points we discussed which did not result in updates include:
> >>> 5) whether to automatically update the `min.compaction.lag.ms` config
> on
> >>> changelog topics when 

[VOTE] KIP-889 Versioned State Stores

[Victoria Xia]

Hi all,

I'd like to start a vote on KIP-889 for introducing versioned key-value
state stores to Kafka Streams:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores

The discussion thread has been open for a few weeks now and has converged
among the current participants.

Thanks,
Victoria

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1439

[Apache Jenkins Server]

See 

[VOTE] 3.3.2 RC0

[Chris Egerton]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.3.2.

This is a bugfix release with several fixes since the release of 3.3.1. A
few of the major issues include:

* KAFKA-14358 Users should not be able to create a regular topic name
__cluster_metadata
KAFKA-14379 Consumer should refresh preferred read replica on update
metadata
* KAFKA-13586 Prevent exception thrown during connector update from
crashing distributed herder


Release notes for the 3.3.2 release:
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/RELEASE_NOTES.html



*** Please download, test and vote by Tuesday, December 20, 10pm UTC

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cegerton/kafka-3.3.2-rc0/javadoc/

* Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
https://github.com/apache/kafka/releases/tag/3.3.2-rc0

* Documentation:
https://kafka.apache.org/33/documentation.html

* Protocol:
https://kafka.apache.org/33/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/135/testReport/

System tests:
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1670984851--apache--3.3--22af3f29ce/2022-12-13--001./2022-12-13--001./report.html
(initial with three flaky failures)
Follow-up system tests:
https://home.apache.org/~cegerton/system_tests/2022-12-14--015/report.html,
https://home.apache.org/~cegerton/system_tests/2022-12-14--016/report.html,
http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1671061000--apache--3.3--69fbaf2457/2022-12-14--001./2022-12-14--001./report.html

(Note that the exact commit used for some of the system test runs will not
precisely match the commit for the release candidate, but that all
differences between those two commits should have no effect on the
relevance or accuracy of the test results.)

Thanks,

Chris

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Rajini Sivaram]

Congratulations, Viktor!

Regards,

Rajini


On Thu, Dec 15, 2022 at 11:41 AM Ron Dagostino  wrote:

> Congrats to you too, Victor!
>
> Ron
>
> > On Dec 15, 2022, at 4:59 AM, Viktor Somogyi-Vass <
> viktor.somo...@cloudera.com.invalid> wrote:
> >
> > Thank you everyone! :)
> >
> >> On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >> Congratulations Viktor!
> >>
> >>> On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
> >>>  wrote:
> >>>
> >>> Congratulations, Viktor!
> >>
>

Re: [ANNOUNCE] New committer: Ron Dagostino

[Rajini Sivaram]

Congratulations, Ron! Well deserved!!

Regards,

Rajini

On Thu, Dec 15, 2022 at 11:42 AM Ron Dagostino  wrote:

> Thank you, everyone!
>
> Ron
>
> > On Dec 15, 2022, at 5:09 AM, Bruno Cadonna  wrote:
> >
> > Congrats Ron!
> >
> > Best,
> > Bruno
> >
> >> On 15.12.22 10:23, Viktor Somogyi-Vass wrote:
> >> Congrats Ron! :)
> >>> On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >>> wrote:
> >>> Congratulations Ron!
> >>>
> >>> On Thu, Dec 15, 2022 at 9:41 AM Eslam Farag 
> wrote:
> 
>  Congratulations, Ron ☺️
> 
>  On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley 
> wrote:
> 
> > Congratulations!
> >
> > On Thu, 15 Dec 2022 at 07:40, Satish Duggana <
> satish.dugg...@gmail.com
> 
> > wrote:
> >
> >> Congratulations, Ron!!
> >>
> >> On Thu, 15 Dec 2022 at 07:48, ziming deng  >
> >> wrote:
> >>
> >>> Congratulations, Ron!
> >>> Well deserved!
> >>>
> >>> --
> >>> Ziming
> >>>
>  On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> 
>  Congratulations, Ron!
>  Well deserved!
> 
>  Luke
> >>>
> >>>
> >>
> >
> >>>
>

[jira] [Resolved] (KAFKA-14427) Add support for ZK migration multi-ops transaction

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-14427.
--
Resolution: Fixed

> Add support for ZK migration multi-ops transaction
> --
>
> Key: KAFKA-14427
> URL: https://issues.apache.org/jira/browse/KAFKA-14427
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14448) ZK brokers register with KRaft during migration

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-14448.
--
Resolution: Fixed

> ZK brokers register with KRaft during migration
> ---
>
> Key: KAFKA-14448
> URL: https://issues.apache.org/jira/browse/KAFKA-14448
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14428) Add Records, RPCs and Configs for KIP-866

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-14428.
--
Resolution: Fixed

> Add Records, RPCs and Configs for KIP-866
> -
>
> Key: KAFKA-14428
> URL: https://issues.apache.org/jira/browse/KAFKA-14428
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14427) Add support for ZK migration multi-ops transaction

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur reopened KAFKA-14427:
--
  Assignee: Colin McCabe  (was: David Arthur)

> Add support for ZK migration multi-ops transaction
> --
>
> Key: KAFKA-14427
> URL: https://issues.apache.org/jira/browse/KAFKA-14427
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: David Arthur
>Assignee: Colin McCabe
>Priority: Major
> Fix For: 3.4.1
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14427) Add support for ZK migration multi-ops transaction

[David Arthur (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur resolved KAFKA-14427.
--
Resolution: Fixed

> Add support for ZK migration multi-ops transaction
> --
>
> Key: KAFKA-14427
> URL: https://issues.apache.org/jira/browse/KAFKA-14427
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1438

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-14496) Wrong Base64 encoder used by OIDC OAuthBearerLoginCallbackHandler

[Endre Vig (Jira)]

Endre Vig created KAFKA-14496:
-

 Summary: Wrong Base64 encoder used by OIDC 
OAuthBearerLoginCallbackHandler
 Key: KAFKA-14496
 URL: https://issues.apache.org/jira/browse/KAFKA-14496
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.3.1
Reporter: Endre Vig


Currently our team is setting up a blueprint for our Kafka consumers/producers 
to provide guidelines on how to connect to our broker using the OIDC security 
mechanism. The blueprint is written in Java using the latest 3.3.1 Kafka 
library dependencies managed by Spring Boot 3.0.0.

While trying to use the new built-in 
{{org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler}}
 introduced by [KIP-768: Extend SASL/OAUTHBEARER with Support for 
OIDC|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575],
 we've noticed that some calls to retrieve the token work out well, while some 
of them (seemingly randomly) are failing with 401 Unauthorized.

After some debugging we've got to the conclusion that the faulty behavior is 
caused by 
{{org.apache.kafka.common.security.oauthbearer.secured.HttpAccessTokenRetriever#formatAuthorizationHeader:}}
{code:java}
static String formatAuthorizationHeader(String clientId, String clientSecret) {
    clientId = sanitizeString("the token endpoint request client ID parameter", 
clientId);
    clientSecret = sanitizeString("the token endpoint request client secret 
parameter", clientSecret);    String s = String.format("%s:%s", clientId, 
clientSecret);
    String encoded = Base64.getUrlEncoder().encodeToString(Utils.utf8(s));
    return String.format("Basic %s", encoded);
} {code}
The above code is using {{java.util.Base64#getUrlEncoder}} on line 311 to 
encode the authorization header value, which is using the alphabet described in 
[section 5 of the RFC|https://www.rfc-editor.org/rfc/rfc4648#section-5] during 
the encoding algorithm. As stated by the Basic Authentication Scheme 
[definition|https://www.rfc-editor.org/rfc/rfc7617#section-2] however, [section 
4 of the RFC|https://www.rfc-editor.org/rfc/rfc4648#section-4] should be used:

??4. and obtains the basic-credentials by encoding this octet sequence using 
Base64 ([RFC4648], Section 4) into a sequence of US-ASCII characters 
([RFC0020]).??

The difference between the 2 alphabets are only on two characters (62: '+' vs. 
'-' and 63: '/' vs. '_'), that's why the 401 Unauthorized response arises only 
for certain credential values.

Here's a concrete example use case:

 
{code:java}
String s = String.format("%s:%s", "SOME_RANDOM_LONG_USER_01234", 
"9Q|0`8i~ute-n9ksjLWb\\50\"AX@UUED5E");
System.out.println(Base64.getUrlEncoder().encodeToString(Utils.utf8(s))); {code}
would print out:

 

 
{code:java}
U09NRV9SQU5ET01fTE9OR19VU0VSXzAxMjM0OjlRfDBgOGl-dXRlLW45a3NqTFdiXDUwIkFYQFVVRUQ1RQ==
 {code}
while

 

 
{code:java}
String s = String.format("%s:%s", "SOME_RANDOM_LONG_USER_01234", 
"9Q|0`8i~ute-n9ksjLWb\\50\"AX@UUED5E");
System.out.println(Base64.getEncoder().encodeToString(Utils.utf8(s))); {code}
would give:

 

 
{code:java}
U09NRV9SQU5ET01fTE9OR19VU0VSXzAxMjM0OjlRfDBgOGl+dXRlLW45a3NqTFdiXDUwIkFYQFVVRUQ1RQ==
 {code}
Please notice the '-' vs. '+' characters.

 

The 2 code snippets above would not behave differently for different 
credentials, where the encoded result doesn't use the 62th character of the 
alphabet:
{code:java}
String s = String.format("%s:%s", "SHORT_USER_01234", 
"9Q|0`8i~ute-n9ksjLWb\\50\"AX@UUED5E");
System.out.println(Base64.getEncoder().encodeToString(Utils.utf8(s))); {code}
{code:java}
U0hPUlRfVVNFUl8wMTIzNDo5UXwwYDhpfnV0ZS1uOWtzakxXYlw1MCJBWEBVVUVENUU=

{code}
So as a *conclusion* I would suggest that line 311 of 
{{HttpAccessTokenRetriever}} should be modified to use 
{{Base64.getEncoder().encodeToString(...)}} instead of 
{{Base64.getUrlEncoder().encodeToString(...).}} 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Ron Dagostino

[Ron Dagostino]

Thank you, everyone!

Ron

> On Dec 15, 2022, at 5:09 AM, Bruno Cadonna  wrote:
> 
> Congrats Ron!
> 
> Best,
> Bruno
> 
>> On 15.12.22 10:23, Viktor Somogyi-Vass wrote:
>> Congrats Ron! :)
>>> On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
>>> wrote:
>>> Congratulations Ron!
>>> 
>>> On Thu, Dec 15, 2022 at 9:41 AM Eslam Farag  wrote:
 
 Congratulations, Ron ☺️
 
 On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley  wrote:
 
> Congratulations!
> 
> On Thu, 15 Dec 2022 at 07:40, Satish Duggana >>> 
> wrote:
> 
>> Congratulations, Ron!!
>> 
>> On Thu, 15 Dec 2022 at 07:48, ziming deng 
>> wrote:
>> 
>>> Congratulations, Ron!
>>> Well deserved!
>>> 
>>> --
>>> Ziming
>>> 
 On Dec 15, 2022, at 09:16, Luke Chen  wrote:
 
 Congratulations, Ron!
 Well deserved!
 
 Luke
>>> 
>>> 
>> 
> 
>>> 

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Ron Dagostino]

Congrats to you too, Victor!

Ron

> On Dec 15, 2022, at 4:59 AM, Viktor Somogyi-Vass 
>  wrote:
> 
> Thank you everyone! :)
> 
>> On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
>> wrote:
>> 
>> Congratulations Viktor!
>> 
>>> On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
>>>  wrote:
>>> 
>>> Congratulations, Viktor!
>> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1437

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 528987 lines...]
[2022-12-15T11:22:20.187Z] 
[2022-12-15T11:22:20.187Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testPropagateLogDir() PASSED
[2022-12-15T11:22:20.187Z] 
[2022-12-15T11:22:20.187Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testGetDataAndStat() STARTED
[2022-12-15T11:22:20.187Z] 
[2022-12-15T11:22:20.187Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testGetDataAndStat() PASSED
[2022-12-15T11:22:20.187Z] 
[2022-12-15T11:22:20.187Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testReassignPartitionsInProgress() STARTED
[2022-12-15T11:22:21.140Z] 
[2022-12-15T11:22:21.140Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testReassignPartitionsInProgress() PASSED
[2022-12-15T11:22:21.140Z] 
[2022-12-15T11:22:21.140Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() STARTED
[2022-12-15T11:22:21.140Z] 
[2022-12-15T11:22:21.140Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() PASSED
[2022-12-15T11:22:21.140Z] 
[2022-12-15T11:22:21.140Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testCreateTopLevelPaths() STARTED
[2022-12-15T11:22:22.093Z] 
[2022-12-15T11:22:22.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2022-12-15T11:22:22.094Z] 
[2022-12-15T11:22:22.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2022-12-15T11:22:22.094Z] 
[2022-12-15T11:22:22.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2022-12-15T11:22:22.094Z] 
[2022-12-15T11:22:22.094Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2022-12-15T11:22:23.047Z] 
[2022-12-15T11:22:23.047Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testAclMethods() STARTED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testAclMethods() PASSED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2022-12-15T11:22:24.000Z] 
[2022-12-15T11:22:24.000Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2022-12-15T11:22:24.950Z] 
[2022-12-15T11:22:24.950Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2022-12-15T11:22:24.950Z] 
[2022-12-15T11:22:24.950Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 165 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2022-12-15T11:22:24.950Z] 
[2022-12-15T11:22:24.950Z] Gradle Test Run :core:integrationTest > Gradle Test 

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Bruno Cadonna]

Congrats Viktor!

Best,
Bruno

On 15.12.22 10:58, Viktor Somogyi-Vass wrote:

Thank you everyone! :)

On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
wrote:


Congratulations Viktor!

On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
 wrote:


Congratulations, Viktor!

Re: [ANNOUNCE] New committer: Ron Dagostino

[Bruno Cadonna]

Congrats Ron!

Best,
Bruno

On 15.12.22 10:23, Viktor Somogyi-Vass wrote:

Congrats Ron! :)

On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
wrote:


Congratulations Ron!

On Thu, Dec 15, 2022 at 9:41 AM Eslam Farag  wrote:


Congratulations, Ron ☺️

On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley  wrote:


Congratulations!

On Thu, 15 Dec 2022 at 07:40, Satish Duggana 


wrote:


Congratulations, Ron!!

On Thu, 15 Dec 2022 at 07:48, ziming deng 
wrote:


Congratulations, Ron!
Well deserved!

--
Ziming


On Dec 15, 2022, at 09:16, Luke Chen  wrote:

Congratulations, Ron!
Well deserved!

Luke

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Viktor Somogyi-Vass]

Thank you everyone! :)

On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
wrote:

> Congratulations Viktor!
>
> On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
>  wrote:
> >
> > Congratulations, Viktor!
>

Re: [ANNOUNCE] New committer: Ron Dagostino

[Viktor Somogyi-Vass]

Congrats Ron! :)

On Thu, Dec 15, 2022 at 10:22 AM Mickael Maison 
wrote:

> Congratulations Ron!
>
> On Thu, Dec 15, 2022 at 9:41 AM Eslam Farag  wrote:
> >
> > Congratulations, Ron ☺️
> >
> > On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley  wrote:
> >
> > > Congratulations!
> > >
> > > On Thu, 15 Dec 2022 at 07:40, Satish Duggana  >
> > > wrote:
> > >
> > > > Congratulations, Ron!!
> > > >
> > > > On Thu, 15 Dec 2022 at 07:48, ziming deng 
> > > > wrote:
> > > >
> > > > > Congratulations, Ron!
> > > > > Well deserved!
> > > > >
> > > > > --
> > > > > Ziming
> > > > >
> > > > > > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> > > > > >
> > > > > > Congratulations, Ron!
> > > > > > Well deserved!
> > > > > >
> > > > > > Luke
> > > > >
> > > > >
> > > >
> > >
>

[jira] [Created] (KAFKA-14495) Improve the RemoteIndexCacheTest

[Luke Chen (Jira)]

Luke Chen created KAFKA-14495:
-

 Summary: Improve the RemoteIndexCacheTest
 Key: KAFKA-14495
 URL: https://issues.apache.org/jira/browse/KAFKA-14495
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


https://github.com/apache/kafka/pull/11390/files#r1049392445

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Ron Dagostino

[Mickael Maison]

Congratulations Ron!

On Thu, Dec 15, 2022 at 9:41 AM Eslam Farag  wrote:
>
> Congratulations, Ron ☺️
>
> On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley  wrote:
>
> > Congratulations!
> >
> > On Thu, 15 Dec 2022 at 07:40, Satish Duggana 
> > wrote:
> >
> > > Congratulations, Ron!!
> > >
> > > On Thu, 15 Dec 2022 at 07:48, ziming deng 
> > > wrote:
> > >
> > > > Congratulations, Ron!
> > > > Well deserved!
> > > >
> > > > --
> > > > Ziming
> > > >
> > > > > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> > > > >
> > > > > Congratulations, Ron!
> > > > > Well deserved!
> > > > >
> > > > > Luke
> > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Mickael Maison]

Congratulations Viktor!

On Thu, Dec 15, 2022 at 10:06 AM Tamas Barnabas Egyed
 wrote:
>
> Congratulations, Viktor!

Re: [ANNOUNCE] New committer: Viktor Somogyi-Vass

[Tamas Barnabas Egyed]

Congratulations, Viktor!

Re: [ANNOUNCE] New committer: Ron Dagostino

[Eslam Farag]

Congratulations, Ron ☺️

On Thu, 15 Dec 2022 at 10:40 AM Tom Bentley  wrote:

> Congratulations!
>
> On Thu, 15 Dec 2022 at 07:40, Satish Duggana 
> wrote:
>
> > Congratulations, Ron!!
> >
> > On Thu, 15 Dec 2022 at 07:48, ziming deng 
> > wrote:
> >
> > > Congratulations, Ron!
> > > Well deserved!
> > >
> > > --
> > > Ziming
> > >
> > > > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> > > >
> > > > Congratulations, Ron!
> > > > Well deserved!
> > > >
> > > > Luke
> > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Ron Dagostino

[Tom Bentley]

Congratulations!

On Thu, 15 Dec 2022 at 07:40, Satish Duggana 
wrote:

> Congratulations, Ron!!
>
> On Thu, 15 Dec 2022 at 07:48, ziming deng 
> wrote:
>
> > Congratulations, Ron!
> > Well deserved!
> >
> > --
> > Ziming
> >
> > > On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> > >
> > > Congratulations, Ron!
> > > Well deserved!
> > >
> > > Luke
> >
> >
>