[jira] [Created] (KAFKA-16476) Check whether TestInfo contains correct parameter name

[PoAn Yang (Jira)]

PoAn Yang created KAFKA-16476:
-

 Summary: Check whether TestInfo contains correct parameter name
 Key: KAFKA-16476
 URL: https://issues.apache.org/jira/browse/KAFKA-16476
 Project: Kafka
  Issue Type: Test
Reporter: PoAn Yang
Assignee: PoAn Yang


After KAFKA-16472, we fixed missing parameter name in `TestInfo#getDisplayName` 
in Java. However, we don't have check for cases like users don't give empty or 
incorrect parameter name in test function.

For example, before the fix in KAFKA-16472, 
`DeleteOffsetsConsumerGroupCommandIntegrationTest#testDeleteOffsetsNonExistingGroup`
 shows `(String).`.

For incorrect parameter name, if users give parameter name like `Quorum`, but 
not `quorum`, the `TestInfoUtils#isKRaft` still can't work as expected.

Probably, we can add check like containing `.`, but not `=` for empty case and 
containing `zk` or `kraft, but not `quorum` for incorrect parameter name case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #127

[Apache Jenkins Server]

See 

[ANNOUNCE] Apache Kafka 3.6.2

[Manikumar]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.6.2

This is a bug fix release and it includes fixes and improvements from 28 JIRAs.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.6.2/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and Scala 2.13) from:
https://kafka.apache.org/downloads#3.6.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 40 contributors to this release!
(Please report an unintended omission)

Akhilesh Chaganti, Andrew Schofield, Anton Liauchuk, Bob Barrett,
Bruno Cadonna, Cheng-Kai, Zhang, Chia-Ping Tsai, Chris Egerton, Colin
P. McCabe, Colin Patrick McCabe, David Arthur, David Jacot, David Mao,
Divij Vaidya, Edoardo Comar, Emma Humber, Gaurav Narula, Greg Harris,
hudeqi, Ismael Juma, Jason Gustafson, Jim Galasyn, Joel Hamill, Johnny
Hsu, José Armando García Sancio, Justine Olshan, Luke Chen, Manikumar
Reddy, Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Mike
Lloyd, Paolo Patierno, PoAn Yang, Ron Dagostino, Sagar Rao, Stanislav
Kozlovski, Walker Carlson, Yash Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,

Manikumar
Release Manager for Apache Kafka 3.6.2

Re: [DISCUSS] KIP-932: Queues for Kafka

[David Arthur]

Andrew, thanks for the KIP! This is a pretty exciting effort.

I've finally made it through the KIP, still trying to grok the whole thing.
Sorry if some of my questions are basic :)


Concepts:

70. Does the Group Coordinator communicate with the Share Coordinator over
RPC or directly in-process?

71. For preventing name collisions with regular consumer groups, could we
define a reserved share group prefix? E.g., the operator defines "sg_" as a
prefix for share groups only, and if a regular consumer group tries to use
that name it fails.

72. When a consumer tries to use a share group, or a share consumer tries
to use a regular group, would INVALID_GROUP_ID make more sense
than INCONSISTENT_GROUP_PROTOCOL?



Share Group Membership:

73. What goes in the Metadata field for TargetAssignment#Member and
Assignment?

74. Under Trigger a rebalance, it says we rebalance when the partition
metadata changes. Would this be for any change, or just certain ones? For
example, if a follower drops out of the ISR and comes back, we probably
don't need to rebalance.

75. "For a share group, the group coordinator does *not* persist the
assignment" Can you explain why this is not needed?

76. " If the consumer just failed to heartbeat due to a temporary pause, it
could in theory continue to fetch and acknowledge records. When it finally
sends a heartbeat and realises it’s been kicked out of the group, it should
stop fetching records because its assignment has been revoked, and rejoin
the group."

A consumer with a long pause might still deliver some buffered records, but
if the share group coordinator has expired its session, it wouldn't accept
acknowledgments for that share consumer. In such a case, is any kind of
error raised to the application like "hey, I know we gave you these
records, but really we shouldn't have" ?


-

Record Delivery and acknowledgement

77. If we guarantee that a ShareCheckpoint is written at least every so
often, could we add a new log compactor that avoids compacting ShareDelta-s
that are still "active" (i.e., not yet superceded by a new
ShareCheckpoint). Mechnically, this could be done by keeping the LSO no
greater than the oldest "active" ShareCheckpoint. This might let us remove
the DeltaIndex thing.

78. Instead of the State in the ShareDelta/Checkpoint records, how about
MessageState? (State is kind of overloaded/ambiguous)

79. One possible limitation with the current persistence model is that all
the share state is stored in one topic. It seems like we are going to be
storing a lot more state than we do in __consumer_offsets since we're
dealing with message-level acks. With aggressive checkpointing and
compaction, we can mitigate the storage requirements, but the throughput
could be a limiting factor. Have we considered other possibilities for
persistence?


Cheers,
David

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

I'm hoping this covers the majority of comments. I will go ahead and open
the vote in the next day or so.

Thanks,
Justine

On Wed, Apr 3, 2024 at 3:31 PM Justine Olshan  wrote:

> Find and replace has failed me :(
>
> Group version seems a little vague, but we can update it. Hopefully find
> and replace won't fail me again, otherwise I will get another email on this.
>
> Justine
>
> On Wed, Apr 3, 2024 at 12:15 PM David Jacot 
> wrote:
>
>> Thanks, Justine.
>>
>> * Should we also use `group.version` (GV) as I suggested in my previous
>> message in order to be consistent?
>> * Should we add both names to the `Public Interfaces` section?
>> * There is still at least one usage of `transaction.protocol.verison` in
>> the KIP too.
>>
>> Best,
>> David
>>
>> On Wed, Apr 3, 2024 at 6:29 PM Justine Olshan
>> 
>> wrote:
>>
>> > I had missed the David's message yesterday about the naming for
>> transaction
>> > version vs transaction protocol version.
>> >
>> > After some offline discussion with Jun, Artem, and David, we agreed that
>> > transaction version is simpler and conveys more than just protocol
>> changes
>> > (flexible records for example)
>> >
>> > I will update the KIP as well as KIP-890
>> >
>> > Thanks,
>> > Justine
>> >
>> > On Tue, Apr 2, 2024 at 2:50 PM Justine Olshan 
>> > wrote:
>> >
>> > > Updated!
>> > >
>> > > Justine
>> > >
>> > > On Tue, Apr 2, 2024 at 2:40 PM Jun Rao 
>> wrote:
>> > >
>> > >> Hi, Justine,
>> > >>
>> > >> Thanks for the reply.
>> > >>
>> > >> 21. Sounds good. It would be useful to document that.
>> > >>
>> > >> 22. Should we add the IV in "metadata.version=17 has no dependencies"
>> > too?
>> > >>
>> > >> Jun
>> > >>
>> > >>
>> > >> On Tue, Apr 2, 2024 at 11:31 AM Justine Olshan
>> > >> 
>> > >> wrote:
>> > >>
>> > >> > Jun,
>> > >> >
>> > >> > 21. Next producer ID field doesn't need to be populated for TV 1.
>> We
>> > >> don't
>> > >> > have the same need to retain this since it is written directly to
>> the
>> > >> > transaction log in InitProducerId. It is only needed for KIP-890
>> part
>> > 2
>> > >> /
>> > >> > TV 2.
>> > >> >
>> > >> > 22. We can do that.
>> > >> >
>> > >> > Justine
>> > >> >
>> > >> > On Tue, Apr 2, 2024 at 10:41 AM Jun Rao 
>> > >> wrote:
>> > >> >
>> > >> > > Hi, Justine,
>> > >> > >
>> > >> > > Thanks for the reply.
>> > >> > >
>> > >> > > 21. What about the new NextProducerId field? Will that be
>> populated
>> > >> with
>> > >> > TV
>> > >> > > 1?
>> > >> > >
>> > >> > > 22. In the dependencies output, should we show both IV and level
>> for
>> > >> > > metadata.version too?
>> > >> > >
>> > >> > > Jun
>> > >> > >
>> > >> > > On Mon, Apr 1, 2024 at 4:43 PM Justine Olshan
>> > >> > > > >> > > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > Hi Jun,
>> > >> > > >
>> > >> > > > 20. I can update the KIP.
>> > >> > > >
>> > >> > > > 21. This is used to complete some of the work with KIP-360. (We
>> > use
>> > >> > > > previous producer ID there, but never persisted it which was in
>> > the
>> > >> KIP
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89068820
>> > >> )
>> > >> > > > The KIP also mentions including previous epoch but we
>> explained in
>> > >> this
>> > >> > > KIP
>> > >> > > > how we can figure this out.
>> > >> > > >
>> > >> > > > Justine
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > On Mon, Apr 1, 2024 at 3:56 PM Jun Rao
>> 
>> > >> > wrote:
>> > >> > > >
>> > >> > > > > Hi, Justine,
>> > >> > > > >
>> > >> > > > > Thanks for the updated KIP. A couple of more comments.
>> > >> > > > >
>> > >> > > > > 20. Could we show the output of version-mapping?
>> > >> > > > >
>> > >> > > > > 21. "Transaction version 1 will include the flexible fields
>> in
>> > the
>> > >> > > > > transaction state log, and transaction version 2 will include
>> > the
>> > >> > > changes
>> > >> > > > > to the transactional protocol as described by KIP-890 (epoch
>> > bumps
>> > >> > and
>> > >> > > > > implicit add partitions.)"
>> > >> > > > >   So TV 1 enables the writing of new tagged fields like
>> > >> > PrevProducerId?
>> > >> > > > But
>> > >> > > > > those fields are only usable after the epoch bump, right?
>> What
>> > >> > > > > functionality does TV 1 achieve?
>> > >> > > > >
>> > >> > > > > Jun
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Mon, Apr 1, 2024 at 2:06 PM Justine Olshan
>> > >> > > > > > >> > > > > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > I have also updated the KIP to mention the feature tool's
>> > >> > --metadata
>> > >> > > > flag
>> > >> > > > > > will be deprecated.
>> > >> > > > > > It will still work for users as they learn the new flag,
>> but a
>> > >> > > warning
>> > >> > > > > > indicating the alternatives will be shown.
>> > >> > > > > >
>> > >> > > > > > Justine
>> > >> > > > > >
>> > >> > > > > > On Thu, Mar 28, 2024 at 11:03 AM Justine Olshan <
>> > >> > > jols...@confluent.io>
>> > >> > > > > > wrote:
>> > >> > > > > >
>>

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

[Sophie Blee-Goldman]

Before I go into my own feedback, which I know not everyone has the time to
read, I just want to announce that we plan to go over this KIP during the
next Streams project sync on April 11. If you are interested in joining the
discussion and need an invitation, just reach out to me in a separate
thread. All are welcome.

Now on to the KIP itself...

Made a pass over the latest KIP and discussion, and had just a few (mostly
minor) points. I'll start with the trivial:

1. nit: the config name is a bit awkward with the double "assignor" in the
name, in my opinion we should either remove the "partition.assignor" part
of the name altogether

2. Just to clarify a point about the `#previousOwnerForPartition` method,
that was originally needed not because users might mix up the
task->partition mapping, but in order to ensure an assignment that
satisfies the cooperative rebalancing protocol of not assigning any
partition to a thread if it had a (different) previous owner this
generation. Since there's no thread-level assignment in this KIP, and
perhaps more importantly I believe the StreamsPartitionAssignor should
still be responsible for guaranteeing cooperative rebalancing semantics
rather than forcing this on the user's TaskAssignor, I completely agree
that we don't need the  `#previousOwnerForPartition` API any more

3. On that note, while I agree that we should leave the thread-level
assignment within a node up to the StreamsPartitionAssignor, and leave it
out of this KIP, one thing that might be valuable is the ability to read
the final thread-level assignment, for example so users could implement
metrics or other monitoring of assignment. Right now the only way to know
which partition/task is assigned to which thread is to look at the logs,
which is hardly optimal. Have you considered including some kind of API for
the StreamsPartitionAssignor to inform the TaskAssignor of the eventual
thread-level assignment that it decides on?
FWIW I am personally fine with leaving this out of the current KIP, as the
current proposal would make it easy to do this as a quick follow-up KIP. It
also would become less useful if/when we decouple the consumers from the
processing threads and do away with thread-level assignment altogether,
though it might be a while before we completely switch over, so it could
still be useful to know the thread-level assignment for a while beyond 3.8.
Just wondering if you've thought about it

4. Lastly, regarding Almog's point 2/6, I do think it would be reasonable
to consolidate the top-level APIs in NodeAssignment in favor of a single
API. For example, something like this:

public interface NodeAssignment {
  ProcessID processId();

  long followupRebalanceDeadline();

  Map assignment();

  static class AssignedTask {
boolean isActive;
boolean isStateful;
  }
}

This would make things easier on the users, as they could build up this map
one task at a time, while also helping to enforce the requirement that a
Node not contain the active and standby version of the same task. The
StreamsPartitionAssignor can then loop through the map and separate the
assigned tasks into the sub-maps that it uses to perform the thread-level
assignment from there. Assuming this is what you had in mind for #2 Almog,
I'm in favor

However, I'm less in favor of combining APIs in NodeState, ie point #6 in
your/Almog's reply. If the goal is to keep things simple for users and make
it so that someone who only wanted to modify the existing assignment logic,
rather than re-build it from the ground up, could do so without a huge
hassle, then keeping all of these individual APIs would be necessary.
There's also a performance element to this, as (IIRC) most of those "extra"
APIs such as sorted tasks by lag are simply exposing pre-computed metadata
that gets attached to each node while the StreamsPartitionAssignor is
processing the rebalance inputs. Moving those to a static utility class
would mean having to recompute everything, as they are getters rather than
utilities, and due to the assignment complexity we have to be performance
sensitive. It will probably make more sense when you see it implemented.



On Sat, Nov 18, 2023 at 4:56 PM Guozhang Wang 
wrote:

> Hi Rohan,
>
> I took another look at the updated wiki page and do not have any major
> questions. Regarding returning a plugin object v.s. configuring a
> plugin object, I do not have a strong opinion except that the latter
> seems more consistent with existing patterns. Just curious, any other
> motivations to go with the latter from you?
>
>
> Guozhang
>
> On Thu, Nov 9, 2023 at 11:19 PM Rohan Desai 
> wrote:
> >
> > Thanks for the feedback so far! I think pretty much all of it is
> > reasonable. I'll reply to it inline:
> >
> > > 1. All the API logic is granular at the Task level, except the
> > previousOwnerForPartition func. I’m not clear what’s the motivation
> behind
> > it, does our controller also want to change how the partitions->tasks
> > mappin

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #168

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2782

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

[Jun Rao]

Hi, Abhijeet,

Thanks for the KIP. Left a few comments.

1. "A drawback of using the last-tiered-offset is that this new follower
would possess only a limited number of locally stored segments. Should it
ascend to the role of leader, there is a risk of needing to fetch these
segments from the remote storage, potentially impacting broker performance."
Since we support consumers fetching from followers, this is a potential
issue on the follower side too. In theory, it's possible for a segment to
be tiered immediately after rolling. In that case, there could be very
little data after last-tiered-offset. It would be useful to think through
how to address this issue.

2. ListOffsetsRequest:
2.1 Typically, we need to bump up the version of the request if we add a
new value for timestamp. See
https://github.com/apache/kafka/pull/10760/files#diff-fac7080d67da905a80126d58fc1745c9a1409de7ef7d093c2ac66a888b134633
.
2.2 Since this changes the inter broker request protocol, it would be
useful to have a section on upgrade (e.g. new IBP/metadata.version).

3. "Instead of fetching Earliest-Pending-Upload-Offset, it could fetch the
last-tiered-offset from the leader, and make a separate leader call to
fetch leader epoch for the following offset."
Why do we need to make a separate call for the leader epoch?
ListOffsetsResponse include both the offset and the corresponding epoch.

4. "Check if the follower replica is empty and if the feature to use
last-tiered-offset is enabled."
Why do we need to check if the follower replica is empty?

5. It can be confirmed by checking if the leader's Log-Start-Offset is the
same as the Leader's Local-Log-Start-Offset.
How does the follower know Local-Log-Start-Offset?

Jun

On Sat, Mar 30, 2024 at 5:51 AM Abhijeet Kumar 
wrote:

> Hi Christo,
>
> Thanks for reviewing the KIP.
>
> The follower needs the earliest-pending-upload-offset (and the
> corresponding leader epoch) from the leader.
> This is the first offset the follower will have locally.
>
> Regards,
> Abhijeet.
>
>
>
> On Fri, Mar 29, 2024 at 1:14 PM Christo Lolov 
> wrote:
>
> > Heya!
> >
> > First of all, thank you very much for the proposal, you have explained
> the
> > problem you want solved very well - I think a faster bootstrap of an
> empty
> > replica is definitely an improvement!
> >
> > For my understanding, which concrete offset do you want the leader to
> give
> > back to a follower - earliest-pending-upload-offset or the
> > latest-tiered-offset? If it is the second, then I believe KIP-1005 ought
> to
> > already be exposing that offset as part of the ListOffsets API, no?
> >
> > Best,
> > Christo
> >
> > On Wed, 27 Mar 2024 at 18:23, Abhijeet Kumar  >
> > wrote:
> >
> > > Hi All,
> > >
> > > I have created KIP-1023 to introduce follower fetch from tiered offset.
> > > This feature will be helpful in significantly reducing Kafka
> > > rebalance/rebuild times when the cluster is enabled with tiered
> storage.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset
> > >
> > > Feedback and suggestions are welcome.
> > >
> > > Regards,
> > > Abhijeet.
> > >
> >
>

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Jun,
Thanks for your comments and thorough review.

I’ve added a new ShareGroupMemberMetadata record which I refer to below.
Each member in a share group has a ShareGroupMemberMetadata record.
Previously, members needed to rejoin the share group if the coordinator changed,
but now the member ID is persisted and the members can continue to heartbeat 
with
the new coordinator with no disruption.

40. Here’s the flow for serving the initial ShareGroupHeartbeat request.
a) The group coordinator writes a ConsumerGroupMetadata record
a ShareGroupMemberMetadata record, and a ShareGroupPartitionMetadata record.
The assignments are not persisted, although the members are.
b) It sends an InitializeShareGroupState RPC to the share coordinator, but it
does NOT initialise the SPSO at this point. The state epoch is the group epoch.
c) The share coordinator serves the InitializeShareGroupState RPC. It writes
a ShareCheckpoint record to the __share_group_state topic and waits for it to 
replicate.
Then it responds to the RPC.
d) Back in the group coordinator, it writes an updated 
ShareGroupPartitionMetadata
to record the completed initialization. It then responds to the 
ShareGroupHeartbeat
record.

Only (a) needs to complete to consider the group created. A partition can only 
be
assigned to members in the share group once (b) to (d) are completed.

I’ve added more detail in the KIP.

41. It is the group coordinator that initializes the ShareGroupState. The state 
might
not contain the SPSO initially, so when the share-partition leader reads the 
state
and sees that the SPSO is not initialised (-1), it will set it to the initial 
value based
of the group.share.auto.offset.reset configuration.

I’ll clarify this in the KIP.

42. The group epoch is updated when membership changes or the group metadata
changes, such as when the number of partitions in a subscribed topic changes.
I don’t think the group epoch needs to be bumped when the share group offset is 
altered.

43. The group coordinator is notified when the number of partitions changes. If 
any
Share groups are subscribed to the topic which changed, the new partition will 
be
eligible for assignment to the members of the group and will be notified of the 
new
assignment on the new heartbeat response. The group epoch will be bumped.

44. There are similarities between fetch sessions from KIP-227 and share
sessions in this KIP. When the share session epoch is 0, it’s equivalent to
a full fetch from KIP-227 - all of the topic-partitions to be included in the 
session
are listed. When the share session epoch is neither 0 nor -1, it’s equivalent o
an incremental fetch from KIP-227 - only changes to the list of topic-partitions
are included, PLUS any piggybacked acknowledgements. When the share session
epoch is -1, it’s the final request and the share session is closed.

A share session is used for both ShareFetch and ShareAcknowledge requests and
the sequence of epochs is shared by them too.

45. Yes, I have fixed ShareGroupPartitionMetadataValue’s definition.

46. In the common case, they’re FileRecords. We just look at the batch header.
I believe the comment about zero-copy being impossible is out of date and I have
removed it.

47.
47.1. If a consumer processes multiple consecutive batches, it can send a
single AcknowledgementBatch that covers multiple batches.
47.2. I’m a bit on the fence about the BaseOffset/LastOffset question. Your
example is correct - =<1, 4> and then
=<5, 7>. I could rename BaseOffset to FirstOffset,
but I would also want to rename LastOffset to EndOffset. Is that a worthwhile
improvement? This is also related to question 56 below. If we can come up with
a comprehensive list of when the different terms are used, I’m happy to change.
Note that I’ve used StartOffset for the SPSO already, so I avoiding using that
name for batches.

48.
48.1. The cache contains a subset of the batch headers (base/lastOffset),
record states for the in-flight records, delivery counts, and the acquiring
member ID.
48.2. The number of in-flight records is controlled by the broker configuration
group.share.record.lock.partition.limit. I think I’ve set the default rather 
low (200),
but that seems appropriate for an early access delivery. Once we’ve had it 
running
with thousands of records in flight, then we can look to change the limits.

49. The broker does not ever need to decompress a batch. When a subset of
records are redelivered, the entire batch is sent to the client with just a 
subset
of the records acquired. The client decompresses the batch, skips over the
unacquired records, and delivers the acquired records to the application.

50. Yes, the endMarker record batches are sent to the client. When the client
collects the fetched records, it adds the endMarker record as a “gap" in the set
of acknowledgements waiting to be sent to the broker. The purpose of a gap is
to indicate to the broker that it sent an acquired record which was not a 
consumable
record at

[jira] [Created] (KAFKA-16475) Create unit test for TopicImageNode

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-16475:


 Summary: Create unit test for TopicImageNode
 Key: KAFKA-16475
 URL: https://issues.apache.org/jira/browse/KAFKA-16475
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16474) AsyncKafkaConsumer might rapidly send out successive heartbeat causing partitions getting revoked

[Philip Nee (Jira)]

Philip Nee created KAFKA-16474:
--

 Summary: AsyncKafkaConsumer might rapidly send out successive 
heartbeat causing partitions getting revoked
 Key: KAFKA-16474
 URL: https://issues.apache.org/jira/browse/KAFKA-16474
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


KAFKA-16389

We've discovered that in some uncommon cases, the consumer could send out 
successive heartbeats without waiting for the response to come back.  this 
might result in causing the consumer to revoke its just assigned assignments in 
some cases.  For example:

 

The consumer first sends out a heartbeat with epoch=0 and memberId='' 

The consumer then rapidly sends out another heartbeat with epoch=0 and 
memberId='' because it has not gotten any response and thus not updating its 
local state

 

The consumer receives assignments from the first heartbeat and reconciles its 
assignment.

 

Since the second heartbeat has epoch=0 and memberId='', the server will think 
this is a new member joining and therefore send out an empty assignment.  

 

The consumer receives the response from the second heartbeat.  Revoke all of 
its partitions.

 

There are 2 issues associate with this bug:
 # inflight logic
 # rapid poll: In the KAFKA-16389 we've observe consumer polling interval to be 
a few ms.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16473) KafkaDockerWrapper uses wrong cluster ID when formatting log dir

[Sebastian Marsching (Jira)]

Sebastian Marsching created KAFKA-16473:
---

 Summary: KafkaDockerWrapper uses wrong cluster ID when formatting 
log dir
 Key: KAFKA-16473
 URL: https://issues.apache.org/jira/browse/KAFKA-16473
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Sebastian Marsching


There is a bug in {{{}KafkaDockerWrapper{}}}, that causes {{Some()}} to be used when formatting the log dir when 
Kafka is started for the first time inside a Docker container.

More specifically, the problem is in {{{}formatStorageCmd{}}}: The code uses 
{{{}env.get("CLUSTER_ID"){}}}, but this returns an {{Option}} not a 
{{{}String{}}}.

The code should instead check whether the environment variable is set, raising 
an exception if it is not set.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16472) Integration tests in Java don't really run kraft case

[PoAn Yang (Jira)]

PoAn Yang created KAFKA-16472:
-

 Summary: Integration tests in Java don't really run kraft case
 Key: KAFKA-16472
 URL: https://issues.apache.org/jira/browse/KAFKA-16472
 Project: Kafka
  Issue Type: Test
Reporter: PoAn Yang
Assignee: PoAn Yang


Following test cases don't really run kraft case. The reason is that the test 
info doesn't contain parameter name, so it always returns false in 
TestInfoUtils#isKRaft.
 * TopicCommandIntegrationTest
 * DeleteConsumerGroupsTest
 * AuthorizerIntegrationTest
 * DeleteOffsetsConsumerGroupCommandIntegrationTest

 

We can add `options.compilerArgs += '-parameters'` after 
[https://github.com/apache/kafka/blob/376e9e20dbf7c7aeb6f6f666d47932c445eb6bd1/build.gradle#L264-L273]
 to fix it.

 

Also, we have to add `String quorum` to cases in 
DeleteOffsetsConsumerGroupCommandIntegrationTest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Thanks all for voting.

I'm now closing the vote. The vote passes with
- 3 +1 bindings votes from Divij Vaidya, Justine, and Manikumar
- 3 +1 non-binding votes from Andrew, Jakub, and  Lianet
- 0 -1 votes

I'll go ahead and finish the release process.


Thanks,

On Thu, Apr 4, 2024 at 8:53 PM Manikumar  wrote:

> +1 (binding) from my side also.
>
> - verified the signatures, artifacts
> - ran the tests on the source archive
> - verified the quickstarts
>
>
> Thanks
>
> On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
> wrote:
>
>> Thanks for clarifying Manikumar!
>>
>> +1 (binding) from me
>>
>> Justine
>>
>> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
>> wrote:
>>
>> > Hi Manikumar
>> >
>> > I verified the following:
>> > - all non-minor commits in the branch are captured in release notes
>> > - signature on the artifact match the public signature of Manikumar
>> > - basic topic creation & produce / consumer works with JDK8 + ARM +
>> Kafka
>> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
>> >
>> > Things look good to me. We don't need another RC for fixing docs.
>> >
>> > +1 (binding) from me.
>> >
>> > --
>> > Divij Vaidya
>> >
>> >
>> >
>> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
>> > wrote:
>> >
>> > > Hi Justine,
>> > >
>> > > Thanks for catching this. looks like we have missed updating
>> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
>> > >
>> > > I will make sure to use the correct version when updating docs for
>> 3.6.2
>> > > release.
>> > > I will also update 3.5 and 3.6 branches with the correct heading and
>> also
>> > > update the release wiki.
>> > >
>> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > we will remove the "-SNAPSHOT" suffix while updating the website
>> docs. we
>> > > may need to automate this in the release script.
>> > >
>> > >
>> > > [1]
>> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
>> > > [2]
>> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
>> > > > > >
>> > > wrote:
>> > >
>> > > > Thanks for clarifying!
>> > > > I took a look at the documentation.html file in there, and it said
>> 3.4.
>> > > Is
>> > > > that expected?
>> > > >
>> > > > There are some files that request fullDot version and that seemed
>> > closer
>> > > to
>> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
>> > > > The upgrade.html file also looked ok.
>> > > >
>> > > > Thanks for running the release and answering my questions!
>> > > > Justine
>> > > >
>> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar <
>> manikumar.re...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Justine,
>> > > > >
>> > > > > Yes, it is intended. For bug fix releases website docs will be
>> > updated
>> > > > > during the final release process.
>> > > > > We can verify the site-docs artifacts here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
>> > > > > These site-docs artifacts will be used to update website docs.
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
>> > > > > 
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Manikumar,
>> > > > > >
>> > > > > > I've verified the keys, scanned the artifacts, and other docs.
>> > > > > > I built from source and ran with a ZK cluster (since I saw that
>> we
>> > > > > updated
>> > > > > > ZK version in this release)
>> > > > > > I ran a few tests on this cluster.
>> > > > > >
>> > > > > > I also ran the 2.12 binary.
>> > > > > >
>> > > > > > I noticed the docs link (
>> > > > https://kafka.apache.org/36/documentation.html)
>> > > > > > mentions 3.6.1 as the latest. Is that intended?
>> > > > > > I will give my final vote when we figure this out.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Justine
>> > > > > >
>> > > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
>> > wrote:
>> > > > > >
>> > > > > > > Hi Manikumar, I did the following checks:
>> > > > > > >
>> > > > > > > - downloaded and built from src
>> > > > > > > - ran all unit test and integration test for clients
>> > > > > > > - ran quickstart with Kraft mode
>> > > > > > > - ran simple workloads with the console consumer/producer
>> > > > > > > - checked all links
>> > > > > > >
>> > > > > > > All looks good to me with this.
>> > > > > > >
>> > > > > > > +1 (non-binding)
>> > > > > > >
>> > > > > > > Thanks!
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar <
>> maniku...@apache.org>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Gentle reminder. Please download, test and vote for the
>> > release.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > >
>> > > > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar <
>> > maniku...@apache.org>
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi All,
>> > > > > > > > >
>> > > > > >

Re: [VOTE] 3.6.2 RC2

[Manikumar]

+1 (binding) from my side also.

- verified the signatures, artifacts
- ran the tests on the source archive
- verified the quickstarts


Thanks

On Thu, Apr 4, 2024 at 7:46 PM Justine Olshan 
wrote:

> Thanks for clarifying Manikumar!
>
> +1 (binding) from me
>
> Justine
>
> On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya 
> wrote:
>
> > Hi Manikumar
> >
> > I verified the following:
> > - all non-minor commits in the branch are captured in release notes
> > - signature on the artifact match the public signature of Manikumar
> > - basic topic creation & produce / consumer works with JDK8 + ARM + Kafka
> > 3.6.2 + Scala 2.12 + ZK + compression (zstd)
> >
> > Things look good to me. We don't need another RC for fixing docs.
> >
> > +1 (binding) from me.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Thanks for catching this. looks like we have missed updating
> > > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
> > >
> > > I will make sure to use the correct version when updating docs for
> 3.6.2
> > > release.
> > > I will also update 3.5 and 3.6 branches with the correct heading and
> also
> > > update the release wiki.
> > >
> > > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > we will remove the "-SNAPSHOT" suffix while updating the website docs.
> we
> > > may need to automate this in the release script.
> > >
> > >
> > > [1]
> https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
> > > [2]
> https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
> > >
> > >
> > > Thanks,
> > >
> > > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
> >  > > >
> > > wrote:
> > >
> > > > Thanks for clarifying!
> > > > I took a look at the documentation.html file in there, and it said
> 3.4.
> > > Is
> > > > that expected?
> > > >
> > > > There are some files that request fullDot version and that seemed
> > closer
> > > to
> > > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > > The upgrade.html file also looked ok.
> > > >
> > > > Thanks for running the release and answering my questions!
> > > > Justine
> > > >
> > > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > >
> > > > > Yes, it is intended. For bug fix releases website docs will be
> > updated
> > > > > during the final release process.
> > > > > We can verify the site-docs artifacts here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > > > > These site-docs artifacts will be used to update website docs.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Manikumar,
> > > > > >
> > > > > > I've verified the keys, scanned the artifacts, and other docs.
> > > > > > I built from source and ran with a ZK cluster (since I saw that
> we
> > > > > updated
> > > > > > ZK version in this release)
> > > > > > I ran a few tests on this cluster.
> > > > > >
> > > > > > I also ran the 2.12 binary.
> > > > > >
> > > > > > I noticed the docs link (
> > > > https://kafka.apache.org/36/documentation.html)
> > > > > > mentions 3.6.1 as the latest. Is that intended?
> > > > > > I will give my final vote when we figure this out.
> > > > > >
> > > > > > Thanks,
> > > > > > Justine
> > > > > >
> > > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
> > wrote:
> > > > > >
> > > > > > > Hi Manikumar, I did the following checks:
> > > > > > >
> > > > > > > - downloaded and built from src
> > > > > > > - ran all unit test and integration test for clients
> > > > > > > - ran quickstart with Kraft mode
> > > > > > > - ran simple workloads with the console consumer/producer
> > > > > > > - checked all links
> > > > > > >
> > > > > > > All looks good to me with this.
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar  >
> > > > wrote:
> > > > > > >
> > > > > > > > Gentle reminder. Please download, test and vote for the
> > release.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar <
> > maniku...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > System test runs are green. There were 13 test failures in
> > the
> > > > > first
> > > > > > > run.
> > > > > > > > > All the failed tests passed in the second run.
> > > > > > > > >
> > > > > > > > > System test results:
> > > > > > > > >
> > > > https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > On Thu, Mar 28, 2024 at 3:21 PM Manikumar <
> > > maniku...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hello K

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #126

[Apache Jenkins Server]

See 

Re: [VOTE] 3.6.2 RC2

[Justine Olshan]

Thanks for clarifying Manikumar!

+1 (binding) from me

Justine

On Thu, Apr 4, 2024 at 3:19 AM Divij Vaidya  wrote:

> Hi Manikumar
>
> I verified the following:
> - all non-minor commits in the branch are captured in release notes
> - signature on the artifact match the public signature of Manikumar
> - basic topic creation & produce / consumer works with JDK8 + ARM + Kafka
> 3.6.2 + Scala 2.12 + ZK + compression (zstd)
>
> Things look good to me. We don't need another RC for fixing docs.
>
> +1 (binding) from me.
>
> --
> Divij Vaidya
>
>
>
> On Thu, Apr 4, 2024 at 10:04 AM Manikumar 
> wrote:
>
> > Hi Justine,
> >
> > Thanks for catching this. looks like we have missed updating
> > `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
> >
> > I will make sure to use the correct version when updating docs for 3.6.2
> > release.
> > I will also update 3.5 and 3.6 branches with the correct heading and also
> > update the release wiki.
> >
> > >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > we will remove the "-SNAPSHOT" suffix while updating the website docs. we
> > may need to automate this in the release script.
> >
> >
> > [1] https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
> > [2] https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
> >
> >
> > Thanks,
> >
> > On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan
>  > >
> > wrote:
> >
> > > Thanks for clarifying!
> > > I took a look at the documentation.html file in there, and it said 3.4.
> > Is
> > > that expected?
> > >
> > > There are some files that request fullDot version and that seemed
> closer
> > to
> > > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > > The upgrade.html file also looked ok.
> > >
> > > Thanks for running the release and answering my questions!
> > > Justine
> > >
> > > On Wed, Apr 3, 2024 at 10:21 AM Manikumar 
> > > wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > Yes, it is intended. For bug fix releases website docs will be
> updated
> > > > during the final release process.
> > > > We can verify the site-docs artifacts here:
> > > >
> > > >
> > >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > > > These site-docs artifacts will be used to update website docs.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > > > 
> > > > wrote:
> > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > I've verified the keys, scanned the artifacts, and other docs.
> > > > > I built from source and ran with a ZK cluster (since I saw that we
> > > > updated
> > > > > ZK version in this release)
> > > > > I ran a few tests on this cluster.
> > > > >
> > > > > I also ran the 2.12 binary.
> > > > >
> > > > > I noticed the docs link (
> > > https://kafka.apache.org/36/documentation.html)
> > > > > mentions 3.6.1 as the latest. Is that intended?
> > > > > I will give my final vote when we figure this out.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M. 
> wrote:
> > > > >
> > > > > > Hi Manikumar, I did the following checks:
> > > > > >
> > > > > > - downloaded and built from src
> > > > > > - ran all unit test and integration test for clients
> > > > > > - ran quickstart with Kraft mode
> > > > > > - ran simple workloads with the console consumer/producer
> > > > > > - checked all links
> > > > > >
> > > > > > All looks good to me with this.
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar 
> > > wrote:
> > > > > >
> > > > > > > Gentle reminder. Please download, test and vote for the
> release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar <
> maniku...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > System test runs are green. There were 13 test failures in
> the
> > > > first
> > > > > > run.
> > > > > > > > All the failed tests passed in the second run.
> > > > > > > >
> > > > > > > > System test results:
> > > > > > > >
> > > https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > On Thu, Mar 28, 2024 at 3:21 PM Manikumar <
> > maniku...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hello Kafka users, developers and client-developers,
> > > > > > > >>
> > > > > > > >> This is the second candidate we are considering for the
> > release
> > > of
> > > > > > > Apache
> > > > > > > >> Kafka 3.6.2.
> > > > > > > >>
> > > > > > > >> This is a bugfix release with several fixes, including
> > > dependency
> > > > > > > >> version bumps for CVEs.
> > > > > > > >>
> > > > > > > >> Release notes for the 3.6.2 release:
> > > > > > > >>
> > > > >
> > https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
> > > > >

[jira] [Created] (KAFKA-16471) SslTransportLayer may leak SSLEngine resources

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16471:
-

 Summary: SslTransportLayer may leak SSLEngine resources
 Key: KAFKA-16471
 URL: https://issues.apache.org/jira/browse/KAFKA-16471
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 3.6.1, 3.7.0
Reporter: Gaurav Narula


{{SslTransportLayer}} does not invoke {{SSLEngine::closeInbound}} in 
{{close()}} after flushing the {{close_notify}} TLS alert.

While this isn't a problem for the default JDK SSLEngine, it results in 
resource leak in Netty/OpenSSL based SSLEngine which frees native resources 
only when {{closeInbound}} is invoked.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

[Omnia Ibrahim]

Thanks Chris for the feedback
> 1. It'd be nice to mention that increasing the max offset lag to INT_MAX
> could work as a partial workaround for users on existing versions (though
> of course this wouldn't prevent creation of the syncs topic).
I updated the KIP

> 2. Will it be illegal to disable offset syncs if other features that rely
> on offset syncs are explicitly enabled in the connector config? If they're
> not explicitly enabled then it should probably be fine to silently disable
> them, but I'd be interested in your thoughts.
The rest of the features that relays on this is controlled by 
emit.checkpoints.enabled (enabled by default) and sync.group.offsets.enabled 
(disabled by default) which are part of MirrorCheckpointConnector config not 
MirrorSourceConnector, I was thinking that MirrorCheckpointConnector should 
fail to start if emit.offset-syncs.enabled is disabled while 
emit.checkpoints.enabled and/or sync.group.offsets.enabled are enabled as no 
point of creating this connector if the main part is disabled. WDYT?

Thanks 
Omnia
 
> On 3 Apr 2024, at 12:45, Chris Egerton  wrote:
> 
> Hi Omnia,
> 
> Thanks for the KIP! Two small things come to mind:
> 
> 1. It'd be nice to mention that increasing the max offset lag to INT_MAX
> could work as a partial workaround for users on existing versions (though
> of course this wouldn't prevent creation of the syncs topic).
> 
> 2. Will it be illegal to disable offset syncs if other features that rely
> on offset syncs are explicitly enabled in the connector config? If they're
> not explicitly enabled then it should probably be fine to silently disable
> them, but I'd be interested in your thoughts.
> 
> Cheers,
> 
> Chris
> 
> On Wed, Apr 3, 2024, 20:41 Luke Chen  wrote:
> 
>> Hi Omnia,
>> 
>> Thanks for the KIP!
>> It LGTM!
>> But I'm not an expert of MM2, it would be good to see if there is any other
>> comment from MM2 experts.
>> 
>> Thanks.
>> Luke
>> 
>> On Thu, Mar 14, 2024 at 6:08 PM Omnia Ibrahim 
>> wrote:
>> 
>>> Hi everyone, I would like to start a discussion thread for KIP-1031:
>>> Control offset translation in MirrorSourceConnector
>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector
>>> 
>>> Thanks
>>> Omnia
>>> 
>> 

Re: [VOTE] 3.6.2 RC2

[Divij Vaidya]

Hi Manikumar

I verified the following:
- all non-minor commits in the branch are captured in release notes
- signature on the artifact match the public signature of Manikumar
- basic topic creation & produce / consumer works with JDK8 + ARM + Kafka
3.6.2 + Scala 2.12 + ZK + compression (zstd)

Things look good to me. We don't need another RC for fixing docs.

+1 (binding) from me.

--
Divij Vaidya



On Thu, Apr 4, 2024 at 10:04 AM Manikumar  wrote:

> Hi Justine,
>
> Thanks for catching this. looks like we have missed updating
> `docs/documentation.html` in kafka repo during 3.5 and 3.6 release.
>
> I will make sure to use the correct version when updating docs for 3.6.2
> release.
> I will also update 3.5 and 3.6 branches with the correct heading and also
> update the release wiki.
>
> >what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> we will remove the "-SNAPSHOT" suffix while updating the website docs. we
> may need to automate this in the release script.
>
>
> [1] https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
> [2] https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36
>
>
> Thanks,
>
> On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan  >
> wrote:
>
> > Thanks for clarifying!
> > I took a look at the documentation.html file in there, and it said 3.4.
> Is
> > that expected?
> >
> > There are some files that request fullDot version and that seemed closer
> to
> > what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> > The upgrade.html file also looked ok.
> >
> > Thanks for running the release and answering my questions!
> > Justine
> >
> > On Wed, Apr 3, 2024 at 10:21 AM Manikumar 
> > wrote:
> >
> > > Hi Justine,
> > >
> > > Yes, it is intended. For bug fix releases website docs will be updated
> > > during the final release process.
> > > We can verify the site-docs artifacts here:
> > >
> > >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > > These site-docs artifacts will be used to update website docs.
> > >
> > >
> > > Thanks,
> > >
> > > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > I've verified the keys, scanned the artifacts, and other docs.
> > > > I built from source and ran with a ZK cluster (since I saw that we
> > > updated
> > > > ZK version in this release)
> > > > I ran a few tests on this cluster.
> > > >
> > > > I also ran the 2.12 binary.
> > > >
> > > > I noticed the docs link (
> > https://kafka.apache.org/36/documentation.html)
> > > > mentions 3.6.1 as the latest. Is that intended?
> > > > I will give my final vote when we figure this out.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M.  wrote:
> > > >
> > > > > Hi Manikumar, I did the following checks:
> > > > >
> > > > > - downloaded and built from src
> > > > > - ran all unit test and integration test for clients
> > > > > - ran quickstart with Kraft mode
> > > > > - ran simple workloads with the console consumer/producer
> > > > > - checked all links
> > > > >
> > > > > All looks good to me with this.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks!
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar 
> > wrote:
> > > > >
> > > > > > Gentle reminder. Please download, test and vote for the release.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar 
> > > > wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > System test runs are green. There were 13 test failures in the
> > > first
> > > > > run.
> > > > > > > All the failed tests passed in the second run.
> > > > > > >
> > > > > > > System test results:
> > > > > > >
> > https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > On Thu, Mar 28, 2024 at 3:21 PM Manikumar <
> maniku...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hello Kafka users, developers and client-developers,
> > > > > > >>
> > > > > > >> This is the second candidate we are considering for the
> release
> > of
> > > > > > Apache
> > > > > > >> Kafka 3.6.2.
> > > > > > >>
> > > > > > >> This is a bugfix release with several fixes, including
> > dependency
> > > > > > >> version bumps for CVEs.
> > > > > > >>
> > > > > > >> Release notes for the 3.6.2 release:
> > > > > > >>
> > > >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
> > > > > > >>
> > > > > > >> *** Please download, test and vote by by Wednesday, April 3rd
> > > > > > >>
> > > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > >> https://kafka.apache.org/KEYS
> > > > > > >>
> > > > > > >> * Release artifacts to be voted upon (source and binary):
> > > > > > >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/
> > > > > > >>
> > > > > > >>
> > > > > > >> * Maven artifacts to be voted upon:
>

[jira] [Resolved] (KAFKA-16359) kafka-clients-3.7.0.jar published to Maven Central is defective

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16359.
---
Fix Version/s: 3.8.0
   3.7.1
   Resolution: Fixed

> kafka-clients-3.7.0.jar published to Maven Central is defective
> ---
>
> Key: KAFKA-16359
> URL: https://issues.apache.org/jira/browse/KAFKA-16359
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.7.0
>Reporter: Jeremy Norris
>Assignee: Apoorv Mittal
>Priority: Critical
> Fix For: 3.8.0, 3.7.1
>
>
> The {{kafka-clients-3.7.0.jar}} that has been published to Maven Central is 
> defective: it's {{META-INF/MANIFEST.MF}} bogusly include a {{Class-Path}} 
> element:
> {code}
> Manifest-Version: 1.0
> Class-Path: zstd-jni-1.5.5-6.jar lz4-java-1.8.0.jar snappy-java-1.1.10
> .5.jar slf4j-api-1.7.36.jar
> {code}
> This bogus {{Class-Path}} element leads to compiler warnings for projects 
> that utilize it as a dependency:
> {code}
> [WARNING] [path] bad path element 
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/zstd-jni-1.5.5-6.jar":
>  no such file or directory
> [WARNING] [path] bad path element 
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/lz4-java-1.8.0.jar": 
> no such file or directory
> [WARNING] [path] bad path element 
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/snappy-java-1.1.10.5.jar":
>  no such file or directory
> [WARNING] [path] bad path element 
> ".../.m2/repository/org/apache/kafka/kafka-clients/3.7.0/slf4j-api-1.7.36.jar":
>  no such file or directory
> {code}
> Either the {{kafka-clients-3.7.0.jar}} needs to be respun and published 
> without the bogus {{Class-Path}} element in it's {{META-INF/MANIFEST.MF}} or 
> a new release should be published that corrects this defect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16470) kafka-dump-log --offsets-decoder should support new records

[David Jacot (Jira)]

David Jacot created KAFKA-16470:
---

 Summary: kafka-dump-log --offsets-decoder should support new 
records
 Key: KAFKA-16470
 URL: https://issues.apache.org/jira/browse/KAFKA-16470
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

[Luke Chen]

Hi Kamal,

Thanks for sharing! I didn't know about the change before v2.8.
If that's the case, then we have to reconsider the solution of this PR.
Is making it optional a good solution? Or should we recover the snapshot if
not found before uploading it?
But what if the topic is created before v2.8.0 and old log segments are
deleted, how could we recover all the producer snapshot for old logs?

Thanks.
Luke


On Wed, Apr 3, 2024 at 11:59 PM Arpit Goyal 
wrote:

> Thanks @Kamal Chandraprakash   Greg Harris
> I currently do not have detailed understanding on the behaviour when empty
> producer snapshot  restored. I will try to test out the behaviour.Meanwhile
> I would request other community members if they can chime in and assist if
> they are already aware of the behaviour mentioned above.
> Thanks and Regards
> Arpit Goyal
> 8861094754
>
>
> On Tue, Mar 26, 2024 at 4:04 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi,
> >
> > Sorry for the late reply. Greg has raised some good points:
> >
> > > Does an empty producer snapshot have the same behavior as a
> > non-existent snapshot when restored?
> >
> > Producer snapshots maintain the status about the ongoing txn to either
> > COMMIT/ABORT the transaction. In the older version (<2.8), we maintain
> > the producer snapshots only for the recent segments. If such a topic gets
> > onboarded to tiered storage and the recently built replica becomes then
> > leader,
> > then it might break the producer.
> >
> > Assume there are 100 segments for a partition, and the producer snapshots
> > are available only for the recent 2 segments. Then, tiered storage is
> > enabled
> > for that topic, 90/100 segments are uploaded and local-log-segments are
> > cleared upto segment 50. If a new follower builds the state from remote,
> it
> > will
> > have the empty producer snapshots and start reading the data from the
> > leader
> > from segment-51. If a transaction is started at segment-40, then it will
> > break the
> > client.
> >
> > We also have to check the impact of expiring producer-ids before the
> > default
> > expiration time of 7 days: *transactional.id.expiration.ms
> > *
> >
> > > 2. Why were empty producer snapshots not backfilled for older data
> > when clusters were upgraded from 2.8?
> >
> > https://github.com/apache/kafka/pull/7929 -- It was not required at that
> > time.
> > With tiered storage, we need the snapshot file for each segment to
> reliably
> > build the follower state from remote storage.
> >
> > > 3. Do producer snapshots need to be available contiguously, or can
> > earlier snapshots be empty while later segments do not exist?
> >
> > I assume you refer to "while later segments do exist". Each snapshot file
> > will contain
> > the cumulative/complete data of all the previous segments. So, a recent
> > segment
> > snapshot is enough to build the producer state. We need to figure out a
> > solution to
> > build the complete producer state for replicas that built the state using
> > the remote.
> >
> > Arpit,
> > We have to deep dive into each of them to come up with the proper
> solution.
> >
> > --
> > Kamal
> >
> >
> > On Tue, Mar 26, 2024 at 3:55 AM Greg Harris  >
> > wrote:
> >
> > > Hi Arpit,
> > >
> > > I think creating empty producer snapshots would be
> > > backwards-compatible for the tiered storage plugins, but I'm not aware
> > > of what the other compatibility/design concerns might be. Maybe you or
> > > another reviewer can answer these questions:
> > > 1. Does an empty producer snapshot have the same behavior as a
> > > non-existent snapshot when restored?
> > > 2. Why were empty producer snapshots not backfilled for older data
> > > when clusters were upgraded from 2.8?
> > > 3. Do producer snapshots need to be available contiguously, or can
> > > earlier snapshots be empty while later segments do not exist?
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Sat, Mar 23, 2024 at 3:24 AM Arpit Goyal 
> > > wrote:
> > > >
> > > > Yes Luke,
> > > > I am also in favour of creating producer snapshot at run time if
> > > > foundEmpty  as this would only be required for topics migrated from <
> > 2.8
> > > > version. This will not break the existing contract with the plugin.
> > Yes,
> > > > metrics do not make sense here as of now.
> > > > Greg, @Kamal Chandraprakash   WDYT ?
> > > > Arpit Goyal
> > > > 8861094754
> > > >
> > > >
> > > > On Sat, Mar 23, 2024 at 3:05 PM Luke Chen  wrote:
> > > >
> > > > > Hi Arpit,
> > > > >
> > > > > I'm in favor of creating an empty producer snapshot since it's only
> > for
> > > > > topics <= v2.8.
> > > > > About the metric, I don't know what we expect users to know.
> > > > > I think we can implement with the empty producer snapshot method,
> > > without
> > > > > the metric.
> > > > > And add them if users are requested it.
> > > > > WDYT?
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Sat, Mar 23, 2024 at 1:24 PM A

Re: Kafka capabilities

[Kafka Life]

Dear Kafka experts , Could anyone having this data share the details please

On Wed, Apr 3, 2024 at 3:42 PM Kafka Life  wrote:

> Hi Kafka users
>
> Does any one have a document or ppt that showcases the capabilities of
> Kafka along with any cost management capability?
> i have a customer who is still using IBM MQM and rabbit MQ. I want the
> client to consider kafka for messaging and data streaming. I wanted to seek
> your expert help if you have any document or ppt i can propose it as an
> example. could you pls help.
>
> thanks and regards
> KrisG
>

Re: [VOTE] 3.6.2 RC2

[Manikumar]

Hi Justine,

Thanks for catching this. looks like we have missed updating
`docs/documentation.html` in kafka repo during 3.5 and 3.6 release.

I will make sure to use the correct version when updating docs for 3.6.2
release.
I will also update 3.5 and 3.6 branches with the correct heading and also
update the release wiki.

>what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
we will remove the "-SNAPSHOT" suffix while updating the website docs. we
may need to automate this in the release script.


[1] https://github.com/apache/kafka/blob/3.6/docs/documentation.html#L36
[2] https://github.com/apache/kafka/blob/3.5/docs/documentation.html#L36


Thanks,

On Thu, Apr 4, 2024 at 3:50 AM Justine Olshan 
wrote:

> Thanks for clarifying!
> I took a look at the documentation.html file in there, and it said 3.4. Is
> that expected?
>
> There are some files that request fullDot version and that seemed closer to
> what was expected: "fullDotVersion": "3.6.2-SNAPSHOT"
> The upgrade.html file also looked ok.
>
> Thanks for running the release and answering my questions!
> Justine
>
> On Wed, Apr 3, 2024 at 10:21 AM Manikumar 
> wrote:
>
> > Hi Justine,
> >
> > Yes, it is intended. For bug fix releases website docs will be updated
> > during the final release process.
> > We can verify the site-docs artifacts here:
> >
> >
> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/kafka_2.12-3.6.2-site-docs.tgz
> > These site-docs artifacts will be used to update website docs.
> >
> >
> > Thanks,
> >
> > On Wed, Apr 3, 2024 at 10:30 PM Justine Olshan
> > 
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > I've verified the keys, scanned the artifacts, and other docs.
> > > I built from source and ran with a ZK cluster (since I saw that we
> > updated
> > > ZK version in this release)
> > > I ran a few tests on this cluster.
> > >
> > > I also ran the 2.12 binary.
> > >
> > > I noticed the docs link (
> https://kafka.apache.org/36/documentation.html)
> > > mentions 3.6.1 as the latest. Is that intended?
> > > I will give my final vote when we figure this out.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Wed, Apr 3, 2024 at 7:25 AM Lianet M.  wrote:
> > >
> > > > Hi Manikumar, I did the following checks:
> > > >
> > > > - downloaded and built from src
> > > > - ran all unit test and integration test for clients
> > > > - ran quickstart with Kraft mode
> > > > - ran simple workloads with the console consumer/producer
> > > > - checked all links
> > > >
> > > > All looks good to me with this.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks!
> > > >
> > > >
> > > >
> > > > On Wed, Apr 3, 2024, 2:19 a.m. Manikumar 
> wrote:
> > > >
> > > > > Gentle reminder. Please download, test and vote for the release.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > On Fri, Mar 29, 2024 at 4:57 PM Manikumar 
> > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > System test runs are green. There were 13 test failures in the
> > first
> > > > run.
> > > > > > All the failed tests passed in the second run.
> > > > > >
> > > > > > System test results:
> > > > > >
> https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > On Thu, Mar 28, 2024 at 3:21 PM Manikumar 
> > > > wrote:
> > > > > >
> > > > > >> Hello Kafka users, developers and client-developers,
> > > > > >>
> > > > > >> This is the second candidate we are considering for the release
> of
> > > > > Apache
> > > > > >> Kafka 3.6.2.
> > > > > >>
> > > > > >> This is a bugfix release with several fixes, including
> dependency
> > > > > >> version bumps for CVEs.
> > > > > >>
> > > > > >> Release notes for the 3.6.2 release:
> > > > > >>
> > > https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
> > > > > >>
> > > > > >> *** Please download, test and vote by by Wednesday, April 3rd
> > > > > >>
> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > > > >> https://kafka.apache.org/KEYS
> > > > > >>
> > > > > >> * Release artifacts to be voted upon (source and binary):
> > > > > >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/
> > > > > >>
> > > > > >>
> > > > > >> * Maven artifacts to be voted upon:
> > > > > >>
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >>
> > > > > >> * Javadoc:
> > > > > >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/
> > > > > >>
> > > > > >> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> > > > > >> https://github.com/apache/kafka/releases/tag/3.6.2-rc2
> > > > > >>
> > > > > >> * Documentation:
> > > > > >> https://kafka.apache.org/36/documentation.html
> > > > > >>
> > > > > >> * Protocol:
> > > > > >> https://kafka.apache.org/36/protocol.html
> > > > > >>
> > > > > >> * Successful Jenkins builds for the 3.6 branch:
> > > > > >> Unit/integration tests:
> > > > > >> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/167/
> > (test
> > > > > >> failures p