Re: Unexpected loss of Offsets

[Luke Chen]

Hi James and Divij,

The community will consider to release a 2nd patch to an old version (i.e.
v2.8.2) unless there's a very critical issue.

I think you could ask Amazon MSK to support Kafka v3.0 and newer version to
have the fix. (and also many new features)
Or, maybe Amazon MSK can build from kafka v2.8 branch by themselves?

Note: Kafka v3.0 doesn't deprecate/remove Zookeeper.

Thank you.
Luke



On Mon, May 2, 2022 at 8:10 PM Divij Vaidya  wrote:

> Luke / James
>
> I agree that this bug is critical enough to release a new patch. Plus,
> there are 10 more bug fixes
> 
>  with
> major/blocker priority waiting to be released in 2.8.2.
>
> I will be happy to assist / perform the release process for 2.8.2 or
> assist in any other way I can. Luke, please let me know how we want to
> proceed ahead on this.
>
> Regards,
> Divij Vaidya
>
>
>
> On Fri, Apr 29, 2022 at 5:09 AM James Olsen  wrote:
>
>> Luke,
>>
>> I would argue that https://issues.apache.org/jira/browse/KAFKA-13636 is
>> a critical defect as it can have a very serious impact.
>>
>> We run on AWS MSK which supports these versions:
>> https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.html.
>> We are currently on 2.7.2.
>>
>> I note that MSK does not support any 3.x (maybe they're not ready for the
>> Zookeeper removal).  So I suspect we're going to need a 2.x if MSK is going
>> to adopt it any time soon.  I'd be happier with a 2.7.3 incorporating
>> KAFKA-13636 in order to minimise the risk of introducing other issues, or
>> the 2.8.2 if that's not possible.
>>
>> What can we do to make this happen ASAP?
>>
>> Regards, James.
>>
>> On 29/04/2022, at 14:50, Luke Chen > show...@gmail.com>> wrote:
>>
>> Hi James,
>>
>> So far, v2.8.2 is not planned, yet. And usually, the patch release only
>> has one, that is, v2.8.0, and v2.8.1.
>> But there are of course some exceptions that some releases have 2 or 3
>> patch releases.
>>
>> For KAFKA-13658, you can check KAFKA-13658<
>> https://issues.apache.org/jira/browse/KAFKA-13658>, which is included in
>> v3.0.1, v3.1.1, and v3.2.0.
>> So far, the v3.0.1 is released, and v3.1.1 and v3.2.0 will be coming soon.
>>
>> Thank you.
>> Luke
>>
>> On Fri, Apr 29, 2022 at 8:53 AM James Olsen > ja...@inaseq.com>> wrote:
>> Luke,
>>
>> Do you know if 2.8.2 will be released anytime soon?  It appears to be
>> waiting on https://issues.apache.org/jira/browse/KAFKA-13805 for which
>> fixes are available.
>>
>> Regards, James.
>>
>> On 11/04/2022, at 14:22, Luke Chen > show...@gmail.com>> wrote:
>>
>> Hi James,
>>
>> This looks like this known issue KAFKA-13636
>> , which should be
>> fixed
>> in the newer version.
>>
>> Thank you.
>> Luke
>>
>> On Mon, Apr 11, 2022 at 9:18 AM James Olsen > ja...@inaseq.com>> wrote:
>>
>> I recently observed the following series of events for a particular
>> partition (MyTopic-6):
>>
>> 2022-03-18 03:18:28,562 INFO
>> [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator]
>> 'executor-thread-2' [Consumer clientId=consumer-MyTopicService-group-3,
>> groupId=MyTopicService-group] Setting offset for partition MyTopic-6 to
>> the
>> committed offset FetchPosition{offset=438, offsetEpoch=Optional.empty,
>> currentLeader=LeaderAndEpoch{leader=Optional[b-2.redacted.kafka.us<
>> http://b-2.redacted.kafka.us/><
>> http://b-2.redacted.kafka.us>-
>> east-1.amazonaws.com:9094 (id: 2 rack:
>> use1-az4)], epoch=64}}
>>
>> -- RESTART (bring up new consumer node)
>>
>> 2022-04-01 15:17:47,943 INFO
>> [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator]
>> 'executor-thread-6' [Consumer clientId=consumer-MyTopicService-group-7,
>> groupId=MyTopicService-group] Setting offset for partition MyTopic-6 to
>> the
>> committed offset FetchPosition{offset=449, offsetEpoch=Optional.empty,
>> currentLeader=LeaderAndEpoch{leader=Optional[b-2.redacted.kafka.us<
>> http://b-2.redacted.kafka.us/><
>> http://b-2.redacted.kafka.us>-
>> east-1.amazonaws.com:9094 (id: 2 rack:
>> use1-az4)], epoch=64}}
>>
>> -- REBALANCE (drop old consumer node)
>>
>> 2022-04-01 15:18:24,414 INFO
>> [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator]
>> 'executor-thread-2' [Consumer clientId=consumer-MyTopicService-group-3,
>> groupId=MyTopicService-group] Found no committed offset for partition
>> MyTopic-6
>> 2022-04-01 15:18:24,474 INFO
>> [org.apache.kafka.clients.consumer.internals.SubscriptionState]
>> 'executor-thread-2' [Consumer clientId=consumer-MyTopicService-group-3,
>> groupId=MyTopicService-group] Resetting offset for partition MyTopic-6 to
>> position FetchPosition{offset=411, offsetEpoch=Optional.empty,
>> currentLeader=LeaderAndEpoch{leader=Optional[b-2.redacted.kafka.us<

Re: [VOTE] 3.2.0 RC1

[Bill Bejeck]

Hi Bruno,

Thanks for running the 3.2 release!

I did the following:

   1. Built from source
   2. Ran the unit tests
   3. Validated keys and checksums
   4. Lightly checked the Javadoc
   5. Ran the quickstart and the Kafka Streams demo app
   6. Looked over the docs
   7. Ran the KRaft quickstart including the dump-log and metadata-shell

In the docs, I found that in section 1.3 on the main page the  `Kafka
Streams Demo` and `app development tutorial` still point to version 2.5.
But since this is a document-only change only we can directly update the
kafka-site repo.

+1(binding)

Thanks,
Bill


On Tue, May 3, 2022 at 10:07 AM Bruno Cadonna  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for release of Apache Kafka 3.2.0.
>
> * log4j 1.x is replaced with reload4j (KAFKA-9366)
> * StandardAuthorizer for KRaft (KIP-801)
> * Send a hint to the partition leader to recover the partition (KIP-704)
> * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> * kafka-console-producer writes headers and null values (KIP-798 and
> KIP-810)
> * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> * Static membership protocol lets the leader skip assignment (KIP-814)
> * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> * Connect APIs list all connector plugins and retrieve their
> configuration (KIP-769)
> * TimestampConverter SMT supports different unix time precisions (KIP-808)
> * Connect source tasks handle producer exceptions (KIP-779)
>
>
> Release notes for the 3.2.0 release:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, May 10th, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
>
> * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> https://github.com/apache/kafka/releases/tag/3.2.0-rc1
>
> * Documentation:
> https://kafka.apache.org/32/documentation.html
>
> * Protocol:
> https://kafka.apache.org/32/protocol.html
>
> * Successful Jenkins builds for the 3.2 branch:
> Unit/integration tests: I'll share a link once the builds complete
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
>
> /**
>
> Thanks,
> Bruno
>

Re: [VOTE] 3.1.1 RC1

[Jakub Scholz]

+1 (non-binding). I used the Scala 2.13 binaries and staged Maven artifacts
and ran various tests with them. Thanks for doing the release.

Jakub

On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison  wrote:

> Hi Tom,
>
> Thanks for running this release!
>
> I've done the following:
> - Checked signatures and checksums
> - Checked javadocs/maven artifacts
> - Built from source and run all tests with Java 11
> - Ran quickstart on Scala 2.13 artifact with Java 11
>
> It looks like the website has not been updated yet, I still only see
> 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> the notable changes section.
>
> +1 (binding)
>
> Thanks,
> Mickael
>
>
>
> On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley  wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the first candidate for release of Apache Kafka 3.1.1.
> >
> > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > since 3.1.0.
> >
> > Release notes for the 3.1.1 release:
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/31/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/31/protocol.html
> >
> > * Successful Jenkins builds for the 3.1 branch:
> > I will share a link one the build is complete.
> >
> > /**
> >
> > Thanks,
> > Tom
>

[VOTE] 3.2.0 RC1

[Bruno Cadonna]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition (KIP-704)
* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and 
KIP-810)

* JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
* Static membership protocol lets the leader skip assignment (KIP-814)
* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their 
configuration (KIP-769)

* TimestampConverter SMT supports different unix time precisions (KIP-808)
* Connect source tasks handle producer exceptions (KIP-779)


Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, May 10th, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc1

* Documentation:
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
Unit/integration tests: I'll share a link once the builds complete
System tests: https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/

/**

Thanks,
Bruno

Re: Kafka Question

[Robin Moffatt]

Kafka itself includes Kafka Streams (
https://kafka.apache.org/31/documentation/streams/), so you can do this
processing in Kafka. There's a Filter transformation that would be a good
place to start:
https://kafka.apache.org/31/documentation/streams/developer-guide/dsl-api.html#stateless-transformations


-- 

Robin Moffatt | Principal Developer Advocate | ro...@confluent.io | @rmoff


On Tue, 3 May 2022 at 03:09, Liam Clarke-Hutchinson 
wrote:

> Hi Emily,
>
> Nope, Kafka doesn't have that capability built in, it's just a distributed
> log that's great for streaming events. However, you can easily write a
> program that consumes those events from Kafka and then does what you want
> :)
>
> Cheers,
>
> Liam
>
> On Tue, 3 May 2022 at 06:30, Emily Schepisi
>  wrote:
>
> > Hello,
> >
> > I have a question about Kafka. If I put an upper and lower control limit
> on
> > the data, and the log records an event where the upper or lower control
> > limit is breached, will Kafka be able to send a notification via email or
> > text message to the user?
> >
> > Example: I'm tracking the daily temperature and set the upper control
> limit
> > at 80 degrees and the lower control limit at 50 degrees. The event log on
> > Kafka recorded the temperature on Monday at 90 degrees, so it's higher
> than
> > the upper control limit. Does Kafka have the capability to send a text
> > message or email to let me know that the temperature is outside of the
> > control limit?
> >
> > Thank you,
> >
> > Emily
> >
>

REMINDER - Travel Assistance available for ApacheCon NA New Orleans 2022

[Gavin McDonald]

Hi All Contributors and Committers,

This is a first reminder email that travel
assistance applications for ApacheCon NA 2022 are now open!

We will be supporting ApacheCon North America in New Orleans, Louisiana,
on October 3rd through 6th, 2022.

TAC exists to help those that would like to attend ApacheCon events, but
are unable to do so for financial reasons. This year, We are supporting
both committers and non-committers involved with projects at the
Apache Software Foundation, or open source projects in general.

For more info on this year's applications and qualifying criteria, please
visit the TAC website at http://www.apache.org/travel/
Applications are open and will close on the 1st of July 2022.

Important: Applicants have until the closing date above to submit their
applications (which should contain as much supporting material as required
to efficiently and accurately process their request), this will enable TAC
to announce successful awards shortly afterwards.

As usual, TAC expects to deal with a range of applications from a diverse
range of backgrounds. We therefore encourage (as always) anyone thinking
about sending in an application to do so ASAP.

Why should you attend as a TAC recipient? We encourage you to read stories
from
past recipients at https://apache.org/travel/stories/ . Also note that
previous TAC recipients have gone on to become Committers, PMC Members, ASF
Members, Directors of the ASF Board and Infrastructure Staff members.
Others have gone from Committer to full time Open Source Developers!

How far can you go! - Let TAC help get you there.