Re: [DISCUSS] Broker behaviour when returning records

[Luke Chen]

Hi Megan,

Answering your questions below:

1. Why can't the broker remove messages from the first batch? So that
fetch.max.bytes is not violated.

The documentation has answered your question:

If the first record batch in the first non-empty partition of the fetch is
larger than this limit, the batch will still be returned to ensure that the
consumer can make progress.

ref;
https://kafka.apache.org/documentation/#consumerconfigs_max.partition.fetch.bytes


2. How does the broker choose which partitions (from those that the
consumer is assigned to) contribute messages to a response batch?

As far as I know, suppose consumer A subscribed to partition 1,2,3,  the
broker just read as many data as the 1st partition picked (let's say,
partition 1), and not exceed the max fetch limit. If it did not exceed,
continue to the partition 2, and so on.

3. Why does the broker send records in batches to the consumer?

For performance

Thank you.
Luke


On Sat, Jan 22, 2022 at 6:22 AM Megan Niu (BLOOMBERG/ TORONTO) <
mni...@bloomberg.net> wrote:

> Hi all,
> I have some questions about how brokers batch records to send to consumers.
>
> One of the configuration properties for a consumer is fetch.max.bytes.
> Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about
> fetch.max.bytes :
>
> "This property lets you specify the maximum bytes that Kafka will return
> whenever the consumer polls a broker (50 MB by default). [...] Note that
> records are sent to the client in batches, and if the first record-batch
> that the broker has to send exceeds this size, the batch will be sent and
> the limit will be ignored."
>
> My questions:
> 1. Why can't the broker remove messages from the first batch? So that
> fetch.max.bytes is not violated.
> 2. How does the broker choose which partitions (from those that the
> consumer is assigned to) contribute messages to a response batch?
> 3. Why does the broker send records in batches to the consumer?

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Tom,

1. > I think you mean tools-log4j.properties there, right?

Thanks for reading carefully. Yes, you are right. It was an error.

2. > Can the log4j-appender log a warning about its own deprecation?

That's a good suggestion. I hadn't thought of this part - adding a WARNing
message to the Log4jAppender would help, like:

> log4j-appender is deprecated and will be removed in a future release. For
migration, please refer to the latest documentation.

I just updated the proposal and the PR accordingly.

Best,
Dongjin

On Fri, Jan 21, 2022 at 9:32 PM Tom Bentley  wrote:

> Hi Dongjin,
>
> Many thanks for the KIP. I have just a couple of minor points.
>
> 1. In the Proposed Changes section there is the sentence "For backward
> compatibility, these tools will use the log4j configuration
> file (tools-log4j2.properties) by default.", I think you mean
> tools-log4j.properties there, right?
>
> 2. Can the log4j-appender log a warning about its own deprecation? The
> "DEPRECATED: using log4j 1.x configuration. To use log4j 2.x
> configuration," message is a bit confusing since it suggests that it's just
> the use of 1.x configuration that's deprecated (which it is), but it's also
> the use of the appender itself which is deprecated.
>
> Kind regards,
>
> Tom
>
> On Thu, 20 Jan 2022 at 05:01, Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I just filed the issue based on the discussion here:
> >
> > KAFKA-13604: Add pluggable logging framework support (
> > https://issues.apache.org/jira/browse/KAFKA-13604)
> >
> > Thanks for posing a great issue, Viktor and Haruki!
> >
> > Best,
> > Dongjin
> >
> > On Thu, Jan 20, 2022 at 2:54 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Hi folks,
> > >
> > > >a) I think this makes sense. But I think this shouldn't block this
> > > KIP, it can be done after moving to log4j2.
> > >
> > > Absolutely.
> > >
> > > >However, this limitation may change in the future - slf4j will support
> > > dynamic logging levels in 2.0:
> > >
> > > This would be awesome, let's keep an eye on this.
> > >
> > > >Kafka avoided all the log4shell vulnerabilities exactly because we
> > DIDN'T
> > > move to log4j 2.x.
> > >
> > > My first reaction was how lucky we are (and all Kafka users) that we
> > didn't
> > > have to do anything on that weekend! Blessed are the lazy. :)
> > >
> > > Viktor
> > >
> > > On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe 
> wrote:
> > >
> > > > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > > > > Hi Dongjin,
> > > > >
> > > > > We're also looking into this matter as our stack was also affected
> by
> > > all
> > > > > the log4j hell and users increasingly pushing us to upgrade to
> log4j2
> > > or
> > > > > logback because of the existing vulnerabilities in log4j1.
> > > > > Regarding the points raised by Haruki:
> > > > >
> > > >
> > > > I've heard the same thing from other people -- that there is now more
> > > > interest in moving from log4j1.x to log4j2.x, after all the recent
> > > > vulnerabilities in the latter. I found this a bit strange. Kafka
> > avoided
> > > > all the log4shell vulnerabilities exactly because we DIDN'T move to
> > log4j
> > > > 2.x. (Yes, I am aware that there is a longstanding vulnerability in
> > that
> > > > one log sink in log4j 1.x, but you can just not use that one.)
> > > >
> > > > I haven't thought about this very hard. Maybe it's still a good idea
> to
> > > > move to log4j2. But it's odd that nobody is commenting about how in
> > this
> > > > case, not updating actually prevented a major security incident for
> > > Kafka.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > a) In my opinion the best would be to make the dynamic logger
> support
> > > > > (Log4jController and LoggingResource) pluggable for log4j2 and
> > logback
> > > > (so
> > > > > an interface could be used to define the dynamic logging control
> > > methods
> > > > > and a config to specify the implementation). That way we're not
> bound
> > > to
> > > > > either logback or log4j and seems like a low-effort thing to do.
> > > > > Additionally this could be used in Connect too in LoggingResource.
> > > > >
> > > > > b) I think testing dependencies aren't that important from the user
> > > > > perspective, it's fine to either use log4j2 or logback, whichever
> is
> > > > > easier. Kafka is either used from the distribution (tgz) or pulled
> in
> > > > > through maven, but test dependencies shouldn't be exposed to the
> > world.
> > > > >
> > > > > c) I would support deprecating the appender in favor of the log4j2
> > > Kafka
> > > > > appender. VerifiableLog4jAppender is intended as a testing tool
> > anyway,
> > > > so
> > > > > I think it's less important to change this to logback.
> > > > >
> > > > > Future vulnerabilities will always be found in either logback or
> > log4j2
> > > > or
> > > > > any other logging framework, so I think the safest approach is to
> > allow
> > > > > users to choose their implementation, while in tests I think 

Re: State store guarantees - KStreams 2.8.1

[Matthias J. Sax]

Well, it's unclear what the remote lookup does... As Kafka Streams does 
not implement this part, my best guess at the moment is to blame it on a 
bug in the remote request implementation.


Are you using some of-the-shelf implementation for the remove lookup 
part or are you using something build in-house?



-Matthias



On 1/21/22 13:13, Jiří Syrový wrote:

I agree it sounds a bit off, but it seems that even a host that is not
marked as active allows me to query it's store and gives me a result that
is not null.

This application has an API that either queries local or remote
store (basically via HTTP API of active host), but the weird part is I get
the local response from both instances instead of expected one remote (on
non-active and non-standby host) and one local.
In principle the code to query stores looks like this

 streams
   .store(
 StoreQueryParameters
   .fromNameAndType(
 storeName,
 QueryableStoreTypes.keyValueStore[K, V]()
   )
   )
   .get(key)


And responses look like this:
$ curl //123 *(response from instance A)*
*{"meta":"KeyQueryMetadata
{activeHost=HostInfo{host='ip-1-2-3-4.eu-west-1.compute.internal',
port=8080}, standbyHosts=[],
partition=0}","value":{"timestamp":"2022-01-21T00:00:02.433Z","enabled":true},"hostname":"ip-1-2-3-4.eu-west-1.compute.internal"}*
$ curl //123 *(response from instance B)*
*{"meta":"KeyQueryMetadata
{activeHost=HostInfo{host='ip-1-2-3-4.eu-west-1.compute.internal',
port=8080}, standbyHosts=[],
partition=0}","value":{"timestamp":"2022-01-21T15:55:27.807Z","enabled":false},"hostname":"ip-9-8-7-6.eu-west-1.compute.internal"}*

This behaviour is not random and is 100% reproducible. I can try to create
a minimal code example that will demonstrate it.

On Fri, 21 Jan 2022 at 18:20, Matthias J. Sax  wrote:


but instance A returns

result X for a partition I and instance B returns result Y for the same
partition I.


This sound a little off. As you stated, if both instances agree on the
active host, the active host must either be instance A or instance B,
and thus you can query partition I only on instance A or instance B. The
non-active instance should not return any data for a partition it does
not host.

Can you elaborate?

-Matthias

On 1/21/22 4:47 AM, Jiří Syrový wrote:

Hi everyone,

I'm trying for a while to answer myself a question about what are

actually

guarantees for state stores in regards to consistency when connected to
transformers.

I have an application where a single (persistent, rocksdb backed) state
store is connected to multiple transformers. Each transformer might both
read (get) and write (put) data into the state store. All transformers
receive data from multiple input topics in the same way (the same key,

same

number of partitions) that before sending it to transformers merged
together.

All transformers are located in the same sub-topology.

What I observed is that even with 0 standby replicas I might get
inconsistent results when querying this state store connected to multiple
transformers. I have 2 instances and metadata on both instances agree on
the active host for this state store and partition, but instance A

returns

result X for a partition I and instance B returns result Y for the same
partition I.

Any suggestions if this is a bug or is my assumption incorrect that the
same state store should give the same result for the same key (same
partition) in 2 distinct transformers fed from the same input?

Thanks,
Jiri

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #626

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13502) Support configuring BROKER_LOGGER on controller-only KRaft nodes

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-13502.

Resolution: Fixed

This issue was fixed by KAFKA-13552.

> Support configuring BROKER_LOGGER on controller-only KRaft nodes
> 
>
> Key: KAFKA-13502
> URL: https://issues.apache.org/jira/browse/KAFKA-13502
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin McCabe
>Priority: Major
>  Labels: kip-500
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13552) Unable to dynamically change broker log levels on KRaft

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-13552.

Resolution: Fixed

> Unable to dynamically change broker log levels on KRaft
> ---
>
> Key: KAFKA-13552
> URL: https://issues.apache.org/jira/browse/KAFKA-13552
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Ron Dagostino
>Assignee: Colin McCabe
>Priority: Major
>
> It is currently not possible to dynamically change the log level in KRaft.  
> For example:
> kafka-configs.sh --bootstrap-server  --alter --add-config 
> "kafka.server.ReplicaManager=DEBUG" --entity-type broker-loggers 
> --entity-name 0
> Results in:
> org.apache.kafka.common.errors.InvalidRequestException: Unexpected resource 
> type BROKER_LOGGER.
> The code to process this request is in ZkAdminManager.alterLogLevelConfigs(). 
>  This needs to be moved out of there, and the functionality has to be 
> processed locally on the broker instead of being forwarded to the KRaft 
> controller.
> It is also an open question as to how we can dynamically alter log levels for 
> a remote KRaft controller.  Connecting directly to it is one possible 
> solution, but that may not be desirable since generally connecting directly 
> to the controller is not necessary.  The ticket for this particular spect of 
> the issue is https://issues.apache.org/jira/browse/KAFKA-13502



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [kafka-clients] Re: [VOTE] 3.1.0 RC1

[Israel Ekpo]

Thanks for managing and running this release, David!

I am excited about this particular release and I appreciate every effort
and contribution that made this possible.

On Fri, Jan 21, 2022 at 8:39 AM 'David Jacot' via kafka-clients <
kafka-clie...@googlegroups.com> wrote:

> Thanks for all the votes!
>
> I'm also +1 (binding) given that:
>
> * Ran the release and generated 3.1.0 RC1.
> * Verified all checksums and signatures.
> * Built from source and ran unit tests.
> * Installed 3.1.0 RC1 and performed the quickstart for both ZK and KRaft.
> * Spotchecked the Javadocs and the site doc.
>
> I will close the vote shortly.
>
> Best,
> David
>
> On Thu, Jan 20, 2022 at 1:21 PM Mickael Maison 
> wrote:
> >
> > Thanks David for running this release!
> >
> > I checked signatures and keys, built from source and ran unit tests
> > with Java 11. I also ran the quickstart.
> >
> > +1 (binding)
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jan 19, 2022 at 9:20 PM Bill Bejeck  wrote:
> > >
> > > Hi David,
> > >
> > > Thanks for running the release!
> > >
> > > I did the following:
> > >
> > >1. Validated all signatures and keys
> > >2. Built from source
> > >3. Ran all the unit tests
> > >4. Spot checked the java docs, and the site docs
> > >5. Ran a portion of the quick start
> > >6. Ran the Kafka Streams demo application
> > >
> > > I noticed one small error on the quick start page: the  "Kafka Streams
> demo and
> > > the app development tutorial" links still reference the 2.5 release.
> > > This is a very minor issue and it doesn't require a new RC, we can
> simply
> > > update the links once released.
> > >
> > > +1(binding)
> > >
> > > Thanks,
> > > Bill
> > >
> > >
> > > On Wed, Jan 19, 2022 at 8:44 AM David Jacot
> 
> > > wrote:
> > >
> > > > Thanks for all the votes! We are still missing two binding votes from
> > > > PMC members.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Tue, Jan 18, 2022 at 5:00 PM Israel Ekpo 
> wrote:
> > > > >
> > > > > Performed the following validations using the tools available here:
> > > > >
> > > > > https://github.com/izzyacademy/apache-kafka-release-party
> > > > >
> > > > >- Verified signatures, keys and hashes for release artifacts
> > > > >- Deployed Multi-Node Cluster in Legacy Mode (with Zookeeper)
> > > > >- Deployed Multi-Node Cluster in KRaft Mode (without Zookeeper)
> > > > >- Can confirm that KAFKA-13456 works as expected after
> switching to
> > > > 3.1.0
> > > > >- Briefly Walked Through 3.1 Documentation, Javadocs and
> Protocol
> > > > Pages
> > > > >
> > > > > +1 (non-binding) for the release candidate
> > > > >
> > > > > Thanks for running this release
> > > > >
> > > > > Israel Ekpo
> > > > > Lead Instructor, IzzyAcademy.com
> > > > > https://www.youtube.com/c/izzyacademy
> > > > > https://izzyacademy.com/
> > > > >
> > > > >
> > > > > On Wed, Jan 12, 2022 at 7:59 AM David Jacot 
> wrote:
> > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the second candidate for release of Apache Kafka 3.1.0.
> > > > > >
> > > > > > * Apache Kafka supports Java 17
> > > > > > * The FetchRequest supports Topic IDs (KIP-516)
> > > > > > * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> > > > > > * Add broker count metrics (KIP-748)
> > > > > > * Differentiate consistently metric latency measured in millis
> and
> > > > > > nanos (KIP-773)
> > > > > > * The eager rebalance protocol is deprecated (KAFKA-13439)
> > > > > > * Add TaskId field to StreamsException (KIP-783)
> > > > > > * Custom partitioners in foreign-key joins (KIP-775)
> > > > > > * Fetch/findSessions queries with open endpoints for
> > > > > > SessionStore/WindowStore (KIP-766)
> > > > > > * Range queries with open endpoints (KIP-763)
> > > > > > * Add total blocked time metric to Streams (KIP-761)
> > > > > > * Add additional configuration to control MirrorMaker2 internal
> topics
> > > > > > naming convention (KIP-690)
> > > > > >
> > > > > > Release notes for the 3.1.0 release:
> > > > > >
> https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Monday, January 17, 9am PT
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.1.0-rc1
> > > > > >
> > > > > > * Documentation:
> > > > > > 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #625

[Apache Jenkins Server]

See 

[DISCUSS] Broker behaviour when returning records

[Megan Niu (BLOOMBERG/ TORONTO)]

Hi all,
I have some questions about how brokers batch records to send to consumers.

One of the configuration properties for a consumer is fetch.max.bytes.
Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about 
fetch.max.bytes :

"This property lets you specify the maximum bytes that Kafka will return 
whenever the consumer polls a broker (50 MB by default). [...] Note that 
records are sent to the client in batches, and if the first record-batch that 
the broker has to send exceeds this size, the batch will be sent and the limit 
will be ignored."

My questions:
1. Why can't the broker remove messages from the first batch? So that 
fetch.max.bytes is not violated.
2. How does the broker choose which partitions (from those that the consumer is 
assigned to) contribute messages to a response batch?
3. Why does the broker send records in batches to the consumer?

[GitHub] [kafka-site] dajac commented on a change in pull request #393: MINOR: Update site for Apache Kafka 3.1.0 release

[GitBox]

dajac commented on a change in pull request #393:
URL: https://github.com/apache/kafka-site/pull/393#discussion_r790003684



##
File path: downloads.html
##
@@ -6,12 +6,58 @@

 Download
 
-3.0.0 is the latest release. The current stable version is 3.0.0.
+3.1.0 is the latest release. The current stable version is 3.1.0.
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html;>procedures and using 
these https://downloads.apache.org/kafka/KEYS;>KEYS.
 
 
+
+3.1.0
+
+
+Released January 24, 2022
+
+
+https://downloads.apache.org/kafka/3.1.0/RELEASE_NOTES.html;>Release 
Notes
+
+
+Source download: https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka-3.1.0-src.tgz;>kafka-3.1.0-src.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka-3.1.0-src.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka-3.1.0-src.tgz.sha512;>sha512)
+
+
+Binary downloads:
+
+Scala 2.12 - https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.12-3.1.0.tgz;>kafka_2.12-3.1.0.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka_2.12-3.1.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka_2.12-3.1.0.tgz.sha512;>sha512)
+Scala 2.13 - https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.13-3.1.0.tgz;>kafka_2.13-3.1.0.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz.sha512;>sha512)
+
+We build for multiple versions of Scala. This only matters if you 
are using Scala and you want a version
+built for the same Scala version you use. Otherwise any version 
should work (2.13 is recommended).
+
+
+
+
+Kafka 3.0.0 includes a number of significant new features. Here is a 
summary of some notable changes:

Review comment:
   Good catch, fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #624

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 508953 lines...]
[2022-01-21T21:14:07.519Z] > Task :connect:api:javadocJar
[2022-01-21T21:14:07.519Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-01-21T21:14:07.519Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-01-21T21:14:07.519Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-01-21T21:14:07.519Z] > Task :connect:json:publishToMavenLocal
[2022-01-21T21:14:07.519Z] > Task :connect:api:testJar
[2022-01-21T21:14:07.519Z] > Task :connect:api:testSrcJar
[2022-01-21T21:14:07.519Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-01-21T21:14:07.519Z] > Task :connect:api:publishToMavenLocal
[2022-01-21T21:14:09.273Z] 
[2022-01-21T21:14:09.273Z] > Task :streams:javadoc
[2022-01-21T21:14:09.273Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:102:
 warning - Tag @link: reference not found: this#getResult()
[2022-01-21T21:14:09.273Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-01-21T21:14:09.273Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-01-21T21:14:09.273Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:147:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-01-21T21:14:09.273Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:147:
 warning - Tag @link: reference not found: this#isFailure()
[2022-01-21T21:14:10.214Z] 5 warnings
[2022-01-21T21:14:11.153Z] 
[2022-01-21T21:14:11.153Z] > Task :streams:javadocJar
[2022-01-21T21:14:13.787Z] > Task :clients:javadoc
[2022-01-21T21:14:15.823Z] > Task :clients:javadocJar
[2022-01-21T21:14:15.823Z] 
[2022-01-21T21:14:15.823Z] > Task :clients:srcJar
[2022-01-21T21:14:15.823Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-01-21T21:14:15.823Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-01-21T21:14:16.761Z] 
[2022-01-21T21:14:16.761Z] > Task :clients:testJar
[2022-01-21T21:14:16.875Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-01-21T21:14:16.876Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-01-21T21:14:17.672Z] > Task :clients:testSrcJar
[2022-01-21T21:14:17.672Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-01-21T21:14:17.672Z] > Task :clients:publishToMavenLocal
[2022-01-21T21:14:31.713Z] > Task :core:compileScala
[2022-01-21T21:16:36.352Z] > Task :core:classes
[2022-01-21T21:16:36.352Z] > Task :core:compileTestJava NO-SOURCE
[2022-01-21T21:16:55.369Z] > Task :core:compileTestScala
[2022-01-21T21:18:44.325Z] > Task :core:testClasses
[2022-01-21T21:18:54.513Z] > Task :streams:compileTestJava
[2022-01-21T21:18:54.513Z] > Task :streams:testClasses
[2022-01-21T21:18:55.447Z] > Task :streams:testJar
[2022-01-21T21:18:56.384Z] > Task :streams:testSrcJar
[2022-01-21T21:18:56.384Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-01-21T21:18:56.384Z] > Task :streams:publishToMavenLocal
[2022-01-21T21:18:56.384Z] 
[2022-01-21T21:18:56.384Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-01-21T21:18:56.384Z] 
[2022-01-21T21:18:56.384Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-01-21T21:18:56.384Z] 
[2022-01-21T21:18:56.384Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-01-21T21:18:56.384Z] 
[2022-01-21T21:18:56.384Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure 

[GitHub] [kafka-site] bbejeck commented on a change in pull request #393: MINOR: Update site for Apache Kafka 3.1.0 release

[GitBox]

bbejeck commented on a change in pull request #393:
URL: https://github.com/apache/kafka-site/pull/393#discussion_r78637



##
File path: downloads.html
##
@@ -6,12 +6,58 @@

 Download
 
-3.0.0 is the latest release. The current stable version is 3.0.0.
+3.1.0 is the latest release. The current stable version is 3.1.0.
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html;>procedures and using 
these https://downloads.apache.org/kafka/KEYS;>KEYS.
 
 
+
+3.1.0
+
+
+Released January 24, 2022
+
+
+https://downloads.apache.org/kafka/3.1.0/RELEASE_NOTES.html;>Release 
Notes
+
+
+Source download: https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka-3.1.0-src.tgz;>kafka-3.1.0-src.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka-3.1.0-src.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka-3.1.0-src.tgz.sha512;>sha512)
+
+
+Binary downloads:
+
+Scala 2.12 - https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.12-3.1.0.tgz;>kafka_2.12-3.1.0.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka_2.12-3.1.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka_2.12-3.1.0.tgz.sha512;>sha512)
+Scala 2.13 - https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.13-3.1.0.tgz;>kafka_2.13-3.1.0.tgz
 (https://downloads.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz.sha512;>sha512)
+
+We build for multiple versions of Scala. This only matters if you 
are using Scala and you want a version
+built for the same Scala version you use. Otherwise any version 
should work (2.13 is recommended).
+
+
+
+
+Kafka 3.0.0 includes a number of significant new features. Here is a 
summary of some notable changes:

Review comment:
   Just noticed this should this be `3.1.0` here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: State store guarantees - KStreams 2.8.1

[Jiří Syrový]

I agree it sounds a bit off, but it seems that even a host that is not
marked as active allows me to query it's store and gives me a result that
is not null.

This application has an API that either queries local or remote
store (basically via HTTP API of active host), but the weird part is I get
the local response from both instances instead of expected one remote (on
non-active and non-standby host) and one local.
In principle the code to query stores looks like this

streams
  .store(
StoreQueryParameters
  .fromNameAndType(
storeName,
QueryableStoreTypes.keyValueStore[K, V]()
  )
  )
  .get(key)


And responses look like this:
$ curl //123 *(response from instance A)*
*{"meta":"KeyQueryMetadata
{activeHost=HostInfo{host='ip-1-2-3-4.eu-west-1.compute.internal',
port=8080}, standbyHosts=[],
partition=0}","value":{"timestamp":"2022-01-21T00:00:02.433Z","enabled":true},"hostname":"ip-1-2-3-4.eu-west-1.compute.internal"}*
$ curl //123 *(response from instance B)*
*{"meta":"KeyQueryMetadata
{activeHost=HostInfo{host='ip-1-2-3-4.eu-west-1.compute.internal',
port=8080}, standbyHosts=[],
partition=0}","value":{"timestamp":"2022-01-21T15:55:27.807Z","enabled":false},"hostname":"ip-9-8-7-6.eu-west-1.compute.internal"}*

This behaviour is not random and is 100% reproducible. I can try to create
a minimal code example that will demonstrate it.

On Fri, 21 Jan 2022 at 18:20, Matthias J. Sax  wrote:

> > but instance A returns
> >> result X for a partition I and instance B returns result Y for the same
> >> partition I.
>
> This sound a little off. As you stated, if both instances agree on the
> active host, the active host must either be instance A or instance B,
> and thus you can query partition I only on instance A or instance B. The
> non-active instance should not return any data for a partition it does
> not host.
>
> Can you elaborate?
>
> -Matthias
>
> On 1/21/22 4:47 AM, Jiří Syrový wrote:
> > Hi everyone,
> >
> > I'm trying for a while to answer myself a question about what are
> actually
> > guarantees for state stores in regards to consistency when connected to
> > transformers.
> >
> > I have an application where a single (persistent, rocksdb backed) state
> > store is connected to multiple transformers. Each transformer might both
> > read (get) and write (put) data into the state store. All transformers
> > receive data from multiple input topics in the same way (the same key,
> same
> > number of partitions) that before sending it to transformers merged
> > together.
> >
> > All transformers are located in the same sub-topology.
> >
> > What I observed is that even with 0 standby replicas I might get
> > inconsistent results when querying this state store connected to multiple
> > transformers. I have 2 instances and metadata on both instances agree on
> > the active host for this state store and partition, but instance A
> returns
> > result X for a partition I and instance B returns result Y for the same
> > partition I.
> >
> > Any suggestions if this is a bug or is my assumption incorrect that the
> > same state store should give the same result for the same key (same
> > partition) in 2 distinct transformers fed from the same input?
> >
> > Thanks,
> > Jiri
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #68

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

[Jason Gustafson]

For a little background on why `AlterIsr` returns the state back in the
response, originally the idea was that the partition leader could use the
response to reset its own state after a failed request. The tricky thing
for ISR changes is always ensuring that the partition leader reflects a
worst-case assumption about the current ISR to avoid advancing the high
watermark incorrectly. Ultimately I do not think we used error responses as
intended, so we might not have a hard requirement to keep all of these
fields. I'd probably suggest sticking with the current approach and
revisiting in a separate KIP, but I don't feel strongly about it.

In regard to the naming of `IsLeaderRecovering`, I agree it still seems a
bit awkward. I kind of liked the idea of turning it into a `PartitionState`
field instead. That would also address the inconsistent type in the
`PartitionChangeRecord`.

1. Changing the name of the AlterIsr RPC to AlterPartition RPC.


> 2. Change the name of the field "CurrentIsrVersion" to
> "PartitionEpoch". This is the name that we use in the KRaft
> implementation and the PartitionRecord.


Both of these suggestions make sense to me.

Best,
Jason

On Fri, Jan 21, 2022 at 11:07 AM José Armando García Sancio
 wrote:

> Hi all,
>
> The following suggestions are not strictly required to implement this
> KIP but what do we think about:
>
> 1. Changing the name of the AlterIsr RPC to AlterPartition RPC.
>
> 2. Change the name of the field "CurrentIsrVersion" to
> "PartitionEpoch". This is the name that we use in the KRaft
> implementation and the PartitionRecord.
>
> Thanks
> -José
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.0 #169

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

[José Armando García Sancio]

Hi all,

The following suggestions are not strictly required to implement this
KIP but what do we think about:

1. Changing the name of the AlterIsr RPC to AlterPartition RPC.

2. Change the name of the field "CurrentIsrVersion" to
"PartitionEpoch". This is the name that we use in the KRaft
implementation and the PartitionRecord.

Thanks
-José

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

[José Armando García Sancio]

Thanks Raman and Colin for your feedback.

Raman wrote:
> - Could you please explain the following about backward compatibility.
> If a leader has been elected unclean. And we decide to roll the
> cluster back when the leader is in the middle of recovery, leader will
> simply not be able to recover when we roll back because it will lose
> its local copy of the unclean flag. Even if the controller sends
> another `LISR` to the leader, it will ignore the flag. So, the leader
> will not be able to carry on its recovery workflow. It will treat the
> situation as if it was not elected unclean and carry on with expanding
> ISR.-

Yes, that is correct. Note that the default value of
"IsLeaderRecovering" in the AlterIsr request is false. That means that
if the controller supports this feature, the validation will succeed
because the size of the ISR is greater than 1 and the
"IsLeaderRecovering" flag is false. If the controller doesn't support
this feature then the ISR will get expanded and the value stored in ZK
will not have the "IsLeaderRecovering".

Raman wrote:
> Is there a specific need for adding "isUnclean '' flag to
> `AlterIsrResponse`. A potential sequence of events will be:
> 1. Controller sets the flag at `ZK` and informs the leader via the
> `LeaderAndIsrRequest`
> 2. Leader undertakes the necessary recovery and sends out a
> `AlterIsrRequest` to controller, with "isUnclean" flag reset
> 3. Controller will reset the flag at `ZK` if the `AlterIsrRequest` goes 
> through.
> Shouldn't the partition level error code at `AlterIsrResponse` be
> enough for the leader to know of a successful update.

I am mainly returning the recorded value for consistency with the
existing fields in the AlterIsrResponse. The implementation on the
broker (AlterIsrManager and Partition) takes advantage of this by
updating it's local state using the response values if the
"CurrentIsrVersion" validates.

Colin wrote:
> "isLeaderRecovering" sounds pretty awkward. If we want to call this "leader 
> recovery" then maybe the flag could be something like "inLeaderRecovery." 
> Actually, how about "inElectionRecovery" to emphasize the fact that we are 
> recovering from an unclean leader election?

I like the name "inElectionRecovery". I'll update the KIP to use this
name. Thanks.

Colin wrote:
> It seems like we both agree that a partition could be stuck in "election 
> recovery" forever if it is running pre-KIP-704 software and there are no 
> available followers to be added. For example, if there are two replicas, and 
> one of them went down and the other was elected uncleanly as leader. Is the 
> argument that being in election recovery forever in this case is not a 
> problem?

You are correct. The partition state can stay in this state forever if
the leader doesn't support this feature, and the replication factor is
1 or the followers never catch up to the leader. I was trying to argue
that this is okay because the leader and future leader will because as
expected.

Colin wrote:
> Can you given an example of a case where a broker would set recovery to true 
> in the AlterIsr RPC? If we can't think of any, then we don't need to add this 
> flag.

That's correct. In the controller I was going to enforce the invariant
that the partition leader is not allowed to change the
"InElectionRecovery'' from false to true. I was going to allow the
broker to send an AlterIsr RPC that doesn't change any state (ISR or
"InElectionRecovery"). I don't think that we should make it implicit
that any AlterIsr RPC that passes the "CurrentIsrVersion" check means
that the leader has recovered from an unclean election.

The current implementation of AlterIsrManager and Partition on the
broker update the leader replica state using the field in the
response. This means that we have to return "InElectionRecovery" in
the response. I think it is awkward for the response to contain this
field but for the request to not contain this field.

Thanks
-José

[GitHub] [kafka-site] junrao merged pull request #392: adding new vulnerability info - CVE-2022-23307

[GitBox]

junrao merged pull request #392:
URL: https://github.com/apache/kafka-site/pull/392


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-814: Static membership protocol should let the leader skip assignment

[Jason Gustafson]

Hey David,

Thanks for the proposal. This was a tricky bug and I think your approach is
probably the best way forward.

It would be helpful to add a little more detail to the proposal. When the
coordinator detects that the static leader is returning, it will set
`skipAssignment` to true in the `JoinGroup` response. I believe the intent
is to return all member subscriptions in this response so that the leader
can monitor all topics subscribed in the group (which might be different
from the consumer's own subscription). The leader will then send an empty
`SyncGroup` request to collect its own assignment. Do I have that right?

I think there might still be an edge case in this proposal (assuming I've
understood it correctly). In between the time that the leader shuts down
and is restarted, it is possible that new partitions are added to one of
the subscribed topics. The returning leader would not know about it
because it has no way to collect the full assignment. Do you think this is
a problem?

Thanks,
Jason

On Wed, Jan 19, 2022 at 7:27 AM David Jacot  wrote:

> Hi folks,
>
> I'd like to start a discussion for KIP-814: Static membership protocol
> should let the
> leader skip assignment. This is a small extension to the static
> membership protocol
> to address KAFKA-13435.
>
> The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw.
>
> Please let me know what you think.
>
> Best,
> David
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #623

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 507111 lines...]
[2022-01-21T18:40:21.986Z] 
[2022-01-21T18:40:21.986Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToCommitToMultiplePartitions[exactly_once] PASSED
[2022-01-21T18:40:21.986Z] 
[2022-01-21T18:40:21.986Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskFails[exactly_once] STARTED
[2022-01-21T18:40:33.448Z] 
[2022-01-21T18:40:33.448Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskFails[exactly_once] PASSED
[2022-01-21T18:40:33.448Z] 
[2022-01-21T18:40:33.448Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToPerformMultipleTransactions[exactly_once] STARTED
[2022-01-21T18:40:34.214Z] 
[2022-01-21T18:40:34.214Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigratePersistentWindowStoreToTimestampedWindowStoreUsingPapi PASSED
[2022-01-21T18:40:34.214Z] 
[2022-01-21T18:40:34.214Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi STARTED
[2022-01-21T18:40:37.337Z] 
[2022-01-21T18:40:37.337Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToPerformMultipleTransactions[exactly_once] PASSED
[2022-01-21T18:40:37.337Z] 
[2022-01-21T18:40:37.337Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToCommitMultiplePartitionOffsets[exactly_once] STARTED
[2022-01-21T18:40:41.140Z] 
[2022-01-21T18:40:41.140Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToCommitMultiplePartitionOffsets[exactly_once] PASSED
[2022-01-21T18:40:41.140Z] 
[2022-01-21T18:40:41.140Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldCommitCorrectOffsetIfInputTopicIsTransactional[exactly_once] STARTED
[2022-01-21T18:40:49.893Z] 
[2022-01-21T18:40:49.893Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldCommitCorrectOffsetIfInputTopicIsTransactional[exactly_once] PASSED
[2022-01-21T18:40:49.893Z] 
[2022-01-21T18:40:49.893Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRunWithTwoSubtopologies[exactly_once] STARTED
[2022-01-21T18:40:54.114Z] 
[2022-01-21T18:40:54.114Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRunWithTwoSubtopologies[exactly_once] PASSED
[2022-01-21T18:40:54.114Z] 
[2022-01-21T18:40:54.114Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once] 
STARTED
[2022-01-21T18:41:10.359Z] 
[2022-01-21T18:41:10.359Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once] 
PASSED
[2022-01-21T18:41:10.359Z] 
[2022-01-21T18:41:10.359Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldWriteLatestOffsetsToCheckpointOnShutdown[exactly_once] STARTED
[2022-01-21T18:41:20.035Z] 
[2022-01-21T18:41:20.035Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldWriteLatestOffsetsToCheckpointOnShutdown[exactly_once] PASSED
[2022-01-21T18:41:20.035Z] 
[2022-01-21T18:41:20.035Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskFailsWithState[exactly_once] STARTED
[2022-01-21T18:41:25.158Z] 
[2022-01-21T18:41:25.158Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldProxyKeyValueStoreToTimestampedKeyValueStoreUsingPapi PASSED
[2022-01-21T18:41:25.158Z] 
[2022-01-21T18:41:25.158Z] 
org.apache.kafka.streams.integration.StoreUpgradeIntegrationTest > 
shouldMigrateInMemoryWindowStoreToTimestampedWindowStoreUsingPapi STARTED
[2022-01-21T18:41:49.070Z] 
[2022-01-21T18:41:49.070Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldNotViolateEosIfOneTaskFailsWithState[exactly_once] PASSED
[2022-01-21T18:41:49.070Z] 
[2022-01-21T18:41:49.070Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRunWithTwoSubtopologiesAndMultiplePartitions[exactly_once] STARTED
[2022-01-21T18:41:50.122Z] 
[2022-01-21T18:41:50.122Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRunWithTwoSubtopologiesAndMultiplePartitions[exactly_once] PASSED
[2022-01-21T18:41:50.122Z] 
[2022-01-21T18:41:50.122Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRestartAfterClose[exactly_once] STARTED
[2022-01-21T18:41:59.989Z] 
[2022-01-21T18:41:59.989Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRestartAfterClose[exactly_once] PASSED
[2022-01-21T18:41:59.989Z] 
[2022-01-21T18:41:59.989Z] 
org.apache.kafka.streams.integration.EosIntegrationTest > 
shouldBeAbleToRunWithEosEnabled[exactly_once_v2] STARTED
[2022-01-21T18:42:03.449Z] 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #67

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13412) Retry of initTransactions after timeout may cause invalid transition

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13412.
-
Fix Version/s: 3.2.0
   3.1.1
   Resolution: Fixed

> Retry of initTransactions after timeout may cause invalid transition
> 
>
> Key: KAFKA-13412
> URL: https://issues.apache.org/jira/browse/KAFKA-13412
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
>
> If `initTransactions()` cannot be completed before the timeout defined by 
> `max.block.ms`, then the call will raise a `TimeoutException`. The user is 
> expected to retry this, which is what Kafka Streams does. However, the 
> producer will keep retrying the `InitProducerId` request in the background 
> and it is possible for it to return before the retry call to 
> `initTransaction()`. This leads to the following exception:
> {code}
> org.apache.kafka.common.KafkaException: TransactionalId blah: Invalid 
> transition attempted from state READY to state INITIALIZING
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:1077)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:1070)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.lambda$initializeTransactions$1(TransactionManager.java:336)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.handleCachedTransactionRequestResult(TransactionManager.java:1198)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.initializeTransactions(TransactionManager.java:333)
>   at 
> org.apache.kafka.clients.producer.internals.TransactionManager.initializeTransactions(TransactionManager.java:328)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:597)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13611) Failed reconfiguration of tasks can cause missing offset replications in MirrorCheckpointConnector

[Jira]

Aljoscha Pörtner created KAFKA-13611:


 Summary: Failed reconfiguration of tasks can cause missing offset 
replications in MirrorCheckpointConnector
 Key: KAFKA-13611
 URL: https://issues.apache.org/jira/browse/KAFKA-13611
 Project: Kafka
  Issue Type: Improvement
  Components: mirrormaker
Affects Versions: 3.0.0, 2.8.1, 2.7.2, 2.6.3, 3.1.0, 2.6.2, 2.7.1, 2.8.0, 
2.6.1, 2.7.0, 2.5.1, 2.6.0, 2.4.1, 2.5.0, 2.4.0
Reporter: Aljoscha Pörtner


Because the _knownConsumerGroups_ are stored within a variable an not queried 
every time _refreshConsumerGroups_ gets executed, errors within the task 
reconfiguration aren't recognized and the reconfiguration will not be retried 
until a new consumer group is added. This can lead to missing offset updates in 
the target cluster because the consumer group is not picked up by a task until 
a completely new consumer group is added and the task reconfiguration is 
successful.

 
{code:java}
private void refreshConsumerGroups()throws 
InterruptedException, ExecutionException {List consumerGroups = 
findConsumerGroups();Set newConsumerGroups = new HashSet<>();   
 newConsumerGroups.addAll(consumerGroups);
newConsumerGroups.removeAll(knownConsumerGroups);Set 
deadConsumerGroups = new HashSet<>();
deadConsumerGroups.addAll(knownConsumerGroups);
deadConsumerGroups.removeAll(consumerGroups);if 
(!newConsumerGroups.isEmpty() || !deadConsumerGroups.isEmpty()) {
log.info("Found {} consumer groups for {}. {} are new. {} were removed. 
Previously had {}.",consumerGroups.size(), sourceAndTarget, 
newConsumerGroups.size(), deadConsumerGroups.size(),
knownConsumerGroups.size());log.debug("Found new consumer groups: 
{}", newConsumerGroups);knownConsumerGroups = consumerGroups;   
 context.requestTaskReconfiguration();}} {code}
[Code|https://github.com/apache/kafka/blob/ca37f14076adbaa302a558a750be197c202f1038/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java#L124]

For an example how the problem can be triggered take a look at the following 
[issue|https://github.com/strimzi/strimzi-kafka-operator/issues/3688].



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: State store guarantees - KStreams 2.8.1

[Matthias J. Sax]

but instance A returns

result X for a partition I and instance B returns result Y for the same
partition I.


This sound a little off. As you stated, if both instances agree on the 
active host, the active host must either be instance A or instance B, 
and thus you can query partition I only on instance A or instance B. The 
non-active instance should not return any data for a partition it does 
not host.


Can you elaborate?

-Matthias

On 1/21/22 4:47 AM, Jiří Syrový wrote:

Hi everyone,

I'm trying for a while to answer myself a question about what are actually
guarantees for state stores in regards to consistency when connected to
transformers.

I have an application where a single (persistent, rocksdb backed) state
store is connected to multiple transformers. Each transformer might both
read (get) and write (put) data into the state store. All transformers
receive data from multiple input topics in the same way (the same key, same
number of partitions) that before sending it to transformers merged
together.

All transformers are located in the same sub-topology.

What I observed is that even with 0 standby replicas I might get
inconsistent results when querying this state store connected to multiple
transformers. I have 2 instances and metadata on both instances agree on
the active host for this state store and partition, but instance A returns
result X for a partition I and instance B returns result Y for the same
partition I.

Any suggestions if this is a bug or is my assumption incorrect that the
same state store should give the same result for the same key (same
partition) in 2 distinct transformers fed from the same input?

Thanks,
Jiri

[jira] [Resolved] (KAFKA-13388) Kafka Producer nodes stuck in CHECKING_API_VERSIONS

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13388.
-
Fix Version/s: 3.0.1
   3.2.0
   3.1.1
 Reviewer: Rajini Sivaram
   Resolution: Fixed

> Kafka Producer nodes stuck in CHECKING_API_VERSIONS
> ---
>
> Key: KAFKA-13388
> URL: https://issues.apache.org/jira/browse/KAFKA-13388
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: David Hoffman
>Assignee: David Jacot
>Priority: Critical
> Fix For: 3.0.1, 3.2.0, 3.1.1
>
> Attachments: Screen Shot 2021-10-25 at 10.28.48 AM.png, 
> image-2021-10-21-13-42-06-528.png
>
>
> I have been seeing expired batch errors in my app.
> {code:java}
> org.apache.kafka.common.errors.TimeoutException: Expiring 51 record(s) for 
> xxx-17:120002 ms has passed since batch creation
> {code}
>  I would have assumed a request timout or connection timeout should have also 
> been logged. I could not find any other associated errors. 
> I added some instrumenting to my app and have traced this down to broker 
> connections hanging in CHECKING_API_VERSIONS state. -It appears there is no 
> effective timeout for Kafka Producer broker connections in 
> CHECKING_API_VERSIONS state.-
> In the code see the after the NetworkClient connects to a broker node it 
> makes a request to check api versions, when it receives the response it marks 
> the node as ready. -I am seeing that sometimes a reply is not received for 
> the check api versions request the connection just hangs in 
> CHECKING_API_VERSIONS state until it is disposed I assume after the idle 
> connection timeout.-
> Update: not actually sure what causes the connection to get stuck in 
> CHECKING_API_VERSIONS.
> -I am guessing the connection setup timeout should be still in play for this, 
> but it is not.- 
>  -There is a connectingNodes set that is consulted when checking timeouts and 
> the node is removed- 
>  -when ClusterConnectionStates.checkingApiVersions(String id) is called to 
> transition the node into CHECKING_API_VERSIONS-



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13591) Fix flaky test ControllerIntegrationTest.testTopicIdCreatedOnUpgrade

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13591.
-
Fix Version/s: 3.2.0
   3.1.1
 Reviewer: Jason Gustafson
   Resolution: Fixed

> Fix flaky test ControllerIntegrationTest.testTopicIdCreatedOnUpgrade
> 
>
> Key: KAFKA-13591
> URL: https://issues.apache.org/jira/browse/KAFKA-13591
> Project: Kafka
>  Issue Type: Bug
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Minor
> Fix For: 3.2.0, 3.1.1
>
>
> {noformat}
> org.opentest4j.AssertionFailedError: expected: not equal but was: 
>   at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
>   at 
> org.junit.jupiter.api.AssertNotEquals.failEqual(AssertNotEquals.java:276)
>   at 
> org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:265)
>   at 
> org.junit.jupiter.api.AssertNotEquals.assertNotEquals(AssertNotEquals.java:260)
>   at 
> org.junit.jupiter.api.Assertions.assertNotEquals(Assertions.java:2798)
>   at 
> kafka.controller.ControllerIntegrationTest.testTopicIdCreatedOnUpgrade(ControllerIntegrationTest.scala:1140)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13610) Make log.cleaner.enable dynamically configurable

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-13610:


 Summary: Make log.cleaner.enable dynamically configurable
 Key: KAFKA-13610
 URL: https://issues.apache.org/jira/browse/KAFKA-13610
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13609) Fix the exception type thrown from dynamic broker config validation

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-13609:


 Summary: Fix the exception type thrown from dynamic broker config 
validation
 Key: KAFKA-13609
 URL: https://issues.apache.org/jira/browse/KAFKA-13609
 Project: Kafka
  Issue Type: Bug
Reporter: Colin McCabe


Currently, we throw InvalidRequestException if the broker configuration fails 
validation. However, it would be more appropriate to throw 
InvalidConfigurationException in this case.

This change might require a KIP since it has compatibility implications.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #66

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13608) Implement Position restoration for all in-memory state stores

[Vicky Papavasileiou (Jira)]

Vicky Papavasileiou created KAFKA-13608:
---

 Summary: Implement Position restoration for all in-memory state 
stores
 Key: KAFKA-13608
 URL: https://issues.apache.org/jira/browse/KAFKA-13608
 Project: Kafka
  Issue Type: Sub-task
Reporter: Vicky Papavasileiou


In-memory state stores restore their state from the changelog (as opposed to 
RocksDB stores that restore from disk). In-memory stores currently don't handle 
restoring of the Position



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #392: adding new vulnerability info - CVE-2022-23307

[GitBox]

scott-confluent commented on a change in pull request #392:
URL: https://github.com/apache/kafka-site/pull/392#discussion_r789758488



##
File path: cve-list.html
##
@@ -9,6 +9,31 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://nvd.nist.gov/vuln/detail/CVE-2022-23307;>CVE-2022-23307 
Deserialization of Untrusted Data Flaw in Apache Log4j logging library in 
versions 1.x
+
+  This CVE identified a flaw where it allows an attacker to send a 
malicious request with serialized data to the component running log4j 
1.x to be deserialized when the chainsaw component is run. Chainsaw is a 
standalone GUI for viewing log entries in log4j. An attacker not only needs to 
be able to generate malicious log entries, but also, have the necessary access 
and permissions to start chainsaw (or if it is already enabled by a customer / 
consumer of Apache Kafka).
+  
+  
+  
+
+  Versions affected
+  All AK versions
+
+
+  Fixed versions
+  NA

Review comment:
   Updated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #168

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 418447 lines...]
[2022-01-21T14:52:19.667Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() STARTED
[2022-01-21T14:52:21.539Z] 
[2022-01-21T14:52:21.539Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() PASSED
[2022-01-21T14:52:21.539Z] 
[2022-01-21T14:52:21.539Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() STARTED
[2022-01-21T14:52:26.255Z] 
[2022-01-21T14:52:26.255Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() PASSED
[2022-01-21T14:52:26.255Z] 
[2022-01-21T14:52:26.255Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() STARTED
[2022-01-21T14:52:30.037Z] 
[2022-01-21T14:52:30.037Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() PASSED
[2022-01-21T14:52:30.037Z] 
[2022-01-21T14:52:30.037Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() STARTED
[2022-01-21T14:52:31.823Z] 
[2022-01-21T14:52:31.823Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() PASSED
[2022-01-21T14:52:31.823Z] 
[2022-01-21T14:52:31.823Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
STARTED
[2022-01-21T14:52:34.517Z] 
[2022-01-21T14:52:34.517Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
PASSED
[2022-01-21T14:52:34.517Z] 
[2022-01-21T14:52:34.517Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() STARTED
[2022-01-21T14:52:39.373Z] 
[2022-01-21T14:52:39.373Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() PASSED
[2022-01-21T14:52:39.373Z] 
[2022-01-21T14:52:39.373Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() STARTED
[2022-01-21T14:52:44.140Z] 
[2022-01-21T14:52:44.140Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() PASSED
[2022-01-21T14:52:44.140Z] 
[2022-01-21T14:52:44.140Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() STARTED
[2022-01-21T14:52:47.727Z] 
[2022-01-21T14:52:47.727Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() PASSED
[2022-01-21T14:52:47.727Z] 
[2022-01-21T14:52:47.727Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() STARTED
[2022-01-21T14:52:49.733Z] 
[2022-01-21T14:52:49.733Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() PASSED
[2022-01-21T14:52:49.733Z] 
[2022-01-21T14:52:49.733Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() STARTED
[2022-01-21T14:52:54.401Z] 
[2022-01-21T14:52:54.401Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() PASSED
[2022-01-21T14:52:54.401Z] 
[2022-01-21T14:52:54.401Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() STARTED
[2022-01-21T14:52:59.070Z] 
[2022-01-21T14:52:59.070Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() PASSED
[2022-01-21T14:52:59.070Z] 
[2022-01-21T14:52:59.070Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() STARTED
[2022-01-21T14:53:02.983Z] 
[2022-01-21T14:53:02.983Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() PASSED
[2022-01-21T14:53:02.983Z] 
[2022-01-21T14:53:02.983Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() STARTED
[2022-01-21T14:53:04.831Z] 
[2022-01-21T14:53:04.831Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() PASSED
[2022-01-21T14:53:04.831Z] 
[2022-01-21T14:53:04.831Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithNonExistingFeatureZNode()
 STARTED
[2022-01-21T14:53:07.760Z] 
[2022-01-21T14:53:07.760Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithNonExistingFeatureZNode()
 PASSED
[2022-01-21T14:53:07.760Z] 
[2022-01-21T14:53:07.760Z] ControllerIntegrationTest > testAlterIsrErrors() 
STARTED
[2022-01-21T14:53:09.538Z] 
[2022-01-21T14:53:09.538Z] ControllerIntegrationTest > testAlterIsrErrors() 
PASSED
[2022-01-21T14:53:09.538Z] 
[2022-01-21T14:53:09.538Z] ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection() STARTED
[2022-01-21T14:53:16.927Z] 
[2022-01-21T14:53:16.927Z] ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection() PASSED
[2022-01-21T14:53:16.927Z] 
[2022-01-21T14:53:16.927Z] ControllerIntegrationTest > testTopicCreation() 
STARTED
[2022-01-21T14:53:18.721Z] 
[2022-01-21T14:53:18.721Z] ControllerIntegrationTest > testTopicCreation() 
PASSED
[2022-01-21T14:53:18.721Z] 
[2022-01-21T14:53:18.721Z] ControllerIntegrationTest > 

[jira] [Created] (KAFKA-13607) Cannot use PEM certificate coding when parent defined file-based

[Piotr Smolinski (Jira)]

Piotr Smolinski created KAFKA-13607:
---

 Summary: Cannot use PEM certificate coding when parent defined 
file-based
 Key: KAFKA-13607
 URL: https://issues.apache.org/jira/browse/KAFKA-13607
 Project: Kafka
  Issue Type: Bug
  Components: clients, config, KafkaConnect
Affects Versions: 3.0.0, 2.7.1
Reporter: Piotr Smolinski


The problem applies to the situation when we create a Kafka client based on 
prepopulated config. If we have only partial control on the input we can 
attempt to reset some values.

KIP-651 added a new cool feature to use PEM coding of certificates as an 
alternative to file stores. I have observed a problem in Confluent Replicator. 
We have shifted the common configuration to the worker level and assumed the 
connectors define only what is specific for them. The security setup is mTLS, 
i.e. we need both client cert and trusted chain. Our default configuration has 
both in #PKCS12 files, but we had to reverse the replication direction and 
redefine the destination coordinates. For these we have certificates, but 
having KIP-651 we could specify them as connector params as opposed to the 
worker deployment change.

It came out that we cannot override **ssl.keystore.location**, 
**ssl.keystore.password**, etc. simply with empty values, because the code in 
the **DefaultSslEngineFactory** checks if the entry is null. We can only 
override it to empty string.

**DefaultSslEngineFactory** should treat the unexpected configuration entries 
as absent when they are **null**, but also when the given entry is an empty 
string.

For a workaround I have created a hacky patch that fixes the behaviour:

https://github.com/piotrsmolinski/kafka-ssl-fix

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

__consumer_offsets auto creation with `auto.create.topics.enable = false`

[Dominic Evans]

Hi,

Kafka 2.8.x changed the behaviour of auto topic creation (KAFKA-9751) so that 
it's now done asynchronously by a forwarded request to the controller rather 
than being immediately started by the broker that received the MetadataRequest. 
In 2.7.x and earlier the client used to get LEADER_NOT_AVAILABLE on the first 
MetadataResponse as the topic creation was immediately in-progress and the 
partitions and leaders were just being assigned. With 2.8.x onward, instead 
they now get UNKNOWN_TOPIC_OR_PARTITION until the Controller has actioned the 
create. That seems fine and is easy to handle.

However, historically the __consumer_offsets topic was special cased such that 
it would get auto-created on a Metadata request even if normal topic 
auto-creation (`auto.create.topics.enable`) was disabled in the broker. Does 
anyone know if something has changed there?

If I startup a brand new empty 2.8.x cluster and just send MetadataRequests for 
the __consumer_offsets topic  I always get UNKNOWN_TOPIC_OR_PARTITION  
responses and the auto-creation doesn't seem to ever get actioned for that 
alone. As you'd expect, if I start up a consumer group the JoinGroup is 
sufficient to cause the __consumer_offsets topic to be created. On 2.7.x and 
earlier, just sending MetadataRequests is sufficient for the __consumer_offsets 
topic to be created.

The behaviour can be observed using (e.g.,) kcat (neé kafkacat) after starting 
a brand new cluster with auto.create.topics.enable=false:
> kcat -L -b localhost:29091,localhost:29092,localhost:29093 -t 
> __consumer_offsets -d metadata,topic

And no auto-creation ever happens despite allowAutoCreate true in the Metadata 
request body:

Kafka (Metadata v4 Request)
Length: 42
API Key: Metadata (3)
API Version: 4
Correlation ID: 3
Client ID: rdkafka
Topic
Topic Name: __consumer_offsets
Allow Auto Topic Creation: True

Is this a regression or an intentional change in behaviour? I couldn't find any 
reference to this in the changelog / release notes.

Best regards,
Dom
-- 
Dominic Evans
IBM Software Group, Hursley Park, UK

[RESULTS] [VOTE] Release Kafka version 3.1.0

[David Jacot]

This vote passes with 7 +1 votes (4 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Randall Hauch
* Bill Bejeck
* Mickael Maison
* David Jacot

Committers:
* No votes

Community:
* Michal Tóth
* Jakub Scholz
* Israel Ekpo

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/thread/mvmm4xcpf3020oqgjxgo8gj80szom14f

I'll continue with the release process and the release announcement
will follow in the next few days.

Best,
David

Re: [VOTE] 3.1.0 RC1

[David Jacot]

Thanks for all the votes!

I'm also +1 (binding) given that:

* Ran the release and generated 3.1.0 RC1.
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Installed 3.1.0 RC1 and performed the quickstart for both ZK and KRaft.
* Spotchecked the Javadocs and the site doc.

I will close the vote shortly.

Best,
David

On Thu, Jan 20, 2022 at 1:21 PM Mickael Maison  wrote:
>
> Thanks David for running this release!
>
> I checked signatures and keys, built from source and ran unit tests
> with Java 11. I also ran the quickstart.
>
> +1 (binding)
>
> Thanks,
> Mickael
>
> On Wed, Jan 19, 2022 at 9:20 PM Bill Bejeck  wrote:
> >
> > Hi David,
> >
> > Thanks for running the release!
> >
> > I did the following:
> >
> >1. Validated all signatures and keys
> >2. Built from source
> >3. Ran all the unit tests
> >4. Spot checked the java docs, and the site docs
> >5. Ran a portion of the quick start
> >6. Ran the Kafka Streams demo application
> >
> > I noticed one small error on the quick start page: the  "Kafka Streams demo 
> > and
> > the app development tutorial" links still reference the 2.5 release.
> > This is a very minor issue and it doesn't require a new RC, we can simply
> > update the links once released.
> >
> > +1(binding)
> >
> > Thanks,
> > Bill
> >
> >
> > On Wed, Jan 19, 2022 at 8:44 AM David Jacot 
> > wrote:
> >
> > > Thanks for all the votes! We are still missing two binding votes from
> > > PMC members.
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Jan 18, 2022 at 5:00 PM Israel Ekpo  wrote:
> > > >
> > > > Performed the following validations using the tools available here:
> > > >
> > > > https://github.com/izzyacademy/apache-kafka-release-party
> > > >
> > > >- Verified signatures, keys and hashes for release artifacts
> > > >- Deployed Multi-Node Cluster in Legacy Mode (with Zookeeper)
> > > >- Deployed Multi-Node Cluster in KRaft Mode (without Zookeeper)
> > > >- Can confirm that KAFKA-13456 works as expected after switching to
> > > 3.1.0
> > > >- Briefly Walked Through 3.1 Documentation, Javadocs and Protocol
> > > Pages
> > > >
> > > > +1 (non-binding) for the release candidate
> > > >
> > > > Thanks for running this release
> > > >
> > > > Israel Ekpo
> > > > Lead Instructor, IzzyAcademy.com
> > > > https://www.youtube.com/c/izzyacademy
> > > > https://izzyacademy.com/
> > > >
> > > >
> > > > On Wed, Jan 12, 2022 at 7:59 AM David Jacot  wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second candidate for release of Apache Kafka 3.1.0.
> > > > >
> > > > > * Apache Kafka supports Java 17
> > > > > * The FetchRequest supports Topic IDs (KIP-516)
> > > > > * Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
> > > > > * Add broker count metrics (KIP-748)
> > > > > * Differentiate consistently metric latency measured in millis and
> > > > > nanos (KIP-773)
> > > > > * The eager rebalance protocol is deprecated (KAFKA-13439)
> > > > > * Add TaskId field to StreamsException (KIP-783)
> > > > > * Custom partitioners in foreign-key joins (KIP-775)
> > > > > * Fetch/findSessions queries with open endpoints for
> > > > > SessionStore/WindowStore (KIP-766)
> > > > > * Range queries with open endpoints (KIP-763)
> > > > > * Add total blocked time metric to Streams (KIP-761)
> > > > > * Add additional configuration to control MirrorMaker2 internal topics
> > > > > naming convention (KIP-690)
> > > > >
> > > > > Release notes for the 3.1.0 release:
> > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Monday, January 17, 9am PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.1.0-rc1
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/31/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/31/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > Unit/integration tests:
> > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/
> > > > > System tests:
> > > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > >

State store guarantees - KStreams 2.8.1

[Jiří Syrový]

Hi everyone,

I'm trying for a while to answer myself a question about what are actually
guarantees for state stores in regards to consistency when connected to
transformers.

I have an application where a single (persistent, rocksdb backed) state
store is connected to multiple transformers. Each transformer might both
read (get) and write (put) data into the state store. All transformers
receive data from multiple input topics in the same way (the same key, same
number of partitions) that before sending it to transformers merged
together.

All transformers are located in the same sub-topology.

What I observed is that even with 0 standby replicas I might get
inconsistent results when querying this state store connected to multiple
transformers. I have 2 instances and metadata on both instances agree on
the active host for this state store and partition, but instance A returns
result X for a partition I and instance B returns result Y for the same
partition I.

Any suggestions if this is a bug or is my assumption incorrect that the
same state store should give the same result for the same key (same
partition) in 2 distinct transformers fed from the same input?

Thanks,
Jiri

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #167

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 416052 lines...]
[2022-01-21T12:34:47.748Z] KafkaZkClientTest > 
testLogDirEventNotificationsDeletion() PASSED
[2022-01-21T12:34:47.748Z] 
[2022-01-21T12:34:47.748Z] KafkaZkClientTest > testGetLogConfigs() STARTED
[2022-01-21T12:34:47.748Z] 
[2022-01-21T12:34:47.748Z] KafkaZkClientTest > testGetLogConfigs() PASSED
[2022-01-21T12:34:47.748Z] 
[2022-01-21T12:34:47.748Z] KafkaZkClientTest > testBrokerSequenceIdMethods() 
STARTED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > testBrokerSequenceIdMethods() 
PASSED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > testAclMethods() STARTED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > testAclMethods() PASSED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > 
testCreateSequentialPersistentPath() STARTED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > 
testCreateSequentialPersistentPath() PASSED
[2022-01-21T12:34:48.673Z] 
[2022-01-21T12:34:48.673Z] KafkaZkClientTest > testConditionalUpdatePath() 
STARTED
[2022-01-21T12:34:49.598Z] 
[2022-01-21T12:34:49.598Z] KafkaZkClientTest > testConditionalUpdatePath() 
PASSED
[2022-01-21T12:34:49.598Z] 
[2022-01-21T12:34:49.598Z] KafkaZkClientTest > 
testGetAllTopicsInClusterTriggersWatch() STARTED
[2022-01-21T12:34:49.598Z] 
[2022-01-21T12:34:49.598Z] KafkaZkClientTest > 
testGetAllTopicsInClusterTriggersWatch() PASSED
[2022-01-21T12:34:49.598Z] 
[2022-01-21T12:34:49.598Z] KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testDeletePath() STARTED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testDeletePath() PASSED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testGetBrokerMethods() STARTED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testGetBrokerMethods() PASSED
[2022-01-21T12:34:50.523Z] 
[2022-01-21T12:34:50.523Z] KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2022-01-21T12:34:51.447Z] 
[2022-01-21T12:34:51.447Z] KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2022-01-21T12:34:51.447Z] 
[2022-01-21T12:34:51.447Z] KafkaZkClientTest > 
testCreateTokenChangeNotification() STARTED
[2022-01-21T12:34:51.447Z] 
[2022-01-21T12:34:51.447Z] KafkaZkClientTest > 
testCreateTokenChangeNotification() PASSED
[2022-01-21T12:34:51.447Z] 
[2022-01-21T12:34:51.447Z] KafkaZkClientTest > testGetTopicsAndPartitions() 
STARTED
[2022-01-21T12:34:52.372Z] 
[2022-01-21T12:34:52.372Z] KafkaZkClientTest > testGetTopicsAndPartitions() 
PASSED
[2022-01-21T12:34:52.372Z] 
[2022-01-21T12:34:52.372Z] KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] STARTED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[1] PASSED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] STARTED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testChroot(boolean) > 
kafka.zk.KafkaZkClientTest.testChroot(boolean)[2] PASSED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testRegisterBrokerInfo() STARTED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testRegisterBrokerInfo() PASSED
[2022-01-21T12:34:53.296Z] 
[2022-01-21T12:34:53.296Z] KafkaZkClientTest > testRetryRegisterBrokerInfo() 
STARTED
[2022-01-21T12:34:54.220Z] 
[2022-01-21T12:34:54.220Z] KafkaZkClientTest > testRetryRegisterBrokerInfo() 
PASSED
[2022-01-21T12:34:54.220Z] 
[2022-01-21T12:34:54.220Z] KafkaZkClientTest > testConsumerOffsetPath() STARTED
[2022-01-21T12:34:54.220Z] 
[2022-01-21T12:34:54.220Z] KafkaZkClientTest > testConsumerOffsetPath() PASSED
[2022-01-21T12:34:54.220Z] 
[2022-01-21T12:34:54.220Z] KafkaZkClientTest > 
testDeleteRecursiveWithControllerEpochVersionCheck() STARTED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > 
testDeleteRecursiveWithControllerEpochVersionCheck() PASSED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > testTopicAssignments() STARTED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > testTopicAssignments() PASSED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > 
testControllerManagementMethods() STARTED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > 
testControllerManagementMethods() PASSED
[2022-01-21T12:34:55.145Z] 
[2022-01-21T12:34:55.145Z] KafkaZkClientTest > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Tom Bentley]

Hi Dongjin,

Many thanks for the KIP. I have just a couple of minor points.

1. In the Proposed Changes section there is the sentence "For backward
compatibility, these tools will use the log4j configuration
file (tools-log4j2.properties) by default.", I think you mean
tools-log4j.properties there, right?

2. Can the log4j-appender log a warning about its own deprecation? The
"DEPRECATED: using log4j 1.x configuration. To use log4j 2.x
configuration," message is a bit confusing since it suggests that it's just
the use of 1.x configuration that's deprecated (which it is), but it's also
the use of the appender itself which is deprecated.

Kind regards,

Tom

On Thu, 20 Jan 2022 at 05:01, Dongjin Lee  wrote:

> Hi all,
>
> I just filed the issue based on the discussion here:
>
> KAFKA-13604: Add pluggable logging framework support (
> https://issues.apache.org/jira/browse/KAFKA-13604)
>
> Thanks for posing a great issue, Viktor and Haruki!
>
> Best,
> Dongjin
>
> On Thu, Jan 20, 2022 at 2:54 AM Viktor Somogyi-Vass
>  wrote:
>
> > Hi folks,
> >
> > >a) I think this makes sense. But I think this shouldn't block this
> > KIP, it can be done after moving to log4j2.
> >
> > Absolutely.
> >
> > >However, this limitation may change in the future - slf4j will support
> > dynamic logging levels in 2.0:
> >
> > This would be awesome, let's keep an eye on this.
> >
> > >Kafka avoided all the log4shell vulnerabilities exactly because we
> DIDN'T
> > move to log4j 2.x.
> >
> > My first reaction was how lucky we are (and all Kafka users) that we
> didn't
> > have to do anything on that weekend! Blessed are the lazy. :)
> >
> > Viktor
> >
> > On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe  wrote:
> >
> > > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > > > Hi Dongjin,
> > > >
> > > > We're also looking into this matter as our stack was also affected by
> > all
> > > > the log4j hell and users increasingly pushing us to upgrade to log4j2
> > or
> > > > logback because of the existing vulnerabilities in log4j1.
> > > > Regarding the points raised by Haruki:
> > > >
> > >
> > > I've heard the same thing from other people -- that there is now more
> > > interest in moving from log4j1.x to log4j2.x, after all the recent
> > > vulnerabilities in the latter. I found this a bit strange. Kafka
> avoided
> > > all the log4shell vulnerabilities exactly because we DIDN'T move to
> log4j
> > > 2.x. (Yes, I am aware that there is a longstanding vulnerability in
> that
> > > one log sink in log4j 1.x, but you can just not use that one.)
> > >
> > > I haven't thought about this very hard. Maybe it's still a good idea to
> > > move to log4j2. But it's odd that nobody is commenting about how in
> this
> > > case, not updating actually prevented a major security incident for
> > Kafka.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > a) In my opinion the best would be to make the dynamic logger support
> > > > (Log4jController and LoggingResource) pluggable for log4j2 and
> logback
> > > (so
> > > > an interface could be used to define the dynamic logging control
> > methods
> > > > and a config to specify the implementation). That way we're not bound
> > to
> > > > either logback or log4j and seems like a low-effort thing to do.
> > > > Additionally this could be used in Connect too in LoggingResource.
> > > >
> > > > b) I think testing dependencies aren't that important from the user
> > > > perspective, it's fine to either use log4j2 or logback, whichever is
> > > > easier. Kafka is either used from the distribution (tgz) or pulled in
> > > > through maven, but test dependencies shouldn't be exposed to the
> world.
> > > >
> > > > c) I would support deprecating the appender in favor of the log4j2
> > Kafka
> > > > appender. VerifiableLog4jAppender is intended as a testing tool
> anyway,
> > > so
> > > > I think it's less important to change this to logback.
> > > >
> > > > Future vulnerabilities will always be found in either logback or
> log4j2
> > > or
> > > > any other logging framework, so I think the safest approach is to
> allow
> > > > users to choose their implementation, while in tests I think we're
> free
> > > to
> > > > use whatever we want as that shouldn't be constrained by
> > vulnerabilities.
> > > >
> > > > Viktor
> > > >
> > > > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada 
> > > wrote:
> > > >
> > > >> Thanks for the clarification.
> > > >>
> > > >> About 2, I wan't aware of those concerns.
> > > >> Let me check them first.
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >> 2021年12月23日(木) 13:37 Dongjin Lee :
> > > >>
> > > >> > Hi Haruki,
> > > >> >
> > > >> >
> > > >> > Thanks for organizing the issue.
> > > >> >
> > > >> >
> > > >> > If the community prefers logback, I will gladly change the
> > dependency
> > > and
> > > >> > update the PR. However, it has the following issues:
> > > >> >
> > > >> >
> > > >> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 +
> > KIP-719
> > > >> are
> 

Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT

[Tom Bentley]

Hi Julien,

Thanks! A couple of other points sprang to mind:

1. seconds is a unit, but millis etc are really just prefixes. I can
imagine what my old physics teacher would have to say about mixing these
concepts :-). I would have preferred to use the abbreviations s, ms, µs and
ns (ms in particular would be consistent with the unit names used in
configs), but I'm quite sure many people would struggle to know how to type
µ. Maybe 'us' is an acceptable alternative (it's a pretty common
convention), or perhaps we should use 'seconds', 'milliseconds',
'microseconds' and 'nanoseconds' as the allowed values, even though they're
a bit long.

2. I think the truncation and sub-milliseconds behaviour when converting
to/from java.util.Date etc. should be part of the documentation of the
config.

Apart from these minor points this is looking good to me.

Kind regards,

Tom

On Wed, 19 Jan 2022 at 22:06, Julien Chanaud 
wrote:

> Hi Tom and thanks for your review,
>
> I agree and I have renamed the field name to unix.precision.
> I've replaced all references to the word "epoch" which I mistakenly used to
> describe measurements throughout the KIP.
> I have also modified the KIP name as well.
> Before : Add support for unix epoch precision in TimestampConverter SMT
> Now : Add support for different unix precisions in TimestampConverter SMT
>
> Let me know what you think,
>
> Julien
>
>
> Le mer. 19 janv. 2022 à 19:27, Tom Bentley  a écrit :
>
> > Hi Julien,
> >
> > I wonder if the name epoch.precision is a little confusing. An epoch is a
> > point in time chosen as a origin for a particular calendar system. As
> such
> > it doesn't have a precision. It's only measurements from this point in
> time
> > which have a precision. In the unix case, precisions of seconds, ms, µs
> and
> > ns seem to make the most sense. So I wonder if the name should be
> > unix.precision instead. That also makes it clearer that it only applies
> in
> > the type=unix case. Wdyt?
> >
> > Kind regards,
> >
> > Tom
> >
> > On Thu, 13 Jan 2022 at 13:42, Julien Chanaud 
> > wrote:
> >
> > > Hi Mickael,
> > >
> > > Thank you very much for your feedback and direction.
> > >
> > > I have added the documentation to the "Public Interfaces" chapter
> > > (formatted in a table as I've seen in other KIPs) and I'll put this
> > > KIP to a vote next week as per your suggestion.
> > >
> > > Regards,
> > >
> > > Julien
> > >
> > > Le mer. 12 janv. 2022 à 18:19, Mickael Maison
> > >  a écrit :
> > > >
> > > > Hi Julien,
> > > >
> > > > Thanks for the KIP. I looks like a useful improvement to the
> > > > TimestampConverter SMT.
> > > >
> > > > I'd suggest adding the documentation for the new setting to the KIP.
> > > > I've had to go check your PR to fully understand how you want to use
> > > > it, both for input and output.
> > > > Apart from that, if you don't get any further feedback, feel free to
> > > > start a vote.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Dec 21, 2021 at 2:19 PM Julien Chanaud <
> > chanaud.jul...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Bumping this KIP discussion.
> > > > > It's a small change, entirely backward compatible and I'd love your
> > > > > feedback on it.
> > > > > Thanks,
> > > > > Julien
> > > > >
> > > > >
> > > > > Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud <
> > chanaud.jul...@gmail.com>
> > > a écrit :
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I would like to start a discussion for KIP-808
> > > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT
> > > > > >
> > > > > > This seems like a simple change but I suspect there are several
> > > things to consider, most notably regarding the java.util.Date object,
> > which
> > > is at the heart of the conversions.
> > > > > >
> > > > > > Let me know what you think.
> > > > > >
> > > > > > Julien
> > > > > >
> > >
> > >
> >
>

[jira] [Created] (KAFKA-13606) MirrorCheckpointTask doesn't check offsets sync result

[Stanislav Savulchik (Jira)]

Stanislav Savulchik created KAFKA-13606:
---

 Summary: MirrorCheckpointTask doesn't check offsets sync result
 Key: KAFKA-13606
 URL: https://issues.apache.org/jira/browse/KAFKA-13606
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 3.0.0, 2.8.1, 2.7.2, 2.7.1, 2.8.0, 2.7.0
Reporter: Stanislav Savulchik


{{MirrorCheckpointTask}} doesn't check the result of calling 
{{AdminClient#alterConsumerGroupOffsets}} method that could return a failed 
Future for instance due to incorrect ACL in the target kafka cluster.

[https://github.com/apache/kafka/blob/3.0.0/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointTask.java#L302]

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)