Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #840

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-6204) Interceptor and MetricsReporter should implement java.io.Closeable

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xavier Léauté resolved KAFKA-6204.
--
Fix Version/s: 3.2.0
 Assignee: Xavier Léauté
   Resolution: Fixed

> Interceptor and MetricsReporter should implement java.io.Closeable
> --
>
> Key: KAFKA-6204
> URL: https://issues.apache.org/jira/browse/KAFKA-6204
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Charly Molter
>Assignee: Xavier Léauté
>Priority: Minor
> Fix For: 3.2.0
>
>
> The serializers and deserializers extends the Closeable interface, even 
> ConsumerInterceptors and ProducerInterceptors implement it.
> ConsumerInterceptor, ProducerInterceptor and MetricsReporter do not extend 
> the Closeable interface.
> Maybe they should for coherency with the rest of the apis.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #839

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #21

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 523355 lines...]
[2022-04-05T22:37:50.150Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task :connect:api:jar UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-04-05T22:37:50.150Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task :connect:json:jar UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-04-05T22:37:50.150Z] > Task :connect:api:javadocJar
[2022-04-05T22:37:50.150Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-04-05T22:37:50.150Z] > Task :connect:json:publishToMavenLocal
[2022-04-05T22:37:50.150Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-04-05T22:37:50.150Z] > Task :connect:api:testJar
[2022-04-05T22:37:50.150Z] > Task :connect:api:testSrcJar
[2022-04-05T22:37:50.150Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-04-05T22:37:50.150Z] > Task :connect:api:publishToMavenLocal
[2022-04-05T22:37:52.922Z] 
[2022-04-05T22:37:52.922Z] > Task :streams:javadoc
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-04-05T22:37:52.922Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-04-05T22:37:52.922Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2022-04-05T22:37:53.850Z] 12 warnings
[2022-04-05T22:37:53.850Z] 
[2022-04-05T22:37:53.850Z] > Task :streams:javadocJar
[2022-04-05T22:37:53.850Z] 
[2022-04-05T22:37:53.850Z] > Task :clients:javadoc
[2022-04-05T22:37:53.850Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-04-05T22:37:54.777Z] 1 warning
[2022-04-05T22:37:55.704Z] 
[2022-04-05T22:37:55.704Z] > Task :clients:javadocJar
[2022-04-05T22:37:56.632Z] 
[2022-04-05T22:37:56.632Z] > Task :clients:srcJar

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #838

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #103

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #20

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #837

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #196

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 419821 lines...]
[2022-04-05T19:17:08.421Z] 
[2022-04-05T19:17:08.421Z] KafkaZkClientTest > 
testSetGetAndDeletePartitionReassignment() STARTED
[2022-04-05T19:17:08.421Z] 
[2022-04-05T19:17:08.421Z] KafkaZkClientTest > 
testSetGetAndDeletePartitionReassignment() PASSED
[2022-04-05T19:17:08.421Z] 
[2022-04-05T19:17:08.421Z] KafkaZkClientTest > 
testIsrChangeNotificationsDeletion() STARTED
[2022-04-05T19:17:08.421Z] 
[2022-04-05T19:17:08.421Z] KafkaZkClientTest > 
testIsrChangeNotificationsDeletion() PASSED
[2022-04-05T19:17:08.421Z] 
[2022-04-05T19:17:08.421Z] KafkaZkClientTest > testGetDataAndVersion() STARTED
[2022-04-05T19:17:09.446Z] 
[2022-04-05T19:17:09.446Z] KafkaZkClientTest > testGetDataAndVersion() PASSED
[2022-04-05T19:17:09.446Z] 
[2022-04-05T19:17:09.446Z] KafkaZkClientTest > testGetChildren() STARTED
[2022-04-05T19:17:09.446Z] 
[2022-04-05T19:17:09.446Z] KafkaZkClientTest > testGetChildren() PASSED
[2022-04-05T19:17:09.446Z] 
[2022-04-05T19:17:09.446Z] KafkaZkClientTest > testSetAndGetConsumerOffset() 
STARTED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testSetAndGetConsumerOffset() 
PASSED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testClusterIdMethods() STARTED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testClusterIdMethods() PASSED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > 
testEntityConfigManagementMethods() STARTED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > 
testEntityConfigManagementMethods() PASSED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testUpdateLeaderAndIsr() STARTED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testUpdateLeaderAndIsr() PASSED
[2022-04-05T19:17:10.644Z] 
[2022-04-05T19:17:10.644Z] KafkaZkClientTest > testUpdateBrokerInfo() STARTED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testUpdateBrokerInfo() PASSED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testCreateRecursive() STARTED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testCreateRecursive() PASSED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testGetConsumerOffsetNoData() 
STARTED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testGetConsumerOffsetNoData() 
PASSED
[2022-04-05T19:17:11.847Z] 
[2022-04-05T19:17:11.847Z] KafkaZkClientTest > testDeleteTopicPathMethods() 
STARTED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > testDeleteTopicPathMethods() 
PASSED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > testSetTopicPartitionStatesRaw() 
STARTED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > testSetTopicPartitionStatesRaw() 
PASSED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > testAclManagementMethods() 
STARTED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > testAclManagementMethods() PASSED
[2022-04-05T19:17:12.981Z] 
[2022-04-05T19:17:12.981Z] KafkaZkClientTest > 
testPreferredReplicaElectionMethods() STARTED
[2022-04-05T19:17:13.917Z] 
[2022-04-05T19:17:13.917Z] KafkaZkClientTest > 
testPreferredReplicaElectionMethods() PASSED
[2022-04-05T19:17:13.917Z] 
[2022-04-05T19:17:13.917Z] KafkaZkClientTest > testPropagateLogDir() STARTED
[2022-04-05T19:17:13.917Z] 
[2022-04-05T19:17:13.917Z] KafkaZkClientTest > testPropagateLogDir() PASSED
[2022-04-05T19:17:13.917Z] 
[2022-04-05T19:17:13.917Z] KafkaZkClientTest > testGetDataAndStat() STARTED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > testGetDataAndStat() PASSED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > 
testReassignPartitionsInProgress() STARTED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > 
testReassignPartitionsInProgress() PASSED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > 
testChrootExistsAndRootIsLocked() STARTED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > 
testChrootExistsAndRootIsLocked() PASSED
[2022-04-05T19:17:14.943Z] 
[2022-04-05T19:17:14.943Z] KafkaZkClientTest > testCreateTopLevelPaths() STARTED
[2022-04-05T19:17:16.178Z] 
[2022-04-05T19:17:16.178Z] KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2022-04-05T19:17:16.178Z] 
[2022-04-05T19:17:16.178Z] KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2022-04-05T19:17:16.178Z] 
[2022-04-05T19:17:16.178Z] KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2022-04-05T19:17:16.178Z] 
[2022-04-05T19:17:16.178Z] KafkaZkClientTest > 

[jira] [Created] (KAFKA-13801) Kafka server does not respect MetricsReporter interface contract for dynamically configured reporters

[Jira]

Xavier Léauté created KAFKA-13801:
-

 Summary: Kafka server does not respect MetricsReporter interface 
contract for dynamically configured reporters
 Key: KAFKA-13801
 URL: https://issues.apache.org/jira/browse/KAFKA-13801
 Project: Kafka
  Issue Type: Bug
Reporter: Xavier Léauté


MetricsReporter.contextChange contract states the method should always
be called first before MetricsReporter.init is called. This is done
correctly for reporters enabled by default (e.g. JmxReporter) but not
for metrics reporters configured dynamically



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Chris Egerton]

Hi Jorge,

Looking good! I have a few comments left but all but one or two are minor.

1. The motivation section states "This KIP is aimed to include support for
nested structures on the existing SMTs... and to include the abstractions
to reuse this in future SMTs". A good implementation of this KIP will
definitely isolate reusable logic into a separate abstraction that can be
easily pulled in to the SMTs we want to add nested field support to, but
unless we plan on making this kind of abstraction publicly available as
some kind of utility method or class that external SMT developers can
leverage, we can probably leave this part out as it's more of an
implementation detail.

2. The Cast example is a little misleading, isn't it? It demonstrates the
escape syntax for fields with dot literals in their names, but it doesn't
demonstrate a way to actually use the Cast (or any other) SMT to access a
nested field in a record, which is the whole point of the KIP. I like the
example of escape syntax but we should probably also add one for nested
field access.

3. With the InsertField SMT, I'm wondering what the specific behavior will
be when some or all of the "middle layer" nested fields are missing. For
example, if I have a record with a value of { "k1": "v1 } and I apply
InsertField with topic.field = n1.n2.n3.topic, what will happen? Will the
updated value become { "k1": "v1", "n1": { "n2": { "n3": "topic" }}}, will
an exception be thrown, or something else? This seems analogous to the
command line mkdir command, which (at least on some operating systems)
fails by default if you try to create a new nested directory where anything
but the last element in the path doesn't exist, but can be invoked with a
flag that instructs it to go ahead and create all levels of nested
directory instead. I'm leaning on the side of "just create everything" but
would be interested in your thoughts, and either way, we should probably
make sure the intended behavior is well-defined before voting.

4. Similarly, what will the behavior be if any of the field elements
specified with InsertField already exist in the record value? Will we just
overwrite them? What's the behavior of InsertField today under similar
circumstances?

Cheers,

Chris

On Thu, Mar 31, 2022 at 4:15 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Thanks, Chris! Much appreciated all the feedback here.
>
> 1. You nailed it setting the design goal here: "it shouldn't be impossible
> to use this new feature for any field name, no matter how convoluted. It's
> fine if edge cases introduce difficulty (such as less-readable
> configurations), but it's not fine if they can't be addressed at all."
> Back to the previous proposals (using only dots as separators) we have 2
> alternatives:
> 1. escaping with backslashes
> 2. escaping with dots itself
>
> I'll lean for alternative 2, as you proposed before. Feels to me that
> backslashes have more potential to lead to confusion in JSON configs, and
> CSV seems like a good precedent to use the same character to escape itself.
> KIP is updated to reflect this.
>
> 2. Thanks! I'll add an example, and stick with the current approach
> defining the style per individual transform configuration.
>
> 3. Yes, thanks! KIP updated.
>
> 4. Of course. KIP updated.
>
> On Mon, 28 Mar 2022 at 21:59, Chris Egerton 
> wrote:
>
> > Hi Jorge,
> >
> > Thanks for addressing my comments; the KIP looks up-to-date and pretty
> > readable now, and the rejected alternatives section does a great job of
> > outlining the discussion so far and providing context for anyone else who
> > might want to join in.
> >
> > 1. Thoughts on choice of delimiter:
> > - I like the optimization for simple cases, but I think the new proposal
> is
> > a little too restrictive. What if there's a field whose name contains all
> > of the permitted options (currently just ".", ",", and "/")?
> > - If we expand the set of permitted delimiters to allow for any
> > single-character string, configuration complexity will increase and
> > readability may decrease
> > - Also worth pointing out that there is some convention for doubling a
> > delimiter character as an escape mechanism with formats like CSV [1]
> > - Overall I think we may be approaching the saturation point for
> productive
> > discussion on delimiter syntax so I don't want to spend too much more of
> > your time on it. I think the one point I'd like to leave for now is that
> it
> > shouldn't be impossible to use this new feature for any field name, no
> > matter how convoluted. It's fine if edge cases introduce difficulty (such
> > as less-readable configurations), but it's not fine if they can't be
> > addressed at all.
> >
> > 2.
> > The configuration style where you define "transforms.field.style" in the
> > connector config, and then this applies to all SMTs for the connector, is
> > very interesting. However, it doesn't follow convention for existing
> SMTs.
> > Right now, if you want to 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #102

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13800) Remove force cast of TimeWindowKStreamImpl in tests of https://github.com/apache/kafka/pull/11896

[Hao Li (Jira)]

Hao Li created KAFKA-13800:
--

 Summary: Remove force cast of TimeWindowKStreamImpl in tests of 
https://github.com/apache/kafka/pull/11896
 Key: KAFKA-13800
 URL: https://issues.apache.org/jira/browse/KAFKA-13800
 Project: Kafka
  Issue Type: Improvement
Reporter: Hao Li


We can remove the cast after `emitStrategy` is added to public api



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13782) Producer may fail to add the correct partition to transaction

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13782.
-
Resolution: Fixed

> Producer may fail to add the correct partition to transaction
> -
>
> Key: KAFKA-13782
> URL: https://issues.apache.org/jira/browse/KAFKA-13782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 3.2.0, 3.1.1
>
>
> In KAFKA-13412, we changed the logic to add partitions to transactions in the 
> producer. The intention was to ensure that the partition is added in 
> `TransactionManager` before the record is appended to the 
> `RecordAccumulator`. However, this does not take into account the possibility 
> that the originally selected partition may be changed if `abortForNewBatch` 
> is set in `RecordAppendResult` in the call to `RecordAccumulator.append`. 
> When this happens, the partitioner can choose a different partition, which 
> means that the `TransactionManager` would be tracking the wrong partition.
> I think the consequence of this is that the batches sent to this partition 
> would get stuck in the `RecordAccumulator` until they timed out because we 
> validate before sending that the partition has been added correctly to the 
> transaction.
> Note that KAFKA-13412 has not been included in any release, so there are no 
> affected versions.
> Thanks to [~alivshits] for identifying the bug.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13794) Producer batch lost silently in TransactionManager

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13794.
-
Fix Version/s: 3.1.1
   3.0.2
   Resolution: Fixed

> Producer batch lost silently in TransactionManager
> --
>
> Key: KAFKA-13794
> URL: https://issues.apache.org/jira/browse/KAFKA-13794
> Project: Kafka
>  Issue Type: Bug
>Reporter: xuexiaoyue
>Priority: Major
> Fix For: 3.1.1, 3.0.2
>
>
> Under the case of idempotence is enabled, when a batch reaches its 
> request.timeout.ms but not yet reaches delivery.timeout.ms, it will be 
> retried and wait for another request.timeout.ms. During the time of this 
> interval, the delivery.timeout.ms may be reached and Sender will remove this 
> in flight batch and bump the producer epoch because of the unresolved 
> sequence, then the sequence of this partition will be reset to 0.
> At this time, if a new batch is sent to the same partition and the former 
> batch reaches request.timeout.ms again, we will see an exception being thrown 
> out by NetworkClient:
> {code:java}
> [ERROR] [kafka-producer-network-thread | producer-1] 
> org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] 
> Uncaught error in request completion:
>  java.lang.IllegalStateException: We are re-enqueueing a batch which is not 
> tracked as part of the in flight requests. batch.topicPartition: 
> txn_test_1648891362900-2; batch.baseSequence: 0
>    at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.insertInSequenceOrder(RecordAccumulator.java:388)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.reenqueue(RecordAccumulator.java:334)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.Sender.reenqueueBatch(Sender.java:668)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:622)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:548)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:836)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109) 
> ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:583)
>  ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:575) 
> ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at 
> org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) 
> ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) 
> ~[kafka-transaction-test-1.0-SNAPSHOT.jar:?]
>    at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102] {code}
> The cause of this is the inflightBatchesBySequence in TransactionManager is 
> not being remove correctly. One batch may be removed by another batch with 
> the same sequence number.
> The potential consequence of this I can think out is that the send progress 
> will be blocked until the latter batch being expired by delivery.timeout.ms
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (KAFKA-13778) Fetch from follower should never run the preferred read replica selection

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13778.
-
Fix Version/s: 3.3.0
 Reviewer: David Jacot
   Resolution: Fixed

> Fetch from follower should never run the preferred read replica selection
> -
>
> Key: KAFKA-13778
> URL: https://issues.apache.org/jira/browse/KAFKA-13778
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.2.0
>Reporter: zhaobo
>Assignee: zhaobo
>Priority: Minor
> Fix For: 3.3.0
>
>
> The design purpose of the code is that only the leader broker can determine 
> the preferred read-replica.
>  
> {code:java}
> readFromLocalLog()
> 
> // If we are the leader, determine the preferred read-replica
> val preferredReadReplica = clientMetadata.flatMap(
>   metadata => findPreferredReadReplica(partition, metadata, replicaId, 
> fetchInfo.fetchOffset, fetchTimeMs)) {code}
>  
> But in fact, since the broker does not judge whether it is the leader or not, 
> the follower will also execute the preferred read-replica selection.
> {code:java}
> partition.leaderReplicaIdOpt.flatMap { leaderReplicaId =>
>   // Don't look up preferred for follower fetches via normal replication and
>   if (Request.isValidBrokerId(replicaId))
> None
>   else { {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] KIP-813 Shared State Stores

[Bill Bejeck]

Thanks for the KIP, Daan.

I've caught up on the discussion thread and I've gone over the KIP.  This
seems like a good addition to me.

+1 (binding)

Thanks,
Bill

On Fri, Apr 1, 2022 at 2:13 PM Matthias J. Sax  wrote:

> +1 (binding)
>
>
> On 4/1/22 6:47 AM, John Roesler wrote:
> > Thanks for the KIP, Daan!
> >
> > I’m +1 (binding)
> >
> > -John
> >
> > On Tue, Mar 29, 2022, at 06:01, Daan Gertis wrote:
> >> I would like to start a vote on this one:
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-813%3A+Shareable+State+Stores
> >>
> >> Cheers,
> >> D.
>

[jira] [Created] (KAFKA-13799) Improve documentation for Kafka zero-copy

[RivenSun (Jira)]

RivenSun created KAFKA-13799:


 Summary: Improve documentation for Kafka zero-copy
 Key: KAFKA-13799
 URL: https://issues.apache.org/jira/browse/KAFKA-13799
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: RivenSun


Via documentation https://kafka.apache.org/documentation/#maximizingefficiency
and [https://kafka.apache.org/documentation/#networklayer] ,
We can know that Kafka combines pagecache and zero-copy when reading messages 
in files on disk, which greatly improves the consumption rate of messages.
But after browsing the source code:
Look directly at the *FileRecords.writeTo(...)* method,
1. Only PlaintextTransportLayer.transferFrom() uses fileChannel.transferTo(), 
and the bottom layer calls the sendfile method to implement zero-copy data 
transfer.
2. The logic of the SslTransportLayer.transferFrom() method: 
{code:java}
fileChannel.read(fileChannelBuffer, pos) 
-> 
sslEngine.wrap(src, netWriteBuffer) 
-> 
flush(ByteBuffer buf) && socketChannel.write(buf){code}
That is, first read the data on the disk or directly from the page cache, then 
encrypt the data, and finally send the encrypted data to the network. 
{*}FileChannel.transferTo() is not used in the whole process{*}.

 

Conclusion: 

PlaintextTransportLayer and SslTransportLayer both use pagecache, but 
SslTransportLayer does not implement zero-copy.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #836

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13791) Fix FetchResponse#`fetchData` and `forgottenTopics`: Assignment of lazy-initialized members should be the last step with double-checked locking

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-13791.
---
Fix Version/s: 3.3.0
   Resolution: Fixed

> Fix FetchResponse#`fetchData` and `forgottenTopics`: Assignment of 
> lazy-initialized members should be the last step with double-checked locking
> ---
>
> Key: KAFKA-13791
> URL: https://issues.apache.org/jira/browse/KAFKA-13791
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.0.1
>Reporter: YunKui Lu
>Priority: Trivial
> Fix For: 3.3.0
>
>
> Double-checked locking can be used for lazy initialization of volatile 
> fields, but only if field assignment is the last step in the synchronized 
> block. Otherwise, you run the risk of threads accessing a half-initialized 
> object.
> The problem is consistent with 
> [KAFKA-13777|https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13777]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)