Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1485

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1486

[Apache Jenkins Server]

See 

Re: [VOTE] 3.3.2 RC1

[Mickael Maison]

+1 binding

- Checked signatures
- Ran quickstart with Scala 2.13 binaries
- Ran unit and integration tests

Thanks Chris for running this release


On Wed, Jan 4, 2023 at 11:37 AM Satish Duggana  wrote:
>
> Hi Chris,
> Thanks for running the release.
>
> +1 (binding)
>
> - Verified signatures and artifacts
> - Ran testAll/releaseTarGzAll successfully with no failures.
> - Ran through quickstart of core/streams on builds generated from the tag.
> - Ran a few internal apps targeting to topics on 3 node cluster.
>
> On Wed, 4 Jan 2023 at 00:49, Manikumar  wrote:
> >
> > Hi Chris,
> >
> > +1 (binding)
> >
> > - verified the signatures, artifacts
> > - ran the tests on the source archive
> > - verified the quickstarts
> >
> > Thanks for running the release!
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Fri, Dec 23, 2022 at 4:44 PM Federico Valeri  
> > wrote:
> > >
> > > Hi, I did the following to validate the release:
> > >
> > > - Checksums and signatures ok
> > > - Build from source using Java 17 and Scala 2.13 ok
> > > - Unit and integration tests ok
> > > - Quickstart in both ZK and KRaft modes ok
> > > - Test app with staging Maven artifacts ok
> > >
> > > +1 (non binding)
> > >
> > > Thanks
> > > Fede
> > >
> > > On Wed, Dec 21, 2022 at 11:21 PM Chris Egerton  
> > > wrote:
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for release of Apache Kafka 3.3.2.
> > > >
> > > > This is a bugfix release with several fixes since the release of 3.3.1. 
> > > > A
> > > > few of the major issues include:
> > > >
> > > > * KAFKA-14358 Users should not be able to create a regular topic name
> > > > __cluster_metadata
> > > > KAFKA-14379 Consumer should refresh preferred read replica on update
> > > > metadata
> > > > * KAFKA-13586 Prevent exception thrown during connector update from
> > > > crashing distributed herder
> > > >
> > > >
> > > > Release notes for the 3.3.2 release:
> > > > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday, January 6, 2023, 10pm UTC
> > > > (this date is chosen to accommodate the various upcoming holidays that
> > > > members of the community will be taking and give everyone enough time to
> > > > test out the release candidate, without unduly delaying the release)
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~cegerton/kafka-3.3.2-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.3 branch) is the 3.3.2 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.3.2-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/33/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/33/protocol.html
> > > >
> > > > The most recent build has had test failures. These all appear to be due 
> > > > to
> > > > flakiness, but it would be nice if someone more familiar with the failed
> > > > tests could confirm this. I may update this thread with passing build 
> > > > links
> > > > if I can get one, or start a new release vote thread if test failures 
> > > > must
> > > > be addressed beyond re-running builds until they pass.
> > > >
> > > > Unit/integration tests:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.3/142/testReport/
> > > >
> > > > José, would it be possible to re-run the system tests for 3.3 on the 
> > > > latest
> > > > commit for 3.3 (e3212f2), and share the results on this thread?
> > > >
> > > > Cheers,
> > > >
> > > > Chris

[jira] [Created] (KAFKA-14575) Move ClusterTool

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14575:
--

 Summary: Move ClusterTool
 Key: KAFKA-14575
 URL: https://issues.apache.org/jira/browse/KAFKA-14575
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison
Assignee: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14576) Move ConsoleConsumer to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14576:
--

 Summary: Move ConsoleConsumer to tools
 Key: KAFKA-14576
 URL: https://issues.apache.org/jira/browse/KAFKA-14576
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14577) Move ConsoleProducer to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14577:
--

 Summary: Move ConsoleProducer to tools
 Key: KAFKA-14577
 URL: https://issues.apache.org/jira/browse/KAFKA-14577
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14578) Move ConsumerPerformance to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14578:
--

 Summary: Move ConsumerPerformance to tools
 Key: KAFKA-14578
 URL: https://issues.apache.org/jira/browse/KAFKA-14578
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14580) Move EndToEndLatency to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14580:
--

 Summary: Move EndToEndLatency to tools
 Key: KAFKA-14580
 URL: https://issues.apache.org/jira/browse/KAFKA-14580
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14579) Move DumpLogSegments to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14579:
--

 Summary: Move DumpLogSegments to tools
 Key: KAFKA-14579
 URL: https://issues.apache.org/jira/browse/KAFKA-14579
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14582) Move JmxTool to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14582:
--

 Summary: Move JmxTool to tools
 Key: KAFKA-14582
 URL: https://issues.apache.org/jira/browse/KAFKA-14582
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14581) Move GetOffsetShell to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14581:
--

 Summary: Move GetOffsetShell to tools
 Key: KAFKA-14581
 URL: https://issues.apache.org/jira/browse/KAFKA-14581
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14584) Move StateChangeLogMerger to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14584:
--

 Summary: Move StateChangeLogMerger to tools
 Key: KAFKA-14584
 URL: https://issues.apache.org/jira/browse/KAFKA-14584
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14585) Move StorageTool to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14585:
--

 Summary: Move StorageTool to tools
 Key: KAFKA-14585
 URL: https://issues.apache.org/jira/browse/KAFKA-14585
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14583) Move ReplicaVerificationTool to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14583:
--

 Summary: Move ReplicaVerificationTool to tools
 Key: KAFKA-14583
 URL: https://issues.apache.org/jira/browse/KAFKA-14583
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14586) Move StreamsResetter to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14586:
--

 Summary: Move StreamsResetter to tools
 Key: KAFKA-14586
 URL: https://issues.apache.org/jira/browse/KAFKA-14586
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14587) Move AclCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14587:
--

 Summary: Move AclCommand to tools
 Key: KAFKA-14587
 URL: https://issues.apache.org/jira/browse/KAFKA-14587
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14588) Move ConfigCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14588:
--

 Summary: Move ConfigCommand to tools
 Key: KAFKA-14588
 URL: https://issues.apache.org/jira/browse/KAFKA-14588
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14589) Move ConsumerGroupCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14589:
--

 Summary: Move ConsumerGroupCommand to tools
 Key: KAFKA-14589
 URL: https://issues.apache.org/jira/browse/KAFKA-14589
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14592) Move FeatureCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14592:
--

 Summary: Move FeatureCommand to tools
 Key: KAFKA-14592
 URL: https://issues.apache.org/jira/browse/KAFKA-14592
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14591) Move DeleteRecordsCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14591:
--

 Summary: Move DeleteRecordsCommand to tools
 Key: KAFKA-14591
 URL: https://issues.apache.org/jira/browse/KAFKA-14591
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14590) Move DelegationTokenCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14590:
--

 Summary: Move DelegationTokenCommand to tools
 Key: KAFKA-14590
 URL: https://issues.apache.org/jira/browse/KAFKA-14590
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14593) Move LeaderElectionCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14593:
--

 Summary: Move LeaderElectionCommand to tools
 Key: KAFKA-14593
 URL: https://issues.apache.org/jira/browse/KAFKA-14593
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14594) Move LogDirsCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14594:
--

 Summary: Move LogDirsCommand to tools
 Key: KAFKA-14594
 URL: https://issues.apache.org/jira/browse/KAFKA-14594
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14596) Move TopicCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14596:
--

 Summary: Move TopicCommand to tools
 Key: KAFKA-14596
 URL: https://issues.apache.org/jira/browse/KAFKA-14596
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14595) Move ReassignPartitionsCommand to tools

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-14595:
--

 Summary: Move ReassignPartitionsCommand to tools
 Key: KAFKA-14595
 URL: https://issues.apache.org/jira/browse/KAFKA-14595
 Project: Kafka
  Issue Type: Sub-task
Reporter: Mickael Maison






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14597) [Streams] record-e2e-latency-max is not reporting correct metrics

[Atul Jain (Jira)]

Atul Jain created KAFKA-14597:
-

 Summary: [Streams] record-e2e-latency-max is not reporting correct 
metrics 
 Key: KAFKA-14597
 URL: https://issues.apache.org/jira/browse/KAFKA-14597
 Project: Kafka
  Issue Type: Bug
Reporter: Atul Jain
 Attachments: process-latency-max.jpg, record-e2e-latency-max.jpg

I was following this KIP documentation 
([https://cwiki.apache.org/confluence/display/KAFKA/KIP-613%3A+Add+end-to-end+latency+metrics+to+Streams])
 and kafka streams documentation 
([https://kafka.apache.org/documentation/#kafka_streams_monitoring:~:text=node%2Did%3D(%5B%2D.%5Cw%5D%2B)-,record%2De2e%2Dlatency%2Dmax,-The%20maximum%20end])
 . Based on these documentations , the *record-e2e-latency-max* should monitor 
the full end to end latencies, which includes both *consumption latencies* and  
{*}processing delays{*}.

However, based on my observations , record-e2e-latency-max seems to be only 
measuring the consumption latencies. processing delays can be measured using 
*process-latency-max* .I am checking all this using a simple topology 
consisting of source, processors and sink (code added). I have added some sleep 
time (of 3 seconds) in one of the processors to ensure some delays in the 
processing logic. These delays are not getting accounted in the 
record-e2e-latency-max but are accounted in process-latency-max. 
process-latency-max was observed to be 3002 ms which accounts for sleep time of 
3 seconds. However, record-e2e-latency-max observed in jconsole is 422ms, which 
does not account for 3 seconds of sleep time.


Code describing my topology:
static Topology buildTopology(String inputTopic, String outputTopic) \{
log.info("Input topic: " + inputTopic + " and output topic: " + 
outputTopic);

Serde stringSerde = Serdes.String();
StreamsBuilder builder = new StreamsBuilder();
builder.stream(inputTopic, Consumed.with(stringSerde, stringSerde))
.peek((k,v) -> log.info("Observed event: key" + k + " value: " 
+ v))
.mapValues(s -> {
try {
System.out.println("sleeping for 3 seconds");
Thread.sleep(3000);
}
catch (InterruptedException e) \{
e.printStackTrace();
}
return  s.toUpperCase();
})
.peek((k,v) -> log.info("Transformed event: key" + k + " value: 
" + v))
.to(outputTopic, Produced.with(stringSerde, stringSerde));
return builder.build();
}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #29

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1487

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-890 Server Side Defense

[Jason Gustafson]

Hi Justine,

Thanks for the proposal.

I was thinking about the implementation a little bit. In the current
proposal, the behavior depends on whether we have an old or new client. For
old clients, we send `DescribeTransactions` and verify the result and for
new clients, we send `AddPartitionsToTxn`. We might be able to simplify the
implementation if we can use the same request type. For example, what if we
bump the protocol version for `AddPartitionsToTxn` and add a `validateOnly`
flag? For older versions, we can set `validateOnly=true` so that the
request only returns successfully if the partition had already been added.
For new versions, we can set `validateOnly=false` and the partition will be
added to the transaction. The other slightly annoying thing that this would
get around is the need to collect the transaction state for all partitions
even when we only care about a subset.

Some additional improvements to consider:

- We can give `AddPartitionsToTxn` better batch support for inter-broker
usage. Currently we only allow one `TransactionalId` to be specified, but
the broker may get some benefit being able to batch across multiple
transactions.
- Another small improvement is skipping topic authorization checks for
`AddPartitionsToTxn` when the request is from a broker. Perhaps we can add
a field for the `LeaderId` or something like that and require CLUSTER
permission when set.

Best,
Jason



On Mon, Dec 19, 2022 at 3:56 PM Jun Rao  wrote:

> Hi, Justine,
>
> Thanks for the explanation. It makes sense to me now.
>
> Jun
>
> On Mon, Dec 19, 2022 at 1:42 PM Justine Olshan
> 
> wrote:
>
> > Hi Jun,
> >
> > My understanding of the mechanism is that when we get to the last epoch,
> we
> > increment to the fencing/last epoch and if any further requests come in
> for
> > this producer ID they are fenced. Then the producer gets a new ID and
> > restarts with epoch/sequence 0. The fenced epoch sticks around for the
> > duration of producer.id.expiration.ms and blocks any late messages
> there.
> > The new ID will get to take advantage of the improved semantics around
> > non-zero start sequences. So I think we are covered.
> >
> > The only potential issue is overloading the cache, but hopefully the
> > improvements (lowered producer.id.expiration.ms) will help with that.
> Let
> > me know if you still have concerns.
> >
> > Thanks,
> > Justine
> >
> > On Mon, Dec 19, 2022 at 10:24 AM Jun Rao 
> wrote:
> >
> > > Hi, Justine,
> > >
> > > Thanks for the explanation.
> > >
> > > 70. The proposed fencing logic doesn't apply when pid changes, is that
> > > right? If so, I am not sure how complete we are addressing this issue
> if
> > > the pid changes more frequently.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Fri, Dec 16, 2022 at 9:16 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for replying!
> > > >
> > > > 70.We already do the overflow mechanism, so my change would just make
> > it
> > > > happen more often.
> > > > I was also not suggesting a new field in the log, but in the
> response,
> > > > which would be gated by the client version. Sorry if something there
> is
> > > > unclear. I think we are starting to diverge.
> > > > The goal of this KIP is to not change to the marker format at all.
> > > >
> > > > 71. Yes, I guess I was going under the assumption that the log would
> > just
> > > > look at its last epoch and treat it as the current epoch. I suppose
> we
> > > can
> > > > have some special logic that if the last epoch was on a marker we
> > > actually
> > > > expect the next epoch or something like that. We just need to
> > distinguish
> > > > based on whether we had a commit/abort marker.
> > > >
> > > > 72.
> > > > > if the producer epoch hasn't been bumped on the
> > > > broker, it seems that the stucked message will fail the sequence
> > > validation
> > > > and will be ignored. If the producer epoch has been bumped, we ignore
> > the
> > > > sequence check and the stuck message could be appended to the log.
> So,
> > is
> > > > the latter case that we want to guard?
> > > >
> > > > I'm not sure I follow that "the message will fail the sequence
> > > validation".
> > > > In some of these cases, we had an abort marker (due to an error) and
> > then
> > > > the late message comes in with the correct sequence number. This is a
> > > case
> > > > covered by the KIP.
> > > > The latter case is actually not something we've considered here. I
> > think
> > > > generally when we bump the epoch, we are accepting that the sequence
> > does
> > > > not need to be checked anymore. My understanding is also that we
> don't
> > > > typically bump epoch mid transaction (based on a quick look at the
> > code)
> > > > but let me know if that is the case.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Thu, Dec 15, 2022 at 12:23 PM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Justine,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 70.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #30

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1488

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #31

[Apache Jenkins Server]

See