[jira] [Created] (KAFKA-13957) Flaky Test StoreQueryIntegrationTest.shouldQuerySpecificActivePartitionStores

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-13957:
--

 Summary: Flaky Test 
StoreQueryIntegrationTest.shouldQuerySpecificActivePartitionStores
 Key: KAFKA-13957
 URL: https://issues.apache.org/jira/browse/KAFKA-13957
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: A. Sophie Blee-Goldman


Failed on a local build so I have the full logs (attached)
{code:java}
java.lang.AssertionError: Unexpected exception thrown while getting the value 
from store.
Expected: is (a string containing "Cannot get state store source-table because 
the stream thread is PARTITIONS_ASSIGNED, not RUNNING" or a string containing 
"The state store, source-table, may have migrated to another instance" or a 
string containing "Cannot get state store source-table because the stream 
thread is STARTING, not RUNNING")
 but: was "The specified partition 1 for store source-table does not exist."
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest.verifyRetrievableException(StoreQueryIntegrationTest.java:539)
at 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest.lambda$shouldQuerySpecificActivePartitionStores$5(StoreQueryIntegrationTest.java:241)
at 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest.until(StoreQueryIntegrationTest.java:557)
at 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest.shouldQuerySpecificActivePartitionStores(StoreQueryIntegrationTest.java:183)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:833) {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13718) kafka-topics describe topic with default config will show `segment.bytes` overridden config

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-13718.
---
Fix Version/s: 3.3.0
   Resolution: Fixed

> kafka-topics describe topic with default config will show `segment.bytes` 
> overridden config 
> 
>
> Key: KAFKA-13718
> URL: https://issues.apache.org/jira/browse/KAFKA-13718
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 3.1.0, 2.8.1, 3.0.0
>Reporter: Luke Chen
>Assignee: Richard Joerger
>Priority: Major
>  Labels: newbie, newbie++
> Fix For: 3.3.0
>
>
> Following the quickstart guide[1], when describing the topic just created 
> with default config, I found there's a overridden config shown:
> _> bin/kafka-topics.sh --describe --topic quickstart-events 
> --bootstrap-server localhost:9092_
> _Topic: quickstart-events   TopicId: 06zRrzDCRceR9zWAf_BUWQ    
> PartitionCount: 1    ReplicationFactor: 1    *Configs: 
> segment.bytes=1073741824*_
>     _Topic: quickstart-events    Partition: 0    Leader: 0    Replicas: 0    
> Isr: 0_
>  
> This config result should be empty as in Kafka quick start page. Although the 
> config value is what we expected (default 1GB value), this info display still 
> confuse users.
>  
> Note: I checked the 2.8.1 build, this issue also happened.
>  
> [1]: [https://kafka.apache.org/quickstart]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[José Armando García Sancio]

Thanks for proposing this improvement David Jacot. I think it is going
to make the graceful shutdown process much more efficient.

+1 (binding) from me.

Re: [DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[José Armando García Sancio]

David Jacot wrote:
> At the moment, the KIP stipulates that the broker remains in
> InControlledShutdown state until it is re-registered with a new
> incarnation id. This implies that a broker can be both fenced and in
> controlled shutdown state. We could make them mutually exclusive but I
> think that there is value in the current proposal because we are able
> to differentiate if a broker was fenced due to the controlled shutdown
>or not.

Thanks David. Is this the reason why the BrokerRegistrationChangeRecord says:

> { "name": "InControlledShutdown", "type": "int8", "versions": "1+", 
> "taggedVersions": "1+", "tag": 1,
> "about": "0 if no change, 1 if the broker is in controlled shutdown." }

In other words the only way to change the InControlShutdown to false
is to create a new registration with a new incarnation id.

> The broker will leave this state when it registers itself with a new 
> incarnation id.

-José

[jira] [Created] (KAFKA-13956) kafka-storage should accept a release version

[David Arthur (Jira)]

David Arthur created KAFKA-13956:


 Summary: kafka-storage should accept a release version
 Key: KAFKA-13956
 URL: https://issues.apache.org/jira/browse/KAFKA-13956
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Arthur
Assignee: David Arthur


In addition to providing the feature flag level for metadata.version, we should 
allow users to pass in the more friendly release version (e.g., 3.0, 3.1-IV0). 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Jun Rao]

Hi, Luke,

Thanks for the reply.

10. You are saying it's difficult to track the number of segments to
recover. But how do we
implement 
kafka.log:type=LogManager,name=remainingSegmentsToRecovery,dir=([-._\/\w\d\s]+),threadNum=([0-9]+),
which tracks at the segment level?

Jun

On Thu, Jun 2, 2022 at 3:39 AM Luke Chen  wrote:

> Hi Jun,
>
> Thanks for the comment.
>
> Yes, I've tried to work on this way to track the number of remaining
> segments, but it will change the design in UnifiedLog, so I only track the
> logs number.
> Currently, we will load all segments and recover those segments if needed
> "during creating UnifiedLog instance". And also get the log offsets here
> <
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/UnifiedLog.scala#L1819-L1842
> >
> .
> That is, if we want to get all segments to be recovered before running log
> recovery, we need to break the logic in UnifiedLog, to create a partial
> UnifiedLog instance, and add more info to it later, which I think is just
> making the codes more complicated.
>
>
> Thank you.
> Luke
>
>
>
> On Thu, May 26, 2022 at 2:57 AM Jun Rao  wrote:
>
> > Hi, Luke,
> >
> > Thanks for the KIP. Just one comment.
> >
> > 10. For kafka.log:type=LogManager,name=remainingLogsToRecovery, could we
> > instead track the number of remaining segments? This monitors the
> progress
> > at a finer granularity and is also consistent with the thread level
> metric.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, May 25, 2022 at 7:47 AM Tom Bentley  wrote:
> >
> > > Thanks Luke! LGTM.
> > >
> > > On Sun, 22 May 2022 at 05:18, Luke Chen  wrote:
> > >
> > > > Hi Tom and Raman,
> > > >
> > > > Thanks for your comments.
> > > >
> > > > > 1. There's not a JIRA for this KIP (or the JIRA link needs
> updating).
> > > > 2. Similarly the link to this discussion thread needs updating.
> > > > > Please update the links to JIRA and the discussion thread.
> > > >
> > > > Yes, thanks for the reminder. I've updated the KIP.
> > > >
> > > > > 3. I wonder whether we need to keep these metrics (with value 0)
> once
> > > the
> > > > broker enters the running state. Do you see it as valuable? A benefit
> > of
> > > > removing the metrics would be a reduction on storage required for
> > metric
> > > > stores which are recording these metrics.
> > > >
> > > > Yes, removing the metrics after log recovery completed is a good
> idea.
> > > > Updated the KIP.
> > > >
> > > > > 4. I think the KIP's public interfaces section could be a bit
> > clearer.
> > > > Previous KIPs which added metrics usually used a table, with the
> MBean
> > > > name, metric type and description. SeeKIP-551 for example (or
> KIP-748,
> > > > KIP-608). Similarly you could use a table in the proposed changes
> > section
> > > > rather than describing the tree you'd see in an MBean console.
> > > >
> > > > Good point! Updated the KIP to use a table to list the MBean name,
> > metric
> > > > type and descriptions.
> > > >
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Fri, May 20, 2022 at 9:13 AM Raman Verma
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > The change is useful and simple. Thanks.
> > > > > Please update the links to JIRA and the discussion thread.
> > > > >
> > > > > Best Regards,
> > > > > Raman Verma
> > > > >
> > > > > On Thu, May 19, 2022 at 8:57 AM Tom Bentley 
> > > wrote:
> > > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Thanks for the KIP. I think the idea makes sense and would
> provide
> > > > useful
> > > > > > observability of log recovery. I have a few comments.
> > > > > >
> > > > > > 1. There's not a JIRA for this KIP (or the JIRA link needs
> > updating).
> > > > > > 2. Similarly the link to this discussion thread needs updating.
> > > > > > 3. I wonder whether we need to keep these metrics (with value 0)
> > once
> > > > the
> > > > > > broker enters the running state. Do you see it as valuable? A
> > benefit
> > > > of
> > > > > > removing the metrics would be a reduction on storage required for
> > > > metric
> > > > > > stores which are recording these metrics.
> > > > > > 4. I think the KIP's public interfaces section could be a bit
> > > clearer.
> > > > > > Previous KIPs which added metrics usually used a table, with the
> > > MBean
> > > > > > name, metric type and description. SeeKIP-551 for example (or
> > > KIP-748,
> > > > > > KIP-608). Similarly you could use a table in the proposed changes
> > > > section
> > > > > > rather than describing the tree you'd see in an MBean console.
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > > > On Wed, 11 May 2022 at 09:08, Luke Chen 
> wrote:
> > > > > >
> > > > > > > > And if people start using RemainingLogs and RemainingSegments
> > and
> > > > > then
> > > > > > > REALLY FEEL like they need RemainingBytes, then we can always
> add
> > > it
> > > > > in the
> > > > > > > future.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > 

Re: [VOTE] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Arthur]

Thanks, David. I think the KIP looks good. +1 (binding) from me

-David

On Wed, Jun 1, 2022 at 12:54 PM Colin McCabe  wrote:

> Thanks, David. With these changes, I am +1 (binding)
>
> Great to see this KIP moving forward!
>
> Colin
>
>
> On Wed, Jun 1, 2022, at 02:04, David Jacot wrote:
> > Hi Colin,
> >
> > Thanks for your feedback! Please find my answers below.
> >
> >> However, I wonder if this will be feasible for ZK-based brokers to do.
> We still support pre-topic-id IBP versions there, right? This might end up
> being more complex than you were hoping.
> >
> > That's right. We support pre-topic-id IBP versions for ZK-based
> > brokers. We will only use version 2 if all the topics have an ID when
> > the request is constructed. Version 1 is used otherwise. I have
> > already implemented this part in the draft PR if you want to see how
> > it looks. It brings a little more complexity in the request
> > building/handling but that works. I think that it is definitely worth
> > it.
> >
> >> We should add a comment in AlterPartitionResponse about the new error
> code INELIGIBLE_REPLICA. This is very important for error codes so we can
> track which ones are returned in which RPC version.
> >
> > Totally. I have those comments in the PR but I forgot to add them in
> > the KIP. I will fix this.
> >
> >> I also wonder if we should add a special error code for the case where
> the AlterPartitions call completes a reassignment AND the completed
> reassignment no longer includes the previous leader. We have been
> overloading FENCED_LEADER_EPOCH for this case, but this is coonfusing to
> operators (especially since this RPC does not support error messages, as
> opposed to codes)
> >
> > Interesting. I was not aware of this one. We don't do this in the ZK
> > controller, right? I do agree that using FENCED_LEADER_EPOCH is
> > confusing here. We could introduce a NEW_LEADER_ELECTED error code for
> > this purpose.
> >
> >> One thing, though, is that we should define how this interacts with
> replica placement. It seems to me that replicas should not be able to be
> placed on these inControlledShutdown nodes (unless done manually via the
> explicit placement API).
> >
> > Yeah, I agree that we need to clarify this. If not mistaken, replicas
> > can be placed on fenced nodes so I don't see why it should be
> > different for inControlledShutdown nodes. Otherwise we will again have
> > that create topic issue when the cluster has only 3 nodes. However I
> > think that we have to guarantee two other invariants:
> > 1) an inControlledShutdown node should not be added to the ISR when a
> > partition is created.
> > 2) an inControlledShutdown node should not be picked as a leader for a
> > partition. (e.g. KAFKA-13944)
> >
> > Let me add this part to the KIP.
> >
> >> I also think we should spell out the fact that once you go into
> controlled shutdown, you don't come out except by creating a new broker
> instance. (new incarnation ID). This also makes me wonder if we need to
> support the shutting down -> not shutting down transition in
> BrokerRegistrationChangeRecord, since we don't plan on using that
> transition.
> >
> > That makes sense. I will add this and remove that transition from
> > BrokerRegistrationChangeRecord.
> >
> >> Finally, RegisterBrokerRecord / BrokerRegistrationChangeRecord should
> be bumped to the next RPC version since we have added a new field. You will
> also need to assign yourself a new IBP / MetadataVersion. (For
> BrokerRegistrationChangeRecord it would be possible to avoid the version
> bump, since we're using tagged fields, but it's better to have it for
> consistency, I think.)
> >
> > Noted.
> >
> > Let me update the KIP to incorporate all your feedback.
> >
> > Cheers,
> > David
> >
> > On Tue, May 31, 2022 at 9:42 PM Colin McCabe  wrote:
> >>
> >> > We should add a comment in AlterPartitionResponse about the new error
> >> > code INELIGIBLE_REPLICA. This is very important for error codes so we
> >> > can track which ones are returned in which RPC version. I also wonder
> >>
> >> Here I'm referring to AlterPartitionResponse.json
> >>
> >> cheers,
> >> Colin
> >>
> >> >
> >> >
> >> > On Tue, May 31, 2022, at 08:36, David Jacot wrote:
> >> >> Hi folks,
> >> >>
> >> >> I'd like to start a vote for KIP-841:
> >> >> https://cwiki.apache.org/confluence/x/phmhD.
> >> >>
> >> >> Thanks,
> >> >> David
>


-- 
David Arthur

Log4j 1.2 update timeline?

[Cleveland, Michael]

Good morning,

I was wondering if there was a timeline when the version of Log4j will be 
updated when using Kafka?

Thank you,
Michael Cleveland

__
The information contained in this e-mail message and any attachments may be 
privileged and confidential. If the reader of this message is not the intended 
recipient or an agent responsible for delivering it to the intended recipient, 
you are hereby notified that any review, dissemination, distribution or copying 
of this communication is strictly prohibited. If you have received this 
communication in error, please notify the sender immediately by replying to 
this e-mail and delete the message and any attachments from your computer.
__

Re: [DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Arthur]

Good point about the unclean election behavior. I agree we wouldn't want to
elect a replica was on a shutting down broker.

> Right. There is a note about this in the Compatibility section.

Thanks :) I missed that

-David




On Thu, Jun 2, 2022 at 12:19 PM David Jacot 
wrote:

> Thanks for your feedback, David. Please find my answers below.
>
> > When an unclean election is performed, I'm assuming we will ignore the
> two
> new invariants proposed in the KIP? If so, maybe we should clarify that.
>
> I am not sure about this. What's the point of electing a broker which
> is fenced or shutting down in this case? At the moment, I think that
> we already prevent a fenced broker from being uncleanly elected if not
> mistaken. The KIP just extends this to include shutting down brokers
> as well.
>
> > I understand the use case of persisting the in-controller-shutdown state
> for controller failover, but are we planning on using this state on the
> leader side for ISR eligibility? Or will we just rely on the controller to
> check this state.
>
> Yes. The KIP explains that the leader will also enforce this based on
> the information it gets via the metadata log. The idea is to prevent
> unnecessary AlterPartition requests to the controller.
>
> > I think we should bump the IBP/MetadataVersion since we are adding a new
> field to a metadata record. Otherwise we might have trouble downgrading the
> software if this field has been serialized.
>
> Right. There is a note about this in the Compatibility section.
>
> Best,
> David
>
> On Thu, Jun 2, 2022 at 6:04 PM David Arthur  wrote:
> >
> > David, thanks for the KIP!
> >
> > When an unclean election is performed, I'm assuming we will ignore the
> two
> > new invariants proposed in the KIP? If so, maybe we should clarify that.
> >
> > I understand the use case of persisting the in-controller-shutdown state
> > for controller failover, but are we planning on using this state on the
> > leader side for ISR eligibility? Or will we just rely on the controller
> to
> > check this state.
> >
> > I think we should bump the IBP/MetadataVersion since we are adding a new
> > field to a metadata record. Otherwise we might have trouble downgrading
> the
> > software if this field has been serialized.
> >
> > Cheers,
> > David
> >
> > On Thu, Jun 2, 2022 at 11:04 AM David Jacot  >
> > wrote:
> >
> > > Hi José,
> > >
> > > At the moment, the KIP stipulates that the broker remains in
> > > InControlledShutdown state until it is re-registered with a new
> > > incarnation id. This implies that a broker can be both fenced and in
> > > controlled shutdown state. We could make them mutually exclusive but I
> > > think that there is value in the current proposal because we are able
> > > to differentiate if a broker was fenced due to the controlled shutdown
> > > or not.
> > >
> > > Best,
> > > David
> > >
> > > On Wed, Jun 1, 2022 at 7:32 PM José Armando García Sancio
> > >  wrote:
> > > >
> > > > Thanks for the updates to the KIP.
> > > >
> > > > I like enumerating invariants. Is it safe to say that if
> > > > `InControlledShutdown` is true then `Fenced` must be false.
> > >
> >
> >
> > --
> > David Arthur
>


-- 
David Arthur

Re: [DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Jacot]

Thanks for your feedback, David. Please find my answers below.

> When an unclean election is performed, I'm assuming we will ignore the two
new invariants proposed in the KIP? If so, maybe we should clarify that.

I am not sure about this. What's the point of electing a broker which
is fenced or shutting down in this case? At the moment, I think that
we already prevent a fenced broker from being uncleanly elected if not
mistaken. The KIP just extends this to include shutting down brokers
as well.

> I understand the use case of persisting the in-controller-shutdown state
for controller failover, but are we planning on using this state on the
leader side for ISR eligibility? Or will we just rely on the controller to
check this state.

Yes. The KIP explains that the leader will also enforce this based on
the information it gets via the metadata log. The idea is to prevent
unnecessary AlterPartition requests to the controller.

> I think we should bump the IBP/MetadataVersion since we are adding a new
field to a metadata record. Otherwise we might have trouble downgrading the
software if this field has been serialized.

Right. There is a note about this in the Compatibility section.

Best,
David

On Thu, Jun 2, 2022 at 6:04 PM David Arthur  wrote:
>
> David, thanks for the KIP!
>
> When an unclean election is performed, I'm assuming we will ignore the two
> new invariants proposed in the KIP? If so, maybe we should clarify that.
>
> I understand the use case of persisting the in-controller-shutdown state
> for controller failover, but are we planning on using this state on the
> leader side for ISR eligibility? Or will we just rely on the controller to
> check this state.
>
> I think we should bump the IBP/MetadataVersion since we are adding a new
> field to a metadata record. Otherwise we might have trouble downgrading the
> software if this field has been serialized.
>
> Cheers,
> David
>
> On Thu, Jun 2, 2022 at 11:04 AM David Jacot 
> wrote:
>
> > Hi José,
> >
> > At the moment, the KIP stipulates that the broker remains in
> > InControlledShutdown state until it is re-registered with a new
> > incarnation id. This implies that a broker can be both fenced and in
> > controlled shutdown state. We could make them mutually exclusive but I
> > think that there is value in the current proposal because we are able
> > to differentiate if a broker was fenced due to the controlled shutdown
> > or not.
> >
> > Best,
> > David
> >
> > On Wed, Jun 1, 2022 at 7:32 PM José Armando García Sancio
> >  wrote:
> > >
> > > Thanks for the updates to the KIP.
> > >
> > > I like enumerating invariants. Is it safe to say that if
> > > `InControlledShutdown` is true then `Fenced` must be false.
> >
>
>
> --
> David Arthur

Re: [DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Arthur]

David, thanks for the KIP!

When an unclean election is performed, I'm assuming we will ignore the two
new invariants proposed in the KIP? If so, maybe we should clarify that.

I understand the use case of persisting the in-controller-shutdown state
for controller failover, but are we planning on using this state on the
leader side for ISR eligibility? Or will we just rely on the controller to
check this state.

I think we should bump the IBP/MetadataVersion since we are adding a new
field to a metadata record. Otherwise we might have trouble downgrading the
software if this field has been serialized.

Cheers,
David

On Thu, Jun 2, 2022 at 11:04 AM David Jacot 
wrote:

> Hi José,
>
> At the moment, the KIP stipulates that the broker remains in
> InControlledShutdown state until it is re-registered with a new
> incarnation id. This implies that a broker can be both fenced and in
> controlled shutdown state. We could make them mutually exclusive but I
> think that there is value in the current proposal because we are able
> to differentiate if a broker was fenced due to the controlled shutdown
> or not.
>
> Best,
> David
>
> On Wed, Jun 1, 2022 at 7:32 PM José Armando García Sancio
>  wrote:
> >
> > Thanks for the updates to the KIP.
> >
> > I like enumerating invariants. Is it safe to say that if
> > `InControlledShutdown` is true then `Fenced` must be false.
>


-- 
David Arthur

Re: [DISCUSS] KIP-841: Fenced replicas should not be allowed to join the ISR in KRaft

[David Jacot]

Hi José,

At the moment, the KIP stipulates that the broker remains in
InControlledShutdown state until it is re-registered with a new
incarnation id. This implies that a broker can be both fenced and in
controlled shutdown state. We could make them mutually exclusive but I
think that there is value in the current proposal because we are able
to differentiate if a broker was fenced due to the controlled shutdown
or not.

Best,
David

On Wed, Jun 1, 2022 at 7:32 PM José Armando García Sancio
 wrote:
>
> Thanks for the updates to the KIP.
>
> I like enumerating invariants. Is it safe to say that if
> `InControlledShutdown` is true then `Fenced` must be false.

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #126

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 504843 lines...]
[2022-06-02T14:24:19.997Z] 
[2022-06-02T14:24:19.997Z] ControllerIntegrationTest > 
testTopicIdCreatedOnUpgradeMultiBrokerScenario() PASSED
[2022-06-02T14:24:19.997Z] 
[2022-06-02T14:24:19.997Z] ControllerIntegrationTest > 
testPreemptionWithCallbacks() STARTED
[2022-06-02T14:24:22.974Z] 
[2022-06-02T14:24:22.974Z] ControllerIntegrationTest > 
testPreemptionWithCallbacks() PASSED
[2022-06-02T14:24:22.974Z] 
[2022-06-02T14:24:22.974Z] ControllerIntegrationTest > 
testControllerDetectsBouncedBrokers() STARTED
[2022-06-02T14:24:28.362Z] 
[2022-06-02T14:24:28.362Z] ControllerIntegrationTest > 
testControllerDetectsBouncedBrokers() PASSED
[2022-06-02T14:24:28.362Z] 
[2022-06-02T14:24:28.362Z] ControllerIntegrationTest > testControlledShutdown() 
STARTED
[2022-06-02T14:24:31.116Z] 
[2022-06-02T14:24:31.116Z] ControllerIntegrationTest > testControlledShutdown() 
PASSED
[2022-06-02T14:24:31.116Z] 
[2022-06-02T14:24:31.116Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() STARTED
[2022-06-02T14:24:33.930Z] 
[2022-06-02T14:24:33.930Z] ControllerIntegrationTest > 
testPreemptionOnControllerShutdown() PASSED
[2022-06-02T14:24:33.930Z] 
[2022-06-02T14:24:33.930Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() STARTED
[2022-06-02T14:24:37.800Z] 
[2022-06-02T14:24:37.800Z] ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress() PASSED
[2022-06-02T14:24:37.800Z] 
[2022-06-02T14:24:37.800Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() STARTED
[2022-06-02T14:24:41.565Z] 
[2022-06-02T14:24:41.565Z] ControllerIntegrationTest > 
testNoTopicIdPersistsThroughControllerReelection() PASSED
[2022-06-02T14:24:41.565Z] 
[2022-06-02T14:24:41.565Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() STARTED
[2022-06-02T14:24:44.367Z] 
[2022-06-02T14:24:44.367Z] ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown() PASSED
[2022-06-02T14:24:44.367Z] 
[2022-06-02T14:24:44.367Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
STARTED
[2022-06-02T14:24:46.128Z] 
[2022-06-02T14:24:46.128Z] ControllerIntegrationTest > testTopicIdsAreAdded() 
PASSED
[2022-06-02T14:24:46.128Z] 
[2022-06-02T14:24:46.128Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() STARTED
[2022-06-02T14:24:50.197Z] 
[2022-06-02T14:24:50.197Z] ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica() PASSED
[2022-06-02T14:24:50.197Z] 
[2022-06-02T14:24:50.197Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() STARTED
[2022-06-02T14:24:56.314Z] 
[2022-06-02T14:24:56.314Z] ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline() PASSED
[2022-06-02T14:24:56.314Z] 
[2022-06-02T14:24:56.314Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() STARTED
[2022-06-02T14:24:59.915Z] 
[2022-06-02T14:24:59.915Z] ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled() PASSED
[2022-06-02T14:24:59.915Z] 
[2022-06-02T14:24:59.915Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() STARTED
[2022-06-02T14:25:03.194Z] 
[2022-06-02T14:25:03.194Z] ControllerIntegrationTest > 
testTopicIdMigrationAndHandlingWithOlderVersion() PASSED
[2022-06-02T14:25:03.194Z] 
[2022-06-02T14:25:03.194Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() STARTED
[2022-06-02T14:25:08.155Z] 
[2022-06-02T14:25:08.155Z] ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica() PASSED
[2022-06-02T14:25:08.155Z] 
[2022-06-02T14:25:08.155Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() STARTED
[2022-06-02T14:25:12.491Z] 
[2022-06-02T14:25:12.491Z] ControllerIntegrationTest > 
testPartitionReassignmentToBrokerWithOfflineLogDir() PASSED
[2022-06-02T14:25:12.491Z] 
[2022-06-02T14:25:12.491Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() STARTED
[2022-06-02T14:25:18.020Z] 
[2022-06-02T14:25:18.020Z] ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica() PASSED
[2022-06-02T14:25:18.020Z] 
[2022-06-02T14:25:18.020Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() STARTED
[2022-06-02T14:25:20.192Z] 
[2022-06-02T14:25:20.192Z] ControllerIntegrationTest > 
testMetadataPropagationOnControlPlane() PASSED
[2022-06-02T14:25:20.192Z] 
[2022-06-02T14:25:20.192Z] ControllerIntegrationTest > 
testControllerFeatureZNodeSetupWhenFeatureVersioningIsEnabledWithNonExistingFeatureZNode()
 STARTED
[2022-06-02T14:25:22.465Z] 
[2022-06-02T14:25:22.465Z] ControllerIntegrationTest > 

[jira] [Created] (KAFKA-13955) Fix failing KRaftClusterTest tests

[Luke Chen (Jira)]

Luke Chen created KAFKA-13955:
-

 Summary: Fix failing KRaftClusterTest tests
 Key: KAFKA-13955
 URL: https://issues.apache.org/jira/browse/KAFKA-13955
 Project: Kafka
  Issue Type: Test
Reporter: Luke Chen
Assignee: dengziming


Tests are failing with timeout exception

java.util.concurrent.TimeoutException: 
testCreateClusterAndPerformReassignment() timed out after 120 seconds

 

Failing tests:

Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testIncrementalAlterConfigs()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testSetLog4jConfigurations()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testLegacyAlterConfigs()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testCreateClusterAndPerformReassignment()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testUnregisterBroker()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testCreateClusterAndCreateAndManyTopics()
Build / JDK 8 and Scala 2.12 / 
kafka.server.KRaftClusterTest.testCreateClusterAndCreateListDeleteTopic()



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Luke Chen]

Hi Jun,

Thanks for the comment.

Yes, I've tried to work on this way to track the number of remaining
segments, but it will change the design in UnifiedLog, so I only track the
logs number.
Currently, we will load all segments and recover those segments if needed
"during creating UnifiedLog instance". And also get the log offsets here

.
That is, if we want to get all segments to be recovered before running log
recovery, we need to break the logic in UnifiedLog, to create a partial
UnifiedLog instance, and add more info to it later, which I think is just
making the codes more complicated.


Thank you.
Luke



On Thu, May 26, 2022 at 2:57 AM Jun Rao  wrote:

> Hi, Luke,
>
> Thanks for the KIP. Just one comment.
>
> 10. For kafka.log:type=LogManager,name=remainingLogsToRecovery, could we
> instead track the number of remaining segments? This monitors the progress
> at a finer granularity and is also consistent with the thread level metric.
>
> Thanks,
>
> Jun
>
> On Wed, May 25, 2022 at 7:47 AM Tom Bentley  wrote:
>
> > Thanks Luke! LGTM.
> >
> > On Sun, 22 May 2022 at 05:18, Luke Chen  wrote:
> >
> > > Hi Tom and Raman,
> > >
> > > Thanks for your comments.
> > >
> > > > 1. There's not a JIRA for this KIP (or the JIRA link needs updating).
> > > 2. Similarly the link to this discussion thread needs updating.
> > > > Please update the links to JIRA and the discussion thread.
> > >
> > > Yes, thanks for the reminder. I've updated the KIP.
> > >
> > > > 3. I wonder whether we need to keep these metrics (with value 0) once
> > the
> > > broker enters the running state. Do you see it as valuable? A benefit
> of
> > > removing the metrics would be a reduction on storage required for
> metric
> > > stores which are recording these metrics.
> > >
> > > Yes, removing the metrics after log recovery completed is a good idea.
> > > Updated the KIP.
> > >
> > > > 4. I think the KIP's public interfaces section could be a bit
> clearer.
> > > Previous KIPs which added metrics usually used a table, with the MBean
> > > name, metric type and description. SeeKIP-551 for example (or KIP-748,
> > > KIP-608). Similarly you could use a table in the proposed changes
> section
> > > rather than describing the tree you'd see in an MBean console.
> > >
> > > Good point! Updated the KIP to use a table to list the MBean name,
> metric
> > > type and descriptions.
> > >
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, May 20, 2022 at 9:13 AM Raman Verma
>  > >
> > > wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > The change is useful and simple. Thanks.
> > > > Please update the links to JIRA and the discussion thread.
> > > >
> > > > Best Regards,
> > > > Raman Verma
> > > >
> > > > On Thu, May 19, 2022 at 8:57 AM Tom Bentley 
> > wrote:
> > > > >
> > > > > Hi Luke,
> > > > >
> > > > > Thanks for the KIP. I think the idea makes sense and would provide
> > > useful
> > > > > observability of log recovery. I have a few comments.
> > > > >
> > > > > 1. There's not a JIRA for this KIP (or the JIRA link needs
> updating).
> > > > > 2. Similarly the link to this discussion thread needs updating.
> > > > > 3. I wonder whether we need to keep these metrics (with value 0)
> once
> > > the
> > > > > broker enters the running state. Do you see it as valuable? A
> benefit
> > > of
> > > > > removing the metrics would be a reduction on storage required for
> > > metric
> > > > > stores which are recording these metrics.
> > > > > 4. I think the KIP's public interfaces section could be a bit
> > clearer.
> > > > > Previous KIPs which added metrics usually used a table, with the
> > MBean
> > > > > name, metric type and description. SeeKIP-551 for example (or
> > KIP-748,
> > > > > KIP-608). Similarly you could use a table in the proposed changes
> > > section
> > > > > rather than describing the tree you'd see in an MBean console.
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Tom
> > > > >
> > > > > On Wed, 11 May 2022 at 09:08, Luke Chen  wrote:
> > > > >
> > > > > > > And if people start using RemainingLogs and RemainingSegments
> and
> > > > then
> > > > > > REALLY FEEL like they need RemainingBytes, then we can always add
> > it
> > > > in the
> > > > > > future.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > Thanks James!
> > > > > > Luke
> > > > > >
> > > > > > On Wed, May 11, 2022 at 3:57 PM James Cheng <
> wushuja...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Luke,
> > > > > > >
> > > > > > > Thanks for the detailed explanation. I agree that the current
> > > > proposal of
> > > > > > > RemainingLogs and RemainingSegments will greatly improve the
> > > > situation,
> > > > > > and
> > > > > > > that we can go ahead with the KIP as is.
> > > > > > >
> > > > > > > If RemainingBytes were straight-forward to implement, then I’d
> > like
> > > > to
> > > > > > > have it. But we can live without it for now. And if 

[jira] [Resolved] (KAFKA-13845) Add support for reading KRaft snapshots in kafka-dump-log

[dengziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dengziming resolved KAFKA-13845.

Resolution: Fixed

> Add support for reading KRaft snapshots in kafka-dump-log
> -
>
> Key: KAFKA-13845
> URL: https://issues.apache.org/jira/browse/KAFKA-13845
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Arthur
>Assignee: dengziming
>Priority: Minor
>  Labels: kip-500
>
> Even though the metadata snapshots use the same format as log segments, the 
> kafka-dump-log tool (DumpLogSegments.scala) does not support the file 
> extension or file name pattern.
> For example, a metadata snapshot will be named like:
> {code:java}
> __cluster_metadata-0/0004-01.checkpoint{code}
> whereas regular log segments (including the metadata log) are named like:
> {code:java}
> __cluster_metadata-0/.log {code}
>  
> We need to enhance the tool to support snapshots. 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-846: Processor-level Streams metrics for records/bytes Producedd

[Sophie Blee-Goldman]

Thanks all!

This KIP has passed with two +1 (non-binding) votes from Sagar and Walker,
and
five +1 (binding) votes from Bruno, Bill, John, Guozhang, and myself.

Cheers,
Sophie

On Wed, Jun 1, 2022 at 9:53 PM Guozhang Wang  wrote:

> +1 (binding)
>
> Thanks Sophie!
>
> On Tue, May 31, 2022 at 11:28 AM Walker Carlson
>  wrote:
>
> > +1 non binding
> >
> > On Tue, May 31, 2022 at 12:19 PM John Roesler 
> wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, May 30, 2022, at 13:00, Bill Bejeck wrote:
> > > > +1 (binding)
> > > >
> > > > -Bill
> > > >
> > > > On Mon, May 30, 2022 at 4:49 AM Sagar 
> > wrote:
> > > >
> > > >> +1 (non-binding).
> > > >>
> > > >> Thanks!
> > > >> Sagar.
> > > >>
> > > >> On Mon, May 30, 2022 at 1:11 PM Bruno Cadonna 
> > > wrote:
> > > >>
> > > >> > +1 (binding)
> > > >> >
> > > >> > Thanks!
> > > >> > Bruno
> > > >> >
> > > >> > On 30.05.22 09:36, Sophie Blee-Goldman wrote:
> > > >> > > Hey all,
> > > >> > >
> > > >> > >   I'd like to kick off the voting thread for the KIP I proposed
> to
> > > add
> > > >> > > processor-level "bytes/records produced" metrics to Kafka
> Streams.
> > > >> > >
> > > >> > > Thanks!
> > > >> > >
> > > >> > > KIP-846: Task-level Streams metrics for bytes/records Produced
> > > >> > > <
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211886093
> > > >> > >
> > > >> > >
> > > >> > > Cheers,
> > > >> > > Sophie
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
>
> --
> -- Guozhang
>

[jira] [Resolved] (KAFKA-12657) Flaky Tests BlockingConnectorTest.testWorkerRestartWithBlockInConnectorStop

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-12657.
---
Resolution: Fixed

> Flaky Tests BlockingConnectorTest.testWorkerRestartWithBlockInConnectorStop
> ---
>
> Key: KAFKA-12657
> URL: https://issues.apache.org/jira/browse/KAFKA-12657
> Project: Kafka
>  Issue Type: Test
>  Components: KafkaConnect
>Reporter: Matthias J. Sax
>Assignee: Chris Egerton
>Priority: Critical
>  Labels: flaky-test
>
> [https://github.com/apache/kafka/pull/10506/checks?check_run_id=2327377745]
> {quote} {{org.opentest4j.AssertionFailedError: Condition not met within 
> timeout 6. Worker did not complete startup in time ==> expected:  
> but was: 
>   at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
>   at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
>   at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:193)
>   at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:319)
>   at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:367)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:316)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:300)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:290)
>   at 
> org.apache.kafka.connect.integration.BlockingConnectorTest.setup(BlockingConnectorTest.java:133)}}
> {quote}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13954) Consumer Topic Offset reset to 0，baseoffset is wrong，Consumers do not consume

[huangyoutai (Jira)]

huangyoutai created KAFKA-13954:
---

 Summary: Consumer Topic Offset reset to 0，baseoffset is 
wrong，Consumers do not consume
 Key: KAFKA-13954
 URL: https://issues.apache.org/jira/browse/KAFKA-13954
 Project: Kafka
  Issue Type: Bug
Reporter: huangyoutai
 Attachments: access1-0.log, consumer-offset.log, 
企业微信截图_16541590838472.png, 企业微信截图_1654159127326.png

Has anyone encountered this problem? kafka is standalone. The producer is using 
librdkafka



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-845: HasField predicate for kafka connect

[Kumud Kumar Srivatsava Tirupati]

Hi Chris,
Thanks for your comments.

I am totally aligned with your comment on nested field names which include
dots. I will incorporate the same based on how the KIP-821 discussion goes
(maybe this parser could be a utility that can be reused in other areas as
well).

But, I still believe this logic of predicate checking shouldn't be made a
part of each individual SMT. After all, that is what the predicates are for
right?
If you see, the current predicates that we have are:
org.apache.kafka.connect.transforms.predicates.TopicNameMatches
org.apache.kafka.connect.transforms.predicates.HasHeaderKey
org.apache.kafka.connect.transforms.predicates.RecordIsTombstone
They only allow you filter/transform based on the metadata of the record.

`HasField` predicate is something very similar but is more powerful because
it allows users to skip transformations at SMT level or drop them from the
transformation chain altogether using the current `
org.apache.kafka.connect.transforms.Filter`. So, the use cases are not just
limited to skipping some SMTs.
If this makes sense, I should probably add this and give an example in the
KIP as well.

*---*
*Thanks and Regards,*
*Kumud Kumar Srivatsava Tirupati*


On Wed, 1 Jun 2022 at 07:09, Chris Egerton  wrote:

> Hi Kumud,
>
> Thanks for the KIP. I'm a little bit skeptical about the necessity for this
> predicate but I think we may be able to satisfy your requirements with a
> slightly different approach. The motivation section deals largely with
> skipping the invocation of SMTs that expect a certain field to be present
> in a record, and will fail if it is not present. This seems like a
> reasonable use case; even when your data has a fixed schema, optional
> fields are still possible, and preventing SMTs from being used on optional
> fields seems unnecessarily restrictive. In fact, it seems like such a
> reasonable use case that I wonder if it'd be worth investing in new
> SMT-level properties to handle this case, instead of requiring users to
> configure a predicate separately alongside them? Something like an
> "on.field.missing" property with options of "skip", "fail", and/or "warn"
> could do the trick.
>
> It's also worth noting that the proposed syntax for the "field.path"
> property in the HasField predicate would make it impossible to refer to
> field names that have dots in them. It hasn't been finalized yet but if we
> do end up going this route, we should probably use the same syntax that's
> agreed upon for KIP-821 [1].
>
> [1] -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-821%3A+Connect+Transforms+support+for+nested+structures
>
> Cheers,
>
> Chris
>
> On Thu, May 26, 2022 at 5:28 AM Sagar  wrote:
>
> > Hi Kumud,
> >
> > Thanks for that. I don't have any other comments at this point on the
> KIP.
> > LGTM overall.
> >
> > Thanks!
> > Sagar.
> >
> > On Wed, May 25, 2022 at 5:14 PM Sagar  wrote:
> >
> > > Thanks for the KIP Kumud.
> > >
> > > Can you please add a couple of examples on how this would behave with
> > > different combinations. I think that way it would be easier to
> > understand.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > > On Wed, May 25, 2022 at 4:59 PM Kumud Kumar Srivatsava Tirupati <
> > > kumudkumartirup...@gmail.com> wrote:
> > >
> > >> Hi all,
> > >> I have written a KIP for having a new HasField predicate for kafka
> > connect
> > >> transforms and would like to start a discussion. Please share your
> > >> thoughts
> > >> on the same.
> > >>
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-845%3A+%27HasField%27+predicate+for+kafka+connect
> > >>
> > >> *---*
> > >> *Thanks and Regards,*
> > >> *Kumud Kumar Srivatsava Tirupati*
> > >>
> > >
> >
>