Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1789

[Apache Jenkins Server]

See 

Re: Test failures

[Sagar]

Hi Greg,

The fix for 14929 has already been included as part of this pr :
https://github.com/apache/kafka/pull/13594

I can create a separate pr just for that flaky test if needed. Let me know .

Sagar.

On Sat, 22 Apr 2023 at 3:20 AM, Greg Harris 
wrote:

> Hey all,
>
> We just landed a fix for https://issues.apache.org/jira/browse/KAFKA-14905
> which was causing all of
> those MirrorConnectorsWithCustomForwardingAdminIntegrationTest failures,
> and we will monitor the build for any re-occurrances.
> Unfortunately we discovered another test flake that was introduced recently
> but that should have a straightforward resolution:
> https://issues.apache.org/jira/browse/KAFKA-14929
> Thanks Ismael for merging a fix for
> https://issues.apache.org/jira/browse/KAFKA-8115 but it appears that there
> is still more investigation needed there, as the test is still failing
> occasionally.
>
> Thanks,
> Greg
>
> On Fri, Apr 14, 2023 at 12:18 PM Ismael Juma  wrote:
>
> > Thanks Greg! I really appreciate the help.
> >
> > Ismael
> >
> > On Fri, Apr 14, 2023 at 12:08 PM Greg Harris
>  > >
> > wrote:
> >
> > > Hey Ismael,
> > >
> > > We're working to stabilize the Connect/MM2 tests with the following
> > issues:
> > >
> > > * https://issues.apache.org/jira/browse/KAFKA-14905 to address
> > > MirrorConectorsWithCustomForwardingAdminIntegrationTest with tentative
> > open
> > > PR
> > > * https://issues.apache.org/jira/browse/KAFKA-14901 to address
> > > ExactlyOnceSourceIntegrationTest caused by an (apparent) kafka bug
> > >
> > > Looking at the other failures in Connect/MM2 for that build in
> > particular,
> > > it appears that most of them include Embedded Kafka not coming
> > up/shutting
> > > down cleanly:
> > > * MirrorConnectorsIntegrationBaseTest
> > > * MirrorConnectorsIntegrationExactlyOnceTest
> > > * MirrorConnectorsIntegrationSSLTest
> > > * ConnectorClientPolicyIntegrationTest
> > > * ConnectorTopicsIntegrationTest
> > > * ExactlyOnceSourceIntegrationTest
> > > * OffsetsApiIntegrationTest
> > > I'll start investigating these failures to learn more.
> > >
> > > I also have a few older flaky test improvements that have not been
> > reviewed
> > > or merged yet:
> > > * https://issues.apache.org/jira/browse/KAFKA-8115 to
> > > address CoordinatorTest (reappeared in the linked build)
> > > * https://issues.apache.org/jira/browse/KAFKA-14345 to address
> > > (Dynamic)ConnectionQuotaTest
> > >
> > > It also appears that the flakey AuthorizerTest should be addressed by
> > > https://github.com/apache/kafka/pull/13543 which is on trunk now but
> > > wasn't
> > > at the time of the above run.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Fri, Apr 14, 2023 at 10:25 AM Ismael Juma 
> wrote:
> > >
> > > > Thanks Justine!
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Apr 14, 2023 at 9:53 AM Justine Olshan
> > > > 
> > > > wrote:
> > > >
> > > > > Hey Ismael -- thanks for bringing this up.
> > > > > I've filed https://issues.apache.org/jira/browse/KAFKA-14904 and
> am
> > > > > working
> > > > > on it now.
> > > > >
> > > > > I hope the other tests get fixed soon.
> > > > >
> > > > > On Fri, Apr 14, 2023 at 6:47 AM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > It looks like there are a lot of test failures in the master
> > branch.
> > > I
> > > > > > don't know which commits introduced them, but can you please
> check
> > if
> > > > > > commit(s) you merged or contributed are the reason and fix it
> asap?
> > > If
> > > > > it's
> > > > > > easy to fix the tests, let's do that - otherwise we should revert
> > the
> > > > > > faulty commit. And let's please be more careful going forward
> when
> > it
> > > > > comes
> > > > > > to the PRs we merge.
> > > > > >
> > > > > > An example from one of the builds, but there are many like this:
> > > > > >
> > > > > > Build / JDK 17 and Scala 2.13 /
> > > > > > kafka.api.TransactionsBounceTest.testWithGroupMetadata()
> > > > > > Build / JDK 17 and Scala 2.13 /
> > > > > > kafka.api.TransactionsBounceTest.testWithGroupId()
> > > > > > Build / JDK 17 and Scala 2.13 /
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAllowAllAccess(String).quorum=kraft
> > > > > > Build / JDK 17 and Scala 2.13 /
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAuthorizeThrowsOnNonLiteralResource(String).quorum=kraft
> > > > > > Build / JDK 8 and Scala 2.12 /
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitionsMixed(String).quorum=zk
> > > > > > Build / JDK 8 and Scala 2.12 /
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAllowAllAccess(String).quorum=kraft
> > > > > > Build / JDK 8 and Scala 2.12 /
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testDeleteAclOnPrefixedResource(String).quorum=kraft

[GitHub] [kafka-site] atu-sharm commented on pull request #506: KAFKA-14925: Hosting JS files locally

[via GitHub]

atu-sharm commented on PR #506:
URL: https://github.com/apache/kafka-site/pull/506#issuecomment-1518434152

   @mimaison can you review please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] atu-sharm opened a new pull request, #506: KAFKA-14925: Hosting JS files locally

[via GitHub]

atu-sharm opened a new pull request, #506:
URL: https://github.com/apache/kafka-site/pull/506

   Adding one JS file which is recommended to be hosted along with the website 
so that we don't have a dependency on external resources


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #115

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1788

[Apache Jenkins Server]

See 

Re: Test failures

[Greg Harris]

Hey all,

We just landed a fix for https://issues.apache.org/jira/browse/KAFKA-14905
which was causing all of
those MirrorConnectorsWithCustomForwardingAdminIntegrationTest failures,
and we will monitor the build for any re-occurrances.
Unfortunately we discovered another test flake that was introduced recently
but that should have a straightforward resolution:
https://issues.apache.org/jira/browse/KAFKA-14929
Thanks Ismael for merging a fix for
https://issues.apache.org/jira/browse/KAFKA-8115 but it appears that there
is still more investigation needed there, as the test is still failing
occasionally.

Thanks,
Greg

On Fri, Apr 14, 2023 at 12:18 PM Ismael Juma  wrote:

> Thanks Greg! I really appreciate the help.
>
> Ismael
>
> On Fri, Apr 14, 2023 at 12:08 PM Greg Harris  >
> wrote:
>
> > Hey Ismael,
> >
> > We're working to stabilize the Connect/MM2 tests with the following
> issues:
> >
> > * https://issues.apache.org/jira/browse/KAFKA-14905 to address
> > MirrorConectorsWithCustomForwardingAdminIntegrationTest with tentative
> open
> > PR
> > * https://issues.apache.org/jira/browse/KAFKA-14901 to address
> > ExactlyOnceSourceIntegrationTest caused by an (apparent) kafka bug
> >
> > Looking at the other failures in Connect/MM2 for that build in
> particular,
> > it appears that most of them include Embedded Kafka not coming
> up/shutting
> > down cleanly:
> > * MirrorConnectorsIntegrationBaseTest
> > * MirrorConnectorsIntegrationExactlyOnceTest
> > * MirrorConnectorsIntegrationSSLTest
> > * ConnectorClientPolicyIntegrationTest
> > * ConnectorTopicsIntegrationTest
> > * ExactlyOnceSourceIntegrationTest
> > * OffsetsApiIntegrationTest
> > I'll start investigating these failures to learn more.
> >
> > I also have a few older flaky test improvements that have not been
> reviewed
> > or merged yet:
> > * https://issues.apache.org/jira/browse/KAFKA-8115 to
> > address CoordinatorTest (reappeared in the linked build)
> > * https://issues.apache.org/jira/browse/KAFKA-14345 to address
> > (Dynamic)ConnectionQuotaTest
> >
> > It also appears that the flakey AuthorizerTest should be addressed by
> > https://github.com/apache/kafka/pull/13543 which is on trunk now but
> > wasn't
> > at the time of the above run.
> >
> > Thanks,
> > Greg
> >
> > On Fri, Apr 14, 2023 at 10:25 AM Ismael Juma  wrote:
> >
> > > Thanks Justine!
> > >
> > > Ismael
> > >
> > > On Fri, Apr 14, 2023 at 9:53 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Hey Ismael -- thanks for bringing this up.
> > > > I've filed https://issues.apache.org/jira/browse/KAFKA-14904 and am
> > > > working
> > > > on it now.
> > > >
> > > > I hope the other tests get fixed soon.
> > > >
> > > > On Fri, Apr 14, 2023 at 6:47 AM Ismael Juma 
> wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > It looks like there are a lot of test failures in the master
> branch.
> > I
> > > > > don't know which commits introduced them, but can you please check
> if
> > > > > commit(s) you merged or contributed are the reason and fix it asap?
> > If
> > > > it's
> > > > > easy to fix the tests, let's do that - otherwise we should revert
> the
> > > > > faulty commit. And let's please be more careful going forward when
> it
> > > > comes
> > > > > to the PRs we merge.
> > > > >
> > > > > An example from one of the builds, but there are many like this:
> > > > >
> > > > > Build / JDK 17 and Scala 2.13 /
> > > > > kafka.api.TransactionsBounceTest.testWithGroupMetadata()
> > > > > Build / JDK 17 and Scala 2.13 /
> > > > > kafka.api.TransactionsBounceTest.testWithGroupId()
> > > > > Build / JDK 17 and Scala 2.13 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAllowAllAccess(String).quorum=kraft
> > > > > Build / JDK 17 and Scala 2.13 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAuthorizeThrowsOnNonLiteralResource(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.TopicCommandIntegrationTest.testDescribeUnderMinIsrPartitionsMixed(String).quorum=zk
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAllowAllAccess(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testDeleteAclOnPrefixedResource(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testNoAclFoundOverride(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testAuthorizeThrowsOnNonLiteralResource(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.security.authorizer.AuthorizerTest.testGetAclsPrincipal(String).quorum=kraft
> > > > > Build / JDK 8 and Scala 2.12 /
> > > > >
> > > 

[jira] [Created] (KAFKA-14929) Flaky KafkaStatusBackingStoreFormatTest#putTopicStateRetriableFailure

[Greg Harris (Jira)]

Greg Harris created KAFKA-14929:
---

 Summary: Flaky 
KafkaStatusBackingStoreFormatTest#putTopicStateRetriableFailure
 Key: KAFKA-14929
 URL: https://issues.apache.org/jira/browse/KAFKA-14929
 Project: Kafka
  Issue Type: Test
  Components: KafkaConnect
Reporter: Greg Harris
 Fix For: 3.5.0


This test recently started flaky-failing with the following stack trace:
{noformat}
org.mockito.exceptions.verification.TooFewActualInvocations: 
kafkaBasedLog.send(, , );Wanted 2 times:-> at 
org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:376)But was 
1 time:-> at 
org.apache.kafka.connect.storage.KafkaStatusBackingStore.sendTopicStatus(KafkaStatusBackingStore.java:315)
   at 
app//org.apache.kafka.connect.util.KafkaBasedLog.send(KafkaBasedLog.java:376)   
 at 
app//org.apache.kafka.connect.storage.KafkaStatusBackingStoreFormatTest.putTopicStateRetriableFailure(KafkaStatusBackingStoreFormatTest.java:219)
at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
 Method) at 
java.base@11.0.16.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at 
java.base@11.0.16.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at 
java.base@11.0.16.1/java.lang.reflect.Method.invoke(Method.java:566){noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Jason Gustafson]

Hey Colin,

The KIP makes sense overall. Nice to clarify the contract between clients
and the controllers. The use of `DirectToKRaftControllerQuorum` will help
prevent misconfiguration. In fact, I wonder if we can return a fatal error
instead of NOT_CONTROLLER so that the client would immediately fail. For
example, could we use INVALID_REQUEST or something like that? Either that
or we need to make sure clients treat NOT_CONTROLLER as a fatal error.
Without that, it would probably get picked up with default retry logic and
the user might not see the problem.

I also wonder if we can relax the requirements on the Metadata request so
that we can use it to list topics and partition state (e.g. URPs).  It
would be useful to query the controllers as the metadata source of truth
when we suspect that the broker metadata may have diverged.

Thanks,
Jason

On Thu, Apr 20, 2023 at 5:53 PM Colin McCabe  wrote:

> On Wed, Apr 19, 2023, at 20:56, Philip Nee wrote:
> > Hey Colin,
> >
> > I still need to finish reading and understanding the KIP, but I have a
> > couple of comments despite being ignorant of most of the KRaft stuff.
> > (Sorry!)
> >
> > Firstly, does it make sense to create an extension of the current
> > AdminClient only to handle these specific KRaft use cases? It seems
> > cumbersome to have two sets of bootstrap configurations to make the
> > AdminClient generic enough to handle these specific cases, instead, maybe
> > it is more obvious (to me) to just extend the AdminClient. What I'm
> > thinking is KraftAdminClient which continuously uses *bootstrap.servers*,
> > but make this class only serves the Kraft controllers APIs.
>
> Hi Philip,
>
> Thanks for taking a look.
>
> We would not want to create a new Admin client class in order to
> communicate directly with the controllers. The RPCs accepted by the
> controllers have the same format as the those accepted by the brokers.
> There is no difference in what is sent over the wire or the data structures
> that are used in the client.
>
> I know you mentioned you haven't had time to read all the KRaft stuff (and
> there is a lot, I understand). But this is one area that would probably be
> clarified if you were able to read at least KIP-500 and KIP-590. RPCs sent
> to the broker are forwarded to the controller (mostly) without modification.
>
> >
> > Secondly, if we want to continue with the design, I'm not yet sure why we
> > can't continue using the *bootstrap.servers*? I assume when the client
> gets
> > the metadata, it should know who it is talking to. I'm just reconsidering
> > your alternative again.
> >
> > A bad idea: Why don't we continue using *bootstrap.servers* but have a
> > separated config like *kraft.controller* = true/false. I feel like most
> > users might not know what is a controller and causes some mistakes down
> the
> > road.
> >
>
> Well, you called it a bad idea, and I can't help but agree! :)
>
> I think "the user might not know what a controller is" is a good sign that
> they shouldn't be interacting with the controller. Like many AdminClient
> APIs, this one is intended for use by administrators only. Most users
> indeed should not need to know what a controller is or interact with it
> directly. If they do interact it should be very clear that they are doing
> so. The --controller-bootstrap flag makes it very clear, as does the
> separate configuration.
>
> Let me give an example of the kind of problems that arise if you want to
> reuse bootstrap.servers for this purpose.
>
> If the user grasb a 3.4 Kafka AdminClient and set bootstrap.servers to a
> set of controller servers, and set direct.to.controller to true, the
> unknown (in 3.4) configuration will be ignored, and a normal metadata
> request will be sent without the direct to controller flag. In that case it
> will give back an error. Confusing, right?
>
> Using controller.servers clarifies this situation because the 3.4 client
> will not support that config, and will complain about the lack of
> bootstrap.servers.
>
> Actually, the situation could get even more confusing than what I
> described since some older preproduction versions of the KRaft controller
> did implement the METADATA RPC. So if you send them a METADATA request
> without any special information, it's not clear what you'll get.  Indeed,
> it would be dependent on the client version and the controller version.
>
> The bottom line is that reusing the bootstrap.servers configuration here
> is not a good idea.
>
> best,
> Colin
>
> > Thanks,
> > P
> >
> > On Wed, Apr 19, 2023 at 2:18 PM Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> I wrote a short KIP about allowing AdminClient to talk directly with the
> >> KRaft controller quorum. Check it out here:
> >>
> >> https://cwiki.apache.org/confluence/x/Owo0Dw
> >>
> >> best,
> >> Colin
> >>
>

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Hey Jason,

Thanks again. I've updated the KIP.

P

On Fri, Apr 21, 2023 at 9:56 AM Jason Gustafson 
wrote:

> Hey Philip, that sounds good to me.
>
> -Jason
>
>
>
> On Thu, Apr 20, 2023 at 1:20 PM Philip Nee  wrote:
>
> > Hey Jason,
> >
> > Thanks for the response. I agree with your suggestions. Just to clarify,
> we
> > want the timeout, bootstrap.resolve.timeout.ms to only bound the DNS
> > resolution time, and we can throw a fatal, BootstrapResolutionException
> (so
> > not connection exception anymore) afterward.
> >
> > I think that aligns with the goal of this KIP.
> >
> > P
> >
> > On Thu, Apr 20, 2023 at 9:23 AM Jason Gustafson
>  > >
> > wrote:
> >
> > > Hey Philip,
> > >
> > > Yeah, I see your point. Here is the challenge I'm considering. Today,
> the
> > > client does not forward connection-related errors to the application.
> It
> > is
> > > debatable whether it should, but that is the behavior that applications
> > > expect today. The only case of a fatal error is DNS resolution and the
> > > application must handle it because they cannot even construct the
> client.
> > > Once we have this KIP, we will introduce a separate fatal failure
> > mechanism
> > > through the `BootstrapConnectionException`. And we will use it not only
> > for
> > > DNS failures, but general bootstrap connection failures . The problem
> is
> > > that existing applications will not know that this should be treated
> as a
> > > fatal error. So they may continue to retry. Does it help in this
> > situation
> > > to poison the client so that it cannot be used? I'm not sure. Perhaps
> it
> > > would reduce the risk a bit if we change `
> > bootstrap.connection.timeout.ms`
> > > to `bootstrap.resolve.timeout.ms` or something like that? Then we
> retain
> > > the current behavior if DNS succeeds, but connections fail. I know I'm
> > the
> > > one that suggested generalizing the configuration, but I feel some
> > > hesitation after thinking about it more.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Wed, Apr 19, 2023 at 8:10 PM Philip Nee 
> wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > Thanks for your review.  I think if we make it a retriable error,
> does
> > it
> > > > make sense to have a configurable timeout still? as we expect the
> user
> > to
> > > > continue to retry anyway.
> > > >
> > > > I'm considering the case of bad configuration. If the user retries
> the
> > > > error, then we rely on the error/warning to alert the user.  In this
> > > case,
> > > > maybe we continue using the proposed behavior, i.e. warn on each poll
> > > after
> > > > the timeout period.
> > > >
> > > > If you agree that a configuration is needed, maybe we can call this
> > > > *bootstrap.auto.retry.ms
> > > >  *instead, to indicate a
> configurable
> > > > period of automatic retry. What do you think?
> > > >
> > > > Cheers,
> > > > P
> > > >
> > > > On Wed, Apr 19, 2023 at 7:17 PM Jason Gustafson
> > >  > > > >
> > > > wrote:
> > > >
> > > > > Hey Phillip,
> > > > >
> > > > > The KIP looks good. 5 minutes seems like a reasonable tradeoff. I
> do
> > > > wonder
> > > > > if it is necessary to treat bootstrap timeout as a fatal error
> > though.
> > > It
> > > > > seems possible that the exception might be caught by handlers in
> > > existing
> > > > > applications which may not expect that the client needs to be
> > > restarted.
> > > > > Perhaps it would be safer to make it retriable? As long as the
> > > > application
> > > > > continues trying to use the client, we could continue trying to
> reach
> > > the
> > > > > bootstrap servers perhaps? That would be closer to behavior today
> > which
> > > > > only treats DNS resolution failures as fatal. What do you think?
> > > > >
> > > > > Best,
> > > > > Jason
> > > > >
> > > > > On Mon, Apr 10, 2023 at 1:53 PM Philip Nee 
> > > wrote:
> > > > >
> > > > > > Thanks, everyone: I'm starting a vote today.  Here's the recap
> for
> > > some
> > > > > of
> > > > > > the questions:
> > > > > >
> > > > > > John: I changed the proposal to throw a non-retriable exception
> > after
> > > > the
> > > > > > timeout elapses. I feel it might be necessary to poison the
> client
> > > > after
> > > > > > retry expires, as it might indicate a real issue.
> > > > > > Ismael: The proposal is to add a configuration for the retry and
> it
> > > > will
> > > > > > throw a non-retriable exception after the time expires.
> > > > > > Chris: Addressed some unclarity that you mentioned, and a new API
> > > won't
> > > > > be
> > > > > > introduced in this KIP.  Maybe up for future discussion.
> > > > > > Jason: I'm proposing adding a timeout config and a bootstrap
> > > exception
> > > > > per
> > > > > > your suggestion.
> > > > > > Kirk: I'm proposing throwing a non-retriable exception in the
> > network
> > > > > > client. See previous comment.
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 27, 2023 at 9:36 AM Chris Egerton
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > 

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Josep Prat]

Hi Mickael,

Greg Harris managed to fix a flaky test in
https://github.com/apache/kafka/pull/13575, I cherry-picked it to the 3.5
(and 3.4) branch. I updated the Jira to reflect that is now fixed on 3.5.0
as well as 3.6.0.
Let me know if I forgot anything.

Best,

On Fri, Apr 21, 2023 at 3:44 PM Mickael Maison 
wrote:

> Hi,
>
> Just a quick reminder that code freeze is next week.
> We still have 27 JIRAs targeting 3.5 [0] including quite a few bugs
> and flaky test issues opened recently. If you have time, take one of
> these items or help with the reviews.
>
> I'll send another update next once we've entered code freeze.
>
> 0:
> https://issues.apache.org/jira/browse/KAFKA-13421?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.5.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC
>
> Thanks,
> Mickael
>
> On Thu, Apr 20, 2023 at 9:14 PM Mickael Maison 
> wrote:
> >
> > Hi Ron,
> >
> > Yes feel free to merge that fix. Thanks for letting me know!
> >
> > Mickael
> >
> > On Thu, Apr 20, 2023 at 8:15 PM Ron Dagostino  wrote:
> > >
> > > Hi Mickael.  I would like to merge
> > > https://github.com/apache/kafka/pull/13532 (KAFKA-14887: No shutdown
> > > for ZK session expiration in feature processing) to the 3.5 branch.
> > > It is a very small and focused fix that can cause unexpected broker
> > > shutdowns when there is instability in the connectivity to ZooKeeper.
> > > The risk is very low.
> > >
> > > Ron
> > >
> > >
> > > On Tue, Apr 18, 2023 at 9:57 AM Mickael Maison <
> mickael.mai...@gmail.com> wrote:
> > > >
> > > > Hi David,
> > > >
> > > > Thanks for the update. I've marked KAFKA-14869 as fixed in 3.5.0, I
> > > > guess you'll only resolve this ticket once you merge the backports to
> > > > earlier branches. The ticket will have to be resolved to run the
> > > > release but that should leave you enough time.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Apr 18, 2023 at 3:42 PM David Jacot
>  wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > FYI - I just merged the two PRs for KIP-915 to trunk/3.5. We are
> all good.
> > > > >
> > > > > Cheers,
> > > > > David
> > > > >
> > > > > On Mon, Apr 17, 2023 at 5:10 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Chris,
> > > > > >
> > > > > > I was looking at that just now! As you said, the PRs merged
> provide
> > > > > > some functionality so I think it's fine to deliver the KIP
> across 2
> > > > > > releases.
> > > > > > I left a comment in
> https://issues.apache.org/jira/browse/KAFKA-14876
> > > > > > to document what's in 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > > On Mon, Apr 17, 2023 at 5:05 PM Chris Egerton
> 
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > It looks like we missed the feature freeze cutoff for part but
> not all of
> > > > > > > KIP-875 [1]. The features we've been able to merge so far are
> the new
> > > > > > > STOPPED state for connectors [2] and the API for reading
> offsets [3]. The
> > > > > > > features we have not been able to merge yet are the APIs for
> altering and
> > > > > > > resetting offsets.
> > > > > > >
> > > > > > > The already-merged features are useful on their own and I
> believe it
> > > > > > should
> > > > > > > be acceptable to release them separately from the
> not-yet-merged ones,
> > > > > > but
> > > > > > > wanted to double-check with you that it's alright to split
> this KIP
> > > > > > across
> > > > > > > two or more releases, starting with 3.5.0.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > Chris
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > > > > [2] - https://github.com/apache/kafka/pull/13424
> > > > > > > [3] - https://github.com/apache/kafka/pull/13434
> > > > > > >
> > > > > > > On Fri, Apr 14, 2023 at 10:13 AM Matthias J. Sax <
> mj...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks a lot!
> > > > > > > >
> > > > > > > > On 4/14/23 5:32 AM, Mickael Maison wrote:
> > > > > > > > > Hi Matthias,
> > > > > > > > >
> > > > > > > > > I merged the PR before cutting the 3.5 branch.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mickael
> > > > > > > > >
> > > > > > > > > On Fri, Apr 14, 2023 at 2:31 PM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Hi David,
> > > > > > > > >>
> > > > > > > > >> I've created the 3.5 branch. Feel free to cherry pick
> these 2
> > > > > > commits
> > > > > > > > >> when they are ready.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> Mickael
> > > > > > > > >>
> > > > > > > > >> On Fri, Apr 14, 2023 at 11:23 AM Satish Duggana
> > > > > > > > >>  wrote:
> > > > > > > > 

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Matthew Benedict de Detrich]

Congratulations Mickael and many thanks to Jun for the work over the years.

On Fri, Apr 21, 2023 at 5:10 PM Jun Rao  wrote:

> Hi, everyone,
>
> After more than 10 years, I am stepping down as the PMC chair of Apache
> Kafka. We now have a new chair Mickael Maison, who has been a PMC member
> since 2020. I plan to continue to contribute to Apache Kafka myself.
>
> Congratulations, Mickael!
>
> Jun
>


-- 

Matthew de Detrich

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491603708037

*w:* aiven.io *e:* matthew.dedetr...@aiven.io

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #170

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 502629 lines...]
[2023-04-21T17:47:13.674Z] [INFO] 

[2023-04-21T17:47:13.674Z] [INFO] Reactor Summary for Kafka Streams :: 
Quickstart 3.3.3-SNAPSHOT:
[2023-04-21T17:47:13.674Z] [INFO] 
[2023-04-21T17:47:13.674Z] [INFO] Kafka Streams :: Quickstart 
 SUCCESS [  3.283 s]
[2023-04-21T17:47:13.674Z] [INFO] streams-quickstart-java 
 SUCCESS [  1.592 s]
[2023-04-21T17:47:13.674Z] [INFO] 

[2023-04-21T17:47:13.674Z] [INFO] BUILD SUCCESS
[2023-04-21T17:47:13.674Z] [INFO] 

[2023-04-21T17:47:13.674Z] [INFO] Total time:  5.307 s
[2023-04-21T17:47:13.674Z] [INFO] Finished at: 2023-04-21T17:47:13Z
[2023-04-21T17:47:13.674Z] [INFO] 

[Pipeline] dir
[2023-04-21T17:47:14.182Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
[2023-04-21T17:47:17.110Z] + + mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java 
-DarchetypeVersion=3.3.3-SNAPSHOTecho -DgroupId=streams.examples Y 
-DartifactId=streams.examples
[2023-04-21T17:47:17.110Z]  -Dversion=0.1 -Dpackage=myapps
[2023-04-21T17:47:18.296Z] [INFO] Scanning for projects...
[2023-04-21T17:47:18.296Z] [INFO] 
[2023-04-21T17:47:18.296Z] [INFO] --< 
org.apache.maven:standalone-pom >---
[2023-04-21T17:47:18.296Z] [INFO] Building Maven Stub Project (No POM) 1
[2023-04-21T17:47:18.296Z] [INFO] [ pom 
]-
[2023-04-21T17:47:18.296Z] [INFO] 
[2023-04-21T17:47:18.296Z] [INFO] >>> maven-archetype-plugin:3.2.1:generate 
(default-cli) > generate-sources @ standalone-pom >>>
[2023-04-21T17:47:18.296Z] [INFO] 
[2023-04-21T17:47:18.296Z] [INFO] <<< maven-archetype-plugin:3.2.1:generate 
(default-cli) < generate-sources @ standalone-pom <<<
[2023-04-21T17:47:18.296Z] [INFO] 
[2023-04-21T17:47:18.296Z] [INFO] 
[2023-04-21T17:47:18.296Z] [INFO] --- maven-archetype-plugin:3.2.1:generate 
(default-cli) @ standalone-pom ---
[2023-04-21T17:47:20.379Z] [INFO] Generating project in Interactive mode
[2023-04-21T17:47:20.895Z] [WARNING] Archetype not found in any catalog. 
Falling back to central repository.
[2023-04-21T17:47:20.895Z] [WARNING] Add a repository with id 'archetype' in 
your settings.xml if archetype's repository is elsewhere.
[2023-04-21T17:47:20.895Z] [INFO] Using property: groupId = streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Using property: artifactId = streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Using property: version = 0.1
[2023-04-21T17:47:20.895Z] [INFO] Using property: package = myapps
[2023-04-21T17:47:20.895Z] Confirm properties configuration:
[2023-04-21T17:47:20.895Z] groupId: streams.examples
[2023-04-21T17:47:20.895Z] artifactId: streams.examples
[2023-04-21T17:47:20.895Z] version: 0.1
[2023-04-21T17:47:20.895Z] package: myapps
[2023-04-21T17:47:20.895Z]  Y: : [INFO] 

[2023-04-21T17:47:20.895Z] [INFO] Using following parameters for creating 
project from Archetype: streams-quickstart-java:3.3.3-SNAPSHOT
[2023-04-21T17:47:20.895Z] [INFO] 

[2023-04-21T17:47:20.895Z] [INFO] Parameter: groupId, Value: streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Parameter: artifactId, Value: streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Parameter: version, Value: 0.1
[2023-04-21T17:47:20.895Z] [INFO] Parameter: package, Value: myapps
[2023-04-21T17:47:20.895Z] [INFO] Parameter: packageInPathFormat, Value: myapps
[2023-04-21T17:47:20.895Z] [INFO] Parameter: package, Value: myapps
[2023-04-21T17:47:20.895Z] [INFO] Parameter: version, Value: 0.1
[2023-04-21T17:47:20.895Z] [INFO] Parameter: groupId, Value: streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Parameter: artifactId, Value: streams.examples
[2023-04-21T17:47:20.895Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/workspace/Kafka_kafka_3.3/streams/quickstart/test-streams-archetype/streams.examples
[2023-04-21T17:47:20.895Z] [INFO] 

[2023-04-21T17:47:20.895Z] [INFO] BUILD SUCCESS
[2023-04-21T17:47:20.895Z] [INFO] 

[2023-04-21T17:47:20.895Z] [INFO] Total time:  3.013 s
[2023-04-21T17:47:20.895Z] [INFO] Finished at: 2023-04-21T17:47:20Z
[2023-04-21T17:47:20.895Z] [INFO] 

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Matthias J. Sax]

Congrats Mickael!

And thanks a lot for taking on this additional task! Glad to have you!


-Matthias

On 4/21/23 9:40 AM, Viktor Somogyi-Vass wrote:

Jun, thank you for all your hard work! Also, congrats Mickael, it is very
well deserved :)

Best,
Viktor

On Fri, Apr 21, 2023, 18:15 Adam Bellemare  wrote:


Thank you for all your hard work Jun - that's a decade-long legacy!
And congratulations to you Mickael!

On Fri, Apr 21, 2023 at 11:20 AM Josep Prat 
wrote:


Thanks Jun for your work as Chair all these years!
Congratulations Mickael!

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:


Hi, everyone,

After more than 10 years, I am stepping down as the PMC chair of Apache
Kafka. We now have a new chair Mickael Maison, who has been a PMC

member

since 2020. I plan to continue to contribute to Apache Kafka myself.

Congratulations, Mickael!

Jun

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1787

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Jason Gustafson]

Hey Philip, that sounds good to me.

-Jason



On Thu, Apr 20, 2023 at 1:20 PM Philip Nee  wrote:

> Hey Jason,
>
> Thanks for the response. I agree with your suggestions. Just to clarify, we
> want the timeout, bootstrap.resolve.timeout.ms to only bound the DNS
> resolution time, and we can throw a fatal, BootstrapResolutionException (so
> not connection exception anymore) afterward.
>
> I think that aligns with the goal of this KIP.
>
> P
>
> On Thu, Apr 20, 2023 at 9:23 AM Jason Gustafson  >
> wrote:
>
> > Hey Philip,
> >
> > Yeah, I see your point. Here is the challenge I'm considering. Today, the
> > client does not forward connection-related errors to the application. It
> is
> > debatable whether it should, but that is the behavior that applications
> > expect today. The only case of a fatal error is DNS resolution and the
> > application must handle it because they cannot even construct the client.
> > Once we have this KIP, we will introduce a separate fatal failure
> mechanism
> > through the `BootstrapConnectionException`. And we will use it not only
> for
> > DNS failures, but general bootstrap connection failures . The problem is
> > that existing applications will not know that this should be treated as a
> > fatal error. So they may continue to retry. Does it help in this
> situation
> > to poison the client so that it cannot be used? I'm not sure. Perhaps it
> > would reduce the risk a bit if we change `
> bootstrap.connection.timeout.ms`
> > to `bootstrap.resolve.timeout.ms` or something like that? Then we retain
> > the current behavior if DNS succeeds, but connections fail. I know I'm
> the
> > one that suggested generalizing the configuration, but I feel some
> > hesitation after thinking about it more.
> >
> > Thanks,
> > Jason
> >
> > On Wed, Apr 19, 2023 at 8:10 PM Philip Nee  wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for your review.  I think if we make it a retriable error, does
> it
> > > make sense to have a configurable timeout still? as we expect the user
> to
> > > continue to retry anyway.
> > >
> > > I'm considering the case of bad configuration. If the user retries the
> > > error, then we rely on the error/warning to alert the user.  In this
> > case,
> > > maybe we continue using the proposed behavior, i.e. warn on each poll
> > after
> > > the timeout period.
> > >
> > > If you agree that a configuration is needed, maybe we can call this
> > > *bootstrap.auto.retry.ms
> > >  *instead, to indicate a configurable
> > > period of automatic retry. What do you think?
> > >
> > > Cheers,
> > > P
> > >
> > > On Wed, Apr 19, 2023 at 7:17 PM Jason Gustafson
> >  > > >
> > > wrote:
> > >
> > > > Hey Phillip,
> > > >
> > > > The KIP looks good. 5 minutes seems like a reasonable tradeoff. I do
> > > wonder
> > > > if it is necessary to treat bootstrap timeout as a fatal error
> though.
> > It
> > > > seems possible that the exception might be caught by handlers in
> > existing
> > > > applications which may not expect that the client needs to be
> > restarted.
> > > > Perhaps it would be safer to make it retriable? As long as the
> > > application
> > > > continues trying to use the client, we could continue trying to reach
> > the
> > > > bootstrap servers perhaps? That would be closer to behavior today
> which
> > > > only treats DNS resolution failures as fatal. What do you think?
> > > >
> > > > Best,
> > > > Jason
> > > >
> > > > On Mon, Apr 10, 2023 at 1:53 PM Philip Nee 
> > wrote:
> > > >
> > > > > Thanks, everyone: I'm starting a vote today.  Here's the recap for
> > some
> > > > of
> > > > > the questions:
> > > > >
> > > > > John: I changed the proposal to throw a non-retriable exception
> after
> > > the
> > > > > timeout elapses. I feel it might be necessary to poison the client
> > > after
> > > > > retry expires, as it might indicate a real issue.
> > > > > Ismael: The proposal is to add a configuration for the retry and it
> > > will
> > > > > throw a non-retriable exception after the time expires.
> > > > > Chris: Addressed some unclarity that you mentioned, and a new API
> > won't
> > > > be
> > > > > introduced in this KIP.  Maybe up for future discussion.
> > > > > Jason: I'm proposing adding a timeout config and a bootstrap
> > exception
> > > > per
> > > > > your suggestion.
> > > > > Kirk: I'm proposing throwing a non-retriable exception in the
> network
> > > > > client. See previous comment.
> > > > >
> > > > >
> > > > > On Mon, Feb 27, 2023 at 9:36 AM Chris Egerton
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Philip,
> > > > > >
> > > > > > Yeah,  "DNS resolution should occur..." seems like a better fit.
> 
> > > > > >
> > > > > > One other question I have is whether we should expose some kind
> of
> > > > public
> > > > > > API for performing preflight validation of the bootstrap URLs. If
> > we
> > > > > change
> > > > > > the behavior of a client configured with a silly typo (e.g.,
> > > > > > "loclahost 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #114

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 520538 lines...]
[2023-04-21T16:25:28.694Z] 
[2023-04-21T16:25:28.694Z] > Task :connect:runtime:integrationTest
[2023-04-21T16:25:28.694Z] 
[2023-04-21T16:25:28.694Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testCreateInternalTopicsWithFewerReplicasThanBrokers PASSED
[2023-04-21T16:25:28.694Z] 
[2023-04-21T16:25:28.694Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testCreateInternalTopicsWithDefaultSettings STARTED
[2023-04-21T16:25:52.638Z] 
[2023-04-21T16:25:52.638Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testCreateInternalTopicsWithDefaultSettings PASSED
[2023-04-21T16:25:52.638Z] 
[2023-04-21T16:25:52.638Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
 STARTED
[2023-04-21T16:25:52.638Z] 
[2023-04-21T16:25:52.638Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
 PASSED
[2023-04-21T16:25:52.638Z] 
[2023-04-21T16:25:52.638Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testFailToCreateInternalTopicsWithMoreReplicasThanBrokers STARTED
[2023-04-21T16:25:53.669Z] 
[2023-04-21T16:25:53.669Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testFailToCreateInternalTopicsWithMoreReplicasThanBrokers PASSED
[2023-04-21T16:25:53.669Z] 
[2023-04-21T16:25:53.669Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi STARTED
[2023-04-21T16:25:57.185Z] 
[2023-04-21T16:25:57.185Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi PASSED
[2023-04-21T16:25:57.185Z] 
[2023-04-21T16:25:57.185Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured STARTED
[2023-04-21T16:25:57.185Z] 
[2023-04-21T16:25:57.185Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured SKIPPED
[2023-04-21T16:25:57.185Z] 
[2023-04-21T16:25:57.185Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testTopicsAreCreatedWhenTopicCreationIsEnabled STARTED
[2023-04-21T16:26:05.413Z] 
[2023-04-21T16:26:05.413Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 121 > 
MirrorConnectorsWithCustomForwardingAdminIntegrationTest > 
testOneWayReplicationWithFrequentOffsetSyncs() PASSED
[2023-04-21T16:26:05.413Z] 
[2023-04-21T16:26:05.413Z] Gradle Test Run :connect:mirror:integrationTest > 
Gradle Test Executor 121 > 
MirrorConnectorsWithCustomForwardingAdminIntegrationTest > 
testReplicationIsCreatingTopicsUsingProvidedForwardingAdmin() STARTED
[2023-04-21T16:26:07.015Z] 
[2023-04-21T16:26:07.015Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testTopicsAreCreatedWhenTopicCreationIsEnabled PASSED
[2023-04-21T16:26:07.015Z] 
[2023-04-21T16:26:07.015Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testTopicsAreCreatedWhenAutoCreateTopicsIsEnabledAtTheBroker STARTED
[2023-04-21T16:26:20.760Z] 
[2023-04-21T16:26:20.760Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testTopicsAreCreatedWhenAutoCreateTopicsIsEnabledAtTheBroker PASSED
[2023-04-21T16:26:20.760Z] 
[2023-04-21T16:26:20.760Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 125 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testSwitchingToTopicCreationEnabled STARTED

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Viktor Somogyi-Vass]

Jun, thank you for all your hard work! Also, congrats Mickael, it is very
well deserved :)

Best,
Viktor

On Fri, Apr 21, 2023, 18:15 Adam Bellemare  wrote:

> Thank you for all your hard work Jun - that's a decade-long legacy!
> And congratulations to you Mickael!
>
> On Fri, Apr 21, 2023 at 11:20 AM Josep Prat 
> wrote:
>
> > Thanks Jun for your work as Chair all these years!
> > Congratulations Mickael!
> >
> > Best,
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:
> >
> > > Hi, everyone,
> > >
> > > After more than 10 years, I am stepping down as the PMC chair of Apache
> > > Kafka. We now have a new chair Mickael Maison, who has been a PMC
> member
> > > since 2020. I plan to continue to contribute to Apache Kafka myself.
> > >
> > > Congratulations, Mickael!
> > >
> > > Jun
> > >
> >
>

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Adam Bellemare]

Thank you for all your hard work Jun - that's a decade-long legacy!
And congratulations to you Mickael!

On Fri, Apr 21, 2023 at 11:20 AM Josep Prat 
wrote:

> Thanks Jun for your work as Chair all these years!
> Congratulations Mickael!
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:
>
> > Hi, everyone,
> >
> > After more than 10 years, I am stepping down as the PMC chair of Apache
> > Kafka. We now have a new chair Mickael Maison, who has been a PMC member
> > since 2020. I plan to continue to contribute to Apache Kafka myself.
> >
> > Congratulations, Mickael!
> >
> > Jun
> >
>

[GitHub] [kafka-site] machi1990 closed pull request #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

machi1990 closed pull request #504: KAFKA-10883: link to get started in the top 
navigation header
URL: https://github.com/apache/kafka-site/pull/504


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] machi1990 commented on pull request #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

machi1990 commented on PR #504:
URL: https://github.com/apache/kafka-site/pull/504#issuecomment-1517993660

   Thanks @mimaison I've edited the JIRA to capture what has been discussed 
here. I'll close this PR as it is also mentioned in the JIRA that once the page 
is redisigned/finished off it has to be linked.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Josep Prat]

Thanks Jun for your work as Chair all these years!
Congratulations Mickael!

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Fri, Apr 21, 2023, 17:10 Jun Rao  wrote:

> Hi, everyone,
>
> After more than 10 years, I am stepping down as the PMC chair of Apache
> Kafka. We now have a new chair Mickael Maison, who has been a PMC member
> since 2020. I plan to continue to contribute to Apache Kafka myself.
>
> Congratulations, Mickael!
>
> Jun
>

Re: [ANNOUNCE] New PMC chair: Mickael Maison

[Federico Valeri]

Congrats Mickael, very well deserved, and thanks Jun for covering this
role so far.

On Fri, Apr 21, 2023 at 5:09 PM Jun Rao  wrote:
>
> Hi, everyone,
>
> After more than 10 years, I am stepping down as the PMC chair of Apache
> Kafka. We now have a new chair Mickael Maison, who has been a PMC member
> since 2020. I plan to continue to contribute to Apache Kafka myself.
>
> Congratulations, Mickael!
>
> Jun

[ANNOUNCE] New PMC chair: Mickael Maison

[Jun Rao]

Hi, everyone,

After more than 10 years, I am stepping down as the PMC chair of Apache
Kafka. We now have a new chair Mickael Maison, who has been a PMC member
since 2020. I plan to continue to contribute to Apache Kafka myself.

Congratulations, Mickael!

Jun

[GitHub] [kafka-site] junrao merged pull request #505: change VP of Kafka

[via GitHub]

junrao merged PR #505:
URL: https://github.com/apache/kafka-site/pull/505


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mimaison commented on pull request #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

mimaison commented on PR #504:
URL: https://github.com/apache/kafka-site/pull/504#issuecomment-1517955010

   Yes feel free to edit the JIRA. I've granted you permissions so you should 
be able to self assign tickets if you want.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on pull request #505: change VP of Kafka

[via GitHub]

junrao commented on PR #505:
URL: https://github.com/apache/kafka-site/pull/505#issuecomment-1517951266

   cc @mimaison 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] machi1990 commented on pull request #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

machi1990 commented on PR #504:
URL: https://github.com/apache/kafka-site/pull/504#issuecomment-1517920328

   Hi @mimaison thanks for the review. You are right. I landed on this issue 
when I was trying to understand how the website is generated. I didn't 
appreciate that the other buttons don't work: when I navigated to the 
`/get-started` link and seeing the "Introduction" content, I wrongly assumed 
that all worked fine and we could quickly add the link and have the quick win. 
However, it seems that the "get-started" page hasn't been revisited since 
https://github.com/apache/kafka-site/pull/269 and it has always been the case 
that's it's not linked in the top navigation menu. I wonder if there are plans 
to rework the content, @scott-confluent  any thoughts ?
   
   > So before enabling it, I think we should properly finish it. WDYT?
   
   I'd agree, thanks. In this case, I think the JIRA 
https://issues.apache.org/jira/browse/KAFKA-10883 could be edited in the way it 
reflects this sentiment i.e the page needs to be properly finished and then 
linked in the top navigation menu. WDYT? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-14928) Metrics collection contends on lock with log cleaning

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-14928:


 Summary: Metrics collection contends on lock with log cleaning
 Key: KAFKA-14928
 URL: https://issues.apache.org/jira/browse/KAFKA-14928
 Project: Kafka
  Issue Type: Bug
Reporter: Divij Vaidya
Assignee: Divij Vaidya
 Fix For: 3.6.0


In LogCleanerManager.scala, calculation of a metric requires a lock [1]. This 
same lock is required by core log cleaner functionality such as 
"grabFilthiestCompactedLog". This might lead to a situation where metric 
calculation holding the lock for an extended period of time may affect the core 
functionality of log cleaning.

This outcome of this task is to prevent expensive metric calculation from 
blocking log cleaning/compaction activity.

[1] 
https://github.com/apache/kafka/blob/dd63d88ac3ea7a9a55a6dacf9c5473e939322a55/core/src/main/scala/kafka/log/LogCleanerManager.scala#L102



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #146

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 295380 lines...]
[2023-04-21T13:49:39.509Z] > Task :connect:json:jar UP-TO-DATE
[2023-04-21T13:49:39.509Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-04-21T13:49:39.509Z] > Task :connect:api:javadocJar
[2023-04-21T13:49:39.509Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-04-21T13:49:39.509Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-04-21T13:49:39.509Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-04-21T13:49:39.509Z] > Task :connect:api:testJar
[2023-04-21T13:49:39.509Z] > Task :connect:json:publishToMavenLocal
[2023-04-21T13:49:39.509Z] > Task :connect:api:testSrcJar
[2023-04-21T13:49:39.509Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-04-21T13:49:39.509Z] > Task :connect:api:publishToMavenLocal
[2023-04-21T13:49:41.307Z] > Task :streams:javadoc
[2023-04-21T13:49:41.307Z] > Task :streams:javadocJar
[2023-04-21T13:49:41.307Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-04-21T13:49:41.307Z] > Task :streams:testClasses UP-TO-DATE
[2023-04-21T13:49:41.307Z] > Task :streams:testJar
[2023-04-21T13:49:41.307Z] 
[2023-04-21T13:49:41.307Z] > Task :clients:javadoc
[2023-04-21T13:49:41.307Z] 
/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2023-04-21T13:49:41.307Z] 
[2023-04-21T13:49:41.307Z] > Task :streams:testSrcJar
[2023-04-21T13:49:41.307Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-04-21T13:49:41.307Z] > Task :streams:publishToMavenLocal
[2023-04-21T13:49:42.831Z] 
[2023-04-21T13:49:42.831Z] > Task :clients:javadoc
[2023-04-21T13:49:42.831Z] 1 warning
[2023-04-21T13:49:42.831Z] 
[2023-04-21T13:49:42.831Z] > Task :clients:javadocJar
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] > Task :clients:srcJar
[2023-04-21T13:49:44.667Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2023-04-21T13:49:44.667Z]   - Gradle detected a problem with the following 
location: '/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/generated/java'. 
Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] > Task :clients:testJar
[2023-04-21T13:49:44.667Z] > Task :clients:testSrcJar
[2023-04-21T13:49:44.667Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-04-21T13:49:44.667Z] > Task :clients:publishToMavenLocal
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure correctness.
[2023-04-21T13:49:44.667Z] Please consult deprecation warnings for more details.
[2023-04-21T13:49:44.667Z] 
[2023-04-21T13:49:44.667Z] BUILD SUCCESSFUL in 28s
[2023-04-21T13:49:44.667Z] 77 actionable tasks: 34 executed, 43 up-to-date
[Pipeline] sh
[2023-04-21T13:49:49.861Z] + grep ^version= gradle.properties
[2023-04-21T13:49:49.861Z] + cut -d= -f 2
[Pipeline] dir
[2023-04-21T13:49:50.553Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-04-21T13:49:53.466Z] + mvn clean install -Dgpg.skip
[2023-04-21T13:49:53.466Z] [INFO] Scanning for projects...
[2023-04-21T13:49:55.277Z] [INFO] 

[2023-04-21T13:49:55.277Z] [INFO] Reactor Build Order:
[2023-04-21T13:49:55.277Z] [INFO] 
[2023-04-21T13:49:55.277Z] [INFO] Kafka Streams :: Quickstart   
 [pom]
[2023-04-21T13:49:55.277Z] [INFO] streams-quickstart-java   
 [maven-archetype]
[2023-04-21T13:49:55.277Z] [INFO] 
[2023-04-21T13:49:55.277Z] [INFO] < 
org.apache.kafka:streams-quickstart >-
[2023-04-21T13:49:55.277Z] [INFO] Building Kafka Streams :: Quickstart 
3.1.3-SNAPSHOT[1/2]
[2023-04-21T13:49:55.277Z] [INFO] 

[jira] [Created] (KAFKA-14927) Dynamic configs not validated when using kafka-configs and --add-config-file

[Justin Daines (Jira)]

Justin Daines created KAFKA-14927:
-

 Summary: Dynamic configs not validated when using kafka-configs 
and --add-config-file
 Key: KAFKA-14927
 URL: https://issues.apache.org/jira/browse/KAFKA-14927
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 3.3.2
Reporter: Justin Daines


Using {{kafka-configs}} should validate dynamic configurations before applying. 
It is possible to send a file with invalid configurations. 

For example a file containing the following:

 
{code:java}
{
  "routes": {
    "crn:///kafka=*": {
      "management": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events-denied"
      },
      "describe": {
        "allowed": "",
        "denied": "confluent-audit-log-events-denied"
      },
      "authentication": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events-denied-authn"
      },
      "authorize": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events-denied-authz"
      },
      "interbroker": {
        "allowed": "",
        "denied": ""
      }
    },
    "crn:///kafka=*/group=*": {
      "consume": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events"
      }
    },
    "crn:///kafka=*/topic=*": {
      "produce": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events"
      },
      "consume": {
        "allowed": "confluent-audit-log-events_audit",
        "denied": "confluent-audit-log-events"
      }
    }
  },
  "destinations": {
    "topics": {
      "confluent-audit-log-events": {
        "retention_ms": 777600
      },
      "confluent-audit-log-events-denied": {
        "retention_ms": 777600
      },
      "confluent-audit-log-events-denied-authn": {
        "retention_ms": 777600
      },
      "confluent-audit-log-events-denied-authz": {
        "retention_ms": 777600
      },
      "confluent-audit-log-events_audit": {
        "retention_ms": 777600
      }
    }
  },
  "default_topics": {
    "allowed": "confluent-audit-log-events_audit",
    "denied": "confluent-audit-log-events"
  },
  "excluded_principals": [
    "User:schemaregistryUser",
    "User:ANONYMOUS",
    "User:appSA",
    "User:admin",
    "User:connectAdmin",
    "User:connectorSubmitter",
    "User:connectorSA",
    "User:schemaregistryUser",
    "User:ksqlDBAdmin",
    "User:ksqlDBUser",
    "User:controlCenterAndKsqlDBServer",
    "User:controlcenterAdmin",
    "User:restAdmin",
    "User:appSA",
    "User:clientListen",
    "User:superUser"
  ]
} {code}
 

 
{code:java}
kafka-configs --bootstrap-server $KAFKA_BOOTSTRAP --entity-type brokers 
--entity-default --alter --add-config-file audit-log.json {code}
 

Yields the following dynamic configs:

 
{code:java}
Default configs for brokers in the cluster are:
  "destinations"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"destinations"=null}
  "confluent-audit-log-events-denied-authn"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events-denied-authn"=null}
  "routes"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"routes"=null}
  "User=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"User=null}
  },=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:},=null}
  "excluded_principals"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"excluded_principals"=null}
  "confluent-audit-log-events_audit"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events_audit"=null}
  "authorize"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"authorize"=null}
  "default_topics"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"default_topics"=null}
  "topics"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"topics"=null}
  ]=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:]=null}
  "interbroker"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"interbroker"=null}
  "produce"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"produce"=null}
  "denied"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"denied"=null}
  "confluent-audit-log-events-denied"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events-denied"=null}
  "confluent-audit-log-events"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events"=null}
  "crn=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"crn=null}
  "management"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"management"=null}
  "describe"=null sensitive=true 
synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"describe"=null}
  

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Mickael Maison]

Hi,

Just a quick reminder that code freeze is next week.
We still have 27 JIRAs targeting 3.5 [0] including quite a few bugs
and flaky test issues opened recently. If you have time, take one of
these items or help with the reviews.

I'll send another update next once we've entered code freeze.

0: 
https://issues.apache.org/jira/browse/KAFKA-13421?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%203.5.0%20AND%20status%20not%20in%20(resolved%2C%20closed)%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC

Thanks,
Mickael

On Thu, Apr 20, 2023 at 9:14 PM Mickael Maison  wrote:
>
> Hi Ron,
>
> Yes feel free to merge that fix. Thanks for letting me know!
>
> Mickael
>
> On Thu, Apr 20, 2023 at 8:15 PM Ron Dagostino  wrote:
> >
> > Hi Mickael.  I would like to merge
> > https://github.com/apache/kafka/pull/13532 (KAFKA-14887: No shutdown
> > for ZK session expiration in feature processing) to the 3.5 branch.
> > It is a very small and focused fix that can cause unexpected broker
> > shutdowns when there is instability in the connectivity to ZooKeeper.
> > The risk is very low.
> >
> > Ron
> >
> >
> > On Tue, Apr 18, 2023 at 9:57 AM Mickael Maison  
> > wrote:
> > >
> > > Hi David,
> > >
> > > Thanks for the update. I've marked KAFKA-14869 as fixed in 3.5.0, I
> > > guess you'll only resolve this ticket once you merge the backports to
> > > earlier branches. The ticket will have to be resolved to run the
> > > release but that should leave you enough time.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Tue, Apr 18, 2023 at 3:42 PM David Jacot  
> > > wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > FYI - I just merged the two PRs for KIP-915 to trunk/3.5. We are all 
> > > > good.
> > > >
> > > > Cheers,
> > > > David
> > > >
> > > > On Mon, Apr 17, 2023 at 5:10 PM Mickael Maison 
> > > > 
> > > > wrote:
> > > >
> > > > > Hi Chris,
> > > > >
> > > > > I was looking at that just now! As you said, the PRs merged provide
> > > > > some functionality so I think it's fine to deliver the KIP across 2
> > > > > releases.
> > > > > I left a comment in https://issues.apache.org/jira/browse/KAFKA-14876
> > > > > to document what's in 3.5.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Mon, Apr 17, 2023 at 5:05 PM Chris Egerton 
> > > > > 
> > > > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > It looks like we missed the feature freeze cutoff for part but not 
> > > > > > all of
> > > > > > KIP-875 [1]. The features we've been able to merge so far are the 
> > > > > > new
> > > > > > STOPPED state for connectors [2] and the API for reading offsets 
> > > > > > [3]. The
> > > > > > features we have not been able to merge yet are the APIs for 
> > > > > > altering and
> > > > > > resetting offsets.
> > > > > >
> > > > > > The already-merged features are useful on their own and I believe it
> > > > > should
> > > > > > be acceptable to release them separately from the not-yet-merged 
> > > > > > ones,
> > > > > but
> > > > > > wanted to double-check with you that it's alright to split this KIP
> > > > > across
> > > > > > two or more releases, starting with 3.5.0.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Chris
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect
> > > > > > [2] - https://github.com/apache/kafka/pull/13424
> > > > > > [3] - https://github.com/apache/kafka/pull/13434
> > > > > >
> > > > > > On Fri, Apr 14, 2023 at 10:13 AM Matthias J. Sax 
> > > > > wrote:
> > > > > >
> > > > > > > Thanks a lot!
> > > > > > >
> > > > > > > On 4/14/23 5:32 AM, Mickael Maison wrote:
> > > > > > > > Hi Matthias,
> > > > > > > >
> > > > > > > > I merged the PR before cutting the 3.5 branch.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > > On Fri, Apr 14, 2023 at 2:31 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hi David,
> > > > > > > >>
> > > > > > > >> I've created the 3.5 branch. Feel free to cherry pick these 2
> > > > > commits
> > > > > > > >> when they are ready.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Mickael
> > > > > > > >>
> > > > > > > >> On Fri, Apr 14, 2023 at 11:23 AM Satish Duggana
> > > > > > > >>  wrote:
> > > > > > > >>>
> > > > > > > >>> Thanks Luke for helping with the reviews and adding a few 
> > > > > > > >>> tests in
> > > > > a
> > > > > > > >>> couple of PRs.
> > > > > > > >>>
> > > > > > > >>> Hi Mickael,
> > > > > > > >>> I raised 3 PRs recently for tiered storage, one is merged. The
> > > > > other 2
> > > > > > > >>> PRs are in the critical path of non-tiered storage changes 
> > > > > > > >>> also.
> > > > > > > >>> Especially, in consumer fetch and retention cleanup paths. 
> > > > > > > >>> These
> > > > > need
> > > > > > > >>> to be thoroughly reviewed and avoid any 

[GitHub] [kafka-site] mimaison commented on pull request #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

mimaison commented on PR #504:
URL: https://github.com/apache/kafka-site/pull/504#issuecomment-1517838156

   Hi @machi1990, thanks for the PR!
   
   I agree that the behavior of the menu is a bit strange. The issue is that 
the "get started" page does not seem finished, so I wonder if it's the reason 
it's currently not reachable. 
   
   For example, it says "All you need to know to get up and running with Apache 
Kafka can be found here. **Etc. Etc.**". Also the buttons don't seem to work.
   
   So before enabling it, I think we should properly finish it. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] machi1990 opened a new pull request, #504: KAFKA-10883: link to get started in the top navigation header

[via GitHub]

machi1990 opened a new pull request, #504:
URL: https://github.com/apache/kafka-site/pull/504

   Closes https://issues.apache.org/jira/browse/KAFKA-10883
   
   Hi @mimaison can you've a look at this? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Viktor Somogyi-Vass]

Hi Daniel,

I think this is a useful addition, it helps resolving issues and
escalations, and improves overall traceability.
Changing the logging context may imply the risk of making certain log
parsers unable to work on new logs. As I see we by default disable this
feature which solves this problem, however I also think that by disabling
it by default it isn't much of a help because users may not know about this
configuration and would not benefit from these when they face problems. So
overall I'd like to go with default=true and wanted to put this out here
for the community to discuss whether it's a problem.
Also, what was the reasoning behind rejecting the second alternative? As I
see that would be a viable option and maybe a bit more idiomatic to the
logging framework.

A minor note: please update the JIRA link in the KIP to point to the right
one.

Best,
Viktor

On Thu, Apr 13, 2023 at 2:19 PM Dániel Urbán  wrote:

> Hi everyone,
>
> I would like to bump this thread. I think this would be very useful for any
> MM2 users, as the current logs with certain architectures (e.g. fan-out)
> are impossible to decipher.
> I already submitted a PR to demonstrate the proposed solution:
> https://github.com/apache/kafka/pull/13475
>
> Thanks for your comments in advance,
> Daniel
>
> Dániel Urbán  ezt írta (időpont: 2023. márc. 30.,
> Cs, 18:24):
>
> > Hello everyone,
> >
> > I would like to kick off a discussion about KIP-916:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-916%3A+MM2+distributed+mode+flow+log+context
> >
> > The KIP aims to enhance the diagnostic information for MM2 distributed
> > mode. MM2 relies on multiple Connect worker instances nested in a single
> > process. In Connect, Connector names are guaranteed to be unique in a
> > single process, but in MM2, this is not true. Because of this, the
> > diagnostics provided by Connect (client.ids, log context) do not ensure
> > that logs are distinguishable for different flows (Connect workers)
> inside
> > an MM2 process.
> >
> > Thanks for all you input in advance,
> > Daniel
> >
>

Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener

[Dániel Urbán]

Thanks for the comments Viktor.
1. My original motivation was IdentityReplicationPolicy based monitoring.
The current MirrorClient implementation cannot list the replica topics of
the target cluster. I think relying on the topic-partition level metrics is
a complex solution. Instead, I would like to make it simple to collect all
the replicated topics of a flow, without relying on the name of the topics.
Then, I simply tried to generalize the approach.
2. Checkpoint metrics are reported per (group, topic, partition), it means
that there is no metric associated with a group. If a filter picks up a
group, but the group doesn't have committed offsets for any of the
replicated partitions, there is no metric to be eagerly registered. This is
a difference between how topic replication and group checkpointing works -
empty topics are still picked up for partition creation and to consume from
them. Groups are only picked up if they have committed offsets already.
3. Not exactly sure what is the added value of listing all
topic-partitions, but that information is available where the filtering
happens. For groups, we don't have anything else besides the group name, so
we cannot really provide more info at that point without significantly
changing the refresh group logic.

Thanks,
Daniel

Viktor Somogyi-Vass  ezt írta
(időpont: 2023. ápr. 21., P, 11:43):

> Hi all,
>
> A couple of comments:
> 1) Regarding the motivation: is the motivation simply monitoring related or
> are there any other reasons to this?
> 2) Can we change monitoring to be identical to filters, so that what is
> actively filtered, we monitor exactly those topics and groups? (So group
> metrics aren't added lazily when a checkpoint is created but when the
> filter is changed.)
> 3) Not sure if we want to widen the scope but since these are interfaces
> I'd use TopicPartition and some kind of GroupDescription classes (not sure
> if the latter exists) instead of Strings. If later on we'll need extra
> properties for these then it can be added on easier.
>
> Best,
> Viktor
>
> On Wed, Apr 19, 2023 at 1:42 PM Dániel Urbán 
> wrote:
>
> > I wouldn't really include a non-existent group (same as we don't care
> about
> > a non-existent topic), that doesn't really matter.
> > I think having an existing group which doesn't have an offset to
> checkpoint
> > is equivalent to a topic having no records to replicate from the
> monitoring
> > perspective.
> >
> > I think the precise way to put it is to monitor the topics and groups
> > picked up by the filtering logic of MM2. "The list currently replicated"
> is
> > not a good definition, as an empty topic would still be interesting for
> > monitoring purposes, even if there is no message to replicate.
> > I think the core motivation is to capture the output of the
> > TopicFilter/GroupFilter + the extra, built-in logic of MM2 related to
> > filtering (e.g. internal topics are never replicated, the heartbeats
> topics
> > are always replicated, and so on). This logic is too complex to reproduce
> > in an external monitoring system, as it would need to use the exact same
> > TopicFilter/GroupFilter configs as MM2 is using, and then implement the
> > additional built-in logic of MM2 to finally get the topics and groups
> > picked up by the replication.
> >
> > I think this would be useful in any replication setups (finding the
> > effective list of filtered topics and groups), but especially useful when
> > using the IdentityReplicationPolicy. One gap related to the identity
> policy
> > is that we cannot find the replica topics of a specific flow, even when
> > using MirrorClient, or having access to the source and target Kafka
> > clusters, as the "traditional" way of finding replica topics is based on
> > topic naming and the ReplicationPolicy.
> >
> > Thanks,
> > Daniel
> >
> > hudeqi <16120...@bjtu.edu.cn> ezt írta (időpont: 2023. ápr. 19., Sze,
> > 10:58):
> >
> > > Thanks for your reply, Daniel.
> > > Regarding the group list, do you mean that if the group of the source
> > > cluster has not committed an offset (the group does not exist or the
> > group
> > > has not committed an offset to the topic being replicated), then the
> > > current metric cannot be collected? Then this involves the question of
> > > motivation: Do we want to monitor the topic list and group list we
> > > configured, or the topic list and group list that are currently being
> > > replicated? If it is the latter, shouldn't it be detected for a group
> > that
> > > has not committed an offset? I don't know if I understand correctly.
> > >
> > > best,
> > > hudeqi
> > >
> > >
> > >  -原始邮件-
> > >  发件人: "Dániel Urbán" 
> > >  发送时间: 2023-04-19 15:50:01 (星期三)
> > >  收件人: dev@kafka.apache.org
> > >  抄送:
> > >  主题: Re: Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener
> > > 
> > > 
> >
>

[jira] [Created] (KAFKA-14926) Remove metrics on Log Cleaner shutdown

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-14926:


 Summary: Remove metrics on Log Cleaner shutdown
 Key: KAFKA-14926
 URL: https://issues.apache.org/jira/browse/KAFKA-14926
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Divij Vaidya
Assignee: Divij Vaidya
 Fix For: 3.6.0


We register metrics with the KafkaMetricsGroup in LogCleaner.scala but we don't 
remove them on shutdown.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener

[Viktor Somogyi-Vass]

Hi all,

A couple of comments:
1) Regarding the motivation: is the motivation simply monitoring related or
are there any other reasons to this?
2) Can we change monitoring to be identical to filters, so that what is
actively filtered, we monitor exactly those topics and groups? (So group
metrics aren't added lazily when a checkpoint is created but when the
filter is changed.)
3) Not sure if we want to widen the scope but since these are interfaces
I'd use TopicPartition and some kind of GroupDescription classes (not sure
if the latter exists) instead of Strings. If later on we'll need extra
properties for these then it can be added on easier.

Best,
Viktor

On Wed, Apr 19, 2023 at 1:42 PM Dániel Urbán  wrote:

> I wouldn't really include a non-existent group (same as we don't care about
> a non-existent topic), that doesn't really matter.
> I think having an existing group which doesn't have an offset to checkpoint
> is equivalent to a topic having no records to replicate from the monitoring
> perspective.
>
> I think the precise way to put it is to monitor the topics and groups
> picked up by the filtering logic of MM2. "The list currently replicated" is
> not a good definition, as an empty topic would still be interesting for
> monitoring purposes, even if there is no message to replicate.
> I think the core motivation is to capture the output of the
> TopicFilter/GroupFilter + the extra, built-in logic of MM2 related to
> filtering (e.g. internal topics are never replicated, the heartbeats topics
> are always replicated, and so on). This logic is too complex to reproduce
> in an external monitoring system, as it would need to use the exact same
> TopicFilter/GroupFilter configs as MM2 is using, and then implement the
> additional built-in logic of MM2 to finally get the topics and groups
> picked up by the replication.
>
> I think this would be useful in any replication setups (finding the
> effective list of filtered topics and groups), but especially useful when
> using the IdentityReplicationPolicy. One gap related to the identity policy
> is that we cannot find the replica topics of a specific flow, even when
> using MirrorClient, or having access to the source and target Kafka
> clusters, as the "traditional" way of finding replica topics is based on
> topic naming and the ReplicationPolicy.
>
> Thanks,
> Daniel
>
> hudeqi <16120...@bjtu.edu.cn> ezt írta (időpont: 2023. ápr. 19., Sze,
> 10:58):
>
> > Thanks for your reply, Daniel.
> > Regarding the group list, do you mean that if the group of the source
> > cluster has not committed an offset (the group does not exist or the
> group
> > has not committed an offset to the topic being replicated), then the
> > current metric cannot be collected? Then this involves the question of
> > motivation: Do we want to monitor the topic list and group list we
> > configured, or the topic list and group list that are currently being
> > replicated? If it is the latter, shouldn't it be detected for a group
> that
> > has not committed an offset? I don't know if I understand correctly.
> >
> > best,
> > hudeqi
> >
> >
> >  -原始邮件-
> >  发件人: "Dániel Urbán" 
> >  发送时间: 2023-04-19 15:50:01 (星期三)
> >  收件人: dev@kafka.apache.org
> >  抄送:
> >  主题: Re: Re: [DISCUSS] KIP-918: MM2 Topic And Group Listener
> > 
> > 
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.0 #219

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 200270 lines...]
[2023-04-21T02:59:59.036Z] There were failing tests. See the report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.0/core/build/reports/tests/integrationTest/index.html
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] FAILURE: Build failed with an exception.
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] * What went wrong:
[2023-04-21T03:00:00.301Z] Execution failed for task 
':streams:upgrade-system-tests-0102:unitTest'.
[2023-04-21T03:00:00.301Z] > Process 'Gradle Test Executor 21' finished with 
non-zero exit value 1
[2023-04-21T03:00:00.301Z]   This problem might be caused by incorrect test 
process configuration.
[2023-04-21T03:00:00.301Z]   Please refer to the test execution section in the 
User Manual at 
https://docs.gradle.org/7.1.1/userguide/java_testing.html#sec:test_execution
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] * Try:
[2023-04-21T03:00:00.301Z] Run with --stacktrace option to get the stack trace. 
Run with --info or --debug option to get more log output. Run with --scan to 
get full insights.
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] * Get more help at https://help.gradle.org
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] See 
https://docs.gradle.org/7.1.1/userguide/command_line_interface.html#sec:command_line_warnings
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] BUILD FAILED in 2h 0m 29s
[2023-04-21T03:00:00.301Z] 202 actionable tasks: 109 executed, 93 up-to-date
[2023-04-21T03:00:00.301Z] 
[2023-04-21T03:00:00.301Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_3.0/build/reports/profile/profile-2023-04-21-00-59-45.html
[2023-04-21T03:00:00.301Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch JDK 16 and Scala 2.12
Cancelling nested steps due to timeout
Cancelling nested steps due to timeout
Cancelling nested steps due to timeout
[2023-04-21T08:43:33.922Z] Sending interrupt signal to process
[2023-04-21T08:43:36.303Z] Sending interrupt signal to process
[2023-04-21T08:43:40.414Z] Sending interrupt signal to process
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] > Task :core:unitTest
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] SocketServerTest > testStagedListenerStartup() 
SKIPPED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] 1607 tests completed, 4 failed, 3 skipped
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] > Task :core:unitTest FAILED
[2023-04-21T08:43:43.273Z] > Task :core:copyDependantLibs
[2023-04-21T08:43:43.273Z] The message received from the daemon indicates that 
the daemon has disappeared.
[2023-04-21T08:43:43.273Z] Build request sent: 
Build{id=7d3455f4-373a-4620-88f6-657b1cd6da2f, 
currentDir=/home/jenkins/jenkins-agent/712657a4/workspace/Kafka_kafka_3.0@2}
[2023-04-21T08:43:43.273Z] Attempting to read last messages from the daemon 
log...
[2023-04-21T08:43:43.273Z] Daemon pid: 3166961
[2023-04-21T08:43:43.273Z]   log file: 
/home/jenkins/.gradle/daemon/7.1.1/daemon-3166961.out.log
[2023-04-21T08:43:43.273Z] - Last  20 lines from daemon log file - 
daemon-3166961.out.log -
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canMakeProgressAfterBackToBackLeaderFailures STARTED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canMakeProgressAfterBackToBackLeaderFailures PASSED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canRecoverFromSingleNodeCommittedDataLoss STARTED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canRecoverFromSingleNodeCommittedDataLoss PASSED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canRecoverAfterAllNodesKilled STARTED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > 
canRecoverAfterAllNodesKilled PASSED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > canElectInitialLeader 
STARTED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] RaftEventSimulationTest > canElectInitialLeader 
PASSED
[2023-04-21T08:43:43.273Z] 
[2023-04-21T08:43:43.273Z] SocketServerTest > testStagedListenerStartup() 
SKIPPED

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #169

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 335220 lines...]
[2023-04-21T03:16:41.879Z] 
[2023-04-21T03:16:41.879Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED
[2023-04-21T03:16:41.879Z] 
[2023-04-21T03:16:41.879Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED
[2023-04-21T03:16:41.879Z] 
[2023-04-21T03:16:41.879Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED
[2023-04-21T03:16:41.879Z] 
[2023-04-21T03:16:41.879Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED
[2023-04-21T03:16:45.427Z] 
[2023-04-21T03:16:45.427Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes PASSED
[2023-04-21T03:16:45.427Z] 
[2023-04-21T03:16:45.427Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog STARTED
[2023-04-21T03:16:48.058Z] 
[2023-04-21T03:16:48.058Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog PASSED
[2023-04-21T03:16:48.058Z] 
[2023-04-21T03:16:48.058Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig STARTED
[2023-04-21T03:16:51.527Z] 
[2023-04-21T03:16:51.527Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig PASSED
[2023-04-21T03:16:51.527Z] 
[2023-04-21T03:16:51.527Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated STARTED
[2023-04-21T03:16:55.105Z] 
[2023-04-21T03:16:55.105Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated PASSED
[2023-04-21T03:16:55.105Z] 
[2023-04-21T03:16:55.105Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault STARTED
[2023-04-21T03:16:58.685Z] 
[2023-04-21T03:16:58.685Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault PASSED
[2023-04-21T03:16:58.685Z] 
[2023-04-21T03:16:58.685Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor STARTED
[2023-04-21T03:16:59.622Z] 
[2023-04-21T03:16:59.622Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor PASSED
[2023-04-21T03:17:00.641Z] 
[2023-04-21T03:17:00.641Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] STARTED
[2023-04-21T03:17:00.641Z] 
[2023-04-21T03:17:00.641Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_UPDATE_true] PASSED
[2023-04-21T03:17:00.641Z] 
[2023-04-21T03:17:00.641Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] STARTED
[2023-04-21T03:17:02.566Z] 
[2023-04-21T03:17:02.566Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_UPDATE_true] PASSED
[2023-04-21T03:17:02.566Z] 
[2023-04-21T03:17:02.566Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] STARTED
[2023-04-21T03:17:04.318Z] 
[2023-04-21T03:17:04.318Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_UPDATE_true] PASSED
[2023-04-21T03:17:04.318Z] 
[2023-04-21T03:17:04.318Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_true] STARTED
[2023-04-21T03:17:17.513Z] 
[2023-04-21T03:17:17.513Z] 
org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade PASSED
[2023-04-21T03:17:17.513Z] 
[2023-04-21T03:17:17.513Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes STARTED
[2023-04-21T03:17:17.513Z] 
[2023-04-21T03:17:17.513Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED
[2023-04-21T03:17:17.513Z] 
[2023-04-21T03:17:17.513Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED
[2023-04-21T03:17:18.559Z] 
[2023-04-21T03:17:18.559Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED
[2023-04-21T03:17:18.559Z] 
[2023-04-21T03:17:18.559Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED
[2023-04-21T03:17:21.945Z] 

Re: [DISCUSS] KIP-921 OpenJDK CRaC support

[Radim Vansa]

Thank you,

now to be tracked as KIP-921: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-921%3A+OpenJDK+CRaC+support


Radim

On 20. 04. 23 15:26, Josep Prat wrote:

Hi Radim,
You should have now permissions to create a KIP.

Best,

On Thu, Apr 20, 2023 at 2:22 PM Radim Vansa  wrote:


Hello,

upon filing a PR [1] with some initial support for OpenJDK CRaC [2][3] I
was directed here to raise a KIP (I don't have the permissions in
wiki/JIRA to create the KIP page yet, though).

In a nutshell, CRaC intends to provide a way to checkpoint (snapshot)
and persist a running Java application and later restore it, possibly on
a different computer. This can be used to significantly speed up the
boot process (from seconds or minutes to tens of milliseconds), live
replication or migration of the heated up application. This is not
entirely transparent to the application; the application can register
for notification when this is happening, and sometime has to assist with
that to prevent unexpected state after restore - e.g. close network
connections and files.

CRaC is not integrated yet into the mainline JDK; JEP is being prepared,
and users are welcome to try out our builds. However even when this gets
into JDK we can't expect users jump onto the latest release immediately;
therefore we provide a facade package org.crac [4] that delegates to the
implementation, if it is present in the running JDK, or provides a no-op
implementation.

With or without the implementation, the support for CRaC in the
application should be designed to have a minimal impact on performance
(few extra objects, some volatile reads...). On the other hand the
checkpoint operation itself can be non-trivial in this matter. Therefore
the main consideration should be about the maintenance costs - keeping a
small JAR in dependencies and some extra code in networking and
persistence.

The support for CRaC does not have to be all-in for all components -
maybe it does not make sense to snapshot a Broker. My PR was for Kafka
Clients because the open network connections need to be handled in a web
application (in my case I am enabling CRaC in Quarkus Superheros [5]
demo). The PR does not handle all possible client-side uses; as I am not
familiar with Kafka I follow the whack-a-mole strategy.

It is possible that the C/R could be handled in a different layer, e.g.
in Quarkus integration code. However our intent is to push the changes
as low in the technology stack as possible, to provide the best fanout
to users without duplicating maintenance efforts. Also having the
support higher up can be fragile and break encapsulation.

Thank you for your consideration, I hope that you'll appreciate our
attempt to innovate the Java ecosystem.

Radim Vansa

PS: I'd appreciate if someone could give me the permissions on wiki to
create a proper KIP! Username: rvansa (both Confluence and JIRA).

[1] https://github.com/apache/kafka/pull/13619

[2] https://wiki.openjdk.org/display/crac

[3] https://github.com/openjdk/crac

[4] https://github.com/CRaC/org.crac

[5] https://quarkus.io/quarkus-workshops/super-heroes/



--
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
      
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B