Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Guozhang Wang]

I'm in favor of the "just pausing the instance itself“ option as well. As
for EOS, the point is that when the processing is paused, we would not
trigger any `producer.send` during the time, and the transaction timeout is
sort of relying on that behavior, so my point was that it's probably better
to also commit the processing before we pause it.


Guozhang

On Fri, May 6, 2022 at 6:12 PM Jim Hughes 
wrote:

> Hi Matthias,
>
> Since the only thing which will be paused is processing the topology, I
> think we can let commits happen naturally.
>
> Good point about getting the paused state to new members; it is seeming
> like the "building block" approach is a good one to keep things simple at
> first.
>
> Cheers,
>
> Jim
>
> On Fri, May 6, 2022 at 8:31 PM Matthias J. Sax  wrote:
>
> > I think it's tricky to propagate a pauseAll() via the rebalance
> > protocol. New members joining the group would need to get paused, too?
> > Could there be weird race conditions with overlapping pauseAll() and
> > resumeAll() calls on different instanced while there could be a errors /
> > network partitions or similar?
> >
> > I would argue that similar to IQ, we provide the basic building blocks,
> > and leave it the user users to implement cross instance management for a
> > pauseAll() scenario. -- Also, if there is really demand, we can always
> > add pauseAll()/resumeAll() as follow up work.
> >
> > About named typologies: I agree to Jim to not include them in this KIP
> > as they are not a public feature yet. If we make named typologies
> > public, the corresponding KIP should extend the pause/resume feature
> > (ie, APIs) accordingly. Of course, the code can (and should) already be
> > setup to support it to be future proof.
> >
> > Good call out about commit and EOS -- to simplify it, I think it might
> > be good to commit also for the at-least-once case?
> >
> >
> > -Matthias
> >
> >
> > On 5/6/22 1:05 PM, Jim Hughes wrote:
> > > Hi Bill,
> > >
> > > Great questions; I'll do my best to reply inline:
> > >
> > > On Fri, May 6, 2022 at 3:21 PM Bill Bejeck  wrote:
> > >
> > >> Hi Jim,
> > >>
> > >> Thanks for the KIP.  I have a couple of meta-questions as well:
> > >>
> > >> 1) Regarding pausing only a subset of running instances, I'm thinking
> > there
> > >> may be a use case for pausing all of them.
> > >> Would it make sense to also allow for pausing all instances by
> > adding a
> > >> method `pauseAll()` or something similar?
> > >>
> > >
> > > Honestly, I'm indifferent on this point.  Presently, I think what I
> have
> > > proposed is the minimal change to get the ability to pause and resume
> > > processing.  If adding a 'pauseAll()' is required, I'd be happy to do
> > that!
> > >
> > >  From Guozhang's email, it sounds like this would require using the
> > > rebalance protocol to trigger the coordination.  Would there be enough
> > room
> > > in that approach to indicate that a named topology is to be paused
> across
> > > all nodes?
> > >
> > >
> > >> 2) Would pausing affect standby tasks?  For example, imagine there
> are 3
> > >> instances A, B, and C.
> > >> A user elects to pause instance C only but it hosts the standby
> > tasks
> > >> for A.
> > >> Would the standby tasks on the paused application continue to read
> > from
> > >> the changelog topic?
> > >>
> > >
> > > Yes, standby tasks would continue reading from the changelog topic.
> All
> > > consumers would continue reading to avoid getting dropped from their
> > > consumer groups.
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > >
> > >
> > >
> > >> Thanks!
> > >> Bill
> > >>
> > >>
> > >> On Fri, May 6, 2022 at 2:44 PM Jim Hughes
>  > >
> > >> wrote:
> > >>
> > >>> Hi Guozhang,
> > >>>
> > >>> Thanks for the feedback; responses inline below:
> > >>>
> > >>> On Fri, May 6, 2022 at 1:09 PM Guozhang Wang 
> > wrote:
> > >>>
> >  Hello Jim,
> > 
> >  Thanks for the proposed KIP. I have some meta questions about it:
> > 
> >  1) Would an instance always pause/resume all of its current owned
> >  topologies (i.e. the named topologies), or are there any scenarios
> > >> where
> > >>> we
> >  only want to pause/resume a subset of them?
> > 
> > >>>
> > >>> An instance may wish to pause some of its named topologies.  I was
> > unsure
> > >>> what to say about named topologies in the KIP since they seem to be
> an
> > >>> internal detail at the moment.
> > >>>
> > >>> I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
> > >>>  public void pauseNamedTopology(final String topologyToPause)
> > >>>  public boolean isNamedTopologyPaused(final String topology)
> > >>>  public void resumeNamedTopology(final String topologyToResume)
> > >>>
> > >>>
> > >>>
> >  2) From a user's perspective, do we want to always issue a
> > >> `pause/resume`
> >  to all the instances or not? For example, we can define the
> semantics
> > >> of
> >  the function as "you only need to call this function on 

[jira] [Created] (KAFKA-13885) Add new metrics for partitioner logic introduced in KIP-794

[Artem Livshits (Jira)]

Artem Livshits created KAFKA-13885:
--

 Summary: Add new metrics for partitioner logic introduced in 
KIP-794
 Key: KAFKA-13885
 URL: https://issues.apache.org/jira/browse/KAFKA-13885
 Project: Kafka
  Issue Type: Improvement
Reporter: Artem Livshits


[https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner]
 introduced new partitioning logic, it would be good to get some observability 
into the logic.  For example, one metric could be the number of brokers that we 
marked unavailable because their latency exceeded 
*partitioner.availability.timeout.ms.*



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi Matthias,

Since the only thing which will be paused is processing the topology, I
think we can let commits happen naturally.

Good point about getting the paused state to new members; it is seeming
like the "building block" approach is a good one to keep things simple at
first.

Cheers,

Jim

On Fri, May 6, 2022 at 8:31 PM Matthias J. Sax  wrote:

> I think it's tricky to propagate a pauseAll() via the rebalance
> protocol. New members joining the group would need to get paused, too?
> Could there be weird race conditions with overlapping pauseAll() and
> resumeAll() calls on different instanced while there could be a errors /
> network partitions or similar?
>
> I would argue that similar to IQ, we provide the basic building blocks,
> and leave it the user users to implement cross instance management for a
> pauseAll() scenario. -- Also, if there is really demand, we can always
> add pauseAll()/resumeAll() as follow up work.
>
> About named typologies: I agree to Jim to not include them in this KIP
> as they are not a public feature yet. If we make named typologies
> public, the corresponding KIP should extend the pause/resume feature
> (ie, APIs) accordingly. Of course, the code can (and should) already be
> setup to support it to be future proof.
>
> Good call out about commit and EOS -- to simplify it, I think it might
> be good to commit also for the at-least-once case?
>
>
> -Matthias
>
>
> On 5/6/22 1:05 PM, Jim Hughes wrote:
> > Hi Bill,
> >
> > Great questions; I'll do my best to reply inline:
> >
> > On Fri, May 6, 2022 at 3:21 PM Bill Bejeck  wrote:
> >
> >> Hi Jim,
> >>
> >> Thanks for the KIP.  I have a couple of meta-questions as well:
> >>
> >> 1) Regarding pausing only a subset of running instances, I'm thinking
> there
> >> may be a use case for pausing all of them.
> >> Would it make sense to also allow for pausing all instances by
> adding a
> >> method `pauseAll()` or something similar?
> >>
> >
> > Honestly, I'm indifferent on this point.  Presently, I think what I have
> > proposed is the minimal change to get the ability to pause and resume
> > processing.  If adding a 'pauseAll()' is required, I'd be happy to do
> that!
> >
> >  From Guozhang's email, it sounds like this would require using the
> > rebalance protocol to trigger the coordination.  Would there be enough
> room
> > in that approach to indicate that a named topology is to be paused across
> > all nodes?
> >
> >
> >> 2) Would pausing affect standby tasks?  For example, imagine there are 3
> >> instances A, B, and C.
> >> A user elects to pause instance C only but it hosts the standby
> tasks
> >> for A.
> >> Would the standby tasks on the paused application continue to read
> from
> >> the changelog topic?
> >>
> >
> > Yes, standby tasks would continue reading from the changelog topic.  All
> > consumers would continue reading to avoid getting dropped from their
> > consumer groups.
> >
> > Cheers,
> >
> > Jim
> >
> >
> >
> >
> >> Thanks!
> >> Bill
> >>
> >>
> >> On Fri, May 6, 2022 at 2:44 PM Jim Hughes  >
> >> wrote:
> >>
> >>> Hi Guozhang,
> >>>
> >>> Thanks for the feedback; responses inline below:
> >>>
> >>> On Fri, May 6, 2022 at 1:09 PM Guozhang Wang 
> wrote:
> >>>
>  Hello Jim,
> 
>  Thanks for the proposed KIP. I have some meta questions about it:
> 
>  1) Would an instance always pause/resume all of its current owned
>  topologies (i.e. the named topologies), or are there any scenarios
> >> where
> >>> we
>  only want to pause/resume a subset of them?
> 
> >>>
> >>> An instance may wish to pause some of its named topologies.  I was
> unsure
> >>> what to say about named topologies in the KIP since they seem to be an
> >>> internal detail at the moment.
> >>>
> >>> I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
> >>>  public void pauseNamedTopology(final String topologyToPause)
> >>>  public boolean isNamedTopologyPaused(final String topology)
> >>>  public void resumeNamedTopology(final String topologyToResume)
> >>>
> >>>
> >>>
>  2) From a user's perspective, do we want to always issue a
> >> `pause/resume`
>  to all the instances or not? For example, we can define the semantics
> >> of
>  the function as "you only need to call this function on any of the
>  application's instances, and all instances would then pause (via the
>  rebalance error codes)", or as "you would call this function for all
> >> the
>  instances of an application". Which one are you referring to?
> 
> >>>
> >>> My initial intent is that one would call this function on any instances
> >> of
> >>> the application that one wishes to pause.  This should allow more
> control
> >>> (in case one wanted to pause a portion of the instances).  On the other
> >>> hand, this approach would put more work on the implementer to
> coordinate
> >>> calling pause or resume across instances.
> >>>
> >>> If the other option is more suitable, happy 

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Matthias J. Sax]

I think it's tricky to propagate a pauseAll() via the rebalance 
protocol. New members joining the group would need to get paused, too? 
Could there be weird race conditions with overlapping pauseAll() and 
resumeAll() calls on different instanced while there could be a errors / 
network partitions or similar?


I would argue that similar to IQ, we provide the basic building blocks, 
and leave it the user users to implement cross instance management for a 
pauseAll() scenario. -- Also, if there is really demand, we can always 
add pauseAll()/resumeAll() as follow up work.


About named typologies: I agree to Jim to not include them in this KIP 
as they are not a public feature yet. If we make named typologies 
public, the corresponding KIP should extend the pause/resume feature 
(ie, APIs) accordingly. Of course, the code can (and should) already be 
setup to support it to be future proof.


Good call out about commit and EOS -- to simplify it, I think it might 
be good to commit also for the at-least-once case?



-Matthias


On 5/6/22 1:05 PM, Jim Hughes wrote:

Hi Bill,

Great questions; I'll do my best to reply inline:

On Fri, May 6, 2022 at 3:21 PM Bill Bejeck  wrote:


Hi Jim,

Thanks for the KIP.  I have a couple of meta-questions as well:

1) Regarding pausing only a subset of running instances, I'm thinking there
may be a use case for pausing all of them.
Would it make sense to also allow for pausing all instances by adding a
method `pauseAll()` or something similar?



Honestly, I'm indifferent on this point.  Presently, I think what I have
proposed is the minimal change to get the ability to pause and resume
processing.  If adding a 'pauseAll()' is required, I'd be happy to do that!

 From Guozhang's email, it sounds like this would require using the
rebalance protocol to trigger the coordination.  Would there be enough room
in that approach to indicate that a named topology is to be paused across
all nodes?



2) Would pausing affect standby tasks?  For example, imagine there are 3
instances A, B, and C.
A user elects to pause instance C only but it hosts the standby tasks
for A.
Would the standby tasks on the paused application continue to read from
the changelog topic?



Yes, standby tasks would continue reading from the changelog topic.  All
consumers would continue reading to avoid getting dropped from their
consumer groups.

Cheers,

Jim





Thanks!
Bill


On Fri, May 6, 2022 at 2:44 PM Jim Hughes 
wrote:


Hi Guozhang,

Thanks for the feedback; responses inline below:

On Fri, May 6, 2022 at 1:09 PM Guozhang Wang  wrote:


Hello Jim,

Thanks for the proposed KIP. I have some meta questions about it:

1) Would an instance always pause/resume all of its current owned
topologies (i.e. the named topologies), or are there any scenarios

where

we

only want to pause/resume a subset of them?



An instance may wish to pause some of its named topologies.  I was unsure
what to say about named topologies in the KIP since they seem to be an
internal detail at the moment.

I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
 public void pauseNamedTopology(final String topologyToPause)
 public boolean isNamedTopologyPaused(final String topology)
 public void resumeNamedTopology(final String topologyToResume)




2) From a user's perspective, do we want to always issue a

`pause/resume`

to all the instances or not? For example, we can define the semantics

of

the function as "you only need to call this function on any of the
application's instances, and all instances would then pause (via the
rebalance error codes)", or as "you would call this function for all

the

instances of an application". Which one are you referring to?



My initial intent is that one would call this function on any instances

of

the application that one wishes to pause.  This should allow more control
(in case one wanted to pause a portion of the instances).  On the other
hand, this approach would put more work on the implementer to coordinate
calling pause or resume across instances.

If the other option is more suitable, happy to do that instead.



3) With EOS, there's a transaction timeout which would determine how

long a

transaction can stay idle before it's force-aborted on the broker

side. I

think when a pause is issued, that means we'd need to immediately

commit

the current transaction for EOS since we do not know how long we could
pause for. Is that right? If yes could you please clarify that in the

doc

as well.



Good point.  My intent is for pause() to wait for the next iteration
through `runOnce()` and then only skip over the processing for paused

tasks

in `taskManager.process(numIterations, time)`.

Do commits live inside that call or do they live across/outside of it?

In

the former case, I think there shouldn't be any issues with EOS.
Otherwise, we may need to work through some details to get EOS right.

Once we figure that out, I can update the KIP.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #913

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi Bill,

Great questions; I'll do my best to reply inline:

On Fri, May 6, 2022 at 3:21 PM Bill Bejeck  wrote:

> Hi Jim,
>
> Thanks for the KIP.  I have a couple of meta-questions as well:
>
> 1) Regarding pausing only a subset of running instances, I'm thinking there
> may be a use case for pausing all of them.
>Would it make sense to also allow for pausing all instances by adding a
> method `pauseAll()` or something similar?
>

Honestly, I'm indifferent on this point.  Presently, I think what I have
proposed is the minimal change to get the ability to pause and resume
processing.  If adding a 'pauseAll()' is required, I'd be happy to do that!

>From Guozhang's email, it sounds like this would require using the
rebalance protocol to trigger the coordination.  Would there be enough room
in that approach to indicate that a named topology is to be paused across
all nodes?


> 2) Would pausing affect standby tasks?  For example, imagine there are 3
> instances A, B, and C.
>A user elects to pause instance C only but it hosts the standby tasks
> for A.
>Would the standby tasks on the paused application continue to read from
> the changelog topic?
>

Yes, standby tasks would continue reading from the changelog topic.  All
consumers would continue reading to avoid getting dropped from their
consumer groups.

Cheers,

Jim




> Thanks!
> Bill
>
>
> On Fri, May 6, 2022 at 2:44 PM Jim Hughes 
> wrote:
>
> > Hi Guozhang,
> >
> > Thanks for the feedback; responses inline below:
> >
> > On Fri, May 6, 2022 at 1:09 PM Guozhang Wang  wrote:
> >
> > > Hello Jim,
> > >
> > > Thanks for the proposed KIP. I have some meta questions about it:
> > >
> > > 1) Would an instance always pause/resume all of its current owned
> > > topologies (i.e. the named topologies), or are there any scenarios
> where
> > we
> > > only want to pause/resume a subset of them?
> > >
> >
> > An instance may wish to pause some of its named topologies.  I was unsure
> > what to say about named topologies in the KIP since they seem to be an
> > internal detail at the moment.
> >
> > I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
> > public void pauseNamedTopology(final String topologyToPause)
> > public boolean isNamedTopologyPaused(final String topology)
> > public void resumeNamedTopology(final String topologyToResume)
> >
> >
> >
> > > 2) From a user's perspective, do we want to always issue a
> `pause/resume`
> > > to all the instances or not? For example, we can define the semantics
> of
> > > the function as "you only need to call this function on any of the
> > > application's instances, and all instances would then pause (via the
> > > rebalance error codes)", or as "you would call this function for all
> the
> > > instances of an application". Which one are you referring to?
> > >
> >
> > My initial intent is that one would call this function on any instances
> of
> > the application that one wishes to pause.  This should allow more control
> > (in case one wanted to pause a portion of the instances).  On the other
> > hand, this approach would put more work on the implementer to coordinate
> > calling pause or resume across instances.
> >
> > If the other option is more suitable, happy to do that instead.
> >
> >
> > > 3) With EOS, there's a transaction timeout which would determine how
> > long a
> > > transaction can stay idle before it's force-aborted on the broker
> side. I
> > > think when a pause is issued, that means we'd need to immediately
> commit
> > > the current transaction for EOS since we do not know how long we could
> > > pause for. Is that right? If yes could you please clarify that in the
> doc
> > > as well.
> > >
> >
> > Good point.  My intent is for pause() to wait for the next iteration
> > through `runOnce()` and then only skip over the processing for paused
> tasks
> > in `taskManager.process(numIterations, time)`.
> >
> > Do commits live inside that call or do they live across/outside of it?
> In
> > the former case, I think there shouldn't be any issues with EOS.
> > Otherwise, we may need to work through some details to get EOS right.
> >
> > Once we figure that out, I can update the KIP.
> >
> > Thanks,
> >
> > Jim
> >
> >
> >
> > >
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Wed, May 4, 2022 at 10:51 AM Jim Hughes
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have written up a KIP for adding the ability to pause and resume
> the
> > > > processing of a topology in AK Streams.  The KIP is here:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > > >
> > > > Thanks in advance for your feedback!
> > > >
> > > > Cheers,
> > > >
> > > > Jim
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Bill Bejeck]

Hi Jim,

Thanks for the KIP.  I have a couple of meta-questions as well:

1) Regarding pausing only a subset of running instances, I'm thinking there
may be a use case for pausing all of them.
   Would it make sense to also allow for pausing all instances by adding a
method `pauseAll()` or something similar?

2) Would pausing affect standby tasks?  For example, imagine there are 3
instances A, B, and C.
   A user elects to pause instance C only but it hosts the standby tasks
for A.
   Would the standby tasks on the paused application continue to read from
the changelog topic?

Thanks!
Bill


On Fri, May 6, 2022 at 2:44 PM Jim Hughes 
wrote:

> Hi Guozhang,
>
> Thanks for the feedback; responses inline below:
>
> On Fri, May 6, 2022 at 1:09 PM Guozhang Wang  wrote:
>
> > Hello Jim,
> >
> > Thanks for the proposed KIP. I have some meta questions about it:
> >
> > 1) Would an instance always pause/resume all of its current owned
> > topologies (i.e. the named topologies), or are there any scenarios where
> we
> > only want to pause/resume a subset of them?
> >
>
> An instance may wish to pause some of its named topologies.  I was unsure
> what to say about named topologies in the KIP since they seem to be an
> internal detail at the moment.
>
> I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
> public void pauseNamedTopology(final String topologyToPause)
> public boolean isNamedTopologyPaused(final String topology)
> public void resumeNamedTopology(final String topologyToResume)
>
>
>
> > 2) From a user's perspective, do we want to always issue a `pause/resume`
> > to all the instances or not? For example, we can define the semantics of
> > the function as "you only need to call this function on any of the
> > application's instances, and all instances would then pause (via the
> > rebalance error codes)", or as "you would call this function for all the
> > instances of an application". Which one are you referring to?
> >
>
> My initial intent is that one would call this function on any instances of
> the application that one wishes to pause.  This should allow more control
> (in case one wanted to pause a portion of the instances).  On the other
> hand, this approach would put more work on the implementer to coordinate
> calling pause or resume across instances.
>
> If the other option is more suitable, happy to do that instead.
>
>
> > 3) With EOS, there's a transaction timeout which would determine how
> long a
> > transaction can stay idle before it's force-aborted on the broker side. I
> > think when a pause is issued, that means we'd need to immediately commit
> > the current transaction for EOS since we do not know how long we could
> > pause for. Is that right? If yes could you please clarify that in the doc
> > as well.
> >
>
> Good point.  My intent is for pause() to wait for the next iteration
> through `runOnce()` and then only skip over the processing for paused tasks
> in `taskManager.process(numIterations, time)`.
>
> Do commits live inside that call or do they live across/outside of it?  In
> the former case, I think there shouldn't be any issues with EOS.
> Otherwise, we may need to work through some details to get EOS right.
>
> Once we figure that out, I can update the KIP.
>
> Thanks,
>
> Jim
>
>
>
> >
> >
> > Guozhang
> >
> >
> >
> > On Wed, May 4, 2022 at 10:51 AM Jim Hughes  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I have written up a KIP for adding the ability to pause and resume the
> > > processing of a topology in AK Streams.  The KIP is here:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > >
> > > Thanks in advance for your feedback!
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

[jira] [Created] (KAFKA-13884) KRaft Obsever are not required to flush on every append

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-13884:
--

 Summary: KRaft Obsever are not required to flush on every append
 Key: KAFKA-13884
 URL: https://issues.apache.org/jira/browse/KAFKA-13884
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


The current implementation of the KRaft Client flushes to disk when observers 
append to the log. This is not required since observer don't participate in 
leader election and the advancement of the high-watermark.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #912

[Apache Jenkins Server]

See 

[DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Hi all,

I created a KIP for adding a mechanism to monitor the health of the
KRaft Controller quorum through metrics. See KIP-835:
https://cwiki.apache.org/confluence/x/0xShD

Thanks for your feedback,
-José

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

The KIP has been updated to reflect the last discussion
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578#KIP832:Allowcreatingaproducer/consumerusingaproducer/consumerconfig-ProposedChanges


Le ven. 6 mai 2022 à 20:44, François Rosière  a
écrit :

> Hello,
>
> No problem to also add a constructor taking the StreamsConfig in the
> TopologyTestDriver.
>
> Summary about the changes to apply:
>
>- Create 2 new constructors in KafkaProducer
>- Create a new constructor in KafkaConsumer and increase de visibility
>of an existing one
>- Create a new constructor in TopologyTestDriver
>
> Kr,
>
> F.
>
> Le ven. 6 mai 2022 à 16:57, John Roesler  a écrit :
>
>> Thanks for the KIP, François!
>>
>> I'm generally in favor of your KIP, since you're
>> proposing to follow the existing pattern of the
>> constructors for both Producer and Consumer,
>> but with the config object instead of Properties
>> or Map configs. Also, because we already have
>> this pattern in Streams, and we are just
>> extending it to Producer and Consumer.
>>
>> Following on the KIP-378 discussion, I do still think
>> this is somewhat of an abuse of the Config objects,
>> and it would be better to have a formal dependency
>> injection interface, but I also don't want to let perfect
>> be the enemy of good. Since it looks like this approach
>> works, and there is also some precedent for it already,
>> I'd be inclined to approve it.
>>
>> Since KIP-378 didn't make it over the finish line, and it
>> seems like a small expansion to your proposal, do you
>> mind also adding the StreamsConfig to the
>> TopologyTestDriver constructors? That way, we can go
>> ahead and resolve both KIPs at once.
>>
>> Thank you,
>> -John
>>
>>
>> On Fri, May 6, 2022, at 06:06, François Rosière wrote:
>> > To stay consistent with existing code, we should simply add 2
>> constructors.
>> > One with ser/deser and one without.
>> > So that, users have the choice to use one or the other.
>> > I updated the KIP accordingly.
>> >
>> > Le ven. 6 mai 2022 à 12:55, François Rosière <
>> francois.rosi...@gmail.com> a
>> > écrit :
>> >
>> >> On the other hand, the KafkaConsumer constructor with a config +
>> >> serializer and deserializer already exists but is not public.
>> >> It would also complexify a bit the caller to not have the
>> >> serializer/deserializer exposed at constructor level.
>> >>
>> >> Once the KIP would have been implemented, for streams, instead of
>> having a
>> >> custom config (already possible), I may simply define a custom
>> >> KafkaClientSupplier reusing the custom configs of both the producer
>> and the
>> >> consumer.
>> >> This supplier currently creates producers and consumers using the
>> >> constructors with a map of config + serializer/deserializer.
>> >>
>> >> So, it seems it's easier to have the constructor with 3 parameters.
>> But in
>> >> any case, it will work if the config can be accessed...
>> >>
>> >> Le ven. 6 mai 2022 à 12:14, François Rosière <
>> francois.rosi...@gmail.com>
>> >> a écrit :
>> >>
>> >>> Hello,
>> >>>
>> >>> We may create a constructor with a single parameter which is the
>> config
>> >>> but then, I would need to give the serializer/deserializer by also
>> >>> overriding the config.
>> >>> Like I would do for the interceptors.
>> >>> So, no real opinion on that, both solutions are ok for me.
>> >>> Maybe easier to take the approach of the single parameter.
>> >>>
>> >>> Hope it respond to the question.
>> >>>
>> >>> Kr,
>> >>>
>> >>> F.
>> >>>
>> >>> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a
>> écrit :
>> >>>
>>  Hi Francois,
>> 
>>  Thank you for updating the KIP!
>> 
>>  Now the motivation of the KIP is much clearer.
>> 
>>  I would still be interested in:
>> 
>>   >> 2. Why do you only want to change/add the constructors that take
>> the
>>   >> properties objects and de/serializers and you do not also want to
>>   >> add/change the constructors that take only the properties?
>> 
>> 
>>  Best,
>>  Bruno
>> 
>>  On 05.05.22 23:15, François Rosière wrote:
>>  > Hello Bruno,
>>  >
>>  > The KIP as been updated. Feel free to give more feedbacks and I
>> will
>>  > complete accordingly.
>>  >
>>  > Kr,
>>  >
>>  > F.
>>  >
>>  > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a
>>  écrit :
>>  >
>>  >> Hi Francois,
>>  >>
>>  >> Thanks for the KIP!
>>  >>
>>  >> Here my first feedback:
>>  >>
>>  >> 1. Could you please extend the motivation section, so that it is
>> clear
>>  >> for a non-Spring dev why the change is needed? Usually, a
>> motivation
>>  >> section benefits a lot from an actual example.
>>  >> Extending the motivation section would also make the KIP more
>>  >> self-contained which is important IMO since this is kind of a log
>> of
>>  the
>>  >> major changes to Kafka. Descriptions of major changes 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

Hello,

No problem to also add a constructor taking the StreamsConfig in the
TopologyTestDriver.

Summary about the changes to apply:

   - Create 2 new constructors in KafkaProducer
   - Create a new constructor in KafkaConsumer and increase de visibility
   of an existing one
   - Create a new constructor in TopologyTestDriver

Kr,

F.

Le ven. 6 mai 2022 à 16:57, John Roesler  a écrit :

> Thanks for the KIP, François!
>
> I'm generally in favor of your KIP, since you're
> proposing to follow the existing pattern of the
> constructors for both Producer and Consumer,
> but with the config object instead of Properties
> or Map configs. Also, because we already have
> this pattern in Streams, and we are just
> extending it to Producer and Consumer.
>
> Following on the KIP-378 discussion, I do still think
> this is somewhat of an abuse of the Config objects,
> and it would be better to have a formal dependency
> injection interface, but I also don't want to let perfect
> be the enemy of good. Since it looks like this approach
> works, and there is also some precedent for it already,
> I'd be inclined to approve it.
>
> Since KIP-378 didn't make it over the finish line, and it
> seems like a small expansion to your proposal, do you
> mind also adding the StreamsConfig to the
> TopologyTestDriver constructors? That way, we can go
> ahead and resolve both KIPs at once.
>
> Thank you,
> -John
>
>
> On Fri, May 6, 2022, at 06:06, François Rosière wrote:
> > To stay consistent with existing code, we should simply add 2
> constructors.
> > One with ser/deser and one without.
> > So that, users have the choice to use one or the other.
> > I updated the KIP accordingly.
> >
> > Le ven. 6 mai 2022 à 12:55, François Rosière 
> a
> > écrit :
> >
> >> On the other hand, the KafkaConsumer constructor with a config +
> >> serializer and deserializer already exists but is not public.
> >> It would also complexify a bit the caller to not have the
> >> serializer/deserializer exposed at constructor level.
> >>
> >> Once the KIP would have been implemented, for streams, instead of
> having a
> >> custom config (already possible), I may simply define a custom
> >> KafkaClientSupplier reusing the custom configs of both the producer and
> the
> >> consumer.
> >> This supplier currently creates producers and consumers using the
> >> constructors with a map of config + serializer/deserializer.
> >>
> >> So, it seems it's easier to have the constructor with 3 parameters. But
> in
> >> any case, it will work if the config can be accessed...
> >>
> >> Le ven. 6 mai 2022 à 12:14, François Rosière <
> francois.rosi...@gmail.com>
> >> a écrit :
> >>
> >>> Hello,
> >>>
> >>> We may create a constructor with a single parameter which is the config
> >>> but then, I would need to give the serializer/deserializer by also
> >>> overriding the config.
> >>> Like I would do for the interceptors.
> >>> So, no real opinion on that, both solutions are ok for me.
> >>> Maybe easier to take the approach of the single parameter.
> >>>
> >>> Hope it respond to the question.
> >>>
> >>> Kr,
> >>>
> >>> F.
> >>>
> >>> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a
> écrit :
> >>>
>  Hi Francois,
> 
>  Thank you for updating the KIP!
> 
>  Now the motivation of the KIP is much clearer.
> 
>  I would still be interested in:
> 
>   >> 2. Why do you only want to change/add the constructors that take
> the
>   >> properties objects and de/serializers and you do not also want to
>   >> add/change the constructors that take only the properties?
> 
> 
>  Best,
>  Bruno
> 
>  On 05.05.22 23:15, François Rosière wrote:
>  > Hello Bruno,
>  >
>  > The KIP as been updated. Feel free to give more feedbacks and I will
>  > complete accordingly.
>  >
>  > Kr,
>  >
>  > F.
>  >
>  > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a
>  écrit :
>  >
>  >> Hi Francois,
>  >>
>  >> Thanks for the KIP!
>  >>
>  >> Here my first feedback:
>  >>
>  >> 1. Could you please extend the motivation section, so that it is
> clear
>  >> for a non-Spring dev why the change is needed? Usually, a
> motivation
>  >> section benefits a lot from an actual example.
>  >> Extending the motivation section would also make the KIP more
>  >> self-contained which is important IMO since this is kind of a log
> of
>  the
>  >> major changes to Kafka. Descriptions of major changes should not
>  >> completely depend on external links (which may become dead in
> future).
>  >> Referencing external resources to point to more details or give
>  context
>  >> is useful, though.
>  >>
>  >> 2. Why do you only want to change/add the constructors that take
> the
>  >> properties objects and de/serializers and you do not also want to
>  >> add/change the constructors that take only the properties?
>  >>
>  

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi Guozhang,

Thanks for the feedback; responses inline below:

On Fri, May 6, 2022 at 1:09 PM Guozhang Wang  wrote:

> Hello Jim,
>
> Thanks for the proposed KIP. I have some meta questions about it:
>
> 1) Would an instance always pause/resume all of its current owned
> topologies (i.e. the named topologies), or are there any scenarios where we
> only want to pause/resume a subset of them?
>

An instance may wish to pause some of its named topologies.  I was unsure
what to say about named topologies in the KIP since they seem to be an
internal detail at the moment.

I intend to add to KafkaStreamsNamedTopologyWrapper methods like:
public void pauseNamedTopology(final String topologyToPause)
public boolean isNamedTopologyPaused(final String topology)
public void resumeNamedTopology(final String topologyToResume)



> 2) From a user's perspective, do we want to always issue a `pause/resume`
> to all the instances or not? For example, we can define the semantics of
> the function as "you only need to call this function on any of the
> application's instances, and all instances would then pause (via the
> rebalance error codes)", or as "you would call this function for all the
> instances of an application". Which one are you referring to?
>

My initial intent is that one would call this function on any instances of
the application that one wishes to pause.  This should allow more control
(in case one wanted to pause a portion of the instances).  On the other
hand, this approach would put more work on the implementer to coordinate
calling pause or resume across instances.

If the other option is more suitable, happy to do that instead.


> 3) With EOS, there's a transaction timeout which would determine how long a
> transaction can stay idle before it's force-aborted on the broker side. I
> think when a pause is issued, that means we'd need to immediately commit
> the current transaction for EOS since we do not know how long we could
> pause for. Is that right? If yes could you please clarify that in the doc
> as well.
>

Good point.  My intent is for pause() to wait for the next iteration
through `runOnce()` and then only skip over the processing for paused tasks
in `taskManager.process(numIterations, time)`.

Do commits live inside that call or do they live across/outside of it?  In
the former case, I think there shouldn't be any issues with EOS.
Otherwise, we may need to work through some details to get EOS right.

Once we figure that out, I can update the KIP.

Thanks,

Jim



>
>
> Guozhang
>
>
>
> On Wed, May 4, 2022 at 10:51 AM Jim Hughes 
> wrote:
>
> > Hi all,
> >
> > I have written up a KIP for adding the ability to pause and resume the
> > processing of a topology in AK Streams.  The KIP is here:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> >
> > Thanks in advance for your feedback!
> >
> > Cheers,
> >
> > Jim
> >
>
>
> --
> -- Guozhang
>

[GitHub] [kafka-site] cadonna merged pull request #407: Fix link to old version

[GitBox]

cadonna merged PR #407:
URL: https://github.com/apache/kafka-site/pull/407


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-10888) Sticky partition leads to uneven product msg, resulting in abnormal delays in some partitions

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-10888.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

merged the PR to trunk. Thanks [~alivshits] for the design, implementation and 
the testing.

>  Sticky partition leads to uneven product msg, resulting in abnormal delays 
> in some partitions
> --
>
> Key: KAFKA-10888
> URL: https://issues.apache.org/jira/browse/KAFKA-10888
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 2.4.1
>Reporter: jr
>Assignee: Artem Livshits
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: image-2020-12-24-21-05-02-800.png, 
> image-2020-12-24-21-09-47-692.png, image-2020-12-24-21-10-24-407.png
>
>
>   110 producers ,550 partitions ,550 consumers , 5 nodes Kafka cluster
>   The producer uses the nullkey+stick partitioner, the total production rate 
> is about 100w tps
> Observed partition delay is abnormal and message distribution is uneven, 
> which leads to the maximum production and consumption delay of the partition 
> with more messages 
> abnormal.
>   I cannot find reason that stick will make the message distribution uneven 
> at this production rate.
>   I can't switch to the round-robin partitioner, which will increase the 
> delay and cpu cost. Is thathe stick partationer design cause uneven message 
> distribution, or this is abnormal. How to solve it?
>   !image-2020-12-24-21-09-47-692.png!
> As shown in the picture, the uneven distribution is concentrated on some 
> partitions and some brokers, there seems to be some rules.
> This problem does not only occur in one cluster, but in many high tps 
> clusters,
> The problem is more obvious on the test cluster we built.
> !image-2020-12-24-21-10-24-407.png!



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13883) KIP-835: Monitor Quorum

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-13883:
--

 Summary: KIP-835: Monitor Quorum
 Key: KAFKA-13883
 URL: https://issues.apache.org/jira/browse/KAFKA-13883
 Project: Kafka
  Issue Type: Improvement
  Components: kraft
Reporter: Jose Armando Garcia Sancio
Assignee: Jose Armando Garcia Sancio


Tracking issue for the implementation of KIP-835.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Guozhang Wang]

Hello Jim,

Thanks for the proposed KIP. I have some meta questions about it:

1) Would an instance always pause/resume all of its current owned
topologies (i.e. the named topologies), or are there any scenarios where we
only want to pause/resume a subset of them?

2) From a user's perspective, do we want to always issue a `pause/resume`
to all the instances or not? For example, we can define the semantics of
the function as "you only need to call this function on any of the
application's instances, and all instances would then pause (via the
rebalance error codes)", or as "you would call this function for all the
instances of an application". Which one are you referring to?

3) With EOS, there's a transaction timeout which would determine how long a
transaction can stay idle before it's force-aborted on the broker side. I
think when a pause is issued, that means we'd need to immediately commit
the current transaction for EOS since we do not know how long we could
pause for. Is that right? If yes could you please clarify that in the doc
as well.



Guozhang



On Wed, May 4, 2022 at 10:51 AM Jim Hughes 
wrote:

> Hi all,
>
> I have written up a KIP for adding the ability to pause and resume the
> processing of a topology in AK Streams.  The KIP is here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
>
> Thanks in advance for your feedback!
>
> Cheers,
>
> Jim
>


-- 
-- Guozhang

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #114

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[John Roesler]

Thanks for the KIP, François!

I'm generally in favor of your KIP, since you're
proposing to follow the existing pattern of the
constructors for both Producer and Consumer,
but with the config object instead of Properties
or Map configs. Also, because we already have
this pattern in Streams, and we are just
extending it to Producer and Consumer.

Following on the KIP-378 discussion, I do still think
this is somewhat of an abuse of the Config objects,
and it would be better to have a formal dependency
injection interface, but I also don't want to let perfect
be the enemy of good. Since it looks like this approach
works, and there is also some precedent for it already,
I'd be inclined to approve it.

Since KIP-378 didn't make it over the finish line, and it
seems like a small expansion to your proposal, do you
mind also adding the StreamsConfig to the
TopologyTestDriver constructors? That way, we can go
ahead and resolve both KIPs at once.

Thank you,
-John


On Fri, May 6, 2022, at 06:06, François Rosière wrote:
> To stay consistent with existing code, we should simply add 2 constructors.
> One with ser/deser and one without.
> So that, users have the choice to use one or the other.
> I updated the KIP accordingly.
>
> Le ven. 6 mai 2022 à 12:55, François Rosière  a
> écrit :
>
>> On the other hand, the KafkaConsumer constructor with a config +
>> serializer and deserializer already exists but is not public.
>> It would also complexify a bit the caller to not have the
>> serializer/deserializer exposed at constructor level.
>>
>> Once the KIP would have been implemented, for streams, instead of having a
>> custom config (already possible), I may simply define a custom
>> KafkaClientSupplier reusing the custom configs of both the producer and the
>> consumer.
>> This supplier currently creates producers and consumers using the
>> constructors with a map of config + serializer/deserializer.
>>
>> So, it seems it's easier to have the constructor with 3 parameters. But in
>> any case, it will work if the config can be accessed...
>>
>> Le ven. 6 mai 2022 à 12:14, François Rosière 
>> a écrit :
>>
>>> Hello,
>>>
>>> We may create a constructor with a single parameter which is the config
>>> but then, I would need to give the serializer/deserializer by also
>>> overriding the config.
>>> Like I would do for the interceptors.
>>> So, no real opinion on that, both solutions are ok for me.
>>> Maybe easier to take the approach of the single parameter.
>>>
>>> Hope it respond to the question.
>>>
>>> Kr,
>>>
>>> F.
>>>
>>> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a écrit :
>>>
 Hi Francois,

 Thank you for updating the KIP!

 Now the motivation of the KIP is much clearer.

 I would still be interested in:

  >> 2. Why do you only want to change/add the constructors that take the
  >> properties objects and de/serializers and you do not also want to
  >> add/change the constructors that take only the properties?


 Best,
 Bruno

 On 05.05.22 23:15, François Rosière wrote:
 > Hello Bruno,
 >
 > The KIP as been updated. Feel free to give more feedbacks and I will
 > complete accordingly.
 >
 > Kr,
 >
 > F.
 >
 > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a
 écrit :
 >
 >> Hi Francois,
 >>
 >> Thanks for the KIP!
 >>
 >> Here my first feedback:
 >>
 >> 1. Could you please extend the motivation section, so that it is clear
 >> for a non-Spring dev why the change is needed? Usually, a motivation
 >> section benefits a lot from an actual example.
 >> Extending the motivation section would also make the KIP more
 >> self-contained which is important IMO since this is kind of a log of
 the
 >> major changes to Kafka. Descriptions of major changes should not
 >> completely depend on external links (which may become dead in future).
 >> Referencing external resources to point to more details or give
 context
 >> is useful, though.
 >>
 >> 2. Why do you only want to change/add the constructors that take the
 >> properties objects and de/serializers and you do not also want to
 >> add/change the constructors that take only the properties?
 >>
 >> 3. I found the following stalled KIP whose motivation is really
 similar
 >> to yours:
 >>
 >>
 >>
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers
 >>
 >> That KIP is also the reason why Kafka Streams still has the
 constructors
 >> with the StreamsConfig parameter. Maybe you want to mention this KIP
 in
 >> yours or even incorporate the remaining topology test driver API
 changes
 >> in your KIP.
 >> Some related links:
 >> - https://github.com/apache/kafka/pull/5344#issuecomment-413350338
 >> - https://github.com/apache/kafka/pull/10484
 >> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #911

[Apache Jenkins Server]

See 

Re: [VOTE] 3.1.1 RC1

[Michal Tóth]

Hello

I have executed some produce/consume system tests which all passed.
Also everything passed from https://github.com/tombentley/kafka-verify-rc
- checking signatures, checksums, with gradle unit & integration tests, etc.

Good from me (non-binding).



pi 6. 5. 2022 o 14:30 David Jacot  napísal(a):

> Thanks for running the release, Tom.
>
> I performed the following validations:
> * Verified all checksums and signatures.
> * Built from source and ran unit tests.
> * Ran the first quickstart steps for both ZK and KRaft.
> * Spotchecked the Javadocs.
>
> I noticed the same issues as others on the website. I checked
> the doc in git and it looks good.
>
> +1 (binding)
>
> Best,
> David
>
> On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun  wrote:
> >
> > +1 (non-binding)
> >
> > RC1 was tested with Apache Spark tests
> >
> > - https://github.com/apache/spark/pull/36135
> >
> > Thanks,
> > Dongjoon.
> >
> > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > Hi Tom,
> > >
> > > I did:
> > > 1. check the signature and checksums
> > > 2. ran quick start with java17 + sacla2.12
> > > 3. browse java docs/documentations
> > >
> > > +1 (non-binding)
> > >
> > > Thanks for running the release.
> > >
> > > Luke
> > >
> > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > Thanks for running the release!
> > > >
> > > > I did the following checks:
> > > >
> > > >1. Validated all the checksum and signatures
> > > >2. Built from source and ran the unit tests (Java 11)
> > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > >4. Did a quick scan of the Javadoc and the documentation
> > > >
> > > > I noticed the same issues as Mickael on the website, but otherwise,
> it
> > > > looks good.
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> wrote:
> > > >
> > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged Maven
> > > > artifacts
> > > > > and ran various tests with them. Thanks for doing the release.
> > > > >
> > > > > Jakub
> > > > >
> > > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison <
> mimai...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > Thanks for running this release!
> > > > > >
> > > > > > I've done the following:
> > > > > > - Checked signatures and checksums
> > > > > > - Checked javadocs/maven artifacts
> > > > > > - Built from source and run all tests with Java 11
> > > > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > > > >
> > > > > > It looks like the website has not been updated yet, I still only
> see
> > > > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention
> reload4j in
> > > > > > the notable changes section.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley <
> tbent...@redhat.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > >
> > > > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > > > >
> > > > > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been
> fixed
> > > > > > > since 3.1.0.
> > > > > > >
> > > > > > > Release notes for the 3.1.1 release:
> > > > > > >
> > > >
> https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > > > > >
> > > > > > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > > > > > >
> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > > > > > https://kafka.apache.org/KEYS
> > > > > > >
> > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > > > > > >
> > > > > > > * Maven artifacts to be voted upon:
> > > > > > >
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > >
> > > > > > > * Javadoc:
> > > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > > > > > >
> > > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > > > > > >
> > > > > > > * Documentation:
> > > > > > > https://kafka.apache.org/31/documentation.html
> > > > > > >
> > > > > > > * Protocol:
> > > > > > > https://kafka.apache.org/31/protocol.html
> > > > > > >
> > > > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > > > I will share a link one the build is complete.
> > > > > > >
> > > > > > > /**
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Tom
> > > > > >
> > > > >
> > > >
> > >
>


-- 
S pozdravom
 Michal Tóth

Re: [VOTE] 3.2.0 RC1

[David Jacot]

Thanks for running the release, Bruno.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the doc and the Javadocs.

+1 (binding)

Best,
David

On Thu, May 5, 2022 at 10:36 AM Jakub Scholz  wrote:
>
> +1 (non-binding).
>
> I used the Scala 2.13 binaries and the staged Maven artifacts and ran
> various tests with them. Thanks for doing the release.
>
> Jakub
>
> On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 3.2.0.
> >
> > * log4j 1.x is replaced with reload4j (KAFKA-9366)
> > * StandardAuthorizer for KRaft (KIP-801)
> > * Send a hint to the partition leader to recover the partition (KIP-704)
> > * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> > * kafka-console-producer writes headers and null values (KIP-798 and
> > KIP-810)
> > * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> > * Static membership protocol lets the leader skip assignment (KIP-814)
> > * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> > * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> > * Connect APIs list all connector plugins and retrieve their
> > configuration (KIP-769)
> > * TimestampConverter SMT supports different unix time precisions (KIP-808)
> > * Connect source tasks handle producer exceptions (KIP-779)
> >
> >
> > Release notes for the 3.2.0 release:
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/32/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/32/protocol.html
> >
> > * Successful Jenkins builds for the 3.2 branch:
> > Unit/integration tests: I'll share a link once the builds complete
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> >
> > /**
> >
> > Thanks,
> > Bruno
> >

Re: [VOTE] 3.1.1 RC1

[David Jacot]

Thanks for running the release, Tom.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the Javadocs.

I noticed the same issues as others on the website. I checked
the doc in git and it looks good.

+1 (binding)

Best,
David

On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun  wrote:
>
> +1 (non-binding)
>
> RC1 was tested with Apache Spark tests
>
> - https://github.com/apache/spark/pull/36135
>
> Thanks,
> Dongjoon.
>
> On 2022/05/05 03:25:25 Luke Chen wrote:
> > Hi Tom,
> >
> > I did:
> > 1. check the signature and checksums
> > 2. ran quick start with java17 + sacla2.12
> > 3. browse java docs/documentations
> >
> > +1 (non-binding)
> >
> > Thanks for running the release.
> >
> > Luke
> >
> > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck  wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for running the release!
> > >
> > > I did the following checks:
> > >
> > >1. Validated all the checksum and signatures
> > >2. Built from source and ran the unit tests (Java 11)
> > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > >4. Did a quick scan of the Javadoc and the documentation
> > >
> > > I noticed the same issues as Mickael on the website, but otherwise, it
> > > looks good.
> > > +1 (binding)
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding). I used the Scala 2.13 binaries and staged Maven
> > > artifacts
> > > > and ran various tests with them. Thanks for doing the release.
> > > >
> > > > Jakub
> > > >
> > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison 
> > > > wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > Thanks for running this release!
> > > > >
> > > > > I've done the following:
> > > > > - Checked signatures and checksums
> > > > > - Checked javadocs/maven artifacts
> > > > > - Built from source and run all tests with Java 11
> > > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > > >
> > > > > It looks like the website has not been updated yet, I still only see
> > > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> > > > > the notable changes section.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley 
> > > > wrote:
> > > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > > >
> > > > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > > > > > since 3.1.0.
> > > > > >
> > > > > > Release notes for the 3.1.1 release:
> > > > > >
> > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/31/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/31/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > > I will share a link one the build is complete.
> > > > > >
> > > > > > /**
> > > > > >
> > > > > > Thanks,
> > > > > > Tom
> > > > >
> > > >
> > >
> >

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #46

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 525071 lines...]
[2022-05-06T12:07:26.432Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-05-06T12:07:26.432Z] > Task :connect:api:jar UP-TO-DATE
[2022-05-06T12:07:26.432Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-05-06T12:07:27.473Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-05-06T12:07:27.473Z] > Task :connect:json:jar UP-TO-DATE
[2022-05-06T12:07:27.473Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-05-06T12:07:27.473Z] > Task :connect:api:javadocJar
[2022-05-06T12:07:27.473Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-05-06T12:07:27.473Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-05-06T12:07:27.473Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-05-06T12:07:27.473Z] > Task :connect:json:publishToMavenLocal
[2022-05-06T12:07:27.473Z] > Task :connect:api:testJar
[2022-05-06T12:07:27.473Z] > Task :connect:api:testSrcJar
[2022-05-06T12:07:27.473Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-05-06T12:07:27.473Z] > Task :connect:api:publishToMavenLocal
[2022-05-06T12:07:29.567Z] 
[2022-05-06T12:07:29.567Z] > Task :streams:javadoc
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-05-06T12:07:29.567Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-05-06T12:07:29.567Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isFailure()
[2022-05-06T12:07:30.607Z] 12 warnings
[2022-05-06T12:07:30.607Z] 
[2022-05-06T12:07:30.607Z] > Task :streams:javadocJar
[2022-05-06T12:07:30.607Z] 
[2022-05-06T12:07:30.607Z] > Task :clients:javadoc
[2022-05-06T12:07:30.607Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-05-06T12:07:31.647Z] 1 warning
[2022-05-06T12:07:32.685Z] 
[2022-05-06T12:07:32.685Z] > Task :clients:javadocJar
[2022-05-06T12:07:33.720Z] 
[2022-05-06T12:07:33.720Z] > Task :clients:srcJar

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #910

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

To stay consistent with existing code, we should simply add 2 constructors.
One with ser/deser and one without.
So that, users have the choice to use one or the other.
I updated the KIP accordingly.

Le ven. 6 mai 2022 à 12:55, François Rosière  a
écrit :

> On the other hand, the KafkaConsumer constructor with a config +
> serializer and deserializer already exists but is not public.
> It would also complexify a bit the caller to not have the
> serializer/deserializer exposed at constructor level.
>
> Once the KIP would have been implemented, for streams, instead of having a
> custom config (already possible), I may simply define a custom
> KafkaClientSupplier reusing the custom configs of both the producer and the
> consumer.
> This supplier currently creates producers and consumers using the
> constructors with a map of config + serializer/deserializer.
>
> So, it seems it's easier to have the constructor with 3 parameters. But in
> any case, it will work if the config can be accessed...
>
> Le ven. 6 mai 2022 à 12:14, François Rosière 
> a écrit :
>
>> Hello,
>>
>> We may create a constructor with a single parameter which is the config
>> but then, I would need to give the serializer/deserializer by also
>> overriding the config.
>> Like I would do for the interceptors.
>> So, no real opinion on that, both solutions are ok for me.
>> Maybe easier to take the approach of the single parameter.
>>
>> Hope it respond to the question.
>>
>> Kr,
>>
>> F.
>>
>> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a écrit :
>>
>>> Hi Francois,
>>>
>>> Thank you for updating the KIP!
>>>
>>> Now the motivation of the KIP is much clearer.
>>>
>>> I would still be interested in:
>>>
>>>  >> 2. Why do you only want to change/add the constructors that take the
>>>  >> properties objects and de/serializers and you do not also want to
>>>  >> add/change the constructors that take only the properties?
>>>
>>>
>>> Best,
>>> Bruno
>>>
>>> On 05.05.22 23:15, François Rosière wrote:
>>> > Hello Bruno,
>>> >
>>> > The KIP as been updated. Feel free to give more feedbacks and I will
>>> > complete accordingly.
>>> >
>>> > Kr,
>>> >
>>> > F.
>>> >
>>> > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a
>>> écrit :
>>> >
>>> >> Hi Francois,
>>> >>
>>> >> Thanks for the KIP!
>>> >>
>>> >> Here my first feedback:
>>> >>
>>> >> 1. Could you please extend the motivation section, so that it is clear
>>> >> for a non-Spring dev why the change is needed? Usually, a motivation
>>> >> section benefits a lot from an actual example.
>>> >> Extending the motivation section would also make the KIP more
>>> >> self-contained which is important IMO since this is kind of a log of
>>> the
>>> >> major changes to Kafka. Descriptions of major changes should not
>>> >> completely depend on external links (which may become dead in future).
>>> >> Referencing external resources to point to more details or give
>>> context
>>> >> is useful, though.
>>> >>
>>> >> 2. Why do you only want to change/add the constructors that take the
>>> >> properties objects and de/serializers and you do not also want to
>>> >> add/change the constructors that take only the properties?
>>> >>
>>> >> 3. I found the following stalled KIP whose motivation is really
>>> similar
>>> >> to yours:
>>> >>
>>> >>
>>> >>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers
>>> >>
>>> >> That KIP is also the reason why Kafka Streams still has the
>>> constructors
>>> >> with the StreamsConfig parameter. Maybe you want to mention this KIP
>>> in
>>> >> yours or even incorporate the remaining topology test driver API
>>> changes
>>> >> in your KIP.
>>> >> Some related links:
>>> >> - https://github.com/apache/kafka/pull/5344#issuecomment-413350338
>>> >> - https://github.com/apache/kafka/pull/10484
>>> >> - https://issues.apache.org/jira/browse/KAFKA-6386
>>> >>
>>> >> Best,
>>> >> Bruno
>>> >>
>>> >>
>>> >> On 04.05.22 22:26, François Rosière wrote:
>>> >>> Hi all,
>>> >>>
>>> >>> KIP-832 has been created to allow implementing Spring managed
>>> >> interceptors
>>> >>> for Producers and Consumers.
>>> >>>
>>> >>> At the moment, interceptors are given as configuration classes to the
>>> >>> producer and consumer configurations. So, the idea here would be to
>>> >> create
>>> >>> 2 new constructors to allow using a Producer and Consumer
>>> configuration
>>> >>> instead of properties or a key value map of configurations entries.
>>> >>> Interceptors could then be given as instances by overriding a config
>>> >> method.
>>> >>> More details can be found in the Spring issue.
>>> >>> https://github.com/spring-projects/spring-kafka/issues/2244
>>> >>>
>>> >>> Any feedback, proposal, vote for this KIP would be more than welcome.
>>> >>>
>>> >>> Kind regards,
>>> >>>
>>> >>> Francois R.
>>> >>>
>>> >>> Le lun. 2 mai 2022 à 21:05, François Rosière <
>>> francois.rosi...@gmail.com>
>>> >> a
>>> >>> écrit :
>>> >>>
>>>  Kip link:
>>> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #113

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

On the other hand, the KafkaConsumer constructor with a config + serializer
and deserializer already exists but is not public.
It would also complexify a bit the caller to not have the
serializer/deserializer exposed at constructor level.

Once the KIP would have been implemented, for streams, instead of having a
custom config (already possible), I may simply define a custom
KafkaClientSupplier reusing the custom configs of both the producer and the
consumer.
This supplier currently creates producers and consumers using the
constructors with a map of config + serializer/deserializer.

So, it seems it's easier to have the constructor with 3 parameters. But in
any case, it will work if the config can be accessed...

Le ven. 6 mai 2022 à 12:14, François Rosière  a
écrit :

> Hello,
>
> We may create a constructor with a single parameter which is the config
> but then, I would need to give the serializer/deserializer by also
> overriding the config.
> Like I would do for the interceptors.
> So, no real opinion on that, both solutions are ok for me.
> Maybe easier to take the approach of the single parameter.
>
> Hope it respond to the question.
>
> Kr,
>
> F.
>
> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a écrit :
>
>> Hi Francois,
>>
>> Thank you for updating the KIP!
>>
>> Now the motivation of the KIP is much clearer.
>>
>> I would still be interested in:
>>
>>  >> 2. Why do you only want to change/add the constructors that take the
>>  >> properties objects and de/serializers and you do not also want to
>>  >> add/change the constructors that take only the properties?
>>
>>
>> Best,
>> Bruno
>>
>> On 05.05.22 23:15, François Rosière wrote:
>> > Hello Bruno,
>> >
>> > The KIP as been updated. Feel free to give more feedbacks and I will
>> > complete accordingly.
>> >
>> > Kr,
>> >
>> > F.
>> >
>> > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a écrit
>> :
>> >
>> >> Hi Francois,
>> >>
>> >> Thanks for the KIP!
>> >>
>> >> Here my first feedback:
>> >>
>> >> 1. Could you please extend the motivation section, so that it is clear
>> >> for a non-Spring dev why the change is needed? Usually, a motivation
>> >> section benefits a lot from an actual example.
>> >> Extending the motivation section would also make the KIP more
>> >> self-contained which is important IMO since this is kind of a log of
>> the
>> >> major changes to Kafka. Descriptions of major changes should not
>> >> completely depend on external links (which may become dead in future).
>> >> Referencing external resources to point to more details or give context
>> >> is useful, though.
>> >>
>> >> 2. Why do you only want to change/add the constructors that take the
>> >> properties objects and de/serializers and you do not also want to
>> >> add/change the constructors that take only the properties?
>> >>
>> >> 3. I found the following stalled KIP whose motivation is really similar
>> >> to yours:
>> >>
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers
>> >>
>> >> That KIP is also the reason why Kafka Streams still has the
>> constructors
>> >> with the StreamsConfig parameter. Maybe you want to mention this KIP in
>> >> yours or even incorporate the remaining topology test driver API
>> changes
>> >> in your KIP.
>> >> Some related links:
>> >> - https://github.com/apache/kafka/pull/5344#issuecomment-413350338
>> >> - https://github.com/apache/kafka/pull/10484
>> >> - https://issues.apache.org/jira/browse/KAFKA-6386
>> >>
>> >> Best,
>> >> Bruno
>> >>
>> >>
>> >> On 04.05.22 22:26, François Rosière wrote:
>> >>> Hi all,
>> >>>
>> >>> KIP-832 has been created to allow implementing Spring managed
>> >> interceptors
>> >>> for Producers and Consumers.
>> >>>
>> >>> At the moment, interceptors are given as configuration classes to the
>> >>> producer and consumer configurations. So, the idea here would be to
>> >> create
>> >>> 2 new constructors to allow using a Producer and Consumer
>> configuration
>> >>> instead of properties or a key value map of configurations entries.
>> >>> Interceptors could then be given as instances by overriding a config
>> >> method.
>> >>> More details can be found in the Spring issue.
>> >>> https://github.com/spring-projects/spring-kafka/issues/2244
>> >>>
>> >>> Any feedback, proposal, vote for this KIP would be more than welcome.
>> >>>
>> >>> Kind regards,
>> >>>
>> >>> Francois R.
>> >>>
>> >>> Le lun. 2 mai 2022 à 21:05, François Rosière <
>> francois.rosi...@gmail.com>
>> >> a
>> >>> écrit :
>> >>>
>>  Kip link:
>> 
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578
>> 
>> 
>> >>>
>> >>
>> >
>>
>

[jira] [Created] (KAFKA-13882) Dockerfile for previewing website

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13882:
---

 Summary: Dockerfile for previewing website
 Key: KAFKA-13882
 URL: https://issues.apache.org/jira/browse/KAFKA-13882
 Project: Kafka
  Issue Type: Task
  Components: docs, website
Reporter: Tom Bentley


Previewing changes to the website/documentation is rather difficult because you 
either have to [hack with the 
HTML|https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Website+Documentation+Changes#ContributingWebsiteDocumentationChanges-KafkaWebsiteRepository]
 or [install 
httpd|https://cwiki.apache.org/confluence/display/KAFKA/Setup+Kafka+Website+on+Local+Apache+Server].
 This is a barrier to contribution.

Having a Dockerfile for previewing the Kafka website (i.e. with httpd properly 
set up) would make it easier for people to contribute website/docs changes.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13881) Add package.java for public package javadoc

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-13881:
---

 Summary: Add package.java for public package javadoc
 Key: KAFKA-13881
 URL: https://issues.apache.org/jira/browse/KAFKA-13881
 Project: Kafka
  Issue Type: Task
Reporter: Tom Bentley


Our public javadoc ([https://kafka.apache.org/31/javadoc/index.html)] doesn't 
have any package descriptions, which is a bit intimidating for anyone who is 
new to the project.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

Hello,

We may create a constructor with a single parameter which is the config but
then, I would need to give the serializer/deserializer by also overriding
the config.
Like I would do for the interceptors.
So, no real opinion on that, both solutions are ok for me.
Maybe easier to take the approach of the single parameter.

Hope it respond to the question.

Kr,

F.

Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a écrit :

> Hi Francois,
>
> Thank you for updating the KIP!
>
> Now the motivation of the KIP is much clearer.
>
> I would still be interested in:
>
>  >> 2. Why do you only want to change/add the constructors that take the
>  >> properties objects and de/serializers and you do not also want to
>  >> add/change the constructors that take only the properties?
>
>
> Best,
> Bruno
>
> On 05.05.22 23:15, François Rosière wrote:
> > Hello Bruno,
> >
> > The KIP as been updated. Feel free to give more feedbacks and I will
> > complete accordingly.
> >
> > Kr,
> >
> > F.
> >
> > Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a écrit :
> >
> >> Hi Francois,
> >>
> >> Thanks for the KIP!
> >>
> >> Here my first feedback:
> >>
> >> 1. Could you please extend the motivation section, so that it is clear
> >> for a non-Spring dev why the change is needed? Usually, a motivation
> >> section benefits a lot from an actual example.
> >> Extending the motivation section would also make the KIP more
> >> self-contained which is important IMO since this is kind of a log of the
> >> major changes to Kafka. Descriptions of major changes should not
> >> completely depend on external links (which may become dead in future).
> >> Referencing external resources to point to more details or give context
> >> is useful, though.
> >>
> >> 2. Why do you only want to change/add the constructors that take the
> >> properties objects and de/serializers and you do not also want to
> >> add/change the constructors that take only the properties?
> >>
> >> 3. I found the following stalled KIP whose motivation is really similar
> >> to yours:
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers
> >>
> >> That KIP is also the reason why Kafka Streams still has the constructors
> >> with the StreamsConfig parameter. Maybe you want to mention this KIP in
> >> yours or even incorporate the remaining topology test driver API changes
> >> in your KIP.
> >> Some related links:
> >> - https://github.com/apache/kafka/pull/5344#issuecomment-413350338
> >> - https://github.com/apache/kafka/pull/10484
> >> - https://issues.apache.org/jira/browse/KAFKA-6386
> >>
> >> Best,
> >> Bruno
> >>
> >>
> >> On 04.05.22 22:26, François Rosière wrote:
> >>> Hi all,
> >>>
> >>> KIP-832 has been created to allow implementing Spring managed
> >> interceptors
> >>> for Producers and Consumers.
> >>>
> >>> At the moment, interceptors are given as configuration classes to the
> >>> producer and consumer configurations. So, the idea here would be to
> >> create
> >>> 2 new constructors to allow using a Producer and Consumer configuration
> >>> instead of properties or a key value map of configurations entries.
> >>> Interceptors could then be given as instances by overriding a config
> >> method.
> >>> More details can be found in the Spring issue.
> >>> https://github.com/spring-projects/spring-kafka/issues/2244
> >>>
> >>> Any feedback, proposal, vote for this KIP would be more than welcome.
> >>>
> >>> Kind regards,
> >>>
> >>> Francois R.
> >>>
> >>> Le lun. 2 mai 2022 à 21:05, François Rosière <
> francois.rosi...@gmail.com>
> >> a
> >>> écrit :
> >>>
>  Kip link:
> 
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578
> 
> 
> >>>
> >>
> >
>

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Luke Chen]

Hi James,

Thanks for your input.

For the `RemainingBytesToRecovery` metric proposal, I think there's one
thing I didn't make it clear.
Currently, when log manager start up, we'll try to load all logs
(segments), and during the log loading, we'll try to recover logs if
necessary.
And the logs loading is using "thread pool" as you thought.

So, here's the problem:
All segments in each log folder (partition) will be loaded in each log
recovery thread, and until it's loaded, we can know how many segments (or
how many Bytes) needed to recover.
That means, if we have 10 partition logs in one broker, and we have 2 log
recovery threads (num.recovery.threads.per.data.dir=2), before the threads
load the segments in each log, we only know how many logs (partitions) we
have in the broker (i.e. RemainingLogsToRecover metric). We cannot know how
many segments/Bytes needed to recover until each thread starts to load the
segments under one log (partition).

So, the example in the KIP, it shows:
Currently, there are still 5 logs (partitions) needed to recover under
/tmp/log1 dir. And there are 2 threads doing the jobs, where one thread has
1 segments needed to recover, and the other one has 3 segments needed
to recover.

   - kafka.log
  - LogManager
 - RemainingLogsToRecover
- /tmp/log1 => 5← there are 5 logs under /tmp/log1
needed to be recovered
- /tmp/log2 => 0
 - RemainingSegmentsToRecover
- /tmp/log1 ← 2 threads are doing log
recovery for /tmp/log1
- 0 => 1 ← there are 1 segments needed to be
   recovered for thread 0
   - 1 => 3
   - /tmp/log2
   - 0 => 0
   - 1 => 0


So, after a while, the metrics might look like this:
It said, now, there are only 4 logs needed to recover in /tmp/log1, and the
thread 0 has 9000 segments left, and thread 1 has 5 segments left (which
should imply the thread already completed 2 logs recovery in the period)

   - kafka.log
  - LogManager
 - RemainingLogsToRecover
- /tmp/log1 => 3← there are 3 logs under /tmp/log1
needed to be recovered
- /tmp/log2 => 0
 - RemainingSegmentsToRecover
- /tmp/log1 ← 2 threads are doing log
recovery for /tmp/log1
- 0 => 9000 ← there are 9000 segments needed to be
   recovered for thread 0
   - 1 => 5
   - /tmp/log2
   - 0 => 0
   - 1 => 0


That said, the `RemainingBytesToRecovery` metric is difficult to achieve as
you expected. I think the current proposal with `RemainingLogsToRecover`
and `RemainingSegmentsToRecover` should already provide enough info for the
log recovery progress.

I've also updated the KIP example to make it clear.


Thank you.
Luke


On Thu, May 5, 2022 at 3:31 AM James Cheng  wrote:

> Hi Luke,
>
> Thanks for adding RemainingSegmentsToRecovery.
>
> Another thought: different topics can have different segment sizes. I
> don't know how common it is, but it is possible. Some topics might want
> small segment sizes to more granular expiration of data.
>
> The downside of RemainingLogsToRecovery and RemainingSegmentsToRecovery is
> that the rate that they will decrement depends on the configuration and
> patterns of the topics and partitions and segment sizes. If someone is
> monitoring those metrics, they might see times where the metric decrements
> slowly, followed by a burst where it decrements quickly.
>
> What about RemainingBytesToRecovery? This would not depend on the
> configuration of the topic or of the data. It would actually be a pretty
> good metric, because I think that this metric would change at a constant
> rate (based on the disk I/O speed that the broker allocates to recovery).
> Because it changes at a constant rate, you would be able to use the
> rate-of-change to predict when it hits zero, which will let you know when
> the broker is going to start up. Like, I would imagine if we graphed
> RemainingBytesToRecovery that we'd see a fairly straight line that is
> decrementing at a steady rate towards zero.
>
> What do you think about adding RemainingBytesToRecovery?
>
> Or, what would you think about making the primary metric be
> RemainingBytesToRecovery, and getting rid of the others?
>
> I don't know if I personally would rather have all 3 metrics, or would
> just use RemainingBytesToRecovery. I'd too would like more community input
> on which of those metrics would be useful to people.
>
> About the JMX metrics, you said that if
> num.recovery.threads.per.data.dir=2, that there might be a separate
> RemainingSegmentsToRecovery counter for each thread. Is that actually how
> the data is structured within the Kafka recovery threads? Does each thread
> get a fixed set of partitions, or is there just one big pool of partitions

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[Bruno Cadonna]

Hi Francois,

Thank you for updating the KIP!

Now the motivation of the KIP is much clearer.

I would still be interested in:

>> 2. Why do you only want to change/add the constructors that take the
>> properties objects and de/serializers and you do not also want to
>> add/change the constructors that take only the properties?


Best,
Bruno

On 05.05.22 23:15, François Rosière wrote:

Hello Bruno,

The KIP as been updated. Feel free to give more feedbacks and I will
complete accordingly.

Kr,

F.

Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna  a écrit :


Hi Francois,

Thanks for the KIP!

Here my first feedback:

1. Could you please extend the motivation section, so that it is clear
for a non-Spring dev why the change is needed? Usually, a motivation
section benefits a lot from an actual example.
Extending the motivation section would also make the KIP more
self-contained which is important IMO since this is kind of a log of the
major changes to Kafka. Descriptions of major changes should not
completely depend on external links (which may become dead in future).
Referencing external resources to point to more details or give context
is useful, though.

2. Why do you only want to change/add the constructors that take the
properties objects and de/serializers and you do not also want to
add/change the constructors that take only the properties?

3. I found the following stalled KIP whose motivation is really similar
to yours:


https://cwiki.apache.org/confluence/display/KAFKA/KIP-378%3A+Enable+Dependency+Injection+for+Kafka+Streams+handlers

That KIP is also the reason why Kafka Streams still has the constructors
with the StreamsConfig parameter. Maybe you want to mention this KIP in
yours or even incorporate the remaining topology test driver API changes
in your KIP.
Some related links:
- https://github.com/apache/kafka/pull/5344#issuecomment-413350338
- https://github.com/apache/kafka/pull/10484
- https://issues.apache.org/jira/browse/KAFKA-6386

Best,
Bruno


On 04.05.22 22:26, François Rosière wrote:

Hi all,

KIP-832 has been created to allow implementing Spring managed

interceptors

for Producers and Consumers.

At the moment, interceptors are given as configuration classes to the
producer and consumer configurations. So, the idea here would be to

create

2 new constructors to allow using a Producer and Consumer configuration
instead of properties or a key value map of configurations entries.
Interceptors could then be given as instances by overriding a config

method.

More details can be found in the Spring issue.
https://github.com/spring-projects/spring-kafka/issues/2244

Any feedback, proposal, vote for this KIP would be more than welcome.

Kind regards,

Francois R.

Le lun. 2 mai 2022 à 21:05, François Rosière 

a

écrit :


Kip link:


https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578

[GitHub] [kafka-site] cadonna commented on pull request #407: Fix link to old version

[GitBox]

cadonna commented on PR #407:
URL: https://github.com/apache/kafka-site/pull/407#issuecomment-1119359385

   Call for review: @bbejeck 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org