[jira] [Created] (KAFKA-12954) Add Support for Scala 3 in 4.0.0

[Josep Prat (Jira)]

Josep Prat created KAFKA-12954:
--

 Summary: Add Support for Scala 3 in 4.0.0
 Key: KAFKA-12954
 URL: https://issues.apache.org/jira/browse/KAFKA-12954
 Project: Kafka
  Issue Type: Improvement
  Components: build
Reporter: Josep Prat
 Fix For: 4.0.0


This is a follow up task from 
https://issues.apache.org/jira/browse/KAFKA-12895, in which Scala 2.12 support 
will be dropped.

It would be good to, at the same time, add support for Scala 3.
Initially it would be enough to only make the code compile with Scala 3 so we 
can generate the proper Scala 3 artifacts, this might be achieved with the 
proper compiler flags and an occasional rewrite.
Follow up tasks could be created to migrate to a more idiomatic Scala 3 writing 
if desired.

If I understand it correctly, this would need a KIP as we are modifying the 
public interfaces (new artifacts). If this is the case, let me know  and I will 
write it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Wiki Permissions Request

[Garmes Amine]

Wiki ID:  mgarmes

I need to create a new KIP page.
Thank you for your help!

Garmes

[jira] [Created] (KAFKA-12955) Fix LogLoader to pass materialized list of segments for deletion

[Kowshik Prakasam (Jira)]

Kowshik Prakasam created KAFKA-12955:


 Summary: Fix LogLoader to pass materialized list of segments for 
deletion
 Key: KAFKA-12955
 URL: https://issues.apache.org/jira/browse/KAFKA-12955
 Project: Kafka
  Issue Type: Sub-task
Reporter: Kowshik Prakasam


Within {{LogLoader.removeAndDeleteSegmentsAsync()}}, we should force 
materialization of the {{segmentsToDelete}} iterable, to make sure the results 
of the iteration remain valid and deterministic. We should also pass only the 
materialized view to the logic that deletes the segments.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Wiki Permissions Request

[Bruno Cadonna]

Hi Garmes,

I gave you the requested permissions.

Thank you for your interest in Apache Kafka!

Best,
Bruno

On 16.06.21 09:40, Garmes Amine wrote:

Wiki ID:  mgarmes

I need to create a new KIP page.
Thank you for your help!

Garmes

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[wenbing shen]

Thanks for your comments Boyang.

I updated the KIP to remove the code details you mentioned that are not
necessary to be shown in the kip.

And I submitted a PR for review: https://github.com/apache/kafka/pull/10889

Thanks,

Wenbing

Boyang Chen  于2021年6月12日周六 上午11:44写道：

> Thanks for the updates Wenbing, the KIP looks good now. One suggestion
> would be removing the implementation details around "MessageReader" and
> "FileMessageReader" in the KIP, which should be reviewed in the actual PR.
> It's ok to briefly mention your high level approach in the `proposed
> changes` section and no need to go deep, since this is not a major feature
> or architectural change.
>
> Other than that, +1 (binding) from me.
>
> Boyang
>
> On Sun, Jun 6, 2021 at 10:54 PM wenbing shen 
> wrote:
>
> > Thanks for your comments Ran.
> >
> > `kafka-console-producer.sh --broker-list localhost:9092 --topic
> > my_topic <(cat file1 file2)`
> > Does this command work properly?
> >
> > You are right, with the help of shell, we can also complete multi-file
> > reading.
> > In the windows environment, perhaps we will also use bat to complete
> > multi-file reading.
> > Adding options from the producer command line tool to support this
> feature
> > is also to make it easier to use kafka.
> >
> > Thanks,
> >
> > Wenbing
> >
> > Ran Lupovich  于2021年6月7日周一 上午11:51写道：
> >
> > >  kafka-console-producer.sh --broker-list localhost:9092 --topic
> > > my_topic <(cat file1 file2)
> > >
> > > בתאריך יום ב׳, 7 ביוני 2021, 05:05, מאת Ran Lupovich ‏<
> > > ranlupov...@gmail.com
> > > >:
> > >
> > > > kafka-console-producer.sh --broker-list localhost:9092 --topic
> my_topic
> > > < my_file.txt
> > > >
> > > >
> > > > ?
> > > >
> > > >
> > > > בתאריך יום א׳, 6 ביוני 2021, 22:43, מאת Boyang Chen ‏<
> > > > reluctanthero...@gmail.com>:
> > > >
> > > >> Thanks for the KIP Wenbing. I have a couple of questions:
> > > >>
> > > >> 1. Could you add more descriptions into the motivation section, as
> of
> > > when
> > > >> the new options are favored and what's the current workaround?
> > > >> 2. Could you separate out the public interface and proposed changes
> > > >> sections, such that the new options are put in the former section
> and
> > > the
> > > >> expected changes are in latter.
> > > >> 3. I don't see a necessity to add support for different file
> > separators,
> > > >> could you elaborate?
> > > >>
> > > >> Boyang
> > > >>
> > > >> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen <
> oliver.shen...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > I'd like to start a vote on KIP-749 to add two options (--files
> and
> > > >> > --files-separator) to ConsoleProducer.
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Wenbing
> > > >> >
> > > >>
> > > >>
> > >
> >
>

Re: [VOTE] KIP-749: Add --files and --file-separator options to the ConsoleProducer

[wenbing shen]

Thanks, Boyang and Luke,

As of present:
- binding: +1 (Boyang)
- non-binding: +1 (Luke)

Can anyone else review this KIP?

Regards,
Wenbing

On 2021/06/16 10:36:14, wenbing shen  wrote: 
> Thanks for your comments Boyang.
> 
> I updated the KIP to remove the code details you mentioned that are not
> necessary to be shown in the kip.
> 
> And I submitted a PR for review: https://github.com/apache/kafka/pull/10889
> 
> Thanks,
> 
> Wenbing
> 
> Boyang Chen  于2021年6月12日周六 上午11:44写道：
> 
> > Thanks for the updates Wenbing, the KIP looks good now. One suggestion
> > would be removing the implementation details around "MessageReader" and
> > "FileMessageReader" in the KIP, which should be reviewed in the actual PR.
> > It's ok to briefly mention your high level approach in the `proposed
> > changes` section and no need to go deep, since this is not a major feature
> > or architectural change.
> >
> > Other than that, +1 (binding) from me.
> >
> > Boyang
> >
> > On Sun, Jun 6, 2021 at 10:54 PM wenbing shen 
> > wrote:
> >
> > > Thanks for your comments Ran.
> > >
> > > `kafka-console-producer.sh --broker-list localhost:9092 --topic
> > > my_topic <(cat file1 file2)`
> > > Does this command work properly?
> > >
> > > You are right, with the help of shell, we can also complete multi-file
> > > reading.
> > > In the windows environment, perhaps we will also use bat to complete
> > > multi-file reading.
> > > Adding options from the producer command line tool to support this
> > feature
> > > is also to make it easier to use kafka.
> > >
> > > Thanks,
> > >
> > > Wenbing
> > >
> > > Ran Lupovich  于2021年6月7日周一 上午11:51写道：
> > >
> > > >  kafka-console-producer.sh --broker-list localhost:9092 --topic
> > > > my_topic <(cat file1 file2)
> > > >
> > > > בתאריך יום ב׳, 7 ביוני 2021, 05:05, מאת Ran Lupovich ‏<
> > > > ranlupov...@gmail.com
> > > > >:
> > > >
> > > > > kafka-console-producer.sh --broker-list localhost:9092 --topic
> > my_topic
> > > > < my_file.txt
> > > > >
> > > > >
> > > > > ?
> > > > >
> > > > >
> > > > > בתאריך יום א׳, 6 ביוני 2021, 22:43, מאת Boyang Chen ‏<
> > > > > reluctanthero...@gmail.com>:
> > > > >
> > > > >> Thanks for the KIP Wenbing. I have a couple of questions:
> > > > >>
> > > > >> 1. Could you add more descriptions into the motivation section, as
> > of
> > > > when
> > > > >> the new options are favored and what's the current workaround?
> > > > >> 2. Could you separate out the public interface and proposed changes
> > > > >> sections, such that the new options are put in the former section
> > and
> > > > the
> > > > >> expected changes are in latter.
> > > > >> 3. I don't see a necessity to add support for different file
> > > separators,
> > > > >> could you elaborate?
> > > > >>
> > > > >> Boyang
> > > > >>
> > > > >> On Sat, Jun 5, 2021 at 5:16 PM wenbing shen <
> > oliver.shen...@gmail.com
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> > I'd like to start a vote on KIP-749 to add two options (--files
> > and
> > > > >> > --files-separator) to ConsoleProducer.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-749:+Add+--files+and+--file-separator+options+to+the+ConsoleProducer
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Wenbing
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > >
> >
> 

[jira] [Created] (KAFKA-12956) Validate the snapshot id when the state machine freeze a snapshot

[Haoran Xuan (Jira)]

Haoran Xuan created KAFKA-12956:
---

 Summary: Validate the snapshot id when the state machine freeze a 
snapshot
 Key: KAFKA-12956
 URL: https://issues.apache.org/jira/browse/KAFKA-12956
 Project: Kafka
  Issue Type: Sub-task
Reporter: Haoran Xuan


This is similar to KAFKA-10800, in this PR, optionally validate the snapshot id 
when `onSnapshotFrozen` is being called. The validation logic will be 
implemented in KAFKA-10800, and this Jira is supposed to directly call that 
logic.

Currently, the `onSnapshotFrozen` can be called by `KafkaRaftClient` and 
`SnapshotWriter`. Do not validate if it is called by `KafkaRaftClient` when 
it's downloading snapshot from leader, do validate if it is called by  
`SnapshotWriter` which implies generating a specific snapshot.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-747 Add support for basic aggregation APIs

[Alexandre Brasil]

Mohan / Mathias,

> > I think extending min/max to non-numeric types makes sense. Wondering
> > why we should require a `Comparator` or if we should require that the
> > types implement `Comparable` instead?
> >
> Good question. This is what it would look like:
>
> KTable min_comparable()
> KTable min_comparator(Comparator comp)

Not sure if I understood Mathias' proposal correctly, but I think that
instead of going with
your original proposal ( KTable min(Function func...)
or mine (KTable min(Comparator comparator...), we could simplify
it a
bit by using a function to extract a Comparable property from the original
value:

KTable min(Function> func...)

> I also think, that min/max should not change the value type. Using
> `Long` for sum() make sense though, and also to require a ` Number>`.

Are there any reasons to limit the sum() to integers? Why not use a Double
instead?

Best regards,
Alexandre

On Wed, Jun 16, 2021 at 1:01 AM Mohan Parthasarathy 
wrote:

> Matthias,
>
> On Mon, Jun 14, 2021 at 9:18 PM Matthias J. Sax  >
> wrote:
>
> > Hi,
> >
> > I think extending min/max to non-numeric types makes sense. Wondering
> > why we should require a `Comparator` or if we should require that the
> > types implement `Comparable` instead?
> >
> > Good question. This is what it would look like:
>
> KTable min_comparable()
> KTable min_comparator(Comparator comp)
>
> For min_comparable to work, you still need the bounds "V extends
> Comparable<
> V>". AFAICT, to avoid the "type parameter V hiding the type V" warning, it
> has to be at the interface level like this:
>
>  KStream>
>
> which is a little messy unless there is a different way to do the same. The
> comparator gives a simple way to define an anonymous function.
>
> What do you think ?
>
>
> > I also think, that min/max should not change the value type. Using
> > `Long` for sum() make sense though, and also to require a ` > Number>`.
> >
> > I guess these are the two possibilities:
>
>  Long sum(Function func)
> Long sum(Function func)
>
> Both should work. "func" can return any subtypes of Number and I don't see
> any advantages with the first version. Can you clarify ?
>
> Thanks
> Mohan
>
>
> >
> > -Matthias
> >
> > On 6/8/21 5:00 PM, Mohan Parthasarathy wrote:
> > > Hi Alex,
> > >
> > > On Tue, Jun 8, 2021 at 2:44 PM Alexandre Brasil <
> > alexandre.bra...@gmail.com>
> > > wrote:
> > >
> > >>
> > >> My point here is that, when we're only interested in a max/min numeric
> > >> value, it doesn't
> > >> matter when we have repeated values, since we'd be only forwarding the
> > >> number downstream,
> > >> so I could disregard when the Comparator returns a zero value (meaning
> > >> equals) and min/max
> > >> would still be semantically correct. But when we're forwarding the
> > original
> > >> object downstream
> > >> instead of its numeric property, it could mean different things
> > >> semantically depending on how
> > >> we handle the repeated values.
> > >>
> > >> As an example, if I were using max() on a stream of Biddings for
> > products
> > >> in an auction, the
> > >> order of the biddings would probably influence the winner if two
> clients
> > >> send Biddings with the
> > >> same value. If we're only forwarding the Bidding value downstream (a
> > double
> > >> value of 100, for
> > >> example), it doesn't matter how repeated values are handled, since the
> > max
> > >> price for this
> > >> auction would still be 100.00, no matter what Bidding got selected in
> > the
> > >> end. But if we're
> > >> forwarding the Biddings downstream instead, then it matters whether
> the
> > >> winning Bidding sent
> > >> downstream was originally posted by Client A or Client B.
> > >>
> > >> I'm not saying that an overloaded method to handle different options
> for
> > >> how repeated values
> > >> should be handled by min/max is mandatory, but it should be clear on
> the
> > >> methods' docs
> > >> what would happen when Comparator.compare() == 0. My preferred option
> > for
> > >> the default
> > >> behaviour is to only forward a new value is smaller/bigger than the
> > >> previous min/max value
> > >> (ignoring compare() == 0), since it would emit less values downstream
> > and
> > >> would be easier
> > >> to read ("I only send a value downstream if it's bigger/smaller than
> the
> > >> previously selected
> > >> value").
> > >>
> > > Thanks for the clarification. I like your suggestion unless someone
> feels
> > > that they want an option to control this (i.e., when compare() == 0,
> > return
> > > the old value vs new value).
> > >
> > >
> > >>
> > >>> Not knowing the schema of the value (V) has its own set of problems.
> > As I
> > >> have alluded to
> > >>> in the proposal, this is a little bit messy. We already have "reduce"
> > >> which can be used to
> > >>> implement sum (mapValues().reduce()).
> > >>> Thinking about it more, do you think "sum" would be useful ? One
> hacky
> > >> way to implement
> > >>> this is to inspect the type of the ret

Re: [DISCUSS] KIP-714: Client metrics and observability

[Magnus Edenhill]

Thanks for your feedback, Travis!

I believe there are different audiences and uses for application (business
logic)
and client (infrastructure) metrics. Kafka clients are part of the
infrastructure,
not the business logic, and should be monitored as such by the organization,
sub-organization, or team, that knows Kafka best and already do Kafka
monitoring - the Kafka operators.


So to be clear, this KIP does not cover application metrics, but Kafka
client metrics.
It does in no way replace or change the way application metrics are
collected, they are
not relevant to the intended use.

An analogy from the telco space are CPEs (customer premises equipment),
e.g. an ADSL router in the customer's home. The network owner - the
infrastructure operator -
monitors the ADSL router metrics for queue pressure, latencies, error
rates, etc, which allows
the operator to effectively troubleshoot customer issues, scale the
network, and foresee
issues, completely without any intervention needed by the end user itself.
This is what we want to achieve with this KIP, extending the infrastructure
operator's
(aka the Kafka cluster operator) monitoring abilities to allow for
end-to-end troubleshooting and observability.


The collection model in the KIP is subscription-based, no metrics will be
collected by default.
Two things need to happen before anything is collected:
 - a metrics plugin needs to be configured on the brokers. This is a custom
plugin to
   serve whatever needs the operator might have for the metrics.
 - client metric subscriptions need to be configured through the Kafka
Admin API to
   select which metrics to collect. The subscription defines what metrics
to collect and at
  what interval; this effectively puts filtering at the edge (client) to
spare central resources.

This functionality is thus opt-in on the cluster side, and opt-out on the
client side, and
great care is taken not to expose any sensitive information in the metrics.


As for what needs to be implemented by a supporting client;
a supporting client does not need to implement all the defined metrics,
each client maintainer may choose
her own subset that makes sense for that given client implementation, and
it is fine to add metrics not
listed in the KIP as long as they're in the client's namespace.
But there's obviously value in having a shared set of common metrics that
all clients provide.
The goal is for all client implementations to support this.


Regards,
Magnus

Den mån 14 juni 2021 kl 16:24 skrev Travis Bischel :

> Hi! I have a few thoughts on this KIP. First, I'd like to thank you for
> the writeup,
> clearly a lot of thought has gone into it and it is very thorough.
> However, I'm not
> convinced it's the right approach from a fundamental level.
>
> Fundamentally, this KIP seems like somewhat of a solution to an
> organizational
> problem. Metrics are organizational concerns, not Kafka operator concerns.
> Clients should make it easy to plug in metrics (this is the approach I
> take in
> my own client), and organizations should have processes such that all
> clients
> gather and ship metrics how that organization desires. If an organization
> is
> set up correctly, there is no reason for metrics to be forwarded through
> Kafka.
> This feels like a solution to an organization not properly setting up how
> processes ship metrics, and in some ways, it's an overbroad solution, and
> in
> other ways, it doesn't cover the entire problem.
>
> From the perspective of Kafka operators, it is easy to see that this KIP is
> nice in that it just dictates what clients should support for metrics and
> that
> the metrics should ship through Kafka. But, from the perspective of an
> observability team, this workflow is basically hijacking the standard flow
> that
> organizations may have. I would rather have applications collect metrics
> and
> ship them the same way every other application does. I'd rather not have to
> configure additional plugins within Kafka to take metrics and forward them.
>
> More importantly, this KIP prescibes cardinality problems, requires that to
> officially support the KIP a client must support all relevant metrics
> within
> the KIP, and requires that a client cannot support other metrics unless
> those
> other metrics also go through a KIP process. It is difficult to imagine
> all of
> these metrics being relevant to every organization, and there is no way
> for an
> organization to filter what is relevant within the client. Instead, the
> filtering is pushed downwards, meaning more network IO and more CPU costs
> to
> filter what is irrelevant and aggregate what needs to be aggregated, and
> more
> time for an organization to setup whatever it is that will be doing this
> filtering and aggregating. Contrast this with a client that enables
> hooking in
> to capture numbers that are relevant within an org itself: the org can
> gather
> what they want, ship only want they want, and ship directly to the
> observability sys

Re: [DISCUSS] KIP-714: Client metrics and observability

[Magnus Edenhill]

Hi Ryanne,

this proposal stems from a need to improve troubleshooting Kafka issues.

As it currently stands, when an application team is experiencing Kafka
service degradation,
or the Kafka operator is seeing misbehaving clients, there are plenty of
steps that needs
to be taken before any client-side metrics can be observed at all, if at
all:
 - Is the application even collecting client metrics? If not it needs to be
reconfigured or implemented, and restarted;
   a restart may have business impact, and may also temporarily? remedy the
problem without giving any further insight
   into what was wrong.
 - Are the desired metrics collected? Where are they stored? For how long?
Is there enough correlating information
   to map it to cluster-side metrics and events? Does the application
on-call know how to find the collected metrics?
 - Export and send these metrics to whoever knows how to interpret them. In
what format? Are all relevant metadata fields
   provided?

The KIP aims to solve all these obstacles by giving the Kafka operator the
tools to collect this information.

Regards,
Magnus


Den tis 15 juni 2021 kl 02:37 skrev Ryanne Dolan :

> Magnus, I think such a substantial change requires more motivation than is
> currently provided. As I read it, the motivation boils down to this: you
> want your clients to phone-home unless they opt-out. As stated in the KIP,
> "there are plenty of existing solutions [...] to send metrics [...] to a
> collector", so the opt-out appears to be the only motivation. Am I missing
> something?
>
> Ryanne
>
> On Wed, Jun 2, 2021 at 7:46 AM Magnus Edenhill  wrote:
>
> > Hey all,
> >
> > I'm proposing KIP-714 to add remote Client metrics and observability.
> > This functionality will allow centralized monitoring and troubleshooting
> of
> > clients and their internals.
> >
> > Please see
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
> >
> > Looking forward to your feedback!
> >
> > Regards,
> > Magnus
> >
>

[jira] [Created] (KAFKA-12957) Refactor Streams Logical Plan Generation

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-12957:
-

 Summary: Refactor Streams Logical Plan Generation
 Key: KAFKA-12957
 URL: https://issues.apache.org/jira/browse/KAFKA-12957
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


There is a general issue of Streams logical plan -> physical plan generation, 
where the physical processor nodes are generated at the parsing phase rather 
than the logical plan compilation phase. The former stage is agnostic to any 
user configurations while only the latter stage have access to it, and hence we 
would not generate physical processor nodes during the parsing phase (i.e. any 
code related to StreamsBuilder), but defer them to the logical plan phase (i.e. 
XXNode.writeToTopology). This has several issues such that many physical 
processor instantiation requires to access the configs, and hence we have to 
defer it to the `init` procedure of the node, which is scattered in many places 
from logical nodes to physical processors.

This would be a big refactoring on Stream's logical plan generation, but I 
think it would worth to get this in a cleaner state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Matthias J. Sax]

Quick follow up. I did a small update to the KIP with regard to
https://issues.apache.org/jira/browse/KAFKA-12909

Israel, Sophie, and Guozhang did agree to this change. I don't think we
need to re-vote.

Please let us know if there are any concerns.


-Matthias

On 1/27/21 12:48 PM, Sophie Blee-Goldman wrote:
> Thanks Bruno, that sounds like a good addition. +1
> 
> On Wed, Jan 27, 2021 at 12:34 PM Bruno Cadonna  wrote:
> 
>> Hi all,
>>
>> During the implementation, we notices that method removeStreamThread()
>> may block indefinitely when the stream thread chosen for removal is
>> blocked and cannot be shut down. Thus, we will add an overload that
>> takes a timeout. The newly added method will throw a TimeoutException,
>> when the timeout is exceeded.
>>
>> We updated the KIP accordingly.
>>
>> KIP: https://cwiki.apache.org/confluence/x/FDd4CQ
>>
>> Best,
>> Bruno
>>
>> On 30.09.20 13:51, Bruno Cadonna wrote:
>>> Thank you all for voting!
>>>
>>> This KIP is accepted with 3 binding +1 (Guozhang, John, Matthias).
>>>
>>> Best,
>>> Bruno
>>>
>>> On 29.09.20 22:24, Matthias J. Sax wrote:
 +1 (binding)

 I am not super happy with the impact on the client state. For example, I
 don't understand why it's ok to scale out if we lose one thread out of
 four, but why it's not ok to scale out if we lose one thread out of one
 (for this case, we would enter ERROR state and cannot add new threads
 afterwards).

 However, this might be an issue for a follow up KIP.


 -Matthias

 On 9/29/20 7:20 AM, John Roesler wrote:
> Thanks, Bruno, this sounds good to me.
> -John
>
> On Tue, Sep 29, 2020, at 03:13, Bruno Cadonna wrote:
>> Hi all,
>>
>> I did two minor modifications to the KIP.
>>
>> - I removed the rather strict guarantee "Dead stream threads are
>> removed
>> from a Kafka Streams client at latest after the next call to
>> KafkaStreams#addStreamThread() or KafkaStreams#removeStreamThread()
>> following the transition to state DEAD."
>> Dead stream threads will be still removed, but the behavior will be
>> less
>> strict.
>>
>> - Added a sentence that states that the Kafka Streams client will
>> transit to ERROR if the last alive stream thread dies exceptionally.
>> This corresponds to the current behavior.
>>
>> I will not restart voting and keep the votes so far.
>>
>> Best,
>> Bruno
>>
>> On 22.09.20 01:19, John Roesler wrote:
>>> I’m +1 also. Thanks, Bruno!
>>> -John
>>>
>>> On Mon, Sep 21, 2020, at 17:08, Guozhang Wang wrote:
 Thanks Bruno. I'm +1 on the KIP.

 On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna 
 wrote:

> Hi,
>
> I would like to restart from zero the voting on KIP-663 that
> proposes to
> add methods to the Kafka Streams client to add and remove stream
> threads
> during execution.
>
>
>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-663%3A+API+to+Start+and+Shut+Down+Stream+Threads
>
>
> Matthias, if you are still +1, please vote again.
>
> Best,
> Bruno
>
> On 04.09.20 23:12, John Roesler wrote:
>> Hi Sophie,
>>
>> Uh, oh, it's never a good sign when the discussion moves
>> into the vote thread :)
>>
>> I agree with you, it seems like a good touch for
>> removeStreamThread() to return the name of the thread that
>> got removed, rather than a boolean flag. Maybe the return
>> value would be `null` if there is no thread to remove.
>>
>> If we go that way, I'd suggest that addStreamThread() also
>> return the name of the newly created thread, or null if no
>> thread can be created right now.
>>
>> I'm not completely sure if I think that callers of this
>> method would know exactly how many threads there are. Sure,
>> if a human being is sitting there looking at the metrics or
>> logs and decides to call the method, it would work out, but
>> I'd expect this kind of method to find its way into
>> automated tooling that reacts to things like current system
>> load or resource saturation. Those kinds of toolchains often
>> are part of a distributed system, and it's probably not that
>> easy to guarantee that the thread count they observe is
>> fully consistent with the number of threads that are
>> actually running. Therefore, an in-situ `int
>> numStreamThreads()` method might not be a bad idea. Then
>> again, it seems sort of optional. A caller can catch an
>> exception or react to a `null` return value just the same
>> either way. Having both add/remove methods behave similarly
>> is probably more

Consumer Reporting error

[Kafka Life]

Hello Kafka experts

The consumer team is reporting issue while consuming the data from the
topic as Singularity Header issue.
Can some one please tell on how to resolve this issue.
Error looks like ;

Starting offset: 1226716

offset: 1226716 position: 0 CreateTime: 1583780622665 isvalid: true
keysize: -1 valuesize: 346 magic: 2 compresscodec: NONE producerId: -1
producerEpoch: -1 sequence: -1 isT*ransactional: false headerKeys:
[singularityheader]* payload: ^@^@^@^A^@^@^@

5.0.0.1246^T5.0.0.1246???\^B,AzLJC6OMQFemD1qRiUdTbg^B^@^BH0aa8db3c-6967-41a3-a7e5-395a5b70a59b^B^P79637040^@^@^@?^C^@^@^B$137@10
@avor-abb113???\??^A^@^@?^A^B^@^BH0186bf9c-53d8-4ec1-ae0b-3f9f6c98c4f4^Rundefined^B?8?>^B^FSIC

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Jesse Feinman]

Hi Moses, 

Specifically, on blocking in the constructor to fetch metadata, while I like 
the idea of specifying the topics in the constructor and fetching the metadata 
then, I think it leads to a few scenarios that could be unexpected. First is if 
you try to use a topic that wasn't included in the constructor which puts us 
back in the same position we're currently in. Second, metadata is only cached 
for a configured period of time, if a topic is published to infrequently then 
the metadata may fall out of cache leaving us in the current situation again. 
And lastly, you call this out as a con in the KIP, but a blocking constructor 
is very unexpected which could result in issues like a service failing to start 
up if metadata is missing.

For those reasons, I don't think it really addresses the core of either issue 
very well so it probably isn't a great option.

Jesse

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #226

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Sophie Blee-Goldman]

Matthias, I'm guessing you meant to send this to the KIP-633 list? This is
KIP-663

On Wed, Jun 16, 2021 at 12:37 PM Matthias J. Sax  wrote:

> Quick follow up. I did a small update to the KIP with regard to
> https://issues.apache.org/jira/browse/KAFKA-12909
>
> Israel, Sophie, and Guozhang did agree to this change. I don't think we
> need to re-vote.
>
> Please let us know if there are any concerns.
>
>
> -Matthias
>
> On 1/27/21 12:48 PM, Sophie Blee-Goldman wrote:
> > Thanks Bruno, that sounds like a good addition. +1
> >
> > On Wed, Jan 27, 2021 at 12:34 PM Bruno Cadonna 
> wrote:
> >
> >> Hi all,
> >>
> >> During the implementation, we notices that method removeStreamThread()
> >> may block indefinitely when the stream thread chosen for removal is
> >> blocked and cannot be shut down. Thus, we will add an overload that
> >> takes a timeout. The newly added method will throw a TimeoutException,
> >> when the timeout is exceeded.
> >>
> >> We updated the KIP accordingly.
> >>
> >> KIP: https://cwiki.apache.org/confluence/x/FDd4CQ
> >>
> >> Best,
> >> Bruno
> >>
> >> On 30.09.20 13:51, Bruno Cadonna wrote:
> >>> Thank you all for voting!
> >>>
> >>> This KIP is accepted with 3 binding +1 (Guozhang, John, Matthias).
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>> On 29.09.20 22:24, Matthias J. Sax wrote:
>  +1 (binding)
> 
>  I am not super happy with the impact on the client state. For
> example, I
>  don't understand why it's ok to scale out if we lose one thread out of
>  four, but why it's not ok to scale out if we lose one thread out of
> one
>  (for this case, we would enter ERROR state and cannot add new threads
>  afterwards).
> 
>  However, this might be an issue for a follow up KIP.
> 
> 
>  -Matthias
> 
>  On 9/29/20 7:20 AM, John Roesler wrote:
> > Thanks, Bruno, this sounds good to me.
> > -John
> >
> > On Tue, Sep 29, 2020, at 03:13, Bruno Cadonna wrote:
> >> Hi all,
> >>
> >> I did two minor modifications to the KIP.
> >>
> >> - I removed the rather strict guarantee "Dead stream threads are
> >> removed
> >> from a Kafka Streams client at latest after the next call to
> >> KafkaStreams#addStreamThread() or KafkaStreams#removeStreamThread()
> >> following the transition to state DEAD."
> >> Dead stream threads will be still removed, but the behavior will be
> >> less
> >> strict.
> >>
> >> - Added a sentence that states that the Kafka Streams client will
> >> transit to ERROR if the last alive stream thread dies exceptionally.
> >> This corresponds to the current behavior.
> >>
> >> I will not restart voting and keep the votes so far.
> >>
> >> Best,
> >> Bruno
> >>
> >> On 22.09.20 01:19, John Roesler wrote:
> >>> I’m +1 also. Thanks, Bruno!
> >>> -John
> >>>
> >>> On Mon, Sep 21, 2020, at 17:08, Guozhang Wang wrote:
>  Thanks Bruno. I'm +1 on the KIP.
> 
>  On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna  >
>  wrote:
> 
> > Hi,
> >
> > I would like to restart from zero the voting on KIP-663 that
> > proposes to
> > add methods to the Kafka Streams client to add and remove stream
> > threads
> > during execution.
> >
> >
> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-663%3A+API+to+Start+and+Shut+Down+Stream+Threads
> >
> >
> > Matthias, if you are still +1, please vote again.
> >
> > Best,
> > Bruno
> >
> > On 04.09.20 23:12, John Roesler wrote:
> >> Hi Sophie,
> >>
> >> Uh, oh, it's never a good sign when the discussion moves
> >> into the vote thread :)
> >>
> >> I agree with you, it seems like a good touch for
> >> removeStreamThread() to return the name of the thread that
> >> got removed, rather than a boolean flag. Maybe the return
> >> value would be `null` if there is no thread to remove.
> >>
> >> If we go that way, I'd suggest that addStreamThread() also
> >> return the name of the newly created thread, or null if no
> >> thread can be created right now.
> >>
> >> I'm not completely sure if I think that callers of this
> >> method would know exactly how many threads there are. Sure,
> >> if a human being is sitting there looking at the metrics or
> >> logs and decides to call the method, it would work out, but
> >> I'd expect this kind of method to find its way into
> >> automated tooling that reacts to things like current system
> >> load or resource saturation. Those kinds of toolchains often
> >> are part of a distributed system, and it's probably not that
> >> easy to guarantee that the thread count they observe is
> >> 

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Israel Ekpo]

Thanks for clarifying that @Sophie it is in regards to KIP-633

On Wed, Jun 16, 2021 at 4:00 PM Sophie Blee-Goldman
 wrote:

> Matthias, I'm guessing you meant to send this to the KIP-633 list? This is
> KIP-663
>
> On Wed, Jun 16, 2021 at 12:37 PM Matthias J. Sax  wrote:
>
> > Quick follow up. I did a small update to the KIP with regard to
> > https://issues.apache.org/jira/browse/KAFKA-12909
> >
> > Israel, Sophie, and Guozhang did agree to this change. I don't think we
> > need to re-vote.
> >
> > Please let us know if there are any concerns.
> >
> >
> > -Matthias
> >
> > On 1/27/21 12:48 PM, Sophie Blee-Goldman wrote:
> > > Thanks Bruno, that sounds like a good addition. +1
> > >
> > > On Wed, Jan 27, 2021 at 12:34 PM Bruno Cadonna 
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> During the implementation, we notices that method removeStreamThread()
> > >> may block indefinitely when the stream thread chosen for removal is
> > >> blocked and cannot be shut down. Thus, we will add an overload that
> > >> takes a timeout. The newly added method will throw a TimeoutException,
> > >> when the timeout is exceeded.
> > >>
> > >> We updated the KIP accordingly.
> > >>
> > >> KIP: https://cwiki.apache.org/confluence/x/FDd4CQ
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >> On 30.09.20 13:51, Bruno Cadonna wrote:
> > >>> Thank you all for voting!
> > >>>
> > >>> This KIP is accepted with 3 binding +1 (Guozhang, John, Matthias).
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>> On 29.09.20 22:24, Matthias J. Sax wrote:
> >  +1 (binding)
> > 
> >  I am not super happy with the impact on the client state. For
> > example, I
> >  don't understand why it's ok to scale out if we lose one thread out
> of
> >  four, but why it's not ok to scale out if we lose one thread out of
> > one
> >  (for this case, we would enter ERROR state and cannot add new
> threads
> >  afterwards).
> > 
> >  However, this might be an issue for a follow up KIP.
> > 
> > 
> >  -Matthias
> > 
> >  On 9/29/20 7:20 AM, John Roesler wrote:
> > > Thanks, Bruno, this sounds good to me.
> > > -John
> > >
> > > On Tue, Sep 29, 2020, at 03:13, Bruno Cadonna wrote:
> > >> Hi all,
> > >>
> > >> I did two minor modifications to the KIP.
> > >>
> > >> - I removed the rather strict guarantee "Dead stream threads are
> > >> removed
> > >> from a Kafka Streams client at latest after the next call to
> > >> KafkaStreams#addStreamThread() or
> KafkaStreams#removeStreamThread()
> > >> following the transition to state DEAD."
> > >> Dead stream threads will be still removed, but the behavior will
> be
> > >> less
> > >> strict.
> > >>
> > >> - Added a sentence that states that the Kafka Streams client will
> > >> transit to ERROR if the last alive stream thread dies
> exceptionally.
> > >> This corresponds to the current behavior.
> > >>
> > >> I will not restart voting and keep the votes so far.
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >> On 22.09.20 01:19, John Roesler wrote:
> > >>> I’m +1 also. Thanks, Bruno!
> > >>> -John
> > >>>
> > >>> On Mon, Sep 21, 2020, at 17:08, Guozhang Wang wrote:
> >  Thanks Bruno. I'm +1 on the KIP.
> > 
> >  On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna <
> br...@confluent.io
> > >
> >  wrote:
> > 
> > > Hi,
> > >
> > > I would like to restart from zero the voting on KIP-663 that
> > > proposes to
> > > add methods to the Kafka Streams client to add and remove
> stream
> > > threads
> > > during execution.
> > >
> > >
> > >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-663%3A+API+to+Start+and+Shut+Down+Stream+Threads
> > >
> > >
> > > Matthias, if you are still +1, please vote again.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 04.09.20 23:12, John Roesler wrote:
> > >> Hi Sophie,
> > >>
> > >> Uh, oh, it's never a good sign when the discussion moves
> > >> into the vote thread :)
> > >>
> > >> I agree with you, it seems like a good touch for
> > >> removeStreamThread() to return the name of the thread that
> > >> got removed, rather than a boolean flag. Maybe the return
> > >> value would be `null` if there is no thread to remove.
> > >>
> > >> If we go that way, I'd suggest that addStreamThread() also
> > >> return the name of the newly created thread, or null if no
> > >> thread can be created right now.
> > >>
> > >> I'm not completely sure if I think that callers of this
> > >> method would know exactly how many threads there are. Sure,
> > >> if a human being is sitting there looking at the metrics or
> > 

Re: [DISCUSS] KIP-739: Block Less on KafkaProducer#send

[Colin McCabe]

Blocking in the constructor to fetch metadata doesn't solve the problem, since 
metadata still needs to be periodically refreshed over time.

best,
Colin

On Wed, Jun 16, 2021, at 09:51, Jesse Feinman wrote:
> Hi Moses, 
> 
> Specifically, on blocking in the constructor to fetch metadata, while I 
> like the idea of specifying the topics in the constructor and fetching 
> the metadata then, I think it leads to a few scenarios that could be 
> unexpected. First is if you try to use a topic that wasn't included in 
> the constructor which puts us back in the same position we're currently 
> in. Second, metadata is only cached for a configured period of time, if 
> a topic is published to infrequently then the metadata may fall out of 
> cache leaving us in the current situation again. And lastly, you call 
> this out as a con in the KIP, but a blocking constructor is very 
> unexpected which could result in issues like a service failing to start 
> up if metadata is missing.
> 
> For those reasons, I don't think it really addresses the core of either 
> issue very well so it probably isn't a great option.
> 
> Jesse
> 

[jira] [Resolved] (KAFKA-12955) Fix LogLoader to pass materialized view of segments for deletion

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-12955.
-
Fix Version/s: 3.0.0
 Assignee: Kowshik Prakasam
   Resolution: Fixed

Merged the PR to trunk.

> Fix LogLoader to pass materialized view of segments for deletion
> 
>
> Key: KAFKA-12955
> URL: https://issues.apache.org/jira/browse/KAFKA-12955
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Kowshik Prakasam
>Assignee: Kowshik Prakasam
>Priority: Critical
> Fix For: 3.0.0
>
>
> Within {{LogLoader.removeAndDeleteSegmentsAsync()}}, we should force 
> materialization of the {{segmentsToDelete}} iterable, to make sure the 
> results of the iteration remain valid and deterministic. We should also pass 
> only the materialized view to the logic that deletes the segments. Otherwise, 
> we could end up deleting the wrong segments asynchronously.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

[John Roesler]

Hello again, all,

Since it would be confusing to continue indefinitely with the old
and now PAPIs both not deprecated, I have decided to go
ahead with deprecating the old PAPI in AK 3.0.

Since KAFKA-10603 has not seen any progress, this means
that we actually do have to go ahead and
deprecate+replace the KStream#process API, so I have
updated the KIP accordingly:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=118172121&selectedPageVersions=19&selectedPageVersions=18

I am implementing this proposal as part of my PR
to deprecate the old PAPI:
https://github.com/apache/kafka/pull/10869

Please let me know if you have any concerns!

Thank you,
-John

On Mon, Oct 12, 2020, at 22:18, John Roesler wrote:
> Hello all,
> 
> While reviewing the KIP document, I realized that I hadn't
> submitted a PR to migrate the KStream.process() method to
> the new API. Upon reflection, I think I'd better defer this
> work for the same reason I deferred all the transform()
> APIs. I believe that the new Processor interface will give
> us the opportunity to collapse all those operations into
> one.
> 
> Doing this work now will not only violate the 2.7 code
> freeze, but it will also take away some of our freedom when
> we take on that future work, since we wouldn't be able to
> re-use the "process" name if we chose.
> 
> Accordingly, I've edited the KIP document to say that we
> will _not_ change that method for now, but leave it for
> future work. I also filed:
> https://issues.apache.org/jira/browse/KAFKA-10603
> 
> Thanks,
> -John
> 
> On Thu, 2020-10-01 at 17:08 -0700, Matthias J. Sax wrote:
> > Thanks John.
> > 
> > SGTM.
> > 
> > On 10/1/20 2:50 PM, John Roesler wrote:
> > > Hello again, all,
> > > 
> > > I'm sorry to make another tweak to this KIP, but during the
> > > implementation of the design we've just agreed on, I
> > > realized that Processors would almost never need to
> > > reference the RecordMetadata. Therefore, I'm proposing to
> > > streamline the API by moving the Optional to
> > > the new ProcessorContext as a method, rather than making it
> > > a method argument to Processor#process.
> > > 
> > > The change is visible here:
> > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=118172121&selectedPageVersions=16&selectedPageVersions=15
> > > 
> > > All of the same semantics and considerations we discussed
> > > still apply, it's just that Processor implementers won't
> > > have to think about it unless they actually _need_ the
> > > topic/partition/offset information from the RecordMetadata.
> > > 
> > > Also, the PR for this part of the KIP is now available here:
> > > https://github.com/apache/kafka/pull/9361
> > > 
> > > I know it's a bit on the heavy side; I've annotated the PR
> > > to try and ease the reviewer's job. I'd greatly appreciate
> > > it if anyone can take the time to review.
> > > 
> > > Thanks,
> > > -John
> > > 
> > > On Wed, 2020-09-30 at 10:16 -0500, John Roesler wrote:
> > > > Thanks, Matthias!
> > > > 
> > > > I can certainly document it. I didn't bother because the old
> > > > Processor, Supplier, and Context will themselves be
> > > > deprecated, so any method that handles them won't be able to
> > > > avoid the deprecation warning. Nevertheless, it doesn't hurt
> > > > just to explicitly deprecated those methods.
> > > > 
> > > > Thanks,
> > > > -John
> > > > 
> > > > On Wed, 2020-09-30 at 08:10 -0700, Matthias J. Sax wrote:
> > > > > Thanks John. I like the proposal.
> > > > > 
> > > > > Btw: I was just going over the KIP and realized that we add new 
> > > > > methods
> > > > > to `StreamBuilder`, `Topology`, and `KStream` that take the new
> > > > > `ProcessorSupplier` class -- should we also deprecate the 
> > > > > corresponding
> > > > > existing ones that take the old `ProcessorSupplier`?
> > > > > 
> > > > > 
> > > > > -Matthias
> > > > > 
> > > > > 
> > > > > On 9/30/20 7:46 AM, John Roesler wrote:
> > > > > > Thanks Paul and Sophie,
> > > > > > 
> > > > > > Your feedback certainly underscores the need to be explicit
> > > > > > in the javadoc about why that parameter is Optional. Getting
> > > > > > this kind of feedback before the release is exactly the kind
> > > > > > of outcome we hope to get from the KIP process!
> > > > > > 
> > > > > > Thanks,
> > > > > > -John
> > > > > > 
> > > > > > On Tue, 2020-09-29 at 22:32 -0500, Paul Whalen wrote:
> > > > > > > John, I totally agree that adding a method to Processor is 
> > > > > > > cumbersome and
> > > > > > > not a good path.  I was imagining maybe a separate interface that 
> > > > > > > could be
> > > > > > > used in the appropriate context, but I don't think that makes too 
> > > > > > > much
> > > > > > > sense - it's just too far away from what Kafka Streams is.  I was
> > > > > > > originally more interested in the "why" Optional than the "how" 
> > > > > > > (I think my
> > > > > > > original reply overplayed the "optional as an argument" concern). 
> > > >

Re: [DISCUSS] KIP-714: Client metrics and observability

[Colin McCabe]

On Sun, Jun 13, 2021, at 21:51, Travis Bischel wrote:
> Hi! I have a few thoughts on this KIP. First, I'd like to thank you for 
> the writeup,
> clearly a lot of thought has gone into it and it is very thorough. 
> However, I'm not
> convinced it's the right approach from a fundamental level.
> 
> Fundamentally, this KIP seems like somewhat of a solution to an organizational
> problem. Metrics are organizational concerns, not Kafka operator concerns.

Hi Travis,

Metrics are certainly Kafka operator concerns. It is very important for cluster 
operators to know things like how many clients there are, what they clients are 
doing, and so forth. This information is needed to administer Kafka. Therefore 
it certainly falls in the domain of the Kafka operations team (and the Kafka 
development team.)

We have added many metrics in the past to make it easier to monitor clients. I 
think this is just another step in that direction.

> Clients should make it easy to plug in metrics (this is the approach I take in
> my own client), and organizations should have processes such that all clients
> gather and ship metrics how that organization desires.
>
> If an organization is set up correctly, there is no reason for metrics to be
> forwarded through Kafka. This feels like a solution to an organization not
> properly setting up how processes ship metrics, and in some ways, it's an
> overbroad solution, and in other ways, it doesn't cover the entire problem.

I think the reason was explained pretty clearly: many admins find it difficult 
to set up monitoring for every client in the organization. In general the team 
which maintains a Kafka cluster is often separate from the teams that use the 
cluster. Therefore rolling out monitoring for clients can be very difficult to 
coordinate.

No metrics will ever cover every possible use-case, but the set proposed here 
does seem useful.

> 
> From the perspective of Kafka operators, it is easy to see that this KIP is
> nice in that it just dictates what clients should support for metrics and that
> the metrics should ship through Kafka. But, from the perspective of an
> observability team, this workflow is basically hijacking the standard flow 
> that
> organizations may have. I would rather have applications collect metrics and
> ship them the same way every other application does. I'd rather not have to
> configure additional plugins within Kafka to take metrics and forward them.

This change doesn't remove any functionality. If you don't want to use KIP-714 
metrics collection, you can simply turn it off and continue collecting metrics 
the way you always have.

> 
> More importantly, this KIP prescibes cardinality problems, requires that to
> officially support the KIP a client must support all relevant metrics within
> the KIP, and requires that a client cannot support other metrics unless those
> other metrics also go through a KIP process. It is difficult to imagine all of
> these metrics being relevant to every organization, and there is no way for an
> organization to filter what is relevant within the client. Instead, the
> filtering is pushed downwards, meaning more network IO and more CPU costs to
> filter what is irrelevant and aggregate what needs to be aggregated, and more
> time for an organization to setup whatever it is that will be doing this
> filtering and aggregating. Contrast this with a client that enables hooking in
> to capture numbers that are relevant within an org itself: the org can gather
> what they want, ship only want they want, and ship directly to the
> observability system they have already set up. As an aside, it may also be
> wise to avoid shipping metrics through Kafka about client interaction with
> Kafka, because if Kafka is having problems, then orgs lose insight into those
> problems. This would be like statuspage using itself for status on its own
> systems.
> 
> Another downside is that by dictating the important metrics, this KIP either
> has two choices: try to choose what is important to every org, and inevitably
> leave out something important to somebody else, or just add everything and let
> the orgs filter. This KIP mostly looks to go with the latter approach, meaning
> orgs will be shipping & filtering. With hooks, an org would be able to gather
> exactly what they want.

I actually do agree with this criticism to some extent. It would be good if the 
broker could specify what metrics it wants, and the clients would send only 
those metrics.

More generally, I'd like to see this split up into several RPCs rather than one 
mega-RPC.

Maybe something like 
1. RegisterClient{Request,Response}
2. ClientMetricsReport{Request,Response}
3. UnregisterClient{Request,Response}

Then the broker can communicate which metrics it wants in 
RegisterClientResponse. It can also assign a client instance ID (which I think 
should be a UUID, not another string).

> 
> As well, I expect that org applications have metrics on the state of the
>

[jira] [Created] (KAFKA-12958) Add similation invariant for leadership and snapshot

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12958:
--

 Summary: Add similation invariant for leadership and snapshot
 Key: KAFKA-12958
 URL: https://issues.apache.org/jira/browse/KAFKA-12958
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jose Armando Garcia Sancio


During the simulation we should add an invariant that notified leaders are 
never asked to load snapshots. The state machine always sees the following 
sequence of callback calls:

Leaders see:
...
handleLeaderChange state machine is notify of leadership
handleSnapshot is never called

Non-leader see:
...
handleLeaderChange state machine is notify that is not leader
handleSnapshot is called 0 or more times



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-714: Client metrics and observability

[Colin McCabe]

Hi Magnus,

Thanks for the KIP. This is certainly something I've been wishing for for a 
while.

Maybe we should emphasize more that the metrics that are being gathered here 
are Kafka metrics, not general application business logic metrics. That seems 
like a point of confusion in some of the replies here. The analogy with a 
telecom gathering metrics about a DSL modem is a good one. These are really 
metrics about the Kafka cluster itself, very similar to the metrics we expose 
about the broker, controller, and so forth.

In my experience, most users want their Kafka clients to be "plug and play" -- 
they want to start up a Kafka client, and do some things. Their focus is on 
their application, not on the details of the infrastructure. If something is 
goes wrong, they want the Kafka team to diagnose the problem and fix it, or at 
least tell them what the issue is. When the Kafka teams tells them they need to 
install and maintain a third-party metrics system to diagnose the problem, this 
can be a very big disappointment. Many users don't have this level of expertise.

A few critiques:

- As I wrote above, I think this could benefit a lot by being split into 
several RPCs. A registration RPC, a report RPC, and an unregister RPC seem like 
logical choices.

- I don't think the client should be able to choose its own UUID. This adds 
complexity and introduces a chance that clients will choose an ID that is not 
unique. We already have an ID that the client itself supplies (clientID) so 
there is no need to introduce another such ID.

- I might be misunderstanding something here, but my reading of this is that 
the client chooses what metrics to send and the broker filters that on the 
broker-side. I think this is backwards -- the broker should inform the client 
about what it wants, and the client should send only that data. (Of course, the 
client may also not know what the broker is asking for, in which case it can 
choose to not send the data). We shouldn't have clients pumping out data that 
nobody wants to read. (sorry if I misinterpreted and this is already the 
case...)

- In general the schema seems to have a bad case of string-itis. UUID, content 
type, and requested metrics are all strings. Since these messages will be sent 
very frequently, it's quite costly to use strings for all these things. We have 
a type for UUID, which uses 16 bytes -- let's use that type for client instance 
ID, rather than a string which will be much larger. Also, since we already send 
clientID in the message header, there is no need to include it again in the 
instance ID.

- I think it would also be nice to have an enum or something for 
AcceptedContentTypes, RequestedMetrics, etc. We know that new additions to 
these categories will require KIPs, so it should be straightforward for the 
project to just have an enum that allows us to communicate these as ints.

- Can you talk about whether you are adding any new library dependencies to the 
Kafka client? It seems like you'd want to add opencensus / opentelemetry, if we 
are using that format here.

- Standard client resource labels: can we send these only in the registration 
RPC?

best,
Colin

On Wed, Jun 16, 2021, at 08:27, Magnus Edenhill wrote:
> Hi Ryanne,
> 
> this proposal stems from a need to improve troubleshooting Kafka issues.
> 
> As it currently stands, when an application team is experiencing Kafka
> service degradation,
> or the Kafka operator is seeing misbehaving clients, there are plenty of
> steps that needs
> to be taken before any client-side metrics can be observed at all, if at
> all:
>  - Is the application even collecting client metrics? If not it needs to be
> reconfigured or implemented, and restarted;
>a restart may have business impact, and may also temporarily? remedy the
> problem without giving any further insight
>into what was wrong.
>  - Are the desired metrics collected? Where are they stored? For how long?
> Is there enough correlating information
>to map it to cluster-side metrics and events? Does the application
> on-call know how to find the collected metrics?
>  - Export and send these metrics to whoever knows how to interpret them. In
> what format? Are all relevant metadata fields
>provided?
> 
> The KIP aims to solve all these obstacles by giving the Kafka operator the
> tools to collect this information.
> 
> Regards,
> Magnus
> 
> 
> Den tis 15 juni 2021 kl 02:37 skrev Ryanne Dolan :
> 
> > Magnus, I think such a substantial change requires more motivation than is
> > currently provided. As I read it, the motivation boils down to this: you
> > want your clients to phone-home unless they opt-out. As stated in the KIP,
> > "there are plenty of existing solutions [...] to send metrics [...] to a
> > collector", so the opt-out appears to be the only motivation. Am I missing
> > something?
> >
> > Ryanne
> >
> > On Wed, Jun 2, 2021 at 7:46 AM Magnus Edenhill  wrote:
> >
> > > Hey all,
> > >
> > > I'm proposing

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Matthias J. Sax]

Yes. Sorry about this mistake.

On 6/16/21 2:29 PM, Israel Ekpo wrote:
> Thanks for clarifying that @Sophie it is in regards to KIP-633
> 
> On Wed, Jun 16, 2021 at 4:00 PM Sophie Blee-Goldman
>  wrote:
> 
>> Matthias, I'm guessing you meant to send this to the KIP-633 list? This is
>> KIP-663
>>
>> On Wed, Jun 16, 2021 at 12:37 PM Matthias J. Sax  wrote:
>>
>>> Quick follow up. I did a small update to the KIP with regard to
>>> https://issues.apache.org/jira/browse/KAFKA-12909
>>>
>>> Israel, Sophie, and Guozhang did agree to this change. I don't think we
>>> need to re-vote.
>>>
>>> Please let us know if there are any concerns.
>>>
>>>
>>> -Matthias
>>>
>>> On 1/27/21 12:48 PM, Sophie Blee-Goldman wrote:
 Thanks Bruno, that sounds like a good addition. +1

 On Wed, Jan 27, 2021 at 12:34 PM Bruno Cadonna 
>>> wrote:

> Hi all,
>
> During the implementation, we notices that method removeStreamThread()
> may block indefinitely when the stream thread chosen for removal is
> blocked and cannot be shut down. Thus, we will add an overload that
> takes a timeout. The newly added method will throw a TimeoutException,
> when the timeout is exceeded.
>
> We updated the KIP accordingly.
>
> KIP: https://cwiki.apache.org/confluence/x/FDd4CQ
>
> Best,
> Bruno
>
> On 30.09.20 13:51, Bruno Cadonna wrote:
>> Thank you all for voting!
>>
>> This KIP is accepted with 3 binding +1 (Guozhang, John, Matthias).
>>
>> Best,
>> Bruno
>>
>> On 29.09.20 22:24, Matthias J. Sax wrote:
>>> +1 (binding)
>>>
>>> I am not super happy with the impact on the client state. For
>>> example, I
>>> don't understand why it's ok to scale out if we lose one thread out
>> of
>>> four, but why it's not ok to scale out if we lose one thread out of
>>> one
>>> (for this case, we would enter ERROR state and cannot add new
>> threads
>>> afterwards).
>>>
>>> However, this might be an issue for a follow up KIP.
>>>
>>>
>>> -Matthias
>>>
>>> On 9/29/20 7:20 AM, John Roesler wrote:
 Thanks, Bruno, this sounds good to me.
 -John

 On Tue, Sep 29, 2020, at 03:13, Bruno Cadonna wrote:
> Hi all,
>
> I did two minor modifications to the KIP.
>
> - I removed the rather strict guarantee "Dead stream threads are
> removed
> from a Kafka Streams client at latest after the next call to
> KafkaStreams#addStreamThread() or
>> KafkaStreams#removeStreamThread()
> following the transition to state DEAD."
> Dead stream threads will be still removed, but the behavior will
>> be
> less
> strict.
>
> - Added a sentence that states that the Kafka Streams client will
> transit to ERROR if the last alive stream thread dies
>> exceptionally.
> This corresponds to the current behavior.
>
> I will not restart voting and keep the votes so far.
>
> Best,
> Bruno
>
> On 22.09.20 01:19, John Roesler wrote:
>> I’m +1 also. Thanks, Bruno!
>> -John
>>
>> On Mon, Sep 21, 2020, at 17:08, Guozhang Wang wrote:
>>> Thanks Bruno. I'm +1 on the KIP.
>>>
>>> On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna <
>> br...@confluent.io

>>> wrote:
>>>
 Hi,

 I would like to restart from zero the voting on KIP-663 that
 proposes to
 add methods to the Kafka Streams client to add and remove
>> stream
 threads
 during execution.



>
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-663%3A+API+to+Start+and+Shut+Down+Stream+Threads


 Matthias, if you are still +1, please vote again.

 Best,
 Bruno

 On 04.09.20 23:12, John Roesler wrote:
> Hi Sophie,
>
> Uh, oh, it's never a good sign when the discussion moves
> into the vote thread :)
>
> I agree with you, it seems like a good touch for
> removeStreamThread() to return the name of the thread that
> got removed, rather than a boolean flag. Maybe the return
> value would be `null` if there is no thread to remove.
>
> If we go that way, I'd suggest that addStreamThread() also
> return the name of the newly created thread, or null if no
> thread can be created right now.
>
> I'm not completely sure if I think that callers of this
> method would know exactly how many threads there are. Sure,
> if a human being is sitting there looking at the metrics or
> logs and decide

Re: [VOTE] KIP-633: Drop 24 hour default of grace period in Streams

[Matthias J. Sax]

Quick follow up. I did a small update to the KIP with regard to
https://issues.apache.org/jira/browse/KAFKA-12909

Israel, Sophie, and Guozhang did agree to this change. I don't think we
need to re-vote.

Please let us know if there are any concerns.


-Matthias

On 4/8/21 9:19 PM, Sophie Blee-Goldman wrote:
> Hey all,
> 
> This KIP has been accepted with
> 
> four +1 (binding) votes from John, Guozhang, Matthias, and myself
> four +1 (non-binding) votes from Leah, Walker, Lotz, and Israel
> 
> Thanks everyone. Israel will take it from here
> 
> On Thu, Apr 8, 2021 at 2:58 PM Sophie Blee-Goldman 
> wrote:
> 
>>> I would also like to volunteer to implement it if that is ok.
>>
>> That would be awesome -- I've been pretty overbooked lately and was
>> literally just about
>> to ask for volunteers to take over this KIP. Perfect timing :)
>>
>> The KIP still has about 4 hours to go on the voting but it looks like
>> it'll pass, so feel free to
>> start working on a PR. Thanks for volunteering!
>>
>> On Thu, Apr 8, 2021 at 2:55 PM Israel Ekpo  wrote:
>>
>>> I have reviewed the KIP. The motivation makes sense and the recommended
>>> API
>>> changes make sense as well.
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>>>
>>> So +1
>>>
>>> I would also like to volunteer to implement it if that is ok.
>>>
>>>
>>>
>>> On Thu, Apr 8, 2021 at 5:42 PM Matthias J. Sax  wrote:
>>>
 +1 (binding)

 On 4/6/21 10:15 AM, Lotz Utfpr wrote:
> Makes sense to me! +1
>
> Apologies for being brief. This email was sent from my mobile phone.
>
>> On 6 Apr 2021, at 18:45, Walker Carlson
>>> 
 wrote:
>>
>> This makes sense to me +1!
>>
>> Walker
>>
>>> On Tue, Apr 6, 2021 at 11:08 AM Guozhang Wang 
 wrote:
>>>
>>> +1. Thanks!
>>>
>>> On Tue, Apr 6, 2021 at 7:00 AM Leah Thomas
 
>>> wrote:
>>>
 Thanks for picking this up, Sophie. +1 from me, non-binding.

 Leah

> On Mon, Apr 5, 2021 at 9:42 PM John Roesler 
 wrote:

> Thanks, Sophie,
>
> I’m +1 (binding)
>
> -John
>
> On Mon, Apr 5, 2021, at 21:34, Sophie Blee-Goldman wrote:
>> Hey all,
>>
>> I'd like to start the voting on KIP-633, to drop the awkward 24
>>> hour
> grace
>> period and improve the API to raise visibility on an important
>>> concept
 in
>> Kafka Streams: grace period nad out-of-order data handling.
>>
>> Here's the KIP:
>>
>

>>>

>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24+hour+default+of+grace+period+in+Streams
>> <
>

>>>

>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-633%3A+Drop+24hr+default+grace+period
>>
>>
>> Cheers,
>> Sophie
>>
>

>>>
>>>
>>> --
>>> -- Guozhang
>>>

>>>
>>
> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #227

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Ryanne Dolan]

Magnus, I think these are arguments for observability in general, but not
why kafka should sit between a client and a metics collector.

Ryanne

On Wed, Jun 16, 2021, 10:27 AM Magnus Edenhill  wrote:

> Hi Ryanne,
>
> this proposal stems from a need to improve troubleshooting Kafka issues.
>
> As it currently stands, when an application team is experiencing Kafka
> service degradation,
> or the Kafka operator is seeing misbehaving clients, there are plenty of
> steps that needs
> to be taken before any client-side metrics can be observed at all, if at
> all:
>  - Is the application even collecting client metrics? If not it needs to be
> reconfigured or implemented, and restarted;
>a restart may have business impact, and may also temporarily? remedy the
> problem without giving any further insight
>into what was wrong.
>  - Are the desired metrics collected? Where are they stored? For how long?
> Is there enough correlating information
>to map it to cluster-side metrics and events? Does the application
> on-call know how to find the collected metrics?
>  - Export and send these metrics to whoever knows how to interpret them. In
> what format? Are all relevant metadata fields
>provided?
>
> The KIP aims to solve all these obstacles by giving the Kafka operator the
> tools to collect this information.
>
> Regards,
> Magnus
>
>
> Den tis 15 juni 2021 kl 02:37 skrev Ryanne Dolan :
>
> > Magnus, I think such a substantial change requires more motivation than
> is
> > currently provided. As I read it, the motivation boils down to this: you
> > want your clients to phone-home unless they opt-out. As stated in the
> KIP,
> > "there are plenty of existing solutions [...] to send metrics [...] to a
> > collector", so the opt-out appears to be the only motivation. Am I
> missing
> > something?
> >
> > Ryanne
> >
> > On Wed, Jun 2, 2021 at 7:46 AM Magnus Edenhill 
> wrote:
> >
> > > Hey all,
> > >
> > > I'm proposing KIP-714 to add remote Client metrics and observability.
> > > This functionality will allow centralized monitoring and
> troubleshooting
> > of
> > > clients and their internals.
> > >
> > > Please see
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
> > >
> > > Looking forward to your feedback!
> > >
> > > Regards,
> > > Magnus
> > >
> >
>

KAFKA-12889 pull request review request

[67]

hi kafka devlopers.

I have created a jira KAFKA-12889 and a pull requst 10818 for fix log cleaner 
may left empty segments about every 2^31 messages.
It has been about two weeks since the pr, but not got any review. ping some 
commiter on github but still no response.
so, can some one review this or tell me what to do next to continue this 
contribution


best wishes and thanks.


qiang

[jira] [Created] (KAFKA-12959) Prioritize assigning standby tasks to threads without any active tasks

[Ravi Bhardwaj (Jira)]

Ravi Bhardwaj created KAFKA-12959:
-

 Summary: Prioritize assigning standby tasks to threads without any 
active tasks
 Key: KAFKA-12959
 URL: https://issues.apache.org/jira/browse/KAFKA-12959
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 2.8.0
Reporter: Ravi Bhardwaj
 Attachments: app1.log, app2.log

Currently while distributing the standby tasks streams does not check if there 
are threads without any tasks or with less number of tasks. This can lead to 
few threads getting assigned both active and standby tasks when are threads 
within the same instance without any tasks assigned.

Example:
{code:java}
App 1:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
 New active tasks: [0_1]
 New standby tasks: [1_0]
 Existing active tasks: []
 Existing standby tasks: [1_0]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
 New active tasks: [1_1]
 New standby tasks: []
 Existing active tasks: [1_1]
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
{code}
 
{code:java}
App2:
[wordcount-lambda-example-client-StreamThread-1] Handle new assignment with:
 New active tasks: [1_0]
 New standby tasks: [1_1]
 Existing active tasks: []
 Existing standby tasks: [1_0, 1_1]
[wordcount-lambda-example-client-StreamThread-2] Handle new assignment with:
 New active tasks: [0_0]
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-3] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
[wordcount-lambda-example-client-StreamThread-4] Handle new assignment with:
 New active tasks: []
 New standby tasks: []
 Existing active tasks: []
 Existing standby tasks: []
{code}
 The standby tasks in both apps is assigned to Thread-1 even though it has an 
active tasks when Thread-3 and Thread-4 didn't have any tasks assigned.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12960) WindowStore and SessionStore do not enforce strict retention time

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-12960:
---

 Summary: WindowStore and SessionStore do not enforce strict 
retention time
 Key: KAFKA-12960
 URL: https://issues.apache.org/jira/browse/KAFKA-12960
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


WindowedStore and SessionStore do not implement a strict retention time in 
general. We should consider to make retention time strict: even if we still 
have some record in the store (due to the segmented implementation), we might 
want to filter expired records on-read. This might benefit PAPI users.

Atm, InMemoryWindow store does already enforce a strict retention time.

As an alternative, we could also inject such a filter in the wrapping 
`MeteredStore` – this might lift the burden from users who implement a custom 
state store.

As an alternative, we could change all DSL operators to verify if data from a 
state store is already expired or not. It might be better to push this 
responsibility into the stores though.

It's especially an issue for stream-stream joins, because the operator relies 
on the retention time to implement it's grace period.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)