Re: [RESULTS] [VOTE] Release Kafka version 2.7.2

[Luke Chen]

Hi Felixzh,
> So, Is version-2.7.2 ready now?
Not yet.
If it is released, you'll see 2.7.2 appeared in the download page here:
https://kafka.apache.org/downloads

Thank you.
Luke

On Mon, Nov 22, 2021 at 3:05 PM felixzh  wrote:

> So, Is version-2.7.2 ready now?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2021-11-12 17:43:23, "Mickael Maison"  wrote:
> >This vote passes with 6 +1 votes (3 bindings) and no 0 or -1 votes.
> >
> >+1 votes
> >PMC Members:
> >* Bill Bejeck
> >* Manikumar Reddy
> >* Konstantine Karantasis
> >
> >Committers:
> >* Tom Bentley
> >
> >Community:
> >* Israel Ekpo
> >* Luke Chen
> >
> >0 votes
> >* No votes
> >
> >-1 votes
> >* No votes
> >
> >Vote thread:
> >https://lists.apache.org/thread/whzns1rj8ythxtfyz18hog1m3y6l215d
> >
> >I'll continue with the release process and the release announcement
> >will follow in the next few days.
> >
> >Mickael
>

Re:[RESULTS] [VOTE] Release Kafka version 2.7.2

[felixzh]

So, Is version-2.7.2 ready now?

















At 2021-11-12 17:43:23, "Mickael Maison"  wrote:
>This vote passes with 6 +1 votes (3 bindings) and no 0 or -1 votes.
>
>+1 votes
>PMC Members:
>* Bill Bejeck
>* Manikumar Reddy
>* Konstantine Karantasis
>
>Committers:
>* Tom Bentley
>
>Community:
>* Israel Ekpo
>* Luke Chen
>
>0 votes
>* No votes
>
>-1 votes
>* No votes
>
>Vote thread:
>https://lists.apache.org/thread/whzns1rj8ythxtfyz18hog1m3y6l215d
>
>I'll continue with the release process and the release announcement
>will follow in the next few days.
>
>Mickael

Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer

[Luke Chen]

Hi Florin,
Thanks for the KIP.

This KIP makes sense to me. Just a comment that the motivation section is
not clearly explain why this KIP is important.
I think John already mentioned a good motivation, which is to support "not
only UTF-8".
You should put that into the KIP, and of course if you have other thoughts,
please also add them into KIP.

Also, in the "public interface" section, there are 3 "Default parsing
pattern", I think you should add 1 remaining case (false, false) to make it
complete.

Otherwise, look good to me.

Thank you.
Luke


On Sun, Nov 21, 2021 at 7:37 PM Florin Akermann 
wrote:

> Hi John,
>
> Thanks for the vote and feedback.
>
> The thought occurred to me too.
>
> Do I understand it correctly: the current version of the
> kafka-console-producer cannot be used for anything other than UTF-8 keys
> and values?
> (There is no other implementation of MessageReader other than the
> ConsoleProducer$LineMessageReader)
> In other words, currently users seem to only apply it with utf-8 strings
> for keys and values?
> This is why I figured I would not deviate from this assumption solely for
> the headers.
>
> I will happily raise another KIP / Jira if there is a need to specify other
> formats / serializers for headers, keys and/or values.
>
> Thanks,
> Florin
>
>
> On Sat, 20 Nov 2021 at 19:34, John Roesler  wrote:
>
> > Hi Florin,
> >
> > Thanks for the KIP!
> >
> > I think the assumption that header values are UTF-8 strings might not
> hold
> > up in the long run, but it seems like we can easily add a property later
> to
> > specify the format. It seems like this scope is probably a handy addition
> > on its own.
> >
> > I’m +1 (binding)
> >
> > Thanks,
> > John
> >
> >
> > On Fri, Nov 19, 2021, at 15:06, flo wrote:
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer
> > >
> >
>

[jira] [Resolved] (KAFKA-12973) Update KIP and dev mailing list

[Jose Armando Garcia Sancio (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Armando Garcia Sancio resolved KAFKA-12973.

Resolution: Fixed

> Update KIP and dev mailing list
> ---
>
> Key: KAFKA-12973
> URL: https://issues.apache.org/jira/browse/KAFKA-12973
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jose Armando Garcia Sancio
>Assignee: Jose Armando Garcia Sancio
>Priority: Major
>
> Update KIP-630 and the Kafka mailing list based on the small implementation 
> deviations from what is documented in the KIP.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Kafka PMC Member: Tom Bentley

[Bill Bejeck]

Congrats Tom!

On Sat, Nov 20, 2021 at 2:37 AM Dongjin Lee  wrote:

> Congratulations, Tom!
>
> Thanks,
> Dongjin
>
> On Sat, Nov 20, 2021 at 9:50 AM Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 11/19/21 2:29 AM, Tom Bentley wrote:
> > > Thanks folks!
> > >
> > > On Fri, Nov 19, 2021 at 5:16 AM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > >> Congratulations Tom!
> > >>
> > >> On Fri, 19 Nov 2021 at 05:53, John Roesler 
> wrote:
> > >>
> > >>> Congratulations, Tom!
> > >>>
> > >>> On Thu, Nov 18, 2021, at 17:53, Konstantine Karantasis wrote:
> >  Congratulations Tom!
> > 
> >  Konstantine
> > 
> > 
> >  On Thu, Nov 18, 2021 at 2:44 PM Luke Chen 
> wrote:
> > 
> > > Congrats, Tom!
> > >
> > > Guozhang Wang  於 2021年11月19日 週五 上午1:13 寫道：
> > >
> > >> Congrats Tom!
> > >>
> > >> Guozhang
> > >>
> > >> On Thu, Nov 18, 2021 at 7:49 AM Jun Rao  >
> > > wrote:
> > >>
> > >>> Hi, Everyone,
> > >>>
> > >>> Tom Bentley has been a Kafka committer since Mar. 15,  2021. He
> > >> has
> > > been
> > >>> very instrumental to the community since becoming a committer.
> > >> It's
> > >>> my
> > >>> pleasure to announce that Tom is now a member of Kafka PMC.
> > >>>
> > >>> Congratulations Tom!
> > >>>
> > >>> Jun
> > >>> on behalf of Apache Kafka PMC
> > >>>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> > >>>
> > >>
> > >
> >
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Re: Do we want to add more SMTs to Apache Kafka?

[Joshua Grisham]

Hi all,

>From my perspective I think that the type of transformations which are
already covered by the existing SMTs is quite good (but anyone else please
say if you feel like you are missing something that feels "standard"), but
the biggest issue is the limitations that many of them have which makes
their usage extremely limited when trying to use them in a real production
scenario.

In my mind, the single biggest gap is the inability to handle nested fields
or anything more than records that essentially look like simple key-value
pairs. (However one exception being if you chain the flatten transform
first then you can apply others on the flattened result, but this is
assuming that the flatten transform can actually handle the message first!
If you have nested arrays then you are toast ;) And wait, maybe you didn't
actually want to flatten anyway?).

I am not sure the best way to approach this (e.g. allow for some kind of
path notation so users can address nested fields directly vs allow for
recursion to match a field name at no matter what level, or both, or
something else?) but I would say that some kind of standardized approach
that was implemented in all of the SMTs (where it makes sense) would
certainly be best! (at least, from a user perspective that the
configuration to address nested fields is consistent across each transform
that allows it).  I did this one way in a proposed change for KIP-683 but
this is only one of the possible ways (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
)

Past that, there are a few tweaks or enhancements which could be made to
some of the existing SMTs which would help prevent them from blocking or
failing for most general scenarios (for example some of the changes I had
proposed in the past but haven't since had the time to follow up on them
fit in this category I think), for example the ability to "cast" a more
complicated structure (such as an array) as a string (Connect API or JSON)
so the record can then be flattened and be inserted into a database table
or something similar will open up a lot of what is IMO currently roadblocks
that users might often hit in Sink scenarios.

Then there are some small tweaks which maybe can be made for specific
cases, some of which Randall already mentioned, such as:

* The Filter implementation is very limited to use mostly due to lack of
some "standard-feeling" predicates (field value filtering is very often
what I think people are looking for) so often the Confluent or other one is
used instead.
* A bit more can be done with InsertField IMO (e.g. giving a wallclock
timestamp instead of the record's produced timestamp is one example that
often seems to pop up).
* Some standardized way to "move" one field to another place e.g. to move
it out of or into a nested record.
* Limitations on only processing one field per transformation, e.g. with
the TimestampConverter like I had proposed with KIP-682 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats)
are just a little annoying feeling and can add to processing time in high
volume scenarios.

(By the way apologies to Randall that I have not had a chance to get back
yet on KIP-682 but will try to do so in the discussion thread in the coming
days if I can!)

And finally I also feel like some of the SMTs are a bit disjointed from
each other when it comes to how the classes are actually designed and how
the configuration works when using them (both from a user implementing the
transform, and a transform developer perspective). Some of the class design
difference might be necessary due to the nature of the transformation
itself, but I wonder if in the future some kind of standardization could be
built into a type of base class or something instead, or some enhancements
to the requirements specified by the interface, which would help to drive a
more standardized approach?  Or maybe at least just a once-through on the
code for all of them to align things like how Config string constants/enums
etc are handled, method names and position within the code, that they are
all refactored in a similar way, etc.

In the end, I do feel it makes sense to try and sort of aim for the 80/20
rule with the standard SMTs to be able to support "real world" scenarios,
but some of these limitations cause them to fall a bit short today.

Hope this is helpful at least to spark other ideas anyway!

Have a nice (rest of the) weekend!
Joshua Grisham


Den lör 20 nov. 2021 kl 01:16 skrev Brandon Brown :

> I agree, if the desire is to keep the internal SMTs collection small then
> providing an ease of discovery like Gunnar suggestions would be extremely
> helpful.
>
> Brandon Brown
>
> > On Nov 19, 2021, at 6:13 PM, Gunnar Morling
>  wrote:
> >
> > Hi all,
> >
> > Just 

Re: [VOTE] KIP-798 Add possibility to write kafka headers in Kafka Console Producer

[Florin Akermann]

Hi John,

Thanks for the vote and feedback.

The thought occurred to me too.

Do I understand it correctly: the current version of the
kafka-console-producer cannot be used for anything other than UTF-8 keys
and values?
(There is no other implementation of MessageReader other than the
ConsoleProducer$LineMessageReader)
In other words, currently users seem to only apply it with utf-8 strings
for keys and values?
This is why I figured I would not deviate from this assumption solely for
the headers.

I will happily raise another KIP / Jira if there is a need to specify other
formats / serializers for headers, keys and/or values.

Thanks,
Florin


On Sat, 20 Nov 2021 at 19:34, John Roesler  wrote:

> Hi Florin,
>
> Thanks for the KIP!
>
> I think the assumption that header values are UTF-8 strings might not hold
> up in the long run, but it seems like we can easily add a property later to
> specify the format. It seems like this scope is probably a handy addition
> on its own.
>
> I’m +1 (binding)
>
> Thanks,
> John
>
>
> On Fri, Nov 19, 2021, at 15:06, flo wrote:
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer
> >
>