We don't have to patch every SMT in the same release, we can definitely move incrementally. We'll just have to note in the release notes and the KIPs page ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) when the change for the KIP was applied to each SMT.
On Thu, Jun 13, 2024 at 11:03 PM Punsak Incham <pun...@mfec.co.th.invalid> wrote: > Thanks! Chris. > > One question : We will need to wait to fix all SMTs in the KIP and release > it to user, or we can split some fixed SMTs to user early? (I noticed that > the PR of Mario ready to merge and it effect only ExtractField and > InsertField) > > > On 2024/06/13 17:01:49 Chris Egerton wrote: > > Hi Punsak, > > > > If nobody has signaled their intent to contribute that work yet (which I > > believe is the case), you are welcome to take it on yourself! > > > > Cheers, > > > > Cheers > > > > On Thu, Jun 13, 2024, 12:52 Punsak Incham <pu...@mfec.co.th.invalid> > wrote: > > > > > My customers are using these SMTs, I think it can affect to their > project > > > (about data correctness) in the future. > > > > > > (You may think why I'm not developing it and send to my customers > > > directly? Because they will use only SMTs that published by Kafka.) > > > > > > On 2024/06/13 16:38:45 Punsak Incham wrote: > > > > Hi all, > > > > I noticed that the PR of Mario effected only InsertField and > > > ExtractField, so I'd like to amend others SMTs in that KIP because I > have > > > experienced to develop custom SMTs. > > > > Can I join to contribute (open PR)? > > > > > > > > On 2024/06/13 15:59:36 Greg Harris wrote: > > > > > Hey Mario, > > > > > > > > > > Thanks for updating the KIP. > > > > > +1 (binding) > > > > > > > > > > Greg > > > > > > > > > > On Thu, Jun 13, 2024 at 1:32 AM Mario Fiore Vitale < > mv...@redhat.com> > > > > > wrote: > > > > > > > > > > > Hi Greg, > > > > > > > > > > > > > However, the KIP still omits the MaskField and ValueToKey > > > > > > transformations. > > > > > > This looks like just a typo to me, should we update the KIP > before > > > closing > > > > > > the vote? > > > > > > > > > > > > Good catch. I have just updated the KIP. > > > > > > > > > > > > I think that we can close the voting and mark it as approved, > right? > > > > > > > > > > > > Thank you all. > > > > > > > > > > > > On Wed, Jun 12, 2024 at 7:21 PM Greg Harris > <gr...@aiven.io.invalid> > > > > > > wrote: > > > > > > > > > > > > > Hi Mario, > > > > > > > > > > > > > > Thank you for mentioning the earlier DISCUSS thread. I found > this > > > comment > > > > > > > from Chris, which was agreed upon and applied to the KIP: > > > > > > > > > > > > > > > Yes, I think we should just do one KIP for all the SMTs. You > > > don't have > > > > > > > to > > > > > > > > implement everything all at once or by yourself, but I don't > see > > > why we > > > > > > > > should require one or more follow-up KIPs to apply the exact > same > > > > > > changes > > > > > > > > to the SMTs we missed the first time. > > > > > > > > > > > > > > However, the KIP still omits the MaskField and ValueToKey > > > > > > transformations. > > > > > > > This looks like just a typo to me, should we update the KIP > before > > > > > > closing > > > > > > > the vote? > > > > > > > > > > > > > > Thanks, > > > > > > > Greg > > > > > > > > > > > > > > On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale < > > > mv...@redhat.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Greg, > > > > > > > > > > > > > > > > The same check was made by Chris Egerton during the > discussion > > > thread. > > > > > > > > > > > > > > > > The original KIP scope was just for the InsertField, > > > ExtractField SMTs, > > > > > > > > then we decided to enlarge the scope of only the KIP to also > > > other > > > > > > > > potential affected SMTs. > > > > > > > > > > > > > > > > As of now the PR scope, instead, is only for InsertField and > > > > > > ExtractField > > > > > > > > SMTs. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Mario. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 12, 2024 at 12:08 AM Greg Harris > > > > > > > <greg.har...@aiven.io.invalid > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Mario, > > > > > > > > > > > > > > > > > > Thanks for the KIP. I'm on-board with this KIP, I just > wanted > > > to > > > > > > > verify a > > > > > > > > > discrepancy I noticed. > > > > > > > > > > > > > > > > > > I checked all of the call-sites of Struct#get(Field) and > > > > > > > > Struct#get(String) > > > > > > > > > in Kafka, and noticed there are some call-sites which are > not > > > > > > included > > > > > > > in > > > > > > > > > the KIP. > > > > > > > > > 1. The Flatten transformation seems to already have the > > > > > > > > > "replace.null.with.default=false" behavior unconditionally. > > > > > > > > > 2. The MaskField transformation unconditionally injects > default > > > > > > values > > > > > > > > for > > > > > > > > > top-level structs. > > > > > > > > > 3. The ValueTokey transformation injects defaults for each > of > > > the > > > > > > > > > configured "fields" > > > > > > > > > 4. The Values#convertToString method injects default > values, > > > but as > > > > > > > this > > > > > > > > > isn't configurable I think we'll need to leave it as-is. > > > > > > > > > > > > > > > > > > Did you intend to add this configuration to the Flatten, > > > MaskField, > > > > > > or > > > > > > > > > ValueToKey transformations, or should those be addressed > in a > > > > > > separate > > > > > > > > > effort? > > > > > > > > > Since the "false" behavior is desirable, I think the > Flatten > > > > > > transform > > > > > > > > > could be left as-is, rather than adding a configuration > that > > > nobody > > > > > > > > needs. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Greg > > > > > > > > > > > > > > > > > > On Tue, Jun 11, 2024 at 9:24 AM Yash Mayya < > ya...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > On Tue, Jun 11, 2024 at 7:40 PM Mario Fiore Vitale < > > > > > > > mvit...@redhat.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > Just wanted to bump up this thread for visibility. > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > On Wed, May 29, 2024 at 5:31 PM Mickael Maison < > > > > > > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Mario, > > > > > > > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > > > > > > > > > > > > > Mickael > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 27, 2024 at 12:06 PM Mario Fiore Vitale < > > > > > > > > > > mvit...@redhat.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > After 7 days I received only one vote. Should I > > > suppose this > > > > > > > will > > > > > > > > > not > > > > > > > > > > > be > > > > > > > > > > > > > approved? > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2024 at 4:14 PM Chris Egerton > > > > > > > > > > <chr...@aiven.io.invalid > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP! +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, May 20, 2024 at 4:22 AM Mario Fiore > Vitale < > > > > > > > > > > > mvit...@redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to call a vote on KIP-1040 which aims > to > > > improve > > > > > > > > > > handling > > > > > > > > > > > of > > > > > > > > > > > > > > > nullable values in InsertField, ExtractField, > and > > > other > > > > > > > > > > > > transformations > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > KIP - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=303794677 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Discussion thread - > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/ggqqqjbg6ccpz8g6ztyj7oxr80q5184n > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks and regards, > > > > > > > > > > > > > > > Mario > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > > > > > Mario Fiore Vitale > > > > > > > > > > > > > > > > > > > > > > > > > > Senior Software Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > Red Hat <https://www.redhat.com/> > > > > > > > > > > > > > <https://www.redhat.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > Mario Fiore Vitale > > > > > > > > > > > > > > > > > > > > > > Senior Software Engineer > > > > > > > > > > > > > > > > > > > > > > Red Hat <https://www.redhat.com/> > > > > > > > > > > > <https://www.redhat.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Mario Fiore Vitale > > > > > > > > > > > > > > > > Senior Software Engineer > > > > > > > > > > > > > > > > Red Hat <https://www.redhat.com/> > > > > > > > > <https://www.redhat.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Mario Fiore Vitale > > > > > > > > > > > > Senior Software Engineer > > > > > > > > > > > > Red Hat <https://www.redhat.com/> > > > > > > <https://www.redhat.com/> > > > > > > > > > > > > > > > > > > > Punsak Incham > > > > Data Platform Engineer (Kafka and Confluent) > > > > MFEC > > > > > > > > > > Punsak Incham > > > Data Platform Engineer (Kafka and Confluent) MFEC > > > > > > > Punsak Incham > Data Platform Engineer (Kafka and Confluent) > MFEC >
