[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841079#comment-17841079 ] Stanislav Spiridonov commented on KAFKA-16585: -- Ok. Let's not make simple things complicated. If the processor is more suitable and doesn't (will not) additional overhead - it is the best option to implement such things. > No way to forward message from punctuation method in the FixedKeyProcessor > -- > > Key: KAFKA-16585 > URL: https://issues.apache.org/jira/browse/KAFKA-16585 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.6.2 >Reporter: Stanislav Spiridonov >Priority: Major > > The FixedKeyProcessorContext can forward only FixedKeyRecord. This class > doesn't have a public constructor and can be created based on existing > records. But such record usually is absent in the punctuation method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[
https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840907#comment-17840907
]
Matthias J. Sax commented on KAFKA-16585:
-
{quote}I can use the regular Processor, but as I understand it add some
overhead comparing with FixedKeyProcessor
{quote}
Where did you get this? The Processor itself does not have overhead. – The only
think that could happen downstream is, that a unnecessary repartition step
could be inserted. We are tackling this via
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-759%3A+Unneeded+repartition+canceling]
{quote}{color:#172b4d}Really, I think FixedKeyProcessor do not need to be
"ensure that the key is not changed". IMHO there is enough to have a key from
the same partition. So, if you will provide the way to generate the
{color}*FixedKeyRecord*{color:#172b4d} from any local store it will be
enough.{color}
{quote}
{color:#172b4d}Well, technically yes, but there is no simply way to
enforce/check this... We would need to serialize the provided key, pipe it
through the Partitioner, and compare the computed partition. Or is there some
other way to do this? – This would be quite expensive to do.{color}
{color:#172b4d}If you feel strong about all this, feel free to do a POC PR and
write a KIP about it, and we can take it from there. I don't see a simple way
to do it, and I believe that using a regular Processor is the right way to go
(especially with KIP-759 on the horizon). {color}
> No way to forward message from punctuation method in the FixedKeyProcessor
> --
>
> Key: KAFKA-16585
> URL: https://issues.apache.org/jira/browse/KAFKA-16585
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.6.2
>Reporter: Stanislav Spiridonov
>Priority: Major
>
> The FixedKeyProcessorContext can forward only FixedKeyRecord. This class
> doesn't have a public constructor and can be created based on existing
> records. But such record usually is absent in the punctuation method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840664#comment-17840664 ] Stanislav Spiridonov commented on KAFKA-16585: -- # I can use the regular Processor, but as I understand it add some overhead comparing with FixedKeyProcessor # Really, I think FixedKeyProcessor do not need to be "ensure that the key is not changed". IMHO there is enough to have a key from the same partition. So, if you will provide the way to generate the *FixedKeyRecord* from any local store it will be enough. # As variant FixedKeyProcessor have to use own internal (optional) store for allowed keys to share them with punktuators, and generate the *FixedKeyRecord* from it. > No way to forward message from punctuation method in the FixedKeyProcessor > -- > > Key: KAFKA-16585 > URL: https://issues.apache.org/jira/browse/KAFKA-16585 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.6.2 >Reporter: Stanislav Spiridonov >Priority: Major > > The FixedKeyProcessorContext can forward only FixedKeyRecord. This class > doesn't have a public constructor and can be created based on existing > records. But such record usually is absent in the punctuation method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839910#comment-17839910 ] Matthias J. Sax commented on KAFKA-16585: - Well, the use-case make sense, but the question is, how can the runtime ensure that the key is not changed? The idea of `FixedKeyProcessor` is to ensure that the key is not changed, but when we allow to set a key, you could set anything and the runtime cannot ensure that the key is "correct". It would be up the user-code to do the right thing... what it unclear from your use-case description is, why can't you use a regular `Processor`? > No way to forward message from punctuation method in the FixedKeyProcessor > -- > > Key: KAFKA-16585 > URL: https://issues.apache.org/jira/browse/KAFKA-16585 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.6.2 >Reporter: Stanislav Spiridonov >Priority: Major > > The FixedKeyProcessorContext can forward only FixedKeyRecord. This class > doesn't have a public constructor and can be created based on existing > records. But such record usually is absent in the punctuation method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[
https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839791#comment-17839791
]
Stanislav Spiridonov commented on KAFKA-16585:
--
If you check the example on
[https://developer.confluent.io/tutorials/kafka-streams-schedule-operations/kstreams.html]
you will see the same situation *transform* vs {*}transformValues{*}.
{noformat}
I used transform in this tutorial as it makes for a better example because you
can use the ProcessorContext.forward method.{noformat}
> No way to forward message from punctuation method in the FixedKeyProcessor
> --
>
> Key: KAFKA-16585
> URL: https://issues.apache.org/jira/browse/KAFKA-16585
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 3.6.2
>Reporter: Stanislav Spiridonov
>Priority: Major
>
> The FixedKeyProcessorContext can forward only FixedKeyRecord. This class
> doesn't have a public constructor and can be created based on existing
> records. But such record usually is absent in the punctuation method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838979#comment-17838979 ] Stanislav Spiridonov commented on KAFKA-16585: -- The case is relatively simple. I have KTable with entities that have to be enrichment with icon attribute from side service. So, the processor maintains the internal store with entities keys and periodically ask the service for update for registered ids. If icon has changes it forward the message with new icon. The key of record is entity key (String), value is a icon (String). BTW I faced into strange behaviour - if I forward new record from another thread it arrived to incorrect processor. So now I just update store from icon KTable instead of forward the record. > No way to forward message from punctuation method in the FixedKeyProcessor > -- > > Key: KAFKA-16585 > URL: https://issues.apache.org/jira/browse/KAFKA-16585 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.6.2 >Reporter: Stanislav Spiridonov >Priority: Major > > The FixedKeyProcessorContext can forward only FixedKeyRecord. This class > doesn't have a public constructor and can be created based on existing > records. But such record usually is absent in the punctuation method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17838730#comment-17838730 ] Matthias J. Sax commented on KAFKA-16585: - Thanks for raising this ticket. Wondering how we could address it though... Given that the "contract" is that the record key is not modified, but there is no input record, how could the key be set in a meaningful way? – The only thing I can think of right now would be to set `key = null`, but it's still semantically questionable... Can you provide more details what you are trying to do? > No way to forward message from punctuation method in the FixedKeyProcessor > -- > > Key: KAFKA-16585 > URL: https://issues.apache.org/jira/browse/KAFKA-16585 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 3.6.2 >Reporter: Stanislav Spiridonov >Priority: Major > > The FixedKeyProcessorContext can forward only FixedKeyRecord. This class > doesn't have a public constructor and can be created based on existing > records. But such record usually is absent in the punctuation method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
