Hi,
I wanted to provide some feedback about the new API introduced in Kafka
streaming 2.7 for adding a Processor to the Topology (using PAPI).
Prior to 2.7, the Processor was typed only by the input key values <K,V>,
and the ProcessorContext was untyped.
In 2.7 ProcessorSupplier & Processor now have the additional params <...,
KForward, VForward> and the ProcessorContext also has <KForward,
VForward>. The *only* use of this appears to be to provide some typing for
the "forward" functions.
One of my common use cases is where I have multiple child nodes each of
which has different output <K,V>. To migrate to the new 2.7 API, seems to
require adding <.., Object, Object> parameters to every type which looks
noisy in the code and does nothing for improving type safety of the API.
I would like to ask why the new type parameters were added when they do not
actually achieve typing for the multiple child use-case?
Personally, I would prefer new methods on the ProcessorContext that allow
me to create a "typed" forwarder per child node.
e.g.
public interface ProcessorContext {
/** Default forwarder. */
<K, V> Forwarder<K, V> forwarder();
/** Forwarder for a specific child. */
<K, V> Forwarder<K, V> forwarder(final String childName);
}
public interface Forwarder<K, V> {
void forward(Record<K, V> record);
}
With my suggested approach you at least get some typing per-topic (child
node). To me this also feels like it better represents the domain model of
single stream input with multiple outputs. Forwarder also feels like a
real entity in the model (similar to Producer), rather than just being a
method hacked into the context.
To me, the new 2.7 API feels like a step backwards. I also get the
impression that the streaming API continues to break each release without a
particular strong model / design behind it (particularly the sometimes
jarring disconnect between PAPI and streaming API).
Having learned from initial implementations, is it time to take a step back
to design a "V2" streaming API which blends the existing PAPI + streaming
API?
I am curious what others think?
Thanks,
Ross
