> and you can't output multiple records or branching logic from a
transform();

For output multiple records in transform, we are currently working on
https://issues.apache.org/jira/browse/KAFKA-4217, I think that should cover
this use case.

For branching the output in transform, I agree this is not perfect but I
think users can follow some patterns like "stream.transform().branch()",
would that work for you?


Guozhang


On Tue, Feb 14, 2017 at 8:29 AM, Mathieu Fenniak <
mathieu.fenn...@replicon.com> wrote:

> On Tue, Feb 14, 2017 at 1:14 AM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > Some thoughts on the mixture usage of DSL / PAPI:
> >
> > There were some suggestions on mixing the usage of DSL and PAPI:
> > https://issues.apache.org/jira/browse/KAFKA-3455, and after thinking it
> a
> > bit more carefully, I'd rather not recommend users following this
> pattern,
> > since in DSL this can always be achieved in process() / transform().
> Hence
> > I think it is okay to prevent such patterns in the new APIs. And for the
> > same reasons, I think we can remove KStreamBuilder#newName() from the
> > public APIs.
> >
>
> I'm not sure that things can always be achieved by process() /
> transform()... there are some limitations to these APIs.  You can't output
> from a process(), and you can't output multiple records or branching logic
> from a transform(); these are things that can be done in the PAPI quite
> easily.
>
> I definitely understand a preference for using process()/transform() where
> possible, but, they don't seem to replace the PAPI.
>
> I would love to be operating in a world that was entirely DSL.  But the DSL
> is limited, and it isn't extensible (... by any stable API).  I don't mind
> reaching into internals today and making my own life difficult to extend
> it, and I'd continue to find a way to do that if you made the APIs distinct
> and split, but I'm just expressing my preference that you not do that. :-)
>
> And about printing the topology for debuggability: I agrees this is a
> > potential drawback, and I'd suggest maintain some functionality to build
> a
> > "dry topology" as Mathieu suggested; the difficulty is that, internally
> we
> > need a different "copy" of the topology for each thread so that they will
> > not share any states, so we cannot directly pass in the topology into
> > KafkaStreams instead of the topology builder. So how about adding a
> > `ToplogyBuilder#toString` function which calls `build()` internally then
> > prints the built dry topology?
> >
>
> Well, this sounds better than KafkaStreams#toString() in that it doesn't
> require a running processor.  But I'd really love to have a simple object
> model for the topology, not a string output, so that I can output my own
> debug format.  I currently have that in the form of
> TopologyBuilder#nodeGroups() & TopologyBuilder#build(Integer).
>
> Mathieu
>



-- 
-- Guozhang

Reply via email to