Hello Ben, 1. Currently Kafka Streams high-level DSL does not take the StreamPartitioner yet, please feel free to file a JIRA so that we can keep track and discuss of whether / how to incorporate it into Kafka Streams DSL.
2. As for now, you can do two work arounds: 1) use `process()` function to manually write to Kafka with customized partitioning: RecordCollector collector = ((RecordCollector.Supplier) context).recordCollector(); collector.send(*...*); 2), you can call addSink() in KStreamBuilder as well since it is extending the TopologyBuilder, but you need to know the upstream processor name (i.e. the parent processor name) which is auto-created in KStreamImpl as "ProcessorType-IndexSuffix", which is a bit hacky. Guozhang On Fri, Mar 18, 2016 at 7:38 AM, Ben Osheroff <b...@zendesk.com.invalid> wrote: > (lemme know if this belongs on the users email list, I'm not sure where > API questions fall) > > Hi, I'm Ben Osheroff, I wrote Maxwell > (http://github.com/zendesk/maxwell) and have been prototyping an engine > to do arbitrary denormalizations of Maxwell's CDC events based on the > kafka-streams library; the elevator pitch is that you can write SQL > joins which the engine compiles down to stream-joins and aggregations > and such. > > Maxwell partitions its stream by mysql database name, which means that > to do stream-joins I need to implement the same (custom) partitioning > algorithm somewhere in my stream processor. I'd prefer not drop down to > the lower level `addSink()` library calls if possible, and I can't > figure out how to mix and match the lower level alls with the higher > level DSL (map/filter/etc). > > So I guess I have two questions: > > 1. Is it somehow possible to add a custom `Sink` to an otherwise high > level stream topology? There's no obvious way to retrieve the topology > names that I can see. > > 2. If not, I'd like to make a feature request that the various stream > building functions (.to, .through) accept an optional > StreamPartitioner. > > 3. Any other ideas about how to pull this off? > > Thanks! > > > - Ben Osheroff > zendesk.com > -- -- Guozhang
