Florian, I'd also have a follow-up question:
> Actually, the set of subtypes is not known in advance this is why I need to > compute output topics from messages. So the branch method is of any help > in my context. For a number of reasons we recommend pre-creating input, intermediate, and output topics of your Kafka Streams application, rather than relying on Kafka's topic-auto-creation feature (the most important reason is that auto-creation will always use the default topic settings, which includes the default number of partitions for topics; these defaults are configured in the broker = server-side, and they might not match what your application needs) . So, in your use case, would you not only need to "route" messages dynamically to output topics, but also need to dynamically create those output topics in the first place? On Tue, Oct 18, 2016 at 4:36 PM, Florian Hussonnois <fhussonn...@gmail.com> wrote: > Thank you Matthias for your answers. > > The mailing list that you linked shows a solution using the Processor API. > > Actually, the set of subtypes is not known in advance this is why I need to > compute output topics from messages. So the branch method is of any help in > my context. > > I think, this feature should be supported by the DSL as the Processor API > solution is not completely safe. > > > 2016-10-18 10:01 GMT+02:00 Damian Guy <damian....@gmail.com>: > > > Hi Florian, > > > > Do you know the set of subtypes in advance? I.e, could you use: > > > > KStream[] branches = stream.branch(predicates); > > > > to split the stream based on the subtypes? > > > > Thanks, > > Damian > > > > > > On Tue, 18 Oct 2016 at 00:43 Matthias J. Sax <matth...@confluent.io> > > wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA512 > > > > > > Hi, > > > > > > using DSL you cannot do this. However, if you use Processor API you > can. > > > > > > There are similar question on the mailing list already. For example: > > > http://search-hadoop.com/m/uyzND1lghNN1tzbf41&subj=kafka+ > stream+to+new+t > > > opic+based+on+message+key > > > > > > As we got this request multiple times already, it might be worth > > > adding it IMHO. Not sure what the opinion of other is? We should make > > > sure that the feature gets accepted before you put a lot of effort in > > > it. :) > > > > > > > > > - -Matthias > > > > > > On 10/17/16 2:10 PM, Florian Hussonnois wrote: > > > > Hi All, > > > > > > > > Currently, it seems not possible with KStream to produce messages > > > > to topics which are not known until runtime. > > > > > > > > For a new project I am evaluating the Kafka Connect / Kafka > > > > Streams architecture but without that feature I cannot retain the > > > > KStreams API. > > > > > > > > Our use case is pretty basic. We have xml messages in input of our > > > > topology. Each message is splitted into subtypes and formatted in > > > > Avro before being sent to a dedicated topic. > > > > > > > > So the output topics depend of the subtype of each message. > > > > > > > > I think it would be nice to add methods into the KStream interface > > > > to provide such feature. > > > > > > > > If you think that feature would be usefull I can create a jira and > > > > contribute to it. Also, do I need to create a new KIP as this > > > > requires changes on a public API ? > > > > > > > > Thanks, > > > > > > > -----BEGIN PGP SIGNATURE----- > > > Comment: GPGTools - https://gpgtools.org > > > > > > iQIcBAEBCgAGBQJYBWIhAAoJECnhiMLycopPfTQQAI69Uii5xd8KvaEo/Aeqs0Xw > > > AzdPHekdVoHANzo1h45W1x3/lnyeMU/n2v09Agsz46cxb+Xbz9NOKGqT3v9Ye0Ic > > > Eyl5yib1B6sWr41rGuYmwDH8zBoC8dPfGZiWhfXL4Sypey3RWzQlVAUWg8Ob4tqF > > > rFeubMjWp7yopKRe/7n//JHF029hVK/ePk1vdEsI+2lBI4N7q9ONT/1wKkeCAtdd > > > CCkI2WP/WbHzCcUVmOL41KoqgQFnmrH7BtLH67jumzEIR16H+ZenGZmS1uzde56E > > > 9mEsl4wmAvfB5GJu6y7JnS7FnQotw7pV7ZneQrA2q8eCZHZqs2fkXf+6ZJNYRir+ > > > rysqt8wJG69ZN9bSNO1Q6/fNbRiSjYi0I7JnzkErP6scfDKlf3bWzlw6Ejc0+iUr > > > Cd0x2m/RlCepVleMT0UZNDlJd0Ml9Q77npP1lyntHVYHjVvtZLdlB5BQYdTMAx3N > > > KCLZ+WkY2CBKcwh/KuMr9kW2eWSxH89JSwEule+1bN4vSKyBA6vtrwDoshf6N23g > > > dEhTiY5NsgkvAe1JEK6d7PLN2Tq1Tq4OJNoP8PZlqW+YSFl41klQUblo8yT1jSlF > > > iCyQS4rgNRabjBs1iZnZNoZ5eodoJMcUyWPhHUYHne+MXuSr1cNNGeNcbS5W0UyE > > > dPCe2IiY4zErzxW/Mjmw > > > =4DpY > > > -----END PGP SIGNATURE----- > > > > > > > > > -- > Florian HUSSONNOIS >
