We can consider adding this feature to with a StreamsAdminClient that we are adding as part of KAFKA-4060. However, I'm still not sure if it should be added on the DSL layer or on the Processor API layer.
Florian, what do you mean that the Processor is not "completely safe"? Do you mean not strong typed? I'm wondering why that would be an issue if you just want to dynamically create topics on-the-fly based on the message content? Guozhang On Tue, Oct 18, 2016 at 7:36 AM, Florian Hussonnois <fhussonn...@gmail.com> wrote: > Thank you Matthias for your answers. > > The mailing list that you linked shows a solution using the Processor API. > > Actually, the set of subtypes is not known in advance this is why I need to > compute output topics from messages. So the branch method is of any help in > my context. > > I think, this feature should be supported by the DSL as the Processor API > solution is not completely safe. > > > 2016-10-18 10:01 GMT+02:00 Damian Guy <damian....@gmail.com>: > > > Hi Florian, > > > > Do you know the set of subtypes in advance? I.e, could you use: > > > > KStream[] branches = stream.branch(predicates); > > > > to split the stream based on the subtypes? > > > > Thanks, > > Damian > > > > > > On Tue, 18 Oct 2016 at 00:43 Matthias J. Sax <matth...@confluent.io> > > wrote: > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA512 > > > > > > Hi, > > > > > > using DSL you cannot do this. However, if you use Processor API you > can. > > > > > > There are similar question on the mailing list already. For example: > > > http://search-hadoop.com/m/uyzND1lghNN1tzbf41&subj=kafka+ > stream+to+new+t > > > opic+based+on+message+key > > > > > > As we got this request multiple times already, it might be worth > > > adding it IMHO. Not sure what the opinion of other is? We should make > > > sure that the feature gets accepted before you put a lot of effort in > > > it. :) > > > > > > > > > - -Matthias > > > > > > On 10/17/16 2:10 PM, Florian Hussonnois wrote: > > > > Hi All, > > > > > > > > Currently, it seems not possible with KStream to produce messages > > > > to topics which are not known until runtime. > > > > > > > > For a new project I am evaluating the Kafka Connect / Kafka > > > > Streams architecture but without that feature I cannot retain the > > > > KStreams API. > > > > > > > > Our use case is pretty basic. We have xml messages in input of our > > > > topology. Each message is splitted into subtypes and formatted in > > > > Avro before being sent to a dedicated topic. > > > > > > > > So the output topics depend of the subtype of each message. > > > > > > > > I think it would be nice to add methods into the KStream interface > > > > to provide such feature. > > > > > > > > If you think that feature would be usefull I can create a jira and > > > > contribute to it. Also, do I need to create a new KIP as this > > > > requires changes on a public API ? > > > > > > > > Thanks, > > > > > > > -----BEGIN PGP SIGNATURE----- > > > Comment: GPGTools - https://gpgtools.org > > > > > > iQIcBAEBCgAGBQJYBWIhAAoJECnhiMLycopPfTQQAI69Uii5xd8KvaEo/Aeqs0Xw > > > AzdPHekdVoHANzo1h45W1x3/lnyeMU/n2v09Agsz46cxb+Xbz9NOKGqT3v9Ye0Ic > > > Eyl5yib1B6sWr41rGuYmwDH8zBoC8dPfGZiWhfXL4Sypey3RWzQlVAUWg8Ob4tqF > > > rFeubMjWp7yopKRe/7n//JHF029hVK/ePk1vdEsI+2lBI4N7q9ONT/1wKkeCAtdd > > > CCkI2WP/WbHzCcUVmOL41KoqgQFnmrH7BtLH67jumzEIR16H+ZenGZmS1uzde56E > > > 9mEsl4wmAvfB5GJu6y7JnS7FnQotw7pV7ZneQrA2q8eCZHZqs2fkXf+6ZJNYRir+ > > > rysqt8wJG69ZN9bSNO1Q6/fNbRiSjYi0I7JnzkErP6scfDKlf3bWzlw6Ejc0+iUr > > > Cd0x2m/RlCepVleMT0UZNDlJd0Ml9Q77npP1lyntHVYHjVvtZLdlB5BQYdTMAx3N > > > KCLZ+WkY2CBKcwh/KuMr9kW2eWSxH89JSwEule+1bN4vSKyBA6vtrwDoshf6N23g > > > dEhTiY5NsgkvAe1JEK6d7PLN2Tq1Tq4OJNoP8PZlqW+YSFl41klQUblo8yT1jSlF > > > iCyQS4rgNRabjBs1iZnZNoZ5eodoJMcUyWPhHUYHne+MXuSr1cNNGeNcbS5W0UyE > > > dPCe2IiY4zErzxW/Mjmw > > > =4DpY > > > -----END PGP SIGNATURE----- > > > > > > > > > -- > Florian HUSSONNOIS > -- -- Guozhang
