Hi Toni, 1. Kafka can create topics on the fly, in case you need it.
https://kafka.apache.org/08/configuration.html auto.create.topics.enabletrueEnable auto creation of topic on the server. If this is set to true then attempts to produce, consume, or fetch metadata for a non-existent topic will automatically create it with the default replication factor and number of partitions. 2. About topic selection based on rules/dictionary, this must be solved on your side. You can use custom-code in your app or an event transport solution, like Flume. Flume 1.6 now includes a sink for Kafka, and it already supports dynamic topics (by using a preprocessor) https://github.com/thilinamb/flume-ng-kafka-sink - *topic*[optional] - The topic in Kafka to which the messages will be published. If this topic is mentioned, every message will be published to the same topic. If dynamic topics are required, it's possible to use a preprocessor instead of a static topic. It's mandatory that either of the parameters *topic* or *preprocessor* is provided, because the topic cannot be null when publishing to Kafka. If none of these parameters are provided, the messages will be published to a default topic called default-flume-topic. Regards. 2015-01-29 0:16 GMT+01:00 Lakshmanan Muthuraman <lakshma...@tokbox.com>: > Hi Toni, > > Couple of thoughts. > > 1. Kafka behaviour need not be changed at run time. Your producers which > push your MAC data into kafka should know to which topic it should write. > Your producer can be flume, log stash or it can be your own custom written > java producer. > > As long as your producer know which topic to write, they can keep creating > new topics as new MAC data comes through your pipeline. > > On Wed, Jan 28, 2015 at 12:10 PM, Toni Cebrián <toni.cebr...@gmail.com> > wrote: > > > Hi, > > > > I'm starting to weight different alternatives for data ingestion and > > I'd like to know whether Kafka meets the problem I have. > > Say we have a set of devices each with its own MAC and then we > receive > > data in Kafka. There is a dictionary defined elsewhere that says each MAC > > to which topic must publish. So I have basically 2 questions: > > New MACs keep comming and the dictionary must be updated accordingly. How > > could I change this Kafka behaviour during runtime? > > A problem for the future. Say that dictionaries are so big that they > don't > > fit in memory. Are there any patterns for bookkeeping internal data > > structures and how route to them? > > > > T. > > > -- David Morales de Frías :: +34 607 010 411 :: @dmoralesdf <https://twitter.com/dmoralesdf> <http://www.stratio.com/> Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 828 6473 // www.stratio.com // *@stratiobd <https://twitter.com/StratioBD>*
