Off the top of my head, I would: - have a Storm topology ready listening on kafka. If you have a few minutes between kafka event and delivery of processed input to clients, I would rather not waste time starting up the topology. - Not implement 1000 topologies. That's at least 1000 jvms. Is the processing in the internal bolts the same or similar enough that the whole requirement can be solved with a single topology that has 3 types of bolts, each of them paralelized as far as your cluster hardware can bear?
Can you provide more information? regards, JG On Mon, Sep 21, 2015 at 10:10 AM, bhasker datta <bhasda...@gmail.com> wrote: > > > On Mon, Sep 21, 2015 at 8:53 AM, bhasker datta <bhasda...@gmail.com> > wrote: > >> Hello, >> I am working on a storm topology where I need to build multiple >> topologies for different client locations. >> >> I have a Kafka server producing data that needs to reach remote client >> machine(after some transformation in the bolts). There are about 1000 of >> those clients. >> when there is new data produced in the Kafka, the clients need to get >> that data in few minutes. >> There is some transformation that needs to happen (which can be handled >> by the bolts) and the data is sent to the remote client via websocket. >> >> e.g: Kafka server -> Kafka spout -> Bolt1 -> bolt2 -> bolt3 -> websocket >> (on a client machine) >> I have 1000 client machines. >> >> I am hoping that this topology can be dynamically created per client. >> When the client is ready, can they initiate this topology and start reading >> the data? >> or can these topologies (1000 of them) be created on the storm and ready >> to send the data? >> >> Is this architecture possible with Storm? >> How efficient is this? >> Is the Strom setup manageable? >> >> thanks for your response. >> Bhasker >> >> > -- Javier González Nicolini
