Not necessarily. I would avoid the term "reimplemented". Btw, Apache Storm is not also the best (streaming) system that can utilize the network and it does not support runtime scale in/out (at least by design). So, can streams preserve its current selling points (ex:dynamicity) while introducing a new communication protocol, which will include the current one (streams-kafka-streams) and the new one (streams-streams)? Are there plans over this?
On Wed, Nov 29, 2017 at 3:56 PM, Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > What you are actually asking is if Kafka Streams should be reimplemented as > Apache Storm? > -wim > > On Wed, 29 Nov 2017 at 15:10 Adrienne Kole <adrienneko...@gmail.com> > wrote: > > > Hi, > > > > The purpose of this email is to get overall intuition for the future > plans > > of streams library. > > > > The main question is that, will it be a single threaded application in > the > > long run and serve microservices use-cases, or are there any plans to > > extend it to multi-node execution framework with less kafka dependency. > > > > Currently, each streams node 'talks' with kafka cluster and they can > > indirectly talk with each other again through kafka. However, especially > if > > kafka is not in the same network with streams nodes (actually this can > > happen if they are in the same network as well) this will cause high > > network overhead and inefficiency. > > > > One solution for this (bypassing network overhead) is to deploy streams > > node on kafka cluster to ensure the data locality. However, this is not > > recommended as the library and kafka can affect each other's performance > > and streams does not necessarily have to know the internal data > > partitioning of kafka. > > > > Another solution would be extending streams library to have a common > > runtime. IMO, preserving the current selling points of streams (like > > dynamic scale in/out) with this kind of extensions can be very good > > improvement. > > > > So my question is that, will streams in the long/short run, will extend > its > > use-cases to massive and efficient stream processing (and compete with > > spark) or stay and strengthen its current position? > > > > Cheers, > > Adrienne > > >
