What you are actually asking is if Kafka Streams should be reimplemented as Apache Storm? -wim
On Wed, 29 Nov 2017 at 15:10 Adrienne Kole <adrienneko...@gmail.com> wrote: > Hi, > > The purpose of this email is to get overall intuition for the future plans > of streams library. > > The main question is that, will it be a single threaded application in the > long run and serve microservices use-cases, or are there any plans to > extend it to multi-node execution framework with less kafka dependency. > > Currently, each streams node 'talks' with kafka cluster and they can > indirectly talk with each other again through kafka. However, especially if > kafka is not in the same network with streams nodes (actually this can > happen if they are in the same network as well) this will cause high > network overhead and inefficiency. > > One solution for this (bypassing network overhead) is to deploy streams > node on kafka cluster to ensure the data locality. However, this is not > recommended as the library and kafka can affect each other's performance > and streams does not necessarily have to know the internal data > partitioning of kafka. > > Another solution would be extending streams library to have a common > runtime. IMO, preserving the current selling points of streams (like > dynamic scale in/out) with this kind of extensions can be very good > improvement. > > So my question is that, will streams in the long/short run, will extend its > use-cases to massive and efficient stream processing (and compete with > spark) or stay and strengthen its current position? > > Cheers, > Adrienne >
