At the time, we chose storm because of a few reasons: - Metron inherited its codebase from OpenSOC, which chose Storm as it predated flink and spark streaming, the two other major contenders in the hadoop stack - Storm was battle tested at the time and, at least then, we had some concern around the maturity and performance of the other contenders - Specifically alternatives involving microbatching with high-throughput topologies like pcap was a concern
Going forward and looking back, I think it was a fine choice, honestly. Ultimately, it quickly became apparent that it was necessary to produce a simplified abstraction on top of the streaming technologies rather than adopting their abstractions as the core point of extension for the architecture. So, for instance, we quickly realized that it was a better experience for people to use Stellar to add enrichments rather than to either modify code *or* flux files directly and possibly bounce topologies. This allowed us to not concern the user with the streaming technology and also, fwiw, makes it easy to pivot to another streaming technology if we so desire in the future. Hope that provides some clarity! Thanks for the very interesting question. :) Casey On Fri, Jan 12, 2018 at 6:29 PM, M. Aaron Bossert <maboss...@gmail.com> wrote: > Perhaps it might be useful for you to articulate your use case? Not to > sound like a generic non-answer, but most of the streaming/CEP frameworks > are pretty good, narrowing down a short list of which ones to use beyond > basic requirements can be highly subjective: does your language of choice > have a “first class” API? do you already have a Hadoop environment that is > more tightly integrated with one framework or another? Do you prefer a > newer framework with newer features and a less mature code base? Or do you > prefer a well tested/tried framework? > > All of these choices boils down to much more than a quantitative > evaluation based on benchmark performance... > > Get Outlook for iOS <https://aka.ms/o0ukef> > ------------------------------ > *From:* Tarik Courdy <tarik.cou...@gmail.com> > *Sent:* Friday, January 12, 2018 5:15:13 PM > *To:* user@metron.apache.org > *Subject:* Motivations for using Apache Storm? > > Good afternoon - > > I've started doing research on various stream processing frameworks and it > seems like there are a ton of them out there. > > Out of curiosity what were the underlying motivations to go with Storm as > opposed to one of the other frameworks out there? > > Thank you for your time. >
