Re: Project Structure, Packaging, and Deployment

2015-04-28 Thread Nathan Leung
If you prefer not to use a separate main class for each topology you can try something like Flux: https://github.com/ptgoetz/flux. Note that it is a work in progress. You will still need to have a yaml document for each topology. With regards to code hierarchy, I find it good to have common libr

Re: Project Structure, Packaging, and Deployment

2015-04-28 Thread Mason Yu
GM Sandon: I can empathize with your challenges. In a previous lifetime, I worked with Websphere 2.0 through 8.X, portals, process server, app server. One installation had over 500+ WAS instances. Pain X 10. Distributed computing is a sword with two edges. Scaling is te

Project Structure, Packaging, and Deployment

2015-04-28 Thread Sandon Jacobs
My company is using storm for various stream-processing solutions, mostly ingesting data from Kafka topics. We have chosen to implement our topologies in Scala, using APIs like Tormenta and Summingbird in the mix as well. We have about 9-10 topologies running in production as we speak. I find t