Sorry I didn't make it clear in my response but market data messages are typically very small, received super-fast and (in my situation) coming from a number of sources that trigger other processes in addition to being aggregated up for large-set, real-time analysis the results of which might be run through a separate Storm Topology.
Craig Charleton [email protected] > On Nov 18, 2015, at 8:24 AM, <[email protected]> > <[email protected]> wrote: > > Hi, > > There was a point in the question that, I think, translates to : Will Storm > be useful if my data packets are small(“Messages are typically short ones”), > but they add up to the size of Big Data ? > > The answer is Yes. > > The other parts of the questions have been answered by others, I hope. > > Regards, > Prajod > > From: Craig Charleton [mailto:[email protected]] > Sent: 18 November 2015 18:39 > To: [email protected] > Cc: John Fang <[email protected]> > Subject: Re: Storm typical application > > Aliza, > > If I may, I would like to share A few random thoughts about your question. > > I worked for a large enterprise software company and our customers were > always struggling with how to use the massive amounts of data that were being > input/created by their systems to understand their business and make > decisions. Traditionally the data had to come to its final resting place > before it could be analyzed for decision support. There was no way to > reformat, clean, analyze, aggregate the data as it was flowing through the > systems, let alone for different user populations to apply their own > perspective to the "streams" without affecting the operations of others. > > That is where I see the value to large organizations. In fact, it was the > limitations of traditional enterprise systems that became obvious once > companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do > things to large volumes of data in real time. They not only needed to > perform these operations quickly, the load was continually growing, so > solutions needed to be able to scale beyond one server on an ongoing basis. > > This is what Storm is for in my opinion. I am currently implementing it to > perform a lot of operations on stock trade and quote information as it is > received from the markets. The number of stocks that need to be handled by > the system is unknown. Therefore I am able to use Storm to write the > operations once and then scale the load across an unlimited number of servers. > > Hope this wasn't too boring. > > > Craig Charleton > [email protected] > > On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <[email protected]> > wrote: > > Hi, > > Thanks for your response. I will try the examples. > > I understand that Storm can do the functionality required in my application, > yet my question is whether it is the right platform. > So far we worked with Karaf-framework for our applications, and I am trying > to understand what should be the motivations to move to Storm framework? > Is it for cases of: > · large amount of real time data processing (Big Data: files, DB, WEB > pages) over distributed machines? > · Large amount of real time events processing – usually control > protocols (network protocols – like routing protocol, VOIP protocols, SNMP, > REST) over distributed machines? > > Thanks, Aliza > > From: John Fang [mailto:[email protected]] > Sent: Wednesday, November 18, 2015 2:21 AM > To: [email protected] > Subject: 答复: Storm typical application > > Yes,storm can do it. I suggest you read some storm’ example.: > https://github.com/apache/storm/tree/master/examples/storm-starter > > > 发件人: Aliza Nagauker [mailto:[email protected]] > 发送时间: 2015年11月17日 23:23 > 收件人: [email protected] > 主题: Storm typical application > > Hello all, > > I am new at Storm. I read Storm Doc and tutorial as published in storm site > and have few basic questions. > I am trying to learn and understand whether Storm is suitable for my > application. > > Is Storm mainly intended for distributed real time applications that has to > handle "massive input data" and apply "data analytics over this data"? > Is it indented to application where the data-size is large and need analytic > over the data itself (word count, search words, convert formats, write it to > DB etc.)? > > Assuming my application is a kind of a Controller that: > �. receive messages from multiple sources: Management Systems, Network > Elements, Internal timers, Internal modules > �. Act upon these messages: update protocol –state-machines, it may > send messages to other servers/applications. > �. Messages are typically short ones – control protocols messages (Not > HTTP pages, Not Documents, Not Database info). > �. We may need to run this application in multiple machines. > > In this case, is Storm is the right choice for this application? > I understand that Storm is indeed very recommended for Distributed Real Time > application, yet, I am not sure it is intended for network applications that > are mainly control application and not Data Processing Applications (Not Big > Data applications) > > I'll appreciate your consult on this. > > Thanks, Aliza > > > > > The information contained in this electronic message and any attachments to > this message are intended for the exclusive use of the addressee(s) and may > contain proprietary, confidential or privileged information. If you are not > the intended recipient, you should not disseminate, distribute or copy this > e-mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. WARNING: Computer viruses can be transmitted via > email. The recipient should check this email and any attachments for the > presence of viruses. The company accepts no liability for any damage caused > by any virus transmitted by this email. www.wipro.com
