Hi, The way you describe the "in memory caching component", it looks very similar to HBase memstore. Any reason for not relying on it?
N. On Fri, Mar 16, 2012 at 4:21 PM, Kleegrewe, Christian < christian.kleegr...@siemens.com> wrote: > Dear all, > > We are currently working on an architecture for a system that should be > serve as an archive for 1000+ measuring components that frequently (~30/s) > send messages containing measurement values (~300 bytes/message). The > archiving system should be capable of not only serving as a long term > storage but also as a kind of streaming data processing and caching > component. There are several functions that should be computed on the > incoming data before finally storing it. > > We suggested an architecture that comprises of: > A message routing component that could route data to calculations and > route calculation results to other components that are interested in these > data. > An in memory caching component that is used for storing up to 10 - 20 > minutes of data before it is written to the long term archive. > An hBase database that is used for the long term storage. > MapReduce framework for doing analytics on the data stored in the hBase > database. > > The complete system should be failsafe and reliable regarding component > failures and it should scale with the number of computers that are utilized. > > Are there any suggestions or feedback to this approach from the community? > and are there any suggestions which tools or systems to use for the message > routing component and the in memory cache. > > Thanks for any help and suggestions > > all the best > > Christian > > > ----------------------------8<----------------------------------------------- > > Siemens AG > Corporate Technology > Corporate Research and Technologies > CT T DE IT3 > Otto-Hahn-Ring 6 > 81739 Munich, Germany > Tel.: +49 89 636-42722 > Fax: +49 89 636-41423 > mailto:christian.kleegr...@siemens.com > > Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard > Cromme; Managing Board: Peter Loescher, Chairman, President and Chief > Executive Officer; Roland Busch, Brigitte Ederer, Klaus Helmrich, Joe > Kaeser, Barbara Kux, Hermann Requardt, Siegfried Russwurm, Peter Y. > Solmssen, Michael Suess; Registered offices: Berlin and Munich, Germany; > Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; > WEEE-Reg.-No. DE 23691322 >
