This seems pretty similar to what you’re asking about: http://data-artisans.com/extending-the-yahoo-streaming-benchmark/
Especially the part where they “...directly exposed the in-flight windows to be queried”, as that sounds like what you meant by “The data cache should have the capability to serve the historical data in milliseconds” — Ken > On Apr 18, 2016, at 10:03pm, Deepak Sharma <deepakmc...@gmail.com> wrote: > > Hi all, > I am looking for an architecture to ingest 10 mils of messages in the real > time streaming mode. > If anyone has worked on similar kind of architecture , can you please point > me to any documentation around the same like what should be the architecture > , which all components/big data ecosystem tools should i consider etc. > The messages has to be in xml/json format , a preprocessor engine or message > enhancer and then finally a processor. > I thought about using data cache as well for serving the data > The data cache should have the capability to serve the historical data in > milliseconds (may be upto 30 days of data) > > > -- > Thanks > Deepak -------------------------- Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr
