Re: Hadoop Real time help

2012-08-22 Thread Niels Basjes
Thanks for the pointers, I have stuff to read now :) On Mon, Aug 20, 2012 at 9:37 AM, Bertrand Dechoux wrote: > The terms are > * ESP : http://en.wikipedia.org/wiki/Event_stream_processing > * CEP : http://en.wikipedia.org/wiki/Complex_event_processing > > By the way, processing streams in real t

Re: Hadoop Real time help

2012-08-20 Thread Mohit Anchlia
One of the most commonly used use case is to perform all IO intensive batch jobs in HDFS and load more structured data or the output of the job into HBase or Solr for quick access. But if your dataset is small that fits into memory then you could also cache it in memory. There are various options d

Re: Hadoop Real time help

2012-08-20 Thread Bertrand Dechoux
The terms are * ESP : http://en.wikipedia.org/wiki/Event_stream_processing * CEP : http://en.wikipedia.org/wiki/Complex_event_processing By the way, processing streams in real time tends toward being a pleonasm. MapReduce follows a batch architecture. You keep data until a given time. You then pr

Re: Hadoop Real time help

2012-08-19 Thread Niels Basjes
Is there a "complete" overview of the tools that allow processing streams of data in realtime? Or even better; what are the terms to google for? -- Met vriendelijke groet, Niels Basjes (Verstuurd vanaf mobiel ) Op 19 aug. 2012 18:22 schreef "Bertrand Dechoux" het volgende: > That's a good ques

Re: Hadoop Real time help

2012-08-19 Thread Bertrand Dechoux
Lucene allows you to build a kind of inverted index "content to document identifier". Solr or ElasticSearch allows to scale the process. However, if I am reading it correctly, you are saying that you can not pre compute a structure (such an index) before the search? If that's true and that you ne

Re: Hadoop Real time help

2012-08-19 Thread mahout user
Thanks Mohit and Bertrand, I am looking into hadoop for search engine as many others. But in case of search engine, I know lucene is there. But in my case i have implemented java classes, they are searching from databases as well as from csv files. But i cant understand if there are GB's of

Re: Hadoop Real time help

2012-08-19 Thread Mohit Anchlia
On Sun, Aug 19, 2012 at 8:44 AM, mahout user wrote: > Hello folks, > > >I am new to hadoop, I just want to get information that how hadoop > framework is usefull for real time service.?can any one explain me..? > > Thanks. > Can you specify your use case? Each use case calls for different de

Re: Hadoop Real time help

2012-08-19 Thread Bertrand Dechoux
That's a good question. More and more people are talking about Hadoop Real Time. One key aspect of this question is whether we are talking about MapReduce or not. MapReduce greatly improves the response time of any data intensive jobs but it is still a batch framework with a noticeable latency. T

Hadoop Real time help

2012-08-19 Thread mahout user
Hello folks, I am new to hadoop, I just want to get information that how hadoop framework is usefull for real time service.?can any one explain me..? Thanks.