Hello, Now I am slowly understanding Hadoop working.
As I want to collect the logs from three machines including Master itself . My small query is which mode should I implement for this?? Standalone Operation Pseudo-Distributed Operation Fully-Distributed Operation Seeking for guidance, Thank you !! *-- Cheers, Mayur* Hi mayur, >> >> Flume is used for data collection. Pig is used for data processing. >> For eg, if you have a bunch of servers that you want to collect the >> logs from and push to HDFS - you would use flume. Now if you need to >> run some analysis on that data, you could use pig to do that. >> >> Sent from my iPhone >> >> On Feb 14, 2013, at 1:39 AM, Mayur Patil <ram.nath241...@gmail.com> >> wrote: >> >> > Hello, >> > >> > I just read about Pig >> > >> >> Pig >> >> A data flow language and execution environment for exploring very >> > large datasets. >> >> Pig runs on HDFS and MapReduce clusters. >> > >> > What the actual difference between Pig and Flume makes in logs >> clustering?? >> > >> > Thank you !! >> > -- >> > Cheers, >> > Mayur. >> > >> > >> > >> >> Hey Mayur, >> >>> >> >>> If you are collecting logs from multiple servers then you can use >> flume >> >>> for the same. >> >>> >> >>> if the contents of the logs are different in format then you can just >> >>> use >> >>> textfileinput format to read and write into any other format you want >> for >> >>> your processing in later part of your projects >> >>> >> >>> first thing you need to learn is how to setup hadoop >> >>> then you can try writing sample hadoop mapreduce jobs to read from >> text >> >>> file and then process them and write the results into another file >> >>> then you can integrate flume as your log collection mechanism >> >>> once you get hold on the system then you can decide more on which >> paths >> >>> you want to follow based on your requirements for storage, compute >> time, >> >>> compute capacity, compression etc >> >>> >> >> -------------- >> >> -------------- >> >> >> >>> Hi, >> >>> >> >>> Please read basics on how hadoop works. >> >>> >> >>> Then start your hands on with map reduce coding. >> >>> >> >>> The tool which has been made for you is flume , but don't see tool >> till >> >>> you complete above two steps. >> >>> >> >>> Good luck , keep us posted. >> >>> >> >>> Regards, >> >>> >> >>> Jagat Singh >> >>> >> >>> ----------- >> >>> Sent from Mobile , short and crisp. >> >>> On 06-Feb-2013 8:32 AM, "Mayur Patil" <ram.nath241...@gmail.com> >> wrote: >> >>> >> >>>> Hello, >> >>>> >> >>>> I am new to Hadoop. I am doing a project in cloud in which I >> >>>> >> >>>> have to use hadoop for Map-reduce. It is such that I am going >> >>>> >> >>>> to collect logs from 2-3 machines having different locations. >> >>>> >> >>>> The logs are also in different formats such as .rtf .log .txt >> >>>> >> >>>> Later, I have to collect and convert them to one format and >> >>>> >> >>>> collect to one location. >> >>>> >> >>>> So I am asking which module of Hadoop that I need to study >> >>>> >> >>>> for this implementation?? Or whole framework should I need >> >>>> >> >>>> to study ?? >> >>>> >> >>>> Seeking for guidance, >> >>>> >> >>>> Thank you !! >> > -- *Cheers, Mayur*.