Hello,
Now I am slowly understanding Hadoop working.
As I want to collect the logs from three machines
including Master itself . My small query is
which mode should I implement for this??
- Standalone Operation
- Pseudo-Distributed Operation
- Fully-Distributed Operation
Seeking for guidance,
Thank you !!
*--
Cheers,
Mayur*
Hi mayur,
>
> Flume is used for data collection. Pig is used for data processing.
> For eg, if you have a bunch of servers that you want to collect the
> logs from and push to HDFS - you would use flume. Now if you need to
> run some analysis on that data, you could use pig to do that.
>
> Sent from my iPhone
>
> On Feb 14, 2013, at 1:39 AM, Mayur Patil <ram.nath241...@gmail.com> wrote:
>
> > Hello,
> >
> > I just read about Pig
> >
> >> Pig
> >> A data flow language and execution environment for exploring very
> > large datasets.
> >> Pig runs on HDFS and MapReduce clusters.
> >
> > What the actual difference between Pig and Flume makes in logs
> clustering??
> >
> > Thank you !!
> > --
> > Cheers,
> > Mayur.
> >
> >
> >
> >> Hey Mayur,
> >>>
> >>> If you are collecting logs from multiple servers then you can use flume
> >>> for the same.
> >>>
> >>> if the contents of the logs are different in format then you can just
> >>> use
> >>> textfileinput format to read and write into any other format you want
> for
> >>> your processing in later part of your projects
> >>>
> >>> first thing you need to learn is how to setup hadoop
> >>> then you can try writing sample hadoop mapreduce jobs to read from text
> >>> file and then process them and write the results into another file
> >>> then you can integrate flume as your log collection mechanism
> >>> once you get hold on the system then you can decide more on which paths
> >>> you want to follow based on your requirements for storage, compute
> time,
> >>> compute capacity, compression etc
> >>>
> >> --------------
> >> --------------
> >>
> >>> Hi,
> >>>
> >>> Please read basics on how hadoop works.
> >>>
> >>> Then start your hands on with map reduce coding.
> >>>
> >>> The tool which has been made for you is flume , but don't see tool till
> >>> you complete above two steps.
> >>>
> >>> Good luck , keep us posted.
> >>>
> >>> Regards,
> >>>
> >>> Jagat Singh
> >>>
> >>> -----------
> >>> Sent from Mobile , short and crisp.
> >>> On 06-Feb-2013 8:32 AM, "Mayur Patil" <ram.nath241...@gmail.com>
> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> I am new to Hadoop. I am doing a project in cloud in which I
> >>>>
> >>>> have to use hadoop for Map-reduce. It is such that I am going
> >>>>
> >>>> to collect logs from 2-3 machines having different locations.
> >>>>
> >>>> The logs are also in different formats such as .rtf .log .txt
> >>>>
> >>>> Later, I have to collect and convert them to one format and
> >>>>
> >>>> collect to one location.
> >>>>
> >>>> So I am asking which module of Hadoop that I need to study
> >>>>
> >>>> for this implementation?? Or whole framework should I need
> >>>>
> >>>> to study ??
> >>>>
> >>>> Seeking for guidance,
> >>>>
> >>>> Thank you !!
>
