You first need to copy data using copyFromLocal to your HDFS and then you can utilize PIG and Hive program for further analysis which run on map reduce. Yes warehouse directory is in HDFS. If you want to run(test) PIG in local then in that case you don't to copy data to HDFS Sent from my BlackBerry, pls excuse typo
-----Original Message----- From: Michael Wang <michael.w...@meredith.com> Date: Mon, 14 May 2012 18:43:47 To: common-user@hadoop.apache.org<common-user@hadoop.apache.org> Reply-To: common-user@hadoop.apache.org Subject: RE: How to load raw log file into HDFS? I have the same question and I am glad to get you guys' help. I am also novice in Hadoop :) I am using pig and hive to analyze the logs. My logs are in <LOCAL_FILE_PATH>. Do I need to use "hadoop fs -copyFromLocal" to put files to <HDFS_FILE_PATH> first, and then load data files to pig or hive from <HDFS_FILE_PATH>? Or can just load logs from Local_file_path directly to pig or hive? After I load the files to hive, I found it is put at /user/hive/warehouse. Is /user/hive/warehouse a HDFS? How do I know what <HDFS_FILE_PATH> are available? -----Original Message----- From: Alexander Fahlke [mailto:alexander.fahlke.mailingli...@googlemail.com] Sent: Monday, May 14, 2012 1:53 AM To: common-user@hadoop.apache.org Subject: Re: How to load raw log file into HDFS? Hi, the best would be to read the documentation and some books to get familar with Hadoop. One of my favourite books is "Hadoop in Action" from Manning ( http://www.manning.com/lam/) This book has an exmple for putting (log)-files into HDFS. Check out the source "listing-3-1" Later you can also check out Cloudera's Flume: https://github.com/cloudera/flume/wiki -- BR Alexander Fahlke Java Developer www.nurago.com | www.fahlke.org On Mon, May 14, 2012 at 7:24 AM, Amith D K <amit...@huawei.com> wrote: > U can even use put/copyFromLocal > > both are similar and does the job via terminal. > > Or u can write a simple client program to do the job :) > > Amith > > > ________________________________________ > From: samir das mohapatra [samir.help...@gmail.com] > Sent: Sunday, May 13, 2012 9:13 PM > To: common-user@hadoop.apache.org > Subject: Re: How to load raw log file into HDFS? > > Hi > To load any file from local > Command: > syntax: hadoop fs -copyFromLocal <LOCAL_FILE_PATH> <HDFS_FILE_PATH> > Example hadoop fs -copyFromLocal input/logs > hdfs://localhost/user/dataset/ > > More Commans: > http://hadoop.apache.org/common/docs/r0.17.1/hdfs_shell.html > > > On Sun, May 13, 2012 at 9:53 AM, AnExplorer <satishtha...@gmail.com> > wrote: > > > > > Hi, I am novice in Hadoop. Kindly suggest how do we load log files into > > hdfs. > > Please suggest the command and steps. > > Thanks in advance!! > > -- > > View this message in context: > > > http://old.nabble.com/How-to-load-raw-log-file-into-HDFS--tp33815208p33815208.html > > Sent from the Hadoop core-user mailing list archive at Nabble.com. > > > > > This electronic message, including any attachments, may contain proprietary, confidential or privileged information for the sole use of the intended recipient(s). You are hereby notified that any unauthorized disclosure, copying, distribution, or use of this message is prohibited. If you have received this message in error, please immediately notify the sender by reply e-mail and delete it.
