The input path is fine. Problem in output path. I am just wonder that it copy the data into local disk (/user/root/) not into hdfs. I dont know why? Is it we give the correct statement to point to hdfs?
Thanks On Mon, Nov 19, 2012 at 3:10 PM, Mohammad Tariq <donta...@gmail.com> wrote: > Try this as your input file path > Path inputFile = new Path("file:///usr/Eclipse/Output.csv"); > > Regards, > Mohammad Tariq > > > > On Mon, Nov 19, 2012 at 8:31 PM, kashif khan <drkashif8...@gmail.com>wrote: > >> when I am applying the command as >> >> $ hadoop fs -put /usr/Eclipse/Output.csv /user/root/Output.csv. >> >> its work fine and file browsing in the hdfs. But i dont know why its not >> work in program. >> >> Many thanks for your cooperation. >> >> Best regards, >> >> >> >> >> On Mon, Nov 19, 2012 at 2:53 PM, Mohammad Tariq <donta...@gmail.com>wrote: >> >>> It would be good if I could have a look on the files. Meantime try some >>> other directories. Also, check the directory permissions once. >>> >>> Regards, >>> Mohammad Tariq >>> >>> >>> >>> On Mon, Nov 19, 2012 at 8:13 PM, kashif khan <drkashif8...@gmail.com>wrote: >>> >>>> >>>> I have tried through root user and made the following changes: >>>> >>>> >>>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>>> Path outputFile = new Path("/user/root/Output1.csv"); >>>> >>>> No result. The following is the log output. The log shows the >>>> destination is null. >>>> >>>> >>>> 2012-11-19 14:36:38,960 INFO FSNamesystem.audit: allowed=true >>>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo >>>> src=/user dst=null perm=null >>>> 2012-11-19 14:36:38,977 INFO FSNamesystem.audit: allowed=true >>>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>>> src=/user dst=null perm=null >>>> 2012-11-19 14:36:39,933 INFO FSNamesystem.audit: allowed=true >>>> ugi=hbase (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>>> src=/hbase/.oldlogs dst=null perm=null >>>> 2012-11-19 14:36:41,147 INFO FSNamesystem.audit: allowed=true >>>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo >>>> src=/user/root dst=null perm=null >>>> 2012-11-19 14:36:41,229 INFO FSNamesystem.audit: allowed=true >>>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>>> src=/user/root dst=null perm=null >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Nov 19, 2012 at 2:29 PM, kashif khan <drkashif8...@gmail.com>wrote: >>>> >>>>> Yeah, My cluster running. When brows http://hadoop1.example.com: >>>>> 50070/dfshealth.jsp. I am getting the main page. Then click on Brows file >>>>> system. I am getting the following: >>>>> >>>>> hbase >>>>> tmp >>>>> user >>>>> >>>>> And when click on user getting: >>>>> >>>>> beeswax >>>>> huuser (I have created) >>>>> root (I have created) >>>>> >>>>> Would you like to see my configuration file. As did not change any >>>>> things, all by default. I have installed CDH4.1 and running on VMs. >>>>> >>>>> Many thanks >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Nov 19, 2012 at 2:04 PM, Mohammad Tariq <donta...@gmail.com>wrote: >>>>> >>>>>> Is your cluster running fine? Are you able to browse Hdfs through the >>>>>> Hdfs Web Console at 50070? >>>>>> >>>>>> Regards, >>>>>> Mohammad Tariq >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Nov 19, 2012 at 7:31 PM, kashif khan >>>>>> <drkashif8...@gmail.com>wrote: >>>>>> >>>>>>> Many thanks. >>>>>>> >>>>>>> I have changed the program accordingly. It does not show any error >>>>>>> but one warring , but when I am browsing the HDFS folder, file is not >>>>>>> copied. >>>>>>> >>>>>>> >>>>>>> public class CopyData { >>>>>>> public static void main(String[] args) throws IOException{ >>>>>>> Configuration conf = new Configuration(); >>>>>>> //Configuration configuration = new Configuration(); >>>>>>> //configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>>> //configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>>> >>>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>>> conf.addResource(new Path >>>>>>> ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>>> FileSystem fs = FileSystem.get(conf); >>>>>>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>>>>>> Path outputFile = new Path("/user/hduser/Output1.csv"); >>>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>>> fs.close(); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> 19-Nov-2012 13:50:32 org.apache.hadoop.util.NativeCodeLoader <clinit> >>>>>>> WARNING: Unable to load native-hadoop library for your platform... >>>>>>> using builtin-java classes where applicable >>>>>>> >>>>>>> Have any idea? >>>>>>> >>>>>>> Many thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 19, 2012 at 1:18 PM, Mohammad Tariq >>>>>>> <donta...@gmail.com>wrote: >>>>>>> >>>>>>>> If it is just copying the files without any processing or change, >>>>>>>> you can use something like this : >>>>>>>> >>>>>>>> public class CopyData { >>>>>>>> >>>>>>>> public static void main(String[] args) throws IOException{ >>>>>>>> >>>>>>>> Configuration configuration = new Configuration(); >>>>>>>> configuration.addResource(new >>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>>>> configuration.addResource(new >>>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>>>> FileSystem fs = FileSystem.get(configuration); >>>>>>>> Path inputFile = new >>>>>>>> Path("/home/mohammad/pc/work/FFT.java"); >>>>>>>> Path outputFile = new Path("/mapout/FFT.java"); >>>>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>>>> fs.close(); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> Obviously you have to modify it as per your requirements like >>>>>>>> continuously polling the targeted directory for new files. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Mohammad Tariq >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Nov 19, 2012 at 6:23 PM, kashif khan < >>>>>>>> drkashif8...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks M Tariq >>>>>>>>> >>>>>>>>> As I am new in Java and Hadoop and have no much experience. I am >>>>>>>>> trying to first write a simple program to upload data into HDFS and >>>>>>>>> gradually move forward. I have written the following simple program to >>>>>>>>> upload the file into HDFS, I dont know why it does not working. >>>>>>>>> could you >>>>>>>>> please check it, if have time. >>>>>>>>> >>>>>>>>> import java.io.BufferedInputStream; >>>>>>>>> import java.io.BufferedOutputStream; >>>>>>>>> import java.io.File; >>>>>>>>> import java.io.FileInputStream; >>>>>>>>> import java.io.FileOutputStream; >>>>>>>>> import java.io.IOException; >>>>>>>>> import java.io.InputStream; >>>>>>>>> import java.io.OutputStream; >>>>>>>>> import java.nio.*; >>>>>>>>> //import java.nio.file.Path; >>>>>>>>> >>>>>>>>> import org.apache.hadoop.conf.Configuration; >>>>>>>>> import org.apache.hadoop.fs.FSDataInputStream; >>>>>>>>> import org.apache.hadoop.fs.FSDataOutputStream; >>>>>>>>> import org.apache.hadoop.fs.FileSystem; >>>>>>>>> import org.apache.hadoop.fs.Path; >>>>>>>>> public class hdfsdata { >>>>>>>>> >>>>>>>>> >>>>>>>>> public static void main(String [] args) throws IOException >>>>>>>>> { >>>>>>>>> try{ >>>>>>>>> >>>>>>>>> >>>>>>>>> Configuration conf = new Configuration(); >>>>>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>>>>> conf.addResource(new Path ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>>>>> FileSystem fileSystem = FileSystem.get(conf); >>>>>>>>> String source = "/usr/Eclipse/Output.csv"; >>>>>>>>> String dest = "/user/hduser/input/"; >>>>>>>>> >>>>>>>>> //String fileName = source.substring(source.lastIndexOf('/') + >>>>>>>>> source.length()); >>>>>>>>> String fileName = "Output1.csv"; >>>>>>>>> >>>>>>>>> if (dest.charAt(dest.length() -1) != '/') >>>>>>>>> { >>>>>>>>> dest = dest + "/" +fileName; >>>>>>>>> } >>>>>>>>> else >>>>>>>>> { >>>>>>>>> dest = dest + fileName; >>>>>>>>> >>>>>>>>> } >>>>>>>>> Path path = new Path(dest); >>>>>>>>> >>>>>>>>> >>>>>>>>> if(fileSystem.exists(path)) >>>>>>>>> { >>>>>>>>> System.out.println("File" + dest + " already exists"); >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> FSDataOutputStream out = fileSystem.create(path); >>>>>>>>> InputStream in = new BufferedInputStream(new >>>>>>>>> FileInputStream(new File(source))); >>>>>>>>> File myfile = new File(source); >>>>>>>>> byte [] b = new byte [(int) myfile.length() ]; >>>>>>>>> int numbytes = 0; >>>>>>>>> while((numbytes = in.read(b)) >= 0) >>>>>>>>> >>>>>>>>> { >>>>>>>>> out.write(b,0,numbytes); >>>>>>>>> } >>>>>>>>> in.close(); >>>>>>>>> out.close(); >>>>>>>>> //bos.close(); >>>>>>>>> fileSystem.close(); >>>>>>>>> } >>>>>>>>> catch(Exception e) >>>>>>>>> { >>>>>>>>> >>>>>>>>> System.out.println(e.toString()); >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> KK >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Nov 19, 2012 at 12:41 PM, Mohammad Tariq < >>>>>>>>> donta...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> You can set your cronjob to execute the program after every 5 >>>>>>>>>> sec. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Mohammad Tariq >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Nov 19, 2012 at 6:05 PM, kashif khan < >>>>>>>>>> drkashif8...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Well, I want to automatically upload the files as the files are >>>>>>>>>>> generating about every 3-5 sec and each file has size about 3MB. >>>>>>>>>>> >>>>>>>>>>> Is it possible to automate the system using put or cp command? >>>>>>>>>>> >>>>>>>>>>> I read about the flume and webHDFS but I am not sure it will >>>>>>>>>>> work or not. >>>>>>>>>>> >>>>>>>>>>> Many thanks >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Nov 19, 2012 at 12:26 PM, Alexander Alten-Lorenz < >>>>>>>>>>> wget.n...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Why do you don't use HDFS related tools like put or cp? >>>>>>>>>>>> >>>>>>>>>>>> - Alex >>>>>>>>>>>> >>>>>>>>>>>> On Nov 19, 2012, at 11:44 AM, kashif khan < >>>>>>>>>>>> drkashif8...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> > HI, >>>>>>>>>>>> > >>>>>>>>>>>> > I am generating files continuously in local folder of my base >>>>>>>>>>>> machine. How >>>>>>>>>>>> > I can now use the flume to stream the generated files from >>>>>>>>>>>> local folder to >>>>>>>>>>>> > HDFS. >>>>>>>>>>>> > I dont know how exactly configure the sources, sinks and hdfs. >>>>>>>>>>>> > >>>>>>>>>>>> > 1) location of folder where files are generating: >>>>>>>>>>>> /usr/datastorage/ >>>>>>>>>>>> > 2) name node address: htdfs://hadoop1.example.com:8020 >>>>>>>>>>>> > >>>>>>>>>>>> > Please let me help. >>>>>>>>>>>> > >>>>>>>>>>>> > Many thanks >>>>>>>>>>>> > >>>>>>>>>>>> > Best regards, >>>>>>>>>>>> > KK >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Alexander Alten-Lorenz >>>>>>>>>>>> http://mapredit.blogspot.com >>>>>>>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >