Try this as your input file path Path inputFile = new Path("file:///usr/Eclipse/Output.csv");
Regards, Mohammad Tariq On Mon, Nov 19, 2012 at 8:31 PM, kashif khan <drkashif8...@gmail.com> wrote: > when I am applying the command as > > $ hadoop fs -put /usr/Eclipse/Output.csv /user/root/Output.csv. > > its work fine and file browsing in the hdfs. But i dont know why its not > work in program. > > Many thanks for your cooperation. > > Best regards, > > > > > On Mon, Nov 19, 2012 at 2:53 PM, Mohammad Tariq <donta...@gmail.com>wrote: > >> It would be good if I could have a look on the files. Meantime try some >> other directories. Also, check the directory permissions once. >> >> Regards, >> Mohammad Tariq >> >> >> >> On Mon, Nov 19, 2012 at 8:13 PM, kashif khan <drkashif8...@gmail.com>wrote: >> >>> >>> I have tried through root user and made the following changes: >>> >>> >>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>> Path outputFile = new Path("/user/root/Output1.csv"); >>> >>> No result. The following is the log output. The log shows the >>> destination is null. >>> >>> >>> 2012-11-19 14:36:38,960 INFO FSNamesystem.audit: allowed=true >>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo >>> src=/user dst=null perm=null >>> 2012-11-19 14:36:38,977 INFO FSNamesystem.audit: allowed=true >>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>> src=/user dst=null perm=null >>> 2012-11-19 14:36:39,933 INFO FSNamesystem.audit: allowed=true >>> ugi=hbase (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>> src=/hbase/.oldlogs dst=null perm=null >>> 2012-11-19 14:36:41,147 INFO FSNamesystem.audit: allowed=true >>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=getfileinfo >>> src=/user/root dst=null perm=null >>> 2012-11-19 14:36:41,229 INFO FSNamesystem.audit: allowed=true >>> ugi=dr.who (auth:SIMPLE) ip=/134.91.36.41 cmd=listStatus >>> src=/user/root dst=null perm=null >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> >>> On Mon, Nov 19, 2012 at 2:29 PM, kashif khan <drkashif8...@gmail.com>wrote: >>> >>>> Yeah, My cluster running. When brows http://hadoop1.example.com: >>>> 50070/dfshealth.jsp. I am getting the main page. Then click on Brows file >>>> system. I am getting the following: >>>> >>>> hbase >>>> tmp >>>> user >>>> >>>> And when click on user getting: >>>> >>>> beeswax >>>> huuser (I have created) >>>> root (I have created) >>>> >>>> Would you like to see my configuration file. As did not change any >>>> things, all by default. I have installed CDH4.1 and running on VMs. >>>> >>>> Many thanks >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Nov 19, 2012 at 2:04 PM, Mohammad Tariq <donta...@gmail.com>wrote: >>>> >>>>> Is your cluster running fine? Are you able to browse Hdfs through the >>>>> Hdfs Web Console at 50070? >>>>> >>>>> Regards, >>>>> Mohammad Tariq >>>>> >>>>> >>>>> >>>>> On Mon, Nov 19, 2012 at 7:31 PM, kashif khan >>>>> <drkashif8...@gmail.com>wrote: >>>>> >>>>>> Many thanks. >>>>>> >>>>>> I have changed the program accordingly. It does not show any error >>>>>> but one warring , but when I am browsing the HDFS folder, file is not >>>>>> copied. >>>>>> >>>>>> >>>>>> public class CopyData { >>>>>> public static void main(String[] args) throws IOException{ >>>>>> Configuration conf = new Configuration(); >>>>>> //Configuration configuration = new Configuration(); >>>>>> //configuration.addResource(new >>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>> //configuration.addResource(new >>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>> >>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>> conf.addResource(new Path ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>> FileSystem fs = FileSystem.get(conf); >>>>>> Path inputFile = new Path("/usr/Eclipse/Output.csv"); >>>>>> Path outputFile = new Path("/user/hduser/Output1.csv"); >>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>> fs.close(); >>>>>> } >>>>>> } >>>>>> >>>>>> 19-Nov-2012 13:50:32 org.apache.hadoop.util.NativeCodeLoader <clinit> >>>>>> WARNING: Unable to load native-hadoop library for your platform... >>>>>> using builtin-java classes where applicable >>>>>> >>>>>> Have any idea? >>>>>> >>>>>> Many thanks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Nov 19, 2012 at 1:18 PM, Mohammad Tariq >>>>>> <donta...@gmail.com>wrote: >>>>>> >>>>>>> If it is just copying the files without any processing or change, >>>>>>> you can use something like this : >>>>>>> >>>>>>> public class CopyData { >>>>>>> >>>>>>> public static void main(String[] args) throws IOException{ >>>>>>> >>>>>>> Configuration configuration = new Configuration(); >>>>>>> configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/core-site.xml")); >>>>>>> configuration.addResource(new >>>>>>> Path("/home/mohammad/hadoop-0.20.205/conf/hdfs-site.xml")); >>>>>>> FileSystem fs = FileSystem.get(configuration); >>>>>>> Path inputFile = new Path("/home/mohammad/pc/work/FFT.java"); >>>>>>> Path outputFile = new Path("/mapout/FFT.java"); >>>>>>> fs.copyFromLocalFile(inputFile, outputFile); >>>>>>> fs.close(); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> Obviously you have to modify it as per your requirements like >>>>>>> continuously polling the targeted directory for new files. >>>>>>> >>>>>>> Regards, >>>>>>> Mohammad Tariq >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 19, 2012 at 6:23 PM, kashif khan <drkashif8...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Thanks M Tariq >>>>>>>> >>>>>>>> As I am new in Java and Hadoop and have no much experience. I am >>>>>>>> trying to first write a simple program to upload data into HDFS and >>>>>>>> gradually move forward. I have written the following simple program to >>>>>>>> upload the file into HDFS, I dont know why it does not working. could >>>>>>>> you >>>>>>>> please check it, if have time. >>>>>>>> >>>>>>>> import java.io.BufferedInputStream; >>>>>>>> import java.io.BufferedOutputStream; >>>>>>>> import java.io.File; >>>>>>>> import java.io.FileInputStream; >>>>>>>> import java.io.FileOutputStream; >>>>>>>> import java.io.IOException; >>>>>>>> import java.io.InputStream; >>>>>>>> import java.io.OutputStream; >>>>>>>> import java.nio.*; >>>>>>>> //import java.nio.file.Path; >>>>>>>> >>>>>>>> import org.apache.hadoop.conf.Configuration; >>>>>>>> import org.apache.hadoop.fs.FSDataInputStream; >>>>>>>> import org.apache.hadoop.fs.FSDataOutputStream; >>>>>>>> import org.apache.hadoop.fs.FileSystem; >>>>>>>> import org.apache.hadoop.fs.Path; >>>>>>>> public class hdfsdata { >>>>>>>> >>>>>>>> >>>>>>>> public static void main(String [] args) throws IOException >>>>>>>> { >>>>>>>> try{ >>>>>>>> >>>>>>>> >>>>>>>> Configuration conf = new Configuration(); >>>>>>>> conf.addResource(new Path("/etc/hadoop/conf/core-site.xml")); >>>>>>>> conf.addResource(new Path ("/etc/hadoop/conf/hdfs-site.xml")); >>>>>>>> FileSystem fileSystem = FileSystem.get(conf); >>>>>>>> String source = "/usr/Eclipse/Output.csv"; >>>>>>>> String dest = "/user/hduser/input/"; >>>>>>>> >>>>>>>> //String fileName = source.substring(source.lastIndexOf('/') + >>>>>>>> source.length()); >>>>>>>> String fileName = "Output1.csv"; >>>>>>>> >>>>>>>> if (dest.charAt(dest.length() -1) != '/') >>>>>>>> { >>>>>>>> dest = dest + "/" +fileName; >>>>>>>> } >>>>>>>> else >>>>>>>> { >>>>>>>> dest = dest + fileName; >>>>>>>> >>>>>>>> } >>>>>>>> Path path = new Path(dest); >>>>>>>> >>>>>>>> >>>>>>>> if(fileSystem.exists(path)) >>>>>>>> { >>>>>>>> System.out.println("File" + dest + " already exists"); >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> FSDataOutputStream out = fileSystem.create(path); >>>>>>>> InputStream in = new BufferedInputStream(new FileInputStream(new >>>>>>>> File(source))); >>>>>>>> File myfile = new File(source); >>>>>>>> byte [] b = new byte [(int) myfile.length() ]; >>>>>>>> int numbytes = 0; >>>>>>>> while((numbytes = in.read(b)) >= 0) >>>>>>>> >>>>>>>> { >>>>>>>> out.write(b,0,numbytes); >>>>>>>> } >>>>>>>> in.close(); >>>>>>>> out.close(); >>>>>>>> //bos.close(); >>>>>>>> fileSystem.close(); >>>>>>>> } >>>>>>>> catch(Exception e) >>>>>>>> { >>>>>>>> >>>>>>>> System.out.println(e.toString()); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Thanks again, >>>>>>>> >>>>>>>> Best regards, >>>>>>>> >>>>>>>> KK >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Nov 19, 2012 at 12:41 PM, Mohammad Tariq < >>>>>>>> donta...@gmail.com> wrote: >>>>>>>> >>>>>>>>> You can set your cronjob to execute the program after every 5 sec. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Mohammad Tariq >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Nov 19, 2012 at 6:05 PM, kashif khan < >>>>>>>>> drkashif8...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Well, I want to automatically upload the files as the files are >>>>>>>>>> generating about every 3-5 sec and each file has size about 3MB. >>>>>>>>>> >>>>>>>>>> Is it possible to automate the system using put or cp command? >>>>>>>>>> >>>>>>>>>> I read about the flume and webHDFS but I am not sure it will work >>>>>>>>>> or not. >>>>>>>>>> >>>>>>>>>> Many thanks >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Nov 19, 2012 at 12:26 PM, Alexander Alten-Lorenz < >>>>>>>>>> wget.n...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Why do you don't use HDFS related tools like put or cp? >>>>>>>>>>> >>>>>>>>>>> - Alex >>>>>>>>>>> >>>>>>>>>>> On Nov 19, 2012, at 11:44 AM, kashif khan < >>>>>>>>>>> drkashif8...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> > HI, >>>>>>>>>>> > >>>>>>>>>>> > I am generating files continuously in local folder of my base >>>>>>>>>>> machine. How >>>>>>>>>>> > I can now use the flume to stream the generated files from >>>>>>>>>>> local folder to >>>>>>>>>>> > HDFS. >>>>>>>>>>> > I dont know how exactly configure the sources, sinks and hdfs. >>>>>>>>>>> > >>>>>>>>>>> > 1) location of folder where files are generating: >>>>>>>>>>> /usr/datastorage/ >>>>>>>>>>> > 2) name node address: htdfs://hadoop1.example.com:8020 >>>>>>>>>>> > >>>>>>>>>>> > Please let me help. >>>>>>>>>>> > >>>>>>>>>>> > Many thanks >>>>>>>>>>> > >>>>>>>>>>> > Best regards, >>>>>>>>>>> > KK >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Alexander Alten-Lorenz >>>>>>>>>>> http://mapredit.blogspot.com >>>>>>>>>>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >