This program can read from local file system as well as HDFS. Try <java program> <Localfile> <HDFSDestination>
Thanks Suresh -----Original Message----- From: Ajey Shah [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 07, 2008 4:04 AM To: core-user@hadoop.apache.org Subject: RE: How do I copy files from my linux file system to HDFS using a java prog? Thanks Suresh. But even this program reads and writes from the HDFS. What i need to do is read from my normal local linux harddrive and write to the HDFS. I'm sorry if I misunderstood your program. Thanks for replying. :) Babu, Suresh wrote: > > > Try this program. Modify the HDFS configuration, if it is different > from the default. > > import java.io.File; > import java.io.IOException; > > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.fs.FileStatus; import > org.apache.hadoop.fs.FileSystem; import > org.apache.hadoop.fs.FSDataInputStream; > import org.apache.hadoop.fs.FSDataOutputStream; > import org.apache.hadoop.fs.Path; > import org.apache.hadoop.io.IOUtils; > > public class HadoopDFSFileReadWrite { > > static void usage () { > System.out.println("Usage : HadoopDFSFileReadWrite <inputfile> > <output file>"); > System.exit(1); > } > > static void printAndExit(String str) { > System.err.println(str); > System.exit(1); > } > > public static void main (String[] argv) throws IOException { > Configuration conf = new Configuration(); > conf.set("fs.default.name", "localhost:9000"); > FileSystem fs = FileSystem.get(conf); > > FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory()); > for(FileStatus status : fileStatus) { > System.out.println("File: " + status.getPath()); > } > > if (argv.length != 2) > usage(); > > // HadoopDFS deals with Path > Path inFile = new Path(argv[0]); > Path outFile = new Path(argv[1]); > > // Check if input/output are valid > if (!fs.exists(inFile)) > printAndExit("Input file not found"); > if (!fs.isFile(inFile)) > printAndExit("Input should be a file"); > if (fs.exists(outFile)) > printAndExit("Output already exists"); > > // Read from and write to new file > FSDataInputStream in = fs.open(inFile); > FSDataOutputStream out = fs.create(outFile); > byte buffer[] = new byte[256]; > try { > int bytesRead = 0; > while ((bytesRead = in.read(buffer)) > 0) { > out.write(buffer, 0, bytesRead); > } > > } catch (IOException e) { > System.out.println("Error while copying file"); > } finally { > in.close(); > out.close(); > } > } > } > > Suresh > > > -----Original Message----- > From: Ajey Shah [mailto:[EMAIL PROTECTED] > Sent: Thursday, May 01, 2008 3:31 AM > To: core-user@hadoop.apache.org > Subject: How do I copy files from my linux file system to HDFS using a > java prog? > > > Hello all, > > I need to copy files from my linux file system to HDFS in a java > program and not manually. This is the piece of code that I have. > > try { > > FileSystem hdfs = FileSystem.get(new Configuration()); > > LocalFileSystem ls = null; > > ls = hdfs.getLocal(hdfs.getConf()); > > hdfs.copyFromLocalFile(false, new > Path(fileName), new Path(outputFile)); > > } catch (Exception e) { > e.printStackTrace(); > } > > The problem is that it searches for the input path on the HDFS and not > my linux file system. > > Can someone point out where I may be wrong. I feel it's some > configuration issue but have not been able to figure it out. > > Thanks. > -- > View this message in context: > http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to > -H DFS-using-a-java-prog--tp16992491p16992491.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. > > > -- View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H DFS-using-a-java-prog--tp16992491p17093646.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
