Please ignore the last two lines (after Thanks)! Akhil
akhil1988 wrote: > > Hi Chris! > > Sorry for the late reply! > > To push the file into HDFS is clear to me and it can be done using "hadoop > fs -put" command also (prior to executing the job), which I generally use. > > The method to access a file in HDFS from Mapper/Reducer is the following: > FileSystem fs = FileSystem.get(conf); > FSDataInputStream din = fs.open("/home/akhil1988/sample.txt"); > > The method (below)that you gave does not work: > Path cachePath= new Path("hdfs:///home/akhil1988/sample.txt"); > BufferedReader wordReader = new BufferedReader(new > FileReader(cachePath.toString())); > > A file in HDFS cannot be accessed through these standard Java function, it > has to be accessed only via the method I have mentioned above. The API > methods for FileSystem class are very limited and it only provides us to > read a data file (containing java primitives) and not any binary files. > > In my specific problem, I am using a API (specific to my research-domain) > which takes a path (String) as input and reads data from this path (which > points to a binary file). So I just need a way in which I can access files > (from tasktrackers) as we do via standard java functions. For this, we > need the files to be present in the local filesystem of the tasktrackers. > That is why I am using DistributedCache. > > I hope I am clear?? And if I am wrong anywhere, please let me know. > > Thanks, > Akhil > > > > > > The API provides only this function to read a data file(containing java > primitives), we cannot read any binary files. > > > > > Well, what I wanted was to have a directory in the local filesystem of the > tasktracker and not the HDFS because of the following reason: > > > > > Chris Curtin-2 wrote: >> >> To push the file to HDFS (put it in the 'a_hdfsDirectory' directory) >> >> Configuration config = new Configuration(); >> FileSystem hdfs = FileSystem.get(config); >> Path srcPath = new Path(a_directory + "/" + outputName); >> Path dstPath = new Path(a_hdfsDirectory + "/" + outputName); >> hdfs.copyFromLocalFile(srcPath, dstPath); >> >> >> to read it from HDFS in your mapper or reducer: >> >> Configuration config = new Configuration(); >> FileSystem hdfs = FileSystem.get(config); >> Path cachePath= new Path(a_hdfsDirectory + "/" + outputName); >> BufferedReader wordReader = new BufferedReader( >> new FileReader(cachePath.toString())); >> >> >> >> On Fri, Jun 26, 2009 at 8:55 PM, akhil1988 <akhilan...@gmail.com> wrote: >> >>> >>> Thanks Chris for your reply! >>> >>> Well, I could not understand much of what has been discussed on that >>> forum. >>> I am unaware of Cascading. >>> >>> My problem is simple - I want a directory to present in the local >>> working >>> directory of tasks so that I can access it from my map task in the >>> following >>> manner : >>> >>> FileInputStream fin = new FileInputStream("Config/file1.config"); >>> >>> where, >>> Config is a directory which contains many files/directories, one of >>> which >>> is >>> file1.config >>> >>> It would be helpful to me if you can tell me what statements to use to >>> distribute a directory to the tasktrackers. >>> The API doc http://hadoop.apache.org/core/docs/r0.20.0/api/index.html >>> says >>> that archives are unzipped on the tasktrackers but I want an example of >>> how >>> to use this in case of a dreictory. >>> >>> Thanks, >>> Akhil >>> >>> >>> >>> Chris Curtin-2 wrote: >>> > >>> > Hi, >>> > >>> > I've found it much easier to write the file to HDFS use the API, then >>> pass >>> > the 'path' to the file in HDFS as a property. You'll need to remember >>> to >>> > clean up the file after you're done with it. >>> > >>> > Example details are in this thread: >>> > >>> http://groups.google.com/group/cascading-user/browse_thread/thread/d5c619349562a8d6# >>> > >>> > Hope this helps, >>> > >>> > Chris >>> > >>> > On Thu, Jun 25, 2009 at 4:50 PM, akhil1988 <akhilan...@gmail.com> >>> wrote: >>> > >>> >> >>> >> Please ask any questions if I am not clear above about the problem I >>> am >>> >> facing. >>> >> >>> >> Thanks, >>> >> Akhil >>> >> >>> >> akhil1988 wrote: >>> >> > >>> >> > Hi All! >>> >> > >>> >> > I want a directory to be present in the local working directory of >>> the >>> >> > task for which I am using the following statements: >>> >> > >>> >> > DistributedCache.addCacheArchive(new >>> URI("/home/akhil1988/Config.zip"), >>> >> > conf); >>> >> > DistributedCache.createSymlink(conf); >>> >> > >>> >> >>> Here Config is a directory which I have zipped and put at the >>> given >>> >> >>> location in HDFS >>> >> > >>> >> > I have zipped the directory because the API doc of DistributedCache >>> >> > (http://hadoop.apache.org/core/docs/r0.20.0/api/index.html) says >>> that >>> >> the >>> >> > archive files are unzipped in the local cache directory : >>> >> > >>> >> > DistributedCache can be used to distribute simple, read-only >>> data/text >>> >> > files and/or more complex types such as archives, jars etc. >>> Archives >>> >> (zip, >>> >> > tar and tgz/tar.gz files) are un-archived at the slave nodes. >>> >> > >>> >> > So, from my understanding of the API docs I expect that the >>> Config.zip >>> >> > file will be unzipped to Config directory and since I have >>> SymLinked >>> >> them >>> >> > I can access the directory in the following manner from my map >>> >> function: >>> >> > >>> >> > FileInputStream fin = new FileInputStream("Config/file1.config"); >>> >> > >>> >> > But I get the FileNotFoundException on the execution of this >>> statement. >>> >> > Please let me know where I am going wrong. >>> >> > >>> >> > Thanks, >>> >> > Akhil >>> >> > >>> >> >>> >> -- >>> >> View this message in context: >>> >> http://www.nabble.com/Using-addCacheArchive-tp24207739p24210836.html >>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com. >>> >> >>> >> >>> > >>> > >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Using-addCacheArchive-tp24207739p24229338.html >>> Sent from the Hadoop core-user mailing list archive at Nabble.com. >>> >>> >> >> > > -- View this message in context: http://www.nabble.com/Using-addCacheArchive-tp24207739p24300929.html Sent from the Hadoop core-user mailing list archive at Nabble.com.