I'm not concerned about disk space usage -- the script we used that deleted the taskTracker cache path has been fixed not to do so.
I'm curious about the exact behavior of jobs that use DistributedCache files. Again, it seems safe from your description to delete files between completed runs. How could the job or the taskTracker distinguish between the files having been deleted and their not having been downloaded from a previous run of the job? Is it state in memory that the taskTracker maintains? On Tue, Sep 27, 2011 at 1:44 PM, Robert Evans <ev...@yahoo-inc.com> wrote: > If you are never ever going to use that file again for any map/reduce task > in the future then yes you can delete it, but I would not recommend it. If > you want to reduce the amount of space that is used by the distributed cache > there is a config parameter for that. > > "local.cache.size" it is the number of bytes per drive that will be used > for storing data in the distributed cache. This is in 0.20 for hadoop I am > not sure if it has changed at all for trunk. It is not documented as far as > I can tell, and it defaults to 10GB. > > --Bobby Evans > > > On 9/27/11 12:04 PM, "Meng Mao" <meng...@gmail.com> wrote: > > From that interpretation, it then seems like it would be safe to delete the > files between completed runs? How could it distinguish between the files > having been deleted and their not having been downloaded from a previous > run? > > On Tue, Sep 27, 2011 at 12:25 PM, Robert Evans <ev...@yahoo-inc.com> > wrote: > > > addCacheFile sets a config value in your jobConf that indicates which > files > > your particular job depends on. When the TaskTracker is assigned to run > > part of your job (map task or reduce task), it will download your > jobConf, > > read it in, and then download the files listed in the conf, if it has not > > already downloaded them from a previous run. Then it will set up the > > directory structure for your job, possibly adding in symbolic links to > these > > files in the working directory for your task. After that it will launch > > your task. > > > > --Bobby Evans > > > > On 9/27/11 11:17 AM, "Meng Mao" <meng...@gmail.com> wrote: > > > > Who is in charge of getting the files there for the first time? The > > addCacheFile call in the mapreduce job? Or a manual setup by the > > user/operator? > > > > On Tue, Sep 27, 2011 at 11:35 AM, Robert Evans <ev...@yahoo-inc.com> > > wrote: > > > > > The problem is the step 4 in the breaking sequence. Currently the > > > TaskTracker never looks at the disk to know if a file is in the > > distributed > > > cache or not. It assumes that if it downloaded the file and did not > > delete > > > that file itself then the file is still there in its original form. It > > does > > > not know that you deleted those files, or if wrote to the files, or in > > any > > > way altered those files. In general you should not be modifying those > > > files. This is not only because it messes up the tracking of those > > files, > > > but because other jobs running concurrently with your task may also be > > using > > > those files. > > > > > > --Bobby Evans > > > > > > > > > On 9/26/11 4:40 PM, "Meng Mao" <meng...@gmail.com> wrote: > > > > > > Let's frame the issue in another way. I'll describe a sequence of > Hadoop > > > operations that I think should work, and then I'll get into what we did > > and > > > how it failed. > > > > > > Normal sequence: > > > 1. have files to be cached in HDFS > > > 2. Run Job A, which specifies those files to be put into > DistributedCache > > > space > > > 3. job runs fine > > > 4. Run Job A some time later. job runs fine again. > > > > > > Breaking sequence: > > > 1. have files to be cached in HDFS > > > 2. Run Job A, which specifies those files to be put into > DistributedCache > > > space > > > 3. job runs fine > > > 4. Manually delete cached files out of local disk on worker nodes > > > 5. Run Job A again, expect it to push out cache copies as needed. > > > 6. job fails because the cache copies didn't get distributed > > > > > > Should this second sequence have broken? > > > > > > On Fri, Sep 23, 2011 at 3:09 PM, Meng Mao <meng...@gmail.com> wrote: > > > > > > > Hmm, I must have really missed an important piece somewhere. This is > > from > > > > the MapRed tutorial text: > > > > > > > > "DistributedCache is a facility provided by the Map/Reduce framework > to > > > > cache files (text, archives, jars and so on) needed by applications. > > > > > > > > Applications specify the files to be cached via urls (hdfs://) in the > > > > JobConf. The DistributedCache* assumes that the files specified via > > > > hdfs:// urls are already present on the FileSystem.* > > > > > > > > *The framework will copy the necessary files to the slave node before > > any > > > > tasks for the job are executed on that node*. Its efficiency stems > from > > > > the fact that the files are only copied once per job and the ability > to > > > > cache archives which are un-archived on the slaves." > > > > > > > > > > > > After some close reading, the two bolded pieces seem to be in > > > contradiction > > > > of each other? I'd always that addCacheFile() would perform the 2nd > > > bolded > > > > statement. If that sentence is true, then I still don't have an > > > explanation > > > > of why our job didn't correctly push out new versions of the cache > > files > > > > upon the startup and execution of JobConfiguration. We deleted them > > > before > > > > our job started, not during. > > > > > > > > On Fri, Sep 23, 2011 at 9:35 AM, Robert Evans <ev...@yahoo-inc.com> > > > wrote: > > > > > > > >> Meng Mao, > > > >> > > > >> The way the distributed cache is currently written, it does not > verify > > > the > > > >> integrity of the cache files at all after they are downloaded. It > > just > > > >> assumes that if they were downloaded once they are still there and > in > > > the > > > >> proper shape. It might be good to file a JIRA to add in some sort > of > > > check. > > > >> Another thing to do is that the distributed cache also includes the > > > time > > > >> stamp of the original file, just incase you delete the file and then > > use > > > a > > > >> different version. So if you want it to force a download again you > > can > > > copy > > > >> it delete the original and then move it back to what it was before. > > > >> > > > >> --Bobby Evans > > > >> > > > >> On 9/23/11 1:57 AM, "Meng Mao" <meng...@gmail.com> wrote: > > > >> > > > >> We use the DistributedCache class to distribute a few lookup files > for > > > our > > > >> jobs. We have been aggressively deleting failed task attempts' > > leftover > > > >> data > > > >> , and our script accidentally deleted the path to our distributed > > cache > > > >> files. > > > >> > > > >> Our task attempt leftover data was here [per node]: > > > >> /hadoop/hadoop-metadata/cache/mapred/local/ > > > >> and our distributed cache path was: > > > >> > > hadoop/hadoop-metadata/cache/mapred/local/taskTracker/archive/<nameNode> > > > >> We deleted this path by accident. > > > >> > > > >> Does this latter path look normal? I'm not that familiar with > > > >> DistributedCache but I'm up right now investigating the issue so I > > > thought > > > >> I'd ask. > > > >> > > > >> After that deletion, the first 2 jobs to run (which are use the > > > >> addCacheFile > > > >> method to distribute their files) didn't seem to push the files out > to > > > the > > > >> cache path, except on one node. Is this expected behavior? Shouldn't > > > >> addCacheFile check to see if the files are missing, and if so, > > > repopulate > > > >> them as needed? > > > >> > > > >> I'm trying to get a handle on whether it's safe to delete the > > > distributed > > > >> cache path when the grid is quiet and no jobs are running. That is, > if > > > >> addCacheFile is designed to be robust against the files it's caching > > not > > > >> being at each job start. > > > >> > > > >> > > > > > > > > > > > > > > > >
