Thanks a lot!
That was it. There was following line in our code:
jobConf.setKeepTaskFilesPattern(.*);
On Fri, Jan 11, 2013 at 2:20 PM, Hemanth Yamijala yhema...@thoughtworks.com
wrote:
Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern
Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern
This is set to .* in the job.xml file you sent. I suspect this may be
causing a problem. Can you please remove this, assuming you have not set it
intentionally ?
Thanks
Hemanth
On Fri, Jan
Thanks for replies!
Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.
Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't
Hi,
On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov itretya...@griddynamics.com
wrote:
Thanks for replies!
Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.
Now, when a job
New York Presbyterian Hospital
From: Hemanth Yamijala [mailto:yhema...@thoughtworks.com]
Sent: Thursday, January 10, 2013 07:37 AM
To: user@hadoop.apache.org user@hadoop.apache.org
Subject: Re: JobCache directory cleanup
Hi,
On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov
itretya
Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.
Sharing your configs for these jobs will definitely help.
Thanks,
+Vinod
On Wed, Jan 9, 2013 at 6:41 AM, Ivan
Good point. Forgot that one :-)
On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli
vino...@hortonworks.com wrote:
Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could
Hello!
I've found that jobcache directory became very large on our cluster, e.g.:
# du -sh /data?/mapred/local/taskTracker/user/jobcache
465G/data1/mapred/local/taskTracker/user/jobcache
464G/data2/mapred/local/taskTracker/user/jobcache
454G
Hi,
Per default (and not configurable) the logs will be persist for 30 days. This
will be configurable in future
(https://issues.apache.org/jira/browse/MAPREDUCE-4643).
- Alex
On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov itretya...@griddynamics.com wrote:
Hello!
I've found that jobcache
Hi Ivan,
Regarding the mapreduce.jobtracker.retiredjobs.cache.size property, the
jobtracker keeps information about a number of completed jobs in memory.
There's a threshold for this, which is a single day by default - as well as
a certain number of jobs per user. Once these limits are hit, the
user@hadoop.apache.org
Subject: Re: JobCache directory cleanup
Thanks a lot Alexander!
What is mapreduce.jobtracker.retiredjobs.cache.size for?
Does cron approach safe for hadoop? Is that only way at the moment?
On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz
wget.n
Hi,
The directory name you have provided is
/data?/mapred/local/taskTracker/persona/jobcache/.
This directory is used by the TaskTracker (slave) daemons to localize job
files when the tasks are run on the slaves.
Hence, I don't think this is related to the parameter
12 matches
Mail list logo
