external jars in .20

2009-06-01 Thread Lance Riedel
We are trying to upgrade to .20 from 19.1 due to several issues we are having. Now are jobs are failing with class not found exceptions. I am very confused about the final state for using external jars in .20. -libjars no long works placing all dependent jars in the jar /lib directory doesn't w

Re: Can not start task tracker because java.lang.NullPointerException

2009-05-22 Thread Lance Riedel
#x27;s not > too much trouble to try out the 19.2 branch from SVN, it would be helpful > in > determining whether this is a problem that's already fixed or if you've > discovered something new. > > Thanks > -Todd > > On Fri, May 22, 2009 at 2:01 PM, Lance Riedel

Re: Can not start task tracker because java.lang.NullPointerException

2009-05-22 Thread Lance Riedel
[dotsp...@domu-12-31-38-00-80-21 hadoop-0.19.1]$ du -sh /tmp 204K/tmp Does this look like a disk error? I had seen that the "org.apache.hadoop.util.DiskChecker$DiskErrorException" is bogus. Thanks! Lance On Fri, May 22, 2009 at 9:33 AM, Lance Riedel wrote: > Version 19.1

Can not start task tracker because java.lang.NullPointerException

2009-05-22 Thread Lance Riedel
Version 19.1 with patches: 4780-2v19.patch (Jira 4780) closeAll3.patch (Jira 3998) I have confirmed that https://issues.apache.org/jira/browse/HADOOP-4924patch is in, so that is not the fix. We are having task trackers die every night with a null pointer exception. Usually 2 or so out of 8 (25%

Re: Constantly getting DiskErrorExceptions - but logged as INFO

2009-05-20 Thread Lance Riedel
Thanks, found it: http://issues.apache.org/jira/browse/HADOOP-4963 Lance On Wed, May 20, 2009 at 8:15 AM, Lance Riedel wrote: > We're still seeing this error in our log files. Is this an expected > output? (the fact that it is INFO makes it seem not so bad, but anythng t

Re: Constantly getting DiskErrorExceptions - but logged as INFO

2009-05-20 Thread Lance Riedel
2009 at 10:34 AM, Lance Riedel wrote: > Trying to figure out what the scenario that would cause the following > errors. Since it is logged as INFO I wasn't sure if it was from speculative > execution, or if there is something more serious happening. Anyone seen > these errors?

Re: Infinite Loop Resending status from task tracker

2009-05-14 Thread Lance Riedel
Sorry, had missed that Todd had created Jira - HADOOP-5761<https://issues.apache.org/jira/browse/HADOOP-5761> Any progress there? Thanks, Lance On Thu, May 14, 2009 at 8:52 AM, Lance Riedel wrote: > Here is the point in the logs where the infinite loop begins - see time > stamp 2

Re: Infinite Loop Resending status from task tracker

2009-05-14 Thread Lance Riedel
doop.mapred.JobTracker: Adding task 'attempt_200905122015_1183_r_14_0' to tip task_200905122015_1183_r_14, for tracker 'tracker_domU-12-31-38-01-AD-91.compute-1.internal:localhost.localdomain/ 127.0.0.1:33929' 2009-05-14 04:03:56,465 INFO org.apache.hadoop.ipc.Server: IPC

Re: Infinite Loop Resending status from task tracker

2009-05-14 Thread Lance Riedel
tting DiskErrorExceptions - but logged as INFO"? I haven't seen responses on that. Thanks! Lance On Thu, May 14, 2009 at 7:48 AM, Lance Riedel wrote: > Here is the latest here.. Haven't heard any more, but every other night we > get 10 gigs logs and tons of failed tasks and h

Constantly getting DiskErrorExceptions - but logged as INFO

2009-05-11 Thread Lance Riedel
Trying to figure out what the scenario that would cause the following errors. Since it is logged as INFO I wasn't sure if it was from speculative execution, or if there is something more serious happening. Anyone seen these errors? They occur a lot. Also, note: We have plenty of disk space on this

Re: Infinite Loop Resending status from task tracker

2009-05-08 Thread Lance Riedel
your mapred.local.dir is on is out of space on that task tracker? 2) Is it possible that you're using a directory under /tmp for mapred.local.dir and some system cron script cleared out /tmp? -Todd On Sat, May 2, 2009 at 9:01 AM, Lance Riedel wrote: Hi Todd, Not sure if this is re

Re: Infinite Loop Resending status from task tracker

2009-05-02 Thread Lance Riedel
there is a separate problem that's a bit harder to track down. Thanks -Todd On Thu, Apr 30, 2009 at 11:17 AM, Lance Riedel wrote: Here are the job tracker logs from the same time (and yes.. there is something there!!): 2009-04-30 02:34:28,484 INFO org.apache.hadoop.mapred.JobTracker: S

Re: Infinite Loop Resending status from task tracker

2009-04-30 Thread Lance Riedel
ipcon wrote: Hey Lance, Did you see any error messages in the JobTracker logs around the time this started? I think I understand how this might happen. Thanks, -Todd On Thu, Apr 30, 2009 at 10:45 AM, Lance Riedel wrote: I have not been able to reproduce. We are using version 19.1 w

Re: Infinite Loop Resending status from task tracker

2009-04-30 Thread Lance Riedel
reproducible? I'm trying to look at the code path that might produce such a behavior and want to make sure I'm looking at the right version. Thanks -Todd On Thu, Apr 30, 2009 at 9:33 AM, Lance Riedel wrote: Has anyone seen this before? Our task tracker produced a 2.7 gig log file

Infinite Loop Resending status from task tracker

2009-04-30 Thread Lance Riedel
Has anyone seen this before? Our task tracker produced a 2.7 gig log file in a few hours. The entry is all the same (every 2 ms): 2009-04-30 02:34:40,207 INFO org.apache.hadoop.mapred.TaskTracker: Resending 'status' to 'ec2-xx-xx-xx-xx.compute-1.amazonaws.com' with reponseId '5341 2009-04-3