Hi Tanya, What version of Hadoop are you running?
Is this a 1-node cluster running in pseudo-distributed mode with 1 physical spinning hard drive? How much intermediate data is being emitted from the Map phase? How many mappers and reducers total is the job running? -- Sameer Farooqui Systems Architect / Hortonworks On Thu, Feb 23, 2012 at 7:08 AM, tanyasch <ta...@tickel.net> wrote: > > Hi, I'm running a job that completes in about a 90 seconds, but takes about > 10-15 minutes to run cleanup. I'm looking for ways to affect or even > monitor the cleanup time. I'd like even advice about whether this is more > of a setup issue (like where I'm storing files, with Accumulo and Hadoop > temporary and log files all writing to the same disk because our cluster is > tiny) or a job issue (can I throw more reducers at it ? the brief > description of the OutputCommitter says it uses available reducers for > cleanup) or a programming issue (in that case I'd post a different > question) > > Basically, I want to know if the first way to go at this is by > reconfiguring > the cluster or if I should be programming my way out of this? Thanks. > > -- > View this message in context: > http://old.nabble.com/Ways-to-improve-job-cleanup-speed-tp33377374p33377374.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. > >
