Yeah, it's against a ~95million row table in hbase. It takes about 30 mins to get to 90% then about 3+ hours to get from 90% to 100%
On Wed, 2012-12-05 at 08:46 -0800, in.abdul wrote: > Hi jay.. > Are you trying to do M-R on HBase Table ? > > > Thanks and regards > Syed Abdul Kather > > > Thanks and Regards, > S SYED ABDUL KATHER > > > > On Wed, Dec 5, 2012 at 9:53 PM, Jay Whittaker [via Lucene] < > ml-node+s472066n402449...@n3.nabble.com> wrote: > > > Hey Ac, > > > > The logs I copied were from the .out files while a job was running. I > > thought that would be the best way to get a good idea of what was > > happening. > > > > Cheers, > > > > Jay > > > > On Tue, 2012-12-04 at 21:59 +0800, [hidden > > email]<http://user/SendEmail.jtp?type=node&node=4024496&i=0>wrote: > > > > > Hi, > > > > > > Have you also checked .out file of the tasktracker in logs? It could > > contain some useful information for the issue. > > > > > > Thanks > > > ac > > > > > > > > > On 4 Dec 2012, at 8:27 PM, Jay Whittaker wrote: > > > > > > > Hey, > > > > > > > > We are running Map reduce jobs against a 12 machine hbase cluster and > > > > for a long time they took approx 30 mins to return a result against > > ~95 > > > > million rows. Without any major changes to the data or any upgrade of > > > > hbase/hadoop they now seem to be taking about 4 hours. and the logs > > are > > > > full of > > > > > > > > 2012-12-04 13:33:15,602 INFO org.apache.hadoop.mapred.TaskTracker: > > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 72 6f 75 > > > > 67 68 74 > > > > ... > > > > 2012-12-04 13:45:17,134 INFO org.apache.hadoop.mapred.TaskTracker: > > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 72 70 > > > > 6c 65 64 65 73 69 67 6e 73 65 72 76 69 63 65 73 > > > > ... > > > > 2012-12-04 13:46:11,515 INFO org.apache.hadoop.mapred.TaskTracker: > > > > attempt_201211210952_0293_m_000031_0 0.0% row: 63 6f 6d 2e 70 75 73 68 > > > > 74 6f 74 61 6c 6b 2d 6f 6e 6c 69 6e 65 > > > > > > > > I presume the 0% is percent complete but I'm not sure as to why the > > time > > > > to complete has now jumped massively. Ganglia shows no major load on > > the > > > > nodes in question so I don't think it's that. > > > > > > > > What steps should I be taking to try troubleshoot the problem? > > > > > > > > Regards, > > > > > > > > Jay > > > > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the discussion > > below: > > > > http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024496.html > > To start a new topic under Hadoop lucene-users, email > > ml-node+s472066n647590...@n3.nabble.com > > To unsubscribe from Lucene, click > > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=472066&code=aW4uYWJkdWxAZ21haWwuY29tfDQ3MjA2NnwxMDczOTUyNDEw> > > . > > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > > > > ----- > THANKS AND REGARDS, > SYED ABDUL KATHER > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Map-Reduce-jobs-taking-a-long-time-at-the-end-tp4024231p4024515.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
