Find Job Status by Job ID

[Laxmi Narsimha Rao Oruganti]

Hey There,

 

We are very new to hadoop and have question.  

 

We are submitting (or creating) job using Hadoop Job API v2 (i.e. namespace 
mapreduce than old one mapred)

 

We submit MR Jobs derived from metadata of our own jobs using 
mapreduce namespace API.  We maintain the Hadoop Job Name in this table.

We want to track the submitted jobs for the progress (and so 
completion) so that we can update our own jobs as complete.

 

1)  Job Tracking API in mapred namespce - Does it work for jobs created 
using new namespace API

2)  If (1) is false, is there any way to track Job Progress using mapreduce 
namespace Job API? New namespace Job Status API requires Job object.  Whereas 
'Job Monitoring' module of ours does not have any job object with it.  

 

Can you please help us with anyway to get Job Status given a Job Name?  We make 
sure job name are unique.

 

Thanks,

Laxmi

Re: Apache hadoop - 1.2.1 source compilation through Maven or ant

[A Laxmi]

To install hadoop, you don’t need to compile it. Just extract all the files
from tar.gz and put it into right place.
(Source:
http://www.danielbit.com/blog/tools-for-linux/install-hadoop-on-ubuntu)


On Sun, Oct 20, 2013 at 10:48 AM, Viswanathan J
jayamviswanat...@gmail.comwrote:

 Hi Ted,

 Thanks for your response. I'm running on Ubuntu, jar built successfully.

 But the jar generated as snapshot version. Also the source which I
 compiled was 1.2.1 but it generated as 1.2.2 snapshot version jar. Is the
 snapshot version because of changes in the source?

 Shall I use that jar in production environment? If yes will that not any
 issue.

 Please help.

 Thanks,
 On Oct 20, 2013 7:59 PM, Ted Yu yuzhih...@gmail.com wrote:

 Can you tell us which OS you're using ?

 What was the error(s) when you ran ant jar ?

 On my Mac, ant jar works.

 Cheers


 On Sun, Oct 20, 2013 at 1:58 AM, Viswanathan J 
 jayamviswanat...@gmail.com wrote:

 Hi,

 Please help to compile the source code of Apache hadoop using mvn or ant.

 I just tried to download the latest hadoop stable source and run ant
 jar but it is not compiling getting errors.

 --
 Regards,
 Viswa.J

Hadoop HBase Pseudo mode - RegionServer disconnects after some time

[A Laxmi]

Hi -

Please find the below log of HBase-master. I have tried all sorts of fixes
mentioned in various threads yet I could not overcome this issue. I made
sure I dont have 127.0.1.1 in /etc/hosts file. I pinged my localhost
(hostname) which gives back the actual IP and not 127.0.0.1 using ping -c 1
localhost. I have 'localhost' in my /etc/hostname and actual IP address
mapped to localhost.localdomain and localhost as alias - something like

/etc/hosts -

192.***.*.*** localhost.localdomain localhost

/etc/hostname -

localhost

I am using *Hadoop 0.20.205.0 and HBase 0.90.6 in Pseudo mode* for storing
crawled data from a crawler - Apache Nutch 2.2.1. I can start Hadoop and
HBase and when I do jps it shows all good, then after that when I start
Nutch crawl after about 40mins of crawling or so, I can see Nutch hanging
up while in about 4th iteration of parsing and at the same time when I do
jps in HBase, I can see everything except HRegionServer. Below is the log.

I tried all possible ways but couldn't overcome this issue. I really need
someone from HBase list to help me with this issue.


2013-10-15 02:02:08,285 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=56 entries
from hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816329235
2013-10-15 02:02:08,285 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 28 of
29: hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816367672, length=64818440
2013-10-15 02:02:08,285 WARN org.apache.hadoop.hbase.util.FSUtils: Running
on HDFS without append enabled may result in data loss
2013-10-15 02:02:08,554 DEBUG org.apache.*hadoop.hbase.master.HMaster: Not
running balancer because processing dead regionserver(s): [127.0.0.1,60020*
,1381814216471]
2013-10-15 02:02:08,556 INFO org.apache.hadoop.hbase.catalo*g.CatalogTracker:
Failed verification of .META.,,1 at address=127.0.0.1:60020;
java.net.ConnectException: Connection refused*
2013-10-15 02:02:08,559 INFO org.apache.hadoop.hbase.catalog.*CatalogTracker:
Current cached META location is not valid*, resetting
2013-10-15 02:02:08,601 WARN org.apache.hadoop.*hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
(2147483647ms)*
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:390)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:422)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:120)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:88)
at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
2013-10-15 02:02:08,842 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs --
HDFS-200 -- not available, dfs.support.append=false
2013-10-15 02:02:08,842 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://localhost:8020/hbase/1_webpage/853ef78be7c0853208e865a9ff13d5fb/recovered.edits/0001556.temp
region=853ef78be7c0853208e865a9ff13d5fb
2013-10-15 02:02:09,443 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=39 entries
from hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816367672
2013-10-15 02:02:09,444 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 29 of
29: hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816657239, length=0

Thanks for your help!