Re: Cannot start resourcemanager

[Siddharth Tiwari]

Hi team, 
Resource manager works now, the capacity scheduler xml had wrong entry somehow. 
But there is another small issue, I have NN HA enabled and wanted to run Hbase 
with it and eventhough I set hbase.rootdir to FsNameservice value it always 
throws exception saying it cannot recognize the nameservice value. I did put 
the core site and hdfs site in hbase conf. Can you help me in setting it up 
with namenode HA in new hadoop-2.2.0 stable release .  Also what versions of 
hive, mahout and pig would be compatible with it. I am using hbase-0.94.12 
release.

Sent from my iPad

> On Oct 17, 2013, at 12:48 PM, "Arun C Murthy"  wrote:
> 
> What command did you use to start the RM?
> 
>> On Oct 17, 2013, at 10:18 AM, Siddharth Tiwari  
>> wrote:
>> 
>> Hi Team,
>> 
>> trying to start resourcemanager in the latest hadoop-2.2.0 stable release. 
>> It throws following error. Please help
>> 
>> 2013-10-17 10:01:51,230 INFO 
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager 
>> metrics system...
>> 2013-10-17 10:01:51,230 INFO 
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
>> system stopped.
>> 2013-10-17 10:01:51,231 INFO 
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
>> system shutdown complete.
>> 2013-10-17 10:01:51,232 FATAL 
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error 
>> starting ResourceManager
>> java.lang.RuntimeException: java.lang.RuntimeException: 
>> java.lang.ClassNotFoundException: Class 
>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>>  not found
>>  at 
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
>>  at 
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333)
>>  at 
>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:263)
>>  at 
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:249)
>>  at 
>> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>  at 
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
>> Class 
>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>>  not found
>>  at 
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
>>  at 
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
>>  ... 5 more
>> Caused by: java.lang.ClassNotFoundException: Class 
>> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>>  not found
>>  at 
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
>>  at 
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
>>  ... 6 more
>> 2013-10-17 10:01:51,239 INFO 
>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG: 
>> /
>> SHUTDOWN_MSG: Shutting down ResourceManager at node1/192.168.147.101
>> 
>> 
>> 
>> **
>> Cheers !!!
>> Siddharth Tiwari
>> Have a refreshing day !!!
>> "Every duty is holy, and devotion to duty is the highest form of worship of 
>> God.” 
>> "Maybe other people will try to limit me but I don't limit myself"
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader of 
> this message is not the intended recipient, you are hereby notified that any 
> printing, copying, dissemination, distribution, disclosure or forwarding of 
> this communication is strictly prohibited. If you have received this 
> communication in error, please contact the sender immediately and delete it 
> from your system. Thank You.

RE: Intermittent DataStreamer Exception while appending to file inside HDFS

[Uma Maheswara Rao G]

Hi Arinto,

You can check 3rd DN logs. Whether any space issues so that node was not 
selected for write etc.

> Does it mean that one of the datanodes was unreachable when we try to append 
> into the files?
It did not select for write itself. If it failed after selected for write means 
you should have get this error while recovery itself.

Regards,
Uma

From: Arinto Murdopo [mailto:ari...@gmail.com]
Sent: 11 October 2013 08:48
To: user@hadoop.apache.org
Subject: Re: Intermittent DataStreamer Exception while appending to file inside 
HDFS

Thank you for the comprehensive answer,
When I inspect our NameNode UI, I see there are 3 datanodes are up.
However, as you mentioned, the log only showed 2 datanodes are up. Does it mean 
that one of the datanodes was unreachable when we try to append into the files?
Best regards,


Arinto
www.otnira.com

On Thu, Oct 10, 2013 at 4:57 PM, Uma Maheswara Rao G 
mailto:mahesw...@huawei.com>> wrote:
Hi Arinto,

Please disable this feature with smaller clusters. 
dfs.client.block.write.replace-datanode-on-failure.policy
Reason for this exception is, you have replication set to 3 and looks like you 
have only 2 nodes in the cluster from the logs. When you first time created 
pipeline we will not do any verification i.e, whether pipeline DNs met the 
replication or not. Above property says only replace DN on failure. But here 
additionally we took advantage of verifying this condition when we reopen the 
pipeline for append. So, here unfortunately it will not meet the replication 
with existing DNs and it will try to add another node. Since you are not having 
any extra nodes in cluster other than selected nodes, it will fail. With the 
current configurations you can not append.


Also please take a look at default configuration description:
dfs.client.block.write.replace-datanode-on-failure.enable
  true
  
If there is a datanode/network failure in the write pipeline,
DFSClient will try to remove the failed datanode from the pipeline
and then continue writing with the remaining datanodes. As a result,
the number of datanodes in the pipeline is decreased.  The feature is
to add new datanodes to the pipeline.

This is a site-wide property to enable/disable the feature.

When the cluster size is extremely small, e.g. 3 nodes or less, cluster
administrators may want to set the policy to NEVER in the default
configuration file or disable this feature.  Otherwise, users may
experience an unusually high rate of pipeline failures since it is
impossible to find new datanodes for replacement.

See also dfs.client.block.write.replace-datanode-on-failure.policy
  


Make this configuration false at your client side.

Regards,
Uma


From: Arinto Murdopo [mailto:ari...@gmail.com]
Sent: 10 October 2013 13:02
To: user@hadoop.apache.org
Subject: Intermittent DataStreamer Exception while appending to file inside HDFS

Hi there,
I have this following exception while I'm appending existing file in my HDFS. 
This error appears intermittently. If the error does not show up, I can append 
the file successfully. If the error appears, I could not append the file.
Here is the error: https://gist.github.com/arinto/d37a56f449c61c9d1d9c
For your convenience, here it is:

13/10/10 14:17:30 WARN hdfs.DFSClient: DataStreamer Exception

java.io.IOException: Failed to add a datanode.  User may turn off this feature 
by setting dfs.client.block.write.replace-datanode-on-failure.policy in 
configuration, where the current policy is DEFAULT.  (Nodes: 
current=[10.0.106.82:50010, 
10.0.106.81:50010], 
original=[10.0.106.82:50010, 
10.0.106.81:50010])

   at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:778)

   at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)

   at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:934)

   at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:461)

Some configuration files:

1. hdfs-site.xml: 
https://gist.github.com/arinto/f5f1522a6f6994ddfc17#file-hdfs-append-datastream-exception-hdfs-site-xml








2. core-site.xml: 
https://gist.github.com/arinto/0c6f40872181fe26f8b1#file-hdfs-append-datastream-exception-core-site-xml







So, any idea how to solve this issue?
Some links that I've found (but unfortunately they do not help)
1. 
StackOverflow,
 our replication factor is 3 and we've never changed the replication factor 
since we setup the cluster.
2. Impala-User mailing 
list

Re: Yarn never use TeraSort#TotalOrderPartitioner when run TeraSort job?

[sam liu]

It's really weird and confusing me. Anyone can help this question?

Thanks!


2013/10/16 sam liu 

> Hi Experts,
>
> In Hadoop-2.0.4, the TeraSort leverage TeraSort#TotalOrderPartitioner as
> its Partitioner: 'job.setPartitionerClass(TotalOrderPartitioner.class);'.
> However, seems Yarn did not execute the methods of
> TeraSort#TotalOrderPartitioner at all. I did some tests to verify it as
> below:
>
> Test 1: Add some code in the method readPartitions() and setConf() in
> TeraSort#TotalOrderPartitioner to print some words and write some word to a
> file.
> Expected Result: Some words should be printed and wrote into a file
> Actual Result: No word was printed and wrote into a file at all
>
> Test 2: Remove all existing methods in TeraSort#TotalOrderPartitioner, but
> only remaining some necessary but empty methods in it
> Expected Result: TeraSort job will ocurr some exception, as the specified
> Partitioner is not implemented at all
> Actual Result: TeraSort job completed successfully without any exception
>
> Above tests confused me a lot, because seems Yarn never use specified
> partitioner TeraSort#TotalOrderPartitioner at all during job execution.
>
> Any one can help provide the reasons?
>
> Thanks very much!
>

Re: Cannot start resourcemanager

[Tsuyoshi OZAWA]

Hi,
could you check environment variables(e.g.
HADOOP_COMMON_HOME/HADOOP_HDFS_HOME/HADOOP_MAPRED_HOME/HADOOP_CONF_DIR)
and send us the contents of etc/yarn-site.conf? In my environment, I
cannot reproduce your problem with 2.2.0 tar ball.
Thanks, Tsuyoshi

On Thu, Oct 17, 2013 at 10:18 AM, Siddharth Tiwari
 wrote:
> Hi Team,
>
> trying to start resourcemanager in the latest hadoop-2.2.0 stable release.
> It throws following error. Please help
>
> 2013-10-17 10:01:51,230 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager
> metrics system...
> 2013-10-17 10:01:51,230 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
> system stopped.
> 2013-10-17 10:01:51,231 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
> system shutdown complete.
> 2013-10-17 10:01:51,232 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> starting ResourceManager
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:263)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:249)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
> ... 6 more
> 2013-10-17 10:01:51,239 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down ResourceManager at node1/192.168.147.101
>
>
>
> **
> Cheers !!!
> Siddharth Tiwari
> Have a refreshing day !!!
> "Every duty is holy, and devotion to duty is the highest form of worship of
> God.”
> "Maybe other people will try to limit me but I don't limit myself"



-- 
- Tsuyoshi

Hadoop HBase Pseudo mode - RegionServer disconnects after some time

[A Laxmi]

Hi -

Please find the below log of HBase-master. I have tried all sorts of fixes
mentioned in various threads yet I could not overcome this issue. I made
sure I dont have 127.0.1.1 in /etc/hosts file. I pinged my localhost
(hostname) which gives back the actual IP and not 127.0.0.1 using ping -c 1
localhost. I have 'localhost' in my /etc/hostname and actual IP address
mapped to localhost.localdomain and localhost as alias - something like

/etc/hosts -

192.***.*.*** localhost.localdomain localhost

/etc/hostname -

localhost

I am using *Hadoop 0.20.205.0 and HBase 0.90.6 in Pseudo mode* for storing
crawled data from a crawler - Apache Nutch 2.2.1. I can start Hadoop and
HBase and when I do jps it shows all good, then after that when I start
Nutch crawl after about 40mins of crawling or so, I can see Nutch hanging
up while in about 4th iteration of parsing and at the same time when I do
jps in HBase, I can see everything except HRegionServer. Below is the log.

I tried all possible ways but couldn't overcome this issue. I really need
someone from HBase list to help me with this issue.


2013-10-15 02:02:08,285 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=56 entries
from hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816329235
2013-10-15 02:02:08,285 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 28 of
29: hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816367672, length=64818440
2013-10-15 02:02:08,285 WARN org.apache.hadoop.hbase.util.FSUtils: Running
on HDFS without append enabled may result in data loss
2013-10-15 02:02:08,554 DEBUG org.apache.*hadoop.hbase.master.HMaster: Not
running balancer because processing dead regionserver(s): [127.0.0.1,60020*
,1381814216471]
2013-10-15 02:02:08,556 INFO org.apache.hadoop.hbase.catalo*g.CatalogTracker:
Failed verification of .META.,,1 at address=127.0.0.1:60020;
java.net.ConnectException: Connection refused*
2013-10-15 02:02:08,559 INFO org.apache.hadoop.hbase.catalog.*CatalogTracker:
Current cached META location is not valid*, resetting
2013-10-15 02:02:08,601 WARN org.apache.hadoop.*hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out
(2147483647ms)*
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:390)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:422)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:120)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:88)
at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
2013-10-15 02:02:08,842 INFO
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs --
HDFS-200 -- not available, dfs.support.append=false
2013-10-15 02:02:08,842 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
path=hdfs://localhost:8020/hbase/1_webpage/853ef78be7c0853208e865a9ff13d5fb/recovered.edits/0001556.temp
region=853ef78be7c0853208e865a9ff13d5fb
2013-10-15 02:02:09,443 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=39 entries
from hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816367672
2013-10-15 02:02:09,444 DEBUG
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog 29 of
29: hdfs://localhost:8020/hbase/.logs/127.0.0.1,60020,1381814216471/
127.0.0.1%3A60020.1381816657239, length=0

Thanks for your help!

Re: Cannot start resourcemanager

[Omkar Joshi]

Hi,

in capacity-scheduler.xml what value you have set it for below property?

 

yarn.scheduler.capacity.resource-calculator


org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator


  The ResourceCalculator implementation to be used to compare

  Resources in the scheduler.

  The default i.e. DefaultResourceCalculator only uses Memory while

  DominantResourceCalculator uses dominant-resource to compare

  multi-dimensional resources such as Memory, CPU etc.



  

Thanks,
Omkar Joshi
*Hortonworks Inc.* 


On Thu, Oct 17, 2013 at 12:48 PM, Arun C Murthy  wrote:

> What command did you use to start the RM?
>
> On Oct 17, 2013, at 10:18 AM, Siddharth Tiwari 
> wrote:
>
> Hi Team,
>
> trying to start resourcemanager in the latest hadoop-2.2.0 stable release.
> It throws following error. Please help
>
> 2013-10-17 10:01:51,230 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager
> metrics system...
> 2013-10-17 10:01:51,230 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
> system stopped.
> 2013-10-17 10:01:51,231 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics
> system shutdown complete.
> 2013-10-17 10:01:51,232 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error
> starting ResourceManager
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:263)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:249)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
> not found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
> ... 6 more
> 2013-10-17 10:01:51,239 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down ResourceManager at node1/192.168.147.101
>
>
>
> ****
> *Cheers !!!*
> *Siddharth Tiwari*
> Have a refreshing day !!!
> *"Every duty is holy, and devotion to duty is the highest form of worship
> of God.” *
> *"Maybe other people will try to limit me but I don't limit myself"*
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: Querying cluster nodes list

[John Lilley]

Ah, never mind.. it is getNodeReports()
john

From: John Lilley [mailto:john.lil...@redpoint.net]
Sent: Thursday, October 17, 2013 1:44 PM
To: user@hadoop.apache.org
Subject: Querying cluster nodes list

I thought mistakenly that getClusterMetrics() would return information about 
the cluster's node, or about a queue's nodes, but this doesn't seem to be true 
- it is only a count.   How can a YARN application query the available node 
list on a cluster and what resources are configured on each node?
Thanks
john

Querying cluster nodes list

[John Lilley]

I thought mistakenly that getClusterMetrics() would return information about 
the cluster's node, or about a queue's nodes, but this doesn't seem to be true 
- it is only a count.   How can a YARN application query the available node 
list on a cluster and what resources are configured on each node?
Thanks
john

Re: Cannot start resourcemanager

[Arun C Murthy]

What command did you use to start the RM?

On Oct 17, 2013, at 10:18 AM, Siddharth Tiwari  
wrote:

> Hi Team,
> 
> trying to start resourcemanager in the latest hadoop-2.2.0 stable release. It 
> throws following error. Please help
> 
> 2013-10-17 10:01:51,230 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager 
> metrics system...
> 2013-10-17 10:01:51,230 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
> system stopped.
> 2013-10-17 10:01:51,231 INFO 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
> system shutdown complete.
> 2013-10-17 10:01:51,232 FATAL 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
> ResourceManager
> java.lang.RuntimeException: java.lang.RuntimeException: 
> java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>  not found
>   at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:263)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:249)
>   at 
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: 
> Class 
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>  not found
>   at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)
>   at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)
>   ... 5 more
> Caused by: java.lang.ClassNotFoundException: Class 
> org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
>  not found
>   at 
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
>   at 
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)
>   ... 6 more
> 2013-10-17 10:01:51,239 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG: 
> /
> SHUTDOWN_MSG: Shutting down ResourceManager at node1/192.168.147.101
> 
> 
> 
> **
> Cheers !!!
> Siddharth Tiwari
> Have a refreshing day !!!
> "Every duty is holy, and devotion to duty is the highest form of worship of 
> God.” 
> "Maybe other people will try to limit me but I don't limit myself"

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Cannot start resourcemanager

[Siddharth Tiwari]

Hi Team,
trying to start resourcemanager in the latest hadoop-2.2.0 stable release. It 
throws following error. Please help
2013-10-17 10:01:51,230 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: 
Stopping ResourceManager metrics system...2013-10-17 10:01:51,230 INFO 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
system stopped.2013-10-17 10:01:51,231 INFO 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics 
system shutdown complete.2013-10-17 10:01:51,232 FATAL 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting 
ResourceManagerjava.lang.RuntimeException: java.lang.RuntimeException: 
java.lang.ClassNotFoundException: Class 
org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
 not found at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1752)   at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getResourceCalculator(CapacitySchedulerConfiguration.java:333)
   at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:263)
  at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:249)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)   
  at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:871)Caused
 by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class 
org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
 not foundat 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1720)   at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1744)   
... 5 moreCaused by: java.lang.ClassNotFoundException: Class 
org.apache.hadoop.yarn.server.resourcemanager.resource.DefaultResourceCalculator
 not found at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626) at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1718)   
... 6 more2013-10-17 10:01:51,239 INFO 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG: 
/SHUTDOWN_MSG: 
Shutting down ResourceManager at node1/192.168.147.101


**

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of 
God.” 

"Maybe other people will try to limit me but I don't limit myself"
  

Is hadoop 1.x series deprecated now ?

[hari]

Hi all,

As the version 2.2.0 is marked stable and is available for
general release, is version 1.x series deprecated now ?

As a Hadoop user, should I assume that the version 1.x series
is not going to be updated with newer updates e.g., newer
versions of HDFS. If that is going to be the case, can it be
marked as deprecated in the website.

Personally, I would have preferred the continuation of version 1.x series
due to its simplicity, as YARN tries to encompass things beyond
mapreduce and Hadoop has so far been about running
mapreduce jobs.

Also, is there any possibility that there would be
resource management features added to
the Hadoop 1.x series with its jobtracker/tasktracker model
(maybe backported from YARN) or
is the 1.x series not even considered for those types of features and
now has been completely abandoned.

Please excuse me if some of the points are not relevant or
important. I was just curious about the status of 1.x series as
I have been using it for quite a while now.

thanks,
Hari

Re: CDH4.4 and HBASE-8912 issue

[Boris Emelyanov]

On 17.10.2013 13:43, Harsh J wrote:

Moving tou...@hbase.apache.org.

HBASE-8912 is still unresolved, and 0.94.13 is just the targeted
version presently.

Are you certain this is the exact issue you're hitting? I believe you
can workaround this by removing the specific bad znode in ZK or so.
When starting up after your major upgrade, did you ensure cleaning
your ZK /hbase znode?

On Thu, Oct 17, 2013 at 2:55 PM, Boris Emelyanov  wrote:

Hello! I've just upgraded my hadoop test-cluster from cdh3 to cdh4.4 with
hbase-0.94.6 and have facedhttps://issues.apache.org/jira/browse/HBASE-8912
issue.

Suggested solution was to update hbase to version 0.94.13, which is absent
in cloudera distribution.

Is it possible to run pure hbase over cloudera hadoop?

Or how can i find if this bug is present in previous versions of cdh?

--
Best regards,

Boris Emelyanov.





Are you certain this is the exact issue you're hitting?


I gues so. Description suits just fine!


I believe you can workaround this by removing the specific bad znode

in ZK or so.

When I disable loadbalancer just after master startup (balance_switch
false), cluster works fine.
But when loadbalancer is enabled and it starts, it allways fails with
exception described in issue, allways with different regions.


When starting up after your major upgrade, did you ensure cleaning

your ZK /hbase znode?

Yes, of course.

--
Best regards,

Boris Emelyanov.

Re: CDH4.4 and HBASE-8912 issue

[Harsh J]

Moving to u...@hbase.apache.org.

HBASE-8912 is still unresolved, and 0.94.13 is just the targeted
version presently.

Are you certain this is the exact issue you're hitting? I believe you
can workaround this by removing the specific bad znode in ZK or so.
When starting up after your major upgrade, did you ensure cleaning
your ZK /hbase znode?

On Thu, Oct 17, 2013 at 2:55 PM, Boris Emelyanov  wrote:
> Hello! I've just upgraded my hadoop test-cluster from cdh3 to cdh4.4 with
> hbase-0.94.6 and have faced https://issues.apache.org/jira/browse/HBASE-8912
> issue.
>
> Suggested solution was to update hbase to version 0.94.13, which is absent
> in cloudera distribution.
>
> Is it possible to run pure hbase over cloudera hadoop?
>
> Or how can i find if this bug is present in previous versions of cdh?
>
> --
> Best regards,
>
> Boris Emelyanov.



-- 
Harsh J

Re: CDH4.4 and HBASE-8912 issue

[Ted Yu]

If I read Lars' comment on the JIRA correctly, HBASE-8912's target was moved to 
0.94.13

It is still open. Meaning, if there is no patch, the target may move to next 
release. 

Cheers

On Oct 17, 2013, at 2:25 AM, Boris Emelyanov  wrote:

> Hello! I've just upgraded my hadoop test-cluster from cdh3 to cdh4.4 with 
> hbase-0.94.6 and have faced https://issues.apache.org/jira/browse/HBASE-8912 
> issue.
> 
> Suggested solution was to update hbase to version 0.94.13, which is absent in 
> cloudera distribution.
> 
> Is it possible to run pure hbase over cloudera hadoop?
> 
> Or how can i find if this bug is present in previous versions of cdh?
> -- 
> Best regards,
> 
> Boris Emelyanov.

Re: is jdk required to run hadoop or jre alone is sufficient

[Harsh J]

No, you do not need JDK to just run Apache Hadoop.

On Thu, Oct 17, 2013 at 2:16 PM, oc tsdb  wrote:
> In general to run just hadoop without using any other tools like sqoop, do
> we need jdk ?
>
>
> On Wed, Oct 16, 2013 at 1:08 PM, Harsh J  wrote:
>>
>> You will need a JDK. Certain tools like Sqoop, etc. have a dependency
>> on JDK for compiling generated code at runtime, and will not function
>> without a JDK.
>>
>> On Wed, Oct 16, 2013 at 10:38 AM, oc tsdb  wrote:
>> > HI ,
>> >
>> > I would like to know if JRE alone is sufficient to run HADOOP services
>> > or
>> > JDK is required ?
>> >
>> > we are planning to install latest stable version of hadoop
>> >
>> > Thanks,
>> >
>> > Oc.tsdb
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

CDH4.4 and HBASE-8912 issue

[Boris Emelyanov]

Hello! I've just upgraded my hadoop test-cluster from cdh3 to cdh4.4 
with hbase-0.94.6 and have faced 
https://issues.apache.org/jira/browse/HBASE-8912 issue.


Suggested solution was to update hbase to version 0.94.13, which is 
absent in cloudera distribution.


Is it possible to run pure hbase over cloudera hadoop?

Or how can i find if this bug is present in previous versions of cdh?

--
Best regards,

Boris Emelyanov.

Re: is jdk required to run hadoop or jre alone is sufficient

[oc tsdb]

In general to run just hadoop without using any other tools like sqoop, do
we need jdk ?


On Wed, Oct 16, 2013 at 1:08 PM, Harsh J  wrote:

> You will need a JDK. Certain tools like Sqoop, etc. have a dependency
> on JDK for compiling generated code at runtime, and will not function
> without a JDK.
>
> On Wed, Oct 16, 2013 at 10:38 AM, oc tsdb  wrote:
> > HI ,
> >
> > I would like to know if JRE alone is sufficient to run HADOOP services or
> > JDK is required ?
> >
> > we are planning to install latest stable version of hadoop
> >
> > Thanks,
> >
> > Oc.tsdb
>
>
>
> --
> Harsh J
>