Re: Can anyone tell me the current typical memory specification, switch size and disk space

[Chris MacKenzie]

Thanks Adaryl,

I’m currently looking at Tom White p298, published May 2012, which
references a 2010 spec. Both Tom and Eric's books where published in 2012
so the information in both will be a tad dated no doubt.

What I need to know is the current:

Processor average spec
Memory spec
Disk storage spec
Network speed.

Can you help me out with that ?

Thanks in advance,

Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: stu...@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk









On 01/08/2014 17:28, "Adaryl "Bob" Wakefield, MBA"
 wrote:

>The book Hadoop Operations by Eric Sammer helped answer a lot of these
>questions for me.
>
>
>Adaryl "Bob" Wakefield, MBA
>Principal
>Mass Street Analytics
>913.938.6685
>www.linkedin.com/in/bobwakefieldmba
>-Original Message-
>From: Chris MacKenzie
>Sent: Friday, August 01, 2014 4:35 AM
>To: user@hadoop.apache.org
>Subject: Can anyone tell me the current typical memory specification,
>switch 
>size and disk space
>
>Hi,
>
>I¹d really appreciate it if someone could let me know the current
>preferred specification for a cluster set up.
>
>On average how many nodes
>Disk space
>Memory
>Switch size
>
>A link to a paper or discussion would be much appreciated.
>
>Thanks in advance
>
>
>Regards,
>
>Chris MacKenzie
>telephone: 0131 332 6967
>email: stu...@chrismackenziephotography.co.uk
>corporate: www.chrismackenziephotography.co.uk
>
>
>
>
>

Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN

[arthur.hk.c...@gmail.com]

Hi,

I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying 
automatic failover, after killing the process of namenode from Active one, the 
name node was not failover to standby node, 

Please advise
Regards
Arthur


2014-08-04 18:54:40,453 WARN 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
roll of the active NN
java.net.ConnectException: Call From standbynode  to  activenode:8020 failed on 
connection exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1414)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
... 11 more
2014-08-04 18:55:03,458 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from 
activenode:54571 Call#17 Retry#1: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:06,683 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from 
activenode:54571 Call#17 Retry#3: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:16,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 
activenode:54602 Call#0 Retry#1: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:19,530 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from 
activenode:54610 Call#17 Retry#5: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:20,756 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 
activenode:54602 Call#0 Retry#3: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
 

Re: Exception in hadoop and java

[Wellington Chevreuil]

These indicates some lib versions conflicts -> UnsupportedOperationException:  
setXIncludeAware is not supported on this JAXP implementation or earlier: class 
gnu.xml.dom.JAXPFactory

That classe is in gnujaxp jar. This chart api probably brought different 
version for this lib, from the version expected by hadoop code to parse 
configurations. You should remove all dependencies for this chart api. Have you 
tried remove this version from your maven repository?

On 3 Aug 2014, at 05:49, Ekta Agrawal  wrote:

> Hi,
> 
> I am writing a code in java that connects to hadoop. Earlier it was running 
> fine. I wanted to add some charts and I used jfree api,it started giving this 
> error.chart is not using hadoop. I removed the chart ,but it keeps coming.If 
> anybody can look into it and help me in understanding that why this error 
> came up and how can I handle it? 
> 
> 14/08/02 21:33:01 ERROR conf.Configuration: Failed to set 
> setXIncludeAware(true) for parser 
> gnu.xml.dom.JAXPFactory@8f2ca6:java.lang.UnsupportedOperationException:  
> setXIncludeAware is not supported on this JAXP implementation or earlier: 
> class gnu.xml.dom.JAXPFactory
> java.lang.UnsupportedOperationException:  setXIncludeAware is not supported 
> on this JAXP implementation or earlier: class gnu.xml.dom.JAXPFactory
> at 
> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:589)
> at 
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1143)
> at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1119)
> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1063)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:470)
> at org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:131)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
> at myclass1.myfunction1(myclass1.java:39)
> at myclass1.main(myclass1.java:25)
> 
> Thanks,
> Ekta

RE: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN

[Brahma Reddy Battula]

HI,


DO you mean Active Namenode which is killed is not transition to STANDBY..?

>>> Here Namenode will not start as standby if you kill..Again you need to 
>>> start manually.

  Automatic failover means when over Active goes down Standy Node will 
transition to Active automatically..it's not like starting killed process and 
making the Active(which is standby.)

Please refer the following doc for same ..( Section : Verifying automatic 
failover)

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html

OR

 DO you mean Standby Namenode is not transition to ACTIVE..?

 Please check ZKFC logs,, Mostly this might not happen from the logs you 
 pasted



Thanks & Regards



Brahma Reddy Battula





From: arthur.hk.c...@gmail.com [arthur.hk.c...@gmail.com]
Sent: Monday, August 04, 2014 4:38 PM
To: user@hadoop.apache.org
Cc: arthur.hk.c...@gmail.com
Subject: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a 
roll of the active NN

Hi,

I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying 
automatic failover, after killing the process of namenode from Active one, the 
name node was not failover to standby node,

Please advise
Regards
Arthur


2014-08-04 18:54:40,453 WARN 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
roll of the active NN
java.net.ConnectException: Call From standbynode  to  activenode:8020 failed on 
connection exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1414)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
... 11 more
2014-08-04 18:55:03,458 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from 
activenode:54571 Call#17 Retry#1: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:06,683 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing from 
activenode:54571 Call#17 Retry#3: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:16,643 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getFileInfo from 
activenode:54602 Call#0 Retry#1: org.apache.hadoop.ipc.StandbyException: 
Operation category READ is not supported in state standby
2014-08-04 18:55:19,530 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 
on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing fr

Re: Exception in hadoop and java

[Ekta Agrawal]

I tried removing gnujaxp jar from my project .It worked. Thanks..:)


On Mon, Aug 4, 2014 at 4:44 PM, Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> These indicates some lib versions conflicts ->
> *UnsupportedOperationException:**  setXIncludeAware is not supported on
> this JAXP implementation or earlier: class gnu.xml.dom.JAXPFactory*
>
> That classe is in gnujaxp jar. This chart api probably brought different
> version for this lib, from the version expected by hadoop code to parse
> configurations. You should remove all dependencies for this chart api. Have
> you tried remove this version from your maven repository?
>
> On 3 Aug 2014, at 05:49, Ekta Agrawal  wrote:
>
> Hi,
>
> I am writing a code in java that connects to hadoop. Earlier it was
> running fine. I wanted to add some charts and I used jfree api,it started
> giving this error.chart is not using hadoop. I removed the chart ,but it
> keeps coming.If anybody can look into it and help me in understanding that
> why this error came up and how can I handle it?
>
> *14/08/02 21:33:01 ERROR conf.Configuration: Failed to set
> setXIncludeAware(true) for parser gnu.xml.dom.JAXPFactory@8f2ca6*
> *:java.lang.**UnsupportedOperationException:*
> *  setXIncludeAware is not supported on this JAXP implementation or
> earlier: class gnu.xml.dom.JAXPFactory java.lang.*
> *UnsupportedOperationException:*
> *  setXIncludeAware is not supported on this JAXP implementation or
> earlier: class gnu.xml.dom.JAXPFactory at javax.xml.parsers.*
> *DocumentBuilderFactory.**setXIncludeAware(**DocumentBuilderFactory.java:*
> *589) at org.apache.hadoop.conf.**Configuration.loadResource(*
> *Configuration.java:1143)at org.apache.hadoop.conf.*
> *Configuration.loadResources(*
> *Configuration.java:1119) at org.apache.hadoop.conf.*
> *Configuration.getProps(*
> *Configuration.java:1063) at org.apache.hadoop.conf.*
> *Configuration.get(*
> *Configuration.java:470) at org.apache.hadoop.fs.*
> *FileSystem.getDefaultUri(*
> *FileSystem.java:131)at org.apache.hadoop.fs.*
> *FileSystem.get(FileSystem.*
> *java:123) at myclass1.myfunction1(**myclass1**.*
> *java:39) at myclass1.main(**myclass1**.java:25)*
>
> Thanks,
> Ekta
>
>
>

Re: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException)

[Ana Gillan]

Hi,

Unfortunately, after I set my user¹s ulimit ­n to  65536, I still get the
same bad performance, killed containers and errors as before.

I collected together a bunch of logs around the moment when the containers
are being killed (application master log, killed container log, hadoop-hdfs
logs, hadoop-yarn logs) and this is what¹s happening in order. The
PrivilegedActionException, is that something? Is my user not in some sort of
correct group?

Thanks in advance!
Ana

2014-08-04 12:21:47,868 INFO [IPC Server handler 4 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from
attempt_1403771939632_0413_m_06_0
2014-08-04 12:21:47,979 INFO [IPC Server handler 5 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from
attempt_1403771939632_0413_m_00_0
2014-08-04 12:21:48,102 INFO [IPC Server handler 6 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1403771939632_0413_m_02_0 is : 0.061734322
2014-08-04 12:21:48,102 INFO [IPC Server handler 6 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
attempt_1403771939632_0413_m_02_0
2014-08-04 12:21:48,110 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock: 
/tmp/hive-zslf023/hive_2014-08-04_12-16-12_475_8240516609553931504/_task_tmp
.-ext-10001/_tmp.06_1. BP-1198908146-10.0.0.11-1399888065184
blk_-300655159790361685_67008{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.0.0.44:50010|RBW],
ReplicaUnderConstruction[10.0.0.43:50010|RBW],
ReplicaUnderConstruction[10.0.0.40:50010|RBW]]}
2014-08-04 12:21:48,145 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.0.0.40:50010 is added to
blk_-300655159790361685_67008{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.0.0.44:50010|RBW],
ReplicaUnderConstruction[10.0.0.43:50010|RBW],
ReplicaUnderConstruction[10.0.0.40:50010|RBW]]} size 0
2014-08-04 12:21:48,145 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.0.0.43:50010 is added to
blk_-300655159790361685_67008{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.0.0.44:50010|RBW],
ReplicaUnderConstruction[10.0.0.43:50010|RBW],
ReplicaUnderConstruction[10.0.0.40:50010|RBW]]} size 0
2014-08-04 12:21:48,146 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: 10.0.0.44:50010 is added to
blk_-300655159790361685_67008{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[10.0.0.44:50010|RBW],
ReplicaUnderConstruction[10.0.0.43:50010|RBW],
ReplicaUnderConstruction[10.0.0.40:50010|RBW]]} size 0
2014-08-04 12:21:48,147 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: 
/tmp/hive-zslf023/hive_2014-08-04_12-16-12_475_8240516609553931504/_task_tmp
.-ext-10001/_tmp.06_1 is closed by
DFSClient_attempt_1403771939632_0413_m_06_1_-1142346950_1
2014-08-04 12:21:48,147 INFO
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions:
422 Total time for transactions(ms): 5 Number of transactions batched in
Syncs: 0 Number of syncs: 186 SyncTimes(ms): 204
2014-08-04 12:21:48,155 INFO [IPC Server handler 7 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1403771939632_0413_m_06_1 is : 0.93497133
2014-08-04 12:21:48,155 INFO [IPC Server handler 7 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
attempt_1403771939632_0413_m_06_1
2014-08-04 12:21:48,176 INFO [IPC Server handler 8 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
attempt_1403771939632_0413_m_06_1 is : 1.0
2014-08-04 12:21:48,176 INFO [IPC Server handler 8 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
attempt_1403771939632_0413_m_06_1
2014-08-04 12:21:48,177 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1403771939632_0413_m_06_1 TaskAttempt Transitioned from RUNNING
to SUCCESS_CONTAINER_CLEANUP
2014-08-04 12:21:48,177 INFO [IPC Server handler 9 on 49784]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
attempt_1403771939632_0413_m_06_1
2014-08-04 12:21:48,178 INFO [ContainerLauncher #4]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
attempt_1403771939632_0413_m_06_1
2014-08-04 12:21:48,178 INFO [ContainerLauncher #4]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
container_1403771939632_0413_01_16 taskAttempt
attempt_1403771939632_0413_m_06_1
2014-08-04 12:21:48,195 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1403771939632_0413_m_06_1 TaskAttempt Transitioned from
SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
2014-08-04 12:21:48,196 INFO [Asy

Re: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN

[arthur.hk.c...@gmail.com]

Hi,

Thanks for your reply.
It was about StandBy Namenode not promoted to Active.
Can you please advise what the path of ZKFC logs?  

"Similar to Namenode status web page, a Cluster Web Console is added in 
federation to monitor the federated cluster at 
http:///dfsclusterhealth.jsp. Any Namenode in the cluster can 
be used to access this web page” 
What is the default port for the cluster console? I tried 8088 but no luck.

Please advise.

Regards
Arthur




On 4 Aug, 2014, at 7:22 pm, Brahma Reddy Battula 
 wrote:

> HI,
> 
> 
> DO you mean Active Namenode which is killed is not transition to STANDBY..?
> 
> >>> Here Namenode will not start as standby if you kill..Again you need to 
> >>> start manually.
> 
>   Automatic failover means when over Active goes down Standy Node will 
> transition to Active automatically..it's not like starting killed process and 
> making the Active(which is standby.)
> 
> Please refer the following doc for same ..( Section : Verifying automatic 
> failover)
> 
> http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html
> 
> OR
> 
>  DO you mean Standby Namenode is not transition to ACTIVE..?
> 
>  Please check ZKFC logs,, Mostly this might not happen from the logs you 
>  pasted
> 
> 
> Thanks & Regards
>  
> Brahma Reddy Battula
>  
> 
> From: arthur.hk.c...@gmail.com [arthur.hk.c...@gmail.com]
> Sent: Monday, August 04, 2014 4:38 PM
> To: user@hadoop.apache.org
> Cc: arthur.hk.c...@gmail.com
> Subject: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger 
> a roll of the active NN
> 
> Hi,
> 
> I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying 
> automatic failover, after killing the process of namenode from Active one, 
> the name node was not failover to standby node, 
> 
> Please advise
> Regards
> Arthur
> 
> 
> 2014-08-04 18:54:40,453 WARN 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
> roll of the active NN
> java.net.ConnectException: Call From standbynode  to  activenode:8020 failed 
> on connection exception: java.net.ConnectException: Connection refused; For 
> more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
> at org.apache.hadoop.ipc.Client.call(Client.java:1414)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
> at 
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
> at 
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:604)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:699)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1462)
> at org.apache.hadoop.ipc.Client.call(Client.java:1381)
> ... 11 more
> 2014-08-04 18:55:03,458 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 1 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getListing 
> from activenode:54571 Call#17 Retry#1: 
> org.apache.hadoop.ipc.StandbyException: Operation category READ is not 
> supported in state standby
> 2014-08-04 18:55:06,683 INFO org.apac

Re: Fair Scheduler issue

[Julien Naour]

We are at 11GB for yarn nodemanager.resource.memory-mb
It seems that the problem is due to the number of CPUs.
Each Spark executor needed too many CPUs in comparaison to available CPUs.
In consequence the Fair Scheduler didn't allow all the available memory
because all CPUs where all-ready used.
Problem solved (or so it seems) by allowing less CPUs by Spark Executors

Thanks,

Julien


2014-08-02 21:13 GMT+02:00 Yehia Elshater :

> Hi Julien,
>
> Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for
> example (the other 3 for OS) ?
>
> Thanks
>
>
>
>
> On 1 August 2014 05:41, Julien Naour  wrote:
>
>> Hello,
>>
>> I'm currently using HDP 2.0 so it's Hadoop 2.2.0.
>> My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
>>
>> Recently we passed from 2 users to 8. We need now a more appropriate
>> Scheduler.
>> We begin with Capacity Scheduler. There was some issues with the
>> different queues particularly when using some spark shell that used some
>> resources for a long time.
>> So we decide to try Fair Scheduler which seems to be a good solution.
>> The problem is that FairScheduler doesn't allow all available resources.
>> It's capped at 73% of the available memory for one jobs 63% for 2 jobs and
>> 45% for 3 jobs. The problem could come from shells that take resources for
>> a long time.
>>
>> We tried some configuration like
>> yarn.scheduler.fair.user-as-default-queue=false
>> or play with the minimum ressources allocated minResources in
>> fair-scheduler.xml but it doesn't seems to resolve the issue.
>>
>> Any advices or good practices to held a good Fair Scheduler?
>>
>> Regards,
>>
>> Julien
>>
>
>

RE: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a roll of the active NN

[Brahma Reddy Battula]

ZKFC LOG:

By Default , it will be under HADOOP_HOME/logs/hadoop_**zkfc.log

Same can be confirmed by using the following commands(to get the log location)

jinfo 7370 | grep -i hadoop.log.dir

ps -eaf | grep -i DFSZKFailoverController | grep -i hadoop.log.dir

WEB Console :

And Default port for NameNode web console is 50070. you can check value of 
"dfs.namenode.http-address" in hdfs-site.xml..

Default values, you can check from the following link..

http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml





Thanks & Regards

Brahma Reddy Battula






From: arthur.hk.c...@gmail.com [arthur.hk.c...@gmail.com]
Sent: Monday, August 04, 2014 6:07 PM
To: user@hadoop.apache.org
Cc: arthur.hk.c...@gmail.com
Subject: Re: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to 
trigger a roll of the active NN

Hi,

Thanks for your reply.
It was about StandBy Namenode not promoted to Active.
Can you please advise what the path of ZKFC logs?

"Similar to Namenode status web page, a Cluster Web Console is added in 
federation to monitor the federated cluster at 
http:///dfsclusterhealth.jsp. Any Namenode in the cluster can 
be used to access this web page”
What is the default port for the cluster console? I tried 8088 but no luck.

Please advise.

Regards
Arthur




On 4 Aug, 2014, at 7:22 pm, Brahma Reddy Battula 
mailto:brahmareddy.batt...@huawei.com>> wrote:

HI,


DO you mean Active Namenode which is killed is not transition to STANDBY..?

>>> Here Namenode will not start as standby if you kill..Again you need to 
>>> start manually.

  Automatic failover means when over Active goes down Standy Node will 
transition to Active automatically..it's not like starting killed process and 
making the Active(which is standby.)

Please refer the following doc for same ..( Section : Verifying automatic 
failover)

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/HDFSHighAvailabilityWithNFS.html

OR

 DO you mean Standby Namenode is not transition to ACTIVE..?

 Please check ZKFC logs,, Mostly this might not happen from the logs you 
 pasted


Thanks & Regards



Brahma Reddy Battula




From: arthur.hk.c...@gmail.com 
[arthur.hk.c...@gmail.com]
Sent: Monday, August 04, 2014 4:38 PM
To: user@hadoop.apache.org
Cc: arthur.hk.c...@gmail.com
Subject: Hadoop 2.4.1 Verifying Automatic Failover Failed: Unable to trigger a 
roll of the active NN

Hi,

I have setup Hadoop 2.4.1 HA Cluster using Quorum Journal, I am verifying 
automatic failover, after killing the process of namenode from Active one, the 
name node was not failover to standby node,

Please advise
Regards
Arthur


2014-08-04 18:54:40,453 WARN 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unable to trigger a 
roll of the active NN
java.net.ConnectException: Call From standbynode  to  activenode:8020 failed on 
connection exception: java.net.ConnectException: Connection refused; For more 
details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
at org.apache.hadoop.ipc.Client.call(Client.java:1414)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy16.rollEditLog(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:139)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:271)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$600(EditLogTailer.java:61)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:313)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:299)
at 
org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at 
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:295)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at 
org.apache.hadoop.net.SocketIOWithTimeout

Re: Ideal number of mappers and reducers to increase performance

[Felix Chern]

The mapper and reducer numbers really depends on what your program is trying to 
do. Without your actual query it’s really difficult to tell why you are having 
this problem.

For example, if you tried to perform a global sum or count, cascalog will only 
use one reducer since this is the only way to do a global sum/count. To avoid 
this behavior you can set a output key that can generally split the reducer. 
e.g. word count example use word as the output key. With this word count output 
you can sum it up in a serial manner or run the global map reduce job with this 
much smaller input.

The mapper number is usually not a performance bottleneck. For your curious, if 
the file is splittable (ie, unzipped text or sequence file), the number of 
mapper number is controlled by the split size in configuration. The smaller the 
split size it is, the more mappers are queued.

In short, your problem is not likely to be a configuration problem, but 
misunderstood the map reduce logic. To solve your problem, can you paste your 
cascalog query and let people take a look?

Felix

On Aug 3, 2014, at 1:51 PM, Sindhu Hosamane  wrote:

> 
> I am not coding in mapreduce. I am running my cascalog queries on hadoop 
> cluster(1 node ) on data of size 280MB. So all the config settings has to be 
> made on hadoop cluster itself.
> As you said , i set the values of mapred.tasktracker.map.tasks.maximum =4  
>  and mapred.tasktracker.reduce.tasks.maximum = 4  
> and then kept tuning it up ways and down ways  like below 
> (4+4)   (5+3) (6+2) (2+6) (3+5) (3+3 ) (10+10)
> 
> But all the time performance remains same .
> Everytime, inspite whatever combination of 
> mapred.tasktracker.map.tasks.maximum and 
> mapred.tasktracker.reduce.tasks.maximum i use -  produces same execution time 
> .
> 
> Then when the above things failed i also tried mapred.reduce.tasks = 4 
> still results are same. No reduction in execution time.
> 
> What other things should i set? Also i made sure hadoop is restarted every 
> time after changing config.
> I have attached my conf folder ..please indicate me what should be added 
> where ?
> I am really stuck ..Your help would be much appreciated. Thank you .
> <(singlenodecuda)conf.zip>
> 
> Regards,
> Sindhu

Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.

[S.L]

The contents are

127.0.0.1   localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
localhost6.localdomain6



On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh <
riteshoneinamill...@gmail.com> wrote:

> check the contents of '/etc/hosts' file
>
>
> On Mon, Aug 4, 2014 at 3:27 AM, S.L  wrote:
>
>> Hi All,
>>
>> I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master and
>> three slave nodes , the slave nodes are listed in the
>> $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the
>> Master Name node on port 9000, however when I start the datanode on any of
>> the slaves I get the following exception .
>>
>> 2014-08-03 08:04:27,952 FATAL
>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>> block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode
>> Uuid null) service to server1.dealyaft.com/170.75.152.162:9000
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException):
>> Datanode denied communication with namenode because hostname cannot be
>> resolved .
>>
>> The following are the contents of my core-site.xml.
>>
>> 
>> 
>> fs.default.name
>> hdfs://server1.mydomain.com:9000
>> 
>> 
>>
>> Also in my hdfs-site.xml  I am not setting any value for dfs.hosts or
>> dfs.hosts.exclude properties.
>>
>> Thanks.
>>
>
>

Re: Ideal number of mappers and reducers to increase performance

[Sindhu Hosamane]

Thanks a lot for your explanation Felix .
MY query is not using global sort/count. But still i am unable to understand - 
even i set the mapped.reduce.tasks=4
when the hadoop job runs i still see 
14/08/03 15:01:48 INFO mapred.MapTask: numReduceTasks: 1
14/08/03 15:01:48 INFO mapred.MapTask: io.sort.mb = 100

Does that look ok , numReduceTasks should be 4 right ?
Also i am pasting my cascalog query below. Please point me where am i wrong. 
why is the performance not increased?

Cascalog code
(def info
  (hfs-delimited  "/users/si/File.txt"
   :delimiter ";"
   :outfields ["?timestamp" "?AIT" "?CET” “?BTT367" ]
   :classes [String String String String  ]
   :skip-header? true))
   
(defn convert-to-long [a]
 (ct/to-long (f/parse custom-formatter a)))

(def info-tap
  (<- [?timestamp  ?BTT367 ]
  ((select-fields info ["?timestamp"  "?BTT367"]) ?timestamp  ?BTT367)))

(defn convert-to-float [a] 
  (try
(if (not= a " ")
  (read-string a))
   (catch Exception e (do 
 nil

 (?<- (stdout) [?timestamp-out ?highest-value](info-tap ?timestamp ?BTT367)
  (convert-to-float ?BTT367 :> ?converted-BTT367 )
  (convert-to-long ?timestamp :> ?converted-timestamp)
  (>= ?converted-timestamp start-value)
  (<= ?converted-timestamp end-value)
  (:sort ?converted-BTT367)(:reverse true)
  (c/limit [1] ?timestamp ?converted-BTT367 :> ?timestamp-out 
?highest-value)) 


Regards,
Sindhu





On 04 Aug 2014, at 19:10, Felix Chern  wrote:

> The mapper and reducer numbers really depends on what your program is trying 
> to do. Without your actual query it’s really difficult to tell why you are 
> having this problem.
> 
> For example, if you tried to perform a global sum or count, cascalog will 
> only use one reducer since this is the only way to do a global sum/count. To 
> avoid this behavior you can set a output key that can generally split the 
> reducer. e.g. word count example use word as the output key. With this word 
> count output you can sum it up in a serial manner or run the global map 
> reduce job with this much smaller input.
> 
> The mapper number is usually not a performance bottleneck. For your curious, 
> if the file is splittable (ie, unzipped text or sequence file), the number of 
> mapper number is controlled by the split size in configuration. The smaller 
> the split size it is, the more mappers are queued.
> 
> In short, your problem is not likely to be a configuration problem, but 
> misunderstood the map reduce logic. To solve your problem, can you paste your 
> cascalog query and let people take a look?
> 
> Felix
> 
> On Aug 3, 2014, at 1:51 PM, Sindhu Hosamane  wrote:
> 
>> 
>> I am not coding in mapreduce. I am running my cascalog queries on hadoop 
>> cluster(1 node ) on data of size 280MB. So all the config settings has to be 
>> made on hadoop cluster itself.
>> As you said , i set the values of mapred.tasktracker.map.tasks.maximum =4  
>>  and mapred.tasktracker.reduce.tasks.maximum = 4  
>> and then kept tuning it up ways and down ways  like below 
>> (4+4)   (5+3) (6+2) (2+6) (3+5) (3+3 ) (10+10)
>> 
>> But all the time performance remains same .
>> Everytime, inspite whatever combination of 
>> mapred.tasktracker.map.tasks.maximum and 
>> mapred.tasktracker.reduce.tasks.maximum i use -  produces same execution 
>> time .
>> 
>> Then when the above things failed i also tried mapred.reduce.tasks = 4 
>> still results are same. No reduction in execution time.
>> 
>> What other things should i set? Also i made sure hadoop is restarted every 
>> time after changing config.
>> I have attached my conf folder ..please indicate me what should be added 
>> where ?
>> I am really stuck ..Your help would be much appreciated. Thank you .
>> <(singlenodecuda)conf.zip>
>> 
>> Regards,
>> Sindhu
> 

Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.

[Satyam Singh]

You have not given namenode uri in /etc/hosts file , thus it can't 
resolve it to ipaddress and your namenode would also be not started.
Preferable practice is to start your cluster through start-dfs.sh 
command, it implicitly starts first namenode and then all its datanodes.


Also make sure you have given ipaddress in salve file, if not then also 
make entry for hostnames in /etc/hosts file



BR,
Satyam

On 08/05/2014 12:21 AM, S.L wrote:


The contents are

127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 
localhost6.localdomain6




On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh 
mailto:riteshoneinamill...@gmail.com>> 
wrote:


check the contents of '/etc/hosts' file


On Mon, Aug 4, 2014 at 3:27 AM, S.L mailto:simpleliving...@gmail.com>> wrote:

Hi All,

I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a
master and three slave nodes , the slave nodes are listed in
the $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from
the slaves to the Master Name node on port 9000, however when
I start the datanode on any of the slaves I get the following
exception .

2014-08-03 08:04:27,952 FATAL
org.apache.hadoop.hdfs.server.datanode.DataNode:
Initialization failed for block pool Block pool
BP-1086620743-170.75.152.162-1407064313305 (Datanode Uuid
null) service to server1.dealyaft.com/170.75.152.162:9000


org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException):
Datanode denied communication with namenode because hostname
cannot be resolved .

The following are the contents of my core-site.xml.



fs.default.name 
hdfs://server1.mydomain.com:9000




Also in my hdfs-site.xml  I am not setting any value for
dfs.hosts or dfs.hosts.exclude properties.

Thanks.

Re: Datanode not allowed to connect to the Namenode in Hadoop 2.3.0 cluster.

[S.L]

when you say /etc/hosts/ file , you mean only on the master of on both the
master and slaves?




On Tue, Aug 5, 2014 at 1:20 AM, Satyam Singh 
wrote:

>  You have not given namenode uri in /etc/hosts file , thus it can't
> resolve it to ipaddress and your namenode would also be not started.
> Preferable practice is to start your cluster through start-dfs.sh command,
> it implicitly starts first namenode and then all its datanodes.
>
> Also make sure you have given ipaddress in salve file, if not then also
> make entry for hostnames in /etc/hosts file
>
>
> BR,
> Satyam
>
> On 08/05/2014 12:21 AM, S.L wrote:
>
>
>  The contents are
>
> 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4
> ::1 localhost localhost.localdomain localhost6
> localhost6.localdomain6
>
>
>
> On Sun, Aug 3, 2014 at 11:21 PM, Ritesh Kumar Singh <
> riteshoneinamill...@gmail.com> wrote:
>
>> check the contents of '/etc/hosts' file
>>
>>
>>  On Mon, Aug 4, 2014 at 3:27 AM, S.L  wrote:
>>
>>>   Hi All,
>>>
>>> I am trying to set up a Apache Hadoop 2.3.0 cluster , I have a master
>>> and three slave nodes , the slave nodes are listed in the
>>> $HADOOP_HOME/etc/hadoop/slaves file and I can telnet from the slaves to the
>>> Master Name node on port 9000, however when I start the datanode on any of
>>> the slaves I get the following exception .
>>>
>>> 2014-08-03 08:04:27,952 FATAL
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
>>> block pool Block pool BP-1086620743-170.75.152.162-1407064313305 (Datanode
>>> Uuid null) service to server1.dealyaft.com/170.75.152.162:9000
>>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException):
>>> Datanode denied communication with namenode because hostname cannot be
>>> resolved .
>>>
>>>  The following are the contents of my core-site.xml.
>>>
>>> 
>>> 
>>> fs.default.name
>>> hdfs://server1.mydomain.com:9000
>>> 
>>> 
>>>
>>>  Also in my hdfs-site.xml  I am not setting any value for dfs.hosts or
>>> dfs.hosts.exclude properties.
>>>
>>>  Thanks.
>>>
>>
>>
>
>

hadoop 2.4.1 build failure on CentOS 6

[Romu]

Hi,

I tried to build hadoop 2.4.1 in a CentOS 6 x86_64 vm but failed.  Maven
3.2.2 (installed from official bin tar ball)


   - Maven: 3.2.2
   - Java: Oracle jdk 1.8.0_11
   - Build command: mvn package -X -e -Pdist,native,docs -Dskiptests
   -Dadditionalparam=-Xdoclint:none

Error message:

[INFO]


[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main . SUCCESS [
 1.993 s]

[INFO] Apache Hadoop Project POM .. SUCCESS [
 1.130 s]

[INFO] Apache Hadoop Annotations .. SUCCESS [
 3.706 s]

[INFO] Apache Hadoop Assemblies ... SUCCESS [
 0.193 s]

[INFO] Apache Hadoop Project Dist POM . SUCCESS [
 3.118 s]

[INFO] Apache Hadoop Maven Plugins  SUCCESS [
 6.921 s]

[INFO] Apache Hadoop MiniKDC .. SUCCESS [
11.463 s]

[INFO] Apache Hadoop Auth . SUCCESS [
52.770 s]

[INFO] Apache Hadoop Auth Examples  SUCCESS [
 2.185 s]

[INFO] Apache Hadoop Common ... FAILURE [
13.456 s]

[INFO] Apache Hadoop NFS .. SKIPPED

[INFO] Apache Hadoop Common Project ... SKIPPED

[INFO] Apache Hadoop HDFS . SKIPPED

[INFO] Apache Hadoop HttpFS ... SKIPPED

[INFO] Apache Hadoop HDFS BookKeeper Journal .. SKIPPED

[INFO] Apache Hadoop HDFS-NFS . SKIPPED

[INFO] Apache Hadoop HDFS Project . SKIPPED

[INFO] hadoop-yarn  SKIPPED

[INFO] hadoop-yarn-api  SKIPPED

[INFO] hadoop-yarn-common . SKIPPED

[INFO] hadoop-yarn-server . SKIPPED

[INFO] hadoop-yarn-server-common .. SKIPPED

[INFO] hadoop-yarn-server-nodemanager . SKIPPED

[INFO] hadoop-yarn-server-web-proxy ... SKIPPED

[INFO] hadoop-yarn-server-applicationhistoryservice ... SKIPPED

[INFO] hadoop-yarn-server-resourcemanager . SKIPPED

[INFO] hadoop-yarn-server-tests ... SKIPPED

[INFO] hadoop-yarn-client . SKIPPED

[INFO] hadoop-yarn-applications ... SKIPPED

[INFO] hadoop-yarn-applications-distributedshell .. SKIPPED

[INFO] hadoop-yarn-applications-unmanaged-am-launcher . SKIPPED

[INFO] hadoop-yarn-site ... SKIPPED

[INFO] hadoop-yarn-project  SKIPPED

[INFO] hadoop-mapreduce-client  SKIPPED

[INFO] hadoop-mapreduce-client-core ... SKIPPED

[INFO] hadoop-mapreduce-client-common . SKIPPED

[INFO] hadoop-mapreduce-client-shuffle  SKIPPED

[INFO] hadoop-mapreduce-client-app  SKIPPED

[INFO] hadoop-mapreduce-client-hs . SKIPPED

[INFO] hadoop-mapreduce-client-jobclient .. SKIPPED

[INFO] hadoop-mapreduce-client-hs-plugins . SKIPPED

[INFO] Apache Hadoop MapReduce Examples ... SKIPPED

[INFO] hadoop-mapreduce ... SKIPPED

[INFO] Apache Hadoop MapReduce Streaming .. SKIPPED

[INFO] Apache Hadoop Distributed Copy . SKIPPED

[INFO] Apache Hadoop Archives . SKIPPED

[INFO] Apache Hadoop Rumen  SKIPPED

[INFO] Apache Hadoop Gridmix .. SKIPPED

[INFO] Apache Hadoop Data Join  SKIPPED

[INFO] Apache Hadoop Extras ... SKIPPED

[INFO] Apache Hadoop Pipes  SKIPPED

[INFO] Apache Hadoop OpenStack support  SKIPPED

[INFO] Apache Hadoop Client ... SKIPPED

[INFO] Apache Hadoop Mini-Cluster . SKIPPED

[INFO] Apache Hadoop Scheduler Load Simulator . SKIPPED

[INFO] Apache Hadoop Tools Dist ... SKIPPED

[INFO] Apache Hadoop Tools  SKIPPED

[INFO] Apache Hadoop Distribution . SKIPPED

[INFO]


[INFO] BUILD FAILURE

[INFO]


[INFO] Total time: 01:38 min

[INFO] Finished at: 2014-08-05T14:01:49+08:00

[INFO] Final Memory: 77M/316M

[INFO]


[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on