[ https://issues.apache.org/jira/browse/AMBARI-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar resolved AMBARI-10977. ------------------------------------ Resolution: Fixed Fixed in 2.1.. > HDFS Rebalance failed with IllegalArgumentException: Does not contain a valid > host:port authority > ------------------------------------------------------------------------------------------------- > > Key: AMBARI-10977 > URL: https://issues.apache.org/jira/browse/AMBARI-10977 > Project: Ambari > Issue Type: Bug > Environment: ambari-2.1.0-376, hdp-2.3.0.0-1880, sles11sp3 > Reporter: Zack Marsh > > The HDFS Rebalance is failing with the following error messages: > stderr: > {code} > 2015-05-06 12:31:46,656 - Error while executing command 'rebalancehdfs': > Traceback (most recent call last): > File > "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", > line 214, in execute > method(env) > File > "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", > line 243, in rebalancehdfs > logoutput = False, > File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", > line 148, in __init__ > self.env.run() > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 152, in run > self.run_action(resource, action) > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 118, in run_action > provider_action() > File > "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", > line 269, in action_run > raise ex > Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export > PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' > ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' > returned 255. May 6, 2015 12:31:46 PM Balancing took 888.0 milliseconds > 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: jolokia1.labs.teradata.com > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037) > at org.apache.hadoop.hdfs.DFSUtil.getNsServiceRpcUris(DFSUtil.java:978) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794) > {code} > stdout: > {code} > Starting balancer with threshold = 10 > Executing command ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export > PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' > ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10' > 2015-05-06 12:31:43,096 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c > 'export > PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' > ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold > 10''] {'logoutput': False, 'on_new_line': handle_new_line} > [balancer] May 6, 2015 12:31:46 PM [balancer] [balancer] Balancing took > 888.0 milliseconds[balancer] > [balancer] 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an > exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: jolokia1.labs.teradata.com > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037) > at org.apache.hadoop.hdfs.DFSUtil[balancer] > .getNsServiceRpcUris(DFSUtil.java:978) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794) > 2015-05-06 12:31:46,656 - Error while executing command 'rebalancehdfs': > Traceback (most recent call last): > File > "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", > line 214, in execute > method(env) > File > "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", > line 243, in rebalancehdfs > logoutput = False, > File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", > line 148, in __init__ > self.env.run() > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 152, in run > self.run_action(resource, action) > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 118, in run_action > provider_action() > File > "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", > line 269, in action_run > raise ex > Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export > PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' > ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' > returned 255. May 6, 2015 12:31:46 PM Balancing took 888.0 milliseconds > 15/05/06 12:31:46 ERROR balancer.Balancer: Exiting balancer due an exception > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: jolokia1.labs.teradata.com > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:213) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:153) > at org.apache.hadoop.hdfs.DFSUtil.getNameServiceUris(DFSUtil.java:1037) > at org.apache.hadoop.hdfs.DFSUtil.getNsServiceRpcUris(DFSUtil.java:978) > at > org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:682) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:794) > {code} > The values of the namenode rpc address properties are as follows: > {code} > dfs.namenode.rpc-address = <NN1 FQDN> > dfs.namenode.rpc-address.<CLUSTER-NAME>.nn1 = <NN1 FQDN>:8020 > dfs.namenode.rpc-address.<CLUSTER-NAME>.nn2 = <NN2FQDN>:8020 > {code} > Setting the plain dfs.namenode.rpc-address property to the Active Namenode at > port 8020 allows the rebalance to succeed. > However, if this property is set to the Stand-by namenode at port 8020 the > rebalance fails with the error: > {code} > 2015-05-06 12:45:59,673 - Error while executing command 'rebalancehdfs': > Traceback (most recent call last): > File > "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", > line 214, in execute > method(env) > File > "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", > line 243, in rebalancehdfs > logoutput = False, > File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", > line 148, in __init__ > self.env.run() > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 152, in run > self.run_action(resource, action) > File > "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", > line 118, in run_action > provider_action() > File > "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", > line 269, in action_run > raise ex > Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export > PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/teradata/bynet/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/opt/teradata/sm3g/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/opt/teradata/dswap/sbin:/usr/tdbms/bin:/opt/teradata/gsctools/bin:/opt/teradata/vmf/bin:/opt/teradata/bin:/var/lib/ambari-agent:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' > ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' > returned 252. 15/05/06 12:45:56 INFO balancer.Balancer: Using a threshold of > 10.0 > 15/05/06 12:45:56 INFO balancer.Balancer: namenodes = > [hdfs://jolokia1.labs.teradata.com:8020, hdfs://JOLOKIA] > 15/05/06 12:45:56 INFO balancer.Balancer: parameters = > Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration > = 5, number of nodes to be excluded = 0, number of nodes to be included = 0] > Time Stamp Iteration# Bytes Already Moved Bytes Left To Move > Bytes Being Moved > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): > Operation category READ is not supported in state standby > at > org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87) > at > org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1785) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1301) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getServerDefaults(FSNamesystem.java:1613) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getServerDefaults(NameNodeRpcServer.java:593) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getServerDefaults(ClientNamenodeProtocolServerSideTranslatorPB.java:383) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) > . Exiting ... > May 6, 2015 12:45:59 PM Balancing took 3.127 seconds > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)