Hi
Slider version .80 on secure cluster.
In my xxx-site.xml files, the
<property>
<name>hadoop.registry.zk.quorum</name>
<value>zk1_host:2181,zk2_host:2181,zk3_host:2181</value>
</property>
However, it appears slider AM uses only the first ZK to connect for
registry - and fails when the first ZK happens to be down.
In the slider AM log
2016-09-30 02:27:27,279 [main] INFO appmaster.SliderAppMaster - Loading
slider-server.xml at
file:/foo/yarn/local/usercache/xx/appcache/application_1474675565244_3660/container_e80_1474675565244_3660_01_000001/confdir/slider-server.xml
2016-09-30 02:27:27,285 [main] INFO appmaster.SliderAppMaster - AM
configuration:
dfs.namenode.kerberos.principal=hdfs/_HOST@ABC
hadoop.registry.zk.quorum=zk1_host:2181
hadoop.registry.zk.root=/registry
Note -- the log shows only the first host, not the quorum string of 3
host:ports
later in log, it tries to connect to ZK1 but since ZK1 is down, the
connection fails. The AM fails start any components as a result.
2016-09-29 23:32:49,768 [main] INFO appmaster.SliderAppMaster - Service
YarnRegistry in state YarnRegistry: STARTED Connection="fixed ZK quorum
"zk1_host:2181" " root="/registry" security disabled
2016-09-29 23:32:49,774 [main-SendThread(bds0211.svc.eng.pdx.wd:2181)] WARN
zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error,
closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
I would expect that if connection to ZK1 failed, then ZK2, 3 ... etc would
be tried .. thats what the ZK quorum is for.
Looking into the code, I see this last "Connection" string is coming
from org.apache.hadoop.registry.client.impl.zk.CuratorService.java
In it, supplyBindingInformation() gets and prints the string in log message.
public BindingInformation supplyBindingInformation() {
BindingInformation binding = new BindingInformation();
String connectString = buildConnectionString();
binding.ensembleProvider = new FixedEnsembleProvider(connectString);
binding.description =
"fixed ZK quorum \"" + connectString + "\"";
return binding;
}
protected String buildConnectionString() {
return getConfig().getTrimmed(KEY_REGISTRY_ZK_QUORUM,
DEFAULT_REGISTRY_ZK_QUORUM);
}
the getConfig() is from org.apache.hadoop.conf.Configuration.java
Its not clear why the value of hadoop.registry.zk.quorum supplied in config
gets trimmed to first host only. Is this the expected behavior ? Or Bug?
It can't be possible to guarantee that the first zookeeper in quorum will
always be reachable .. I would expect multiple nodes in quorum to be tried
for connection
Any thoughts would be appreciated ...
