Hi, Could you please provide logs from containers? Also IGNITE_PATH property incorrect. The property should contains path to apache ignite zip archive. For example: /ignite/apache-ignite-fabric-1.7.0-bin.zip. Also IGNITE_USERS_LIBS not needed here. The property need to use when you want deploy to cluster your own libs.
On Thu, Aug 11, 2016 at 7:54 AM, percent620 <percent...@163.com> wrote: > Thanks vkulichenko's quick response. > > Here is my detailed steps about how to deploy and integration spark with > ignite as below. > > 1、Followed this guidelines about how to deploy ignite-yarn application. > <http://apache-ignite-users.70518.x6.nabble.com/file/n6941/ignite-yarn.png > > > > it's successfully and the log is displayed ok > ================================== > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/ > common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/ > appcache/application_1455892346017_5077/filecache/ > 10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 16/08/10 22:54:57 INFO impl.ContainerManagementProtocolProxy: > yarn.client.max-cached-nodemanagers-proxies : 0 > Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Application master registered. > Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > Aug 10, 2016 10:54:58 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > 16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for : > vmsecdomain010194062066.cm10:61362 > 16/08/10 22:54:59 INFO impl.AMRMClientImpl: Received new token for : > vmsecdomain010194062042.cm10:42077 > Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster > onContainersAllocated > INFO: Launching container: container_1455892346017_5077_02_000002. > 16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > : vmsecdomain010194062066.cm10:61362 > Aug 10, 2016 10:54:59 PM org.apache.ignite.yarn.ApplicationMaster > onContainersAllocated > INFO: Launching container: container_1455892346017_5077_02_000003. > 16/08/10 22:54:59 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > : vmsecdomain010194062042.cm10:42077 > Aug 10, 2016 10:55:08 PM org.apache.ignite.yarn.ApplicationMaster > onContainersCompleted > INFO: Container completed. Container id: > container_1455892346017_5077_02_000002. State: COMPLETE. > Aug 10, 2016 10:55:09 PM org.apache.ignite.yarn.ApplicationMaster > onContainersCompleted > INFO: Container completed. Container id: > container_1455892346017_5077_02_000003. State: COMPLETE. > > > 2、downloaded this apache-ignite-fabric-1.6.0-bin.zip and unzip this file > to > the /u01/XXX/apache-ignite-fabric-1.6.0-bin directory. > > 3、cluster16.properis content is as below > $cat cluster16.properties > # The number of nodes in the cluster. > IGNITE_NODE_COUNT=2 > # The number of CPU Cores for each Apache Ignite node. > IGNITE_RUN_CPU_PER_NODE=1 > # The number of Megabytes of RAM for each Apache Ignite node. > IGNITE_MEMORY_PER_NODE=2048 > # The version ignite which will be run on nodes. > IGNITE_VERSION=1.6.0 > IGNITE_WORK_DIR=/u01/XXX/apache-ignite-fabric-1.6.0-bin/ > IGNITE_XML_CONFIG=/ignite/release16/apache-ignite-fabric-1.6.0-bin/config/ > default-config.xml > IGNITE_RELEASES_DIR=/ignite/release16/ > IGNITE_USERS_LIBS=/u01/XXX/apache-ignite-fabric-1.6.0-bin/libs/ > IGNITE_PATH=/ignite/release16/ > > > hdfs directory is as below > =========================== > ./hdfs dfs -ls /ignite/ > drwxr-xr-x - hbase hbase 0 2016-08-10 17:07 /ignite/release16 > drwxr-xr-x - hbase hbase 0 2016-08-02 06:25 /ignite/releases > drwxr-xr-x - hbase hbase 0 2016-08-10 17:24 /ignite/workdir > -rw-r--r-- 3 hbase hbase 27710331 2016-08-10 17:04 /ignite/yarn > ============= > $./hdfs dfs -ls /ignite/release16 > drwxr-xr-x - hbase hbase 0 2016-08-11 12:44 > /ignite/release16/apache-ignite-fabric-1.6.0-bin > > ============= > > 4、Running spark code on yarn and the code is as below > val igniteContext = new IgniteContext[String, BaseLine](sc,() => new > IgniteConfiguration()) > > > the code is hanging on this and i think that this client can't connect this > server > > 5、from yarn console and found the following error message > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/u01/hbase/hadoop-2.5.0-cdh5.3.0/share/hadoop/ > common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/u01/hbase/tmp/nm-local-dir/usercache/hbase/ > appcache/application_1455892346017_5077/filecache/ > 10/ignite-yarn.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 16/08/10 17:24:16 INFO impl.ContainerManagementProtocolProxy: > yarn.client.max-cached-nodemanagers-proxies : 0 > Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Application master registered. > Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > Aug 10, 2016 5:24:16 PM org.apache.ignite.yarn.ApplicationMaster run > INFO: Making request. Memory: 2,432, cpu 1. > 16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for : > vmsecdomain010194054064.cm10:56307 > 16/08/10 17:24:17 INFO impl.AMRMClientImpl: Received new token for : > vmsecdomain010194062006.cm10:22640 > Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster > onContainersAllocated > INFO: Launching container: container_1455892346017_5077_01_000002. > 16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > : vmsecdomain010194054064.cm10:56307 > Aug 10, 2016 5:24:17 PM org.apache.ignite.yarn.ApplicationMaster > onContainersAllocated > INFO: Launching container: container_1455892346017_5077_01_000003. > 16/08/10 17:24:17 INFO impl.ContainerManagementProtocolProxy: Opening > proxy > : vmsecdomain010194062006.cm10:22640 > Aug 10, 2016 5:24:21 PM org.apache.ignite.yarn.ApplicationMaster > onContainersCompleted > INFO: Container completed. Container id: > container_1455892346017_5077_01_000003. State: COMPLETE. > Aug 10, 2016 5:24:22 PM org.apache.ignite.yarn.ApplicationMaster > onContainersCompleted > INFO: Container completed. Container id: > container_1455892346017_5077_01_000002. State: COMPLETE. > 16/08/10 22:47:26 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:26 WARN ipc.Client: Exception encountered while connecting > to > the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:26 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1. > Trying to fail over immediately. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1455892346017_5077_000001 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException( > RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException( > RPCUtil.java:104) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > ... 8 more > 16/08/10 22:47:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm2 > 16/08/10 22:47:26 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2 > after 1 fail over attempts. Trying to fail over after sleeping for 22339ms. > java.net.ConnectException: Call From > vmsecdomain010194062063.cm10/10.194.62.63 to > vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection > exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.net.NetUtils.wrapWithMessage( > NetUtils.java:783) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) > at org.apache.hadoop.ipc.Client.call(Client.java:1415) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect( > SocketChannelImpl.java:739) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect( > SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) > at org.apache.hadoop.ipc.Client$Connection.setupConnection( > Client.java:606) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams( > Client.java:700) > at org.apache.hadoop.ipc.Client$Connection.access$2800(Client. > java:367) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) > at org.apache.hadoop.ipc.Client.call(Client.java:1382) > ... 12 more > 16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm1 > 16/08/10 22:47:48 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:48 WARN ipc.Client: Exception encountered while connecting > to > the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:48 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1 > after 2 fail over attempts. Trying to fail over immediately. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1455892346017_5077_000001 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException( > RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException( > RPCUtil.java:104) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > ... 8 more > 16/08/10 22:47:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm2 > 16/08/10 22:47:48 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2 > after 3 fail over attempts. Trying to fail over after sleeping for 26407ms. > java.net.ConnectException: Call From > vmsecdomain010194062063.cm10/10.194.62.63 to > vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection > exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.net.NetUtils.wrapWithMessage( > NetUtils.java:783) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) > at org.apache.hadoop.ipc.Client.call(Client.java:1415) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect( > SocketChannelImpl.java:739) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect( > SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) > at org.apache.hadoop.ipc.Client$Connection.setupConnection( > Client.java:606) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams( > Client.java:700) > at org.apache.hadoop.ipc.Client$Connection.access$2800(Client. > java:367) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) > at org.apache.hadoop.ipc.Client.call(Client.java:1382) > ... 12 more > 16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm1 > 16/08/10 22:48:14 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:48:14 WARN ipc.Client: Exception encountered while connecting > to > the server : > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:48:14 WARN security.UserGroupInformation: > PriviledgedActionException as:hbase (auth:SIMPLE) > cause:org.apache.hadoop.ipc.RemoteException(org.apache. > hadoop.security.token.SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > 16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm1 > after 4 fail over attempts. Trying to fail over immediately. > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > AMRMToken from appattempt_1455892346017_5077_000001 > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException( > RPCUtil.java:53) > at > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException( > RPCUtil.java:104) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > Invalid AMRMToken from appattempt_1455892346017_5077_000001 > at org.apache.hadoop.ipc.Client.call(Client.java:1411) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > ... 8 more > 16/08/10 22:48:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm2 > 16/08/10 22:48:14 INFO retry.RetryInvocationHandler: Exception while > invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm2 > after 5 fail over attempts. Trying to fail over after sleeping for 19315ms. > java.net.ConnectException: Call From > vmsecdomain010194062063.cm10/10.194.62.63 to > vmsecdomain010194054035.cm10.tbsite.net:8030 failed on connection > exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.net.NetUtils.wrapWithMessage( > NetUtils.java:783) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) > at org.apache.hadoop.ipc.Client.call(Client.java:1415) > at org.apache.hadoop.ipc.Client.call(Client.java:1364) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy12.allocate(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy13.allocate(Unknown Source) > at > org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl. > allocate(AMRMClientImpl.java:278) > at > org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$ > HeartbeatThread.run(AMRMClientAsyncImpl.java:224) > Caused by: java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at sun.nio.ch.SocketChannelImpl.finishConnect( > SocketChannelImpl.java:739) > at > org.apache.hadoop.net.SocketIOWithTimeout.connect( > SocketIOWithTimeout.java:206) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) > at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) > at org.apache.hadoop.ipc.Client$Connection.setupConnection( > Client.java:606) > at org.apache.hadoop.ipc.Client$Connection.setupIOstreams( > Client.java:700) > at org.apache.hadoop.ipc.Client$Connection.access$2800(Client. > java:367) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463) > at org.apache.hadoop.ipc.Client.call(Client.java:1382) > ... 12 more > 16/08/10 22:48:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing > over to rm1 > > > Can anyone help me on this issue? I'm vary urgent for this issues, thanks > again!!!1 > > > > -- > View this message in context: http://apache-ignite-users. > 70518.x6.nabble.com/Ignite-for-Spark-on-YARN-Deployment-tp6910p6941.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. >
