Just to see if the AM UI is accessible when the CLI fails. Seems like your issue is intermittent. RPC timeout for CLIs are set to 15 secs, so there could be several reasons for which the timeout occurs. Do you see any network/routing issue to connect to the host where the AM is running?
-Gour On 9/21/17, 12:31 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >Hi Gour, > >Will try to access the AM Web UI next time the issue happens. Is there >anything specific that should be checked within the AM UI ? Or is the test >just to see if AM UI is accessible at all ? > >Thanks, > >Manoj > >On Thu, Sep 21, 2017 at 11:26 AM, Gour Saha <gs...@hortonworks.com> wrote: > >> Are you able to go to the RM UI and load the ApplicationMaster web ui >>for >> this app? >> >> -Gour >> >> On 9/21/17, 11:00 AM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >> >> >Any thoughts ? >> > >> >On Mon, Sep 18, 2017 at 3:22 PM, Manoj Samel <manojsamelt...@gmail.com> >> >wrote: >> > >> >> >> >> CDH 5.5.1 cluster with Kerberos, slider version 0.80 >> >> >> >> Sometimes Slider commands start hanging >> >> >> >> slider list <app> --containers >> >> >> >> [r...@s-76zyl02.sys.az1.eng.pdx.wd ~]# slider list spas --containers >> >> 2017-09-18 21:44:45,659 [main] INFO tools.SliderUtils - JVM >>initialized >> >> into secure mode with kerberos realm BIGDATA >> >> Exception: Call From <host running command>/<host_ip> to >> >><slider_AM_HOST> >> >> failed on socket timeout exception: java.net.SocketTimeoutException: >> >> 15000 millis timeout while waiting for channel to be ready for read. >>ch >> >>: >> >> java.nio.channels.SocketChannel[connected local=/<slider >> >> command_host>:46777 remote=<host_running_slider_am>/<IP of host >>running >> >> slider am>:32120]; For more details see: http://wiki.apache.org/ >> >> hadoop/SocketTimeout >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> >> Method) >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance( >> >> NativeConstructorAccessorImpl.java:62) >> >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( >> >> DelegatingConstructorAccessorImpl.java:45) >> >> at >>java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> >> at org.apache.hadoop.net.NetUtils.wrapWithMessage( >> NetUtils.java:791) >> >> at >>org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >> >> at org.apache.hadoop.ipc.Client.call(Client.java:1403) >> >> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. >> >> invoke(ProtobufRpcEngine.java:230) >> >> at com.sun.proxy.$Proxy19.getLiveContainers(Unknown Source) >> >> at >> >>org.apache.slider.server.appmaster.rpc.SliderClusterProtocolProxy. >> >> getLiveContainers(SliderClusterProtocolProxy.java:229) >> >> at >> >>org.apache.slider.client.ipc.SliderClusterOperations.getContainers( >> >> SliderClusterOperations.java:458) >> >> at org.apache.slider.client.SliderClient.getContainers( >> >> SliderClient.java:2763) >> >> at org.apache.slider.client.SliderClient.actionList( >> >> SliderClient.java:2735) >> >> at org.apache.slider.client.SliderClient.exec( >> SliderClient.java:510) >> >> at org.apache.slider.client.SliderClient.runService( >> >> SliderClient.java:424) >> >> at org.apache.slider.core.main.ServiceLauncher.launchService( >> >> ServiceLauncher.java:188) >> >> at >> >>org.apache.slider.core.main.ServiceLauncher.launchServiceRobustly( >> >> ServiceLauncher.java:475) >> >> at org.apache.slider.core.main.ServiceLauncher. >> launchServiceAndExit( >> >> ServiceLauncher.java:403) >> >> at org.apache.slider.core.main.ServiceLauncher.serviceMain( >> >> ServiceLauncher.java:630) >> >> at org.apache.slider.Slider.main(Slider.java:49) >> >> Caused by: java.net.SocketTimeoutException: 15000 millis timeout >>while >> >> waiting for channel to be ready for read. ch : >> >>java.nio.channels.SocketChannel[connected >> >> local=/<Local_IP>:46777 >> >>remote=<Slider_AM_HOST>/<slider_am_host_ip>:32120] >> >> at org.apache.hadoop.net.SocketIOWithTimeout.doIO( >> >> SocketIOWithTimeout.java:164) >> >> at org.apache.hadoop.net.SocketInputStream.read( >> >> SocketInputStream.java:161) >> >> at org.apache.hadoop.net.SocketInputStream.read( >> >> SocketInputStream.java:131) >> >> at java.io.FilterInputStream.read(FilterInputStream.java:133) >> >> at java.io.FilterInputStream.read(FilterInputStream.java:133) >> >> at org.apache.hadoop.ipc.Client$Connection$PingInputStream. >> >> read(Client.java:515) >> >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) >> >> at java.io.BufferedInputStream.read(BufferedInputStream.java:265) >> >> at java.io.DataInputStream.readInt(DataInputStream.java:387) >> >> at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse( >> >> Client.java:1075) >> >> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:970) >> >> 2017-09-18 21:45:01,499 [main] INFO util.ExitUtil - Exiting with >> >>status 56 >> >> >> >> >> >> Slider AM Log Shows no errors. The only warning I can see is about >>TGT >> >> renewer >> >> >> >> 2017-09-18 15:40:57,009 [TGT Renewer for xyz@mydomain] WARN >> >> security.UserGroupInformation - Exception encountered while running >>the >> >> renewal command. Aborting renew thread. ExitCodeException exitCode=1: >> >> kinit: Ticket expired while renewing credentials >> >> 2017-09-18 15:43:29,536 [Socket Reader #1 for port 32120] INFO >> >>ipc.Server >> >> - Auth successful for xyz@mydomain (auth:SIMPLE) >> >> 2017-09-18 15:43:29,537 [Socket Reader #1 for port 32120] INFO >> >>authorize.ServiceAuthorizationManager >> >> - Authorization successful for xyz@mydomain (auth:TOKEN) for >> >> protocol=interface org.apache.slider.server.appmaster.rpc. >> >> SliderClusterProtocolPB >> >> 2017-09-18 15:48:29,569 [Socket Reader #1 for port 32120] INFO >> >>ipc.Server >> >> - Auth successful for xyz@mydomain (auth:SIMPLE) >> >> 2017-09-18 15:48:29,570 [Socket Reader #1 for port 32120] INFO >> >>authorize.ServiceAuthorizationManager >> >> - Authorization successful for xyz@mydomain (auth:TOKEN) for >> >> protocol=interface org.apache.slider.server.appmaster.rpc. >> >> SliderClusterProtocolPB >> >> >> >>
