Hello Jasson We probably have to update the documentation, but when using distributed scheduling, the mapreduce task must be submitted with the following extra configuration:
-Dyarn.resourcemanager.scheduler.address=127.0.0.1:8049 Essentially, the AM should talk to the DistributedScheduler interceptor running on the local NM. Let me know if that works. Cheers -Arun On Wed, Jun 14, 2017 at 12:18 AM, Konstantinos Karanasos < kkarana...@gmail.com> wrote: > Hi Wei, > > Can you please share your yarn-site.xml and the command you used for > running wordcount? > > Thanks, > Konstantinos > > On Tue, Jun 13, 2017 at 11:44 Jasson Chenwei <ynjassionc...@gmail.com> > wrote: > > > hi all > > > > I have set up a distributed scheduler using a new feature in Hadoop-3.0. > My > > Hadoop version is hadoop-3.0.0-appha3. I have enabled the > > opportunistic container and distributed scheduler in > > yarn-site.xml following the guide. But wordcount application master > failed > > to launch as follow: > > > > *2017-06-13 12:34:11,036 INFO [main] org.eclipse.jetty.server.Server: > > Started @5116ms* > > *2017-06-13 12:34:11,036 INFO [main] org.apache.hadoop.yarn.webapp. > WebApps: > > Web app mapreduce started at 45559* > > *2017-06-13 12:34:11,039 INFO [AsyncDispatcher event handler] > > org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: > JOB_CREATE > > job_1497050650910_0004* > > *2017-06-13 12:34:11,041 INFO [main] > > org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class > > java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: > > class org.apache.hadoop.ipc.DefaultRpcScheduler* > > *2017-06-13 12:34:11,042 INFO [Socket Reader #1 for port 36026] > > org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36026* > > *2017-06-13 12:34:11,045 INFO [IPC Server Responder] > > org.apache.hadoop.ipc.Server: IPC Server Responder: starting* > > *2017-06-13 12:34:11,045 INFO [IPC Server listener on 36026] > > org.apache.hadoop.ipc.Server: IPC Server listener on 36026: starting* > > *2017-06-13 12:34:11,083 INFO [main] > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > > nodeBlacklistingEnabled:true* > > *2017-06-13 12:34:11,083 INFO [main] > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > > maxTaskFailuresPerNode is 3* > > *2017-06-13 12:34:11,083 INFO [main] > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: > > blacklistDisablePercent is 33* > > *2017-06-13 12:34:11,090 INFO [main] > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the > > mappers will be scheduled using OPPORTUNISTIC containers* > > *2017-06-13 12:34:11,132 INFO [main] org.apache.hadoop.yarn.client. > RMProxy: > > Connecting to ResourceManager at localhost/127.0.0.1:8030 > > <http://127.0.0.1:8030>* > > *2017-06-13 12:34:11,193 WARN [main] org.apache.hadoop.ipc.Client: > > Exception encountered while connecting to the server : > > > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > > *2017-06-13 12:34:11,203 ERROR [main] > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while > > registering* > > *org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > > AMRMToken from appattempt_1497050650910_0004_000002* > > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method)* > > * at > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:62)* > > * at > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45)* > > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)* > > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException( > RPCUtil.java:53)* > > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException( > RPCUtil.java:80)* > > * at > > > > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException( > RPCUtil.java:119)* > > * at > > > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli > entImpl.java:109)* > > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* > > * at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62)* > > * at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)* > > * at java.lang.reflect.Method.invoke(Method.java:498)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:398)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod( > RetryInvocationHandler.java:163)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call. > invoke(RetryInvocationHandler.java:155)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce( > RetryInvocationHandler.java:95)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:335)* > > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator. > register(RMCommunicator.java:161)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator. > serviceStart(RMCommunicator.java:121)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart( > RMContainerAllocator.java:280)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter. > serviceStart(MRAppMaster.java:966)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.service.CompositeService.serviceStart( > CompositeService.java:120)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster. > serviceStart(MRAppMaster.java:1263)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run( > MRAppMaster.java:1716)* > > * at java.security.AccessController.doPrivileged(Native Method)* > > * at javax.security.auth.Subject.doAs(Subject.java:422)* > > * at > > > > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1857)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( > MRAppMaster.java:1712)* > > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main( > MRAppMaster.java:1643)* > > *Caused by: > > > > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token. > SecretManager$InvalidToken): > > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > > * at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)* > > * at org.apache.hadoop.ipc.Client.call(Client.java:1427)* > > * at org.apache.hadoop.ipc.Client.call(Client.java:1337)* > > * at > > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:227)* > > * at > > > > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker. > invoke(ProtobufRpcEngine.java:1)* > > * at com.sun.proxy.$Proxy85.registerApplicationMaster(Unknown Source)* > > * at > > > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli > entImpl.java:107)* > > * ... 25 more* > > *2017-06-13 12:34:11,207 INFO [main] > > org.apache.hadoop.service.AbstractService: Service RMCommunicator > failed in > > state STARTED; cause: > > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > > AMRMToken from appattempt_1497050650910_0004_000002* > > *org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > > org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid > > AMRMToken from appattempt_1497050650910_0004_000002* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator. > register(RMCommunicator.java:177)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator. > serviceStart(RMCommunicator.java:121)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart( > RMContainerAllocator.java:280)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter. > serviceStart(MRAppMaster.java:966)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.service.CompositeService.serviceStart( > CompositeService.java:120)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster. > serviceStart(MRAppMaster.java:1263)* > > * at > > org.apache.hadoop.service.AbstractService.start( > AbstractService.java:193)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run( > MRAppMaster.java:1716)* > > * at java.security.AccessController.doPrivileged(Native Method)* > > * at javax.security.auth.Subject.doAs(Subject.java:422)* > > * at > > > > org.apache.hadoop.security.UserGroupInformation.doAs( > UserGroupInformation.java:1857)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster( > MRAppMaster.java:1712)* > > * at > > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main( > MRAppMaster.java:1643)* > > *Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: > > Invalid AMRMToken from appattempt_1497050650910_0004_000002* > > * at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method)* > > * at > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance( > NativeConstructorAccessorImpl.java:62)* > > * at > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45)* > > * at java.lang.reflect.Constructor.newInstance(Constructor.java:423)* > > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException( > RPCUtil.java:53)* > > * at > > org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException( > RPCUtil.java:80)* > > * at > > > > org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException( > RPCUtil.java:119)* > > * at > > > > org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBCli > entImpl.registerApplicationMaster(ApplicationMasterProtocolPBCli > entImpl.java:109)* > > * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* > > * at > > > > sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62)* > > * at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43)* > > * at java.lang.reflect.Method.invoke(Method.java:498)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod( > RetryInvocationHandler.java:398)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod( > RetryInvocationHandler.java:163)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call. > invoke(RetryInvocationHandler.java:155)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce( > RetryInvocationHandler.java:95)* > > * at > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke( > RetryInvocationHandler.java:335)* > > * at com.sun.proxy.$Proxy86.registerApplicationMaster(Unknown Source)* > > * at > > > > org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator. > register(RMCommunicator.java:161)* > > * ... 14 more* > > > > > > > > Is there any configuration missing that I have a AMRMToken erros? > > > > I appreciate your help! > > > > Thanks! > > > > > > Wei Chen > > > -- > Konstantinos >
