As per the error, spark user does not have permission to create directory under NodeManager Local Directory or the existing spark user directory is with stale uid or gid.
*Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018* 1. Check if spark user is able to create directory under NM local dir. 2. Remove /d1/hadoop/yarn/local/usercache/spark from all NMs and rerun the job. On Thu, Oct 10, 2019 at 12:13 PM Manuel Sopena Ballesteros < manuel...@garvan.org.au> wrote: > Dear Hadoop community, > > > > I am trying to configure yarn container executor following this document > https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html > > > > I follow all the steps but after restart YARN I can’t start spark thrift > server. > > > > This is the error I can see in yarn > > > > Application application_1570681803028_0018 failed 1 times (global limit > =2; local limit is =1) due to AM Container for > appattempt_1570681803028_0018_000001 exited with exitCode: -1000 Failing > this attempt.Diagnostics: [2019-10-10 16:49:35.322]Application > application_1570681803028_0018 initialization failed (exitCode=255) with > output: main : command provided 0 main : run as user is spark main : > requested yarn user is spark Can't create directory > /d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 > - Permission denied Can't create directory > /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 > - Permission denied Did not create any app directories For more detailed > output, check the application tracking page: > http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018 > Then click on links to logs of each attempt. . Failing the application. > > > > > > This is the content of container-executor.cfg > > > > [luffy@gl-hdp-ctrl01-mlx ~]$ cat > /etc/hadoop/3.1.0.0-78/0/container-executor.cfg > > > > > > #/* > > # * Licensed to the Apache Software Foundation (ASF) under one > > # * or more contributor license agreements. See the NOTICE file > > # * distributed with this work for additional information > > # * regarding copyright ownership. The ASF licenses this file > > # * to you under the Apache License, Version 2.0 (the > > # * "License"); you may not use this file except in compliance > > # * with the License. You may obtain a copy of the License at > > # * > > # * http://www.apache.org/licenses/LICENSE-2.0 > > # * > > # * Unless required by applicable law or agreed to in writing, software > > # * distributed under the License is distributed on an "AS IS" BASIS, > > # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > implied. > > # * See the License for the specific language governing permissions and > > # * limitations under the License. > > # */ > > yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local > > yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log > > yarn.nodemanager.linux-container-executor.group=hadoop > > banned.users=hdfs,yarn,mapred,bin > > # min.user.id=1000 > > min.user.id=80 > > > > [docker] > > module.enabled=false > > docker.binary=/usr/bin/docker > > > docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE > > docker.allowed.devices= > > docker.allowed.networks=host,none,bridge > > docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local, > > > docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log, > > docker.privileged-containers.enabled=false > > docker.trusted.registries= > > docker.allowed.volume-drivers= > > > > [gpu] > > module.enabled=false > > > > [cgroups] > > root= > > yarn-hierarchy= > > > > I was hopping if someone could help me troubleshooting about what YARN is > trying to do and how to fix this configuration issue? > > > > Thank you very much > > > > Manuel > NOTICE > Please consider the environment before printing this email. This message > and any attachments are intended for the addressee named and may contain > legally privileged/confidential/copyright information. If you are not the > intended recipient, you should not read, use, disclose, copy or distribute > this communication. If you have received this message in error please > notify us at once by return email and then delete both messages. We accept > no liability for the distribution of viruses or similar in electronic > communications. This notice should not be removed. >
