Thank you very much Prabhu,
Deleting the /d0 folder fixed the issue
Manuel
From: Prabhu Josephraj [mailto:pjos...@cloudera.com]
Sent: Thursday, October 10, 2019 6:17 PM
To: Manuel Sopena Ballesteros
Cc: user@hadoop.apache.org
Subject: Re: can't start spark thrift after Configuring YARN container executor
As per the error, spark user does not have permission to create directory under
NodeManager Local Directory or the existing spark user directory is with stale
uid or gid.
Permission denied Can't create directory
/d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018
1. Check if spark user is able to create directory under NM local dir.
2. Remove /d1/hadoop/yarn/local/usercache/spark from all NMs and rerun the job.
On Thu, Oct 10, 2019 at 12:13 PM Manuel Sopena Ballesteros
mailto:manuel...@garvan.org.au>> wrote:
Dear Hadoop community,
I am trying to configure yarn container executor following this document
https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html
I follow all the steps but after restart YARN I can’t start spark thrift server.
This is the error I can see in yarn
Application application_1570681803028_0018 failed 1 times (global limit =2;
local limit is =1) due to AM Container for appattempt_1570681803028_0018_01
exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-10-10
16:49:35.322]Application application_1570681803028_0018 initialization failed
(exitCode=255) with output: main : command provided 0 main : run as user is
spark main : requested yarn user is spark Can't create directory
/d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 -
Permission denied Can't create directory
/d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 -
Permission denied Did not create any app directories For more detailed output,
check the application tracking page:
http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018
Then click on links to logs of each attempt. . Failing the application.
This is the content of container-executor.cfg
[luffy@gl-hdp-ctrl01-mlx ~]$ cat /etc/hadoop/3.1.0.0-78/0/container-executor.cfg
#/*
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements. See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership. The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License. You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local
yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=hdfs,yarn,mapred,bin
# min.user.id<http://min.user.id>=1000
min.user.id<http://min.user.id>=80
[docker]
module.enabled=false
docker.binary=/usr/bin/docker
docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
docker.allowed.devices=
docker.allowed.networks=host,none,bridge
docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,
docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log,
docker.privileged-containers.enabled=false
docker.trusted.registries=
docker.allowed.volume-drivers=
[gpu]
module.enabled=false
[cgroups]
root=
yarn-hierarchy=
I was hopping if someone could help me troubleshooting about what YARN is
trying to do and how to fix this configuration issue?
Thank you very much
Manuel
NOTICE
Please consider the environment before printing this email. This message and
any attachments are intended for the addressee named and may contain legally
privileged/confidential/copyright information. If you are not the intended
recipient, you should not read, use, disclose, copy or distribute this
communication. If you have received this message in error please notify us at
once by return email and then delete both messages. We accept no liability for
the distribution of viruses or similar in electronic communications. This
notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and
any attachments are intended for the addressee named and may contain lega