Hi,
Unfortunately, after I set my user¹s ulimit n to 65536, I still get the
same bad performance, killed containers and errors as before.
I collected together a bunch of logs around the moment when the containers
are being killed (application master log, killed container log, hadoop-hdfs
logs,
Hi everyone,
I am having an issue with MapReduce jobs running through Hive being killed
after 600s timeouts and with very simple jobs taking over 3 hours (or just
failing) for a set of files with a compressed size of only 1-2gb. I will try
and provide as much information as I can here, so if
file
/tmp/hive-zslf023/hive_2014-08-02_12-33-59_857_6455822541748133957/_task_tmp.-ext-10001/_tmp.00_0
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/tmp/hive-zslf023/hive_2014-08-02_12-33-59_857_6455822541748133957/_task_tmp
For my own user? It is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 483941
max locked memory (kbytes, -l) 64
Filemax across the cluster is set to over 6 million. I¹ve checked the open
file limits for the accounts used by the Hadoop daemons and they have an
open file limit of 32K. This is confirmed by the various .out files, e.g.
/var/log/hadoop-hdfs/hadoop-hdfs-datanode-slave1.out
Contains open files
:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException)
For my own user? It is as follows:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending
I¹m not sure which user is fetching the data, but I¹m assuming no one
changed that from the default. The data isn¹t huge in size, just in number,
so I suppose the open files limit is not the issue?
I¹m running the job again with mapred.task.timeout=120, but containers
are still being killed
Exception
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
No lease on
/tmp/hive-usnm123/hive_2014-08-02_17-41-52_914_251548734850890001/_task_tmp.-ext-10001/_tmp.06_0:
File does not exist. Holder
Ok, I will request this to be done, as I¹m not an admin, and then get back
to this thread on Monday. Thank you!
From: hadoop hive hadooph...@gmail.com
Reply-To: user@hadoop.apache.org
Date: Saturday, 2 August 2014 18:50
To: user@hadoop.apache.org
Subject: Re:
