You can try snakebite https://github.com/spotify/snakebite.
$ snakebite ls -R path
I just run it to list 705K files and it went fine.
2014-05-30 20:42 GMT+02:00 Harsh J ha...@cloudera.com:
The HADOOP_OPTS gets overriden by HADOOP_CLIENT_OPTS for FsShell
utilities. The right way to extend
Hi,
do you have an idea on how to look at the content of a 530k-files HDFS folder?
(yes, I know it is a bad idea to have such setup, but that’s the status and I’d
like to debug it)
and the only tool that doesn’t go out of memory is hdfs dfs -count folder/“
-ls goes out of memory, -count with the
Hi Guido,
You can set client side heap in HADOOP_OPTS variable before running the ls
command.
export HADOOP_OPTS=-Xmx3g; hadoop fs -ls /
- Bharath
On Fri, May 30, 2014 at 5:22 PM, Guido Serra z...@fsfe.org wrote:
Hi,
do you have an idea on how to look at the content of a 530k-files HDFS
already tried, didn't work (24cores at 100% and a-lot-memory, stilll ...
GC overhead limit exceed)
thanks anyhow
On 05/30/2014 02:43 PM, bharath vissapragada wrote:
Hi Guido,
You can set client side heap in HADOOP_OPTS variable before running
the ls command.
export HADOOP_OPTS=-Xmx3g;
Listing such a directory should not be a big problem. Can you cut and paste the
command output.
Which release are you using?
Sent from phone
On May 30, 2014, at 5:49 AM, Guido Serra z...@fsfe.org wrote:
already tried, didn't work (24cores at 100% and a-lot-memory, stilll ... GC
overhead
guido@hd11 ~ $ export HADOOP_OPTS=-Xmx3g;hdfs dfs -ls /logs/2014-05-28/
14/05/30 13:05:44 WARN retry.RetryInvocationHandler: Exception while
invoking getListing of class ClientNamenodeProtocolTranslatorPB. Trying
to fail over immediately.
14/05/30 13:05:45 WARN retry.RetryInvocationHandler:
forgot to mention… it is CDH 4.6.0
On 30 May 2014, at 15:08, Guido Serra z...@fsfe.org wrote:
guido@hd11 ~ $ export HADOOP_OPTS=-Xmx3g;hdfs dfs -ls /logs/2014-05-28/
14/05/30 13:05:44 WARN
The HADOOP_OPTS gets overriden by HADOOP_CLIENT_OPTS for FsShell
utilities. The right way to extend is to use HADOOP_CLIENT_OPTS
instead, for FsShell and other client applications such as hadoop
fs/hdfs dfs/hadoop jar, etc..
On Fri, May 30, 2014 at 6:13 PM, bharath vissapragada
