Re: How do i find volume failure using java code???

2014-10-19 Thread Mahesh Kumar Vasanthu Somashekar
Hi Cho Ju Il, Some of the objects that you are trying to access, like DatanodeDescriptor, are only internal to namenode, not for clients. Client always get a fresh instance, not the one which namenode updates as part of heartbeat, and that's why getVolumeFailures returns zero.

Re: Spark vs Tez

2014-10-19 Thread Niels Basjes
Very interesting! What makes Tez more scalable than Spark? What architectural thing makes the difference? Niels Basjes On Oct 19, 2014 3:07 AM, Jeff Zhang zjf...@gmail.com wrote: Tez has a feature called pre-warm which will launch JVM before you use it and you can reuse the container

Re: Spark vs Tez

2014-10-19 Thread Mohan Radhakrishnan
Is Tez's architecture similar to Akka's distributed architecture ? I think I remember that Jonas boner mentioned during a presentation on distributed computing about Akka's support for protocols like raft etc. What makes Tez more scalable in this regard ? Thanks, Mohan On Sun, Oct 19, 2014 at

Can add a regular check in DataNode on free disk space?

2014-10-19 Thread sam liu
Hi Experts and Developers, At present, if a DataNode does not has free disk space, we can not get this bad situation from anywhere, including DataNode log. At the same time, under this situation, the hdfs writing operation will fail and return error msg as below. However, from the error msg, user

reg pig issue

2014-10-19 Thread mallik arjun
HI hadoopers, i have the following issue with pig raw_log_entries = LOAD '/user/u377058/rc_app_error' USING TextLoader AS (line:chararray); logs_base = FOREACH raw_log_entries GENERATE FLATTEN( REGEX_EXTRACT_ALL(line,

Re: Can add a regular check in DataNode on free disk space?

2014-10-19 Thread unmesha sreeveni
1. Stop all Hadoop daemons 2. Remove all files from /var/lib/hadoop-hdfs/cache/hdfs/dfs/name 3. Format namenode 4. Start all Hadoop daemons. On Mon, Oct 20, 2014 at 8:26 AM, sam liu samliuhad...@gmail.com wrote: Hi Experts and Developers, At present, if a