To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
The dirs in that env-var are app-specific and are for the app's user to
utilize. You shouldn't have any permission issues working within them.
The LocalDirAllocator is still somewhat MR-bound but you can still
John
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative
./) thats auto
11:58 PM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Hi,
MR does use multiple disks when spilling. But the work directory is also
round-robined to spread I/O.
YARN sets an environment property thats a list (comma separated value) of
directories
John
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the
relative
Thanks, sounds like LOCAL_DIR_ENV is the way to go.
john
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Monday, October 21, 2013 12:11 PM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
The dirs in that env-var are app-specific
@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
This post might help a bit.
http://hortonworks.com/blog/management-of-application-dependencies-in-yarn/
Thanks,
Jian
On Mon, Oct 21, 2013 at 11:11 AM, Harsh J
ha...@cloudera.commailto:ha...@cloudera.com wrote:
The dirs
We have a pure YARN application (no MapReduce) that has need to store a
significant amount of temporary data. How can we know the best location for
these files? How can we ensure that our YARN tasks have write access to these
locations? Is this something that must be configured outside of
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative ./)
thats auto-cleaned up at the end of the container's life.
This is the best place to store the temporary files. This is not something you
need custom
-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: Sunday, October 20, 2013 10:49 AM
To: user@hadoop.apache.org
Subject: Re: temporary file locations for YARN applications
Every container gets its own local work directory (You can use the relative
./) thats auto-cleaned up at the end
