[
https://issues.apache.org/jira/browse/PIG-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776988#comment-13776988
]
Rohini Palaniswamy commented on PIG-2672:
-----------------------------------------
[~aniket486],
bq. Currently jars get copied to /tmp/temp-<random>/ which can be written by
all users
No. They do not. They go into /user/<username>/.staging which is readable
and writable only by that user. Even if it were to go to /tmp/temp- (where the
intermediate files now go), we have the dfs.umaskmode set to 077 so only the
user has rwx and no one else has.
It is good to have shared cluster location, but if someone accidentally
deletes that directory then all user jobs already launched will fail. It would
be good if you can add the check to see if the cachedir is writable before
trying to create it there. People with a multi-tenant environment like us can
then chose to place frequently used jars in the shared cluster location but
protect it with 755 so that others don't write into it.
> Optimize the use of DistributedCache
> ------------------------------------
>
> Key: PIG-2672
> URL: https://issues.apache.org/jira/browse/PIG-2672
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Aniket Mokashi
> Fix For: 0.12.0
>
> Attachments: PIG-2672.patch
>
>
> Pig currently copies jar files to a temporary location in hdfs and then adds
> them to DistributedCache for each job launched. This is inefficient in terms
> of
> * Space - The jars are distributed to task trackers for every job taking
> up lot of local temporary space in tasktrackers.
> * Performance - The jar distribution impacts the job launch time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
