[
https://issues.apache.org/jira/browse/PIG-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880071#comment-13880071
]
Koji Noguchi commented on PIG-2672:
-----------------------------------
bq. The cache will be reused by other pig jobs during and after the week and we
will not be modifying the time of the files. So we can't put that under .Trash
as it will be cleaned up.
Maybe I didn't explain it well enough.
Tasks will always use the cache that is scheduled for deletion next next
Monday.
Given that the longest job (on our cluster) is 1 week, there won't be any jobs
using that cache when it's expunged.
> Optimize the use of DistributedCache
> ------------------------------------
>
> Key: PIG-2672
> URL: https://issues.apache.org/jira/browse/PIG-2672
> Project: Pig
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Fix For: 0.13.0
>
> Attachments: PIG-2672-5.patch, PIG-2672.patch
>
>
> Pig currently copies jar files to a temporary location in hdfs and then adds
> them to DistributedCache for each job launched. This is inefficient in terms
> of
> * Space - The jars are distributed to task trackers for every job taking
> up lot of local temporary space in tasktrackers.
> * Performance - The jar distribution impacts the job launch time.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
