[jira] [Commented] (MAPREDUCE-1729) Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751722#comment-13751722 ] Akira AJISAKA commented on MAPREDUCE-1729: -- [~yamashitasni], thanks for your comment. I'll try to implement your 2nd option. Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly. -- Key: MAPREDUCE-1729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1729 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distributed-cache Reporter: Amareshwari Sriramadasu Assignee: Akira AJISAKA Currently, distributed cache fails the job if the cache file gets modified on the fly. But there should be an option to fail a job or not. See discussions in MAPREDUCE-1288. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1729) Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750707#comment-13750707 ] Shinichi Yamashita commented on MAPREDUCE-1729: --- Hi, When a target file is updated during MapReduce Job, the result will change. Because it may cause inconsistencies in the result of MapReduce Job, the architecture of the current DistributedCache is valid. But it may set the following options, 1. Addition of a property the update of the target file in MapReduce Job 2. It copies a target file before MapReduce Job start in another directory and use that file By any policy, this know-how should document it as an important usage of DistributedCache. Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly. -- Key: MAPREDUCE-1729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1729 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distributed-cache Reporter: Amareshwari Sriramadasu Assignee: Akira AJISAKA Currently, distributed cache fails the job if the cache file gets modified on the fly. But there should be an option to fail a job or not. See discussions in MAPREDUCE-1288. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1729) Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13708801#comment-13708801 ] Akira AJISAKA commented on MAPREDUCE-1729: -- Hi, I need this option. A long-term job failed in our production environment because the file which was used as distributed cache was modified at fixed intervals. As the output of the job would be better if distributed cache is newer, we don't want to fail the job if cache file gets modified on the fly. Our workaround is, copy original file to tmpfile and use tmpfile as distributed cache. If the option exists, we don't need to copy original file before the job begin. Distributed cache should provide an option to fail the job or not, if cache file gets modified on the fly. -- Key: MAPREDUCE-1729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1729 Project: Hadoop Map/Reduce Issue Type: Bug Components: distributed-cache Reporter: Amareshwari Sriramadasu Currently, distributed cache fails the job if the cache file gets modified on the fly. But there should be an option to fail a job or not. See discussions in MAPREDUCE-1288. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
