[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.1.patch rebase code to see the cases running result > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.14.0 > > Attachments: HIVE-860.1.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-860: --- Fix Version/s: (was: 0.13.0) 0.14.0 > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.14.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.14.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.14.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch Running this one again. Some of the failures are quite strange. > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.14.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch I tested manually on a cluster but I am sure tests will be impacted by this change. Let's see. https://reviews.apache.org/r/18200/ > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Fix Version/s: 0.13.0 Affects Version/s: 0.12.0 Status: Patch Available (was: Open) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch Minor update to the pom so the build can bootstrap. > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Status: Open (was: Patch Available) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Status: Patch Available (was: Open) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch Addresses review. > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch This should fix it. The problem is that when ExecDriver is run as main() the instance variable "conf" is null and has to be ignored as opposed to "job". > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Brock Noland > Fix For: 0.13.0 > > Attachments: HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Fix Version/s: (was: 0.14.0) 0.15.0 > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.2.patch trigger a full test in CI > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: (was: HIVE-860.2.patch) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.2.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.2.patch Reattach the patch to see the CI result. Last time the CI failed to start due to the following error: rsync: write failed on "/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-1868/succeeded/TestCliDriver-avro_add_column.q-orc_wide_table.q-query_with_semi.q-and-12-more/hive.log": No space left on device (28) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.3.patch The patch includes the changes: 1. add some needed classes into the HiveAuxClasspathBuilder 2. rebase code since Hadoop20Shim is removed > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.3.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.4.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.3.patch, HIVE-860.4.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.4.patch reattach since the error "No space left on device (28)" > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Ferdinand Xu > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.3.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Assignee: Dong Chen (was: Ferdinand Xu) > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.3.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-860: --- Attachment: HIVE-860.4.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860.1.patch, HIVE-860.2.patch, HIVE-860.2.patch, > HIVE-860.3.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-860: --- Attachment: HIVE-860-debug.4.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860-debug.4.patch, HIVE-860.1.patch, > HIVE-860.2.patch, HIVE-860.2.patch, HIVE-860.3.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-860: -- Attachment: HIVE-860.4.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860-debug.4.patch, HIVE-860.1.patch, > HIVE-860.2.patch, HIVE-860.2.patch, HIVE-860.3.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-860: --- Attachment: HIVE-860.4.patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860-debug.4.patch, HIVE-860.1.patch, > HIVE-860.2.patch, HIVE-860.2.patch, HIVE-860.3.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.4.patch Running tests again so I can look at logs. > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860-debug.4.patch, HIVE-860.1.patch, > HIVE-860.2.patch, HIVE-860.2.patch, HIVE-860.3.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-860) Persistent distributed cache
[ https://issues.apache.org/jira/browse/HIVE-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-860: -- Attachment: HIVE-860.5.patch Rebased [~dongc]'s patch > Persistent distributed cache > > > Key: HIVE-860 > URL: https://issues.apache.org/jira/browse/HIVE-860 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.12.0 >Reporter: Zheng Shao >Assignee: Dong Chen > Fix For: 0.15.0 > > Attachments: HIVE-860-debug.4.patch, HIVE-860.1.patch, > HIVE-860.2.patch, HIVE-860.2.patch, HIVE-860.3.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, HIVE-860.4.patch, > HIVE-860.4.patch, HIVE-860.5.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, > HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch, HIVE-860.patch > > > DistributedCache is shared across multiple jobs, if the hdfs file name is the > same. > We need to make sure Hive put the same file into the same location every time > and do not overwrite if the file content is the same. > We can achieve 2 different results: > A1. Files added with the same name, timestamp, and md5 in the same session > will have a single copy in distributed cache. > A2. Filed added with the same name, timestamp, and md5 will have a single > copy in distributed cache. > A2 has a bigger benefit in sharing but may raise a question on when Hive > should clean it up in hdfs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
