[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Resolution: Fixed Hadoop Flags: Reviewed Target Version/s: 3.0.0 Status: Resolved (was: Patch Available) > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch, HIVE-18264.8.patch, HIVE-18264.8.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Attachment: HIVE-18264.8.patch > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch, HIVE-18264.8.patch, HIVE-18264.8.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Attachment: HIVE-18264.8.patch > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch, HIVE-18264.8.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Attachment: HIVE-18264.7.patch > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, > HIVE-18264.6.patch, HIVE-18264.7.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Attachment: HIVE-18264.6.patch > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, HIVE-18264.6.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Issue Type: Sub-task (was: Bug) Parent: HIVE-16520 > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Description: Currently we have a separate cache for partitions and partition col stats which results in some calls iterating through each of these for retrieving/updating. For example, to modify a partition col stat, currently we need to lock table, partition and partition col stats caches which are all separate hashmaps. We can get better performance by organizing hierarchically. For example, we can have a partition, partition col stats and table col stats cache per table to improve on the previous mechanisms. This will also result in better concurrency, since now instead of locking the whole cache, we can selectively lock the table cache and modify multiple tables in parallel. In addition, currently, the prewarm mechanism populates all the caches initially (it skips tables that do not pass whitelist/blacklist filter) and it is a blocking call. This patch also makes prewarm non-blocking so that the calls for tables that are already cached can be served from the memory and the ones that are not can be served from the rdbms. was:Currently we have a separate cache for partitions and partition col stats which results in some calls iterating through each of these for retrieving/updating. We can get better performance by organizing hierarchically. We should also make prewarm non-blocking > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. For example, to modify a partition col stat, currently > we need to lock table, partition and partition col stats caches which are all > separate hashmaps. We can get better performance by organizing > hierarchically. For example, we can have a partition, partition col stats and > table col stats cache per table to improve on the previous mechanisms. This > will also result in better concurrency, since now instead of locking the > whole cache, we can selectively lock the table cache and modify multiple > tables in parallel. > In addition, currently, the prewarm mechanism populates all the caches > initially (it skips tables that do not pass whitelist/blacklist filter) and > it is a blocking call. This patch also makes prewarm non-blocking so that the > calls for tables that are already cached can be served from the memory and > the ones that are not can be served from the rdbms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Description: Currently we have a separate cache for partitions and partition col stats which results in some calls iterating through each of these for retrieving/updating. We can get better performance by organizing hierarchically. We should also make prewarm non-blocking (was: Currently we have a separate cache for partitions and partition col stats which results in some calls iterating through each of these for retrieving/updating. We can get better performance by organizing hierarchically. ) > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. We can get better performance by organizing > hierarchically. We should also make prewarm non-blocking -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking
[ https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18264: Summary: CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking (was: CachedStore: Store cached partitions/col stats within the table cache ) > CachedStore: Store cached partitions/col stats within the table cache and > make prewarm non-blocking > --- > > Key: HIVE-18264 > URL: https://issues.apache.org/jira/browse/HIVE-18264 > Project: Hive > Issue Type: Bug >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, > HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch > > > Currently we have a separate cache for partitions and partition col stats > which results in some calls iterating through each of these for > retrieving/updating. We can get better performance by organizing > hierarchically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
