[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=638454&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-638454 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 17/Aug/21 00:13 Start Date: 17/Aug/21 00:13 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-899899028 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 638454) Time Spent: 4h 20m (was: 4h 10m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 4h 20m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=641415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-641415 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 25/Aug/21 00:12 Start Date: 25/Aug/21 00:12 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #2128: URL: https://github.com/apache/hive/pull/2128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 641415) Time Spent: 4.5h (was: 4h 20m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 4.5h > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=578252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578252
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 07/Apr/21 09:56
Start Date: 07/Apr/21 09:56
Worklog Time Spent: 10m
Work Description: adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608511275
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isValid = false;
+
+if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
nit: add braces.
Also, should we return true? if validWriteId is not present, we can safely
assume that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 578252)
Time Spent: 20m (was: 10m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=578477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578477
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 07/Apr/21 17:06
Start Date: 07/Apr/21 17:06
Worklog Time Spent: 10m
Work Description: adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608511275
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isValid = false;
+
+if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
nit: add braces.
Also, should we return true? if validWriteId is not present, we can assume
that table is not loaded properly and it should be fetched from metastore db.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 578477)
Time Spent: 0.5h (was: 20m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=578515&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578515
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 07/Apr/21 17:56
Start Date: 07/Apr/21 17:56
Worklog Time Spent: 10m
Work Description: adesh-rao commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r608870539
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isValid = false;
Review comment:
nit: rename the variable to `isStale` instead? Below we are setting
isValid to true when the the writeIdList in CachedStore is stale/older than
client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 578515)
Time Spent: 40m (was: 0.5h)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=584128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584128
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 08:07
Start Date: 16/Apr/21 08:07
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r614644293
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isValid = false;
+
+if (StringUtils.isEmpty(validWriteIdList))
+ return false;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584128)
Time Spent: 50m (was: 40m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=584129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584129
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 16/Apr/21 08:07
Start Date: 16/Apr/21 08:07
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r614644380
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2499,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isValid = false;
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 584129)
Time Spent: 1h (was: 50m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=584130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-584130 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 16/Apr/21 08:08 Start Date: 16/Apr/21 08:08 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-821000521 @kishendas Can you please review this PR mean while I will add UT in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 584130) Time Spent: 1h 10m (was: 1h) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=595061&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-595061 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 12/May/21 04:18 Start Date: 12/May/21 04:18 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-839420970 @kishendas Could you please review this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 595061) Time Spent: 1h 20m (was: 1h 10m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=597399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597399 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 16/May/21 16:06 Start Date: 16/May/21 16:06 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma edited a comment on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660 @kgyrtkirk Could you please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597399) Time Spent: 1h 40m (was: 1.5h) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=597852&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597852 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 17/May/21 18:03 Start Date: 17/May/21 18:03 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660 @kgyrtkirk Could please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 597852) Time Spent: 1h 50m (was: 1h 40m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=598149&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-598149 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 17/May/21 18:38 Start Date: 17/May/21 18:38 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma edited a comment on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-841813660 @kgyrtkirk Could you please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 598149) Time Spent: 2h (was: 1h 50m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=600304&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-600304
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 21/May/21 11:43
Start Date: 21/May/21 11:43
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r636821684
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
A curious question.
I think, we recently added writeId allocation logic for DDL operations. Do
we rollback the changes happened in metastore if DDL op fails?
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Probably, getting the snapshot before fetching the metadata objects
might help to handle some cases.
1. Take snapshot from current thread and cache it ->
ValidWriteIdList=:HWM=10:
2. Fetch table object --> writeid=10
2. Another thread writes table object --> writeid=11
3. In this case, new snapshot would be ValidWriteIdList=:HWM=11: which
doesn't match with cached snapshot
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName,
String tblName, String vali
catName = normalizeIdentifier(catName);
Review comment:
@Override
public Table getTable(String catName, String dbName, String tblName,
String validWriteIds) throws MetaException {
return getTable(catName, dbName, tblName, null, -1);
}
Here, validWriteIds is not passed down.
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
+new
GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName,
tblName
+.getTblValidWriteIds().get(0));
+Deadline.stopTimer();
+ } catch (Exception e) {
+LOG.info("Updating CachedStore: unable to update table
validWriteIdList of catalog: " + catName + ", database: "
++ dbName + ", table: " + tblName, e);
+ }
+ if (validWriteIdList != null) {
+sharedCache.refreshValidWriteIdListInCache(catName, dbName, tblName,
validWriteIdList);
+
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603881
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 08:17
Start Date: 30/May/21 08:17
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642039659
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName,
String tblName, String vali
catName = normalizeIdentifier(catName);
dbName = StringUtils.normalizeIdentifier(dbName);
tblName = StringUtils.normalizeIdentifier(tblName);
-if (!shouldCacheTable(catName, dbName, tblName) || (canUseEvents &&
rawStore.isActiveTransaction())) {
+if (shouldGetTableFromRawStore(catName, dbName, tblName, validWriteIds)) {
Review comment:
I will do that in new PR. Where I will put tableId and writeId check on
all the methods in cacheStore. This PR is mostly around adding validWriteIdList
to TableWrapper Object in cacheStore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603881)
Time Spent: 2h 20m (was: 2h 10m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603882&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603882 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 30/May/21 08:18 Start Date: 30/May/21 08:18 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on a change in pull request #2128: URL: https://github.com/apache/hive/pull/2128#discussion_r642039720 ## File path: standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java ## @@ -38,10 +38,11 @@ import org.apache.hadoop.hive.metastore.Warehouse; import org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest; import org.apache.hadoop.hive.metastore.api.*; -import org.apache.hadoop.hive.metastore.api.utils.DecimalUtils; import org.apache.hadoop.hive.metastore.client.builder.DatabaseBuilder; import org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector; import org.apache.hadoop.hive.metastore.conf.MetastoreConf; +import org.apache.hadoop.hive.metastore.txn.TxnStore; Review comment: I will do that in new PR. Where I will put tableId and writeId check on all the methods and there corresponding tests in cacheStore. This PR is mostly around adding validWriteIdList to TableWrapper Object in cacheStore. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 603882) Time Spent: 2.5h (was: 2h 20m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603884&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603884
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 08:27
Start Date: 30/May/21 08:27
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642040720
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2498,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isStale = true;
+
+if (StringUtils.isEmpty(validWriteIdList))
+ return isStale;
Review comment:
There is background updater thread which is running in certain interval.
Which update all table object. We are targeting eventual consistency for cached
store.
https://cwiki.apache.org/confluence/display/Hive/Synchronized+Metastore+Cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603884)
Time Spent: 2h 40m (was: 2.5h)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603885&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603885
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 08:29
Start Date: 30/May/21 08:29
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642041015
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -573,6 +577,18 @@ static void prewarm(RawStore rawStore) {
Deadline.stopTimer();
cacheObjects.setTableConstraints(tableConstraints);
+ try {
+Deadline.startTimer("getValidWriteIds");
+ValidWriteIdList validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore
+.getValidWriteIds(
+new
GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName,
tblName
+.getTblValidWriteIds().get(0));
+Deadline.stopTimer();
+cacheObjects.setValidWriteIdList(validWriteIdList);
+ } catch (Exception e) {
+LOG.debug(ExceptionUtils.getStackTrace(e));
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603885)
Time Spent: 2h 50m (was: 2h 40m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603886&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603886 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 30/May/21 08:29 Start Date: 30/May/21 08:29 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on a change in pull request #2128: URL: https://github.com/apache/hive/pull/2128#discussion_r642041067 ## File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java ## @@ -1266,7 +1309,7 @@ public Table getTable(String catName, String dbName, String tblName, String vali catName = normalizeIdentifier(catName); Review comment: Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 603886) Time Spent: 3h (was: 2h 50m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603887&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603887
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 08:35
Start Date: 30/May/21 08:35
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642041716
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
+new
GetValidWriteIdsRequest(Collections.singletonList(TableName.getDbTable(dbName,
tblName
+.getTblValidWriteIds().get(0));
+Deadline.stopTimer();
+ } catch (Exception e) {
+LOG.info("Updating CachedStore: unable to update table
validWriteIdList of catalog: " + catName + ", database: "
++ dbName + ", table: " + tblName, e);
+ }
+ if (validWriteIdList != null) {
+sharedCache.refreshValidWriteIdListInCache(catName, dbName, tblName,
validWriteIdList);
+LOG.debug("CachedStore: updated cached table validWriteIdList for
catalog: {}, database: {}, table: {}",
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603887)
Time Spent: 3h 10m (was: 3h)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603889 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 30/May/21 08:45 Start Date: 30/May/21 08:45 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #2128: URL: https://github.com/apache/hive/pull/2128#issuecomment-850964411 @sankarh This PR is about adding validWriteIdList as part of cached store. Current status 1. HIVE-23931 - Send ValidWriteIdList and tableId to get_*_constraints HMS APIs (open) 2. HIVE-23820: [HS2] Send tableId in request for get_table_request API (merged) 3. HIVE-23571: Add ValidWriteIdList to SharedCache.TableWrapper (open) 4. HIVE-23821: Send tableId in request for all the new HMS get_partition APIs (merged) Once above PR get merged. I will raise a PR to add tableId and writedId check to all the methods and there corresponding test cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 603889) Time Spent: 3h 20m (was: 3h 10m) > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603903&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603903
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 11:18
Start Date: 30/May/21 11:18
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642060638
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Since we are target eventual consistency it is fine we fall-back to raw
store. But we can target aggressive refresh rate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603903)
Time Spent: 3.5h (was: 3h 20m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=603904&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-603904
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 30/May/21 11:20
Start Date: 30/May/21 11:20
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r642060827
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
Need to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 603904)
Time Spent: 3h 40m (was: 3.5h)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=609246&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609246
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 09/Jun/21 16:52
Start Date: 09/Jun/21 16:52
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648497223
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
AFAIK, we are targeting strict consistency with "Synchronized metastore
cache" work. Eventual consistency is supported even today with regular refresh
of cache with the interval of 1 min. Pls recheck this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609246)
Time Spent: 3h 50m (was: 3h 40m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=609247&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609247
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 09/Jun/21 16:54
Start Date: 09/Jun/21 16:54
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648497223
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -918,6 +938,29 @@ private void updateAllTableConstraints(RawStore rawStore,
String catName, String
}
}
+private void updateValidWriteIdList(TxnStore txnStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
+ LOG.debug("CachedStore: updating cached table validWriteIdlist for
catalog: {}, database: {}, table: {}", catName,
+ dbName, tblName);
+ ValidWriteIdList validWriteIdList = null;
+ try {
+Deadline.startTimer("getValidWriteIds");
+validWriteIdList =
TxnCommonUtils.createValidReaderWriteIdList(txnStore.getValidWriteIds(
Review comment:
AFAIK, we are targeting strict consistency with "Synchronized metastore
cache" work. Eventual consistency is supported even today with regular refresh
of cache with the interval of 1 min.
The problem is, it returns from cache even if it is invalid which is not
right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609247)
Time Spent: 4h (was: 3h 50m)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[
https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=609252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609252
]
ASF GitHub Bot logged work on HIVE-23571:
-
Author: ASF GitHub Bot
Created on: 09/Jun/21 16:59
Start Date: 09/Jun/21 16:59
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2128:
URL: https://github.com/apache/hive/pull/2128#discussion_r648501960
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -2476,6 +2498,25 @@ public boolean isTableConstraintValid(String catName,
String dbName, String tblN
return isValid;
}
+ public boolean isTableCacheStale(String catName, String dbName, String
tblName, String validWriteIdList) {
+boolean isStale = true;
+
+if (StringUtils.isEmpty(validWriteIdList))
+ return isStale;
Review comment:
Background thread is the current logic to refresh the cache at regular
intervals. Synchronized cache work shouldn't rely on that as it should provide
strict consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609252)
Time Spent: 4h 10m (was: 4h)
> [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
> --
>
> Key: HIVE-23571
> URL: https://issues.apache.org/jira/browse/HIVE-23571
> Project: Hive
> Issue Type: Sub-task
>Reporter: Kishen Das
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in
> deciding whether a given read request can be served from the cache or we have
> to reload it from the backing database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23571) [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper
[ https://issues.apache.org/jira/browse/HIVE-23571?focusedWorklogId=573155&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573155 ] ASF GitHub Bot logged work on HIVE-23571: - Author: ASF GitHub Bot Created on: 28/Mar/21 09:15 Start Date: 28/Mar/21 09:15 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma opened a new pull request #2128: URL: https://github.com/apache/hive/pull/2128 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 573155) Remaining Estimate: 0h Time Spent: 10m > [CachedStore] Add ValidWriteIdList to SharedCache.TableWrapper > -- > > Key: HIVE-23571 > URL: https://issues.apache.org/jira/browse/HIVE-23571 > Project: Hive > Issue Type: Sub-task >Reporter: Kishen Das >Assignee: Ashish Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Add ValidWriteIdList to SharedCache.TableWrapper. This would be used in > deciding whether a given read request can be served from the cache or we have > to reload it from the backing database. -- This message was sent by Atlassian Jira (v8.3.4#803005)
