[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=736225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736225 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 03/Mar/22 18:50 Start Date: 03/Mar/22 18:50 Worklog Time Spent: 10m Work Description: sourabh912 commented on pull request #3042: URL: https://github.com/apache/hive/pull/3042#issuecomment-1058376954 Merged into master. Commit: 3c53faafe230eb7b611d3dfb5057a81bfd1e96d1 Thank you @nrg4878 for the review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736225) Time Spent: 1h (was: 50m) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Attachments: > 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch > > Time Spent: 1h > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=736226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-736226 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 03/Mar/22 18:50 Start Date: 03/Mar/22 18:50 Worklog Time Spent: 10m Work Description: sourabh912 closed pull request #3042: URL: https://github.com/apache/hive/pull/3042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 736226) Time Spent: 1h 10m (was: 1h) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Attachments: > 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=732669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-732669 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 24/Feb/22 20:50 Start Date: 24/Feb/22 20:50 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #3042: URL: https://github.com/apache/hive/pull/3042#issuecomment-1050251249 Looks good to me. +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 732669) Time Spent: 50m (was: 40m) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Attachments: > 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch > > Time Spent: 50m > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=729166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729166 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 17/Feb/22 19:00 Start Date: 17/Feb/22 19:00 Worklog Time Spent: 10m Work Description: sourabh912 commented on pull request #3042: URL: https://github.com/apache/hive/pull/3042#issuecomment-1043307276 @nrg4878 : Please review and share your feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 729166) Time Spent: 40m (was: 0.5h) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Attachments: > 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch > > Time Spent: 40m > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=729165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-729165 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 17/Feb/22 18:59 Start Date: 17/Feb/22 18:59 Worklog Time Spent: 10m Work Description: sourabh912 opened a new pull request #3042: URL: https://github.com/apache/hive/pull/3042 ### What changes were proposed in this pull request? Enabling the property hive.load.dynamic.partitions.scan.specific.partitions othewise a user sees significant performance degradation in insert/insert overwrite queries for large partitioned tables ### Why are the changes needed? Improves performance for insert/insert overwrite queries for large partitioned tables. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Relying on existing tests since it is just a configuration change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 729165) Time Spent: 0.5h (was: 20m) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Attachments: > 0001-HIVE-25178-Reduce-number-of-getPartition-calls-durin.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions for External Tables
[ https://issues.apache.org/jira/browse/HIVE-25178?focusedWorklogId=605113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-605113 ] ASF GitHub Bot logged work on HIVE-25178: - Author: ASF GitHub Bot Created on: 02/Jun/21 11:00 Start Date: 02/Jun/21 11:00 Worklog Time Spent: 10m Work Description: aasha merged pull request #2338: URL: https://github.com/apache/hive/pull/2338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 605113) Time Spent: 20m (was: 10m) > Reduce number of getPartition calls during loadDynamicPartitions for External > Tables > > > Key: HIVE-25178 > URL: https://issues.apache.org/jira/browse/HIVE-25178 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Narayanan Venkateswaran >Priority: Major > Labels: performance, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When dynamic partitions are loaded, Hive::loadDynamicPartition loads all > partitions from HMS causing heavy load on it. This becomes worse when large > number of partitions are present in tables. > Only relevant partitions being loaded in dynamic partitions can be queried > from HMS for partition existence. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958] -- This message was sent by Atlassian Jira (v8.3.4#803005)
