[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548066#comment-13548066 ] Hudson commented on HIVE-2907: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311850 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q * /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q * /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out * /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.10.0 Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.1.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.2.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.3.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.4.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.5.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.6.patch, ASF.LICENSE.NOT.GRANTED--HIVE-2907.D2505.7.patch, HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[
https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13280304#comment-13280304
]
Srinivas commented on HIVE-2907:
I downloaded the source code for Hive-0.9.1. However, it appears that
ObjectStore.java is missing the fix that fetches partition-metadata in
batches. So, it can still cause issues when trying to drop a table with a large
number of partitions.
Proposed fix in method dropTable from ObjectStore.java
==
int partitionBatchSize = HiveConf.getIntVar(getConf(),
ConfVars.METASTORE_BATCH_RETRIEVE_MAX);
// call dropPartition on each of the table's partitions to follow the
// procedure for cleanly dropping partitions.
ListMPartition partsToDelete = listMPartitions(dbName, tableName,
partitionBatchSize);
while (true){
if (partsToDelete != null || partsToDelete.isEmpty()) {
break;
}
for (MPartition mpart : partsToDelete) {
dropPartitionCommon(mpart);
}
}
Hive error when dropping a table with large number of partitions
Key: HIVE-2907
URL: https://issues.apache.org/jira/browse/HIVE-2907
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.9.0
Environment: General. Hive Metastore bug.
Reporter: Mousom Dhar Gupta
Assignee: Mousom Dhar Gupta
Priority: Minor
Fix For: 0.9.0
Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt,
HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch,
HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch,
HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
Original Estimate: 10h
Remaining Estimate: 10h
Running into an Out Of Memory error when trying to drop a table with 128K
partitions.
The methods dropTable in
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
encounter out of memory errors
when dropping tables with lots of partitions because they try to load the
metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13280317#comment-13280317 ] Srinivas commented on HIVE-2907: Thanks for the timely clarification. Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.10.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13274818#comment-13274818 ] Phabricator commented on HIVE-2907: --- kevinwilfong has accepted the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn To: JIRA, kevinwilfong, mousom Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250791#comment-13250791 ] Kevin Wilfong commented on HIVE-2907: - +1 tests passed, will commit. Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251235#comment-13251235 ] Hudson commented on HIVE-2907: -- Integrated in Hive-trunk-h0.21 #1367 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1367/]) HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850) Result = FAILURE kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1311850 Files : * /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java * /hive/trunk/conf/hive-default.xml.template * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java * /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q * /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q * /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out * /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13240846#comment-13240846 ] Phabricator commented on HIVE-2907: --- kevinwilfong has requested changes to the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. I'm seeing a lot of test failures in TestNegativeCliDriver. They seem to work ok when run individually, but when I run ant test or ant test -Dtestcase=TestNegativeCliDriver they fail. Could you take a look? REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Assignee: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239718#comment-13239718 ] Phabricator commented on HIVE-2907: --- kevinwilfong has accepted the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. +1 Will commit if tests pass. REVISION DETAIL https://reviews.facebook.net/D2505 BRANCH svn Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238662#comment-13238662 ] Mousom Dhar Gupta commented on HIVE-2907: - For ObjectStore::dropTable am dropping pratitions in batches. And for DDLTask::dropTable am getting all partition names and then getting partition metadata from those names in batches. The batch size is determined by a config parameter (set to 1000 to start with). Thanks Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238791#comment-13238791 ] Phabricator commented on HIVE-2907: --- kevinwilfong has commented on the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. This looks really good, thanks for the code clean up, just some small comments. INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 It'd be great if we could use the same config here, otherwise alter table will face the same memory issues for tables with lots of partitions. This could be a separate task too. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399-1401 If any client code already assumes that if 0 is max, it returns all partitions, which, given the code in HiveAlterHandler, seems possible, this will break that code. Could we treat 0 as infinity as well? metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1565-1567 Same here regarding 0 = infinity metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 Can you remove this line if it's not needed. metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 Shouldn't this be table should not have existed common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 Could you add a testcase, where you set this really low, e.g. 1, and you drop a table with multiple partitions. Also, could you add a similar test where you make one of the partitions NO_DROP, and make sure it fails. REVISION DETAIL https://reviews.facebook.net/D2505 Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238894#comment-13238894 ] Phabricator commented on HIVE-2907: --- kevinwilfong has commented on the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. Could you add a negative test, in which hive.metastore.batch.retrieve.max=1; and otherwise the same as drop_table_failure3.q, this should check that you are not looping over the same partition to check for NO_DROP. REVISION DETAIL https://reviews.facebook.net/D2505 Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13238964#comment-13238964 ] Phabricator commented on HIVE-2907: --- mousom has commented on the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. May I modify the drop_table_failure3.q test itself to do that? That might serve both purposes. INLINE COMMENTS metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 Can you please make this a separate task Kevin? Thanks common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 like the *.q file I added? metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 yup metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 my bad. done. common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 i will add both in *.q files. metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399 I thought about it but it's just intuitively does not make sense if we return 'ALL' when 0 is passed.And it might be a very difficult to catch bug for a client.But, on the other hand, you are right that this change might break existing clients and that has to be avoided, so I guess I will make this bug a feature :-) REVISION DETAIL https://reviews.facebook.net/D2505 Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table with large number of partitions
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239099#comment-13239099 ] Phabricator commented on HIVE-2907: --- kevinwilfong has commented on the revision HIVE-2907 [jira] Hive error when dropping a table with large number of partitions. INLINE COMMENTS metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:464-465 Why are you checking for values.size() / 2 here? Admittedly, this works, but shouldn't it be just 2? Or shouldn't you use values.size() / 2 in line 461? metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:463 sp returned metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:506-507 Same here re values.size() / 2 vs. 2 metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:505 same here re returned sp metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:512 sp returned metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:470 sp returned REVISION DETAIL https://reviews.facebook.net/D2505 Hive error when dropping a table with large number of partitions Key: HIVE-2907 URL: https://issues.apache.org/jira/browse/HIVE-2907 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Environment: General. Hive Metastore bug. Reporter: Mousom Dhar Gupta Priority: Minor Fix For: 0.9.0 Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch Original Estimate: 10h Remaining Estimate: 10h Running into an Out Of Memory error when trying to drop a table with 128K partitions. The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors when dropping tables with lots of partitions because they try to load the metadata for every partition into memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
