> On Nov. 2, 2017, 10:40 p.m., Peter Vary wrote: > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java > > Lines 265 (patched) > > <https://reviews.apache.org/r/63528/diff/1/?file=1879637#file1879637line265> > > > > Shouldn't we use > > parts = msdb.getPartitions(dbname, name, partitionBatchSize); > > instead > > parts = msdb.getPartitions(dbname, name, -1); > > > > This way we will load only the required amount of partitions, instead > > of all.
Although msdb.getPartitions(dbname, name, partitionBatchSize) only loads partitionBatchSize amount of partitions, it will always return the same set of partitions. This is used in dropPartitions and it's not a problem there as we're also removing the requested set, so it is guaranteed to get new ones in the next call. For our use case here it is sadly not feasable to use this call. Taking a look at RawStore class I can see no iterative way to query the partitions from the DB behind HMS. - Adam ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63528/#review189979 ----------------------------------------------------------- On Nov. 2, 2017, 10:02 p.m., Adam Szita wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/63528/ > ----------------------------------------------------------- > > (Updated Nov. 2, 2017, 10:02 p.m.) > > > Review request for hive, Peter Vary and Barna Zsombor Klara. > > > Bugs: HIVE-17969 > https://issues.apache.org/jira/browse/HIVE-17969 > > > Repository: hive-git > > > Description > ------- > > Refactoring alter table code to use batching of partitions when calling the > heavy removeUnusedColumnDescriptor method > > > Diffs > ----- > > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java > ccadac1ada6aaae884ab39f5d99e91b8c542404e > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java > 62801c53853dbafb7c425cff943ec819dcee4800 > > > Diff: https://reviews.apache.org/r/63528/diff/1/ > > > Testing > ------- > > > Thanks, > > Adam Szita > >
