Re: Review Request 71045: HIVE-21948
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71045/#review216544 --- Ship it! Ship It! - Jesús Camacho Rodríguez On July 11, 2019, 11:32 a.m., Krisztian Kasa wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71045/ > --- > > (Updated July 11, 2019, 11:32 a.m.) > > > Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and > Vineet Garg. > > > Bugs: HIVE-21948 > https://issues.apache.org/jira/browse/HIVE-21948 > > > Repository: hive-git > > > Description > --- > > Implement parallel processing in Pre Upgrade Tool > = > Process tables paralelly using a thread pool for each. Thread pools size can > be defined by command line options. If no options are given the default pool > size is the number of cpu cores. > > I had to remove database parallel processing because the ForkJoinPool > implementation does not support nested parallel streams. > > > Diffs > - > > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java > PRE-CREATION > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java > PRE-CREATION > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java > PRE-CREATION > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java > PRE-CREATION > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java > 0a7354d12b > > upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java > 66213d424c > > upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java > PRE-CREATION > > upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/71045/diff/2/ > > > Testing > --- > > TestPreUpgradeTool UT passed. > > Manually: > 1. deploy ambari and HDP 2.6.5 cluster. > 2. create some hive tables and insert/update values > 3. run PreUpgradeTool and check logs and output script > > > Thanks, > > Krisztian Kasa > >
Re: Review Request 71045: HIVE-21948
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71045/ --- (Updated July 11, 2019, 11:32 a.m.) Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet Garg. Bugs: HIVE-21948 https://issues.apache.org/jira/browse/HIVE-21948 Repository: hive-git Description (updated) --- Implement parallel processing in Pre Upgrade Tool = Process tables paralelly using a thread pool for each. Thread pools size can be defined by command line options. If no options are given the default pool size is the number of cpu cores. I had to remove database parallel processing because the ForkJoinPool implementation does not support nested parallel streams. Diffs (updated) - upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java 0a7354d12b upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java 66213d424c upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java PRE-CREATION upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java PRE-CREATION Diff: https://reviews.apache.org/r/71045/diff/2/ Changes: https://reviews.apache.org/r/71045/diff/1-2/ Testing --- TestPreUpgradeTool UT passed. Manually: 1. deploy ambari and HDP 2.6.5 cluster. 2. create some hive tables and insert/update values 3. run PreUpgradeTool and check logs and output script Thanks, Krisztian Kasa
Review Request 71045: HIVE-21948
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71045/ --- Review request for hive, Jesús Camacho Rodríguez, Zoltan Haindrich, and Vineet Garg. Bugs: HIVE-21948 https://issues.apache.org/jira/browse/HIVE-21948 Repository: hive-git Description --- Implement parallel processing in Pre Upgrade Tool = Process databases and tables paralelly using a thread pool for each. Thread pools size can be defined by command line options. If no options are given the default pool size is the number of cpu cores / 2 Diffs - upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CloseableThreadLocal.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactTablesState.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/CompactionMetaInfo.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/NamedForkJoinWorkerThreadFactory.java PRE-CREATION upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java 0a7354d12b upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/RunOptions.java 66213d424c upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestCloseableThreadLocal.java PRE-CREATION upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestRunOptions.java PRE-CREATION Diff: https://reviews.apache.org/r/71045/diff/1/ Testing --- TestPreUpgradeTool UT passed. Manually: 1. deploy ambari and HDP 2.6.5 cluster. 2. create some hive tables and insert/update values 3. run PreUpgradeTool and check logs and output script Thanks, Krisztian Kasa
