> On Dec. 7, 2016, 4:43 p.m., Mohit Sabharwal wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line > > 1765 > > <https://reviews.apache.org/r/54393/diff/2/?file=1578074#file1578074line1765> > > > > can BlobStorageUtils.areOptimizationsEnabled(conf) return true for non > > blobstores ?
Yes. This only checks if the flag is enabled or not. Useful for users that do not want Bobstore optimizations even if they use Blobstore paths. Another check happens at the end of the method that verifies that the paths are Blobstore. > On Dec. 7, 2016, 4:43 p.m., Mohit Sabharwal wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java, line > > 1842 > > <https://reviews.apache.org/r/54393/diff/2/?file=1578074#file1578074line1842> > > > > throw RuntimeException I don't think this is necessary. It is a non-check exception. The mergeMovePaths() method should never be called with an invalid MoveWork (that's why shouldMergePaths() is called), but I left it in case a user attempts to call it in a different way in a different method. - Sergio ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54393/#review158345 ----------------------------------------------------------- On Dec. 7, 2016, 5:02 p.m., Sergio Pena wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54393/ > ----------------------------------------------------------- > > (Updated Dec. 7, 2016, 5:02 p.m.) > > > Review request for hive. > > > Bugs: HIVE-15361 > https://issues.apache.org/jira/browse/HIVE-15361 > > > Repository: hive-git > > > Description > ------- > > Problem: > - DynamicPartitionCtx and ListBucketingCtx objects weren't set on the new > MoveWork created when merging the two MoveWork objects from the > ConditionalTask. > > Solution > - Set the DynamicPartitionCtx and ListBucketingCtx objects to the new > MoveWork created for the S3 optimization. > > Other changes > - Merge the MoveWork objects inside the createCondTask() method for better > error handling. > - Only merge the MoveWork related to the moveOnlyMoveTask. The MoveWork from > the mergeAndMoveMoveTask may cause other issues that are not correctly tested. > - Two new private methods are added to check and merge the conditional > input/output tasks to the linked MoveWork. > > > Diffs > ----- > > > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_dynamic_partitions.q > PRE-CREATION > itests/hive-blobstore/src/test/queries/clientpositive/insert_into_table.q > 25e2e7007ff539223d9244ca9822aa65d1441eb0 > > itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_dynamic_partitions.q > PRE-CREATION > > itests/hive-blobstore/src/test/queries/clientpositive/insert_overwrite_table.q > 846b2b113f09a74a3f05c13ffb56163e81dc1e8e > > itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out > PRE-CREATION > > itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out > 223cdf4d17e7fe9959a68f22a0aee6ea3872e2a9 > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out > 9b993a6568b63aa21bf3e644832c6347954f6f59 > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out > PRE-CREATION > > itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out > 81bcc7674727d68e881413dc083d4969013bd0fb > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > adc1188f09c8019a8aa60403d5813d6fa4509ceb > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadDesc.java > bcd3125ab4ad20c00fec565e5004ee200c0187d5 > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadFileDesc.java > 9a868a04ce93d5c2ee75b5c6e96a1401cea93133 > ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java > 771a919ccd0bd75fe6197299ae057647ece89a7e > ql/src/java/org/apache/hadoop/hive/ql/plan/MoveWork.java > 9f498c7fb88a7a9f77b8c6739c097a2b26b0c617 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java > e6ec44504685bd9e53f158cc359b8a7b79fd0166 > > Diff: https://reviews.apache.org/r/54393/diff/ > > > Testing > ------- > > All itests/hive-blobstore tests run. > > Added new blobstore tests: > - insert_into_dynamic_partitions.q > - insert_overwrite_dynamic_partitions.q > > Waiting for HiveQA to run the rest of the q-tests. > > > Thanks, > > Sergio Pena > >
