[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578834#comment-16578834 ] Eric Wohlstadter commented on HIVE-20321: - "specific case was query23 and query65" "query23 has group by ss_customer_sk across store_sales" "query65 has group by ss_store_sk, ss_item_sk" > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16578841#comment-16578841 ] Gopal V commented on HIVE-20321: And query51 has "group by ss_item_sk, d_date" > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581335#comment-16581335 ] Hive QA commented on HIVE-20321: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 21s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 45s{color} | {color:red} ql: The patch generated 48 new + 145 unchanged - 14 fixed = 193 total (was 159) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 30s{color} | {color:red} ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 57s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object) is unusual At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 79] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers() may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:[line 90] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines equals but not hashCode At VectorHashKeyWrapperSingleLong.java:hashCode At VectorHashKeyWrapperSingleLong.java:[lines 43-49] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals but not hashCode At VectorHashKeyWrapperTwoLong.java:hashCode At VectorHashKeyWrapperTwoLong.java:[lines 55-63] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13239/dev-support/hive-personality.sh | | git revision | master / 691e654 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13239/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13239/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13239/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assi
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581381#comment-16581381 ] Hive QA commented on HIVE-20321: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935649/HIVE-20321.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 121 failed/errored test(s), 14854 tests executed *Failed tests:* {noformat} TestCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=50) [fileformat_sequencefile.q,dynamic_semijoin_reduction.q,ppd_union.q,explain_ddl.q,vectorization_pushdown.q,udf_to_boolean.q,union_paren.q,join_cond_pushdown_unqual3.q,exim_hidden_files.q,udf_using.q,cross_product_check_1.q,results_cache_quoted_identifiers.q,input_part2.q,udf_int.q,udf_find_in_set.q,insert_values_nonascii.q,vector_groupby_cube1.q,spark_use_ts_stats_for_mapjoin.q,symlink_text_input_format.q,annotate_stats_groupby2.q,inputddl7.q,multi_insert_gby4.q,erasure_simple.q,materialized_view_authorization_sqlstd.q,stats_part.q,overridden_confs.q,interval_arithmetic.q,udf_stddev_samp.q,groupby_multi_single_reducer3.q,vectorization_8.q] org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=256) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12] (batchId=264) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan] (batchId=70) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_view_delete] (batchId=35) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avrocountemptytbl] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_1] (batchId=34) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_3] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketcontext_8] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_simple_select] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_simple_select] (batchId=18) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_date] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[diff_part_input_formats] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[except_all] (batchId=49) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window] (batchId=34) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_rollup_empty] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_2] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_expr_1] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_expr_2] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_by_pos] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partInit] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ptfgroupbyjoin] (batchId=91) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_and] (batchId=2) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_not] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_or] (batchId=40) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[setop_no_distinct] (batchId=86) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_12] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=21) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_multiinsert] (batchId=89) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_unqual_corr_expr] (batchId=8) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_count] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_include_no_sel] (batchId=4) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_join_no_keys] (batchId=46) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_parquet_projection] (batchId=45) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_queries] (batchId=103) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[bucket5] (batchId=155) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[except_distinct] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581881#comment-16581881 ] Matt McCline commented on HIVE-20321: - With patch #3, there are 4 different kinds of vector hash key wrapper classes with these members: 1) VectorHashKeyWrapperEmpty (int hashcode) 2) VectorHashKeyWrapperSingleLong (int hashcode, boolean isNull flag, long value) 3) VectorHashKeyWrapperTwoLong (int hashcode, 2 X boolean isNull flags 2 X long values) 4) VectorHashKeyWrapperGeneral (original class with many members including arrays for all data types, etc) With the new code "framework", we can add special case classes for other keys as desired. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582110#comment-16582110 ] Gopal V commented on HIVE-20321: LGTM - +1 The single key ::equals() implementation has been improved by 15% across the whole query I tested with (& dropped out of the top-5 functions). The only test query which did not improve was {{select count(distinct ss_item_sk), count(distinct ss_customer_sk) from store_sales;}} > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582140#comment-16582140 ] Hive QA commented on HIVE-20321: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 8s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 50 new + 144 unchanged - 15 fixed = 194 total (was 159) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 29s{color} | {color:red} ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object) is unusual At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 77] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers() may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:[line 89] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines equals but not hashCode At VectorHashKeyWrapperSingleLong.java:hashCode At VectorHashKeyWrapperSingleLong.java:[lines 43-49] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals but not hashCode At VectorHashKeyWrapperTwoLong.java:hashCode At VectorHashKeyWrapperTwoLong.java:[lines 55-63] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13259/dev-support/hive-personality.sh | | git revision | master / 463a512 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13259/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13259/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13259/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assi
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582202#comment-16582202 ] Hive QA commented on HIVE-20321: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935795/HIVE-20321.03.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14880 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] (batchId=194) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13259/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13259/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13259/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12935795 - PreCommit-HIVE-Build > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582912#comment-16582912 ] Hive QA commented on HIVE-20321: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 57s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 18s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 50 new + 144 unchanged - 15 fixed = 194 total (was 159) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 26s{color} | {color:red} ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 42s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object) is unusual At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 77] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers() may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:[line 89] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines equals but not hashCode At VectorHashKeyWrapperSingleLong.java:hashCode At VectorHashKeyWrapperSingleLong.java:[lines 43-49] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals but not hashCode At VectorHashKeyWrapperTwoLong.java:hashCode At VectorHashKeyWrapperTwoLong.java:[lines 55-63] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13273/dev-support/hive-personality.sh | | git revision | master / ddc2465 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13273/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assi
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583286#comment-16583286 ] Hive QA commented on HIVE-20321: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 7s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s{color} | {color:red} ql: The patch generated 50 new + 144 unchanged - 15 fixed = 194 total (was 159) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 20s{color} | {color:red} ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object) is unusual At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 77] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers() may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:[line 89] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines equals but not hashCode At VectorHashKeyWrapperSingleLong.java:hashCode At VectorHashKeyWrapperSingleLong.java:[lines 43-49] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals but not hashCode At VectorHashKeyWrapperTwoLong.java:hashCode At VectorHashKeyWrapperTwoLong.java:[lines 55-63] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13283/dev-support/hive-personality.sh | | git revision | master / 8c6b5c6 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13283/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13283/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13283/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assi
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583303#comment-16583303 ] Hive QA commented on HIVE-20321: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935907/HIVE-20321.05.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14884 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=252) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13283/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13283/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13283/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12935907 - PreCommit-HIVE-Build > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch, HIVE-20321.04.patch, HIVE-20321.05.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583785#comment-16583785 ] Hive QA commented on HIVE-20321: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 27s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 7s{color} | {color:blue} ql in master has 2305 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s{color} | {color:red} ql: The patch generated 50 new + 144 unchanged - 15 fixed = 194 total (was 159) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 25s{color} | {color:red} ql generated 4 new + 2304 unchanged - 1 fixed = 2308 total (was 2305) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBase.equals(Object) is unusual At VectorHashKeyWrapperBase.java:VectorHashKeyWrapperBase.java:[line 77] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperBatch.getVectorHashKeyWrappers() may expose internal representation by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:by returning VectorHashKeyWrapperBatch.vectorHashKeyWrappers At VectorHashKeyWrapperBatch.java:[line 89] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperSingleLong defines equals but not hashCode At VectorHashKeyWrapperSingleLong.java:hashCode At VectorHashKeyWrapperSingleLong.java:[lines 43-49] | | | org.apache.hadoop.hive.ql.exec.vector.wrapper.VectorHashKeyWrapperTwoLong defines equals but not hashCode At VectorHashKeyWrapperTwoLong.java:hashCode At VectorHashKeyWrapperTwoLong.java:[lines 55-63] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-13299/dev-support/hive-personality.sh | | git revision | master / 59cf159 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-13299/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-13299/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-13299/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assi
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583809#comment-16583809 ] Hive QA commented on HIVE-20321: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12935961/HIVE-20321.06.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 14884 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/13299/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/13299/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-13299/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12935961 - PreCommit-HIVE-Build > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch, HIVE-20321.04.patch, HIVE-20321.05.patch, > HIVE-20321.06.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583917#comment-16583917 ] Matt McCline commented on HIVE-20321: - Green test run. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch, HIVE-20321.04.patch, HIVE-20321.05.patch, > HIVE-20321.06.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20321) Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 CacheLine
[ https://issues.apache.org/jira/browse/HIVE-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583919#comment-16583919 ] Matt McCline commented on HIVE-20321: - Committed to master. [~gopalv] thank you for your code review and perf checking. > Vectorization: Cut down memory size of 1 col VectorHashKeyWrapper to <1 > CacheLine > - > > Key: HIVE-20321 > URL: https://issues.apache.org/jira/browse/HIVE-20321 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Major > Attachments: HIVE-20321.01.patch, HIVE-20321.02.patch, > HIVE-20321.03.patch, HIVE-20321.04.patch, HIVE-20321.05.patch, > HIVE-20321.06.patch > > > With a full sized LLAP instance, the memory size of the VectorHashKeyWrapper > is bigger than the low Xmx JVMs. > {code} > * 64-bit VM: ** > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper object internals: > OFFSET SIZE > TYPE DESCRIPTION VALUE > 016 > (object header) N/A > 16 4 > int VectorHashKeyWrapper.hashcodeN/A > 20 4 > (alignment/padding gap) > 24 8 > long[] VectorHashKeyWrapper.longValues N/A > 32 8 > double[] VectorHashKeyWrapper.doubleValuesN/A > 40 8 > byte[][] VectorHashKeyWrapper.byteValues N/A > 48 8 >int[] VectorHashKeyWrapper.byteStarts N/A > 56 8 >int[] VectorHashKeyWrapper.byteLengths N/A > 64 8 > org.apache.hadoop.hive.serde2.io.HiveDecimalWritable[] > VectorHashKeyWrapper.decimalValues N/A > 72 8 > java.sql.Timestamp[] VectorHashKeyWrapper.timestampValues N/A > 80 8 > org.apache.hadoop.hive.common.type.HiveIntervalDayTime[] > VectorHashKeyWrapper.intervalDayTimeValues N/A > 88 8 > boolean[] VectorHashKeyWrapper.isNull N/A > 96 8 > org.apache.hadoop.hive.ql.exec.vector.VectorHashKeyWrapper.HashContext > VectorHashKeyWrapper.hashCtx N/A > Instance size: 104 bytes > Space losses: 4 bytes internal + 0 bytes external = 4 bytes total > {code} > Pulling this up to a parent class allows for this to be cut down to 32 bytes > for the single column case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)