[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16483126#comment-16483126 ] Vihang Karajgaonkar commented on HIVE-18629: Oh I see. In that case can we close this issue? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481335#comment-16481335 ] Sergey Shelukhin commented on HIVE-18629: - See the discussion above. [~mmccline] doesn't like it :) It may be superseded by the epic vectorization null fix patch. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16481277#comment-16481277 ] Vihang Karajgaonkar commented on HIVE-18629: [~sershe] Was this not merged for a reason? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370476#comment-16370476 ] Sergey Shelukhin commented on HIVE-18629: - [~mmccline] ping? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361687#comment-16361687 ] Hive QA commented on HIVE-18629: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12910278/HIVE-18629.03.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13158 tests executed *Failed tests:* {noformat} TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=92) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,index_bitmap_auto.q,bucket_num_reducers_acid2.q] org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=241) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat] (batchId=180) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde] (batchId=180) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] (batchId=251) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=222) org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded] (batchId=206) org.apache.hadoop.hive.ql.plan.TestReadEntityDirect.org.apache.hadoop.hive.ql.plan.TestReadEntityDirect (batchId=293) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase (batchId=243) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=235) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=235) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=235) org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill (batchId=236) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9180/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9180/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9180/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12910278 - PreCommit-HIVE-Build > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361643#comment-16361643 ] Hive QA commented on HIVE-18629: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} storage-api: The patch generated 2 new + 4 unchanged - 2 fixed = 6 total (was 6) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 9m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 0808f7d | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9180/yetus/diff-checkstyle-storage-api.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9180/yetus/patch-asflicense-problems.txt | | modules | C: storage-api U: storage-api | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9180/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361527#comment-16361527 ] Sergey Shelukhin commented on HIVE-18629: - [~mmccline] this patch as is will either already set noNulls to false when isNull is set to true, or throw an error > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361518#comment-16361518 ] Matt McCline commented on HIVE-18629: - I think the patch is still wrong. 1) Until HIVE-18622 is available to fix the problem, this focused fix could encounter an output column that is corrupted. 2) For the isRepeated case you need to set noNulls to false when isNull is set to true. I still think this version is best. It is conservative in that it makes sure the output vector isNull entries are set correctly. {code} public void copySelected( boolean selectedInUse, int[] sel, int size, BytesColumnVector output) { // We do not need to do a column reset since we are carefully changing the output. output.isRepeating = false; // Handle repeating case if (isRepeating) { if (noNulls || !isNull[0]) { output.isNull[0] = false; output.setVal(0, vector[0], start[0], length[0]); } else { output.isNull[0] = true; output.noNulls = false; } output.isRepeating = true; return; } // Handle normal case if (noNulls) { /* * Make sure our output results have their isNull entry initialized to false. * NOTE: We leave outputColVector.noNulls flag alone since we don't clear all * the isNull entries. */ if (selectedInUse) { for (int j = 0; j < size; j++) { int i = sel[j]; output.isNull[i] = false; output.setVal(i, vector[i], start[i], length[i]); } } else { Arrays.fill(output.isNull, 0, size, false); for(int i = 0; i < size; ++i) { output.setVal(i, vector[i], start[i], length[i]); } } } else { // Carefully handle NULLs... if (selectedInUse) { for (int j = 0; j < size; j++) { int i = sel[j]; if (!isNull[i]) { output.isNull[i] = false; output.setVal(i, vector[i], start[i], length[i]); } else { output.isNull[i] = true; output.noNulls = false; } } } else { for (int i = 0; i < size; i++) { if (!isNull[i]) { output.isNull[i] = false; output.setVal(i, vector[i], start[i], length[i]); } else { output.isNull[i] = true; output.noNulls = false; } } } } {code} > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361446#comment-16361446 ] Sergey Shelukhin commented on HIVE-18629: - Added an assertion after discussing with [~mmccline] Can you take a look at the updated patch? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.03.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358065#comment-16358065 ] Hive QA commented on HIVE-18629: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909695/HIVE-18629.02.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12995 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_serde] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde] (batchId=180) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1] (batchId=250) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestAddPartitions.testAddPartitionsNullColTypeInSd[Embedded] (batchId=206) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients (batchId=200) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9110/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9110/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9110/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909695 - PreCommit-HIVE-Build > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358032#comment-16358032 ] Hive QA commented on HIVE-18629: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} storage-api: The patch generated 2 new + 4 unchanged - 2 fixed = 6 total (was 6) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 59s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3464df4 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9110/yetus/diff-checkstyle-storage-api.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9110/yetus/diff-checkstyle-ql.txt | | modules | C: storage-api ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9110/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356278#comment-16356278 ] Hive QA commented on HIVE-18629: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909544/HIVE-18629.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12969 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet (batchId=215) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill (batchId=235) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9080/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9080/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909544 - PreCommit-HIVE-Build > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356235#comment-16356235 ] Sergey Shelukhin commented on HIVE-18629: - Updated to also handle isRepeating case. [~mmccline] I don't think the changes around noNulls are necessary... they can only become relevant if input vector is invalid in the first place and has isNull elements when noNulls is set. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.02.patch, > HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356232#comment-16356232 ] Hive QA commented on HIVE-18629: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} storage-api: The patch generated 1 new + 4 unchanged - 2 fixed = 5 total (was 6) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 50s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / ee2c814 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-storage-api.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus/diff-checkstyle-ql.txt | | modules | C: storage-api ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9080/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356214#comment-16356214 ] Vihang Karajgaonkar commented on HIVE-18629: Patch looks good to me. Can we include a test case which would have caused the NPE before the patch? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356009#comment-16356009 ] Sergey Shelukhin commented on HIVE-18629: - No new test failures. cc [~vihangk1] [~mmccline] > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354914#comment-16354914 ] Hive QA commented on HIVE-18629: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12909347/HIVE-18629.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12974 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=175) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=161) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=122) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.client.TestAddAlterDropIndexes.testDropIndexInvalidDB[Embedded] (batchId=206) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=280) org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCTAS (batchId=280) org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap (batchId=282) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9059/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9059/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9059/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 22 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12909347 - PreCommit-HIVE-Build > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354888#comment-16354888 ] Hive QA commented on HIVE-18629: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 38s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} storage-api: The patch generated 1 new + 4 unchanged - 2 fixed = 5 total (was 6) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 16m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 3972bf0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus/diff-checkstyle-storage-api.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus/diff-checkstyle-ql.txt | | modules | C: storage-api ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9059/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354880#comment-16354880 ] Sergey Shelukhin commented on HIVE-18629: - [~vihangk1] updated the patch. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.01.patch, HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354609#comment-16354609 ] Sergey Shelukhin commented on HIVE-18629: - Sorry, yeah, I mixed it up when changing the ifs around. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354585#comment-16354585 ] Vihang Karajgaonkar commented on HIVE-18629: Hi [~sershe] I took a quick look at the attached patch. Shouldn't the snippet below also guard against NPE by adding a if condition {{if (!(output.isNull[i] = isNull[i]))}} after we do a System.arraycopy {code} } else { System.arraycopy(isNull, 0, output.isNull, 0, size); for (int i = 0; i < size; i++) { output.setVal(i, vector[i], start[i], length[i]); } } {code} > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354350#comment-16354350 ] Sergey Shelukhin commented on HIVE-18629: - [~vihangk1] no that makes sense, they might be related. I don't see that the patch there handles copySelected though. Note that If... expr class has two paths; if the condition vector (#1) is repeating it calls copySelected on v2 or v3 (the sides of the case expression); otherwise it goes one by one to look at v1 values to copy values from v2 and v3 one by one. This separation by v1's repeatability should have no bearing on how v2 and v3 are treated; however the separate-setVal logic in ifexpr, under copySelected path, has isNull checks protecting setVal, whereas copySelected doesn't have such checks and calls setVal blindly. So as far as I can tell either this patch is necessary, or null checks on the non-repeating path in IfExpr... are unnecessary. That's just the low level observation I made, I'm not super familiar with vectorization changes recently. So this may be necessary in addition to HIVE-18622. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353493#comment-16353493 ] Vihang Karajgaonkar commented on HIVE-18629: oops didn't see the comment above. I will let [~mmccline] decide. Pls ignore my comment above. > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353490#comment-16353490 ] Vihang Karajgaonkar commented on HIVE-18629: possibly related to HIVE-18622? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18629) copyValues in BytesColumnVector may be missing null checks
[ https://issues.apache.org/jira/browse/HIVE-18629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16353237#comment-16353237 ] Sergey Shelukhin commented on HIVE-18629: - [~mmccline] does this patch make sense? > copyValues in BytesColumnVector may be missing null checks > -- > > Key: HIVE-18629 > URL: https://issues.apache.org/jira/browse/HIVE-18629 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18629.patch > > > {noformat} > Caused by: java.lang.NullPointerException > at java.lang.System.arraycopy(Native Method) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173) > at > org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:333) > at > org.apache.hadoop.hive.ql.exec.vector.expressions..evaluate(IfExprStringGroupColumnStringGroupColumn.java:83) > at > org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:133) > {noformat} > IfExprStringGroupColumnStringGroupColumn code below the v1.isRepeating case > has isNull checks for v2/v3 buffers that copySelected is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)