[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825543#comment-16825543
]
Gopal V commented on HIVE-21531:
Taking HIVE-21647 as the green sign.
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.2.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824816#comment-16824816
]
Jesus Camacho Rodriguez commented on HIVE-21531:
[~ashutosh.bapat], [~sankarh], the two tests above continue timing out
repeatedly, you can check the ptest server.
Going through the logs, I have realized that they are not batched with other
tests due to HIVE-21109, which seems a step in the right direction. However,
this does not seem sufficient to avoid the timeout.
Could you disable them in the meantime and enable them back once you have fixed
the issue (I guess a possible option may be splitting / rewriting part of those
two tests)? Currently, we cannot check in anything into master.
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.2.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824759#comment-16824759
]
Hive QA commented on HIVE-21531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12966814/HIVE-21531.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15923 tests
executed
*Failed tests:*
{noformat}
TestReplAcidTablesWithJsonMessage - did not produce a TEST-*.xml file (likely
timed out) (batchId=256)
TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely
timed out) (batchId=258)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17022/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17022/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17022/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12966814 - PreCommit-HIVE-Build
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.2.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824731#comment-16824731
]
Hive QA commented on HIVE-21531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17022/dev-support/hive-personality.sh
|
| git revision | master / 313eb24 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17022/yetus/whitespace-eol.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17022/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.2.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824551#comment-16824551
]
Ashutosh Chauhan commented on HIVE-21531:
-
+1
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824547#comment-16824547
]
Gopal V commented on HIVE-21531:
cbo_rp_limit.q golden file needs change -
It was changed recently in HIVE-20572 and this will change it back (& add an
extra test).
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814323#comment-16814323
]
Hive QA commented on HIVE-21531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12965383/HIVE-21531.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15898 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=173)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[1]
(batchId=214)
org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[3]
(batchId=214)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16909/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16909/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16909/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12965383 - PreCommit-HIVE-Build
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814288#comment-16814288
]
Hive QA commented on HIVE-21531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2258 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16909/dev-support/hive-personality.sh
|
| git revision | master / 928f3d6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16909/yetus/whitespace-eol.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16909/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.1.patch,
> HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805061#comment-16805061
]
Hive QA commented on HIVE-21531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12964136/HIVE-21531.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15877 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=169)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16754/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16754/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16754/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12964136 - PreCommit-HIVE-Build
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805020#comment-16805020
]
Hive QA commented on HIVE-21531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
43s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16754/dev-support/hive-personality.sh
|
| git revision | master / 1eb3194 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16754/yetus/whitespace-eol.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16754/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-21531.1.patch, HIVE-21531.WIP.patch
>
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void
[jira] [Commented] (HIVE-21531) Vectorization: all NULL hashcodes are not computed using Murmur3
[
https://issues.apache.org/jira/browse/HIVE-21531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803607#comment-16803607
]
Gopal V commented on HIVE-21531:
{code}
$ scala -cp ~/hw/hive/ql/target/hive-exec-3.2.0-SNAPSHOT.jar
Picked up _JAVA_OPTIONS: -Djava.awt.headless=true -Xmx2048m
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0).
Type in expressions for evaluation. Or try :help.
scala> import org.apache.hive.common.util.Murmur3;
import org.apache.hive.common.util.Murmur3
scala> import org.apache.hive.common.util.HashCodeUtil;
import org.apache.hive.common.util.HashCodeUtil
scala> val bytes = Array[Byte]('a','b','c','d');
bytes: Array[Byte] = Array(97, 98, 99, 100)
scala> HashCodeUtil.calculateBytesHashCode(bytes, 0, 4);
res0: Int = 646393889
scala> Murmur3.hash32(bytes, 0, 4, 0);
res1: Int = 1139631978
{code}
> Vectorization: all NULL hashcodes are not computed using Murmur3
>
>
> Key: HIVE-21531
> URL: https://issues.apache.org/jira/browse/HIVE-21531
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
>
> The comments in Vectorized hash computation call out the MurmurHash
> implementation (the one using 0x5bd1e995), while the non-vectorized codepath
> calls out the Murmur3 one (using 0xcc9e2d51).
> The comments here are wrong
> {code}
> /**
>* Batch compute the hash codes for all the serialized keys.
>*
>* NOTE: MAJOR MAJOR ASSUMPTION:
>* We assume that HashCodeUtil.murmurHash produces the same result
>* as MurmurHash.hash with seed = 0 (the method used by
> ReduceSinkOperator for
>* UNIFORM distribution).
>*/
> protected void computeSerializedHashCodes() {
> int offset = 0;
> int keyLength;
> byte[] bytes = output.getData();
> for (int i = 0; i < nonNullKeyCount; i++) {
> keyLength = serializedKeyLengths[i];
> hashCodes[i] = Murmur3.hash32(bytes, offset, keyLength, 0);
> offset += keyLength;
> }
> }
> {code}
> but the wrong comment is followed in the Vector RS operator
> {code}
> System.arraycopy(nullKeyOutput.getData(), 0, nullBytes, 0,
> nullBytesLength);
> nullKeyHashCode = HashCodeUtil.calculateBytesHashCode(nullBytes, 0,
> nullBytesLength);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
