[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861666#comment-16861666
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12971475/HIVE-21796.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16050 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17528/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17528/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17528/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12971475 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.5.patch, HIVE-21796.patch,
> jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861644#comment-16861644
]
Hive QA commented on HIVE-21796:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2260 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17528/dev-support/hive-personality.sh
|
| git revision | master / 4f25b25 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17528/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.5.patch, HIVE-21796.patch,
> jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860993#comment-16860993
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12971428/HIVE-21796.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 16050 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime
(batchId=274)
org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime
(batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17519/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17519/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17519/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12971428 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860953#comment-16860953
]
Hive QA commented on HIVE-21796:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2260 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17519/dev-support/hive-personality.sh
|
| git revision | master / 922b24e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17519/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.4.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859882#comment-16859882 ] Yongzhi Chen commented on HIVE-21796: - [~zmatyus], we have to pass all the tests although the failures may not be related. Please re-run the tests. > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858860#comment-16858860 ] Yongzhi Chen commented on HIVE-21796: - Each field's index in the list is stored in the field object itself (StructFieldImpl index), if it is a valid list, the index in the list must match the value in StructFieldImpl. So two same size valid lists with the same set of value entries should have the same order. Double check the code, fields and the map are all calculated in ArrayWritableObjectInspector, and there are no other ways to change the map, so equal fieldsName combined with equal fields should create the equal map. You change look fine +1 > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858617#comment-16858617 ] Csaba Ringhofer commented on HIVE-21796: [~ychena] > I think to keep the map check, for names and fields, we only check if their > list size equal. That would be a weaker check, because it would ignore the order of the fields. > If we remove map comparing, it is possible a name map to the wrong field? That would be true if fields+fieldNames would be sets, but those are lists and have the same order, so name->field mapping can be always recalculated from them. > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858094#comment-16858094
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12971043/HIVE-21796.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 16082 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=271)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=271)
org.apache.hive.jdbc.TestActivePassiveHA.testConnectionActivePassiveHAServiceDiscovery
(batchId=271)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailover (batchId=271)
org.apache.hive.jdbc.TestActivePassiveHA.testManualFailoverUnauthorized
(batchId=271)
org.apache.hive.jdbc.TestActivePassiveHA.testNoConnectionOnPassive (batchId=271)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17457/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17457/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17457/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12971043 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16858049#comment-16858049
]
Hive QA commented on HIVE-21796:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17457/dev-support/hive-personality.sh
|
| git revision | master / e7f2fccd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17457/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Zoltan Matyus
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857852#comment-16857852 ] Yongzhi Chen commented on HIVE-21796: - It looks for me that this fix may regress HIVE-15130, the original code compare names, fields and the map( from names to fields). If we remove map comparing, it is possible a name map to the wrong field? > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857503#comment-16857503 ] Zoltan Matyus commented on HIVE-21796: -- bq. ERROR: -1 due to 16 failed/errored test(s), 16082 tests executed Failed tests are unrelated, retrying with same content. > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Zoltan Matyus >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch, > HIVE-21796.3.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857191#comment-16857191
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970938/HIVE-21796.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 16082 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_avro]
(batchId=286)
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=272)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17441/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17441/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17441/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970938 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857169#comment-16857169
]
Hive QA commented on HIVE-21796:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
56s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17441/dev-support/hive-personality.sh
|
| git revision | master / e7f2fccd |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17441/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.2.patch,
> HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855973#comment-16855973
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970829/HIVE-21796.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16080 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17418/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17418/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17418/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970829 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855932#comment-16855932
]
Hive QA commented on HIVE-21796:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
55s{color} | {color:blue} ql in master has 2256 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 4 new + 7 unchanged - 0 fixed
= 11 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17418/dev-support/hive-personality.sh
|
| git revision | master / eace41d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17418/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17418/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.1.patch, HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855668#comment-16855668 ] Csaba Ringhofer commented on HIVE-21796: Added new patch that also contains tests (written by [~zmatyus]). > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > Attachments: HIVE-21796.1.patch, HIVE-21796.patch, jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852731#comment-16852731
]
Hive QA commented on HIVE-21796:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12970373/HIVE-21796.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16064 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17365/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17365/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17365/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12970373 - PreCommit-HIVE-Build
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[
https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852705#comment-16852705
]
Hive QA commented on HIVE-21796:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 5m
3s{color} | {color:blue} ql in master has 2259 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17365/dev-support/hive-personality.sh
|
| git revision | master / c280615 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
>
>
> Key: HIVE-21796
> URL: https://issues.apache.org/jira/browse/HIVE-21796
> Project: Hive
> Issue Type: Bug
>Reporter: Csaba Ringhofer
>Assignee: Csaba Ringhofer
>Priority: Major
> Attachments: HIVE-21796.patch, jstack.txt
>
>
> The issue came up during an Impala test when we tried to run it with Hive
> 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a
> 99 level nested column to a similar Orc file, and spent its time in
> ArrayWritableObjectInspector.equals() according to jstack.
> The problem seems to be that equals() calls both fields.equals(that.fields)
> and fieldsByName.equals(that.fieldsByName), and both try to compare all
> nested fields recursively, which leads to the O(2^nesting_depth) complexity.
> The commit that introduced this behavior:
> https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed
> The Impala test:
> https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851114#comment-16851114 ] Csaba Ringhofer commented on HIVE-21796: [~tlipcon] The fan-out occurs by calling both fields.equals(that.fields) and fieldsByName.equals(that.fieldsByName) - both will call equal() for fields at the next level, which will again call both of them... I have attached a jstack where this occurs - java.util.AbstractMap.equals(AbstractMap.java:495) and java.util.AbstractList.equals(AbstractList.java:523) occur in the callstack in random order (that's how I realized what is going on). > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Priority: Major > Attachments: jstack.txt > > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21796) ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time
[ https://issues.apache.org/jira/browse/HIVE-21796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851089#comment-16851089 ] Todd Lipcon commented on HIVE-21796: I looked through this code and it wasn't obvious to me how this ends up being exponential-time. I would have thought that the recursion depth would be exactly the nesting depth without any "fan-out". > ArrayWritableObjectInspector.equals can take O(2^nesting_depth) time > > > Key: HIVE-21796 > URL: https://issues.apache.org/jira/browse/HIVE-21796 > Project: Hive > Issue Type: Bug >Reporter: Csaba Ringhofer >Priority: Major > > The issue came up during an Impala test when we tried to run it with Hive > 3.1. The a query hanged: it tried to insert 1 row from a Parquet file with a > 99 level nested column to a similar Orc file, and spent its time in > ArrayWritableObjectInspector.equals() according to jstack. > The problem seems to be that equals() calls both fields.equals(that.fields) > and fieldsByName.equals(that.fieldsByName), and both try to compare all > nested fields recursively, which leads to the O(2^nesting_depth) complexity. > The commit that introduced this behavior: > https://github.com/apache/hive/commit/98a25f2d831ab27e174bc99792047eaa8ec08b82#diff-8c6363e90d442f239bc252a104f1bfed > The Impala test: > https://github.com/apache/impala/blob/9ee4a5e1940afa47227a92e0f6fba6d4c9909f63/tests/query_test/test_nested_types.py#L612 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
