[jira] [Commented] (HIVE-22458) Add more constraints on showing partitions
[
https://issues.apache.org/jira/browse/HIVE-22458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082126#comment-17082126
]
Hive QA commented on HIVE-22458:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
38s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
53s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch parser passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 1426 unchanged - 1 fixed = 1426 total (was 1427) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21598/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common parser
standalone-metastore/metastore-server ql itests/hcatalog-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21598/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add more constraints on showing parti
[jira] [Commented] (HIVE-22458) Add more constraints on showing partitions
[
https://issues.apache.org/jira/browse/HIVE-22458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082139#comment-17082139
]
Hive QA commented on HIVE-22458:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999739/HIVE-22458.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18217 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21598/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21598/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21598/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999739 - PreCommit-HIVE-Build
> Add more constraints on showing partitions
> --
>
> Key: HIVE-22458
> URL: https://issues.apache.org/jira/browse/HIVE-22458
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22458.2.patch, HIVE-22458.3.patch,
> HIVE-22458.4.patch, HIVE-22458.5.patch, HIVE-22458.branch-1.02.patch,
> HIVE-22458.branch-1.patch, HIVE-22458.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When we showing partitions of a table with thousands of partitions, all the
> partitions will be returned and it's not easy to catch the specified one from
> them, this make showing partitions hard to use. We can add where/limit/order
> by constraints to show partitions like:
> show partitions table_name [partition_specs] where partition_key >= value
> order by partition_key desc limit n;
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23173) User login success/failed attempts should be logged
[ https://issues.apache.org/jira/browse/HIVE-23173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082147#comment-17082147 ] mahesh kumar behera commented on HIVE-23173: why the test code changes are done ? > User login success/failed attempts should be logged > --- > > Key: HIVE-23173 > URL: https://issues.apache.org/jira/browse/HIVE-23173 > Project: Hive > Issue Type: Improvement >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Minor > Attachments: HIVE-23173.1.patch, HIVE-23173.2.patch, > HIVE-23173.3.patch > > > User login success & failure attempts should be logged in server logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23185) Historic queries lost after HS2 restart
[ https://issues.apache.org/jira/browse/HIVE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082153#comment-17082153 ] Zhihua Deng commented on HIVE-23185: The unfinished/some canceled operation logs may remain uncleaned after HS2 restart. > Historic queries lost after HS2 restart > --- > > Key: HIVE-23185 > URL: https://issues.apache.org/jira/browse/HIVE-23185 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Zhihua Deng >Priority: Major > > QueryInfoCache caches historic queries in memory, when HS2 restart due to OOM > or upgrade, the queries are no longer seen at webui. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23184) Upgrade druid to 0.17.1
[
https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082171#comment-17082171
]
Hive QA commented on HIVE-23184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999748/HIVE-23184.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 18216 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz2]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_extractTime]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_floorTime]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_semijoin_reduction_all_types]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=206)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_ts]
(batchId=206)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateEmptyTablePlusCommitDropTableWithPurge
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateEmptyTablePlusCommitDropTableWithoutPurge
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateTablePlusCommitDropTableWithPurge
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitCreateTablePlusCommitDropTableWithoutPurge
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitEmptyInsertTable
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertIntoTable
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertIntoWhenDestinationSegmentFileExist
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertIntoWithConflictingIntervalSegment
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertIntoWithNonExtendableSegment
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertOverwriteTable
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitInsertTable
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testCommitMultiInsertOverwriteTable
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testDeleteSegment
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testInsertIntoAppendOneMorePartition
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testPreCreateTableWhenDataSourceExists
(batchId=316)
org.apache.hadoop.hive.druid.TestDruidStorageHandler.testPreCreateTableWillCreateSegmentsTable
(batchId=316)
org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat.testTimeZone
(batchId=316)
org.apache.hadoop.hive.druid.io.TestHiveDruidSplit.testSerDeser (batchId=316)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidDeserializer
(batchId=316)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidObjectDeserializer
(batchId=316)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidObjectSerializer
(batchId=316)
org.apache.hadoop.hive.druid.serde.TestDruidSerDe.testDruidObjectSerializerwithNullTimestamp
(batchId=316)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testSerDesr (batchId=316)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testTimeStampColumnName
(batchId=316)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21599/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21599/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21599/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999748 - PreCommit-HIVE-Build
> Upgrade druid to 0.17.1
> ---
>
> Key: HIVE-23184
> URL: https://issues.apache.org/jira/browse/HIVE-23184
>
[jira] [Commented] (HIVE-23184) Upgrade druid to 0.17.1
[
https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082172#comment-17082172
]
Hive QA commented on HIVE-23184:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} druid-handler: The patch generated 60 new + 0
unchanged - 0 fixed = 60 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
2s{color} | {color:red} root: The patch generated 60 new + 0 unchanged - 0
fixed = 60 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21599/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21599/yetus/diff-checkstyle-druid-handler.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21599/yetus/diff-checkstyle-root.txt
|
| modules | C: ql druid-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21599/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade druid to 0.17.1
> ---
>
> Key: HIVE-23184
> URL: https://issues.apache.org/jira/browse/HIVE-23184
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
>
[jira] [Commented] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082176#comment-17082176
]
Miklos Gergely commented on HIVE-23183:
---
+1
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23006) Basic compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23006: -- Attachment: HIVE-23006.08.patch > Basic compiler support for Probe MapJoin > > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23006.01.patch, HIVE-23006.02.patch, > HIVE-23006.03.patch, HIVE-23006.04.patch, HIVE-23006.05.patch, > HIVE-23006.06.patch, HIVE-23006.07.patch, HIVE-23006.08.patch > > Time Spent: 3h > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.3.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23137: -- Attachment: (was: HIVE-23137.01.patch) > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23137.01.patch, HIVE-23137.01.patch > > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23137: -- Attachment: HIVE-23137.01.patch > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23137.01.patch, HIVE-23137.01.patch > > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23137: -- Attachment: (was: HIVE-23137.01.patch) > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23137.01.patch > > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22320:
Attachment: HIVE-22320.01.patch
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-22320:
Status: Patch Available (was: In Progress)
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22320 started by László Bodor.
---
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work stopped] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22320 stopped by László Bodor.
---
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work started] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22320 started by László Bodor.
---
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22458) Add more constraints on showing partitions
[ https://issues.apache.org/jira/browse/HIVE-22458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082201#comment-17082201 ] Miklos Gergely commented on HIVE-22458: --- [~dengzh] please also modify the documentation. > Add more constraints on showing partitions > -- > > Key: HIVE-22458 > URL: https://issues.apache.org/jira/browse/HIVE-22458 > Project: Hive > Issue Type: Improvement >Reporter: Zhihua Deng >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22458.2.patch, HIVE-22458.3.patch, > HIVE-22458.4.patch, HIVE-22458.5.patch, HIVE-22458.branch-1.02.patch, > HIVE-22458.branch-1.patch, HIVE-22458.patch > > Time Spent: 50m > Remaining Estimate: 0h > > When we showing partitions of a table with thousands of partitions, all the > partitions will be returned and it's not easy to catch the specified one from > them, this make showing partitions hard to use. We can add where/limit/order > by constraints to show partitions like: > show partitions table_name [partition_specs] where partition_key >= value > order by partition_key desc limit n; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082176#comment-17082176
]
Miklos Gergely edited comment on HIVE-23183 at 4/13/20, 10:05 AM:
--
+1, please make sure that the change is also present in the DDL documentation.
was (Author: mgergely):
+1
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23006) Basic compiler support for Probe MapJoin
[
https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082208#comment-17082208
]
Hive QA commented on HIVE-23006:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 4 new + 374 unchanged - 0
fixed = 378 total (was 374) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 9 new + 112 unchanged - 0
fixed = 121 total (was 112) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
2s{color} | {color:red} ql generated 3 new + 1527 unchanged - 1 fixed = 1530
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Class org.apache.hadoop.hive.ql.exec.TableScanOperator defines
non-transient non-serializable instance field probeDecodeContextSet In
TableScanOperator.java:instance field probeDecodeContextSet In
TableScanOperator.java |
| | Class org.apache.hadoop.hive.ql.plan.MapWork defines non-transient
non-serializable instance field probeDecodeContext In MapWork.java:instance
field probeDecodeContext In MapWork.java |
| | Class org.apache.hadoop.hive.ql.plan.TableScanDesc defines non-transient
non-serializable instance field probeDecodeContext In
TableScanDesc.java:instance field probeDecodeContext In TableScanDesc.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21600/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21600/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21600/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21600/yetus/new-findbugs-ql.html
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21600/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apa
[jira] [Commented] (HIVE-23006) Basic compiler support for Probe MapJoin
[
https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082232#comment-17082232
]
Hive QA commented on HIVE-23006:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999753/HIVE-23006.08.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18218 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21600/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21600/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21600/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999753 - PreCommit-HIVE-Build
> Basic compiler support for Probe MapJoin
>
>
> Key: HIVE-23006
> URL: https://issues.apache.org/jira/browse/HIVE-23006
> Project: Hive
> Issue Type: Sub-task
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23006.01.patch, HIVE-23006.02.patch,
> HIVE-23006.03.patch, HIVE-23006.04.patch, HIVE-23006.05.patch,
> HIVE-23006.06.patch, HIVE-23006.07.patch, HIVE-23006.08.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> The decision of pushing down information to the Record reader (potentially
> reducing decoding time by row-level filtering) should be done at query
> compilation time.
> This patch adds an extra optimisation step with the goal of finding Table
> Scan operators that could reduce the number of rows decoded at runtime using
> extra available information.
> It currently looks for all the available MapJoin operators that could use the
> smaller HashTable on the probing side (where TS is) to filter-out rows that
> would never match.
> To do so the HashTable information is pushed down to the TS properties and
> then propagated as part of MapWork.
> If the a single TS is used by multiple operators (shared-word), this rule can
> not be applied.
> This rule can be extended to support static filter expressions like:
> _select * from sales where sold_state = 'PR';_
> This optimisation manly targets the Tez execution engine running on Llap.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[ https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Pintér updated HIVE-23107: - Attachment: HIVE-23107.12.patch > Remove MIN_HISTORY_LEVEL table > -- > > Key: HIVE-23107 > URL: https://issues.apache.org/jira/browse/HIVE-23107 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Pintér >Assignee: László Pintér >Priority: Major > Attachments: HIVE-23107.01.patch, HIVE-23107.02.patch, > HIVE-23107.03.patch, HIVE-23107.04.patch, HIVE-23107.05.patch, > HIVE-23107.06.patch, HIVE-23107.07.patch, HIVE-23107.08.patch, > HIVE-23107.09.patch, HIVE-23107.10.patch, HIVE-23107.11.patch, > HIVE-23107.12.patch > > > MIN_HISTORY_LEVEL table is used in two places: > * Cleaner uses it to decide if the files can be removed - this could be > replaced by adding a new column to compaction_queue storing the next_txn_id > when the change was committed, and before cleaning checking the minimum open > transaction id in the TXNS table > * Initiator uses it to decide if some items from TXN_TO_WRITE_ID table can > be removed. This could be replaced by using the WRITE_SET.WS_COMMIT_ID. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082247#comment-17082247
]
Hive QA commented on HIVE-23058:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 37 unchanged - 0 fixed
= 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 5 new + 127
unchanged - 0 fixed = 132 total (was 127) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21601/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21601/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21601/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21601/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch,
> HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs when compaction attempt is relaunched after first task attempt
> failure due to preemption by Scheduler or any other reason.
> Since _tmp directory was created by first attempt and
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: (was: HIVE-23058.3.patch) > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[ https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi updated HIVE-23058: Attachment: HIVE-23058.3.patch > Compaction task reattempt fails with FileAlreadyExistsException > --- > > Key: HIVE-23058 > URL: https://issues.apache.org/jira/browse/HIVE-23058 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch, > HIVE_23058.1.patch, HIVE_23058.patch > > > Issue occurs when compaction attempt is relaunched after first task attempt > failure due to preemption by Scheduler or any other reason. > Since _tmp directory was created by first attempt and was left uncleaned > after task attempt failure. Second attempt of the the task fails with > "FileAlreadyExistsException" exception. > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): > > /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5 > for client 10.xx.xx.xxx already exists -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21603) Java 11 preparation: update powermock version
[ https://issues.apache.org/jira/browse/HIVE-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-21603: -- Attachment: HIVE-21603.04.patch > Java 11 preparation: update powermock version > - > > Key: HIVE-21603 > URL: https://issues.apache.org/jira/browse/HIVE-21603 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: László Pintér >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21603.01.patch, HIVE-21603.02.patch, > HIVE-21603.03.patch, HIVE-21603.04.patch > > Time Spent: 20m > Remaining Estimate: 0h > > PowerMock1 has no support for Java11, therefore we need to bump its version > to 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082260#comment-17082260
]
Hive QA commented on HIVE-23058:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999754/HIVE-23058.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18217 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21601/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21601/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21601/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999754 - PreCommit-HIVE-Build
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch,
> HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs when compaction attempt is relaunched after first task attempt
> failure due to preemption by Scheduler or any other reason.
> Since _tmp directory was created by first attempt and was left uncleaned
> after task attempt failure. Second attempt of the the task fails with
> "FileAlreadyExistsException" exception.
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException):
>
> /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5
> for client 10.xx.xx.xxx already exists
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082298#comment-17082298
]
Hive QA commented on HIVE-22320:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
53s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/util: The patch generated 1 new + 44 unchanged
- 0 fixed = 45 total (was 44) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21603/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21603/yetus/diff-checkstyle-itests_util.txt
|
| modules | C: itests/util U: itests/util |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21603/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23186) Strict Check SemanticException Should Properly Quote Table Name
[
https://issues.apache.org/jira/browse/HIVE-23186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23186:
-
> Strict Check SemanticException Should Properly Quote Table Name
> ---
>
> Key: HIVE-23186
> URL: https://issues.apache.org/jira/browse/HIVE-23186
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> https://github.com/apache/hive/blob/029cab297a9ae40d249f63040721f93857398648/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java#L191-L192
> {code:java}
> throw new SemanticException(error + " No partition predicate for
> Alias \""
> + alias + "\" Table \"" + tab.getTableName() + "\"");
> {code}
> Use back ticks and use the database name as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23176) Remove SELECT REGEX Column Feature
[ https://issues.apache.org/jira/browse/HIVE-23176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23176: -- Attachment: HIVE-23176.4.patch > Remove SELECT REGEX Column Feature > -- > > Key: HIVE-23176 > URL: https://issues.apache.org/jira/browse/HIVE-23176 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23176.1.patch, HIVE-23176.2.patch, > HIVE-23176.3.patch, HIVE-23176.4.patch > > > Remove the Hive feature: REGEX Column. > > Hive has this interesting feature for doing REGEX to SELECT multiple columns. > This needs to go. It is not SQL standard and as currently implemented, it > is impossible to determine if a column identifier is a REGEX or the actual > name of the column. If a column name is enclosed in back ticks then any > UTF-8 character is a valid table name. > > [https://dev.mysql.com/doc/refman/8.0/en/identifiers.html] > [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23183:
--
Attachment: HIVE-23183.1.patch
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch, HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22320) Cluster and fs type settings can be replaced with a single minicluster setting in CliConfigs
[
https://issues.apache.org/jira/browse/HIVE-22320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082331#comment-17082331
]
Hive QA commented on HIVE-22320:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999757/HIVE-22320.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17404 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=207)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21603/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21603/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21603/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999757 - PreCommit-HIVE-Build
> Cluster and fs type settings can be replaced with a single minicluster
> setting in CliConfigs
>
>
> Key: HIVE-22320
> URL: https://issues.apache.org/jira/browse/HIVE-22320
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Minor
> Labels: newbie
> Fix For: 4.0.0
>
> Attachments: HIVE-22320.01.patch
>
>
> Fs + cluster types are already linked together in MiniClusterType, so it's
> confusing to set them in 2 steps, separately, if it covers an already defined
> minicluster type.
> https://github.com/apache/hive/blob/master/itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> {code}
> setClusterType(MiniClusterType.TEZ);
> setFsType(QTestMiniClusters.FsType.HDFS);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23187) Make TABLE Token Optional in ANALYZE Statement
[ https://issues.apache.org/jira/browse/HIVE-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23187: - > Make TABLE Token Optional in ANALYZE Statement > -- > > Key: HIVE-23187 > URL: https://issues.apache.org/jira/browse/HIVE-23187 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23187) Make TABLE Token Optional in ANALYZE Statement
[
https://issues.apache.org/jira/browse/HIVE-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23187:
--
Description:
{code:none}
ANALYZE [TABLE] Table1 PARTITION(ds='2008-04-09', hr=11) COMPUTE STATISTICS;
{code}
> Make TABLE Token Optional in ANALYZE Statement
> --
>
> Key: HIVE-23187
> URL: https://issues.apache.org/jira/browse/HIVE-23187
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:none}
> ANALYZE [TABLE] Table1 PARTITION(ds='2008-04-09', hr=11) COMPUTE STATISTICS;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23188) Allow STATS Token in Analyze Table
[ https://issues.apache.org/jira/browse/HIVE-23188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23188: - > Allow STATS Token in Analyze Table > -- > > Key: HIVE-23188 > URL: https://issues.apache.org/jira/browse/HIVE-23188 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23188) Allow STATS Token in Analyze Table
[
https://issues.apache.org/jira/browse/HIVE-23188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23188:
--
Description:
{code:none}
ANALYZE TABLE Table1 PARTITION(ds='2008-04-09', hr=11) COMPUTE
[STATISTICS|STATS];
{code}
Save a few keyboard strokes.
> Allow STATS Token in Analyze Table
> --
>
> Key: HIVE-23188
> URL: https://issues.apache.org/jira/browse/HIVE-23188
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:none}
> ANALYZE TABLE Table1 PARTITION(ds='2008-04-09', hr=11) COMPUTE
> [STATISTICS|STATS];
> {code}
> Save a few keyboard strokes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082362#comment-17082362
]
David Mollitor commented on HIVE-21354:
---
[~pvary] I'm not sure on the exact relationship between table and partition. I
believe they overlap in some meta data, but maybe not all? There might be an
issue of:
* Client 1: Read partition 'a'
* Client 2: Change the table-level meta data
* Client 1: Read partition 'b'
.. but I don't know
Doing a 'DROP' makes sense to lock just the partition... whatever the meta data
change might be is irrelevant because... well,... it's going to be dropped
anyway.
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082368#comment-17082368
]
Hive QA commented on HIVE-23107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
19s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 666 unchanged - 2 fixed = 666 total (was 668) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
24s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 187 unchanged - 3 fixed = 188 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findMinOpenTxnIdForCleaner()
may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at CompactionTxnHandler.java:up java.sql.ResultSet Obligation to clean
up resource created at CompactionTxnHandler.java:[line 1137] is not discharged |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21604/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21604/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21604/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove MIN_HISTORY_LEVEL table
> --
>
> Key: HIVE-23107
> URL: https://issues.apache.org/jira/browse/HIVE-23107
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Pintér
>
[jira] [Assigned] (HIVE-23189) Change Explain ANALYZE to Explain PROFILE
[
https://issues.apache.org/jira/browse/HIVE-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23189:
-
> Change Explain ANALYZE to Explain PROFILE
> -
>
> Key: HIVE-23189
> URL: https://issues.apache.org/jira/browse/HIVE-23189
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> {code:none}
> EXPLAIN
> [EXTENDED|CBO|AST|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION|ANALYZE] query
> {code}
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-TheANALYZEClause
> In Hive, there is an {{EXPLAIN ANALYZE}} query. This can get a bit confusing
> because you can run an {{EXPLAIN ANALYZE}} against an {{ANALYZE TABLE}}
> statement, so you have something like,...
> {code:sql}
> EXPLAIN ANALYZE ANALYZE TABLE `myTable` COMPUTE STATISTICS;
> {code}
> I would like to propose that the name be changed to {{EXPLAIN PROFILE}}.
> This borrows from Apache Impala because it has a {{PROFILE}} command which
> produces the stats that actually occurred during the query run (much like
> this Hive feature).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23189) Change Explain ANALYZE to Explain PROFILE
[
https://issues.apache.org/jira/browse/HIVE-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23189:
--
Target Version/s: 4.0.0
> Change Explain ANALYZE to Explain PROFILE
> -
>
> Key: HIVE-23189
> URL: https://issues.apache.org/jira/browse/HIVE-23189
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> {code:none}
> EXPLAIN
> [EXTENDED|CBO|AST|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION|ANALYZE] query
> {code}
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-TheANALYZEClause
> In Hive, there is an {{EXPLAIN ANALYZE}} query. This can get a bit confusing
> because you can run an {{EXPLAIN ANALYZE}} against an {{ANALYZE TABLE}}
> statement, so you have something like,...
> {code:sql}
> EXPLAIN ANALYZE ANALYZE TABLE `myTable` COMPUTE STATISTICS;
> {code}
> I would like to propose that the name be changed to {{EXPLAIN PROFILE}}.
> This borrows from Apache Impala because it has a {{PROFILE}} command which
> produces the stats that actually occurred during the query run (much like
> this Hive feature).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23187) Make TABLE Token Optional in ANALYZE Statement
[
https://issues.apache.org/jira/browse/HIVE-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082400#comment-17082400
]
David Mollitor commented on HIVE-23187:
---
This is not possible right now because of [HIVE-23187] leads to some confusion
for the parser when the TABLE token is made optional.
> Make TABLE Token Optional in ANALYZE Statement
> --
>
> Key: HIVE-23187
> URL: https://issues.apache.org/jira/browse/HIVE-23187
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:none}
> ANALYZE [TABLE] Table1 PARTITION(ds='2008-04-09', hr=11) COMPUTE STATISTICS;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23190) LLAP: modify IndexCache to pass filesystem object to TezSpillRecord
[ https://issues.apache.org/jira/browse/HIVE-23190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23190: Description: This ticket is about to follow TEZ-4145's changes in Hive's copy of IndexCache class. > LLAP: modify IndexCache to pass filesystem object to TezSpillRecord > --- > > Key: HIVE-23190 > URL: https://issues.apache.org/jira/browse/HIVE-23190 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Priority: Major > > This ticket is about to follow TEZ-4145's changes in Hive's copy of > IndexCache class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23190) LLAP: modify IndexCache to pass filesystem object to TezSpillRecord
[ https://issues.apache.org/jira/browse/HIVE-23190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-23190: --- Assignee: László Bodor > LLAP: modify IndexCache to pass filesystem object to TezSpillRecord > --- > > Key: HIVE-23190 > URL: https://issues.apache.org/jira/browse/HIVE-23190 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > > This ticket is about making the changes introduced in TEZ-4145 in Hive's copy > of IndexCache class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082407#comment-17082407
]
Hive QA commented on HIVE-23107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999766/HIVE-23107.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17404 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21604/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21604/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21604/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999766 - PreCommit-HIVE-Build
> Remove MIN_HISTORY_LEVEL table
> --
>
> Key: HIVE-23107
> URL: https://issues.apache.org/jira/browse/HIVE-23107
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-23107.01.patch, HIVE-23107.02.patch,
> HIVE-23107.03.patch, HIVE-23107.04.patch, HIVE-23107.05.patch,
> HIVE-23107.06.patch, HIVE-23107.07.patch, HIVE-23107.08.patch,
> HIVE-23107.09.patch, HIVE-23107.10.patch, HIVE-23107.11.patch,
> HIVE-23107.12.patch
>
>
> MIN_HISTORY_LEVEL table is used in two places:
> * Cleaner uses it to decide if the files can be removed - this could be
> replaced by adding a new column to compaction_queue storing the next_txn_id
> when the change was committed, and before cleaning checking the minimum open
> transaction id in the TXNS table
> * Initiator uses it to decide if some items from TXN_TO_WRITE_ID table can
> be removed. This could be replaced by using the WRITE_SET.WS_COMMIT_ID.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23190) LLAP: modify IndexCache to pass filesystem object to TezSpillRecord
[ https://issues.apache.org/jira/browse/HIVE-23190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23190: Description: This ticket is about making the changes introduced in TEZ-4145 in Hive's copy of IndexCache class. (was: This ticket is about to follow TEZ-4145's changes in Hive's copy of IndexCache class.) > LLAP: modify IndexCache to pass filesystem object to TezSpillRecord > --- > > Key: HIVE-23190 > URL: https://issues.apache.org/jira/browse/HIVE-23190 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Priority: Major > > This ticket is about making the changes introduced in TEZ-4145 in Hive's copy > of IndexCache class. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23173) User login success/failed attempts should be logged
[
https://issues.apache.org/jira/browse/HIVE-23173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082425#comment-17082425
]
Naresh P R commented on HIVE-23173:
---
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
Testcode is failing randomly with below error
{code:java}
java.lang.AssertionError: expected:<6> but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hive.ql.parse.WarehouseInstance.verifyResults(WarehouseInstance.java:395){code}
Because they have scheduler & same table name on both the testcase. So changed
the table names to be unique per testcase.
> User login success/failed attempts should be logged
> ---
>
> Key: HIVE-23173
> URL: https://issues.apache.org/jira/browse/HIVE-23173
> Project: Hive
> Issue Type: Improvement
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Minor
> Attachments: HIVE-23173.1.patch, HIVE-23173.2.patch,
> HIVE-23173.3.patch
>
>
> User login success & failure attempts should be logged in server logs
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082440#comment-17082440
]
Peter Vary commented on HIVE-21354:
---
[~belugabehr]: My understanding is that having a lock on a partition
automatically locks the table (even without further, table specific lock), and
having a lock on a table prevents further conflicting locks on any partition of
the given table by a different query.
This means that the extra table level lock is not only unnecessary, but it also
prevents parallelism which should be allowed. (DROP PARTITION for p1, INSERT
INTO p2)
The question is whether this code is used for legacy locks as well which might
have different logic. (Most probably not)
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082449#comment-17082449
]
Hive QA commented on HIVE-23058:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 37 unchanged - 0 fixed
= 38 total (was 37) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} itests/hive-unit: The patch generated 5 new + 127
unchanged - 0 fixed = 132 total (was 127) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21605/dev-support/hive-personality.sh
|
| git revision | master / 8259a21 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21605/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21605/yetus/diff-checkstyle-itests_hive-unit.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21605/yetus/whitespace-eol.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21605/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch,
> HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs
[jira] [Updated] (HIVE-23073) Shade netty and upgrade to netty 4.1.48.Final
[ https://issues.apache.org/jira/browse/HIVE-23073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23073: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Laszlo! > Shade netty and upgrade to netty 4.1.48.Final > - > > Key: HIVE-23073 > URL: https://issues.apache.org/jira/browse/HIVE-23073 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23073.01.patch, HIVE-23073.02.patch, > HIVE-23073.03.patch, HIVE-23073.04.patch, HIVE-23073.05.patch, > HIVE-23073.06.patch, HIVE-23073.06.patch, HIVE-23073.06.patch, > HIVE-23073.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23006) Basic compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082458#comment-17082458 ] Jesus Camacho Rodriguez commented on HIVE-23006: +1 > Basic compiler support for Probe MapJoin > > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23006.01.patch, HIVE-23006.02.patch, > HIVE-23006.03.patch, HIVE-23006.04.patch, HIVE-23006.05.patch, > HIVE-23006.06.patch, HIVE-23006.07.patch, HIVE-23006.08.patch > > Time Spent: 3h > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23184) Upgrade druid to 0.17.1
[ https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-23184: Attachment: HIVE-23184.1.patch > Upgrade druid to 0.17.1 > --- > > Key: HIVE-23184 > URL: https://issues.apache.org/jira/browse/HIVE-23184 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23184.1.patch, HIVE-23184.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade to druid latest release 0.17.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23058) Compaction task reattempt fails with FileAlreadyExistsException
[
https://issues.apache.org/jira/browse/HIVE-23058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082487#comment-17082487
]
Hive QA commented on HIVE-23058:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999771/HIVE-23058.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17405 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21605/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21605/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21605/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999771 - PreCommit-HIVE-Build
> Compaction task reattempt fails with FileAlreadyExistsException
> ---
>
> Key: HIVE-23058
> URL: https://issues.apache.org/jira/browse/HIVE-23058
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Riju Trivedi
>Assignee: Riju Trivedi
>Priority: Major
> Attachments: HIVE-23058.2.patch, HIVE-23058.3.patch,
> HIVE_23058.1.patch, HIVE_23058.patch
>
>
> Issue occurs when compaction attempt is relaunched after first task attempt
> failure due to preemption by Scheduler or any other reason.
> Since _tmp directory was created by first attempt and was left uncleaned
> after task attempt failure. Second attempt of the the task fails with
> "FileAlreadyExistsException" exception.
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException):
>
> /warehouse/tablespace/managed/hive/default.db/compaction_test/_tmp_3670bbef-ba7a-4c10-918d-9a2ee17cbd22/base_186/bucket_5
> for client 10.xx.xx.xxx already exists
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23184) Upgrade druid to 0.17.1
[ https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-23184: Attachment: (was: HIVE-23184.1.patch) > Upgrade druid to 0.17.1 > --- > > Key: HIVE-23184 > URL: https://issues.apache.org/jira/browse/HIVE-23184 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23184.1.patch, HIVE-23184.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade to druid latest release 0.17.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23184) Upgrade druid to 0.17.1
[ https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-23184: Attachment: HIVE-23184.1.patch > Upgrade druid to 0.17.1 > --- > > Key: HIVE-23184 > URL: https://issues.apache.org/jira/browse/HIVE-23184 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23184.1.patch, HIVE-23184.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade to druid latest release 0.17.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21603) Java 11 preparation: update powermock version
[
https://issues.apache.org/jira/browse/HIVE-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082488#comment-17082488
]
Hive QA commented on HIVE-21603:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999774/HIVE-21603.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21606/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21606/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21606/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-04-13 16:59:59.056
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-21606/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-04-13 16:59:59.059
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
8259a21..c1836f0 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 8259a21 HIVE-23178 : Add Tez Total Order Partitioner (Roohi
Syeda via Gopal V)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at c1836f0 HIVE-23073 : Shade netty and upgrade to netty
4.1.48.Final (Laszlo Bodor via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-04-13 17:00:00.368
+ rm -rf ../yetus_PreCommit-HIVE-Build-21606
+ mkdir ../yetus_PreCommit-HIVE-Build-21606
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-21606
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21606/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed: pom.xml:193
Falling back to three-way merge...
Applied patch to 'pom.xml' cleanly.
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java:49
Falling back to three-way merge...
Applied patch to
'ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java' with
conflicts.
Going to apply patch with: git apply -p0
error: patch failed: pom.xml:193
Falling back to three-way merge...
Applied patch to 'pom.xml' cleanly.
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java:49
Falling back to three-way merge...
Applied patch to
'ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java' with
conflicts.
U ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-21606
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999774 - PreCommit-HIVE-Build
> Java 11 preparation: update powermock version
> -
>
> Key: HIVE-21603
> URL: https://issues.apache.org/jira/browse/HIVE-21603
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: László Pintér
>Assignee: Panagiotis Garefalakis
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21603.01.patch, HIVE-21603.02.patch,
> HIVE-21603.03.patch, HIVE-21603.04.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> PowerMock1 has no support for Java11, therefore we need to bump its version
> to 2.0.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23006) Basic compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?focusedWorklogId=421434&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-421434 ] ASF GitHub Bot logged work on HIVE-23006: - Author: ASF GitHub Bot Created on: 13/Apr/20 17:01 Start Date: 13/Apr/20 17:01 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #952: HIVE-23006 ProbeDecode compiler support URL: https://github.com/apache/hive/pull/952 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 421434) Time Spent: 3h 10m (was: 3h) > Basic compiler support for Probe MapJoin > > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23006.01.patch, HIVE-23006.02.patch, > HIVE-23006.03.patch, HIVE-23006.04.patch, HIVE-23006.05.patch, > HIVE-23006.06.patch, HIVE-23006.07.patch, HIVE-23006.08.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23006) Basic compiler support for Probe MapJoin
[ https://issues.apache.org/jira/browse/HIVE-23006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23006: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~pgaref]! > Basic compiler support for Probe MapJoin > > > Key: HIVE-23006 > URL: https://issues.apache.org/jira/browse/HIVE-23006 > Project: Hive > Issue Type: Sub-task >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23006.01.patch, HIVE-23006.02.patch, > HIVE-23006.03.patch, HIVE-23006.04.patch, HIVE-23006.05.patch, > HIVE-23006.06.patch, HIVE-23006.07.patch, HIVE-23006.08.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > The decision of pushing down information to the Record reader (potentially > reducing decoding time by row-level filtering) should be done at query > compilation time. > This patch adds an extra optimisation step with the goal of finding Table > Scan operators that could reduce the number of rows decoded at runtime using > extra available information. > It currently looks for all the available MapJoin operators that could use the > smaller HashTable on the probing side (where TS is) to filter-out rows that > would never match. > To do so the HashTable information is pushed down to the TS properties and > then propagated as part of MapWork. > If the a single TS is used by multiple operators (shared-word), this rule can > not be applied. > This rule can be extended to support static filter expressions like: > _select * from sales where sold_state = 'PR';_ > This optimisation manly targets the Tez execution engine running on Llap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082501#comment-17082501
]
David Mollitor edited comment on HIVE-21354 at 4/13/20, 5:15 PM:
-
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this this a a
simple way... it comes up with a list of all the required locks and the first
one is always the table lock, the rest are the required partitions. That is to
say, it takes an explicit lock on the table,... there is no logic for an
implicit table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
was (Author: belugabehr):
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this this a a
simple way... it comes up with a list of all the required locks and the first
one is always the table lock, the rest are the required partitions. That is to
say, it takes an explicit lock on the table,... there is no logic for an
implicit table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082501#comment-17082501
]
David Mollitor commented on HIVE-21354:
---
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this this a a
simple way... it comes up with a list of all the required locks and the first
one is always the table lock, the rest are the required partitions. That is to
say, it takes an explicit lock on the table,... there is no logic for an
implicit table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082501#comment-17082501
]
David Mollitor edited comment on HIVE-21354 at 4/13/20, 5:16 PM:
-
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this in a simple
way... it comes up with a list of all the required locks and the first one is
always the table lock, the rest are the required partitions. That is to say,
it takes an explicit lock on the table,... there is no logic for an implicit
table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
was (Author: belugabehr):
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this this a a
simple way... it comes up with a list of all the required locks and the first
one is always the table lock, the rest are the required partitions. That is to
say, it takes an explicit lock on the table,... there is no logic for an
implicit table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082501#comment-17082501
]
David Mollitor edited comment on HIVE-21354 at 4/13/20, 5:18 PM:
-
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this in a simple
way... it comes up with a list of all the required locks and the first one is
always the table lock, the rest are the required partitions. That is to say,
it takes an explicit lock on the table,... there is no logic for an implicit
table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
What I would expect is, for an INSERT into a specific partition, or TRUNCATE
partition statement, those queries would take a SHARED_READ lock on the
table-level and an EXCLUSIVE lock on the specific partitions.
was (Author: belugabehr):
[~pvary] I do not think that Hive has any logic that says "if a partition of a
table is locked, then the table is locked." I think it does this in a simple
way... it comes up with a list of all the required locks and the first one is
always the table lock, the rest are the required partitions. That is to say,
it takes an explicit lock on the table,... there is no logic for an implicit
table lock:
{code:none}
EXPLAIN LOCKS SELECT * FROM web_logs;
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> SHARED_READ
default.web_logs.date=2015-11-19 -> SHARED_READ
default.web_logs.date=2015-11-20 -> SHARED_READ
default.web_logs.date=2015-11-21 -> SHARED_READ
{code}
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082505#comment-17082505
]
David Mollitor commented on HIVE-21354:
---
... something like:
{code:none}
explain locks alter table web_logs drop partition(`date`='2015-11-18')
LOCK INFORMATION:
default.web_logs -> SHARED_READ
default.web_logs.date=2015-11-18 -> EXCLUSIVE
{code}
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23137: -- Attachment: (was: HIVE-23137.01.patch) > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082513#comment-17082513 ] Miklos Gergely commented on HIVE-23137: --- Spark tests are removed. > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23137) Disable spark tests at precommit
[ https://issues.apache.org/jira/browse/HIVE-23137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23137: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Disable spark tests at precommit > > > Key: HIVE-23137 > URL: https://issues.apache.org/jira/browse/HIVE-23137 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Spark is getting less and less widely supported and actively developed > execution engine mode by the community. Therefore running the tests for it in > every precommit is not reasonable. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23158) Optimize S3A recordReader policy for Random IO formats
[ https://issues.apache.org/jira/browse/HIVE-23158?focusedWorklogId=421462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-421462 ] ASF GitHub Bot logged work on HIVE-23158: - Author: ASF GitHub Bot Created on: 13/Apr/20 17:36 Start Date: 13/Apr/20 17:36 Worklog Time Spent: 10m Work Description: pgaref commented on pull request #972: HIVE-23158 initial patch URL: https://github.com/apache/hive/pull/972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 421462) Time Spent: 20m (was: 10m) > Optimize S3A recordReader policy for Random IO formats > -- > > Key: HIVE-23158 > URL: https://issues.apache.org/jira/browse/HIVE-23158 > Project: Hive > Issue Type: Bug >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-23158.01.patch > > Time Spent: 20m > Remaining Estimate: 0h > > S3A filesystem client (inherited by Hadoop) supports the notion of input > policies. > These policies tune the behaviour of HTTP requests that are used for reading > different filetypes such as TEXT or ORC. > For formats such as ORC and Parquet that do a lot of seek operations, there > is an optimized RANDOM mode that reads files only partially instead of fully > (default). > I am suggesting to add some extra logic as part of HiveInputFormat to make > sure we optimize RecordReader requests for random IO when data is stored on > S3A using formats such as ORC or Parquet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23191) Prevent redundant output descriptor config serialization
[
https://issues.apache.org/jira/browse/HIVE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman reassigned HIVE-23191:
---
> Prevent redundant output descriptor config serialization
>
>
> Key: HIVE-23191
> URL: https://issues.apache.org/jira/browse/HIVE-23191
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
>
> {code:java}
> DagUtils#createVertex(JobConf, BaseWork, Path,
> TezWork, Map){code}
> creates an output descriptor if it is leaf vertex. It uses the same config
> object that is used in processor descriptor. It should not create payload
> from scratch when processor descriptor has the identical payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23191) Prevent redundant output descriptor config serialization
[
https://issues.apache.org/jira/browse/HIVE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mustafa Iman updated HIVE-23191:
Attachment: HIVE-23191.patch
Status: Patch Available (was: Open)
> Prevent redundant output descriptor config serialization
>
>
> Key: HIVE-23191
> URL: https://issues.apache.org/jira/browse/HIVE-23191
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23191.patch
>
>
> {code:java}
> DagUtils#createVertex(JobConf, BaseWork, Path,
> TezWork, Map){code}
> creates an output descriptor if it is leaf vertex. It uses the same config
> object that is used in processor descriptor. It should not create payload
> from scratch when processor descriptor has the identical payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23163) Class TrustDomainAuthenticationTest should be abstract
[ https://issues.apache.org/jira/browse/HIVE-23163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082542#comment-17082542 ] Chinna Rao Lalam commented on HIVE-23163: - +1 > Class TrustDomainAuthenticationTest should be abstract > -- > > Key: HIVE-23163 > URL: https://issues.apache.org/jira/browse/HIVE-23163 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Yikun Jiang >Priority: Major > Attachments: HIVE-23163.1.patch, HIVE-23163.2.patch, > HIVE-23163.3.patch > > > When running tests in pre-commit CI, the test parser will only identify test > classes start with 'Test' > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestPropertiesParser.java#L406] > But when running using `mvn test`, it will also parse classes such as > '*Test.java': > [http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/inclusion-exclusion.html] > so for: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TrustDomainAuthenticationTest.java#L38] > it will also be included in the test, for example: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/TrustDomainAuthenticationTest/testTrustedDomainAuthentication/] > This is because that class TrustDomainAuthenticationTest is actually a parent > class that does not have the parameters for init. The actual tests are its > children classes like the rest in > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/] > they will pass actuall parameters for init: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TestTrustDomainAuthenticationBinary.java#L26] > > we can make this class abstract so that it won't be included when running > 'mvn test', like > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractJdbcTriggersTest.java#L54] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23176) Remove SELECT REGEX Column Feature
[
https://issues.apache.org/jira/browse/HIVE-23176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082558#comment-17082558
]
Hive QA commented on HIVE-23176:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
58s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch parser passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} ql: The patch generated 0 new + 475 unchanged - 1
fixed = 475 total (was 476) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} root: The patch generated 0 new + 475 unchanged - 1
fixed = 475 total (was 476) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21607/dev-support/hive-personality.sh
|
| git revision | master / d45ac51 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: parser ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21607/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove SELECT REGEX Column Feature
> --
>
> Key: HIVE-23176
> URL: https://issues.apache.org/jira/browse/HIVE-23176
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23176.1.patch, HIVE-23176.2.patch,
> HIVE-23176.3.patch, HIVE-23176.4.patch
>
>
> Remove the Hive feature: REGEX Column.
>
> Hive has this interesting feature for doing REGEX to SELECT multiple columns.
> This needs to go. It is not SQL standard and as currently implemented, it
> is impossible to determine if a column identifier is a REGEX or the actual
> name of the column. If a column name is enclosed in back ticks then any
> UTF-8 characte
[jira] [Commented] (HIVE-23176) Remove SELECT REGEX Column Feature
[
https://issues.apache.org/jira/browse/HIVE-23176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082565#comment-17082565
]
Hive QA commented on HIVE-23176:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999781/HIVE-23176.4.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17400 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21607/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21607/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21607/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999781 - PreCommit-HIVE-Build
> Remove SELECT REGEX Column Feature
> --
>
> Key: HIVE-23176
> URL: https://issues.apache.org/jira/browse/HIVE-23176
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23176.1.patch, HIVE-23176.2.patch,
> HIVE-23176.3.patch, HIVE-23176.4.patch
>
>
> Remove the Hive feature: REGEX Column.
>
> Hive has this interesting feature for doing REGEX to SELECT multiple columns.
> This needs to go. It is not SQL standard and as currently implemented, it
> is impossible to determine if a column identifier is a REGEX or the actual
> name of the column. If a column name is enclosed in back ticks then any
> UTF-8 character is a valid table name.
>
> [https://dev.mysql.com/doc/refman/8.0/en/identifiers.html]
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23163) Class TrustDomainAuthenticationTest should be abstract
[ https://issues.apache.org/jira/browse/HIVE-23163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-23163: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) HIVE-23163.3.patch committed to master. Thank you for the patch Yikun Jiang. > Class TrustDomainAuthenticationTest should be abstract > -- > > Key: HIVE-23163 > URL: https://issues.apache.org/jira/browse/HIVE-23163 > Project: Hive > Issue Type: Bug >Reporter: Zhenyu Zheng >Assignee: Yikun Jiang >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23163.1.patch, HIVE-23163.2.patch, > HIVE-23163.3.patch > > > When running tests in pre-commit CI, the test parser will only identify test > classes start with 'Test' > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/testutils/ptest2/src/main/java/org/apache/hive/ptest/execution/conf/UnitTestPropertiesParser.java#L406] > But when running using `mvn test`, it will also parse classes such as > '*Test.java': > [http://maven.apache.org/plugins-archives/maven-surefire-plugin-2.12.4/examples/inclusion-exclusion.html] > so for: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TrustDomainAuthenticationTest.java#L38] > it will also be included in the test, for example: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/TrustDomainAuthenticationTest/testTrustedDomainAuthentication/] > This is because that class TrustDomainAuthenticationTest is actually a parent > class that does not have the parameters for init. The actual tests are its > children classes like the rest in > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/23/testReport/junit/org.apache.hive.service.auth/] > they will pass actuall parameters for init: > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/service/auth/TestTrustDomainAuthenticationBinary.java#L26] > > we can make this class abstract so that it won't be included when running > 'mvn test', like > [https://github.com/apache/hive/blob/d2163cbfb8bacf859fa8572e24c8533bb2dcb0f3/itests/hive-unit/src/test/java/org/apache/hive/jdbc/AbstractJdbcTriggersTest.java#L54] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082571#comment-17082571
]
Peter Vary commented on HIVE-21354:
---
[~belugabehr]: AFAIK checkLock checks locks hierarchically. So it prevents
aquiring table level exclusive lock if a partition level exclusive lock is
already acquired by another query.
Explicitly mentioning table level shared lock like you suggested in your
comment for drop partition is philosophical question IMHO if the handling of
the partition level lock already prevents exclusive lock on a table anyway.
With the current implementation the performance is better with fewer number of
locks.
Having extra table level lock when querying from a single partition is a bug if
the table level lock prevents dropping other partitions, which is not a desired
behaviour.
So it all comes down if the lock check does exact matches, or it checks stuff
hierarchically.
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23184) Upgrade druid to 0.17.1
[ https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082572#comment-17082572 ] Nishant Bangarwa commented on HIVE-23184: - +cc [~ashutoshc] Please review. > Upgrade druid to 0.17.1 > --- > > Key: HIVE-23184 > URL: https://issues.apache.org/jira/browse/HIVE-23184 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23184.1.patch, HIVE-23184.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade to druid latest release 0.17.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082575#comment-17082575
]
Hive QA commented on HIVE-23183:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21608/dev-support/hive-personality.sh
|
| git revision | master / 2bf957f |
| modules | C: parser ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21608/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch, HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082578#comment-17082578
]
David Mollitor commented on HIVE-21354:
---
bq. So it all comes down if the lock check does exact matches, or it checks
stuff hierarchically.
Yes. Exactly :)
I think we are just both guessing on which one is employed. I will need to dig
in to figure it out, unless you can point me at the code that does this
implicit locking check.
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23176) Remove SELECT REGEX Column Feature
[ https://issues.apache.org/jira/browse/HIVE-23176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23176: -- Attachment: HIVE-23176.4.patch > Remove SELECT REGEX Column Feature > -- > > Key: HIVE-23176 > URL: https://issues.apache.org/jira/browse/HIVE-23176 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23176.1.patch, HIVE-23176.2.patch, > HIVE-23176.3.patch, HIVE-23176.4.patch, HIVE-23176.4.patch > > > Remove the Hive feature: REGEX Column. > > Hive has this interesting feature for doing REGEX to SELECT multiple columns. > This needs to go. It is not SQL standard and as currently implemented, it > is impossible to determine if a column identifier is a REGEX or the actual > name of the column. If a column name is enclosed in back ticks then any > UTF-8 character is a valid table name. > > [https://dev.mysql.com/doc/refman/8.0/en/identifiers.html] > [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority Of Partitions Are Locked
[
https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082586#comment-17082586
]
Peter Vary commented on HIVE-21354:
---
HIVE-22888
> Lock The Entire Table If Majority Of Partitions Are Locked
> --
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism.
> When a Hive query interacts with a table which has a lot of partitions, this
> may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal
> to half the total number of partitions, simply create one ZNode lock at the
> table level.
> This would improve performance of many queries, but in particular, a {{select
> count(1) from table}} ... or ... {{select * from table limit 5}} where the
> table has many partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21603) Java 11 preparation: update powermock version
[ https://issues.apache.org/jira/browse/HIVE-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-21603: -- Attachment: HIVE-21603.05.patch > Java 11 preparation: update powermock version > - > > Key: HIVE-21603 > URL: https://issues.apache.org/jira/browse/HIVE-21603 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: László Pintér >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21603.01.patch, HIVE-21603.02.patch, > HIVE-21603.03.patch, HIVE-21603.04.patch, HIVE-21603.05.patch > > Time Spent: 20m > Remaining Estimate: 0h > > PowerMock1 has no support for Java11, therefore we need to bump its version > to 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23184) Upgrade druid to 0.17.1
[ https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-23184: Attachment: HIVE-23184.2.patch > Upgrade druid to 0.17.1 > --- > > Key: HIVE-23184 > URL: https://issues.apache.org/jira/browse/HIVE-23184 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23184.1.patch, HIVE-23184.2.patch, HIVE-23184.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Upgrade to druid latest release 0.17.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23189) Change Explain ANALYZE to Explain PROFILE
[
https://issues.apache.org/jira/browse/HIVE-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23189:
--
Attachment: HIVE-23189.1.patch
> Change Explain ANALYZE to Explain PROFILE
> -
>
> Key: HIVE-23189
> URL: https://issues.apache.org/jira/browse/HIVE-23189
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23189.1.patch
>
>
> {code:none}
> EXPLAIN
> [EXTENDED|CBO|AST|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION|ANALYZE] query
> {code}
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-TheANALYZEClause
> In Hive, there is an {{EXPLAIN ANALYZE}} query. This can get a bit confusing
> because you can run an {{EXPLAIN ANALYZE}} against an {{ANALYZE TABLE}}
> statement, so you have something like,...
> {code:sql}
> EXPLAIN ANALYZE ANALYZE TABLE `myTable` COMPUTE STATISTICS;
> {code}
> I would like to propose that the name be changed to {{EXPLAIN PROFILE}}.
> This borrows from Apache Impala because it has a {{PROFILE}} command which
> produces the stats that actually occurred during the query run (much like
> this Hive feature).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23189) Change Explain ANALYZE to Explain PROFILE
[
https://issues.apache.org/jira/browse/HIVE-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23189:
--
Status: Patch Available (was: Open)
> Change Explain ANALYZE to Explain PROFILE
> -
>
> Key: HIVE-23189
> URL: https://issues.apache.org/jira/browse/HIVE-23189
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23189.1.patch
>
>
> {code:none}
> EXPLAIN
> [EXTENDED|CBO|AST|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION|ANALYZE] query
> {code}
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-TheANALYZEClause
> In Hive, there is an {{EXPLAIN ANALYZE}} query. This can get a bit confusing
> because you can run an {{EXPLAIN ANALYZE}} against an {{ANALYZE TABLE}}
> statement, so you have something like,...
> {code:sql}
> EXPLAIN ANALYZE ANALYZE TABLE `myTable` COMPUTE STATISTICS;
> {code}
> I would like to propose that the name be changed to {{EXPLAIN PROFILE}}.
> This borrows from Apache Impala because it has a {{PROFILE}} command which
> produces the stats that actually occurred during the query run (much like
> this Hive feature).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082620#comment-17082620
]
Hive QA commented on HIVE-23183:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999782/HIVE-23183.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17406 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_materialized]
(batchId=143)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21608/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21608/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21608/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999782 - PreCommit-HIVE-Build
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch, HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23192) "default" database locationUri should be external warehouse root.
[ https://issues.apache.org/jira/browse/HIVE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23192: > "default" database locationUri should be external warehouse root. > - > > Key: HIVE-23192 > URL: https://issues.apache.org/jira/browse/HIVE-23192 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > When creating the default database, the database locationUri should be set to > external warehouse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23184) Upgrade druid to 0.17.1
[
https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082665#comment-17082665
]
Hive QA commented on HIVE-23184:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} itests/qtest-druid in master has 7 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} druid-handler: The patch generated 0 new + 0
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} root: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch qtest-druid passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21609/dev-support/hive-personality.sh
|
| git revision | master / 2bf957f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql druid-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21609/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade druid to 0.17.1
> ---
>
> Key: HIVE-23184
> URL: https://issues.apache.org/jira/browse/HIVE-23184
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: N
[jira] [Commented] (HIVE-23184) Upgrade druid to 0.17.1
[
https://issues.apache.org/jira/browse/HIVE-23184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082668#comment-17082668
]
Hive QA commented on HIVE-23184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999818/HIVE-23184.2.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17406 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 (batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21609/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21609/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21609/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999818 - PreCommit-HIVE-Build
> Upgrade druid to 0.17.1
> ---
>
> Key: HIVE-23184
> URL: https://issues.apache.org/jira/browse/HIVE-23184
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23184.1.patch, HIVE-23184.2.patch, HIVE-23184.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Upgrade to druid latest release 0.17.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23181) Remove snakeyaml lib from Hive distribution
[ https://issues.apache.org/jira/browse/HIVE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roohi Syeda updated HIVE-23181: --- Attachment: HIVE-23181.01.patch > Remove snakeyaml lib from Hive distribution > --- > > Key: HIVE-23181 > URL: https://issues.apache.org/jira/browse/HIVE-23181 > Project: Hive > Issue Type: Bug >Reporter: Roohi Syeda >Assignee: Roohi Syeda >Priority: Major > Attachments: HIVE-23181.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23191) Prevent redundant output descriptor config serialization
[
https://issues.apache.org/jira/browse/HIVE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082692#comment-17082692
]
Hive QA commented on HIVE-23191:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
51s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21610/dev-support/hive-personality.sh
|
| git revision | master / 2bf957f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21610/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Prevent redundant output descriptor config serialization
>
>
> Key: HIVE-23191
> URL: https://issues.apache.org/jira/browse/HIVE-23191
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23191.patch
>
>
> {code:java}
> DagUtils#createVertex(JobConf, BaseWork, Path,
> TezWork, Map){code}
> creates an output descriptor if it is leaf vertex. It uses the same config
> object that is used in processor descriptor. It should not create payload
> from scratch when processor descriptor has the identical payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23181) Remove snakeyaml lib from Hive distribution
[ https://issues.apache.org/jira/browse/HIVE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roohi Syeda updated HIVE-23181: --- Attachment: (was: HIVE-23181.01.patch) > Remove snakeyaml lib from Hive distribution > --- > > Key: HIVE-23181 > URL: https://issues.apache.org/jira/browse/HIVE-23181 > Project: Hive > Issue Type: Bug >Reporter: Roohi Syeda >Assignee: Roohi Syeda >Priority: Major > Attachments: HIVE-23181.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23181) Remove snakeyaml lib from Hive distribution
[ https://issues.apache.org/jira/browse/HIVE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roohi Syeda updated HIVE-23181: --- Attachment: HIVE-23181.01.patch > Remove snakeyaml lib from Hive distribution > --- > > Key: HIVE-23181 > URL: https://issues.apache.org/jira/browse/HIVE-23181 > Project: Hive > Issue Type: Bug >Reporter: Roohi Syeda >Assignee: Roohi Syeda >Priority: Major > Attachments: HIVE-23181.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23098) Allow Operation assertState to Accept a Collection
[
https://issues.apache.org/jira/browse/HIVE-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082703#comment-17082703
]
David Mollitor commented on HIVE-23098:
---
[~ngangam] [~pvary] [~mgergely] Any chance you got a moment to take a peek at
this?
> Allow Operation assertState to Accept a Collection
> --
>
> Key: HIVE-23098
> URL: https://issues.apache.org/jira/browse/HIVE-23098
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23098.1.patch, HIVE-23098.2.patch,
> HIVE-23098.2.patch, HIVE-23098.2.patch, HIVE-23098.3.patch
>
>
> {code:java|title=Operation.java}
> protected final void assertState(List states) throws
> HiveSQLException {
> if (!states.contains(state)) {
> throw new HiveSQLException("Expected states: " + states.toString() + ",
> but found "
> + this.state);
> }
> this.lastAccessTime = System.currentTimeMillis();
> }
> /*/
> public void someMethod() {
> assertState(new
> ArrayList(Arrays.asList(OperationState.FINISHED)));
> }
> {code}
> By allowing {{assertState}} to accept a {{Collection}}, one can save an
> allocation and simplify the code:
> {code:java}
> assertState(Collections.singleton(OperationState.FINISHED));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23193) Review of Subset of Debug Logging
[
https://issues.apache.org/jira/browse/HIVE-23193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-23193:
-
> Review of Subset of Debug Logging
> -
>
> Key: HIVE-23193
> URL: https://issues.apache.org/jira/browse/HIVE-23193
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> bq. Better yet, use parameterized messages
> bq. Will outperform the first form by a factor of at least 30, in case of a
> disabled logging statement.
> http://www.slf4j.org/faq.html
> * Use parameterized logging where appropriate
> * Add logging guards {{if (Log.isDebugEnabled()}} around loops and complex
> debug message
> Simplify the code, remove lines of code, and potentially increase performance
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23193) Review of Subset of Debug Logging
[
https://issues.apache.org/jira/browse/HIVE-23193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23193:
--
Attachment: HIVE-23193.1.patch
> Review of Subset of Debug Logging
> -
>
> Key: HIVE-23193
> URL: https://issues.apache.org/jira/browse/HIVE-23193
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23193.1.patch
>
>
> bq. Better yet, use parameterized messages
> bq. Will outperform the first form by a factor of at least 30, in case of a
> disabled logging statement.
> http://www.slf4j.org/faq.html
> * Use parameterized logging where appropriate
> * Add logging guards {{if (Log.isDebugEnabled()}} around loops and complex
> debug message
> Simplify the code, remove lines of code, and potentially increase performance
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23193) Review of Subset of Debug Logging
[
https://issues.apache.org/jira/browse/HIVE-23193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23193:
--
Status: Patch Available (was: Open)
> Review of Subset of Debug Logging
> -
>
> Key: HIVE-23193
> URL: https://issues.apache.org/jira/browse/HIVE-23193
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23193.1.patch
>
>
> bq. Better yet, use parameterized messages
> bq. Will outperform the first form by a factor of at least 30, in case of a
> disabled logging statement.
> http://www.slf4j.org/faq.html
> * Use parameterized logging where appropriate
> * Add logging guards {{if (Log.isDebugEnabled()}} around loops and complex
> debug message
> Simplify the code, remove lines of code, and potentially increase performance
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23098) Allow Operation assertState to Accept a Collection
[
https://issues.apache.org/jira/browse/HIVE-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082710#comment-17082710
]
Naveen Gangam commented on HIVE-23098:
--
[~belugabehr] I thought I had +1'ed it last week. I had reviewed the patch. It
looks good to me. +1
> Allow Operation assertState to Accept a Collection
> --
>
> Key: HIVE-23098
> URL: https://issues.apache.org/jira/browse/HIVE-23098
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23098.1.patch, HIVE-23098.2.patch,
> HIVE-23098.2.patch, HIVE-23098.2.patch, HIVE-23098.3.patch
>
>
> {code:java|title=Operation.java}
> protected final void assertState(List states) throws
> HiveSQLException {
> if (!states.contains(state)) {
> throw new HiveSQLException("Expected states: " + states.toString() + ",
> but found "
> + this.state);
> }
> this.lastAccessTime = System.currentTimeMillis();
> }
> /*/
> public void someMethod() {
> assertState(new
> ArrayList(Arrays.asList(OperationState.FINISHED)));
> }
> {code}
> By allowing {{assertState}} to accept a {{Collection}}, one can save an
> allocation and simplify the code:
> {code:java}
> assertState(Collections.singleton(OperationState.FINISHED));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23183) Make TABLE Token Optional in TRUNCATE Statement
[
https://issues.apache.org/jira/browse/HIVE-23183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23183:
--
Attachment: HIVE-23183.1.patch
> Make TABLE Token Optional in TRUNCATE Statement
> ---
>
> Key: HIVE-23183
> URL: https://issues.apache.org/jira/browse/HIVE-23183
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23183.1.patch, HIVE-23183.1.patch,
> HIVE-23183.1.patch
>
>
> {code:none}
> TRUNCATE [TABLE] tbl_name
> {code}
> https://dev.mysql.com/doc/refman/8.0/en/truncate-table.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23029) LLAP: Shuffle Handler should support Index Cache configuration
[ https://issues.apache.org/jira/browse/HIVE-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082715#comment-17082715 ] Ashutosh Chauhan commented on HIVE-23029: - +1 > LLAP: Shuffle Handler should support Index Cache configuration > -- > > Key: HIVE-23029 > URL: https://issues.apache.org/jira/browse/HIVE-23029 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-23029.1.patch, Screenshot 2020-03-16 at 12.08.44 > PM.jpg > > > !Screenshot 2020-03-16 at 12.08.44 PM.jpg|width=1592,height=1112! > > Queries like Q78 at large scale misses index cache with unordered edges. (24 > * 1009 = 24216. With the default 10 MB cache size, it can accommodate only > 400+ entries). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23181) Remove snakeyaml lib from Hive distribution
[ https://issues.apache.org/jira/browse/HIVE-23181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23181: Status: Patch Available (was: Open) > Remove snakeyaml lib from Hive distribution > --- > > Key: HIVE-23181 > URL: https://issues.apache.org/jira/browse/HIVE-23181 > Project: Hive > Issue Type: Bug >Reporter: Roohi Syeda >Assignee: Roohi Syeda >Priority: Major > Attachments: HIVE-23181.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23191) Prevent redundant output descriptor config serialization
[
https://issues.apache.org/jira/browse/HIVE-23191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082719#comment-17082719
]
Hive QA commented on HIVE-23191:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12999810/HIVE-23191.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17406 tests
executed
*Failed tests:*
{noformat}
TestLocalSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
TestSparkNegativeCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=264)
TestSparkPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=264)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21610/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21610/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12999810 - PreCommit-HIVE-Build
> Prevent redundant output descriptor config serialization
>
>
> Key: HIVE-23191
> URL: https://issues.apache.org/jira/browse/HIVE-23191
> Project: Hive
> Issue Type: Improvement
>Reporter: Mustafa Iman
>Assignee: Mustafa Iman
>Priority: Major
> Attachments: HIVE-23191.patch
>
>
> {code:java}
> DagUtils#createVertex(JobConf, BaseWork, Path,
> TezWork, Map){code}
> creates an output descriptor if it is leaf vertex. It uses the same config
> object that is used in processor descriptor. It should not create payload
> from scratch when processor descriptor has the identical payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23029) LLAP: Shuffle Handler should support Index Cache configuration
[ https://issues.apache.org/jira/browse/HIVE-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23029: Fix Version/s: 4.0.0 Assignee: Rajesh Balamohan Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~ashutoshc] . Committed to master. > LLAP: Shuffle Handler should support Index Cache configuration > -- > > Key: HIVE-23029 > URL: https://issues.apache.org/jira/browse/HIVE-23029 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-23029.1.patch, Screenshot 2020-03-16 at 12.08.44 > PM.jpg > > > !Screenshot 2020-03-16 at 12.08.44 PM.jpg|width=1592,height=1112! > > Queries like Q78 at large scale misses index cache with unordered edges. (24 > * 1009 = 24216. With the default 10 MB cache size, it can accommodate only > 400+ entries). > -- This message was sent by Atlassian Jira (v8.3.4#803005)
