[jira] [Commented] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063098#comment-17063098
]
Hive QA commented on HIVE-23042:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
25s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 0 fixed = 187 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | A prepared statement is generated from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(Connection, Statement,
OpenTxnRequest) At TxnHandler.java:from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(Connection, Statement,
OpenTxnRequest) At TxnHandler.java:[line 637] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21188/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21188/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server
standalone-metastore/metastore-tools/tools-common
standalone-metastore/metastore-tools/metastore-benchmarks U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21188/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Status: Patch Available (was: Open)
New patch contains fixes for test failures. (not final patch as there is some
cleanup to be done with commented code).
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, Hive Metastore Support for
> Tenant-based storage heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Attachment: HIVE-22995.4.patch
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, Hive Metastore Support for
> Tenant-based storage heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-22995:
-
Status: Open (was: Patch Available)
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, HIVE-22995.4.patch, Hive Metastore Support for
> Tenant-based storage heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063087#comment-17063087
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997132/HIVE-23052.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21187/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21187/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21187/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997132 - PreCommit-HIVE-Build
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch
>
>
> TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the
> next lock id from the metastore. This locks the NEXT_LOCK_ID table for
> concurrent lock requests until the transaction has been committed.
> Since this creates contention, we should reduce the scope of this S4U,
> thereby locking the table for only as long as necessary. Currently, within
> the same txn, after the initial S4U, we also insert the transaction
> components (TXN_COMPONENTS table) before committing, even though they are
> independent and do not make use of the lock id. So moving these txn_component
> inserts before the S4U would decrease the time the next_lock_id table is
> locked and thus increase concurrency.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063065#comment-17063065
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 33 new + 547 unchanged - 37 fixed = 580 total (was 584) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21187/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21187/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21187/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch
>
>
> TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the
> next lock id from the metastore. This locks the NEXT_LOCK_ID table for
> concurrent lock requests until the transaction has been committed.
> Since this creates contention, we should reduce the scope of this S4U,
> thereby locking the table for only as long as necessary. Currently, within
> the same txn, after the initial S4U, we also insert the transaction
> components (TXN_COMPONENTS table) before committing, even though they are
> independent and do not make use of the lock id. So moving these txn_component
> inserts before the S4U would decrease the time the next_lock_id table is
> locked and thus increase concurrency.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19805) TableScanDesc Use Commons Library
[
https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063054#comment-17063054
]
Hive QA commented on HIVE-19805:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997131/HIVE-19805.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18123 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=193)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_fixed_bucket_pruning]
(batchId=190)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21186/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21186/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997131 - PreCommit-HIVE-Build
> TableScanDesc Use Commons Library
> -
>
> Key: HIVE-19805
> URL: https://issues.apache.org/jira/browse/HIVE-19805
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch,
> HIVE-19805.3.patch, HIVE-19805.4.patch, Screenshot from 2018-06-09
> 11-37-08.png
>
>
> Use commons library and remove some code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19805) TableScanDesc Use Commons Library
[
https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063043#comment-17063043
]
Hive QA commented on HIVE-19805:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
50s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 1
fixed = 7 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21186/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21186/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TableScanDesc Use Commons Library
> -
>
> Key: HIVE-19805
> URL: https://issues.apache.org/jira/browse/HIVE-19805
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch,
> HIVE-19805.3.patch, HIVE-19805.4.patch, Screenshot from 2018-06-09
> 11-37-08.png
>
>
> Use commons library and remove some code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063036#comment-17063036
]
Hive QA commented on HIVE-23037:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997125/HIVE-23037.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21185/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21185/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21185/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997125 - PreCommit-HIVE-Build
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[
https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063015#comment-17063015
]
Hive QA commented on HIVE-23037:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21185/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21185/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
> -
>
> Key: HIVE-23037
> URL: https://issues.apache.org/jira/browse/HIVE-23037
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23037.1.patch, HIVE-23037.1.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063009#comment-17063009
]
Hive QA commented on HIVE-23051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997115/HIVE-23051.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21184/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21184/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21184/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997115 - PreCommit-HIVE-Build
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it late checks that the {{statementId}} is greater,... which is
> will always be since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062999#comment-17062999
]
Hive QA commented on HIVE-23051:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 7
fixed = 1 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21184/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21184/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it late checks that the {{statementId}} is greater,... which is
> will always be since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23053) Clean Up Stats Mergers
[
https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062989#comment-17062989
]
Hive QA commented on HIVE-23053:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997114/HIVE-23053.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 18096 tests
executed
*Failed tests:*
{noformat}
TestEncryptedHDFSCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=195)
[encryption_insert_partition_static.q]
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=160)
[mapreduce2.q,bucket6.q,insert_into1.q,empty_dir_in_table.q,alter_table_location3.q,temp_table_drop_partitions_filter4.q,orc_merge1.q,tez_acid_union_dynamic_partition.q,parquet_types_vectorization.q,orc_merge_diff_fs.q,llap_stats.q,llapdecider.q,llap_nullscan.q,orc_ppd_basic.q,rcfile_merge4.q]
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query27]
(batchId=306)
org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat.org.apache.hadoop.hive.ql.parse.TestReplTableMigrationWithJsonFormat
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21183/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21183/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21183/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997114 - PreCommit-HIVE-Build
> Clean Up Stats Mergers
> --
>
> Key: HIVE-23053
> URL: https://issues.apache.org/jira/browse/HIVE-23053
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23053.1.patch
>
>
> * DEBUG log each invocation of merge
> * Use the inherited logger properly
> * Use parameterized logging
> * Simplify code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23053) Clean Up Stats Mergers
[
https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062968#comment-17062968
]
Hive QA commented on HIVE-23053:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 27 unchanged - 2 fixed = 29 total (was 29) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21183/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21183/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21183/yetus/whitespace-eol.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21183/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up Stats Mergers
> --
>
> Key: HIVE-23053
> URL: https://issues.apache.org/jira/browse/HIVE-23053
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23053.1.patch
>
>
> * DEBUG log each invocation of merge
> * Use the inherited logger properly
> * Use parameterized logging
> * Simplify code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062953#comment-17062953
]
Hive QA commented on HIVE-23032:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997109/HIVE-23032.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21182/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21182/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21182/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997109 - PreCommit-HIVE-Build
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch,
> HIVE-23032.3.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062939#comment-17062939 ] Naveen Gangam commented on HIVE-23033: -- [~dlavati] I have discussed this. Because this patch doesnt make any changes to the structure of the schema, either patch should be ok. There wouldn't be any compatibility issues. So +1 for the patch. > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1, 3.1.2 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-23033.1.patch, HIVE-23033.2.branch-3.patch, > HIVE-23033.2.patch > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. > This could be backported to all active branches for 3.x as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Attachment: HIVE-23004.12.patch Status: Patch Available (was: Open) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.12.patch, HIVE-23004.2.patch, > HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch, > HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23004) Support Decimal64 operations across multiple vertices
[ https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23004: Status: Open (was: Patch Available) > Support Decimal64 operations across multiple vertices > - > > Key: HIVE-23004 > URL: https://issues.apache.org/jira/browse/HIVE-23004 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch, > HIVE-23004.11.patch, HIVE-23004.2.patch, HIVE-23004.4.patch, > HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch > > > Support Decimal64 operations across multiple vertices -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062923#comment-17062923
]
Hive QA commented on HIVE-23032:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 13 new + 570 unchanged - 15 fixed = 583 total (was 585) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21182/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21182/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21182/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch,
> HIVE-23032.3.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22957) Support For Filter Expression In MSCK Command
[
https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062909#comment-17062909
]
Hive QA commented on HIVE-22957:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997101/HIVE-22957.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18124 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query18]
(batchId=306)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21181/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21181/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21181/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997101 - PreCommit-HIVE-Build
> Support For Filter Expression In MSCK Command
> -
>
> Key: HIVE-22957
> URL: https://issues.apache.org/jira/browse/HIVE-22957
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22957.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently MSCK command supports full repair of table (all partitions) or some
> subset of partitions based on partitionSpec. The aim of this jira is to
> introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a
> larger subset of partitions can be recovered (added/deleted) without firing a
> full repair might take time if the no. of partitions are huge.
> *Approach*:
> The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR
> TABLE ADD|DROP|SYNC PARTITIONS WHERE
> AND
> *Flow:*
> 1) Parse the where clause and generate filterExpression
> 2) fetch all the partitions from the metastore which matches the filter
> expression
> 3) fetch all the partition file from the filesystem
> 4) remove all the partition path which does not match with the filter
> expression
> 5) Based on ADD | DROP | SYNC do the remaining steps.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22957) Support For Filter Expression In MSCK Command
[
https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062879#comment-17062879
]
Hive QA commented on HIVE-22957:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
57s{color} | {color:blue} parser in master has 3 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch parser passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 121 unchanged - 2 fixed = 121 total (was 123) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21181/dev-support/hive-personality.sh
|
| git revision | master / 213ca2e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: parser standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21181/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support For Filter Expression In MSCK Command
> -
>
> Key: HIVE-22957
> URL: https://issues.apache.org/jira/browse/HIVE-22957
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22957.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently MSCK command
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062854#comment-17062854
]
Hive QA commented on HIVE-22940:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 126 unchanged - 3
fixed = 126 total (was 129) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} root: The patch generated 0 new + 132 unchanged - 3
fixed = 132 total (was 135) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hive-minikdc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch qtest-accumulo passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch qtest-kudu passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch qtest-spark passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
58s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 89m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate.deriveRowType(RelDataTypeFactory,
RelDataType, boolean, ImmutableBitSet, List)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062850#comment-17062850
]
Hive QA commented on HIVE-22940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997095/HIVE-22940.05.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18125 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21180/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21180/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21180/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997095 - PreCommit-HIVE-Build
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch,
> HIVE-22940.03.patch, HIVE-22940.04.patch, HIVE-22940.05.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: Patch Available (was: In Progress) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: In Progress (was: Patch Available) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Attachment: HIVE-23045.5.patch > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch, HIVE-23045.5.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23044) Make sure Cleaner doesn't delete delta directories for running queries
[ https://issues.apache.org/jira/browse/HIVE-23044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Chovan updated HIVE-23044: - Attachment: HIVE-23044.3.branch-3.1.patch > Make sure Cleaner doesn't delete delta directories for running queries > -- > > Key: HIVE-23044 > URL: https://issues.apache.org/jira/browse/HIVE-23044 > Project: Hive > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Zoltan Chovan >Assignee: Zoltan Chovan >Priority: Major > Attachments: HIVE-23044.1.branch-3.1.patch, > HIVE-23044.2.branch-3.1.patch, HIVE-23044.3.branch-3.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062766#comment-17062766
]
Hive QA commented on HIVE-22126:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 44 unchanged - 1
fixed = 44 total (was 45) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} root: The patch generated 0 new + 50 unchanged - 1
fixed = 50 total (was 51) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hive-minikdc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch qtest-accumulo passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch qtest-kudu passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch qtest-spark passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
0s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| |
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveAggregate.deriveRowType(RelDataTypeFactory,
RelDataType, boolean, ImmutableBitSet, List) concatenates
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[
https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062762#comment-17062762
]
Hive QA commented on HIVE-22126:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997094/HIVE-22126.09.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18123 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testRowsAffected (batchId=286)
org.apache.hive.jdbc.TestXSRFFilter.testFilterEnabledWithInjection (batchId=290)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21179/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21179/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21179/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997094 - PreCommit-HIVE-Build
> hive-exec packaging should shade guava
> --
>
> Key: HIVE-22126
> URL: https://issues.apache.org/jira/browse/HIVE-22126
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Eugene Chung
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch,
> HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch,
> HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch,
> HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch
>
>
> The ql/pom.xml includes complete guava library into hive-exec.jar
> https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a
> problems for downstream clients of hive which have hive-exec.jar in their
> classpath since they are pinned to the same guava version as that of hive.
> We should shade guava classes so that other components which depend on
> hive-exec can independently use a different version of guava as needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23042:
--
Attachment: HIVE-23042.3.patch
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23042.2.patch, HIVE-23042.3.patch, HIVE-23042.patch
>
>
> When opening a new transaction we issue 2 queries to update the MIN_OPEN_TXN
> table.
> {code}
>
> values(763, 763)>
> {code}
> This could be archived with a single query faster, if we do not open
> transactions in batch, like:
> {code}
>SELECT ?, MIN("TXN_ID") FROM "TXNS" WHERE "TXN_STATE" = 'o'>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062683#comment-17062683
]
Hive QA commented on HIVE-23042:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997089/HIVE-23042.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18123 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21178/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21178/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21178/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997089 - PreCommit-HIVE-Build
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23042.2.patch, HIVE-23042.patch
>
>
> When opening a new transaction we issue 2 queries to update the MIN_OPEN_TXN
> table.
> {code}
>
> values(763, 763)>
> {code}
> This could be archived with a single query faster, if we do not open
> transactions in batch, like:
> {code}
>SELECT ?, MIN("TXN_ID") FROM "TXNS" WHERE "TXN_STATE" = 'o'>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23047) Calculate the epoch on DB side
[ https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062671#comment-17062671 ] Denys Kuzmenko commented on HIVE-23047: --- +1 > Calculate the epoch on DB side > -- > > Key: HIVE-23047 > URL: https://issues.apache.org/jira/browse/HIVE-23047 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23047.2.patch, HIVE-23047.patch > > > We use TxnHandler.getDbTime to calculate the epoch on the DB server, and > immediately insert the value back again. We would be better of by using sql > to calculate the value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.1.patch Status: Patch Available (was: In Progress) > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch > > > TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the > next lock id from the metastore. This locks the NEXT_LOCK_ID table for > concurrent lock requests until the transaction has been committed. > Since this creates contention, we should reduce the scope of this S4U, > thereby locking the table for only as long as necessary. Currently, within > the same txn, after the initial S4U, we also insert the transaction > components (TXN_COMPONENTS table) before committing, even though they are > independent and do not make use of the lock id. So moving these txn_component > inserts before the S4U would decrease the time the next_lock_id table is > locked and thus increase concurrency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-19805: -- Attachment: HIVE-19805.4.patch > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch, > HIVE-19805.3.patch, HIVE-19805.4.patch, Screenshot from 2018-06-09 > 11-37-08.png > > > Use commons library and remove some code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23035) Scheduled query executor may hang in case TezAMs are launched on-demand
[ https://issues.apache.org/jira/browse/HIVE-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23035: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you Laszlo for reviewing the changes! > Scheduled query executor may hang in case TezAMs are launched on-demand > --- > > Key: HIVE-23035 > URL: https://issues.apache.org/jira/browse/HIVE-23035 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23035.01.patch > > > Right now the schq executor hangs during session initialization - because it > tries to open the tez session while it initializes the SessionState -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062635#comment-17062635
]
Hive QA commented on HIVE-23042:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
26s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 0 fixed = 187 total (was 186) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | A prepared statement is generated from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(Connection, Statement,
OpenTxnRequest) At TxnHandler.java:from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxns(Connection, Statement,
OpenTxnRequest) At TxnHandler.java:[line 637] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21178/dev-support/hive-personality.sh
|
| git revision | master / e182d9c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21178/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server
standalone-metastore/metastore-tools/tools-common
standalone-metastore/metastore-tools/metastore-benchmarks U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21178/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>
[jira] [Comment Edited] (HIVE-23048) Use sequences for TXN_ID generation
[
https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062614#comment-17062614
]
Peter Vary edited comment on HIVE-23048 at 3/19/20, 2:19 PM:
-
We need the following functionality from the backend database:
* Way to generate TXN_ID - sequence, or identity like stuff
* Way to insert single row to TXN data, and retrieve back the generated id
* Way to batch insert row to TXN data, and retrieve back the generated id-s
* Way to get the last transaction id for HWM calculations
Created a small test class to check the different databases, used the following
versions:
* Derby - 10.14.1.0
* MySQL - 5.7.23
* PostgreSQL - 11.5
* Microsoft SQL Server - 2017 GA
* Oracle - XE 11g - had to use ojdbc8.jar driver (downloaded for the 19c
version) to make getGeneratedKeys work for batch
Here are the results:
{code:java}
Checking Apache Derby
Single statement
Gen key: 1
Multi statement
Gen keys: [11]
Checking MySQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Checking PostgreSQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
Checking Microsoft SQL Server
Single statement
Gen key: 1
Multi statement
com.microsoft.sqlserver.jdbc.SQLServerException: The statement must be executed
before any results can be obtained.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:234)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getGeneratedKeys(SQLServerStatement.java:2216)
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.getGeneratedKeys(HikariProxyPreparedStatement.java)
at org.apache.hadoop.hive.a.runCheck(a.java:130)
at org.apache.hadoop.hive.a.main(a.java:171)
Seq state: 11
Checking Oracle
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
{code}
The key takeaways:
* MSSQL and Derby does not support batch insert with getGeneratedKeys.
Workaround could be:
** Insert them row by row
* MySQL, MSSQL, Derby does not support retrieving the last transaction id.
Workaround could be:
** Run specific query, like:
{code:java}
SELECT MAX(TXN_ID) FROM (SELECT MAX(TXN_ID) FROM TXNS UNION ALL SELECT
MAX(CTC_TXNID) FROM COMPLETED_TXN_COMPONENTS)
{code}
was (Author: pvary):
We need the following functionality from the backend database:
* Way to generate TXN_ID - sequence, or identity like stuff
* Way to batch insert row to TXN data, and retrieve back the generated id-s
* Way to get the last transaction id for HWM calculations
Created a small test class to check the different databases, used the following
versions:
* Derby - 10.14.1.0
* MySQL - 5.7.23
* PostgreSQL - 11.5
* Microsoft SQL Server - 2017 GA
* Oracle - XE 11g - had to use ojdbc8.jar driver (downloaded for the 19c
version) to make getGeneratedKeys work for batch
Here are the results:
{code:java}
Checking Apache Derby
Single statement
Gen key: 1
Multi statement
Gen keys: [11]
Checking MySQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Checking PostgreSQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
Checking Microsoft SQL Server
Single statement
Gen key: 1
Multi statement
com.microsoft.sqlserver.jdbc.SQLServerException: The statement must be executed
before any results can be obtained.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:234)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getGeneratedKeys(SQLServerStatement.java:2216)
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.getGeneratedKeys(HikariProxyPreparedStatement.java)
at org.apache.hadoop.hive.a.runCheck(a.java:130)
at org.apache.hadoop.hive.a.main(a.java:171)
Seq state: 11
Checking Oracle
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
{code}
The key takeaways:
* MSSQL and Derby does not support batch insert with getGeneratedKeys.
Workaround could be:
** Insert them row by row
* MySQL, MSSQL, Derby does not support retrieving the last transaction id.
Workaround could be:
** Run specific query, like:
{code:java}
SELECT MAX(TXN_ID) FROM (SELECT MAX(TXN_ID) FROM TXNS UNION ALL SELECT
MAX(CTC_TXNID) FROM COMPLETED_TXN_COMPONENTS)
{code}
> Use sequences for TXN_ID generation
> ---
>
> Key: HIVE-23048
> URL: https://issues.apache.org/jira/browse/HIVE-23048
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
[jira] [Updated] (HIVE-23048) Use sequences for TXN_ID generation
[ https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23048: -- Attachment: DbTest.java > Use sequences for TXN_ID generation > --- > > Key: HIVE-23048 > URL: https://issues.apache.org/jira/browse/HIVE-23048 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: DbTest.java > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23048) Use sequences for TXN_ID generation
[
https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062614#comment-17062614
]
Peter Vary commented on HIVE-23048:
---
We need the following functionality from the backend database:
* Way to generate TXN_ID - sequence, or identity like stuff
* Way to batch insert row to TXN data, and retrieve back the generated id-s
* Way to get the last transaction id for HWM calculations
Created a small test class to check the different databases, used the following
versions:
* Derby - 10.14.1.0
* MySQL - 5.7.23
* PostgreSQL - 11.5
* Microsoft SQL Server - 2017 GA
* Oracle - XE 11g - had to use ojdbc8.jar driver (downloaded for the 19c
version) to make getGeneratedKeys work for batch
Here are the results:
{code:java}
Checking Apache Derby
Single statement
Gen key: 1
Multi statement
Gen keys: [11]
Checking MySQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Checking PostgreSQL
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
Checking Microsoft SQL Server
Single statement
Gen key: 1
Multi statement
com.microsoft.sqlserver.jdbc.SQLServerException: The statement must be executed
before any results can be obtained.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:234)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getGeneratedKeys(SQLServerStatement.java:2216)
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.getGeneratedKeys(HikariProxyPreparedStatement.java)
at org.apache.hadoop.hive.a.runCheck(a.java:130)
at org.apache.hadoop.hive.a.main(a.java:171)
Seq state: 11
Checking Oracle
Single statement
Gen key: 1
Multi statement
Gen keys: [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
Seq state: 11
{code}
The key takeaways:
* MSSQL and Derby does not support batch insert with getGeneratedKeys.
Workaround could be:
** Insert them row by row
* MySQL, MSSQL, Derby does not support retrieving the last transaction id.
Workaround could be:
** Run specific query, like:
{code:java}
SELECT MAX(TXN_ID) FROM (SELECT MAX(TXN_ID) FROM TXNS UNION ALL SELECT
MAX(CTC_TXNID) FROM COMPLETED_TXN_COMPONENTS)
{code}
> Use sequences for TXN_ID generation
> ---
>
> Key: HIVE-23048
> URL: https://issues.apache.org/jira/browse/HIVE-23048
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-19805) TableScanDesc Use Commons Library
[ https://issues.apache.org/jira/browse/HIVE-19805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-19805: -- Priority: Minor (was: Trivial) > TableScanDesc Use Commons Library > - > > Key: HIVE-19805 > URL: https://issues.apache.org/jira/browse/HIVE-19805 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-19805.1.patch, HIVE-19805.2.patch, > HIVE-19805.3.patch, Screenshot from 2018-06-09 11-37-08.png > > > Use commons library and remove some code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21289) Expect EQ and LIKE to Generate the Identical Explain Plans
[
https://issues.apache.org/jira/browse/HIVE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062605#comment-17062605
]
David Mollitor commented on HIVE-21289:
---
[~kgyrtkirk] [~jcamachorodriguez] Are you able to take a peek at this?
> Expect EQ and LIKE to Generate the Identical Explain Plans
> --
>
> Key: HIVE-21289
> URL: https://issues.apache.org/jira/browse/HIVE-21289
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.3.4
>Reporter: David Mollitor
>Priority: Minor
>
> I generated some test data with the UUID function.
> {code:sql}
> explain select * from test_like where a like
> 'abce6254-d437-426b-8873-2cbc153ddfbc';
> explain select * from test_like where a =
> 'abce6254-d437-426b-8873-2cbc153ddfbc';
> {code}
> {code}
> Explain
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_like
> filterExpr: (a like 'abce6254-d437-426b-8873-2cbc153ddfbc')
> (type: boolean)
> Statistics: Num rows: 262144 Data size: 9437184 Basic stats:
> COMPLETE Column stats: NONE
> Filter Operator
> predicate: (a like 'abce6254-d437-426b-8873-2cbc153ddfbc')
> (type: boolean)
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: a (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 131072 Data size: 4718592 Basic
> stats: COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
> {code}
> Explain
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_like
> filterExpr: (a = 'abce6254-d437-426b-8873-2cbc153ddfbc') (type:
> boolean)
> Statistics: Num rows: 262144 Data size: 9437184 Basic stats:
> COMPLETE Column stats: NONE
> Filter Operator
> predicate: (a = 'abce6254-d437-426b-8873-2cbc153ddfbc') (type:
> boolean)
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: 'abce6254-d437-426b-8873-2cbc153ddfbc' (type:
> string)
> outputColumnNames: _col0
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 131072 Data size: 4718592 Basic
> stats: COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
> They may be the same under the covers, but I would expect the EXPLAIN plan to
> be exactly the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21289) Expect EQ and LIKE to Generate the Identical Explain Plans
[
https://issues.apache.org/jira/browse/HIVE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21289:
--
Component/s: CBO
> Expect EQ and LIKE to Generate the Identical Explain Plans
> --
>
> Key: HIVE-21289
> URL: https://issues.apache.org/jira/browse/HIVE-21289
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.3.4
>Reporter: David Mollitor
>Priority: Minor
>
> I generated some test data with the UUID function.
> {code:sql}
> explain select * from test_like where a like
> 'abce6254-d437-426b-8873-2cbc153ddfbc';
> explain select * from test_like where a =
> 'abce6254-d437-426b-8873-2cbc153ddfbc';
> {code}
> {code}
> Explain
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_like
> filterExpr: (a like 'abce6254-d437-426b-8873-2cbc153ddfbc')
> (type: boolean)
> Statistics: Num rows: 262144 Data size: 9437184 Basic stats:
> COMPLETE Column stats: NONE
> Filter Operator
> predicate: (a like 'abce6254-d437-426b-8873-2cbc153ddfbc')
> (type: boolean)
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: a (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 131072 Data size: 4718592 Basic
> stats: COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
> {code}
> Explain
> STAGE DEPENDENCIES:
> Stage-1 is a root stage
> Stage-0 depends on stages: Stage-1
> STAGE PLANS:
> Stage: Stage-1
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: test_like
> filterExpr: (a = 'abce6254-d437-426b-8873-2cbc153ddfbc') (type:
> boolean)
> Statistics: Num rows: 262144 Data size: 9437184 Basic stats:
> COMPLETE Column stats: NONE
> Filter Operator
> predicate: (a = 'abce6254-d437-426b-8873-2cbc153ddfbc') (type:
> boolean)
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> Select Operator
> expressions: 'abce6254-d437-426b-8873-2cbc153ddfbc' (type:
> string)
> outputColumnNames: _col0
> Statistics: Num rows: 131072 Data size: 4718592 Basic stats:
> COMPLETE Column stats: NONE
> File Output Operator
> compressed: false
> Statistics: Num rows: 131072 Data size: 4718592 Basic
> stats: COMPLETE Column stats: NONE
> table:
> input format: org.apache.hadoop.mapred.TextInputFormat
> output format:
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
> serde:
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> ListSink
> {code}
> They may be the same under the covers, but I would expect the EXPLAIN plan to
> be exactly the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062597#comment-17062597
]
Hive QA commented on HIVE-23045:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997088/HIVE-23045.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18128 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query21]
(batchId=307)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21177/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21177/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21177/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997088 - PreCommit-HIVE-Build
> Zookeeper SSL/TLS support
> -
>
> Key: HIVE-23045
> URL: https://issues.apache.org/jira/browse/HIVE-23045
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC, Metastore
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch,
> HIVE-23045.3.patch, HIVE-23045.4.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its
> clients.
> [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide]
> The SSL communication should be possible in the different part of HIVE, where
> it communicates with Zookeeper servers. The Zookeeper clients are used in the
> following places:
> * HiveServer2 PrivilegeSynchronizer
> * HiveServer2 register/remove server from Zookeeper
> * HS2ActivePassiveHARegistryClient
> * ZooKeeperHiveLockManager
> * LLapZookeeperRegistryImpl
> * TezAmRegistryImpl
> * WebHCat ZooKeeperStorage
> * JDBC Driver server lookup
> * Metastore - ZookeeperTokenStore
> * Metastore register/remove server from Zookeeper
> The flag to enable SSL communication and the required parameters should be
> provided by different configuration parameters, corresponding the different
> use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23037) Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus
[ https://issues.apache.org/jira/browse/HIVE-23037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23037: -- Attachment: HIVE-23037.1.patch > Print Logging Information for Exception in AcidUtils tryListLocatedHdfsStatus > - > > Key: HIVE-23037 > URL: https://issues.apache.org/jira/browse/HIVE-23037 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-23037.1.patch, HIVE-23037.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23016) Extract JdbcConnectionParams from Utils Class
[ https://issues.apache.org/jira/browse/HIVE-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062581#comment-17062581 ] David Mollitor commented on HIVE-23016: --- [~anir1ghosh] Would love your assistance. Just FYI, I do have a few open patches that relies on the current location. Can you check back in a week or two? > Extract JdbcConnectionParams from Utils Class > - > > Key: HIVE-23016 > URL: https://issues.apache.org/jira/browse/HIVE-23016 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: n00b, newbie, noob > > And make it its own class. > https://github.com/apache/hive/blob/4700e210ef7945278c4eb313c9ebd810b0224da1/jdbc/src/java/org/apache/hive/jdbc/Utils.java#L72 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23054) Capture Total Byte Size in Column Statistics
[
https://issues.apache.org/jira/browse/HIVE-23054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23054:
--
Description:
Store a counter in HMS column statics for the total number of bytes (raw) in
each column.
Right now, there is no good way to merge the average column length when
performing an INSERT statement into a table. Right now, the code just selects
the maximum value, however, if inserting a single records with a long length
(128 bytes) into a table that has millions of strings with an average length of
4, the average length for the entire data set gets boosted to 128.
{code:java}
aggregateData.setAvgColLen(Math.max(aggregateData.getAvgColLen(),
newData.getAvgColLen()));
{code}
https://github.com/apache/hive/blob/e182d9ce6c09136d13ee889ef069b202f60052ec/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/merge/StringColumnStatsMerger.java#L34
Store the total raw size of all the data in each column. Between the total raw
size, and the average length, one can compute the real average length when
merging the existing data and the newly inserted data.
was:
Store a counter in HMS column statics for the total number of bytes (raw) in
each column.
Right now, there is no good way to merge the average column length when
performing an INSERT statement into a table. Right now, the code just selects
the maximum value, however, if inserting a single records with a long length
(128 bytes) into a table that has millions of strings with an average length of
4, the average length for the entire data set gets boosted to 128.
{code:java}
aggregateData.setAvgColLen(Math.max(aggregateData.getAvgColLen(),
newData.getAvgColLen()));
{code}
https://github.com/apache/hive/blob/e182d9ce6c09136d13ee889ef069b202f60052ec/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/merge/StringColumnStatsMerger.java#L34
Store the total raw size of all the data in each column. Between the total raw
size, and the average length, one can compute the real average length when
merging the exiting data and the newly inserted data.
> Capture Total Byte Size in Column Statistics
>
>
> Key: HIVE-23054
> URL: https://issues.apache.org/jira/browse/HIVE-23054
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Statistics
>Reporter: David Mollitor
>Priority: Major
>
> Store a counter in HMS column statics for the total number of bytes (raw) in
> each column.
> Right now, there is no good way to merge the average column length when
> performing an INSERT statement into a table. Right now, the code just
> selects the maximum value, however, if inserting a single records with a long
> length (128 bytes) into a table that has millions of strings with an average
> length of 4, the average length for the entire data set gets boosted to 128.
> {code:java}
> aggregateData.setAvgColLen(Math.max(aggregateData.getAvgColLen(),
> newData.getAvgColLen()));
> {code}
> https://github.com/apache/hive/blob/e182d9ce6c09136d13ee889ef069b202f60052ec/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/merge/StringColumnStatsMerger.java#L34
> Store the total raw size of all the data in each column. Between the total
> raw size, and the average length, one can compute the real average length
> when merging the existing data and the newly inserted data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[
https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062568#comment-17062568
]
Hive QA commented on HIVE-23045:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
36s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 2 new + 86 unchanged - 0 fixed = 88 total (was 86) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 374 unchanged - 0
fixed = 375 total (was 374) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch llap-client passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 36 unchanged - 1 fixed = 36 total (was 37) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 3
fixed = 7 total (was 10) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 37 unchanged -
1 fixed = 37 total (was 38) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/webhcat/svr: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle
[jira] [Updated] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23051:
--
Attachment: HIVE-23051.2.patch
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch, HIVE-23051.2.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it late checks that the {{statementId}} is greater,... which is
> will always be since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23053) Clean Up Stats Mergers
[ https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23053: -- Attachment: HIVE-23053.1.patch > Clean Up Stats Mergers > -- > > Key: HIVE-23053 > URL: https://issues.apache.org/jira/browse/HIVE-23053 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23053.1.patch > > > * DEBUG log each invocation of merge > * Use the inherited logger properly > * Use parameterized logging > * Simplify code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23053) Clean Up Stats Mergers
[ https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23053: -- Status: Patch Available (was: Open) > Clean Up Stats Mergers > -- > > Key: HIVE-23053 > URL: https://issues.apache.org/jira/browse/HIVE-23053 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23053.1.patch > > > * DEBUG log each invocation of merge > * Use the inherited logger properly > * Use parameterized logging > * Simplify code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23053) Clean Up Stats Mergers
[ https://issues.apache.org/jira/browse/HIVE-23053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23053: - > Clean Up Stats Mergers > -- > > Key: HIVE-23053 > URL: https://issues.apache.org/jira/browse/HIVE-23053 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > > * DEBUG log each invocation of merge > * Use the inherited logger properly > * Use parameterized logging > * Simplify code -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Attachment: HIVE-23032.3.patch > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23047) Calculate the epoch on DB side
[
https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062524#comment-17062524
]
Hive QA commented on HIVE-23047:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997087/HIVE-23047.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18123 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21176/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21176/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21176/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997087 - PreCommit-HIVE-Build
> Calculate the epoch on DB side
> --
>
> Key: HIVE-23047
> URL: https://issues.apache.org/jira/browse/HIVE-23047
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23047.2.patch, HIVE-23047.patch
>
>
> We use TxnHandler.getDbTime to calculate the epoch on the DB server, and
> immediately insert the value back again. We would be better of by using sql
> to calculate the value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-22888:
--
Description:
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
HL_LOCK_TYPE FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 333) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
HL_LOCK_EXT_ID = 333) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR LS.HL_PARTITION
= LBC.HL_PARTITION))
WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
AND (LBC.HL_LOCK_TYPE='e'
AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
IS NOT NULL )
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
LIMIT 1;
{code}
was:
- Created extra (db, tbl, part) index on HIVE_LOCKS table;
- Replaced inner select under checkLocks using multiple IN statements with JOIN
operator;
generated query looks like :
{code}
SELECT LS.* FROM (
SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
HL_LOCK_TYPE FROM HIVE_LOCKS
WHERE HL_LOCK_EXT_ID < 333) LS
INNER JOIN (
SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
HL_LOCK_EXT_ID = 333) LBC
ON LS.HL_DB = LBC.HL_DB
AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
LBC.HL_TABLE
AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR LS.HL_PARTITION
= LBC.HL_PARTITION))
WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
AND (LBC.HL_LOCK_TYPE='e'
AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
IS NOT NULL )
OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
LIMIT 1;
{code}
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch,
> HIVE-22888.11.patch, HIVE-22888.2.patch, HIVE-22888.3.patch,
> HIVE-22888.4.patch, HIVE-22888.5.patch, HIVE-22888.6.patch,
> HIVE-22888.8.patch, HIVE-22888.9.patch, acid-lock-perf-test.pdf
>
>
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23049) Constraint name uniqueness query should set Long as parameter instead of long
[ https://issues.apache.org/jira/browse/HIVE-23049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23049: -- Resolution: Invalid Status: Resolved (was: Patch Available) This is not the issue here. > Constraint name uniqueness query should set Long as parameter instead of long > - > > Key: HIVE-23049 > URL: https://issues.apache.org/jira/browse/HIVE-23049 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23049.01.patch, HIVE-23049.02.patch > > > Running with Oracle19 the parameters present to a query in datanucleus are > not boxed, thus it must be explicitly set to Long and not long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23047) Calculate the epoch on DB side
[
https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062470#comment-17062470
]
Hive QA commented on HIVE-23047:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
17s{color} | {color:blue} standalone-metastore/metastore-server in master has
186 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 23 new + 572 unchanged - 12 fixed = 595 total (was 584) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21176/dev-support/hive-personality.sh
|
| git revision | master / e182d9c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21176/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21176/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Calculate the epoch on DB side
> --
>
> Key: HIVE-23047
> URL: https://issues.apache.org/jira/browse/HIVE-23047
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23047.2.patch, HIVE-23047.patch
>
>
> We use TxnHandler.getDbTime to calculate the epoch on the DB server, and
> immediately insert the value back again. We would be better of by using sql
> to calculate the value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062460#comment-17062460
]
Hive QA commented on HIVE-23004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997084/HIVE-23004.11.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18124 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=126)
org.apache.hive.jdbc.TestSSL.testSSLFetch (batchId=291)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21175/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21175/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21175/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997084 - PreCommit-HIVE-Build
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.10.patch,
> HIVE-23004.11.patch, HIVE-23004.2.patch, HIVE-23004.4.patch,
> HIVE-23004.6.patch, HIVE-23004.7.patch, HIVE-23004.8.patch, HIVE-23004.9.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Description: TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the next lock id from the metastore. This locks the NEXT_LOCK_ID table for concurrent lock requests until the transaction has been committed. Since this creates contention, we should reduce the scope of this S4U, thereby locking the table for only as long as necessary. Currently, within the same txn, after the initial S4U, we also insert the transaction components (TXN_COMPONENTS table) before committing, even though they are independent and do not make use of the lock id. So moving these txn_component inserts before the S4U would decrease the time the next_lock_id table is locked and thus increase concurrency. was: TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the next lock id from the metastore. This locks the NEXT_LOCK_ID table for concurrent lock requests until the transaction has been committed. Since this creates contention, we should reduce the scope of this S4U, thereby locking the table for only as long as necessary. Currently, within the same txn, after the initial S4U, we also insert the transaction components (TXN_COMPONENTS table) before committing, even though they are independent and do not make use of the lock id. So factoring out these txn_component inserts should decrease the time the next_lock_id table is locked and thus increase concurrency. > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the > next lock id from the metastore. This locks the NEXT_LOCK_ID table for > concurrent lock requests until the transaction has been committed. > Since this creates contention, we should reduce the scope of this S4U, > thereby locking the table for only as long as necessary. Currently, within > the same txn, after the initial S4U, we also insert the transaction > components (TXN_COMPONENTS table) before committing, even though they are > independent and do not make use of the lock id. So moving these txn_component > inserts before the S4U would decrease the time the next_lock_id table is > locked and thus increase concurrency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23052 started by Marton Bod. - > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the > next lock id from the metastore. This locks the NEXT_LOCK_ID table for > concurrent lock requests until the transaction has been committed. > Since this creates contention, we should reduce the scope of this S4U, > thereby locking the table for only as long as necessary. Currently, within > the same txn, after the initial S4U, we also insert the transaction > components (TXN_COMPONENTS table) before committing, even though they are > independent and do not make use of the lock id. So factoring out these > txn_component inserts should decrease the time the next_lock_id table is > locked and thus increase concurrency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod reassigned HIVE-23052: - > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > TxnHandler::enqueueLockWithRetry uses select for update (S4U) to retrieve the > next lock id from the metastore. This locks the NEXT_LOCK_ID table for > concurrent lock requests until the transaction has been committed. > Since this creates contention, we should reduce the scope of this S4U, > thereby locking the table for only as long as necessary. Currently, within > the same txn, after the initial S4U, we also insert the transaction > components (TXN_COMPONENTS table) before committing, even though they are > independent and do not make use of the lock id. So factoring out these > txn_component inserts should decrease the time the next_lock_id table is > locked and thus increase concurrency. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062426#comment-17062426
]
Hive QA commented on HIVE-23004:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} serde: The patch generated 6 new + 289 unchanged - 0
fixed = 295 total (was 289) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 28 new + 849 unchanged - 3
fixed = 877 total (was 852) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} serde generated 3 new + 194 unchanged - 3 fixed = 197
total (was 197) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
56s{color} | {color:red} ql generated 2 new + 1530 unchanged - 1 fixed = 1532
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:serde |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNotNullMarker At
BinarySortableDeserializeRead.java:[line 145] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:byte[], byte[]) may expose internal
representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnNullMarker At
BinarySortableDeserializeRead.java:[line 144] |
| | new
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead(TypeInfo[],
DataTypePhysicalVariation[], boolean, boolean[], byte[], byte[]) may expose
internal representation by storing an externally mutable object into
BinarySortableDeserializeRead.columnSortOrderIsDesc At
BinarySortableDeserializeRead.java:byte[],
[jira] [Updated] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Attachment: (was: HIVE-22957.01.patch) > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Attachment: HIVE-22957.01.patch > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Component/s: Standalone Metastore > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?focusedWorklogId=406040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406040 ] ASF GitHub Bot logged work on HIVE-22957: - Author: ASF GitHub Bot Created on: 19/Mar/20 09:52 Start Date: 19/Mar/20 09:52 Worklog Time Spent: 10m Work Description: shameersss1 commented on pull request #959: HIVE-22957: Support For Filter Expression In MSCK Command URL: https://github.com/apache/hive/pull/959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 406040) Remaining Estimate: 0h Time Spent: 10m > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22957: -- Labels: pull-request-available (was: ) > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23049) Constraint name uniqueness query should set Long as parameter instead of long
[ https://issues.apache.org/jira/browse/HIVE-23049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23049: -- Attachment: (was: HIVE-23049.02.patch) > Constraint name uniqueness query should set Long as parameter instead of long > - > > Key: HIVE-23049 > URL: https://issues.apache.org/jira/browse/HIVE-23049 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23049.01.patch, HIVE-23049.02.patch > > > Running with Oracle19 the parameters present to a query in datanucleus are > not boxed, thus it must be explicitly set to Long and not long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23049) Constraint name uniqueness query should set Long as parameter instead of long
[ https://issues.apache.org/jira/browse/HIVE-23049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23049: -- Attachment: HIVE-23049.02.patch > Constraint name uniqueness query should set Long as parameter instead of long > - > > Key: HIVE-23049 > URL: https://issues.apache.org/jira/browse/HIVE-23049 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23049.01.patch, HIVE-23049.02.patch > > > Running with Oracle19 the parameters present to a query in datanucleus are > not boxed, thus it must be explicitly set to Long and not long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062404#comment-17062404 ] Syed Shameerur Rahman edited comment on HIVE-22957 at 3/19/20, 9:46 AM: Submitted initial patch for verification. Currently added support only for *STRING* partition column for predicate filtering with limited operators such as *EQUAL*, *LIKE*, *NOTEQUAL* etc. I will add support for more data types and operator in a separate Jira or as part of this Jira after initial verification of the patch. Partition column predicate can only we cascaded by *AND* logical operator might extend this to *OR* later. [~jcamachorodriguez] [~prasanth_j] Can you please review the patch? was (Author: srahman): Submitted initial patch for verification. Currently added support only for *STRING* partition column for predicate filtering with limited operators such as *EQUAL*, *LIKE*, *NOTEQUAL* etc. I will add support for more data types and operator in a separate Jira or as part of this Jira after initial verification of the patch. [~jcamachorodriguez] [~prasanth_j] Can you please review the patch? > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062404#comment-17062404 ] Syed Shameerur Rahman commented on HIVE-22957: -- Submitted initial patch for verification. Currently added support only for *STRING* partition column for predicate filtering with limited operators such as *EQUAL*, *LIKE*, *NOTEQUAL* etc. I will add support for more data types and operator in a separate Jira or as part of this Jira after initial verification of the patch. [~jcamachorodriguez] [~prasanth_j] Can you please review the patch? > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Comment: was deleted (was: [~prasanth_j] [~rbalamohan] [~jcamachorodriguez] Any Concerns / Suggestions regarding the approach?) > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22957) Support For Filter Expression In MSCK Command
[ https://issues.apache.org/jira/browse/HIVE-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-22957: - Attachment: HIVE-22957.01.patch Status: Patch Available (was: Open) > Support For Filter Expression In MSCK Command > - > > Key: HIVE-22957 > URL: https://issues.apache.org/jira/browse/HIVE-22957 > Project: Hive > Issue Type: Improvement >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22957.01.patch > > > Currently MSCK command supports full repair of table (all partitions) or some > subset of partitions based on partitionSpec. The aim of this jira is to > introduce a filterExp (=, !=, <, >, >=, <=, LIKE) in MSCK command so that a > larger subset of partitions can be recovered (added/deleted) without firing a > full repair might take time if the no. of partitions are huge. > *Approach*: > The initial approach is to add a where clause in MSCK command Eg: MCK REPAIR > TABLE ADD|DROP|SYNC PARTITIONS WHERE > AND > *Flow:* > 1) Parse the where clause and generate filterExpression > 2) fetch all the partitions from the metastore which matches the filter > expression > 3) fetch all the partition file from the filesystem > 4) remove all the partition path which does not match with the filter > expression > 5) Based on ADD | DROP | SYNC do the remaining steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062402#comment-17062402 ] Peter Varga commented on HIVE-23045: [~dkuzmenko] [~pvary] could you take a look at this. Thanks. > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062398#comment-17062398
]
Hive QA commented on HIVE-22995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997083/HIVE-22995.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 107 failed/errored test(s), 18128 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[escape_comments]
(batchId=306)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[authorization_owner_actions_db]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_uses_database_location]
(batchId=40)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[describe_database]
(batchId=42)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=196)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=117)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=117)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultClient.tablesCreateDropAlterTruncate
(batchId=230)
org.apache.hadoop.hive.metastore.TestCatalogNonDefaultSvr.tablesCreateDropAlterTruncate
(batchId=239)
org.apache.hadoop.hive.metastore.TestCatalogOldClient.databases (batchId=230)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterTable
(batchId=233)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTableDatabase
(batchId=233)
org.apache.hadoop.hive.metastore.TestHiveMetastoreTransformer.testTransformerAlterTable
(batchId=250)
org.apache.hadoop.hive.metastore.TestHiveMetastoreTransformer.testTransformerDatabase
(batchId=250)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterTable
(batchId=235)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testTableDatabase
(batchId=235)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testAlterTable
(batchId=239)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testTableDatabase
(batchId=239)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=234)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=234)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testAlterTable
(batchId=241)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testTableDatabase
(batchId=241)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=231)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableDatabase
(batchId=231)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterTable
(batchId=229)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testTableDatabase
(batchId=229)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterTable
(batchId=240)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testTableDatabase
(batchId=240)
org.apache.hadoop.hive.metastore.TestTenantBasedStorageHierarchy.testAlterDatabase
(batchId=254)
org.apache.hadoop.hive.metastore.TestTenantBasedStorageHierarchy.testCreateDatabaseWithExtAndManagedLocations
(batchId=254)
org.apache.hadoop.hive.metastore.TestTenantBasedStorageHierarchy.testCreateDatabaseWithNewLocation
(batchId=254)
org.apache.hadoop.hive.metastore.TestTenantBasedStorageHierarchy.testCreateDatabaseWithOldLocation
(batchId=254)
org.apache.hadoop.hive.metastore.client.TestDatabases.testCreateDatabaseDefaultValues[Embedded]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestDatabases.testCreateDatabaseDefaultValues[Remote]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestExchangePartitions.testExchangePartitionNonExistingPartLocation[Embedded]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestExchangePartitions.testExchangePartitionNonExistingPartLocation[Remote]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestExchangePartitions.testExchangePartitionsNonExistingPartLocation[Embedded]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestExchangePartitions.testExchangePartitionsNonExistingPartLocation[Remote]
(batchId=232)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Embedded]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableChangingDatabase[Embedded]
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableChangingDatabase[Remote]
(batchId=230)
org.apache.hadoop.hive.metastore.tools.schematool.TestSchemaToolForMetastore.testValidateLocations
(batchId=234)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=406018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406018
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 19/Mar/20 09:17
Start Date: 19/Mar/20 09:17
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r394885186
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -193,20 +193,29 @@ private Long getEventFromPreviousDumpMetadata(Path
previousDumpPath) throws Sema
}
private Path getPreviousDumpMetadataPath(Path dumpRoot) throws IOException {
+FileStatus latestUpdatedStatus = null;
FileSystem fs = dumpRoot.getFileSystem(conf);
if (fs.exists(dumpRoot)) {
FileStatus[] statuses = fs.listStatus(dumpRoot);
if (statuses.length > 0) {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406018)
Time Spent: 6h 10m (was: 6h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.2.patch, HIVE-22997.4.patch,
> HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch,
> HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=406019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-406019
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 19/Mar/20 09:17
Start Date: 19/Mar/20 09:17
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r394307271
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -662,17 +690,48 @@ void dumpTable(String dbName, String tblName, String
validTxnList, Path dbRoot,
replLogger.tableLog(tblName, tableSpec.tableHandle.getTableType());
if (tableSpec.tableHandle.getTableType().equals(TableType.EXTERNAL_TABLE)
|| Utils.shouldDumpMetaDataOnly(conf)) {
- return;
+ return Collections.EMPTY_LIST;
+}
+return replPathMappings;
+ }
+
+ private void intitiateDataCopyTasks() {
+Iterator extCopyWorkItr =
work.getDirCopyIterator();
+ReplOperationCompleteAckWork replDumpCompleteAckWork = new
ReplOperationCompleteAckWork(dumpAckFile);
+Task dumpCompleteAckWorkTask =
TaskFactory.get(replDumpCompleteAckWork, conf);
+List> childTasks = new ArrayList<>();
+int maxTasks =
conf.getIntVar(HiveConf.ConfVars.REPL_APPROX_MAX_LOAD_TASKS);
+TaskTracker taskTracker = new TaskTracker(maxTasks);
+while (taskTracker.canAddMoreTasks() && hasMoreCopyWork()) {
+ if (work.replPathIteratorInitialized() && extCopyWorkItr.hasNext()) {
+childTasks.addAll(new ExternalTableCopyTaskBuilder(work,
conf).tasks(taskTracker));
+ } else {
+childTasks.addAll(ReplPathMapping.tasks(work, taskTracker, conf));
+ }
}
-for (ReplPathMapping replPathMapping: replPathMappings) {
- Task copyTask = ReplCopyTask.getLoadCopyTask(
- tuple.replicationSpec, replPathMapping.getSrcPath(),
replPathMapping.getTargetPath(), conf, false);
- this.addDependentTask(copyTask);
- LOG.info("Scheduled a repl copy task from [{}] to [{}]",
- replPathMapping.getSrcPath(), replPathMapping.getTargetPath());
+if (!childTasks.isEmpty()) {
+ boolean ackTaskAdded = false;
+ if (taskTracker.canAddMoreTasks()) {
+childTasks.add(dumpCompleteAckWorkTask);
+ackTaskAdded = true;
+ }
+ if (hasMoreCopyWork() || !ackTaskAdded) {
+DAGTraversal.traverse(childTasks, new
AddDependencyToLeaves(TaskFactory.get(work, conf)));
Review comment:
Ack is being added at the end in these tasks as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 406019)
Time Spent: 6h 20m (was: 6h 10m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.2.patch, HIVE-22997.4.patch,
> HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch,
> HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22940) Make the datasketches functions available as predefined functions
[ https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22940: Attachment: HIVE-22940.05.patch > Make the datasketches functions available as predefined functions > -- > > Key: HIVE-22940 > URL: https://issues.apache.org/jira/browse/HIVE-22940 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch, > HIVE-22940.03.patch, HIVE-22940.04.patch, HIVE-22940.05.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22126: --- Assignee: Eugene Chung (was: Zoltan Haindrich) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23051) Clean up BucketCodec
[
https://issues.apache.org/jira/browse/HIVE-23051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062382#comment-17062382
]
László Bodor commented on HIVE-23051:
-
[~belugabehr]: could you please make some extra efforts to introduce a
TestBucketCodec for unit testing at least "encode" method for different
scenarios/codecs (even the statementId<0 case)? it would be great to have a
test class like that, reflecting the expected behavior of BucketCodec
> Clean up BucketCodec
>
>
> Key: HIVE-23051
> URL: https://issues.apache.org/jira/browse/HIVE-23051
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23051.1.patch
>
>
> A couple of nagging things caught my eye with this class. The first thing:
> {code:java|title=BucketCodec.java}
> int statementId = options.getStatementId() >= 0 ?
> options.getStatementId() : 0;
> assert this.version >=0 && this.version <= MAX_VERSION
> : "Version out of range: " + version;
> if(!(options.getBucketId() >= 0 && options.getBucketId() <=
> MAX_BUCKET_ID)) {
> throw new IllegalArgumentException("bucketId out of range: " +
> options.getBucketId());
> }
> if(!(statementId >= 0 && statementId <= MAX_STATEMENT_ID)) {
> throw new IllegalArgumentException("statementId out of range: " +
> statementId);
> }
> {code}
> {{statementId}} gets capped, if it's less than 0, then it gets rounded up to
> 0. However, it late checks that the {{statementId}} is greater,... which is
> will always be since it's getting rounded.
> # Remove the rounding behavior.
> # Make better error message
> # Fail-fast in the constructor if the version is invalid
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22126: Attachment: HIVE-22126.09.patch > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-22126: --- Assignee: Zoltan Haindrich (was: Eugene Chung) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062380#comment-17062380 ] Zoltan Haindrich commented on HIVE-22126: - I bumped into this issue again - from some other side * if I happen to add a dependency the qtest-accumulo module also needs it to be added as a depand that's completely unexpected - I see it dates back around 5 years - and that this patch also starts fixing it... * the schq_ingest was most likely unrelated ; I think I've landed a patch since then to stabilize it * I suspect TestJdbcWithMiniHS2 was also bumping into sme temporary issue.. fixing this could probably also help in the future as well; I'll reattach the patch to see if those were flaky tests or not.. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch, HIVE-22126.02.patch, > HIVE-22126.03.patch, HIVE-22126.04.patch, HIVE-22126.05.patch, > HIVE-22126.06.patch, HIVE-22126.07.patch, HIVE-22126.08.patch, > HIVE-22126.09.patch, HIVE-22126.09.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22995) Add support for location for managed tables on database
[
https://issues.apache.org/jira/browse/HIVE-22995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062362#comment-17062362
]
Hive QA commented on HIVE-22995:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 15s{color}
| {color:red}
/data/hiveptest/logs/PreCommit-HIVE-Build-21174/patches/PreCommit-HIVE-Build-21174.patch
does not apply to master. Rebase required? Wrong Branch? See
http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21174/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add support for location for managed tables on database
> ---
>
> Key: HIVE-22995
> URL: https://issues.apache.org/jira/browse/HIVE-22995
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22995.1.patch, HIVE-22995.2.patch,
> HIVE-22995.3.patch, Hive Metastore Support for Tenant-based storage
> heirarchy.pdf
>
>
> I have attached the initial spec to this jira.
> Default location for database would be the external table base directory.
> Managed location can be optionally specified.
> {code}
> CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
> [COMMENT database_comment]
> [LOCATION hdfs_path]
> [MANAGEDLOCATION hdfs_path]
> [WITH DBPROPERTIES (property_name=property_value, ...)];
> ALTER (DATABASE|SCHEMA) database_name SET
> MANAGEDLOCATION
> hdfs_path;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062360#comment-17062360
]
Hive QA commented on HIVE-22997:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997074/HIVE-22997.15.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18124 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21173/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21173/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21173/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997074 - PreCommit-HIVE-Build
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
> HIVE-22997.12.patch, HIVE-22997.13.patch, HIVE-22997.14.patch,
> HIVE-22997.15.patch, HIVE-22997.2.patch, HIVE-22997.4.patch,
> HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch,
> HIVE-22997.8.patch, HIVE-22997.9.patch
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[ https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23002: Fix Version/s: 4.0.0 Assignee: Rajesh Balamohan Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thanks [~ashutoshc] > Optimise LazyBinaryUtils.writeVLong > --- > > Key: HIVE-23002 > URL: https://issues.apache.org/jira/browse/HIVE-23002 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-23002.1.patch, HIVE-23002.2.patch, > HIVE-23002.3.patch, Screenshot 2020-03-10 at 5.01.34 AM.jpg > > > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420] > It would be good to add a method which accepts scratch bytes. > > !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23042) Merge queries to a single one for updating MIN_OPEN_TXNS table
[
https://issues.apache.org/jira/browse/HIVE-23042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23042:
--
Attachment: HIVE-23042.2.patch
> Merge queries to a single one for updating MIN_OPEN_TXNS table
> --
>
> Key: HIVE-23042
> URL: https://issues.apache.org/jira/browse/HIVE-23042
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23042.2.patch, HIVE-23042.patch
>
>
> When opening a new transaction we issue 2 queries to update the MIN_OPEN_TXN
> table.
> {code}
>
> values(763, 763)>
> {code}
> This could be archived with a single query faster, if we do not open
> transactions in batch, like:
> {code}
>SELECT ?, MIN("TXN_ID") FROM "TXNS" WHERE "TXN_STATE" = 'o'>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?focusedWorklogId=405972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-405972 ] ASF GitHub Bot logged work on HIVE-23045: - Author: ASF GitHub Bot Created on: 19/Mar/20 07:24 Start Date: 19/Mar/20 07:24 Worklog Time Spent: 10m Work Description: pvargacl commented on pull request #958: HIVE-23045: Zookeeper SSL/TLS support URL: https://github.com/apache/hive/pull/958 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 405972) Remaining Estimate: 0h Time Spent: 10m > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-23045: -- Labels: pull-request-available (was: ) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062339#comment-17062339
]
Hive QA commented on HIVE-22997:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} ql: The patch generated 0 new + 79 unchanged - 2
fixed = 79 total (was 81) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 649
unchanged - 1 fixed = 649 total (was 650) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
53s{color} | {color:red} ql generated 1 new + 1530 unchanged - 1 fixed = 1531
total (was 1531) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.exec.repl.ReplDumpWork is Serializable;
consider declaring a serialVersionUID At ReplDumpWork.java:a serialVersionUID
At ReplDumpWork.java:[lines 41-140] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21173/dev-support/hive-personality.sh
|
| git revision | master / 6f9ae63 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21173/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21173/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.10.patch, HIVE-22997.11.patch,
>
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: Patch Available (was: Open) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Status: Open (was: Patch Available) > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23045) Zookeeper SSL/TLS support
[ https://issues.apache.org/jira/browse/HIVE-23045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23045: --- Attachment: HIVE-23045.4.patch > Zookeeper SSL/TLS support > - > > Key: HIVE-23045 > URL: https://issues.apache.org/jira/browse/HIVE-23045 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, Metastore >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Critical > Attachments: HIVE-23045.1.patch, HIVE-23045.2.patch, > HIVE-23045.3.patch, HIVE-23045.4.patch > > > Zookeeper 3.5.5 server can operate with SSL/TLS secure connection with its > clients. > [https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide] > The SSL communication should be possible in the different part of HIVE, where > it communicates with Zookeeper servers. The Zookeeper clients are used in the > following places: > * HiveServer2 PrivilegeSynchronizer > * HiveServer2 register/remove server from Zookeeper > * HS2ActivePassiveHARegistryClient > * ZooKeeperHiveLockManager > * LLapZookeeperRegistryImpl > * TezAmRegistryImpl > * WebHCat ZooKeeperStorage > * JDBC Driver server lookup > * Metastore - ZookeeperTokenStore > * Metastore register/remove server from Zookeeper > The flag to enable SSL communication and the required parameters should be > provided by different configuration parameters, corresponding the different > use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23047) Calculate the epoch on DB side
[ https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062338#comment-17062338 ] Peter Vary commented on HIVE-23047: --- Fixed the new checkstyle errors introduced by me. Did not change the case indentation since the whole file uses the same. > Calculate the epoch on DB side > -- > > Key: HIVE-23047 > URL: https://issues.apache.org/jira/browse/HIVE-23047 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23047.2.patch, HIVE-23047.patch > > > We use TxnHandler.getDbTime to calculate the epoch on the DB server, and > immediately insert the value back again. We would be better of by using sql > to calculate the value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23047) Calculate the epoch on DB side
[ https://issues.apache.org/jira/browse/HIVE-23047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23047: -- Attachment: HIVE-23047.2.patch > Calculate the epoch on DB side > -- > > Key: HIVE-23047 > URL: https://issues.apache.org/jira/browse/HIVE-23047 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-23047.2.patch, HIVE-23047.patch > > > We use TxnHandler.getDbTime to calculate the epoch on the DB server, and > immediately insert the value back again. We would be better of by using sql > to calculate the value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23016) Extract JdbcConnectionParams from Utils Class
[ https://issues.apache.org/jira/browse/HIVE-23016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062323#comment-17062323 ] Anirban Ghosh commented on HIVE-23016: -- Hi! I'd like to take this up if no one else is working on this already. > Extract JdbcConnectionParams from Utils Class > - > > Key: HIVE-23016 > URL: https://issues.apache.org/jira/browse/HIVE-23016 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Priority: Minor > Labels: n00b, newbie, noob > > And make it its own class. > https://github.com/apache/hive/blob/4700e210ef7945278c4eb313c9ebd810b0224da1/jdbc/src/java/org/apache/hive/jdbc/Utils.java#L72 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062319#comment-17062319
]
Hive QA commented on HIVE-23032:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12997072/HIVE-23032.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 18123 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query21]
(batchId=306)
org.apache.hadoop.hive.metastore.security.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
(batchId=299)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21172/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21172/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21172/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12997072 - PreCommit-HIVE-Build
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
