[jira] [Comment Edited] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530818#comment-16530818 ] Sankar Hariappan edited comment on HIVE-17840 at 7/3/18 5:24 AM: - Thanks for the review [~maheshk114] and [~daijy]! 02.patch is committed to master/branch-3. was (Author: sankarh): Thanks for the review [~maheshk114] and [~daijy]! 02.patch is committed to master. > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Resolution: Fixed Status: Resolved (was: Patch Available) > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Fix Version/s: 3.2.0 > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18118) Explain Extended should indicate if a file being read is an EC file
[
https://issues.apache.org/jira/browse/HIVE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530837#comment-16530837
]
Hive QA commented on HIVE-18118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929973/HIVE-18118.14.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 14653 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=191)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=262)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=262)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12339/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12339/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12339/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929973 - PreCommit-HIVE-Build
> Explain Extended should indicate if a file being read is an EC file
> ---
>
> Key: HIVE-18118
> URL: https://issues.apache.org/jira/browse/HIVE-18118
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18118.1.patch, HIVE-18118.10.patch,
> HIVE-18118.10.patch, HIVE-18118.11.patch, HIVE-18118.11.patch,
> HIVE-18118.12.patch, HIVE-18118.14.patch, HIVE-18118.2.patch,
> HIVE-18118.3.patch, HIVE-18118.4.patch, HIVE-18118.5.patch,
> HIVE-18118.6.patch, HIVE-18118.7.patch, HIVE-18118.8.patch, HIVE-18118.9.patch
>
>
> We already print out the files Hive will read in the explain extended
> command, we just have to modify it to say whether or not its an EC file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18118) Explain Extended should indicate if a file being read is an EC file
[
https://issues.apache.org/jira/browse/HIVE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530832#comment-16530832
]
Hive QA commented on HIVE-18118:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
4s{color} | {color:blue} standalone-metastore in master has 228 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} common: The patch generated 0 new + 96 unchanged - 2
fixed = 96 total (was 98) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} standalone-metastore: The patch generated 0 new +
455 unchanged - 12 fixed = 455 total (was 467) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} ql: The patch generated 0 new + 499 unchanged - 35
fixed = 499 total (was 534) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} standalone-metastore generated 0 new + 226 unchanged
- 2 fixed = 226 total (was 228) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 13s{color} |
{color:black} {color} |
\\
[jira] [Updated] (HIVE-19267) Create/Replicate ACID Write event
[ https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19267: --- Attachment: (was: HIVE-19267.23.patch) > Create/Replicate ACID Write event > - > > Key: HIVE-19267 > URL: https://issues.apache.org/jira/browse/HIVE-19267 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Attachments: HIVE-19267.01-branch-3.patch, HIVE-19267.01.patch, > HIVE-19267.02.patch, HIVE-19267.03.patch, HIVE-19267.04.patch, > HIVE-19267.05.patch, HIVE-19267.06.patch, HIVE-19267.07.patch, > HIVE-19267.08.patch, HIVE-19267.09.patch, HIVE-19267.10.patch, > HIVE-19267.11.patch, HIVE-19267.12.patch, HIVE-19267.13.patch, > HIVE-19267.14.patch, HIVE-19267.15.patch, HIVE-19267.16.patch, > HIVE-19267.17.patch, HIVE-19267.18.patch, HIVE-19267.19.patch, > HIVE-19267.20.patch, HIVE-19267.21.patch, HIVE-19267.22.patch > > > > h1. Replicate ACID write Events > * Create new EVENT_WRITE event with related message format to log the write > operations with in a txn along with data associated. > * Log this event when perform any writes (insert into, insert overwrite, > load table, delete, update, merge, truncate) on table/partition. > * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple > partitions, then need to log one event per partition. > * DbNotificationListener should log this type of event to special metastore > table named "MTxnWriteNotificationLog". > * This table should maintain a map of txn ID against list of > tables/partitions written by given txn. > * The entry for a given txn should be removed by the cleaner thread that > removes the expired events from EventNotificationTable. > h1. Replicate Commit Txn operation (with writes) > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" > metastore table to consolidate the list of tables/partitions modified within > this txn scope. > * Based on the list of tables/partitions modified and table Write ID, need > to compute the list of delta files added by this txn. > * Repl dump should read this message and dump the metadata and delta files > list. > *Target warehouse:* > * Ensure snapshot isolation at target for on-going read txns which shouldn't > view the data replicated from committed txn. (Ensured with open and allocate > write ID events). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Status: Open (was: Patch Available) > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Status: Patch Available (was: Open) > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Attachment: HIVE-17840.01-branch-3.patch > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530818#comment-16530818 ] Sankar Hariappan commented on HIVE-17840: - Thanks for the review [~maheshk114] and [~daijy]! 02.patch is committed to master. > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-17840.01-branch-3.patch, HIVE-17840.01.patch, > HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-17840: Fix Version/s: 4.0.0 > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-17840.01.patch, HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20051) Skip authorization for temp tables
[ https://issues.apache.org/jira/browse/HIVE-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530809#comment-16530809 ] Jason Dere commented on HIVE-20051: --- +1 > Skip authorization for temp tables > -- > > Key: HIVE-20051 > URL: https://issues.apache.org/jira/browse/HIVE-20051 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20051.01.patch, HIVE-20051.02.patch, > HIVE-20051.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530804#comment-16530804 ] Daniel Dai commented on HIVE-17840: --- +1 > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17840.01.patch, HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20051) Skip authorization for temp tables
[
https://issues.apache.org/jira/browse/HIVE-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530798#comment-16530798
]
Hive QA commented on HIVE-20051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929972/HIVE-20051.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14652 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12338/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12338/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12338/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929972 - PreCommit-HIVE-Build
> Skip authorization for temp tables
> --
>
> Key: HIVE-20051
> URL: https://issues.apache.org/jira/browse/HIVE-20051
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20051.01.patch, HIVE-20051.02.patch,
> HIVE-20051.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[ https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530793#comment-16530793 ] mahesh kumar behera commented on HIVE-17840: [~sankarh] Code changes looks fine to me > HiveMetaStore eats exception if transactionalListeners.notifyEvent fail > --- > > Key: HIVE-17840 > URL: https://issues.apache.org/jira/browse/HIVE-17840 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-17840.01.patch, HIVE-17840.02.patch > > > For example, in add_partitions_core, if there's exception in > MetaStoreListenerNotifier.notifyEvent(transactionalListeners,), > transaction rollback but no exception thrown. Client will assume add > partition is successful and take a positive path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530789#comment-16530789 ] Sankar Hariappan commented on HIVE-20062: - +1 Thanks [~sershe] for fixing this! > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20062.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20051) Skip authorization for temp tables
[
https://issues.apache.org/jira/browse/HIVE-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530776#comment-16530776
]
Hive QA commented on HIVE-20051:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 5 new + 12
unchanged - 0 fixed = 17 total (was 12) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12338/dev-support/hive-personality.sh
|
| git revision | master / 1fc1221 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12338/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12338/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Skip authorization for temp tables
> --
>
> Key: HIVE-20051
> URL: https://issues.apache.org/jira/browse/HIVE-20051
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20051.01.patch, HIVE-20051.02.patch,
> HIVE-20051.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20046) remove NUM_FILES check or add a negative test
[
https://issues.apache.org/jira/browse/HIVE-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-20046:
---
Assignee: (was: Sergey Shelukhin)
> remove NUM_FILES check or add a negative test
> -
>
> Key: HIVE-20046
> URL: https://issues.apache.org/jira/browse/HIVE-20046
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> // Since newly initialized empty table has 0 for the parameter.
> if (Long.parseLong(statsParams.get(StatsSetupConst.NUM_FILES)) == 0) {
> return true;
> }
> {noformat}
> This doesn't look safe; # of files could be set to 0 by an invalid update, or
> potentially a parallel update that we cannot see (not sure if this is
> possible; there's some code in metastore that updates basic stats outside of
> the scope of the query).
> It would be better to remove this, and see if it breaks some tests. If we do
> need this, there should be a negative test at some point
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20062: Status: Patch Available (was: Open) > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20062.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20018:
Description:
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
(batchId=238)
{noformat}
Most tests have errors, not being able to find a table/partition in
getPartition. Might be related to catalog handling, or something else.
Some have NPEs, some no obvious errors.
Update: the only two real failures are
{noformat}
[ERROR] TestReplicationScenarios.testBootstrapWithConcurrentDropTable:462
expected: but was:
[ERROR] TestReplicationScenarios.testBootstrapWithDropPartitionedTable:687
expected: but was:
{noformat}
caused by injection failures due to signature changes. Will fix after
HIVE-20062 and HIVE-19975
was:
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
(batchId=238)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
(batchId=238)
{noformat}
Most tests have errors, not being able to find a table/partition in
getPartition. Might be related to catalog handling, or something else.
Some have NPEs, some no obvious errors.
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
> Update: the only two real failures are
> {noformat}
> [ERROR] TestReplicationScenarios.testBootstrapWithConcurrentDropTable:462
> expected: but was:
> [ERROR] TestReplicationScenarios.testBootstrapWithDropPartitionedTable:687
> expected: but was:
> {noformat}
> caused by injection failures due to signature changes. Will fix after
> HIVE-20062 and HIVE-19975
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20018 started by Sergey Shelukhin.
---
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530768#comment-16530768 ] Sergey Shelukhin commented on HIVE-20062: - [~thejas] [~sankarh] can you take a look? this makes sure injection is reset even if the test fails > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20062.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-20062: Attachment: HIVE-20062.patch > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20062.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20062) TestReplicationScenarios doesn't clean injection properly, causing bizarre interdependent failures
[ https://issues.apache.org/jira/browse/HIVE-20062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20062: --- > TestReplicationScenarios doesn't clean injection properly, causing bizarre > interdependent failures > -- > > Key: HIVE-20062 > URL: https://issues.apache.org/jira/browse/HIVE-20062 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19267) Create/Replicate ACID Write event
[
https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530757#comment-16530757
]
Hive QA commented on HIVE-19267:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929970/HIVE-19267.22.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14652 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12337/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12337/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12337/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929970 - PreCommit-HIVE-Build
> Create/Replicate ACID Write event
> -
>
> Key: HIVE-19267
> URL: https://issues.apache.org/jira/browse/HIVE-19267
> Project: Hive
> Issue Type: Sub-task
> Components: repl, Transactions
>Affects Versions: 3.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: ACID, DR, pull-request-available, replication
> Attachments: HIVE-19267.01-branch-3.patch, HIVE-19267.01.patch,
> HIVE-19267.02.patch, HIVE-19267.03.patch, HIVE-19267.04.patch,
> HIVE-19267.05.patch, HIVE-19267.06.patch, HIVE-19267.07.patch,
> HIVE-19267.08.patch, HIVE-19267.09.patch, HIVE-19267.10.patch,
> HIVE-19267.11.patch, HIVE-19267.12.patch, HIVE-19267.13.patch,
> HIVE-19267.14.patch, HIVE-19267.15.patch, HIVE-19267.16.patch,
> HIVE-19267.17.patch, HIVE-19267.18.patch, HIVE-19267.19.patch,
> HIVE-19267.20.patch, HIVE-19267.21.patch, HIVE-19267.22.patch,
> HIVE-19267.23.patch
>
>
>
> h1. Replicate ACID write Events
> * Create new EVENT_WRITE event with related message format to log the write
> operations with in a txn along with data associated.
> * Log this event when perform any writes (insert into, insert overwrite,
> load table, delete, update, merge, truncate) on table/partition.
> * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple
> partitions, then need to log one event per partition.
> * DbNotificationListener should log this type of event to special metastore
> table named "MTxnWriteNotificationLog".
> * This table should maintain a map of txn ID against list of
> tables/partitions written by given txn.
> * The entry for a given txn should be removed by the cleaner thread that
> removes the expired events from EventNotificationTable.
> h1. Replicate Commit Txn operation (with writes)
> Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions
> modified within the txn.
> *Source warehouse:*
> * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog"
> metastore table to consolidate the list of tables/partitions modified within
> this txn scope.
> * Based on the list of tables/partitions modified and table Write ID, need
> to compute the list of delta files added by this txn.
> * Repl dump should read this message and dump the metadata and delta files
> list.
> *Target warehouse:*
> * Ensure snapshot isolation at target for on-going read txns which shouldn't
> view the data replicated from committed txn. (Ensured with open and allocate
> write ID events).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19267) Create/Replicate ACID Write event
[
https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530746#comment-16530746
]
Hive QA commented on HIVE-19267:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
58s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/server-extensions in master has 4 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} standalone-metastore: The patch generated 7 new + 2187
unchanged - 6 fixed = 2194 total (was 2193) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 6 new + 1246 unchanged - 19
fixed = 1252 total (was 1265) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/server-extensions: The patch generated 0
new + 4 unchanged - 3 fixed = 4 total (was 7) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 38 new + 654
unchanged - 5 fixed = 692 total (was 659) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
28s{color} | {color:red} ql generated 1 new + 2285 unchanged - 1 fixed = 2286
total (was 2286) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red} hcatalog/server-extensions generated 1 new + 2
unchanged - 2 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Found reliance on default encoding in
org.apache.hadoop.hive.ql.parse.repl.dump.events.CommitTxnHandler.writer(EventHandler$Context,
Path):in
org.apache.hadoop.hive.ql.parse.repl.dump.events.CommitTxnHandler.writer(EventHandler$Context,
Path): new java.io.OutputStreamWriter(OutputStream) At
CommitTxnHandler.java:[line 52] |
| FindBugs |
[jira] [Comment Edited] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530740#comment-16530740
]
Sergey Shelukhin edited comment on HIVE-20018 at 7/3/18 2:42 AM:
-
Takes 20 minutes to run this test, but I suspect injection thing doesn't get
called because of the signature changes in the patch, then one test fails
verifying that the injection was called; then all the subsequent tests fail
because the injection thing is still sitting there, doing random things like
dropping tables on getTable call.
Would be easier to fix after partition patch where I remove all the spurious
overloads. I will verify that this is actually the case and fix this in master
(the test interdependency); then as part of/after partition patch, fix the
injections in the test to the final version of the methods.
was (Author: sershe):
Takes 20 minutes to run this test, but I suspect injection thing doesn't get
called because of the signature changes in the patch, then one test fails
verifying that the injection was called; then all the subsequent tests fail
because the injection thing is still sitting there.
Would be easier to fix after partition patch where I remove all the spurious
overloads. I will verify that this is actually the case and fix this in master
(the test interdependency); then as part of/after partition patch, fix the
injections in the test to the final version of the methods.
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530740#comment-16530740
]
Sergey Shelukhin commented on HIVE-20018:
-
Takes 20 minutes to run this test, but I suspect injection thing doesn't get
called because of the signature changes in the patch, then one test fails
verifying that the injection was called; then all the subsequent tests fail
because the injection thing is still sitting there.
Would be easier to fix after partition patch where I remove all the spurious
overloads. I will verify that this is actually the case and fix this in master
(the test interdependency); then as part of/after partition patch, fix the
injections in the test to the final version of the methods.
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19267) Create/Replicate ACID Write event
[ https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19267: --- Attachment: HIVE-19267.23.patch > Create/Replicate ACID Write event > - > > Key: HIVE-19267 > URL: https://issues.apache.org/jira/browse/HIVE-19267 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Attachments: HIVE-19267.01-branch-3.patch, HIVE-19267.01.patch, > HIVE-19267.02.patch, HIVE-19267.03.patch, HIVE-19267.04.patch, > HIVE-19267.05.patch, HIVE-19267.06.patch, HIVE-19267.07.patch, > HIVE-19267.08.patch, HIVE-19267.09.patch, HIVE-19267.10.patch, > HIVE-19267.11.patch, HIVE-19267.12.patch, HIVE-19267.13.patch, > HIVE-19267.14.patch, HIVE-19267.15.patch, HIVE-19267.16.patch, > HIVE-19267.17.patch, HIVE-19267.18.patch, HIVE-19267.19.patch, > HIVE-19267.20.patch, HIVE-19267.21.patch, HIVE-19267.22.patch, > HIVE-19267.23.patch > > > > h1. Replicate ACID write Events > * Create new EVENT_WRITE event with related message format to log the write > operations with in a txn along with data associated. > * Log this event when perform any writes (insert into, insert overwrite, > load table, delete, update, merge, truncate) on table/partition. > * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple > partitions, then need to log one event per partition. > * DbNotificationListener should log this type of event to special metastore > table named "MTxnWriteNotificationLog". > * This table should maintain a map of txn ID against list of > tables/partitions written by given txn. > * The entry for a given txn should be removed by the cleaner thread that > removes the expired events from EventNotificationTable. > h1. Replicate Commit Txn operation (with writes) > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" > metastore table to consolidate the list of tables/partitions modified within > this txn scope. > * Based on the list of tables/partitions modified and table Write ID, need > to compute the list of delta files added by this txn. > * Repl dump should read this message and dump the metadata and delta files > list. > *Target warehouse:* > * Ensure snapshot isolation at target for on-going read txns which shouldn't > view the data replicated from committed txn. (Ensured with open and allocate > write ID events). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530731#comment-16530731
]
Sergey Shelukhin commented on HIVE-20018:
-
I think this might be due to horrible injection usage in the test, the tests
use statics to override behaviors and then never reset on failure. So if one
injecting test fails, tests after it may fail.
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20061) add a flag to turn off txn stats
[ https://issues.apache.org/jira/browse/HIVE-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-20061: --- Assignee: Sergey Shelukhin > add a flag to turn off txn stats > > > Key: HIVE-20061 > URL: https://issues.apache.org/jira/browse/HIVE-20061 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junjie Chen updated HIVE-17593: --- Attachment: HIVE-17593.3.patch > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.3.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530682#comment-16530682
]
Alice Fan commented on HIVE-19933:
--
Added results of MySQL and Oracle when dropping an non-existing partition:
MySQL
{code:java}
mysql> ALTER TABLE trb3 TRUNCATE PARTITION part1;
ERROR 1735 (HY000): Unknown partition 'part1' in table 'trb3'
{code}
Oracle
{code:java}
Alter table Q1_SALES_BY_REGION drop partition notexistingpart;
ORA-02149: Specified partition does not exist
{code}
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20052) Arrow serde should fill ArrowColumnVector(Decimal) with the given schema precision/scale
[
https://issues.apache.org/jira/browse/HIVE-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530678#comment-16530678
]
Hive QA commented on HIVE-20052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929971/HIVE-20052.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14651 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12336/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12336/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12336/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929971 - PreCommit-HIVE-Build
> Arrow serde should fill ArrowColumnVector(Decimal) with the given schema
> precision/scale
>
>
> Key: HIVE-20052
> URL: https://issues.apache.org/jira/browse/HIVE-20052
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20052.patch
>
>
> Arrow serde should fill ArrowColumnVector with given precision and scale.
> When it serializes negative values into Arrow, it throws exceptions that the
> precision of the value is not same with the precision of Arrow decimal vector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20052) Arrow serde should fill ArrowColumnVector(Decimal) with the given schema precision/scale
[
https://issues.apache.org/jira/browse/HIVE-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530655#comment-16530655
]
Hive QA commented on HIVE-20052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 25 new + 351 unchanged - 11
fixed = 376 total (was 362) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12336/dev-support/hive-personality.sh
|
| git revision | master / 1fc1221 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12336/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12336/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Arrow serde should fill ArrowColumnVector(Decimal) with the given schema
> precision/scale
>
>
> Key: HIVE-20052
> URL: https://issues.apache.org/jira/browse/HIVE-20052
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20052.patch
>
>
> Arrow serde should fill ArrowColumnVector with given precision and scale.
> When it serializes negative values into Arrow, it throws exceptions that the
> precision of the value is not same with the precision of Arrow decimal vector.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17751) Separate HMS Client and HMS server into separate sub-modules
[ https://issues.apache.org/jira/browse/HIVE-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-17751: -- Attachment: HIVE-17751.08.patch > Separate HMS Client and HMS server into separate sub-modules > > > Key: HIVE-17751 > URL: https://issues.apache.org/jira/browse/HIVE-17751 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-17751.01.patch, HIVE-17751.02.patch, > HIVE-17751.03.patch, HIVE-17751.04.patch, > HIVE-17751.06-standalone-metastore.patch, HIVE-17751.07.patch, > HIVE-17751.08.patch > > > external applications which are interfacing with HMS should ideally only > include HMSClient library instead of one big library containing server as > well. We should ideally have a thin client library so that cross version > support for external applications is easier. We should sub-divide the > standalone module into possibly 3 modules (one for common classes, one for > client classes and one for server) or 2 sub-modules (one for client and one > for server) so that we can generate separate jars for HMS client and server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530654#comment-16530654
]
Alice Fan commented on HIVE-19933:
--
Adding additional Information:
By default, hive.exec.drop.ignorenonexistent is set to True. With that, when
users accidentally drop NON-existing table, view, or partition, it will not
throw exception. It will just give an INFO level message like: Returning Hive
schema: Schema(fieldSchemas:null, properties:null). In my opinion, even though
we keep the flag, it should be set to False by default. It seems weird that
Hive provides "INFO : OK" when user drop non-existing table/view/partition.
An drop non-existing table example:
{code:java}
drop view table_not_existing;
INFO : Compiling
command(queryId=hive_20180702175252_8577258a-d68e-4922-b4ba-b317eff9fda4): drop
view table_not_existing
INFO : Semantic Analysis Completed
{color:red}INFO : Returning Hive schema: Schema(fieldSchemas:null,
properties:null){color}
INFO : Completed compiling
command(queryId=hive_20180702175252_8577258a-d68e-4922-b4ba-b317eff9fda4); Time
taken: 0.002 seconds
INFO : Executing
command(queryId=hive_20180702175252_8577258a-d68e-4922-b4ba-b317eff9fda4): drop
view table_not_existing
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20180702175252_8577258a-d68e-4922-b4ba-b317eff9fda4); Time
taken: 0.003 seconds
INFO : OK
{code}
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junjie Chen updated HIVE-17593: --- Comment: was deleted (was: [~Ferd], I updated qtest result, do you know why it still failed? ) > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[ https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530650#comment-16530650 ] Junjie Chen commented on HIVE-17593: [~Ferd], I updated qtest result, do you know why it still failed? > DataWritableWriter strip spaces for CHAR type before writing, but predicate > generator doesn't do same thing. > > > Key: HIVE-17593 > URL: https://issues.apache.org/jira/browse/HIVE-17593 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.0, 3.0.0 >Reporter: Junjie Chen >Assignee: Junjie Chen >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-17593.2.patch, HIVE-17593.patch > > > DataWritableWriter strip spaces for CHAR type before writing. While when > generating predicate, it does NOT do same striping which should cause data > missing! > In current version, it doesn't cause data missing since predicate is not well > push down to parquet due to HIVE-17261. > Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as > same which will build a predicate with tail spaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20051) Skip authorization for temp tables
[ https://issues.apache.org/jira/browse/HIVE-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530642#comment-16530642 ] Jason Dere edited comment on HIVE-20051 at 7/3/18 12:54 AM: Seems like the patch is doing the right thing, but can you check the test failures? was (Author: jdere): Seems like the patch is doing the right thing, but can you check the failures? > Skip authorization for temp tables > -- > > Key: HIVE-20051 > URL: https://issues.apache.org/jira/browse/HIVE-20051 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20051.01.patch, HIVE-20051.02.patch, > HIVE-20051.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20051) Skip authorization for temp tables
[ https://issues.apache.org/jira/browse/HIVE-20051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530642#comment-16530642 ] Jason Dere commented on HIVE-20051: --- Seems like the patch is doing the right thing, but can you check the failures? > Skip authorization for temp tables > -- > > Key: HIVE-20051 > URL: https://issues.apache.org/jira/browse/HIVE-20051 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20051.01.patch, HIVE-20051.02.patch, > HIVE-20051.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[
https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530631#comment-16530631
]
Hive QA commented on HIVE-17840:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929967/HIVE-17840.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14650 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12335/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12335/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12335/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929967 - PreCommit-HIVE-Build
> HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
> ---
>
> Key: HIVE-17840
> URL: https://issues.apache.org/jira/browse/HIVE-17840
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17840.01.patch, HIVE-17840.02.patch
>
>
> For example, in add_partitions_core, if there's exception in
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,),
> transaction rollback but no exception thrown. Client will assume add
> partition is successful and take a positive path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18249) Remove Thrift dependency on fb303
[ https://issues.apache.org/jira/browse/HIVE-18249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-18249: -- Affects Version/s: (was: 3.0.0) 4.0.0 > Remove Thrift dependency on fb303 > - > > Key: HIVE-18249 > URL: https://issues.apache.org/jira/browse/HIVE-18249 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Labels: thrift > > Looks like we are not really using fb303 and can remove fb303 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18249) Remove Thrift dependency on fb303
[ https://issues.apache.org/jira/browse/HIVE-18249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530619#comment-16530619 ] Alexander Kolbasov commented on HIVE-18249: --- Hive itself isn't using fb303 interfaces. There is a concern that some 3-rd party might use these and might be broken by the change, so we can't remove it in the current branch. We should consider deprecating in in the next release. [~alangates] What do you think about this? > Remove Thrift dependency on fb303 > - > > Key: HIVE-18249 > URL: https://issues.apache.org/jira/browse/HIVE-18249 > Project: Hive > Issue Type: Bug > Components: Hive, Metastore >Affects Versions: 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Labels: thrift > > Looks like we are not really using fb303 and can remove fb303 dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19337) Partition whitelist regex doesn't work (and never did)
[
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19337:
--
Attachment: HIVE-19337.03.branch-2.patch
> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-19337.01.branch-2.patch,
> HIVE-19337.03.branch-2.patch
>
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
> Note that it uses name() method which returns enum name
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in
> Hive3 - probably when [~alangates] reworked access to configuration
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19337) Partition whitelist regex doesn't work (and never did)
[
https://issues.apache.org/jira/browse/HIVE-19337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19337:
--
Attachment: (was: HIVE-19337.02.branch-2.patch)
> Partition whitelist regex doesn't work (and never did)
> --
>
> Key: HIVE-19337
> URL: https://issues.apache.org/jira/browse/HIVE-19337
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.3
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-19337.01.branch-2.patch
>
>
> {{ObjectStore.setConf()}} has the following code:
> {code:java}
> String partitionValidationRegex =
>
> hiveConf.get(HiveConf.ConfVars.METASTORE_PARTITION_NAME_WHITELIST_PATTERN.name());
> {code}
> Note that it uses name() method which returns enum name
> (METASTORE_PARTITION_NAME_WHITELIST_PATTERN) rather then .varname
> As a result the regex will always be null.
> The code was introduced as part of
> HIVE-7223 Support generic PartitionSpecs in Metastore partition-functions
> So looks like this was broken since the original code drop. This is fixed in
> Hive3 - probably when [~alangates] reworked access to configuration
> (HIVE-17733) so it isn't a bug in Hive-3.
> [~stakiar_impala_496e] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17849) alterPartition() may fail to rollback transaction
[
https://issues.apache.org/jira/browse/HIVE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov reassigned HIVE-17849:
-
Assignee: (was: Alexander Kolbasov)
> alterPartition() may fail to rollback transaction
> -
>
> Key: HIVE-17849
> URL: https://issues.apache.org/jira/browse/HIVE-17849
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
>Reporter: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-17849.01-branch-2.3.patch
>
>
> In HiveAlterHandle.alterPartition() there is this code:
> {code}
> try {
> msdb.openTransaction();
> msdb.alterPartition(dbname, name, new_part.getValues(), oldPart);
> if (transactionalListeners != null &&
> !transactionalListeners.isEmpty()) {
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
>
> EventMessage.EventType.ALTER_PARTITION,
> new
> AlterPartitionEvent(new_part, oldPart, tbl, success, handler));
> }
> revertMetaDataTransaction = msdb.commitTransaction();
> } catch (Exception ex2) {
> LOG.error("Attempt to revert partition metadata change failed.
> The revert was attempted " +
> "because associated filesystem rename operation failed with
> exception " + ex.getMessage(), ex2);
> if (!revertMetaDataTransaction) {
> msdb.rollbackTransaction();
> }
> }
> {code}
> Note that there is no {{finally}} clause, so it is possible for some
> unchecked exception to occur in which case the transaction will remain
> active. Once this happens, all subsequent transactions on this thread will
> not behave correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17849) alterPartition() may fail to rollback transaction
[
https://issues.apache.org/jira/browse/HIVE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530611#comment-16530611
]
Alexander Kolbasov commented on HIVE-17849:
---
This isn't a problem in Hive3, only in Hive2
> alterPartition() may fail to rollback transaction
> -
>
> Key: HIVE-17849
> URL: https://issues.apache.org/jira/browse/HIVE-17849
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Attachments: HIVE-17849.01-branch-2.3.patch
>
>
> In HiveAlterHandle.alterPartition() there is this code:
> {code}
> try {
> msdb.openTransaction();
> msdb.alterPartition(dbname, name, new_part.getValues(), oldPart);
> if (transactionalListeners != null &&
> !transactionalListeners.isEmpty()) {
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
>
> EventMessage.EventType.ALTER_PARTITION,
> new
> AlterPartitionEvent(new_part, oldPart, tbl, success, handler));
> }
> revertMetaDataTransaction = msdb.commitTransaction();
> } catch (Exception ex2) {
> LOG.error("Attempt to revert partition metadata change failed.
> The revert was attempted " +
> "because associated filesystem rename operation failed with
> exception " + ex.getMessage(), ex2);
> if (!revertMetaDataTransaction) {
> msdb.rollbackTransaction();
> }
> }
> {code}
> Note that there is no {{finally}} clause, so it is possible for some
> unchecked exception to occur in which case the transaction will remain
> active. Once this happens, all subsequent transactions on this thread will
> not behave correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20039: -- Attachment: HIVE-20039.1.patch > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-20039.1.patch > > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19253) HMS ignores tableType property for external tables
[
https://issues.apache.org/jira/browse/HIVE-19253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated HIVE-19253:
--
Labels: newbie (was: )
> HMS ignores tableType property for external tables
> --
>
> Key: HIVE-19253
> URL: https://issues.apache.org/jira/browse/HIVE-19253
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.2, 3.0.0, 3.1.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Major
> Labels: newbie
>
> When someone creates a table using Thrift API they may think that setting
> tableType to {{EXTERNAL_TABLE}} creates an external table. And boom - their
> table is gone later because HMS will silently change it to managed table.
> here is the offending code:
> {code:java}
> private MTable convertToMTable(Table tbl) throws InvalidObjectException,
> MetaException {
> ...
> // If the table has property EXTERNAL set, update table type
> // accordingly
> String tableType = tbl.getTableType();
> boolean isExternal =
> Boolean.parseBoolean(tbl.getParameters().get("EXTERNAL"));
> if (TableType.MANAGED_TABLE.toString().equals(tableType)) {
> if (isExternal) {
> tableType = TableType.EXTERNAL_TABLE.toString();
> }
> }
> if (TableType.EXTERNAL_TABLE.toString().equals(tableType)) {
> if (!isExternal) { // Here!
> tableType = TableType.MANAGED_TABLE.toString();
> }
> }
> {code}
> So if the EXTERNAL parameter is not set, table type is changed to managed
> even if it was external in the first place - which is wrong.
> More over, in other places code looks at the table property to decide table
> type and some places look at parameter. HMS should really make its mind which
> one to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20040) JDBC: HTTP listen queue is 50 and SYNs are lost
[
https://issues.apache.org/jira/browse/HIVE-20040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530604#comment-16530604
]
Prasanth Jayachandran commented on HIVE-20040:
--
+1
> JDBC: HTTP listen queue is 50 and SYNs are lost
> ---
>
> Key: HIVE-20040
> URL: https://issues.apache.org/jira/browse/HIVE-20040
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-20040.1.patch
>
>
> When testing with 5000 concurrent users, the JDBC HTTP port ends up
> overflowing on SYNs when the HS2 gc pauses.
> This is because each getQueryProgress request is an independent HTTP request,
> so unlike the BINARY mode, there are more connections being established &
> closed in HTTP mode.
> {code}
> LISTEN 0 50 *:10004*:*
> {code}
> This turns into connection errors when enabling
> {{net.ipv4.tcp_abort_on_overflow=1}}, but the better approach is to enqueue
> the connections until the HS2 is done with its GC pause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17840) HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
[
https://issues.apache.org/jira/browse/HIVE-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530602#comment-16530602
]
Hive QA commented on HIVE-17840:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
13s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12335/dev-support/hive-personality.sh
|
| git revision | master / 1fc1221 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore itests/hcatalog-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12335/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveMetaStore eats exception if transactionalListeners.notifyEvent fail
> ---
>
> Key: HIVE-17840
> URL: https://issues.apache.org/jira/browse/HIVE-17840
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Daniel Dai
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-17840.01.patch, HIVE-17840.02.patch
>
>
> For example, in add_partitions_core, if there's exception in
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,),
> transaction rollback but no exception thrown. Client will assume add
> partition is successful and take a positive path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19933) ALTER TABLE DROP PARTITION - Partition Not Found
[
https://issues.apache.org/jira/browse/HIVE-19933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530600#comment-16530600
]
Aihua Xu commented on HIVE-19933:
-
Since we support {{IF EXISTS}} syntax for drop partitions and drop table,
{{hive.exec.drop.ignorenonexistent}} configuration seems not very useful to me.
If {{IF EXISTS}} is specified, then drop table/partition/view will succeed if
they don't exist; if not, then "Not found" message will get displayed. Maybe we
should remove this configuration?
> ALTER TABLE DROP PARTITION - Partition Not Found
>
>
> Key: HIVE-19933
> URL: https://issues.apache.org/jira/browse/HIVE-19933
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.2
>Reporter: BELUGA BEHR
>Assignee: Alice Fan
>Priority: Major
>
> {code:sql}
> ALTER TABLE web_logsz DROP PARTITION (`date`='xyz')
> -- SemanticException [Error 10001]: Table not found web_logsz
> ALTER TABLE web_logs DROP PARTITION (`date`='xyz')
> -- Success.
> {code}
> There is no 'xyz' partition for the 'date' column. To make this more
> consistent, the query should fail if the user tries to drop a partition that
> does not exist
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19902) Provide Metastore micro-benchmarks
[ https://issues.apache.org/jira/browse/HIVE-19902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kolbasov updated HIVE-19902: -- Attachment: HIVE-19902.03.patch > Provide Metastore micro-benchmarks > -- > > Key: HIVE-19902 > URL: https://issues.apache.org/jira/browse/HIVE-19902 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.1.0, 4.0.0 >Reporter: Alexander Kolbasov >Assignee: Alexander Kolbasov >Priority: Major > Attachments: HIVE-19902.01.patch, HIVE-19902.02.patch, > HIVE-19902.03.patch > > > It would be very useful to have metastore benchmarks to be able to track perf > issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[
https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530578#comment-16530578
]
Hive QA commented on HIVE-20006:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 64 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
58s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5
fixed = 423 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1964
unchanged - 13 fixed = 1973 total (was 1977) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4
fixed = 384 total (was 384) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
21s{color} | {color:red} root: The patch generated 14 new + 2766 unchanged - 22
fixed = 2780 total (was 2788) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
26s{color} | {color:red} standalone-metastore generated 4 new + 226 unchanged -
2 fixed = 230 total (was 228) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
7s{color} | {color:red} ql generated 1 new + 2285 unchanged - 1 fixed = 2286
total (was 2286) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged
- 1 fixed = 53 total (was 54) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
4s{color} | {color:green} root generated 0 new + 370 unchanged - 1 fixed = 370
total (was 371) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | Dead store to numCompsWritten in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest) At
[jira] [Commented] (HIVE-20049) hive.groupby.limit.extrastep should be false by default
[ https://issues.apache.org/jira/browse/HIVE-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530577#comment-16530577 ] Ashutosh Chauhan commented on HIVE-20049: - [~abstractdog] What we shall do is remove the config altogether and make code behave as if it was false. > hive.groupby.limit.extrastep should be false by default > --- > > Key: HIVE-20049 > URL: https://issues.apache.org/jira/browse/HIVE-20049 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Ashutosh Chauhan >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20049.01.patch > > > In fact this flag is not needed since FetchTask can enforce limit there is > never a reason to have another vertex purely for limit. > > It was introduced by HIVE-12963 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20006) Make materializations invalidation cache work with multiple active remote metastores
[
https://issues.apache.org/jira/browse/HIVE-20006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530572#comment-16530572
]
Hive QA commented on HIVE-20006:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929966/HIVE-20006.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14619 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=190)
[druidmini_dynamic_partition.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test1.q,druidmini_test_insert.q]
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=238)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=191)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12334/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12334/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12334/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929966 - PreCommit-HIVE-Build
> Make materializations invalidation cache work with multiple active remote
> metastores
>
>
> Key: HIVE-20006
> URL: https://issues.apache.org/jira/browse/HIVE-20006
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch,
> HIVE-19027.03.patch, HIVE-19027.04.patch, HIVE-20006.01.patch,
> HIVE-20006.02.patch, HIVE-20006.patch
>
>
> The main points:
> - Only MVs stored in transactional tables can have a time window value of 0.
> Those are the only MVs that can be guaranteed to not be outdated when a query
> is executed, if we use custom storage handlers to store the materialized
> view, we cannot make any promises.
> - For MVs that +cannot be outdated+, we do not check the metastore. Instead,
> comparison is based on valid write id lists.
> - For MVs that +can be outdated+, we still rely on the invalidation cache.
> ** The window for valid outdated MVs can be specified in intervals of 1
> minute (less than that, it is difficult to have any guarantees about whether
> the MV is actually outdated by less than a minute or not).
> ** The async loading is done every interval / 2 (or probably better, we can
> make it configurable).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20019: - Attachment: HIVE-20019.2.patch > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch, HIVE-20019.2.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530556#comment-16530556 ] Prasanth Jayachandran commented on HIVE-20019: -- bans only if commons-logging or log4j is included as direct dependency. > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20019.1.patch, HIVE-20019.2.patch > > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20049) hive.groupby.limit.extrastep should be false by default
[ https://issues.apache.org/jira/browse/HIVE-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20049: Attachment: HIVE-20049.01.patch > hive.groupby.limit.extrastep should be false by default > --- > > Key: HIVE-20049 > URL: https://issues.apache.org/jira/browse/HIVE-20049 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Ashutosh Chauhan >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-20049.01.patch > > > In fact this flag is not needed since FetchTask can enforce limit there is > never a reason to have another vertex purely for limit. > > It was introduced by HIVE-12963 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20021: --- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20021: --- Affects Version/s: 3.0.0 > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530533#comment-16530533
]
Sergey Shelukhin commented on HIVE-20018:
-
Looks like some tests only fail in combination with other tests, not
independently. No conclusions so far...
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20016) Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20016: Attachment: HIVE-20016.1.patch > Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure > --- > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20016.1.patch > > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20016) Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530531#comment-16530531 ] Yongzhi Chen commented on HIVE-20016: - re-attached the patch to re-run the tests > Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure > --- > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20016.1.patch > > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20049) hive.groupby.limit.extrastep should be false by default
[ https://issues.apache.org/jira/browse/HIVE-20049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-20049: Description: In fact this flag is not needed since FetchTask can enforce limit there is never a reason to have another vertex purely for limit. It was introduced by HIVE-12963 was:In fact this flag is not needed since FetchTask can enforce limit there is never a reason to have another vertex purely for limit. > hive.groupby.limit.extrastep should be false by default > --- > > Key: HIVE-20049 > URL: https://issues.apache.org/jira/browse/HIVE-20049 > Project: Hive > Issue Type: Task > Components: Query Planning >Reporter: Ashutosh Chauhan >Assignee: Laszlo Bodor >Priority: Major > > In fact this flag is not needed since FetchTask can enforce limit there is > never a reason to have another vertex purely for limit. > > It was introduced by HIVE-12963 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20016) Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
[ https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20016: Attachment: (was: HIVE-20016.1.patch) > Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure > --- > > Key: HIVE-20016 > URL: https://issues.apache.org/jira/browse/HIVE-20016 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with: > java.lang.AssertionError: Concurrent Statement failed: > org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError: > Authorization plugins not initialized! > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374) > at > org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-20039 started by Deepak Jaiswal. - > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-20039: -- Status: Patch Available (was: In Progress) > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2, 3.0.0 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-20018:
---
Assignee: Sergey Shelukhin (was: Steve Yeom)
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20001) With doas set to true, running select query as hrt_qa user on external table fails due to permission denied to read /warehouse/tablespace/managed directory.
[
https://issues.apache.org/jira/browse/HIVE-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaume M updated HIVE-20001:
---
Attachment: HIVE-20001.3.patch
Status: Patch Available (was: Open)
> With doas set to true, running select query as hrt_qa user on external table
> fails due to permission denied to read /warehouse/tablespace/managed
> directory.
>
>
> Key: HIVE-20001
> URL: https://issues.apache.org/jira/browse/HIVE-20001
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20001.1.patch, HIVE-20001.1.patch,
> HIVE-20001.2.patch, HIVE-20001.3.patch
>
>
> Hive: With doas set to true, running select query as hrt_qa user on external
> table fails due to permission denied to read /warehouse/tablespace/managed
> directory.
> Steps:
> 1. Create a external table.
> 2. Set doas to true.
> 3. run select count(*) using user hrt_qa.
> Table creation query.
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "drop table if exists test_table purge;
> create external table test_table(id int, age int) row format delimited fields
> terminated by '|' stored as textfile;
> load data inpath '/tmp/table1.dat' overwrite into table test_table;
> {code}
> select count(*) query execution fails
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "select count(*) from test_table where age>30 and
> id<10100;"
> 2018-06-22 10:22:29,328|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Class path contains
> multiple SLF4J bindings.
> 2018-06-22 10:22:29,330|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2018-06-22 10:22:29,335|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Actual binding is of
> type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2018-06-22 10:22:31,408|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Format tsv is deprecated,
> please use tsv2
> 2018-06-22 10:22:31,529|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Connecting to
> jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit
> 2018-06-22 10:22:32,031|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:32 [main]:
> INFO jdbc.HiveConnection: Connected to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,130|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Failed to connect to
> ctr-e138-1518143905142-375925-01-04.hwx.site:10001
> 2018-06-22 10:22:34,244|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:34 [main]:
> WARN jdbc.HiveConnection: Could not open client transport with JDBC Uri:
> jdbc:hive2://ctr-e138-1518143905142-375925-01-04.hwx.site:10001/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit:
> Failed to open new session:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:java.security.AccessControlException: Permission
>
[jira] [Commented] (HIVE-20001) With doas set to true, running select query as hrt_qa user on external table fails due to permission denied to read /warehouse/tablespace/managed directory.
[
https://issues.apache.org/jira/browse/HIVE-20001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530503#comment-16530503
]
ASF GitHub Bot commented on HIVE-20001:
---
GitHub user beltran opened a pull request:
https://github.com/apache/hive/pull/389
HIVE-20001: With doas set to true, running select query as hrt_qa use…
…r on external table fails due to permission denied to read
/warehouse/tablespace/managed directory
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/beltran/hive HIVE-20001-3
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/389.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #389
commit 764d76de31f83b2c5985ad29410135faf4e32998
Author: Jaume Marhuenda
Date: 2018-07-01T03:47:15Z
HIVE-20001: With doas set to true, running select query as hrt_qa user on
external table fails due to permission denied to read
/warehouse/tablespace/managed directory
> With doas set to true, running select query as hrt_qa user on external table
> fails due to permission denied to read /warehouse/tablespace/managed
> directory.
>
>
> Key: HIVE-20001
> URL: https://issues.apache.org/jira/browse/HIVE-20001
> Project: Hive
> Issue Type: Bug
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20001.1.patch, HIVE-20001.1.patch,
> HIVE-20001.2.patch, HIVE-20001.3.patch
>
>
> Hive: With doas set to true, running select query as hrt_qa user on external
> table fails due to permission denied to read /warehouse/tablespace/managed
> directory.
> Steps:
> 1. Create a external table.
> 2. Set doas to true.
> 3. run select count(*) using user hrt_qa.
> Table creation query.
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "drop table if exists test_table purge;
> create external table test_table(id int, age int) row format delimited fields
> terminated by '|' stored as textfile;
> load data inpath '/tmp/table1.dat' overwrite into table test_table;
> {code}
> select count(*) query execution fails
> {code}
> beeline -n hrt_qa -p pwd -u
> "jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit"
> --outputformat=tsv -e "select count(*) from test_table where age>30 and
> id<10100;"
> 2018-06-22 10:22:29,328|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Class path contains
> multiple SLF4J bindings.
> 2018-06-22 10:22:29,330|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2018-06-22 10:22:29,335|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|SLF4J: Actual binding is of
> type [org.apache.logging.slf4j.Log4jLoggerFactory]
> 2018-06-22 10:22:31,408|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Format tsv is deprecated,
> please use tsv2
> 2018-06-22 10:22:31,529|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|Connecting to
> jdbc:hive2://ctr-e138-1518143905142-375925-01-06.hwx.site:2181,ctr-e138-1518143905142-375925-01-05.hwx.site:2181,ctr-e138-1518143905142-375925-01-07.hwx.site:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;principal=hive/_h...@example.com;transportMode=http;httpPath=cliservice;ssl=true;sslTrustStore=/etc/security/serverKeys/hivetruststore.jks;trustStorePassword=changeit
> 2018-06-22 10:22:32,031|INFO|Thread-126|machine.py:111 -
> tee_pipe()||b3a493ec-99be-483e-91fe-4b701ec27ebc|18/06/22 10:22:32 [main]:
> INFO jdbc.HiveConnection: Connected to
>
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20039: --- Status: Open (was: Patch Available) > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2, 3.0.0 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530494#comment-16530494
]
Hive QA commented on HIVE-18453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929960/HIVE-18453.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14654 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12333/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12333/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12333/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929960 - PreCommit-HIVE-Build
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax and
> allows for future compatibility changes to support Parquet ACIDv2 tables
> along with ORC tables.
> The ACIDv2 design is format independent, with the ability to add new
> vectorized input formats with no changes to the design.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20039: --- Description: Left outer join on bucketed table on certain cases gives wrong results. Depending on the order in which the table-scans are walked through, the FilterPruner might end up using the wrong table scan's table properties on the other table. was:Left outer join on bucketed table on certain cases gives wrong results. > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Left outer join on bucketed table on certain cases gives wrong results. > Depending on the order in which the table-scans are walked through, the > FilterPruner might end up using the wrong table scan's table properties on > the other table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20039) Bucket pruning: Left Outer Join on bucketed table gives wrong result
[ https://issues.apache.org/jira/browse/HIVE-20039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20039: --- Summary: Bucket pruning: Left Outer Join on bucketed table gives wrong result (was: Left Outer Join on bucketed table gives wrong result) > Bucket pruning: Left Outer Join on bucketed table gives wrong result > - > > Key: HIVE-20039 > URL: https://issues.apache.org/jira/browse/HIVE-20039 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.2 >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Left outer join on bucketed table on certain cases gives wrong results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18785) Make JSON SerDe First-Class SerDe
[
https://issues.apache.org/jira/browse/HIVE-18785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530491#comment-16530491
]
Aihua Xu commented on HIVE-18785:
-
[~belugabehr] I already updated the page. Are you expecting something else?
{noformat}
Starting in Hive 3.0.0, JsonSerDe is added to Hive Serde as
"org.apache.hadoop.hive.serde2.JsonSerDe" (HIVE-19211).
CREATE TABLE my_table(a string, b bigint, ...)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.JsonSerDe'
STORED AS TEXTFILE;
Or STORED AS JSONFILE is supported starting in Hive 4.0.0 (HIVE-19899), so you
can create table as follows:
CREATE TABLE my_table(a string, b bigint, ...) STORED AS JSONFILE;
{noformat}
> Make JSON SerDe First-Class SerDe
> -
>
> Key: HIVE-18785
> URL: https://issues.apache.org/jira/browse/HIVE-18785
> Project: Hive
> Issue Type: New Feature
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: Aihua Xu
>Priority: Major
>
> According to the [Hive SerDe
> Docs|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RowFormats],
> there are some extra steps involved in getting the JSON SerDe to work:
> {quote}
> ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe'
> STORED AS TEXTFILE
> In some distributions, a reference to hive-hcatalog-core.jar is required.
> ADD JAR /usr/lib/hive-hcatalog/lib/hive-hcatalog-core.jar;
> {quote}
> I would like to propose that we move this SerDe into first-class status:
> {{STORED AS JSONFILE}}
> The user should have to perform no additional steps to use this SerDe.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530480#comment-16530480 ] Sergey Shelukhin commented on HIVE-19975: - Fixed test failures; there were two issues in the patch. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19975.01.patch, HIVE-19975.patch, > branch-19975.01.nogen.patch, branch-19975.nogen.patch > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19975: Attachment: HIVE-19975.01.patch > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19975.01.patch, HIVE-19975.patch, > branch-19975.01.nogen.patch, branch-19975.nogen.patch > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19975: Attachment: branch-19975.01.nogen.patch > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19975.01.patch, HIVE-19975.patch, > branch-19975.01.nogen.patch, branch-19975.nogen.patch > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20059: - Resolution: Fixed Fix Version/s: 3.2.0 4.0.0 Status: Resolved (was: Patch Available) Committed to branch-3 and master. Thanks for the review! > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530474#comment-16530474 ] Prasanth Jayachandran commented on HIVE-20059: -- There are no tests that affects this change as this code path is not triggered by hive. I will commit this patch. > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20016) Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
[
https://issues.apache.org/jira/browse/HIVE-20016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530472#comment-16530472
]
Aihua Xu commented on HIVE-20016:
-
Findbugs warning seems not applicable but how about TestHS2AuthzContext?
Checked the tests and don't feel it's related. Can we rerun to make sure they
pass if they are flaky? Also do you think we should use
synchronized(authenticator or an auth lock) {} block so it will sync with other
methods?
BTW: I created HIVE-20053. I feel we can use Singleton for HiveAuth to
initialize once so we can solve this issue but I notice a few session related
info is used in setupAuth().
> Investigate TestJdbcWithMiniHS2.testParallelCompilation3 random failure
> ---
>
> Key: HIVE-20016
> URL: https://issues.apache.org/jira/browse/HIVE-20016
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-20016.1.patch
>
>
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 failed with:
> java.lang.AssertionError: Concurrent Statement failed:
> org.apache.hive.service.cli.HiveSQLException: java.lang.AssertionError:
> Authorization plugins not initialized!
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.finishTasks(TestJdbcWithMiniHS2.java:374)
> at
> org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3(TestJdbcWithMiniHS2.java:304)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18453) ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet support
[
https://issues.apache.org/jira/browse/HIVE-18453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530468#comment-16530468
]
Hive QA commented on HIVE-18453:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
22s{color} | {color:red} ql generated 4 new + 2281 unchanged - 5 fixed = 2285
total (was 2286) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA227 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46153-46166] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA271 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46490-46503] |
| | Dead store to LA28_141 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA28.specialStateTransition(int,
IntStream) At HiveParser.java:[line 45983] |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA286 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
46626-46639] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12333/dev-support/hive-personality.sh
|
| git revision | master / d83a0be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12333/yetus/new-findbugs-ql.html
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12333/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: Add "CREATE TRANSACTIONAL TABLE" syntax to unify ACID ORC & Parquet
> support
> -
>
> Key: HIVE-18453
> URL: https://issues.apache.org/jira/browse/HIVE-18453
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18453.01.patch, HIVE-18453.02.patch,
> HIVE-18453.03.patch, HIVE-18453.04.patch
>
>
> The ACID table markers are currently done with TBLPROPERTIES which is
> inherently fragile.
> The "create transactional table" offers a way to standardize the syntax
[jira] [Assigned] (HIVE-20060) Refactor HiveSchemaTool and MetastoreSchemaTool
[ https://issues.apache.org/jira/browse/HIVE-20060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-20060: - > Refactor HiveSchemaTool and MetastoreSchemaTool > --- > > Key: HIVE-20060 > URL: https://issues.apache.org/jira/browse/HIVE-20060 > Project: Hive > Issue Type: Task > Components: Beeline, Metastore >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > > These two classes are 95% the same. Now that HIVE-19711 has split > HiveSchemaTool into multiple components it will be much easier to refactor > these so that there is only one version of the code that each shares. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19792) Upgrade orc to 1.5.2 and enable decimal_64 schema evolution tests
[ https://issues.apache.org/jira/browse/HIVE-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19792: - Summary: Upgrade orc to 1.5.2 and enable decimal_64 schema evolution tests (was: Enable schema evolution tests for decimal 64) > Upgrade orc to 1.5.2 and enable decimal_64 schema evolution tests > - > > Key: HIVE-19792 > URL: https://issues.apache.org/jira/browse/HIVE-19792 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19792.1.patch, HIVE-19792.2.patch, > HIVE-19792.3.patch, HIVE-19792.4.patch > > > Following tests are disabled in HIVE-19629 as orc ConvertTreeReaderFactory > does not handle Decimal64ColumnVectors. This jira is to re-enable those tests > after orc supports it. > 1) type_change_test_int_vectorized.q > 2) type_change_test_int.q > 3) orc_schema_evolution_float.q > 4) schema_evol_orc_nonvec_part_all_primitive.q > 5) schema_evol_orc_nonvec_part_all_primitive_llap_io.q > 6) schema_evol_orc_vec_part_all_primitive.q > 7) schema_evol_orc_vec_part_all_primitive_llap_io.q > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19792) Enable schema evolution tests for decimal 64
[ https://issues.apache.org/jira/browse/HIVE-19792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19792: - Attachment: HIVE-19792.4.patch > Enable schema evolution tests for decimal 64 > > > Key: HIVE-19792 > URL: https://issues.apache.org/jira/browse/HIVE-19792 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19792.1.patch, HIVE-19792.2.patch, > HIVE-19792.3.patch, HIVE-19792.4.patch > > > Following tests are disabled in HIVE-19629 as orc ConvertTreeReaderFactory > does not handle Decimal64ColumnVectors. This jira is to re-enable those tests > after orc supports it. > 1) type_change_test_int_vectorized.q > 2) type_change_test_int.q > 3) orc_schema_evolution_float.q > 4) schema_evol_orc_nonvec_part_all_primitive.q > 5) schema_evol_orc_nonvec_part_all_primitive_llap_io.q > 6) schema_evol_orc_vec_part_all_primitive.q > 7) schema_evol_orc_vec_part_all_primitive_llap_io.q > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[
https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530435#comment-16530435
]
Hive QA commented on HIVE-17593:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929939/HIVE-17593.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 14632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.org.apache.hadoop.hive.cli.TestBeeLineDriver
(batchId=261)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=69)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_parquet_types]
(batchId=172)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterPartition
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTable
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterTableCascade
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterViewParititon
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testColumnStatistics
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTable
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testComplexTypeApi
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testConcurrentMetastores
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBLocationChange
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBOwner
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDBOwnerChange
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabase
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocation
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocationWithPermissionProblems
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDropDatabaseCascadeMVMultiDB
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDropTable
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterLastPartition
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFilterSinglePartition
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testFunctionWithResources
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetConfigValue
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetMetastoreUuid
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetSchemaWithNoClassDefFoundError
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetTableObjects
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testGetUUIDInParallel
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testJDOPersistanceManagerCleanup
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitionNames
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitions
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testListPartitionsWihtLimitEnabled
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testNameMethods
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartition
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePartition
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRetriableClientWithConnLifetime
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleFunction
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTable
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSimpleTypeApi
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testStatsFastTrivial
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testSynchronized
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableDatabase
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testTableFilter
(batchId=217)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testValidateTableCols
(batchId=217)
{noformat}
Test results:
[jira] [Assigned] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19860: --- Assignee: Rajkumar Singh (was: Sergey Shelukhin) > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.02.patch, > HIVE-19860.patch, Screen Shot 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530427#comment-16530427 ] Sergey Shelukhin commented on HIVE-19860: - Another attempt, looks like a flaky test failure > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.02.patch, > HIVE-19860.patch, Screen Shot 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19860: Attachment: HIVE-19860.02.patch > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.02.patch, > HIVE-19860.patch, Screen Shot 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19860: --- Assignee: Sergey Shelukhin (was: Rajkumar Singh) > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot > 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19711) Refactor Hive Schema Tool
[ https://issues.apache.org/jira/browse/HIVE-19711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely resolved HIVE-19711. --- Resolution: Fixed > Refactor Hive Schema Tool > - > > Key: HIVE-19711 > URL: https://issues.apache.org/jira/browse/HIVE-19711 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-19711.01.patch, HIVE-19711.02.patch, > HIVE-19711.03.patch, HIVE-19711.04.patch, HIVE-19711.05.patch, > HIVE-19711.06.patch, HIVE-19711.07.patch, HIVE-19711.08.patch > > > HiveSchemaTool is an 1500 lines long class trying to do everything It shold > be cut into multiple classes doing smaller components. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19711) Refactor Hive Schema Tool
[ https://issues.apache.org/jira/browse/HIVE-19711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reopened HIVE-19711: --- > Refactor Hive Schema Tool > - > > Key: HIVE-19711 > URL: https://issues.apache.org/jira/browse/HIVE-19711 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-19711.01.patch, HIVE-19711.02.patch, > HIVE-19711.03.patch, HIVE-19711.04.patch, HIVE-19711.05.patch, > HIVE-19711.06.patch, HIVE-19711.07.patch, HIVE-19711.08.patch > > > HiveSchemaTool is an 1500 lines long class trying to do everything It shold > be cut into multiple classes doing smaller components. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530407#comment-16530407 ] Vineet Garg commented on HIVE-20059: +1 > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20059: - Status: Patch Available (was: Open) > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530406#comment-16530406 ] Prasanth Jayachandran commented on HIVE-20059: -- Tests are not affected by this change. > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17593) DataWritableWriter strip spaces for CHAR type before writing, but predicate generator doesn't do same thing.
[
https://issues.apache.org/jira/browse/HIVE-17593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530403#comment-16530403
]
Hive QA commented on HIVE-17593:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2286 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12332/dev-support/hive-personality.sh
|
| git revision | master / d83a0be |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12332/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> DataWritableWriter strip spaces for CHAR type before writing, but predicate
> generator doesn't do same thing.
>
>
> Key: HIVE-17593
> URL: https://issues.apache.org/jira/browse/HIVE-17593
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.0, 3.0.0
>Reporter: Junjie Chen
>Assignee: Junjie Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-17593.2.patch, HIVE-17593.patch
>
>
> DataWritableWriter strip spaces for CHAR type before writing. While when
> generating predicate, it does NOT do same striping which should cause data
> missing!
> In current version, it doesn't cause data missing since predicate is not well
> push down to parquet due to HIVE-17261.
> Please see ConvertAstTosearchArg.java, getTypes treats CHAR and STRING as
> same which will build a predicate with tail spaces.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20059) Hive streaming should try shade prefix unconditionally on exception
[ https://issues.apache.org/jira/browse/HIVE-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20059: - Attachment: HIVE-20059.1.patch > Hive streaming should try shade prefix unconditionally on exception > --- > > Key: HIVE-20059 > URL: https://issues.apache.org/jira/browse/HIVE-20059 > Project: Hive > Issue Type: Bug > Components: Streaming >Affects Versions: 3.1.0, 4.0.0, 3.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-20059.1.patch > > > AbstractRecordWriter tries hive.classloader.shade.prefix on > ClassNotFoundException but there are instances where OrcOutputFormat from old > hive version gets loaded resulting in ClassCastException. I think we should > try shadeprefix when defined and when any exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20055) SQL injection via metastore ACID APIs (and maybe queries, although that's unlikely)
[
https://issues.apache.org/jira/browse/HIVE-20055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530404#comment-16530404
]
Andrew Sherman commented on HIVE-20055:
---
I'm not sure I completely understand this jira, but often then best way to
avoid sql injection attacks is to code jdbc using prepared statements.
> SQL injection via metastore ACID APIs (and maybe queries, although that's
> unlikely)
> ---
>
> Key: HIVE-20055
> URL: https://issues.apache.org/jira/browse/HIVE-20055
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Thejas M Nair
>Priority: Major
>
> [~thejas] asked me to create this JIRA based on my earlier email :)
> {noformat}
> This might be doable with a specially crafted query, I’m not sure what APIs
> calls have what checks (e.g. via Hive parser) that would prevent the below.
> However, for remote metastore (default on many clusters currently, afaik it’s
> the default for ACID) we expose thrift API that accepts strings e.g.
> get_valid_write_ids.
> That passes the string table names to TxnHandler::getValidWriteIdsForTable,
> that inserts them into the query string w/quoteString call; quoteString
> doesn’t do any validation.
> Some ready made delete statements also exist e.g. "delete from REPL_TXN_MAP
> where RTM_SRC_TXN_ID = " + sourceTxnId + " and RTM_REPL_POLICY = " +
> quoteString(rqst.getReplPolicy());
> I think my replication policy might be {' OR '1' = '1} ;)
> So, SQL injection might be possible thru these APIs.
> I wonder if this class should be switched to parameter based execution?
> DirectSQL could be used as an example, although that uses DataNucleus direct
> sql feature… at least we need some checks on these.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
