[jira] [Commented] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[
https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420178#comment-16420178
]
Hive QA commented on HIVE-19032:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9917/dev-support/hive-personality.sh
|
| git revision | master / 1974397 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9917/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9917/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Disable GROUP BY aggregations with DISTINCT
> --
>
> Key: HIVE-19032
> URL: https://issues.apache.org/jira/browse/HIVE-19032
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19032.01.patch, HIVE-19032.02.patch,
> HIVE-19032.03.patch
>
>
> Vectorized GROUP BY does not support DISTINCT aggregation functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events
[ https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18781: Labels: ACID DR replication (was: ) > Create/Replicate Open, Commit (without writes) and Abort Txn events > --- > > Key: HIVE-18781 > URL: https://issues.apache.org/jira/browse/HIVE-18781 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, replication > Fix For: 3.0.0 > > Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch, > HIVE-18781.03.patch, HIVE-18781.04.patch, HIVE-18781.05.patch, > HIVE-18781.06.patch, HIVE-18781.07.patch, HIVE-18781.08.patch, > HIVE-18781.09.patch, HIVE-18781.10.patch, HIVE-18781.11.patch, > HIVE-18781.11.patch > > > *EVENT_OPEN_TXN:* > *Source Warehouse:* > - Create new event type EVENT_OPEN_TXN with related message format etc. > - When any transaction is opened either by auto-commit mode or > multi-statement mode, need to capture this event. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Open a txn in target warehouse. > - Create a map of source txn ID against target txn ID and persist the same > in metastore. There should be one map per replication policy (DBName.* incase > of DB level replication, DBName.TableName incase of table level replication) > > *EVENT_COMMIT_TXN (Without writes)* > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > - Create EVENT_COMMIT_TXN event type with corresponding message format etc. > *Target warehouse:* > - Repl load should read this event from the dump. > - Validate the source txn ID from the event using the Source-Target Txn ID > map maintained in target metastore. Also, need to check if corresponding > target txn ID is valid. > - If valid, then apply the event and commit the corresponding target > transaction. > - This new event should be idempotent such that if it is applied twice, then > second time it should be loop. > > *EVENT_ABORT_TXN* > Source Warehouse: > - Create new event type EVENT_ABORT_TXN with related message format etc. > - Capture this event when abort the txn. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Validate if source txn ID from the event is there in the source-target txn > ID map. If not there, just noop the event. > - If valid, then Abort the corresponding target txn and remove the entry > from source-target txn ID map. > All these new events should be idempotent such that if it is applied twice, > then second time it should be noop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18725) Improve error handling for subqueries if there is wrong column reference
[
https://issues.apache.org/jira/browse/HIVE-18725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-18725:
-
Attachment: HIVE-18725.02.patch
> Improve error handling for subqueries if there is wrong column reference
>
>
> Key: HIVE-18725
> URL: https://issues.apache.org/jira/browse/HIVE-18725
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-18725.01.patch, HIVE-18725.02.patch
>
>
> If there is a column reference within subquery which doesn't exist Hive
> throws misleading error message.
> e.g.
> {code:sql}
> select * from table1 where table1.col1 IN (select col2 from table2 where
> table2.col1=table1.non_existing_column) and table1.col1 IN (select 4);
> {code}
> The above query, assuming table1 doesn't have non_existing_column, will throw
> following misleading error:
> {noformat}
> FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'col1':
> Only 1 SubQuery expression is supported.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19058) add object owner to HivePrivilegeObject
[
https://issues.apache.org/jira/browse/HIVE-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420168#comment-16420168
]
Hive QA commented on HIVE-19058:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916923/HIVE-19058.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 156 failed/errored test(s), 13436 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19062: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420164#comment-16420164 ] Vineet Garg commented on HIVE-19062: Thanks for fixing this [~asherman]. This was due to HIVE-18953. I have pushed this to master. > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19062) Update constraint_partition_columns.q.out
[ https://issues.apache.org/jira/browse/HIVE-19062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19062: --- Fix Version/s: 3.0.0 > Update constraint_partition_columns.q.out > - > > Key: HIVE-19062 > URL: https://issues.apache.org/jira/browse/HIVE-19062 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19062.1.patch > > > TestNegativeCliDriver is error-prone at present but if you run > constraint_partition_columns.q on its own you get a diff. I think this is a > simple regression caused by [HIVE-18726] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19078) Add separate option to display query explain on HS2
[ https://issues.apache.org/jira/browse/HIVE-19078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhudeep Petwal reassigned HIVE-19078: --- Assignee: Madhudeep Petwal > Add separate option to display query explain on HS2 > --- > > Key: HIVE-19078 > URL: https://issues.apache.org/jira/browse/HIVE-19078 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Madhudeep Petwal >Priority: Major > > currently the same option (ConfVars.HIVE_LOG_EXPLAIN_OUTPUT) controls both > explain print to console; and display on hs2 ; seeing it also on the console > eats up scroll space very fast > https://github.com/apache/hive/blob/b750a16be5a8e378ac11a88ace244e826d23cfe5/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L684 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420148#comment-16420148
]
Vineet Garg commented on HIVE-19059:
[~jcamachorodriguez] Thanks for reviewing. I have uploaded a patch with review
comments addressed. But I am still waiting on first test run to come :)
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch, HIVE-19059.2.patch,
> HIVE-19059.3.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19059:
---
Status: Open (was: Patch Available)
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch, HIVE-19059.2.patch,
> HIVE-19059.3.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19059:
---
Attachment: HIVE-19059.3.patch
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch, HIVE-19059.2.patch,
> HIVE-19059.3.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19059:
---
Status: Patch Available (was: Open)
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch, HIVE-19059.2.patch,
> HIVE-19059.3.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19058) add object owner to HivePrivilegeObject
[
https://issues.apache.org/jira/browse/HIVE-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420140#comment-16420140
]
Hive QA commented on HIVE-19058:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 17 new + 8 unchanged - 0 fixed
= 25 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9916/dev-support/hive-personality.sh
|
| git revision | master / 3e3292b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9916/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9916/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9916/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> add object owner to HivePrivilegeObject
> ---
>
> Key: HIVE-19058
> URL: https://issues.apache.org/jira/browse/HIVE-19058
> Project: Hive
> Issue Type: Bug
> Components: Security
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19058.01.patch, HIVE-19058.02.patch
>
>
> this can enable HiveAuthorizer to create policies based on the owner of the
> object - for example, only let the owner of a table read/write it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420135#comment-16420135 ] Matt McCline commented on HIVE-19024: - Committed to master. [~gopalv] thank you for your code review. > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch, > HIVE-19024.03.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Resolution: Fixed Status: Resolved (was: Patch Available) > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch, > HIVE-19024.03.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Fix Version/s: 3.0.0 > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0 > > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch, > HIVE-19024.03.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18814) Support Add Partition For Acid tables
[
https://issues.apache.org/jira/browse/HIVE-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420130#comment-16420130
]
Hive QA commented on HIVE-18814:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916921/HIVE-18814.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9915/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9915/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9915/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-30 04:09:36.760
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9915/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-30 04:09:36.776
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3e3292b HIVE-19029 - Load Data should prevent loading acid files
(Eugene Koifman, reviewed by Jason Dere)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 3e3292b HIVE-19029 - Load Data should prevent loading acid files
(Eugene Koifman, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-30 04:09:41.286
+ rm -rf ../yetus_PreCommit-HIVE-Build-9915
+ mkdir ../yetus_PreCommit-HIVE-Build-9915
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9915
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9915/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:447
Falling back to three-way merge...
Applied patch to 'ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java'
with conflicts.
Going to apply patch with: git apply -p0
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java:447
Falling back to three-way merge...
Applied patch to 'ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java'
with conflicts.
U ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12916921 - PreCommit-HIVE-Build
> Support Add Partition For Acid tables
> -
>
> Key: HIVE-18814
> URL: https://issues.apache.org/jira/browse/HIVE-18814
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18814.01.patch, HIVE-18814.02.patch
>
>
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual%2BDDL#LanguageManualDDL-AddPartitions]
> Add Partition command creates a {{Partition}} metadata object and sets the
> location to the directory containing data files.
> In current master (Hive 3.0), Add partition on an acid table doesn't fail and
> at read time the data is decorated with row__id but the original transaction
> is 0. I suspect in earlier Hive versions this will throw or return no data.
> Since this new partition didn't have data before, assigning txnid:0 isn't
> going to generate duplicate IDs but it could violate Snapshot Isolation in
> multi stmt txns. Suppose txnid:7 runs {{select * from T}}. Then txnid:8
> adds a partition to T. Now if txnid:7 runs the same query again, it will see
> the data in the new partition.
> This can't be release like this since a delete on this data (added via Add
>
[jira] [Commented] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[
https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420127#comment-16420127
]
Hive QA commented on HIVE-19024:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916918/HIVE-19024.03.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13059 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Commented] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420122#comment-16420122 ] Nishant Bangarwa commented on HIVE-19049: - [~ashutoshc] thanks for the review. It seems some of the test failures are related to this patch. have attached an addendum patch to handle the case when envContext is null. > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19049: Status: Patch Available (was: Reopened) > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa reopened HIVE-19049: - > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19049: Attachment: (was: HIVE-19049.addendum.patch) > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19049: Attachment: HIVE-19049.addendum.patch > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Attachment: HIVE-19019.06.patch
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch,
> HIVE-19019.06.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Status: Patch Available (was: In Progress)
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch,
> HIVE-19019.06.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Status: In Progress (was: Patch Available)
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19049) Add support for Alter table add columns for Druid
[ https://issues.apache.org/jira/browse/HIVE-19049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19049: Attachment: HIVE-19049.addendum.patch > Add support for Alter table add columns for Druid > - > > Key: HIVE-19049 > URL: https://issues.apache.org/jira/browse/HIVE-19049 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19049.01.patch, HIVE-19049.02.patch, > HIVE-19049.addendum.patch, HIVE-19049.patch > > > Add support for Alter table add columns for Druid. > Currently it is not supported and throws exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420115#comment-16420115 ] Daniel Dai commented on HIVE-18841: --- +1 pending test. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[
https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420104#comment-16420104
]
Hive QA commented on HIVE-19024:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9914/dev-support/hive-personality.sh
|
| git revision | master / 3e3292b |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9914/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9914/yetus/patch-asflicense-problems.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9914/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Disable complex type constants for VectorUDFAdaptor
> --
>
> Key: HIVE-19024
> URL: https://issues.apache.org/jira/browse/HIVE-19024
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch,
> HIVE-19024.03.patch
>
>
> Currently, complex type constants are not detected and cause execution
> failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19059) Support DEFAULT keyword with INSERT and UPDATE
[
https://issues.apache.org/jira/browse/HIVE-19059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19059:
---
Labels: TODOC3.0 (was: )
> Support DEFAULT keyword with INSERT and UPDATE
> --
>
> Key: HIVE-19059
> URL: https://issues.apache.org/jira/browse/HIVE-19059
> Project: Hive
> Issue Type: New Feature
> Components: SQL
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-19059.1.patch, HIVE-19059.2.patch
>
>
> Support DEFAULT keyword in INSERT e.g.
> {code:sql}
> INSERT INTO TABLE t values (DEFAULT, DEFAULT)
> {code}
> or with UPDATE
> {code:sql}
> UPDATE TABLE t SET col1=DEFAULT WHERE col2 > 4
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420091#comment-16420091
]
Hive QA commented on HIVE-19019:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916915/HIVE-19019.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9913/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9913/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9913/
Messages:
{noformat}
This message was trimmed, see log for full details
Applied patch to 'ql/src/test/results/clientnegative/alter_non_native.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out:120
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/beeline/materialized_view_create_rewrite.q.out'
cleanly.
error: patch failed: ql/src/test/results/clientpositive/complex_alias.q.out:17
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/complex_alias.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/druid/druidmini_mv.q.out:511
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/druid/druidmini_mv.q.out'
cleanly.
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/druid/druidmini_test_alter.q.out' cleanly.
error: patch failed: ql/src/test/results/clientpositive/druid_basic3.q.out:265
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/druid_basic3.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/explainuser_1.q.out:310
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/explainuser_1.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out:119
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_dummy.q.out:119
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_dummy.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out:155
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/materialized_view_create_rewrite_multi_db.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/sharedworkext.q.out:616
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/sharedworkext.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/subquery_in_having.q.out:1573
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/subquery_in_having.q.out' cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/subquery_notin.q.out:2973
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/subquery_notin.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/subquery_select.q.out:5046
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/subquery_select.q.out' cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_outer_reference_windowed.q.out:1031
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/vector_outer_reference_windowed.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out:137
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_windowing.q.out:3878
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/llap/vector_windowing.q.out' cleanly.
error: patch failed:
ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out:120
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/materialized_view_create_rewrite.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out:156
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/materialized_view_create_rewrite_multi_db.q.out'
cleanly.
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/materialized_view_rewrite_1.q.out' cleanly.
Falling back to three-way merge...
[jira] [Commented] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420088#comment-16420088
]
Hive QA commented on HIVE-19018:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916909/HIVE-19018.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 13059 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-18781) Create/Replicate Open, Commit (without writes) and Abort Txn events
[ https://issues.apache.org/jira/browse/HIVE-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-18781: --- Attachment: HIVE-18781.11.patch > Create/Replicate Open, Commit (without writes) and Abort Txn events > --- > > Key: HIVE-18781 > URL: https://issues.apache.org/jira/browse/HIVE-18781 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18781.01.patch, HIVE-18781.02.patch, > HIVE-18781.03.patch, HIVE-18781.04.patch, HIVE-18781.05.patch, > HIVE-18781.06.patch, HIVE-18781.07.patch, HIVE-18781.08.patch, > HIVE-18781.09.patch, HIVE-18781.10.patch, HIVE-18781.11.patch, > HIVE-18781.11.patch > > > *EVENT_OPEN_TXN:* > *Source Warehouse:* > - Create new event type EVENT_OPEN_TXN with related message format etc. > - When any transaction is opened either by auto-commit mode or > multi-statement mode, need to capture this event. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Open a txn in target warehouse. > - Create a map of source txn ID against target txn ID and persist the same > in metastore. There should be one map per replication policy (DBName.* incase > of DB level replication, DBName.TableName incase of table level replication) > > *EVENT_COMMIT_TXN (Without writes)* > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > - Create EVENT_COMMIT_TXN event type with corresponding message format etc. > *Target warehouse:* > - Repl load should read this event from the dump. > - Validate the source txn ID from the event using the Source-Target Txn ID > map maintained in target metastore. Also, need to check if corresponding > target txn ID is valid. > - If valid, then apply the event and commit the corresponding target > transaction. > - This new event should be idempotent such that if it is applied twice, then > second time it should be loop. > > *EVENT_ABORT_TXN* > Source Warehouse: > - Create new event type EVENT_ABORT_TXN with related message format etc. > - Capture this event when abort the txn. > - Repl dump should read this event from EventNotificationTable and dump the > message. > *Target Warehouse:* > - Repl load should read the event from the dump and get the message. > - Validate if source txn ID from the event is there in the source-target txn > ID map. If not there, just noop the event. > - If valid, then Abort the corresponding target txn and remove the entry > from source-target txn ID map. > All these new events should be idempotent such that if it is applied twice, > then second time it should be noop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Status: Patch Available (was: Open) > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Attachment: HIVE-19083.1.patch > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Attachment: (was: HIVE-HIVE-19083.1.patch) > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Description: Partition clause should be optional for * INSERT INTO VALUES * INSERT OVERWRITE * INSERT SELECT > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19083.1.patch > > > Partition clause should be optional for > * INSERT INTO VALUES > * INSERT OVERWRITE > * INSERT SELECT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Status: Open (was: Patch Available) > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Attachment: HIVE-HIVE-19083.1.patch > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420065#comment-16420065
]
Vineet Garg commented on HIVE-18792:
Since the patch I uploaded only caters to one requirement I have created a
sub-task to cover that (HIVE-19083)
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19083: --- Status: Patch Available (was: Open) > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-HIVE-19083.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18792:
---
Attachment: (was: HIVE-18792.2.patch)
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18792) Allow standard compliant syntax for insert on partitioned tables
[
https://issues.apache.org/jira/browse/HIVE-18792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-18792:
---
Attachment: (was: HIVE-18792.1.patch)
> Allow standard compliant syntax for insert on partitioned tables
>
>
> Key: HIVE-18792
> URL: https://issues.apache.org/jira/browse/HIVE-18792
> Project: Hive
> Issue Type: Improvement
> Components: SQL
>Reporter: Ashutosh Chauhan
>Assignee: Vineet Garg
>Priority: Major
>
> Following works:
> {code}
> create table t1 (a int, b int, c int);
> create table t2 (a int, b int, c int) partitioned by (d int);
> insert into t1 values (1,2,3);
> insert into t1 (c, b, a) values (1,2,3);
> insert into t1 (a,b) values (1,2);
> {code}
> For partitioned tables it should work similarly but doesn't. All of
> following fails:
> {code}
> insert into t2 values (1,2,3,4);
> insert into t2 (a, b, c, d) values (1,2,3,4);
> insert into t2 (c,d) values (1,2);
> insert into t2 (a,b) values (1,2);
> {code}
> All of above should work. Also note following works:
> {code}
> insert into t2 partition(d) values (1,2,3,4);
> insert into t2 partition(d=4) values (1,2,3);
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19083) Make partition clause optional for INSERT
[ https://issues.apache.org/jira/browse/HIVE-19083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-19083: -- > Make partition clause optional for INSERT > - > > Key: HIVE-19083 > URL: https://issues.apache.org/jira/browse/HIVE-19083 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18955) HoS: Unable to create Channel from class NioServerSocketChannel
[
https://issues.apache.org/jira/browse/HIVE-18955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420062#comment-16420062
]
Rui Li commented on HIVE-18955:
---
Re-upload the patch for another ptest run.
[~stakiar], do you think we can downgrade netty as [~yuchaoran2011] suggested?
> HoS: Unable to create Channel from class NioServerSocketChannel
> ---
>
> Key: HIVE-18955
> URL: https://issues.apache.org/jira/browse/HIVE-18955
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Blocker
> Attachments: HIVE-18955.1.patch, HIVE-18955.1.patch
>
>
> Hit the issue when trying launch spark job. Stack trace:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> io.netty.channel.DefaultChannelId.newInstance()Lio/netty/channel/DefaultChannelId;
> at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:111)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at io.netty.channel.AbstractChannel.(AbstractChannel.java:83)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.nio.AbstractNioChannel.(AbstractNioChannel.java:84)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.nio.AbstractNioMessageChannel.(AbstractNioMessageChannel.java:42)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:86)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:72)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_151]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[?:1.8.0_151]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ~[?:1.8.0_151]
> at
> io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> ... 32 more
> {noformat}
> It seems we have conflicts versions of class
> {{io.netty.channel.DefaultChannelId}} from async-http-client.jar and
> netty-all.jar
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18955) HoS: Unable to create Channel from class NioServerSocketChannel
[
https://issues.apache.org/jira/browse/HIVE-18955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-18955:
--
Attachment: HIVE-18955.1.patch
> HoS: Unable to create Channel from class NioServerSocketChannel
> ---
>
> Key: HIVE-18955
> URL: https://issues.apache.org/jira/browse/HIVE-18955
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Blocker
> Attachments: HIVE-18955.1.patch, HIVE-18955.1.patch
>
>
> Hit the issue when trying launch spark job. Stack trace:
> {noformat}
> Caused by: java.lang.NoSuchMethodError:
> io.netty.channel.DefaultChannelId.newInstance()Lio/netty/channel/DefaultChannelId;
> at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:111)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at io.netty.channel.AbstractChannel.(AbstractChannel.java:83)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.nio.AbstractNioChannel.(AbstractNioChannel.java:84)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.nio.AbstractNioMessageChannel.(AbstractNioMessageChannel.java:42)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:86)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:72)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_151]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[?:1.8.0_151]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ~[?:1.8.0_151]
> at
> io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
> ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
> ... 32 more
> {noformat}
> It seems we have conflicts versions of class
> {{io.netty.channel.DefaultChannelId}} from async-http-client.jar and
> netty-all.jar
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-7139) Period in username can't be parsed
[ https://issues.apache.org/jira/browse/HIVE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420052#comment-16420052 ] Jerry Chi edited comment on HIVE-7139 at 3/30/18 1:46 AM: -- Is there really no planned fix or workaround for this? Thanks! If there is no planned fix, then shouldn't user names containing periods be banned in the first place? was (Author: jerrychi): Is there really no planned fix or workaround for this? Thanks! > Period in username can't be parsed > -- > > Key: HIVE-7139 > URL: https://issues.apache.org/jira/browse/HIVE-7139 > Project: Hive > Issue Type: Bug > Components: Authentication >Affects Versions: 0.13.0 >Reporter: Andrew Musselman >Priority: Minor > > We have an environment where usernames have period characters between their > first and last names. > This breaks parsing while trying to grant permissions, e.g., > hive> grant all on database mydb to user first.last; > FAILED: ParseException line 1:40 missing EOF at '.' near 'first' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-7139) Period in username can't be parsed
[ https://issues.apache.org/jira/browse/HIVE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420052#comment-16420052 ] Jerry Chi commented on HIVE-7139: - Is there really no planned fix or workaround for this? Thanks! > Period in username can't be parsed > -- > > Key: HIVE-7139 > URL: https://issues.apache.org/jira/browse/HIVE-7139 > Project: Hive > Issue Type: Bug > Components: Authentication >Affects Versions: 0.13.0 >Reporter: Andrew Musselman >Priority: Minor > > We have an environment where usernames have period characters between their > first and last names. > This breaks parsing while trying to grant permissions, e.g., > hive> grant all on database mydb to user first.last; > FAILED: ParseException line 1:40 missing EOF at '.' near 'first' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19043) Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for Complex Types
[ https://issues.apache.org/jira/browse/HIVE-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19043: Status: In Progress (was: Patch Available) > Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for > Complex Types > -- > > Key: HIVE-19043 > URL: https://issues.apache.org/jira/browse/HIVE-19043 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19043.01.patch, HIVE-19043.02.patch, > HIVE-19043.03.patch > > > Issues were revealed by vectorizing create_struct_table.q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19043) Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for Complex Types
[ https://issues.apache.org/jira/browse/HIVE-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19043: Attachment: HIVE-19043.03.patch > Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for > Complex Types > -- > > Key: HIVE-19043 > URL: https://issues.apache.org/jira/browse/HIVE-19043 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19043.01.patch, HIVE-19043.02.patch, > HIVE-19043.03.patch > > > Issues were revealed by vectorizing create_struct_table.q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19043) Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for Complex Types
[ https://issues.apache.org/jira/browse/HIVE-19043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19043: Status: Patch Available (was: In Progress) > Vectorization: LazySimpleDeserializeRead fewer fields handling is broken for > Complex Types > -- > > Key: HIVE-19043 > URL: https://issues.apache.org/jira/browse/HIVE-19043 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19043.01.patch, HIVE-19043.02.patch, > HIVE-19043.03.patch > > > Issues were revealed by vectorizing create_struct_table.q -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19037) Vectorization: Miscellaneous cleanup
[
https://issues.apache.org/jira/browse/HIVE-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19037:
Status: Patch Available (was: In Progress)
> Vectorization: Miscellaneous cleanup
>
>
> Key: HIVE-19037
> URL: https://issues.apache.org/jira/browse/HIVE-19037
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19037.01.patch, HIVE-19037.02.patch,
> HIVE-19037.03.patch, HIVE-19037.04.patch
>
>
> # Extraneous INFO logging in VectorReduceSinkCommonOperator
> # NPE in EXPLAIN for some SelectColumnIsTrue vector expressions
> # Don't throw instantiation exception when UDF has NULL SCALAR that cannot
> match vectorization expression description. Catch it early and don't try to
> create vectorization expression for UDF. (try VectorUDFAdaptor instead).
> # Support vectorizing SELECT NULL.
> # Support constant casting from \{STRING|CHAR|VARCHAR} to
> \{LONG|DOUBLE|DECIMAL} case.
> # Suppress fatal Vectorizer exceptions. Put single line exception stack
> trace innotVectorizedReason. Add test only variable
> hive.test.vectorizer.suppress.fatal.exceptions to turn that off so Vectorizer
> execution failure can be found more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19037) Vectorization: Miscellaneous cleanup
[
https://issues.apache.org/jira/browse/HIVE-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19037:
Attachment: HIVE-19037.04.patch
> Vectorization: Miscellaneous cleanup
>
>
> Key: HIVE-19037
> URL: https://issues.apache.org/jira/browse/HIVE-19037
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19037.01.patch, HIVE-19037.02.patch,
> HIVE-19037.03.patch, HIVE-19037.04.patch
>
>
> # Extraneous INFO logging in VectorReduceSinkCommonOperator
> # NPE in EXPLAIN for some SelectColumnIsTrue vector expressions
> # Don't throw instantiation exception when UDF has NULL SCALAR that cannot
> match vectorization expression description. Catch it early and don't try to
> create vectorization expression for UDF. (try VectorUDFAdaptor instead).
> # Support vectorizing SELECT NULL.
> # Support constant casting from \{STRING|CHAR|VARCHAR} to
> \{LONG|DOUBLE|DECIMAL} case.
> # Suppress fatal Vectorizer exceptions. Put single line exception stack
> trace innotVectorizedReason. Add test only variable
> hive.test.vectorizer.suppress.fatal.exceptions to turn that off so Vectorizer
> execution failure can be found more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19037) Vectorization: Miscellaneous cleanup
[
https://issues.apache.org/jira/browse/HIVE-19037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19037:
Status: In Progress (was: Patch Available)
> Vectorization: Miscellaneous cleanup
>
>
> Key: HIVE-19037
> URL: https://issues.apache.org/jira/browse/HIVE-19037
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19037.01.patch, HIVE-19037.02.patch,
> HIVE-19037.03.patch
>
>
> # Extraneous INFO logging in VectorReduceSinkCommonOperator
> # NPE in EXPLAIN for some SelectColumnIsTrue vector expressions
> # Don't throw instantiation exception when UDF has NULL SCALAR that cannot
> match vectorization expression description. Catch it early and don't try to
> create vectorization expression for UDF. (try VectorUDFAdaptor instead).
> # Support vectorizing SELECT NULL.
> # Support constant casting from \{STRING|CHAR|VARCHAR} to
> \{LONG|DOUBLE|DECIMAL} case.
> # Suppress fatal Vectorizer exceptions. Put single line exception stack
> trace innotVectorizedReason. Add test only variable
> hive.test.vectorizer.suppress.fatal.exceptions to turn that off so Vectorizer
> execution failure can be found more easily.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420048#comment-16420048
]
Hive QA commented on HIVE-19018:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9912/dev-support/hive-personality.sh
|
| git revision | master / 3e3292b |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9912/yetus/patch-asflicense-problems.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9912/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19032: Status: Patch Available (was: In Progress) > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch, HIVE-19032.02.patch, > HIVE-19032.03.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420030#comment-16420030
]
Hive QA commented on HIVE-17661:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916911/HIVE-17661.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13058 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
[jira] [Updated] (HIVE-19058) add object owner to HivePrivilegeObject
[ https://issues.apache.org/jira/browse/HIVE-19058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19058: -- Attachment: HIVE-19058.02.patch > add object owner to HivePrivilegeObject > --- > > Key: HIVE-19058 > URL: https://issues.apache.org/jira/browse/HIVE-19058 > Project: Hive > Issue Type: Bug > Components: Security >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-19058.01.patch, HIVE-19058.02.patch > > > this can enable HiveAuthorizer to create policies based on the owner of the > object - for example, only let the owner of a table read/write it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18814) Support Add Partition For Acid tables
[
https://issues.apache.org/jira/browse/HIVE-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18814:
--
Attachment: HIVE-18814.02.patch
> Support Add Partition For Acid tables
> -
>
> Key: HIVE-18814
> URL: https://issues.apache.org/jira/browse/HIVE-18814
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18814.01.patch, HIVE-18814.02.patch
>
>
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual%2BDDL#LanguageManualDDL-AddPartitions]
> Add Partition command creates a {{Partition}} metadata object and sets the
> location to the directory containing data files.
> In current master (Hive 3.0), Add partition on an acid table doesn't fail and
> at read time the data is decorated with row__id but the original transaction
> is 0. I suspect in earlier Hive versions this will throw or return no data.
> Since this new partition didn't have data before, assigning txnid:0 isn't
> going to generate duplicate IDs but it could violate Snapshot Isolation in
> multi stmt txns. Suppose txnid:7 runs {{select * from T}}. Then txnid:8
> adds a partition to T. Now if txnid:7 runs the same query again, it will see
> the data in the new partition.
> This can't be release like this since a delete on this data (added via Add
> partition) will use row_ids with txnid:0 so a later upgrade that sees
> un-compacted may generate row_ids with different txnid (assuming this is
> fixed by then)
>
> One option is follow Load Data approach and create a new delta_x_x/ and
> move/copy the data there.
>
> Another is to allocate a new writeid and save it in Partition metadata. This
> could then be used to decorate data with ROW__IDs. This avoids move/copy but
> retains data "outside" of the table tree which make it more likely that this
> data will be modified in some way which can really break things if done after
> and SQL update/delete on this data have happened.
>
> It performs no validations on add (except for partition spec) so any file
> with any format can be added. It allows add to bucketed tables as well.
> Seems like a very dangerous command. Maybe a better option is to block it
> and advise using Load Data. Alternatively, make this do Add partition
> metadata op followed by Load Data.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19032: Attachment: HIVE-19032.03.patch > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch, HIVE-19032.02.patch, > HIVE-19032.03.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19032) Vectorization: Disable GROUP BY aggregations with DISTINCT
[ https://issues.apache.org/jira/browse/HIVE-19032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19032: Status: In Progress (was: Patch Available) > Vectorization: Disable GROUP BY aggregations with DISTINCT > -- > > Key: HIVE-19032 > URL: https://issues.apache.org/jira/browse/HIVE-19032 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19032.01.patch, HIVE-19032.02.patch, > HIVE-19032.03.patch > > > Vectorized GROUP BY does not support DISTINCT aggregation functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419994#comment-16419994 ] Thejas M Nair commented on HIVE-18841: -- I think we can add that as well in follow up patch, if we see asks for this. The current ask from users that we have seen is to be able to allow/deny access to permanent functions (eg encryption related ones), by name of the functions. > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Status: Patch Available (was: In Progress) > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch, > HIVE-19024.03.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Attachment: HIVE-19024.03.patch > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch, > HIVE-19024.03.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19024) Vectorization: Disable complex type constants for VectorUDFAdaptor
[ https://issues.apache.org/jira/browse/HIVE-19024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19024: Status: In Progress (was: Patch Available) > Vectorization: Disable complex type constants for VectorUDFAdaptor > -- > > Key: HIVE-19024 > URL: https://issues.apache.org/jira/browse/HIVE-19024 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19024.01.patch, HIVE-19024.02.patch > > > Currently, complex type constants are not detected and cause execution > failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419989#comment-16419989
]
Hive QA commented on HIVE-17661:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 1 new + 136 unchanged - 1
fixed = 137 total (was 137) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9911/dev-support/hive-personality.sh
|
| git revision | master / 3e3292b |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9911/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9911/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9911/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18021) Insert overwrite on acid table with Union All optimizations
[
https://issues.apache.org/jira/browse/HIVE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419951#comment-16419951
]
Eugene Koifman commented on HIVE-18021:
---
I think it may generate the same ROW__IDs in each subdir for IOW case - need to
check.
> Insert overwrite on acid table with Union All optimizations
> ---
>
> Key: HIVE-18021
> URL: https://issues.apache.org/jira/browse/HIVE-18021
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> This is a followup from HIVE-14988.
> T is unbucketed acid table
> {noformat}
> insert into T select a,b from S union all select a,b from S1
> {noformat}
> will create a separate subdirectory for each leg of the union in the target
> table
> (automatically on Tez, with some props enabled on MR)
> Regular Insert will make each subdirectory be a delta_x_x_0, delta_x_x_1.
> See HIVE-15899.
> There is no such suffix mechanism for base_x/.
> Need to figure how this should work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419949#comment-16419949
]
Eugene Koifman commented on HIVE-17661:
---
+1
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419947#comment-16419947 ] Eugene Koifman edited comment on HIVE-18747 at 3/29/18 11:46 PM: - There are a few new checkstyle problems, otherwise +1 patch 3 is the change in TxnHandler.getValidWriteIdsForTable() specific to this ticket or a general fix? was (Author: ekoifman): +1 patch 3 is the change in TxnHandler.getValidWriteIdsForTable() specific to this ticket or a general fix? > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Status: Patch Available (was: In Progress)
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Status: In Progress (was: Patch Available)
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18747) Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL.
[ https://issues.apache.org/jira/browse/HIVE-18747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419947#comment-16419947 ] Eugene Koifman commented on HIVE-18747: --- +1 patch 3 is the change in TxnHandler.getValidWriteIdsForTable() specific to this ticket or a general fix? > Cleaner for TXN_TO_WRITE_ID table entries using MIN_HISTORY_LEVEL. > -- > > Key: HIVE-18747 > URL: https://issues.apache.org/jira/browse/HIVE-18747 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Minor > Labels: ACID, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18747.01.patch, HIVE-18747.02.patch, > HIVE-18747.03.patch > > > Per table write ID implementation (HIVE-18192) maintains a map between txn ID > and table write ID in TXN_TO_WRITE_ID meta table. > The entries in this table is used to generate ValidWriteIdList for the given > ValidTxnList to ensure snapshot isolation. > When table or database is dropped, then these entries are cleaned-up. But, it > is necessary to clean-up for active tables too for better performance. > TXN_TO_WRITE_ID table keeps a mapping of Transaction ID to Write ID. The > state of each Write ID (open, committed, aborted) is determined by the state > of the parent transaction. In order to be able to get a WriteIdList that is > accurate wrt ValidTxnList that is locked in at the start of the transaction, > we have to retain txnid<->writeid mapping even after the transaction ends. > This is because a reader at Snapshot Isolation that started when transaction > X was open, should continue to ignore the data written by X even after X > commits. > So we need a mechanism to know when it is safe to remove TXN_TO_WRITE_ID. > There are 2 parts to it. When txn X is opened, it records Y=select > min(txn_id) from TXNS where txn_state=’o’ in MIN_HISTORY(txnid,opentxnid) > table, i.e. it adds (X, Y) to MIN_HISTORY. On commit (and abort) of X, it > removes its own entry from MIN_HISTORY. In the absence of Aborted > transactions, MIN_HISTORY gives us the smallest open txnid across all active > reader snapshots. Let Z=select min(opentxnid) from MIN_HISTORY. We can > delete entries from TXN_TO_WRITE_ID once TXN_TO_WRITE_ID.T2W_TXNID < Z since > every active reader sees txns < Z as committed. > If S is aborted txns, we retain the metadata about it in TXNS as long as any > data written S may be visible to some reader in the system so that the reader > knows to skip this data. The rules for when that is are complex but wrt to > TXN_TO_WRITE_ID, if A=select min(TXN_ID) from TXNS where TXN_STATE=’a’, then > it’s safe to delete from TXN_TO_WRITE_ID when TXN_TO_WRITE_ID.T2W_TXNID < > min(Z,A). > If no open or aborted txns exist in the system, then we need to enable > cleanup using latest allocated value of NEXT_TXN_ID table. Delete condition > would be TXN_TO_WRITE_ID.T2W_TXNID < min(Z,A,NEXT_TXN_ID.ntxn_next). > Also, it is proposed to trigger cleanup on TXN_TO_WRITE_ID from initiator > immediately after cleaning up aborted txns metadata from TXNS table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19019) Vectorization: When vectorized, orc_merge_incompat_schema.q throws HiveException "Not implemented yet" from VectorExpressionWriterMap
[
https://issues.apache.org/jira/browse/HIVE-19019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-19019:
Attachment: HIVE-19019.05.patch
> Vectorization: When vectorized, orc_merge_incompat_schema.q throws
> HiveException "Not implemented yet" from VectorExpressionWriterMap
> -
>
> Key: HIVE-19019
> URL: https://issues.apache.org/jira/browse/HIVE-19019
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19019.01.patch, HIVE-19019.02.patch,
> HIVE-19019.03.patch, HIVE-19019.04.patch, HIVE-19019.05.patch
>
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> orc_merge_incompat_schema.q triggers this call stack:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Not implemented
> yet
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory$19.writeValue(VectorExpressionWriterFactory.java:1496)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFArgDesc.getDeferredJavaObject(VectorUDFArgDesc.java:123)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.setResult(VectorUDFAdaptor.java:199)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.udf.VectorUDFAdaptor.evaluate(VectorUDFAdaptor.java:151)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:955)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.flushDeserializerBatch(VectorMapOperator.java:630)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:698)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1210)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:829)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:154)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> ~[hadoop-mapreduce-client-core-3.0.0-beta1.jar:?]
> {noformat}
> The complex types in VectorExpressionWriterFactory are not fully implemented.
> Also, null_cast.q, nullMap.q, and nested_column_pruning.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18052) Run p-tests on mm tables
[
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419941#comment-16419941
]
Hive QA commented on HIVE-18052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916513/HIVE-18052.18.patch
{color:green}SUCCESS:{color} +1 due to 80 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1270 failed/errored test(s), 12463 tests
executed
*Failed tests:*
{noformat}
TestCopyUtils - did not produce a TEST-*.xml file (likely timed out)
(batchId=231)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=96)
[jira] [Commented] (HIVE-18052) Run p-tests on mm tables
[
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419932#comment-16419932
]
Hive QA commented on HIVE-18052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
12s{color} | {color:red} root: The patch generated 13 new + 2217 unchanged - 9
fixed = 2230 total (was 2226) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch core passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hcatalog-pig-adapter passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/webhcat/java-client: The patch generated 0
new + 107 unchanged - 1 fixed = 107 total (was 108) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 7 new + 884
unchanged - 7 fixed = 891 total (was 891) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 3 new + 157 unchanged - 0
fixed = 160 total (was 157) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} service: The patch generated 3 new + 62 unchanged - 1
fixed = 65 total (was 63) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419911#comment-16419911
]
Sergey Shelukhin commented on HIVE-17661:
-
[~ekoifman] does this make sense?
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17661:
Status: Patch Available (was: Open)
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-17661:
---
Assignee: Sergey Shelukhin
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17661) DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
[
https://issues.apache.org/jira/browse/HIVE-17661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-17661:
Attachment: HIVE-17661.patch
> DBTxnManager.acquireLocks() - MM tables should use shared lock for Insert
> -
>
> Key: HIVE-17661
> URL: https://issues.apache.org/jira/browse/HIVE-17661
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-17661.patch
>
>
> {noformat}
> case INSERT:
> assert t != null;
> if(AcidUtils.isFullAcidTable(t)) {
> compBuilder.setShared();
> }
> else {
> if
> (conf.getBoolVar(HiveConf.ConfVars.HIVE_TXN_STRICT_LOCKING_MODE)) {
> {noformat}
> _if(AcidUtils.isFullAcidTable(t)) {_
> should probably be
> _if(AcidUtils.isAcidTable(t)) {_
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17861) MM tables - multi-IOW is broken
[ https://issues.apache.org/jira/browse/HIVE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419910#comment-16419910 ] Sergey Shelukhin commented on HIVE-17861: - Just tried to change multi_... tests in mm_all to non-transactional, the results are different. > MM tables - multi-IOW is broken > --- > > Key: HIVE-17861 > URL: https://issues.apache.org/jira/browse/HIVE-17861 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > > After HIVE-17856, see if multi IOW was commented out because of IOW issues or > because it's broken in addition to IOW being broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17861) MM tables - multi-IOW is broken
[ https://issues.apache.org/jira/browse/HIVE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17861: Summary: MM tables - multi-IOW is broken (was: MM tables - multi-IOW may be broken) > MM tables - multi-IOW is broken > --- > > Key: HIVE-17861 > URL: https://issues.apache.org/jira/browse/HIVE-17861 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > > After HIVE-17856, see if multi IOW was commented out because of IOW issues or > because it's broken in addition to IOW being broken. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18570: Component/s: Transactions > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > If we treat IOW foo like DELETE FROM foo (to reason about it w.r.t. ACID > semantics), it seems to me this sequence of events is only possible under > read-uncommitted isolation level (so, 2 deletes rows written by 1). > Under any other isolation level rows written by 1 must survive, or there must > be some lock based change in sequence or conflict. > Update: to clarify, if 1 ran an update on rows instead of an insert, and 2 > still ran an IOW/delete, row lock conflict (or equivalent) should cause one > of them to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18570: Fix Version/s: 3.0.0 > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0 > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > If we treat IOW foo like DELETE FROM foo (to reason about it w.r.t. ACID > semantics), it seems to me this sequence of events is only possible under > read-uncommitted isolation level (so, 2 deletes rows written by 1). > Under any other isolation level rows written by 1 must survive, or there must > be some lock based change in sequence or conflict. > Update: to clarify, if 1 ran an update on rows instead of an insert, and 2 > still ran an IOW/delete, row lock conflict (or equivalent) should cause one > of them to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18570: Priority: Blocker (was: Major) > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Blocker > Fix For: 3.0.0 > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > If we treat IOW foo like DELETE FROM foo (to reason about it w.r.t. ACID > semantics), it seems to me this sequence of events is only possible under > read-uncommitted isolation level (so, 2 deletes rows written by 1). > Under any other isolation level rows written by 1 must survive, or there must > be some lock based change in sequence or conflict. > Update: to clarify, if 1 ran an update on rows instead of an insert, and 2 > still ran an IOW/delete, row lock conflict (or equivalent) should cause one > of them to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15995) Syncing metastore table with serde schema
[
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419908#comment-16419908
]
Vihang Karajgaonkar commented on HIVE-15995:
hmm.. this was unexpected. I think we can address this in a separate JIRA.
Technically it looks like a different problem. Can you create a different JIRA
for that? Rest of the patch looks good to me. +1
> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch,
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch,
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can
> alter the 'avro.schema.url' property to update table schema to the next
> version. Updating properties however doesn't affect column list stored in
> metastore DB so the table is not in the newest version when returned from
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
> PARTITIONED BY (dt string)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> LOCATION
> '/tmp/schema-evolution/user_test1'
> TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' =
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18570: Labels: (was: transactions) > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > If we treat IOW foo like DELETE FROM foo (to reason about it w.r.t. ACID > semantics), it seems to me this sequence of events is only possible under > read-uncommitted isolation level (so, 2 deletes rows written by 1). > Under any other isolation level rows written by 1 must survive, or there must > be some lock based change in sequence or conflict. > Update: to clarify, if 1 ran an update on rows instead of an insert, and 2 > still ran an IOW/delete, row lock conflict (or equivalent) should cause one > of them to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18841) Support authorization of UDF usage in hive
[ https://issues.apache.org/jira/browse/HIVE-18841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419897#comment-16419897 ] Daniel Dai commented on HIVE-18841: --- I imagine authorize all UDFs in a particular jar might be useful, is it possible to pass function resources as a parameter to ReadEntity? > Support authorization of UDF usage in hive > -- > > Key: HIVE-18841 > URL: https://issues.apache.org/jira/browse/HIVE-18841 > Project: Hive > Issue Type: New Feature >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Critical > Attachments: HIVE-18841.1.patch > > > It should be possible to create authorization policies on UDF usage. > ie, it should be possible to control who can use certain UDF in their queries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19048) Initscript errors are ignored
[
https://issues.apache.org/jira/browse/HIVE-19048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419892#comment-16419892
]
Bharathkrishna Guruvayoor Murali commented on HIVE-19048:
-
I think --force is present for a reason, that if the user wants to continue
execution even after there is an error in init script.
By default, the init script works as expected (errors are not ignored by
default).
Hence, please let me know if I can close this Jira if there are no other
comments on the behavior.
> Initscript errors are ignored
> -
>
> Key: HIVE-19048
> URL: https://issues.apache.org/jira/browse/HIVE-19048
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Zoltan Haindrich
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
>
> I've been running some queries for a while when I've noticed that my
> initscript has an error; and beeline stops interpreting the initscript after
> encountering the first error.
> {code}
> echo 'invalid;' > init.sql
> echo 'select 1;' > s1.sql
> beeline -u jdbc:hive2://localhost:1/ -n hive -i init.sql -f s1.sql
> [...]
> Running init script init.sql
> 0: jdbc:hive2://localhost:1/> invalid;
> Error: Error while compiling statement: FAILED: ParseException line 1:0
> cannot recognize input near 'invalid' '' '' (state=42000,code=4)
> 0: jdbc:hive2://localhost:1/> select 1;
> [...]
> $ echo $?
> 0
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Status: Patch Available (was: In Progress)
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18741) Add support for Import into Acid table
[ https://issues.apache.org/jira/browse/HIVE-18741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18741: -- Description: This should follow Load Data approach (or use load data directly) Note that import supports partition spec Does import support loading files not created by Export? If so, similarly to HIVE-19029 - should check for Acid meta columns and reject was: This should follow Load Data approach (or use load data directly) Note that import supports partition spec > Add support for Import into Acid table > -- > > Key: HIVE-18741 > URL: https://issues.apache.org/jira/browse/HIVE-18741 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > This should follow Load Data approach (or use load data directly) > Note that import supports partition spec > Does import support loading files not created by Export? If so, similarly to > HIVE-19029 - should check for Acid meta columns and reject -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Status: In Progress (was: Patch Available)
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Attachment: (was: HIVE-19018.1.patch)
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19018) beeline -e now requires semicolon even when used with query from command line
[
https://issues.apache.org/jira/browse/HIVE-19018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19018:
Attachment: HIVE-19018.1.patch
> beeline -e now requires semicolon even when used with query from command line
> -
>
> Key: HIVE-19018
> URL: https://issues.apache.org/jira/browse/HIVE-19018
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19018.1.patch
>
>
> Right now if you execute {{beeline -u "jdbc:hive2://" -e "select 3"}},
> beeline console will wait for you to enter ';". It's a regression from the
> old behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19081) Add partition should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-19081:
-
> Add partition should prevent loading acid files
> ---
>
> Key: HIVE-19081
> URL: https://issues.apache.org/jira/browse/HIVE-19081
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> similar to HIVE-19029
> {{Alter Table T add Partition ...} T is acid should check to make sure input
> files were not copied from another Acid table, i.e. make sure the files don't
> have Acid metadata columns.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19029:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19029.01.patch, HIVE-19029.02.patch,
> HIVE-19029.04.patch
>
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18021) Insert overwrite on acid table with Union All optimizations
[
https://issues.apache.org/jira/browse/HIVE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419880#comment-16419880
]
Sergey Shelukhin commented on HIVE-18021:
-
Is there a correctness problem from separate directories?
> Insert overwrite on acid table with Union All optimizations
> ---
>
> Key: HIVE-18021
> URL: https://issues.apache.org/jira/browse/HIVE-18021
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> This is a followup from HIVE-14988.
> T is unbucketed acid table
> {noformat}
> insert into T select a,b from S union all select a,b from S1
> {noformat}
> will create a separate subdirectory for each leg of the union in the target
> table
> (automatically on Tez, with some props enabled on MR)
> Regular Insert will make each subdirectory be a delta_x_x_0, delta_x_x_1.
> See HIVE-15899.
> There is no such suffix mechanism for base_x/.
> Need to figure how this should work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19029) Load Data should prevent loading acid files
[
https://issues.apache.org/jira/browse/HIVE-19029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419871#comment-16419871
]
Eugene Koifman commented on HIVE-19029:
---
tried various failed tests locally - they either pass or fail w/o the patch in
the same way
committed to master
thanks Jason for the review
> Load Data should prevent loading acid files
> ---
>
> Key: HIVE-19029
> URL: https://issues.apache.org/jira/browse/HIVE-19029
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19029.01.patch, HIVE-19029.02.patch,
> HIVE-19029.04.patch
>
>
> {{Load Data into T}} where T is acid should check to make sure input files
> were not copied from another Acid table, i.e. make sure the files don't have
> Acid metadata columns.
> AcidUtils.MetaData.isRawFormat()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18570) ACID IOW implemented using base may delete too much data
[ https://issues.apache.org/jira/browse/HIVE-18570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18570: Labels: transactions (was: ) > ACID IOW implemented using base may delete too much data > > > Key: HIVE-18570 > URL: https://issues.apache.org/jira/browse/HIVE-18570 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Labels: transactions > > Suppose we have a table with delta_0 insert data. > Txn 1 starts an insert into delta_1. > Txn 2 starts an IOW into base_2. > Txn 2 commits. > Txn 1 commits after txn 2 but its results would be invisible. > If we treat IOW foo like DELETE FROM foo (to reason about it w.r.t. ACID > semantics), it seems to me this sequence of events is only possible under > read-uncommitted isolation level (so, 2 deletes rows written by 1). > Under any other isolation level rows written by 1 must survive, or there must > be some lock based change in sequence or conflict. > Update: to clarify, if 1 ran an update on rows instead of an insert, and 2 > still ran an IOW/delete, row lock conflict (or equivalent) should cause one > of them to fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18827) useless dynamic value exceptions strike back
[
https://issues.apache.org/jira/browse/HIVE-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419864#comment-16419864
]
Sergey Shelukhin commented on HIVE-18827:
-
hmm
> useless dynamic value exceptions strike back
>
>
> Key: HIVE-18827
> URL: https://issues.apache.org/jira/browse/HIVE-18827
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18827.1.patch
>
>
> Looking at ~master, I can see tons of exceptions like this in LLAP log:
> {noformat}
> 2018-02-27T14:07:51,989 WARN [IO-Elevator-Thread-12
> (1515669035295_0909_1_08_000117_0)] impl.RecordReaderImpl:
> NoDynamicValuesException when evaluating predicate. Skipping ORC PPD. Stats:
> numberOfValues: 9750
> intStatistics {
> minimum: 11335
> maximum: 560
> sum: 27648854404
> }
> hasNull: true
> Predicate: (BETWEEN ss_addr_sk
> DynamicValue(RS_27_customer_address_ca_address_sk_min)
> DynamicValue(RS_27_customer_address_ca_address_sk_max))
> org.apache.hadoop.hive.ql.plan.DynamicValue$NoDynamicValuesException: Value
> does not exist in registry: RS_27_customer_address_ca_address_sk_min
> at
> org.apache.hadoop.hive.ql.exec.tez.DynamicValueRegistryTez.getValue(DynamicValueRegistryTez.java:77)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:137)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:423)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:848)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:835)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:335)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:276)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:273)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> ~[hadoop-common-3.0.0.3.0.0.0-776.jar:?]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:273)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:110)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.9.2-SNAPSHOT.jar:0.9.2-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19064) Add mode to support delimited identifiers enclosed within double quotation
[
https://issues.apache.org/jira/browse/HIVE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419858#comment-16419858
]
Hive QA commented on HIVE-19064:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12916512/HIVE-19064.01.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 98 failed/errored test(s), 13435 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=95)
