[jira] [Commented] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438485#comment-15438485 ] Prasanth Jayachandran commented on HIVE-14515: -- +1 > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438484#comment-15438484 ] Prasanth Jayachandran commented on HIVE-14515: -- >From ~18 minutes to ~6 min. Great! > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Attachment: HIVE-14515.02.patch > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Status: Patch Available (was: In Progress) > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch, HIVE-14515.02.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Status: In Progress (was: Patch Available) > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14437) Vectorization: Optimize key misses in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14437: --- Resolution: Fixed Fix Version/s: 2.2.0 Release Note: Vectorization: Optimize key misses in VectorMapJoinFastBytesHashTable (Gopal V, reviewed by Matt McCline) Status: Resolved (was: Patch Available) > Vectorization: Optimize key misses in VectorMapJoinFastBytesHashTable > - > > Key: HIVE-14437 > URL: https://issues.apache.org/jira/browse/HIVE-14437 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.2.0 > > Attachments: HIVE-14437.1.patch > > > Currently, the lookup in VectorMapJoinFastBytesHashTable proceeds until the > max number of metric put conflicts have been reached. > This can have a fast-exit when encountering the first empty slot during the > probe, to speed up looking for non-existent keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14437) Vectorization: Optimize key misses in VectorMapJoinFastBytesHashTable
[ https://issues.apache.org/jira/browse/HIVE-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438447#comment-15438447 ] Gopal V commented on HIVE-14437: Pushed to master, thanks [~mmccline]. > Vectorization: Optimize key misses in VectorMapJoinFastBytesHashTable > - > > Key: HIVE-14437 > URL: https://issues.apache.org/jira/browse/HIVE-14437 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V > Fix For: 2.2.0 > > Attachments: HIVE-14437.1.patch > > > Currently, the lookup in VectorMapJoinFastBytesHashTable proceeds until the > max number of metric put conflicts have been reached. > This can have a fast-exit when encountering the first empty slot during the > probe, to speed up looking for non-existent keys. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Status: Patch Available (was: In Progress) > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Status: In Progress (was: Patch Available) > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14648) LLAP: Avoid private pages in the SSD cache
[ https://issues.apache.org/jira/browse/HIVE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14648: --- Attachment: HIVE-14648.1.patch > LLAP: Avoid private pages in the SSD cache > -- > > Key: HIVE-14648 > URL: https://issues.apache.org/jira/browse/HIVE-14648 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HIVE-14648.1.patch > > > There's no reason for the SSD cache to have private mappings to the cache > file, there's only one reader and the memory overheads aren't worth it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14648) LLAP: Avoid private pages in the SSD cache
[ https://issues.apache.org/jira/browse/HIVE-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-14648: --- Status: Patch Available (was: Open) > LLAP: Avoid private pages in the SSD cache > -- > > Key: HIVE-14648 > URL: https://issues.apache.org/jira/browse/HIVE-14648 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Blocker > Fix For: 2.2.0 > > Attachments: HIVE-14648.1.patch > > > There's no reason for the SSD cache to have private mappings to the cache > file, there's only one reader and the memory overheads aren't worth it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Status: Patch Available (was: Reopened) > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[
https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438428#comment-15438428
]
Matt McCline commented on HIVE-14515:
-
Got rid of the FetchWork Q files. Removed "MapWork" from other Q file names.
Separated ACID update/delete queries into separate files.
I found that the thing that reduced time the most was combining queries.
>From my laptop:
{code}
Done query: schema_evol_orc_acid_part.q elapsedTime=25s
Done query: schema_evol_orc_acid_part_update.q elapsedTime=11s
Done query: schema_evol_orc_acid_table.q elapsedTime=19s
Done query: schema_evol_orc_acid_table_update.q elapsedTime=8s
Done query: schema_evol_orc_acidvec_part.q elapsedTime=23s
Done query: schema_evol_orc_acidvec_part_update.q elapsedTime=10s
Done query: schema_evol_orc_acidvec_table.q elapsedTime=18s
Done query: schema_evol_orc_acidvec_table_update.q elapsedTime=7s
Done query: schema_evol_orc_nonvec_part.q elapsedTime=17s
Done query: schema_evol_orc_nonvec_part_all_complex.q elapsedTime=8s
Done query: schema_evol_orc_nonvec_part_all_primitive.q elapsedTime=15s
Done query: schema_evol_orc_nonvec_table.q elapsedTime=13s
Done query: schema_evol_orc_vec_part.q elapsedTime=16s
Done query: schema_evol_orc_vec_part_all_complex.q elapsedTime=7s
Done query: schema_evol_orc_vec_part_all_primitive.q elapsedTime=14s
Done query: schema_evol_orc_vec_table.q elapsedTime=12s
Done query: schema_evol_stats.q elapsedTime=3s
Done query: schema_evol_text_nonvec_part.q elapsedTime=16s
Done query: schema_evol_text_nonvec_part_all_complex.q elapsedTime=7s
Done query: schema_evol_text_nonvec_part_all_primitive.q elapsedTime=14s
Done query: schema_evol_text_nonvec_table.q elapsedTime=12s
Done query: schema_evol_text_vec_part.q elapsedTime=16s
Done query: schema_evol_text_vec_part_all_complex.q elapsedTime=7s
Done query: schema_evol_text_vec_part_all_primitive.q elapsedTime=14s
Done query: schema_evol_text_vec_table.q elapsedTime=12s
Done query: schema_evol_text_vecrow_part.q elapsedTime=16s
Done query: schema_evol_text_vecrow_part_all_complex.q elapsedTime=7s
Done query: schema_evol_text_vecrow_part_all_primitive.q elapsedTime=14s
Done query: schema_evol_text_vecrow_table.q elapsedTime=12s
{code}
> Schema evolution uses slow INSERT INTO .. VALUES
>
>
> Key: HIVE-14515
> URL: https://issues.apache.org/jira/browse/HIVE-14515
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14515.01.patch
>
>
> Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14515) Schema evolution uses slow INSERT INTO .. VALUES
[ https://issues.apache.org/jira/browse/HIVE-14515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14515: Attachment: HIVE-14515.01.patch > Schema evolution uses slow INSERT INTO .. VALUES > > > Key: HIVE-14515 > URL: https://issues.apache.org/jira/browse/HIVE-14515 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14515.01.patch > > > Use LOAD DATA LOCAL INPATH and INSERT INTO TABLE ... SELECT * FROM instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14610) CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result in input30
[
https://issues.apache.org/jira/browse/HIVE-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438407#comment-15438407
]
Hive QA commented on HIVE-14610:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825166/HIVE-14610.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10460 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input30]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1000/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1000/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1000/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825166 - PreCommit-HIVE-MASTER-Build
> CBO: Calcite Operator To Hive Operator(Calcite Return Path): Fix wrong result
> in input30
>
>
> Key: HIVE-14610
> URL: https://issues.apache.org/jira/browse/HIVE-14610
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-14610.1.patch, HIVE-14610.2.patch
>
>
> Setup:
> {code}
> set hive.mapred.mode=nonstrict;
> create table dest30(a int);
> create table tst_dest30(a int);
> set hive.test.mode=true;
> set hive.test.mode.prefix=tst_;
> insert overwrite table dest30
> select count(1) from src;
> {code}
> Query:
> {code} select * from tst_dest30; {code}
> Expected Result:
> {code} 10 {code}
> Actual Result:
> {code} 500 {code}
> Turning on hive test mode is suppose to generate sampling (implicit
> tablesample clause) but return path isn't honoring hive.test.mode flag
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438382#comment-15438382 ] Ke Jia commented on HIVE-13589: --- Hi [~vihangk1], I update the patch according to your suggestion. Thank you. > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438371#comment-15438371
]
Sergey Shelukhin edited comment on HIVE-14652 at 8/26/16 3:02 AM:
--
The fix (and also a refactor of the class to not have a million-line method).
I have a vague feeling that most of the logic in this method is bogus, but it
may be just because I am missing something, because it apparently works. The
main question is, why do we evaluate UDFs on partition values from the pruned
set for the filters that we purport to remove, if we have just used the same
filters to prune the partitions, so one of the two should be true - either we
cannot eliminate the filter, or the final result of all the expressions is
known to be true (or not matter). So we'd insta-bail as soon as we'd see any
disagreement after evaluation; or have a walk state that indicates the value
doesn't matter.
I don't really know if that's the case or if I'm missing something here.
So for now the fix is to change the new IN logic introduced by HIVE-11424 to
follow the same twisted logic.
Let's see what that breaks.
The problem is that HIVE-11424 changes IN to true if there's a column on the
left side, but, as described above, this IN was used to filter the partitions,
so in the NOT IN case, IN is guaranteed to be false. So, while the "regular"
logic would have confirmed that and then applied NOT to the false constant, the
current code results in NOT being applied to the true constant.
cc [~jcamachorodriguez] [~ashutoshc]
EDIT: I think the old IN logic for UDF on the left hand side might also be
broken the same way, need to take a look
was (Author: sershe):
The fix (and also a refactor of the class to not have a million-line method).
I have a vague feeling that most of the logic in this method is bogus, but it
may be just because I am missing something, because it apparently works. The
main question is, why do we evaluate UDFs on partition values from the pruned
set for the filters that we purport to remove, if we have just used the same
filters to prune the partitions, so one of the two should be true - either we
cannot eliminate the filter, or the final result of all the expressions is
known to be true (or not matter). So we'd insta-bail as soon as we'd see any
disagreement after evaluation; or have a walk state that indicates the value
doesn't matter.
I don't really know if that's the case or if I'm missing something here.
So for now the fix is to change the new IN logic introduced by HIVE-11424 to
follow the same twisted logic.
Let's see what that breaks.
The problem is that HIVE-11424 changes IN to true if there's a column on the
left side, but, as described above, this IN was used to filter the partitions,
so in the NOT IN case, IN is guaranteed to be false. So, while the "regular"
logic would have confirmed that and then applied NOT to the false constant, the
current code results in NOT being applied to the true constant.
cc [~jcamachorodriguez] [~ashutoshc]
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated HIVE-13589: -- Attachment: HIVE-13589.5.patch > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14652:
Status: Patch Available (was: Open)
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14652:
Priority: Blocker (was: Major)
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14652:
Attachment: HIVE-14652.patch
The fix (and also a refactor of the class to not have a million-line method).
I have a vague feeling that most of the logic in this method is bogus, but it
may be just because I am missing something, because it apparently works. The
main question is, why do we evaluate UDFs on partition values from the pruned
set for the filters that we purport to remove, if we have just used the same
filters to prune the partitions, so one of the two should be true - either we
cannot eliminate the filter, or the final result of all the expressions is
known to be true (or not matter). So we'd insta-bail as soon as we'd see any
disagreement after evaluation; or have a walk state that indicates the value
doesn't matter.
I don't really know if that's the case or if I'm missing something here.
So for now the fix is to change the new IN logic introduced by HIVE-11424 to
follow the same twisted logic.
Let's see what that breaks.
The problem is that HIVE-11424 changes IN to true if there's a column on the
left side, but, as described above, this IN was used to filter the partitions,
so in the NOT IN case, IN is guaranteed to be false. So, while the "regular"
logic would have confirmed that and then applied NOT to the false constant, the
current code results in NOT being applied to the true constant.
cc [~jcamachorodriguez] [~ashutoshc]
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14652.patch
>
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12353) When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.
[ https://issues.apache.org/jira/browse/HIVE-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438355#comment-15438355 ] Eugene Koifman commented on HIVE-12353: --- I've update https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions but it could use some editing. > When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it > should not. > --- > > Key: HIVE-12353 > URL: https://issues.apache.org/jira/browse/HIVE-12353 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12353.2.patch, HIVE-12353.3.patch, > HIVE-12353.4.patch, HIVE-12353.6.patch, HIVE-12353.7.patch, > HIVE-12353.8.patch, HIVE-12353.patch > > > One of the things that this method does is delete entries from TXN_COMPONENTS > for partition that it was trying to compact. > This causes Aborted transactions in TXNS to become empty according to > CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be > deleted. > Once they are deleted, data that belongs to these txns is deemed committed... > We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) > states. We should also not delete then entry from markedCleaned() > We'll have separate process that cleans 'f' and 's' records after X minutes > (or after > N records for a given partition exist). > This allows SHOW COMPACTIONS to show some history info and how many times > compaction failed on a given partition (subject to retention interval) so > that we don't have to call markCleaned() on Compactor failures at the same > time preventing Compactor to constantly getting stuck on the same bad > partition/table. > Ideally we'd want to include END_TIME field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14536) Unit test code cleanup
[ https://issues.apache.org/jira/browse/HIVE-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14536: -- Attachment: HIVE-14536.5.patch The patch after reviews, and with HIVE-14625 > Unit test code cleanup > -- > > Key: HIVE-14536 > URL: https://issues.apache.org/jira/browse/HIVE-14536 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-14536.5.patch, HIVE-14536.patch > > > Clean up the itest infrastructure, to create a readable, easy to understand > code -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438310#comment-15438310
]
Sergey Shelukhin commented on HIVE-14652:
-
I think this is partition condition remover. IN like this would return true
(i.e. match any row?) and then not true will become false... great.
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-12353) When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.
[ https://issues.apache.org/jira/browse/HIVE-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438309#comment-15438309 ] Eugene Koifman edited comment on HIVE-12353 at 8/26/16 2:11 AM: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowCompactions has been updated was (Author: ekoifman): https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowCompactions > When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it > should not. > --- > > Key: HIVE-12353 > URL: https://issues.apache.org/jira/browse/HIVE-12353 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12353.2.patch, HIVE-12353.3.patch, > HIVE-12353.4.patch, HIVE-12353.6.patch, HIVE-12353.7.patch, > HIVE-12353.8.patch, HIVE-12353.patch > > > One of the things that this method does is delete entries from TXN_COMPONENTS > for partition that it was trying to compact. > This causes Aborted transactions in TXNS to become empty according to > CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be > deleted. > Once they are deleted, data that belongs to these txns is deemed committed... > We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) > states. We should also not delete then entry from markedCleaned() > We'll have separate process that cleans 'f' and 's' records after X minutes > (or after > N records for a given partition exist). > This allows SHOW COMPACTIONS to show some history info and how many times > compaction failed on a given partition (subject to retention interval) so > that we don't have to call markCleaned() on Compactor failures at the same > time preventing Compactor to constantly getting stuck on the same bad > partition/table. > Ideally we'd want to include END_TIME field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12353) When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.
[ https://issues.apache.org/jira/browse/HIVE-12353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438309#comment-15438309 ] Eugene Koifman commented on HIVE-12353: --- https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowCompactions > When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it > should not. > --- > > Key: HIVE-12353 > URL: https://issues.apache.org/jira/browse/HIVE-12353 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-12353.2.patch, HIVE-12353.3.patch, > HIVE-12353.4.patch, HIVE-12353.6.patch, HIVE-12353.7.patch, > HIVE-12353.8.patch, HIVE-12353.patch > > > One of the things that this method does is delete entries from TXN_COMPONENTS > for partition that it was trying to compact. > This causes Aborted transactions in TXNS to become empty according to > CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be > deleted. > Once they are deleted, data that belongs to these txns is deemed committed... > We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) > states. We should also not delete then entry from markedCleaned() > We'll have separate process that cleans 'f' and 's' records after X minutes > (or after > N records for a given partition exist). > This allows SHOW COMPACTIONS to show some history info and how many times > compaction failed on a given partition (subject to retention interval) so > that we don't have to call markCleaned() on Compactor failures at the same > time preventing Compactor to constantly getting stuck on the same bad > partition/table. > Ideally we'd want to include END_TIME field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11994) SHOW COMPACTIONS should include meta info about the job
[ https://issues.apache.org/jira/browse/HIVE-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438305#comment-15438305 ] Eugene Koifman edited comment on HIVE-11994 at 8/26/16 2:09 AM: This comment is wrong. SHOW COMPACTIONS does show history so this ticket is about adding meta info. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowCompactions is update with additional "state" values was (Author: ekoifman): This comment is wrong. SHOW COMPACTIONS does show history so this ticket is about adding meta info. > SHOW COMPACTIONS should include meta info about the job > --- > > Key: HIVE-11994 > URL: https://issues.apache.org/jira/browse/HIVE-11994 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman > > Currently output of SHOW COMPACTIONS only has current information. > it would very useful if it could retain the data for a configurable period of > time after the Cleaner handles the entry. Perhaps the entry can go into 'h' > state for 'historical'. > Such historical data can be used to drive alert generating logic. > We can add a column with a JSON doc which can include various stats about the > job: number of delta files, size in MB, time taken. Also can be useful to > collect job properties from Hadoop, such as RAM used for example. > All of this can enhance the type/quality of alerts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11994) SHOW COMPACTIONS should include meta info about the job
[ https://issues.apache.org/jira/browse/HIVE-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11994: -- Assignee: (was: Eugene Koifman) > SHOW COMPACTIONS should include meta info about the job > --- > > Key: HIVE-11994 > URL: https://issues.apache.org/jira/browse/HIVE-11994 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman > > Currently output of SHOW COMPACTIONS only has current information. > it would very useful if it could retain the data for a configurable period of > time after the Cleaner handles the entry. Perhaps the entry can go into 'h' > state for 'historical'. > Such historical data can be used to drive alert generating logic. > We can add a column with a JSON doc which can include various stats about the > job: number of delta files, size in MB, time taken. Also can be useful to > collect job properties from Hadoop, such as RAM used for example. > All of this can enhance the type/quality of alerts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11994) SHOW COMPACTIONS should include history and meta info
[ https://issues.apache.org/jira/browse/HIVE-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438305#comment-15438305 ] Eugene Koifman commented on HIVE-11994: --- This comment is wrong. SHOW COMPACTIONS does show history so this ticket is about adding meta info. > SHOW COMPACTIONS should include history and meta info > - > > Key: HIVE-11994 > URL: https://issues.apache.org/jira/browse/HIVE-11994 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > Currently output of SHOW COMPACTIONS only has current information. > it would very useful if it could retain the data for a configurable period of > time after the Cleaner handles the entry. Perhaps the entry can go into 'h' > state for 'historical'. > Such historical data can be used to drive alert generating logic. > We can add a column with a JSON doc which can include various stats about the > job: number of delta files, size in MB, time taken. Also can be useful to > collect job properties from Hadoop, such as RAM used for example. > All of this can enhance the type/quality of alerts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11994) SHOW COMPACTIONS should include meta info about the job
[ https://issues.apache.org/jira/browse/HIVE-11994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-11994: -- Summary: SHOW COMPACTIONS should include meta info about the job (was: SHOW COMPACTIONS should include history and meta info) > SHOW COMPACTIONS should include meta info about the job > --- > > Key: HIVE-11994 > URL: https://issues.apache.org/jira/browse/HIVE-11994 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > Currently output of SHOW COMPACTIONS only has current information. > it would very useful if it could retain the data for a configurable period of > time after the Cleaner handles the entry. Perhaps the entry can go into 'h' > state for 'historical'. > Such historical data can be used to drive alert generating logic. > We can add a column with a JSON doc which can include various stats about the > job: number of delta files, size in MB, time taken. Also can be useful to > collect job properties from Hadoop, such as RAM used for example. > All of this can enhance the type/quality of alerts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11956) SHOW LOCKS should indicate what acquired the lock
[ https://issues.apache.org/jira/browse/HIVE-11956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438302#comment-15438302 ] Eugene Koifman commented on HIVE-11956: --- https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks updated > SHOW LOCKS should indicate what acquired the lock > - > > Key: HIVE-11956 > URL: https://issues.apache.org/jira/browse/HIVE-11956 > Project: Hive > Issue Type: Improvement > Components: CLI, Transactions >Affects Versions: 0.14.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-11956.2.patch, HIVE-11956.patch > > > This can be a queryId, Flume agent id, Storm bolt id, etc. This would > dramatically help diagnosing issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14554) Hive ptest should delete the itests/thirdparty directory everytime it builds hive
[
https://issues.apache.org/jira/browse/HIVE-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438296#comment-15438296
]
Hive QA commented on HIVE-14554:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825167/HIVE-14554.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/999/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/999/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-999/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825167 - PreCommit-HIVE-MASTER-Build
> Hive ptest should delete the itests/thirdparty directory everytime it builds
> hive
> -
>
> Key: HIVE-14554
> URL: https://issues.apache.org/jira/browse/HIVE-14554
> Project: Hive
> Issue Type: Task
> Components: Testing Infrastructure
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14554.1.patch
>
>
> The {{itests/thridparty}} directory is created by hive on spark when
> downloading the spark-assembly file. Hive ptest should delete this directory
> everytime it runs a new set of tests to avoid conflicts when a new spark
> tarball is submitted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11685) Restarting Metastore kills Compactions - store Hadoop job id in COMPACTION_QUEUE
[ https://issues.apache.org/jira/browse/HIVE-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438295#comment-15438295 ] Eugene Koifman commented on HIVE-11685: --- With HIVE-11388 adding JobID in SHOW COMPACTIONS is even more important to be able to grep (now one of several) metastore logs to see which one ran this compaction > Restarting Metastore kills Compactions - store Hadoop job id in > COMPACTION_QUEUE > > > Key: HIVE-11685 > URL: https://issues.apache.org/jira/browse/HIVE-11685 > Project: Hive > Issue Type: Improvement > Components: Metastore, Transactions >Affects Versions: 1.0.1 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > CompactorMR submits MR job to do compaction and waits for completion. > If the metastore need to be restarted, it will kill in-flight compactions. > I ideally we'd want to add job ID to the COMPACTION_QUEUE table (and include > that in SHOW COMPACTIONS) and poll for it or register a callback so that the > job survives Metastore restart > Also, > when running revokeTimedoutWorker() make sure to use this JobId to kill the > job is it's still running. > Alternatively, if it's still running, maybe just a assign a new worker_id and > let it continue to run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438281#comment-15438281 ] Prasanth Jayachandran edited comment on HIVE-14627 at 8/26/16 1:46 AM: --- I looked at all minimr tests and they seem to use only 3 tables from the initial tables (src, srcpart and part). Also none of the tests seem to require column stats. So I created another q_test_init.sql file that loads only these table. With that I repeated the experiment mentioned in the description, the entire test took only 2m33s. The tests that are deleted does not have the corresponding qfile (must have been deleted but not removed from properties file). [~sseth] Can you please take a look? Also we can now increase the batch size for minimr tests (maybe 10? There are 50 tests now, so 5 batches). was (Author: prasanth_j): I looked at all minimr tests and they seem to use only 3 tables from the initial tables (src, srcpart and part). Also none of the tests seem to require column stats. So I created another q_test_init.sql file that loads only these table. With that I repeated the experiment mentioned in the description, the entire test took only 2m33s. The tests that are deleted does not have the corresponding qfile (must have been deleted but not removed from properties file). [~sseth] Can you please take a look? Also we can now increase the batch size for minimr tests (maybe 10?). > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch, HIVE-14627.2.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14627: - Attachment: HIVE-14627.2.patch orc_mr_pathalias.q has been removed from minimr as it is not required to run on minimr. It just tests for prefix matching in different tables doing joins. This is a test for HiveInputFormat. Running it in TestCliDriver will alone be sufficient. > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch, HIVE-14627.2.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438281#comment-15438281 ] Prasanth Jayachandran commented on HIVE-14627: -- I looked at all minimr tests and they seem to use only 3 tables from the initial tables (src, srcpart and part). Also none of the tests seem to require column stats. So I created another q_test_init.sql file that loads only these table. With that I repeated the experiment mentioned in the description, the entire test took only 2m33s. The tests that are deleted does not have the corresponding qfile (must have been deleted but not removed from properties file). [~sseth] Can you please take a look? Also we can now increase the batch size for minimr tests (maybe 10?). > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[
https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438277#comment-15438277
]
Ferdinand Xu commented on HIVE-13589:
-
Hi [~vihangk1]
{noformat}
2. Will this work if if the password is really an empty string (non-null but
empty)?
{noformat}
Now option "- p" required a parameter which means empty string will not be
allowed as a password to pass in. The only way to login with an empty string is
by prompting.
> beeline - support prompt for password with '-u' option
> --
>
> Key: HIVE-13589
> URL: https://issues.apache.org/jira/browse/HIVE-13589
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Thejas M Nair
>Assignee: Ke Jia
> Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch,
> HIVE-13589.3.patch, HIVE-13589.4.patch
>
>
> Specifying connection string using commandline options in beeline is
> convenient, as it gets saved in shell command history, and it is easy to
> retrieve it from there.
> However, specifying the password in command prompt is not secure as it gets
> displayed on screen and saved in the history.
> It should be possible to specify '-p' without an argument to make beeline
> prompt for password.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14652) incorrect results for not in on partition columns
[
https://issues.apache.org/jira/browse/HIVE-14652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438275#comment-15438275
]
Sergey Shelukhin commented on HIVE-14652:
-
The partition pruning is performed correctly and the partition is added to
inputs, locks and all
> incorrect results for not in on partition columns
> -
>
> Key: HIVE-14652
> URL: https://issues.apache.org/jira/browse/HIVE-14652
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.2.0
>Reporter: stephen sprague
>Assignee: Sergey Shelukhin
>
> {noformat}
> create table foo (i int) partitioned by (s string);
> insert overwrite table foo partition(s='foo') select cint from alltypesorc
> limit 10;
> insert overwrite table foo partition(s='bar') select cint from alltypesorc
> limit 10;
> select * from foo where s not in ('bar');
> {noformat}
> No results. IN ... works correctly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14627: - Attachment: HIVE-14627.1.patch > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14627) Improvements to MiniMr tests
[ https://issues.apache.org/jira/browse/HIVE-14627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14627: - Status: Patch Available (was: Open) > Improvements to MiniMr tests > > > Key: HIVE-14627 > URL: https://issues.apache.org/jira/browse/HIVE-14627 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14627.1.patch > > > Currently MiniMr is extremely slow, I ran udf_using.q on MiniMr and following > are the execution time breakdown > Total time - 13m59s > Junit reported time for testcase - 50s > Most of the time is spent in creating/loading/analyzing initial tables - ~12m > Cleanup - ~1m > There is huge overhead for running MiniMr tests when compared to the actual > test runtime. > Ran the same test without init script. > Total time - 2m17s > Junit reported time for testcase - 52s > Also I noticed some tests that doesn't have to run on MiniMr (like > udf_using.q that does not require MiniMr. It just reads/write to hdfs which > we can do in MiniTez/MiniLlap which are way faster). Most tests access only > very few initial tables to read few rows from it. We can fix those tests to > load just the table that is required for the table instead of all initial > tables. Also we can remove q_init_script.sql initialization for MiniMr after > rewriting and moving over the unwanted tests which should cut down the > runtime a lot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14635) establish a separate path for FSOP to write into final path
[ https://issues.apache.org/jira/browse/HIVE-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438235#comment-15438235 ] Sergey Shelukhin edited comment on HIVE-14635 at 8/26/16 1:07 AM: -- The patch that makes one simple query write to final destination, with some temporary q files, todos and logging to verify and debug. I am running q files on LLAP for debug as it's more realistic than CliDriver in terms of what runs where, but is faster than MiniTez. Maybe I should run MiniTez to get separate processes. Also refactors some very long methods for clarity This is intended to be committed into hive-14535 branch. As per subtask division, this is not used for read, FSOP conflicts are not really handled, and all the special cases will break :) [~gopalv] [~prasanth_j] [~ashutoshc] any comments? I assume branch patch does not need a +1 was (Author: sershe): The patch that makes one simple query write to final destination, with some temporary q files, todos and logging to verify and debug. I am running q files on LLAP for debug as it's more realistic than CliDriver in terms of what runs where, but is faster than MiniTez. Maybe I should run MiniTez to get separate processes. This is intended to be committed into hive-14535 branch. As per subtask division, this is not used for read, FSOP conflicts are not really handled, and all the special cases will break :) [~gopalv] [~prasanth_j] [~ashutoshc] any comments? I assume branch patch does not need a +1 > establish a separate path for FSOP to write into final path > --- > > Key: HIVE-14635 > URL: https://issues.apache.org/jira/browse/HIVE-14635 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hive-14535 > > Attachments: HIVE-14635.branch.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14635) establish a separate path for FSOP to write into final path
[ https://issues.apache.org/jira/browse/HIVE-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14635: Fix Version/s: hive-14535 > establish a separate path for FSOP to write into final path > --- > > Key: HIVE-14635 > URL: https://issues.apache.org/jira/browse/HIVE-14635 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hive-14535 > > Attachments: HIVE-14635.branch.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14635) establish a separate path for FSOP to write into final path
[ https://issues.apache.org/jira/browse/HIVE-14635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14635: Attachment: HIVE-14635.branch.patch The patch that makes one simple query write to final destination, with some temporary q files, todos and logging to verify and debug. I am running q files on LLAP for debug as it's more realistic than CliDriver in terms of what runs where, but is faster than MiniTez. Maybe I should run MiniTez to get separate processes. This is intended to be committed into hive-14535 branch. As per subtask division, this is not used for read, FSOP conflicts are not really handled, and all the special cases will break :) [~gopalv] [~prasanth_j] [~ashutoshc] any comments? I assume branch patch does not need a +1 > establish a separate path for FSOP to write into final path > --- > > Key: HIVE-14635 > URL: https://issues.apache.org/jira/browse/HIVE-14635 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hive-14535 > > Attachments: HIVE-14635.branch.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Status: Patch Available (was: Open) > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Attachment: HIVE-14651.01.patch Patch adds a local cluster for tez/llap. It doesn't enable it for either of them though. There's some random failures which need to be debugged. Also contains some minor qtestutil refactoring. Fixes a bug where the config for minillap would be lost for sessions launched after the first one (config read from minillapcluster only once earileri). This manifests as some AMs end up trying to talk to LLAP on the FixedRegistry port - 15001. [~prasanth_j] - could you please take a look. I'll create follow up jiras to enable tests. > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14651.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3827) LATERAL VIEW doesn't work with union all statement
[ https://issues.apache.org/jira/browse/HIVE-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438211#comment-15438211 ] Hajime Osako commented on HIVE-3827: This seems reproducible with Hive 1.2.1. Is anyone looking into this Jira? Or supposed to be fixed? > LATERAL VIEW doesn't work with union all statement > -- > > Key: HIVE-3827 > URL: https://issues.apache.org/jira/browse/HIVE-3827 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.9.0 > Environment: hive0.9.0 hadoop 0.20.205 >Reporter: cyril liao > > LATER VIEW lose data working with union all. > query NO.1: > SELECT > 1 as from_pid, > 1 as to_pid, > cid as from_path, > (CASE WHEN pid=0 THEN cid ELSE pid END) as to_path, > 0 as status > FROM > (SELECT union_map(c_map) AS c_map > FROM > (SELECT collect_map(id,parent_id)AS c_map > FROM > wl_channels > GROUP BY id,parent_id > )tmp > )tmp2 > LATERAL VIEW recursion_concat(c_map) a AS cid, pid > this query returns about 1 rows ,and their status is 0. > query NO.2: > select > a.from_pid as from_pid, > a.to_pid as to_pid, > a.from_path as from_path, > a.to_path as to_path, > a.status as status > from wl_dc_channels a > where a.status <> 0 > this query returns about 100 rows ,and their status is 1 or 2. > query NO.3: > select > from_pid, > to_pid, > from_path, > to_path, > status > from > ( > SELECT > 1 as from_pid, > 1 as to_pid, > cid as from_path, > (CASE WHEN pid=0 THEN cid ELSE pid END) as to_path, > 0 as status > FROM > (SELECT union_map(c_map) AS c_map > FROM > (SELECT collect_map(id,parent_id)AS c_map > FROM > wl_channels > GROUP BY id,parent_id > )tmp > )tmp2 > LATERAL VIEW recursion_concat(c_map) a AS cid, pid > union all > select > a.from_pid as from_pid, > a.to_pid as to_pid, > a.from_path as from_path, > a.to_path as to_path, > a.status as status > from wl_dc_channels a > where a.status <> 0 > ) unin_tbl > this query has the same result as query NO.2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14619) CASE folding can produce wrong expression
[
https://issues.apache.org/jira/browse/HIVE-14619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14619:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Jesus!
> CASE folding can produce wrong expression
> -
>
> Key: HIVE-14619
> URL: https://issues.apache.org/jira/browse/HIVE-14619
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-14619.patch
>
>
> This is a case that was not covered in the testsuite. For the following query:
> {code:sql}
> select (CASE WHEN key = value THEN '1' WHEN true THEN '0' ELSE NULL END)
> from src
> {code}
> Currently, we end up folding the select expression to '0', as we fail bail
> out in the second statement and fail to account that there are two different
> possible values for the CASE expression ('1' and '0').
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14561) Minor ptest2 improvements
[ https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14561: -- Status: Patch Available (was: Reopened) > Minor ptest2 improvements > - > > Key: HIVE-14561 > URL: https://issues.apache.org/jira/browse/HIVE-14561 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14561.01.patch, HIVE-14561.02.patch > > > Re-purposed to track a few more improvements. > - Update spring framework to work with Java8 > - Change elapseTime logging to milliseconds from seconds > - Add thread name to log files. > - Allow an empty logsEndPoint if outputDir is not specified > - Log configuration when starting in a web server > - Allow tests to be run even if no qtests property is set > - Fix an exception on test completion when using FixedExecutionContextProvider -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Component/s: Testing Infrastructure > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14619) CASE folding can produce wrong expression
[
https://issues.apache.org/jira/browse/HIVE-14619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438193#comment-15438193
]
Hive QA commented on HIVE-14619:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825313/HIVE-14619.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/998/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/998/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-998/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825313 - PreCommit-HIVE-MASTER-Build
> CASE folding can produce wrong expression
> -
>
> Key: HIVE-14619
> URL: https://issues.apache.org/jira/browse/HIVE-14619
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-14619.patch
>
>
> This is a case that was not covered in the testsuite. For the following query:
> {code:sql}
> select (CASE WHEN key = value THEN '1' WHEN true THEN '0' ELSE NULL END)
> from src
> {code}
> Currently, we end up folding the select expression to '0', as we fail bail
> out in the second statement and fail to account that there are two different
> possible values for the CASE expression ('1' and '0').
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14636) pass information from FSOP/TezTask to MoveTask to take care of speculative execution and failed tasks
[ https://issues.apache.org/jira/browse/HIVE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438172#comment-15438172 ] Sergey Shelukhin edited comment on HIVE-14636 at 8/26/16 12:20 AM: --- After some discussions and looking at code, it looks like for now we will write a manifest into the FS to replace the FSOp-specific commit logic. MoveTask will go off these manifests and delete them There's no way to pass the payload from FSOp running inside tasks back to the client where MoveTask lives without involving some intermediate storage. As a side note, it appears that Hive does not actually support speculative execution provided by Tez, so we never call canCommit. Update: Hive actually just chooses the biggest file among different task attempts, so this can still be coordinated by choosing the biggest file among those in manifests. Retroactive failures are impossible for final outputs, and at any rate it doesn't seem like they should happen against persistent FS if Hive could write final outputs from intermediate stages - otherwise it would mean HDFS/S3/... is losing data. So \*waves hands vigorously\* manifests will not collide the same way the existing FSOP moves do not collide. cc [~gopalv] [~sseth] [~ashutoshc] was (Author: sershe): After some discussions and looking at code, it looks like for now we will write a manifest into the FS to replace the FSOp-specific commit logic. MoveTask will go off these manifests and delete them There's no way to pass the payload from FSOp running inside tasks back to the client where MoveTask lives without involving some intermediate storage. As a side note, it appears that Hive does not actually support speculative execution provided by Tez, so we never call canCommit. Retroactive failures are impossible for final outputs, and at any rate it doesn't seem like they should happen against persistent FS if Hive could write final outputs from intermediate stages - otherwise it would mean HDFS/S3/... is losing data. So \*waves hands vigorously\* manifests will not collide the same way the existing FSOP moves do not collide. cc [~gopalv] [~sseth] [~ashutoshc] > pass information from FSOP/TezTask to MoveTask to take care of speculative > execution and failed tasks > - > > Key: HIVE-14636 > URL: https://issues.apache.org/jira/browse/HIVE-14636 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14651) Add a local cluster for Tez and LLAP
[ https://issues.apache.org/jira/browse/HIVE-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14651: -- Issue Type: Sub-task (was: Task) Parent: HIVE-13503 > Add a local cluster for Tez and LLAP > > > Key: HIVE-14651 > URL: https://issues.apache.org/jira/browse/HIVE-14651 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14636) pass information from FSOP/TezTask to MoveTask to take care of speculative execution and failed tasks
[ https://issues.apache.org/jira/browse/HIVE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438172#comment-15438172 ] Sergey Shelukhin commented on HIVE-14636: - After some discussions and looking at code, it looks like for now we will write a manifest into the FS to replace the FSOp-specific commit logic. MoveTask will go off these manifests and delete them There's no way to pass the payload from FSOp running inside tasks back to the client where MoveTask lives without involving some intermediate storage. As a side note, it appears that Hive does not actually support speculative execution provided by Tez, so we never call canCommit. Retroactive failures are impossible for final outputs, and at any rate it doesn't seem like they should happen against persistent FS if Hive could write final outputs from intermediate stages - otherwise it would mean HDFS/S3/... is losing data. So \*waves hands vigorously\* manifests will not collide the same way the existing FSOP moves do not collide. > pass information from FSOP/TezTask to MoveTask to take care of speculative > execution and failed tasks > - > > Key: HIVE-14636 > URL: https://issues.apache.org/jira/browse/HIVE-14636 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14636) pass information from FSOP/TezTask to MoveTask to take care of speculative execution and failed tasks
[ https://issues.apache.org/jira/browse/HIVE-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-14636: --- Assignee: Sergey Shelukhin > pass information from FSOP/TezTask to MoveTask to take care of speculative > execution and failed tasks > - > > Key: HIVE-14636 > URL: https://issues.apache.org/jira/browse/HIVE-14636 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14487) Add REBUILD statement for materialized views
[
https://issues.apache.org/jira/browse/HIVE-14487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438157#comment-15438157
]
Eugene Koifman commented on HIVE-14487:
---
What happens to the view while it's being rebuilt? Is it still usable in it's
non-materialized version or does raise an error?
> Add REBUILD statement for materialized views
>
>
> Key: HIVE-14487
> URL: https://issues.apache.org/jira/browse/HIVE-14487
> Project: Hive
> Issue Type: Sub-task
> Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Alan Gates
>
> Support for rebuilding existing materialized views. The statement is the
> following:
> {code:sql}
> ALTER MATERIALIZED VIEW [db_name.]materialized_view_name REBUILD;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14462) Reduce number of partition check calls in add_partitions
[ https://issues.apache.org/jira/browse/HIVE-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438140#comment-15438140 ] Rajesh Balamohan commented on HIVE-14462: - [~ashutoshc] - I have reverted the metastore changes in .8 version. > Reduce number of partition check calls in add_partitions > > > Key: HIVE-14462 > URL: https://issues.apache.org/jira/browse/HIVE-14462 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14462.1.patch, HIVE-14462.2.patch, > HIVE-14462.3.patch, HIVE-14462.4.patch, HIVE-14462.6.patch, > HIVE-14462.7.patch, HIVE-14462.8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
[ https://issues.apache.org/jira/browse/HIVE-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-14612: - Resolution: Fixed Status: Resolved (was: Patch Available) > org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout > failure > > > Key: HIVE-14612 > URL: https://issues.apache.org/jira/browse/HIVE-14612 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14612.1.patch > > > Failing for some time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14462) Reduce number of partition check calls in add_partitions
[ https://issues.apache.org/jira/browse/HIVE-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-14462: Attachment: HIVE-14462.8.patch > Reduce number of partition check calls in add_partitions > > > Key: HIVE-14462 > URL: https://issues.apache.org/jira/browse/HIVE-14462 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14462.1.patch, HIVE-14462.2.patch, > HIVE-14462.3.patch, HIVE-14462.4.patch, HIVE-14462.6.patch, > HIVE-14462.7.patch, HIVE-14462.8.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
[ https://issues.apache.org/jira/browse/HIVE-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438123#comment-15438123 ] Prasanth Jayachandran commented on HIVE-14612: -- I see. Makes sense. +1 > org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout > failure > > > Key: HIVE-14612 > URL: https://issues.apache.org/jira/browse/HIVE-14612 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14612.1.patch > > > Failing for some time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
[
https://issues.apache.org/jira/browse/HIVE-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438121#comment-15438121
]
Hari Sankar Sivarama Subramaniyan edited comment on HIVE-14612 at 8/25/16 11:21
PM:
[~prasanth_j] The reason we are changing the layout is because the console
layout is defined in hive-log4j2.properties via the following lines:
{code}
rootLogger.appenderRef.console.ref = console
# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
{code}
And, the hardcoded patterns are used only if the layout for the console was set
to null, which in this case is not so.
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java#L213
Also, I tested via the debugger (with and without HIVE-14405) to ensure that
this is the case.
was (Author: hsubramaniyan):
[~prasanth_j] The reason we are changing the layout is because the console
layout is defined in hive-log4j2.properties via the following lines:
rootLogger.appenderRef.console.ref = console
# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
And, the hardcoded patterns are used only if the layout for the console was set
to null, which in this case is not so.
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java#L213
Also, I tested via the debugger (with and without HIVE-14405) to ensure that
this is the case.
> org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
> failure
>
>
> Key: HIVE-14612
> URL: https://issues.apache.org/jira/browse/HIVE-14612
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14612.1.patch
>
>
> Failing for some time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
[
https://issues.apache.org/jira/browse/HIVE-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438121#comment-15438121
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-14612:
--
[~prasanth_j] The reason we are changing the layout is because the console
layout is defined in hive-log4j2.properties via the following lines:
rootLogger.appenderRef.console.ref = console
# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
And, the hardcoded patterns are used only if the layout for the console was set
to null, which in this case is not so.
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java#L213
Also, I tested via the debugger (with and without HIVE-14405) to ensure that
this is the case.
> org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
> failure
>
>
> Key: HIVE-14612
> URL: https://issues.apache.org/jira/browse/HIVE-14612
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-14612.1.patch
>
>
> Failing for some time
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14650) Select fails when ORC file has more columns than table schema
[
https://issues.apache.org/jira/browse/HIVE-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Mink updated HIVE-14650:
-
Priority: Minor (was: Major)
> Select fails when ORC file has more columns than table schema
> -
>
> Key: HIVE-14650
> URL: https://issues.apache.org/jira/browse/HIVE-14650
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
>Reporter: Jeff Mink
>Priority: Minor
>
> When SELECTing from a Hive ORC table, the following IndexOutOfBoundsException
> is thrown if the underlying ORC file has 4 or more columns than the Hive
> schema (where N is the number of columns in the ORC file).
> {noformat}
> Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> 16/08/25 15:22:19 ERROR CliDriver: Failed with exception
> java.io.IOException:java.lang.IndexOutOfBoundsException: toIndex = N
> java.io.IOException: java.lang.IndexOutOfBoundsException: toIndex = N
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:507)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:414)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1686)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IndexOutOfBoundsException: toIndex = 6
> at java.util.ArrayList.subListRangeCheck(ArrayList.java:1004)
> at java.util.ArrayList.subList(ArrayList.java:996)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.getSchemaOnRead(RecordReaderFactory.java:161)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderFactory.createTreeReader(RecordReaderFactory.java:66)
> at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:202)
> at
> org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:541)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1216)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1113)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:673)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:323)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:445)
> ... 15 more
> {noformat}
> This error appears to be related to the patch of HIVE-10591.
> Steps to reproduce (Hive QL):
> {noformat}
> DROP TABLE IF EXISTS orc_drop_column;
> CREATE TABLE orc_drop_column (`id` int, `name` string, `description` string,
> `somevalue` double, `someflag` boolean, `somedate` timestamp) STORED AS ORC;
> INSERT INTO TABLE orc_drop_column select * from (select 1, 'my_name',
> 'my_desc', 5.5, true, '2016-08-25 06:00:00') a;
> ALTER TABLE orc_drop_column SET SERDE
> 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe';
> ALTER TABLE orc_drop_column REPLACE COLUMNS (
> `id` int,
> `name` string
> );
> ALTER TABLE orc_drop_column SET SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde';
> SELECT id, name FROM orc_drop_column;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14642) handle insert overwrite, load
[ https://issues.apache.org/jira/browse/HIVE-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14642: Summary: handle insert overwrite, load (was: handle insert overwrite) > handle insert overwrite, load > - > > Key: HIVE-14642 > URL: https://issues.apache.org/jira/browse/HIVE-14642 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13555) Add nullif udf
[
https://issues.apache.org/jira/browse/HIVE-13555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-13555:
---
Assignee: Zoltan Haindrich
> Add nullif udf
> --
>
> Key: HIVE-13555
> URL: https://issues.apache.org/jira/browse/HIVE-13555
> Project: Hive
> Issue Type: Sub-task
> Components: UDF
>Reporter: Ashutosh Chauhan
>Assignee: Zoltan Haindrich
>
> {{nullif(exp1, exp2)}} is shorthand for: {{case when exp1 = exp2 then null
> else exp1}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437866#comment-15437866 ] Peter Vary commented on HIVE-14426: --- The tests failures are not related > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14426.2.patch, HIVE-14426.3.patch, > HIVE-14426.4.patch, HIVE-14426.5.patch, HIVE-14426.6.patch, HIVE-14426.patch > > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14426) Extensive logging on info level in WebHCat
[
https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437859#comment-15437859
]
Hive QA commented on HIVE-14426:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825521/HIVE-14426.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10462 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_part_exist]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view_partitioned]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partitions_json]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/997/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/997/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-997/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825521 - PreCommit-HIVE-MASTER-Build
> Extensive logging on info level in WebHCat
> --
>
> Key: HIVE-14426
> URL: https://issues.apache.org/jira/browse/HIVE-14426
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14426.2.patch, HIVE-14426.3.patch,
> HIVE-14426.4.patch, HIVE-14426.5.patch, HIVE-14426.6.patch, HIVE-14426.patch
>
>
> There is an extensive logging in WebHCat at info level, and even some
> sensitive information could be logged
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11957) SHOW TRANSACTIONS should show queryID/agent id of the creator
[ https://issues.apache.org/jira/browse/HIVE-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11957: - Attachment: HIVE-11957.1.patch patch 1 for adding txn start time and last heartbeat time to SHOW TRANSACTIONS output > SHOW TRANSACTIONS should show queryID/agent id of the creator > - > > Key: HIVE-11957 > URL: https://issues.apache.org/jira/browse/HIVE-11957 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-11957.1.patch > > > this would be very useful for debugging > should also include heartbeat/create timestamps > would be nice to support some filtering/sorting options, like sort by create > time, agent id. filter by table, database, etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11957) SHOW TRANSACTIONS should show queryID/agent id of the creator
[ https://issues.apache.org/jira/browse/HIVE-11957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11957: - Target Version/s: 2.2.0 Status: Patch Available (was: Open) > SHOW TRANSACTIONS should show queryID/agent id of the creator > - > > Key: HIVE-11957 > URL: https://issues.apache.org/jira/browse/HIVE-11957 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-11957.1.patch > > > this would be very useful for debugging > should also include heartbeat/create timestamps > would be nice to support some filtering/sorting options, like sort by create > time, agent id. filter by table, database, etc -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14618) beeline fetch logging delays before query completion
[ https://issues.apache.org/jira/browse/HIVE-14618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437825#comment-15437825 ] Tao Li commented on HIVE-14618: --- Submitted HIVE-14618.3.patch to fix the test issue. > beeline fetch logging delays before query completion > > > Key: HIVE-14618 > URL: https://issues.apache.org/jira/browse/HIVE-14618 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14618.1.patch, HIVE-14618.2.patch, > HIVE-14618.3.patch > > > Beeline has a thread that fetches logs from HS2. However, it uses the same > HiveStatement object to also wait for query completion using a long-poll > (with default interval of 5 seconds). > The jdbc client has a lock around the thrift api calls, resulting in the > getLogs api blocking on the query completion check. ie the logs would get > shown only every 5 seconds by default. > cc [~vgumashta] [~gopalv] [~thejas] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14603) AssertionError when running explainuser_3.q under MiniLlap
[
https://issues.apache.org/jira/browse/HIVE-14603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437823#comment-15437823
]
Siddharth Seth commented on HIVE-14603:
---
The steBatchSize exception happens when moving from the HBase metastore to the
sql metastore, even with MinTezCluster. Will attach a small patch for this
shortly. [~mmccline] - may need some help from you on this. We should at least
figure out if these are product issues or test issues.
> AssertionError when running explainuser_3.q under MiniLlap
> --
>
> Key: HIVE-14603
> URL: https://issues.apache.org/jira/browse/HIVE-14603
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>
> When running explainuser_3.q under TestMiniLlapCliDriver, following error is
> thrown
> {code}
> ], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) :
> attempt_1471903600528_0001_32_01_00_3:java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.AssertionError
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: java.lang.AssertionError
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:395)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:237)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: java.lang.AssertionError
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setBatchSize(VectorizedBatchUtil.java:125)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:459)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:386)
> ... 18 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14618) beeline fetch logging delays before query completion
[ https://issues.apache.org/jira/browse/HIVE-14618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Li updated HIVE-14618: -- Attachment: HIVE-14618.3.patch > beeline fetch logging delays before query completion > > > Key: HIVE-14618 > URL: https://issues.apache.org/jira/browse/HIVE-14618 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14618.1.patch, HIVE-14618.2.patch, > HIVE-14618.3.patch > > > Beeline has a thread that fetches logs from HS2. However, it uses the same > HiveStatement object to also wait for query completion using a long-poll > (with default interval of 5 seconds). > The jdbc client has a lock around the thrift api calls, resulting in the > getLogs api blocking on the query completion check. ie the logs would get > shown only every 5 seconds by default. > cc [~vgumashta] [~gopalv] [~thejas] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14418) Hive config validation prevents unsetting the settings
[
https://issues.apache.org/jira/browse/HIVE-14418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14418:
Attachment: HIVE-14418.04.patch
Updated to reuse ResetProcessor
> Hive config validation prevents unsetting the settings
> --
>
> Key: HIVE-14418
> URL: https://issues.apache.org/jira/browse/HIVE-14418
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14418.01.patch, HIVE-14418.02.patch,
> HIVE-14418.03.patch, HIVE-14418.04.patch, HIVE-14418.patch
>
>
> {noformat}
> hive> set hive.tez.task.scale.memory.reserve.fraction.max=;
> Query returned non-zero code: 1, cause: 'SET
> hive.tez.task.scale.memory.reserve.fraction.max=' FAILED because
> hive.tez.task.scale.memory.reserve.fraction.max expects FLOAT type value.
> hive> set hive.tez.task.scale.memory.reserve.fraction.max=null;
> Query returned non-zero code: 1, cause: 'SET
> hive.tez.task.scale.memory.reserve.fraction.max=null' FAILED because
> hive.tez.task.scale.memory.reserve.fraction.max expects FLOAT type value.
> {noformat}
> unset also doesn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-14647) Typo fixes in Beeline help
[ https://issues.apache.org/jira/browse/HIVE-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran resolved HIVE-14647. -- Resolution: Fixed Fix Version/s: 2.2.0 Thanks [~mbalassi] for the contribution! > Typo fixes in Beeline help > -- > > Key: HIVE-14647 > URL: https://issues.apache.org/jira/browse/HIVE-14647 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 2.2.0 >Reporter: Márton Balassi >Assignee: Márton Balassi > Fix For: 2.2.0 > > > https://github.com/apache/hive/pull/99 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14546) TestMiniTezCliDriver plan diffs
[
https://issues.apache.org/jira/browse/HIVE-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Sankar Sivarama Subramaniyan resolved HIVE-14546.
--
Resolution: Cannot Reproduce
> TestMiniTezCliDriver plan diffs
> ---
>
> Key: HIVE-14546
> URL: https://issues.apache.org/jira/browse/HIVE-14546
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> The following tests are failing consistently for the last some days in
> precommit runs:
> {code}
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez_join_hash]
> 56 sec 5
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_2]
> 35 sec 19
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1]
> 58 sec 19
>
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part1]
>2.8 sec 19
>
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[transform_ppr1]
>2.4 sec 19
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14546) TestMiniTezCliDriver plan diffs
[
https://issues.apache.org/jira/browse/HIVE-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437728#comment-15437728
]
Hari Sankar Sivarama Subramaniyan commented on HIVE-14546:
--
Verified these have been resolved by HIVE-14502. Closing as cannot reproduce.
> TestMiniTezCliDriver plan diffs
> ---
>
> Key: HIVE-14546
> URL: https://issues.apache.org/jira/browse/HIVE-14546
> Project: Hive
> Issue Type: Sub-task
>Reporter: Hari Sankar Sivarama Subramaniyan
>Assignee: Hari Sankar Sivarama Subramaniyan
>
> The following tests are failing consistently for the last some days in
> precommit runs:
> {code}
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[tez_join_hash]
> 56 sec 5
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_2]
> 35 sec 19
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1]
> 58 sec 19
>
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part1]
>2.8 sec 19
>
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[transform_ppr1]
>2.4 sec 19
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14647) Typo fixes in Beeline help
[ https://issues.apache.org/jira/browse/HIVE-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437725#comment-15437725 ] ASF GitHub Bot commented on HIVE-14647: --- Github user asfgit closed the pull request at: https://github.com/apache/hive/pull/99 > Typo fixes in Beeline help > -- > > Key: HIVE-14647 > URL: https://issues.apache.org/jira/browse/HIVE-14647 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 2.2.0 >Reporter: Márton Balassi >Assignee: Márton Balassi > > https://github.com/apache/hive/pull/99 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14619) CASE folding can produce wrong expression
[
https://issues.apache.org/jira/browse/HIVE-14619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437716#comment-15437716
]
Ashutosh Chauhan commented on HIVE-14619:
-
+1
> CASE folding can produce wrong expression
> -
>
> Key: HIVE-14619
> URL: https://issues.apache.org/jira/browse/HIVE-14619
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Attachments: HIVE-14619.patch
>
>
> This is a case that was not covered in the testsuite. For the following query:
> {code:sql}
> select (CASE WHEN key = value THEN '1' WHEN true THEN '0' ELSE NULL END)
> from src
> {code}
> Currently, we end up folding the select expression to '0', as we fail bail
> out in the second statement and fail to account that there are two different
> possible values for the CASE expression ('1' and '0').
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Sharma updated HIVE-13966: Attachment: HIVE-13966.pdf > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14462) Reduce number of partition check calls in add_partitions
[ https://issues.apache.org/jira/browse/HIVE-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437704#comment-15437704 ] Ashutosh Chauhan commented on HIVE-14462: - Thanks Sergey for pointing that out. My bad I missed it. Reverted the patch. [~rajesh.balamohan] Lets revert changes from HiveMetastore.java totally for this patch and take that improvement in a follow-up. We can get in msck related changes in this jira. > Reduce number of partition check calls in add_partitions > > > Key: HIVE-14462 > URL: https://issues.apache.org/jira/browse/HIVE-14462 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14462.1.patch, HIVE-14462.2.patch, > HIVE-14462.3.patch, HIVE-14462.4.patch, HIVE-14462.6.patch, HIVE-14462.7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14647) Typo fixes in Beeline help
[ https://issues.apache.org/jira/browse/HIVE-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14647: - Reporter: Márton Balassi (was: Prasanth Jayachandran) > Typo fixes in Beeline help > -- > > Key: HIVE-14647 > URL: https://issues.apache.org/jira/browse/HIVE-14647 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 2.2.0 >Reporter: Márton Balassi >Assignee: Márton Balassi > > https://github.com/apache/hive/pull/99 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14561) Minor ptest2 improvements
[ https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14561: -- Attachment: HIVE-14561.02.patch > Minor ptest2 improvements > - > > Key: HIVE-14561 > URL: https://issues.apache.org/jira/browse/HIVE-14561 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14561.01.patch, HIVE-14561.02.patch > > > Re-purposed to track a few more improvements. > - Update spring framework to work with Java8 > - Change elapseTime logging to milliseconds from seconds > - Add thread name to log files. > - Allow an empty logsEndPoint if outputDir is not specified > - Log configuration when starting in a web server > - Allow tests to be run even if no qtests property is set > - Fix an exception on test completion when using FixedExecutionContextProvider -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: (was: HIVE-13930.06.patch) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: (was: HIVE-13930.03.patch) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: (was: HIVE-13930.05.patch) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: (was: HIVE-13930.04.patch) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.07.patch, HIVE-13930.08.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13930: Attachment: HIVE-13930.08.patch Incorporated HIVE-14319, and added some logging for the spark test (it passes locally) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.06.patch, HIVE-13930.07.patch, HIVE-13930.08.patch, > HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-14561) Minor ptest2 improvements
[ https://issues.apache.org/jira/browse/HIVE-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth reopened HIVE-14561: --- [~ashutoshc] pointed out that this caused build failures. Reverted and re-opening. > Minor ptest2 improvements > - > > Key: HIVE-14561 > URL: https://issues.apache.org/jira/browse/HIVE-14561 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14561.01.patch > > > Re-purposed to track a few more improvements. > - Update spring framework to work with Java8 > - Change elapseTime logging to milliseconds from seconds > - Add thread name to log files. > - Allow an empty logsEndPoint if outputDir is not specified > - Log configuration when starting in a web server > - Allow tests to be run even if no qtests property is set > - Fix an exception on test completion when using FixedExecutionContextProvider -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11388) Allow ACID Compactor components to run in multiple metastores
[ https://issues.apache.org/jira/browse/HIVE-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437611#comment-15437611 ] Eugene Koifman commented on HIVE-11388: --- This has been documented. The only real change is in "hive.compactor.initiator.on" of https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-NewConfigurationParametersforTransactions. > Allow ACID Compactor components to run in multiple metastores > - > > Key: HIVE-11388 > URL: https://issues.apache.org/jira/browse/HIVE-11388 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-11388.2.patch, HIVE-11388.4.patch, > HIVE-11388.5.patch, HIVE-11388.6.patch, HIVE-11388.7.patch, > HIVE-11388.branch-1.patch, HIVE-11388.patch > > > (this description is no loner accurate; see further comments) > org.apache.hadoop.hive.ql.txn.compactor.Initiator is a thread that runs > inside the metastore service to manage compactions of ACID tables. There > should be exactly 1 instance of this thread (even with multiple Thrift > services). > This is documented in > https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-Configuration > but not enforced. > Should add enforcement, since more than 1 Initiator could cause concurrent > attempts to compact the same table/partition - which will not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-14426: -- Attachment: HIVE-14426.6.patch Review proposed changes > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14426.2.patch, HIVE-14426.3.patch, > HIVE-14426.4.patch, HIVE-14426.5.patch, HIVE-14426.6.patch, HIVE-14426.patch > > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14249) Add simple materialized views with manual rebuilds
[
https://issues.apache.org/jira/browse/HIVE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437556#comment-15437556
]
Hive QA commented on HIVE-14249:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825412/HIVE-14249.05.patch
{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10482 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_part_exist]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view_partitioned]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partitions_json]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[schemeAuthority2]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/990/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/990/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-990/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825412 - PreCommit-HIVE-MASTER-Build
> Add simple materialized views with manual rebuilds
> --
>
> Key: HIVE-14249
> URL: https://issues.apache.org/jira/browse/HIVE-14249
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views, Parser
>Reporter: Alan Gates
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-10459.2.patch, HIVE-14249.05.patch
>
>
> This patch is a start at implementing simple views. It doesn't have enough
> testing yet (e.g. there's no negative testing). And I know it fails in the
> partitioned case. I suspect things like security and locking don't work
> properly yet either. But I'm posting it as a starting point.
> In this initial patch I'm just handling simple materialized views with manual
> rebuilds. In later JIRAs we can add features such as allowing the optimizer
> to rewrite queries to use materialized views rather than tables named in the
> queries, giving the optimizer the ability to determine when a materialized
> view is stale, etc.
> Also, I didn't rebase this patch against trunk after the migration from
> svn->git so it may not apply cleanly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14625) Minor qtest fixes
[ https://issues.apache.org/jira/browse/HIVE-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14625: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) > Minor qtest fixes > - > > Key: HIVE-14625 > URL: https://issues.apache.org/jira/browse/HIVE-14625 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14625.01.patch, HIVE-14625.02.patch, > HIVE-14625.03.patch > > > Log times for CoreCliDriver > Exit early if cleanup and createsSources fails > Turn PerfLogger off for ptests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14625) Minor qtest fixes
[ https://issues.apache.org/jira/browse/HIVE-14625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437512#comment-15437512 ] Siddharth Seth commented on HIVE-14625: --- Can't reproduce the test failure locally. Will leave fixing it for HIVE-14628. Committing this. Thanks for the review [~prasanth_j] > Minor qtest fixes > - > > Key: HIVE-14625 > URL: https://issues.apache.org/jira/browse/HIVE-14625 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14625.01.patch, HIVE-14625.02.patch, > HIVE-14625.03.patch > > > Log times for CoreCliDriver > Exit early if cleanup and createsSources fails > Turn PerfLogger off for ptests -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14637) edit or split MoveTask to commit job results to metastore
[ https://issues.apache.org/jira/browse/HIVE-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14637: Summary: edit or split MoveTask to commit job results to metastore (was: edit or split MoveTask to commit files to metastore) > edit or split MoveTask to commit job results to metastore > - > > Key: HIVE-14637 > URL: https://issues.apache.org/jira/browse/HIVE-14637 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14535) add micromanaged tables to Hive (metastore keeps track of the files)
[ https://issues.apache.org/jira/browse/HIVE-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437494#comment-15437494 ] Sergey Shelukhin edited comment on HIVE-14535 at 8/25/16 7:30 PM: -- Also it will allow flexibility wrt the actual model (file list, txn, dag; prefix, directory, etc.), as e.g. metastore plumbing and special cases can be done separately, etc. was (Author: sershe): Also it will allow flexibility wrt the actual model as e.g. metastore plumbing and special cases can be done separately, etc. > add micromanaged tables to Hive (metastore keeps track of the files) > > > Key: HIVE-14535 > URL: https://issues.apache.org/jira/browse/HIVE-14535 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Design doc: > https://docs.google.com/document/d/1b3t1RywfyRb73-cdvkEzJUyOiekWwkMHdiQ-42zCllY > Feel free to comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14535) add micromanaged tables to Hive (metastore keeps track of the files)
[ https://issues.apache.org/jira/browse/HIVE-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437494#comment-15437494 ] Sergey Shelukhin commented on HIVE-14535: - Also it will allow flexibility wrt the actual model as e.g. metastore plumbing and special cases can be done separately, etc. > add micromanaged tables to Hive (metastore keeps track of the files) > > > Key: HIVE-14535 > URL: https://issues.apache.org/jira/browse/HIVE-14535 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Design doc: > https://docs.google.com/document/d/1b3t1RywfyRb73-cdvkEzJUyOiekWwkMHdiQ-42zCllY > Feel free to comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14535) add micromanaged tables to Hive (metastore keeps track of the files)
[ https://issues.apache.org/jira/browse/HIVE-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437493#comment-15437493 ] Sergey Shelukhin commented on HIVE-14535: - I think the patch will take too long for me to just do as one patch (maybe it's just cause I'm not very familiar with this part of the codebase). I think I'll create a branch to be able to commit broken code and split the work in subtasks... > add micromanaged tables to Hive (metastore keeps track of the files) > > > Key: HIVE-14535 > URL: https://issues.apache.org/jira/browse/HIVE-14535 > Project: Hive > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Design doc: > https://docs.google.com/document/d/1b3t1RywfyRb73-cdvkEzJUyOiekWwkMHdiQ-42zCllY > Feel free to comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14612) org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout failure
[ https://issues.apache.org/jira/browse/HIVE-14612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437490#comment-15437490 ] Prasanth Jayachandran commented on HIVE-14612: -- I don't understand why HIVE-14405 is changing the pattern layout of the non-verbose in operation logging. IIRC the default is non-verbose mode for operation logging, since operation logs goes to separate log files (LogDivertAppender) it is expected to have a different layout (%m %n). Also the patterns are hardcoded https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java#L57 the expected result is that log lines start with log level. > org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout > failure > > > Key: HIVE-14612 > URL: https://issues.apache.org/jira/browse/HIVE-14612 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-14612.1.patch > > > Failing for some time -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437484#comment-15437484 ] Sergey Shelukhin commented on HIVE-13930: - org.apache.hive.spark.client.TestSparkClient.testJobSubmission and org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] are two "real" failures (at least on HiveQA), the rest are known for a long time or recently broken elsewhere > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.06.patch, HIVE-13930.07.patch, HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14462) Reduce number of partition check calls in add_partitions
[ https://issues.apache.org/jira/browse/HIVE-14462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437478#comment-15437478 ] Sergey Shelukhin commented on HIVE-14462: - It appears that the 3 failures that add partition if it doesn't already exist are broken by this patch :( Perhaps they can be special cased to do the extra check only in this case? org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[add_part_exist] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_view_partitioned] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partitions_json] > Reduce number of partition check calls in add_partitions > > > Key: HIVE-14462 > URL: https://issues.apache.org/jira/browse/HIVE-14462 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14462.1.patch, HIVE-14462.2.patch, > HIVE-14462.3.patch, HIVE-14462.4.patch, HIVE-14462.6.patch, HIVE-14462.7.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14624) LLAP: Use FQDN for all communication
[
https://issues.apache.org/jira/browse/HIVE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437471#comment-15437471
]
Gopal V commented on HIVE-14624:
The impl has to use getCanonicalHostName() instead of just getHostName(), since
"hostname" and "hostname -f" might not match.
> LLAP: Use FQDN for all communication
> -
>
> Key: HIVE-14624
> URL: https://issues.apache.org/jira/browse/HIVE-14624
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Gopal V
> Fix For: 2.2.0
>
>
> {code}
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java:
> + socketAddress.getHostName());
> llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapFixedRegistryImpl.java:
> host = socketAddress.getHostName();
> llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java:
> public static String getHostName() {
> llap-common/src/java/org/apache/hadoop/hive/llap/metrics/MetricsUtils.java:
>return InetAddress.getLocalHost().getHostName();
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:
> String name = address.getHostName();
> llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java:
> builder.setAmHost(address.getHostName());
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java:
>nodeId = LlapNodeId.getInstance(localAddress.get().getHostName(),
> localAddress.get().getPort());
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
> localAddress.get().getHostName(), vertex.getDagName(),
> qIdProto.getDagIndex(),
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
> new ExecutionContextImpl(localAddress.get().getHostName()), env,
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java:
>String hostName = MetricsUtils.getHostName();
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java:
> .setBindAddress(addr.getHostName())
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
> request.getContainerIdString(), executionContext.getHostName(),
> vertex.getDagName(),
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
>String displayName = "LlapDaemonCacheMetrics-" +
> MetricsUtils.getHostName();
> llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java:
>displayName = "LlapDaemonIOMetrics-" + MetricsUtils.getHostName();
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TestLlapDaemonProtocolServerImpl.java:
> new LlapProtocolClientImpl(new Configuration(),
> serverAddr.getHostName(),
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java:
> builder.setAmHost(getAddress().getHostName());
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java:
> String displayName = "LlapTaskSchedulerMetrics-" +
> MetricsUtils.getHostName();
> {code}
> In systems where the hostnames do not match FQDN, calling the
> getCanonicalHostName() will allow for resolution of the hostname when
> accessing from a different base domain.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
