[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095077#comment-17095077
]
Hive QA commented on HIVE-23266:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001470/HIVE-23266.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 17178 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=93)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionNamesPartialValsBogusCatalog[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionNamesPartialValsBogusCatalog[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionsWithPartialValuesBogusCatalog[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionsWithPartialValuesBogusCatalog[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNoDb[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNoDb[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNoTable[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNoTable[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoDb[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoDb[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoTable[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoTable[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNoDb[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNoDb[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNoTable[Embedded]
(batchId=152)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNoTable[Remote]
(batchId=152)
org.apache.hadoop.hive.metastore.tools.metatool.TestHiveMetaTool.testExecuteJDOQL
(batchId=174)
org.apache.hadoop.hive.metastore.tools.metatool.TestMetaToolTaskExecuteJDOQLQuery.testIllegalQuery
(batchId=156)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesByValuesNoTable[Embedded]
(batchId=269)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesByValuesNoTable[Remote]
(batchId=269)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByValuesNoTable[Embedded]
(batchId=269)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByValuesNoTable[Remote]
(batchId=269)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsWithAuthByValuesNoTable[Embedded]
(batchId=269)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsWithAuthByValuesNoTable[Remote]
(batchId=269)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22005/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22005/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22005/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001470 - PreCommit-HIVE-Build
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.2.patch,
> HIVE-23266.2.patch, HIVE-23266.3.patch, HIVE-23266.4.patch,
[jira] [Commented] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095058#comment-17095058
]
Hive QA commented on HIVE-23266:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
16s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 15 new + 268 unchanged - 28 fixed = 283 total (was 296) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
23s{color} | {color:red} standalone-metastore/metastore-server generated 3 new
+ 186 unchanged - 1 fixed = 189 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Impossible cast from String to
org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At
ObjectStore.java:org.apache.hadoop.hive.metastore.model.MPartition in
org.apache.hadoop.hive.metastore.ObjectStore.listPartitionsPsWithAuth(String,
String, String, List, short, String, List) At ObjectStore.java:[line 3162] |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(Table,
List, List, String) At ObjectStore.java:is not thrown in
org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(Table,
List, List, String) At ObjectStore.java:[line 9136] |
| |
org.apache.hadoop.hive.metastore.ObjectStore.getMPartitionColumnStatistics(Table,
List, List, String) concatenates strings using + in a loop At
ObjectStore.java:strings using + in a loop At ObjectStore.java:[line 9116] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22005/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22005/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22005/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22005/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
[jira] [Updated] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22737:
Status: Patch Available (was: Open)
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.5.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22737:
Attachment: HIVE-22737.5.patch
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.5.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22737) Concurrency: FunctionRegistry::getFunctionInfo is static object locked
[
https://issues.apache.org/jira/browse/HIVE-22737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22737:
Status: Open (was: Patch Available)
> Concurrency: FunctionRegistry::getFunctionInfo is static object locked
> --
>
> Key: HIVE-22737
> URL: https://issues.apache.org/jira/browse/HIVE-22737
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer, UDF
>Reporter: Gopal Vijayaraghavan
>Assignee: Ashutosh Chauhan
>Priority: Major
> Attachments: FunctionRegistry-lock.png, HIVE-22737.2.patch,
> HIVE-22737.3.patch, HIVE-22737.4.patch, HIVE-22737.patch
>
>
> The lock is inside a HS2-wide static object
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java#L191
> {code}
> // registry for system functions
> private static final Registry system = new Registry(true);
> {code}
> And this is the lock itself
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Registry.java#L332
> {code}
> public FunctionInfo getFunctionInfo(String functionName) throws
> SemanticException {
> lock.lock();
> {code}
> !FunctionRegistry-lock.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[ https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23294: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Rajesh! > Remove sync bottleneck in TezConfigurationFactory > - > > Key: HIVE-23294 > URL: https://issues.apache.org/jira/browse/HIVE-23294 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot > 2020-04-24 at 1.53.20 PM.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53] > [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628] > It ends up locking for property names in the config. For short running > queries with concurrency, this is an issue. > > !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Attachment: HIVE-23089.6.patch
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch, HIVE-23089.2.patch,
> HIVE-23089.3.patch, HIVE-23089.4.patch, HIVE-23089.5.patch,
> HIVE-23089.6.patch, HIVE-23089.6.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Status: Patch Available (was: Open)
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch, HIVE-23089.2.patch,
> HIVE-23089.3.patch, HIVE-23089.4.patch, HIVE-23089.5.patch,
> HIVE-23089.6.patch, HIVE-23089.6.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Status: Open (was: Patch Available)
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch, HIVE-23089.2.patch,
> HIVE-23089.3.patch, HIVE-23089.4.patch, HIVE-23089.5.patch,
> HIVE-23089.6.patch, HIVE-23089.6.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23304) Two SQL execution results with the same semantics are not the same
[
https://issues.apache.org/jira/browse/HIVE-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gumengchao updated HIVE-23304:
--
Description:
The simplified question is like:
sql_1: select * from (subquery)
sql_2: create temporary table with subquery
and then select * from temporary table
the result of sql_1 and sql_2 is diffrent.
The specific questions are as follows:
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
The result is below.But this is not the result I want
{noformat}
4.063136456211812 162 1150
4.476190476190476 904 1150
4.249370277078086 951 1150
4.1416791604197912301150
3.6464646464646466 19661150
4.163043478260869 23301150
3.7039473684210527 31631150
4.387453874538745 33071150
4.047363717605005 36711150
3.8265682656826567 36751150{noformat}
So I divided sql into two steps
fisrt: create temporary table with subquery
{noformat}
create temporary table tmp as
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10{noformat}
second: use temporary table replace subquery
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a INNER JOIN tmp b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
the result
{noformat}
4.52054794520548745 1150
4.4498902706656915 750 1150
4.476190476190476 904 1150
4.280748663101604 905 1150
3.7314890154597236 10941150
4.189 12361150
4.2104377104377112561150
3.747422680412371 12791150
4.0739348370927315 20641150
4.125390450691656 29971150{noformat}
Why the results of two executions are different?
was:
The simplified question is like
sql_1: select * from (subquery)
sql_2: create temporary table with subquery
and then select * from temporary table
the result of sql_1 and sql_2 is dirffrent.
The specific questions are as follows:
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON
[jira] [Updated] (HIVE-23304) Two SQL execution results with the same semantics are not the same
[
https://issues.apache.org/jira/browse/HIVE-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gumengchao updated HIVE-23304:
--
Description:
The simplified question is like
sql_1: select * from (subquery)
sql_2: create temporary table with subquery
and then select * from temporary table
the result of sql_1 and sql_2 is dirffrent.
The specific questions are as follows:
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
The result is below.But this is not the result I want
{noformat}
4.063136456211812 162 1150
4.476190476190476 904 1150
4.249370277078086 951 1150
4.1416791604197912301150
3.6464646464646466 19661150
4.163043478260869 23301150
3.7039473684210527 31631150
4.387453874538745 33071150
4.047363717605005 36711150
3.8265682656826567 36751150{noformat}
So I divided sql into two steps
fisrt: create temporary table with subquery
{noformat}
create temporary table tmp as
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10{noformat}
second: use temporary table replace subquery
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a INNER JOIN tmp b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
the result
{noformat}
4.52054794520548745 1150
4.4498902706656915 750 1150
4.476190476190476 904 1150
4.280748663101604 905 1150
3.7314890154597236 10941150
4.189 12361150
4.2104377104377112561150
3.747422680412371 12791150
4.0739348370927315 20641150
4.125390450691656 29971150{noformat}
Why the results of two executions are different?
was:
The simplified question is
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
The result is below.But this is not the result I want
{noformat}
4.063136456211812 162 1150
4.476190476190476 904 1150
4.249370277078086 951 1150
4.14167916041979
[jira] [Commented] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095049#comment-17095049
]
Hive QA commented on HIVE-23124:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001467/HIVE-23124.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17178 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat16]
(batchId=86)
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
(batchId=239)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22004/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22004/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22004/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001467 - PreCommit-HIVE-Build
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch, HIVE-23124.2.patch,
> HIVE-23124.3.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23304) Two SQL execution results with the same semantics are not the same
[
https://issues.apache.org/jira/browse/HIVE-23304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gumengchao updated HIVE-23304:
--
Description:
The simplified question is
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
The result is below.But this is not the result I want
{noformat}
4.063136456211812 162 1150
4.476190476190476 904 1150
4.249370277078086 951 1150
4.1416791604197912301150
3.6464646464646466 19661150
4.163043478260869 23301150
3.7039473684210527 31631150
4.387453874538745 33071150
4.047363717605005 36711150
3.8265682656826567 36751150{noformat}
So I divided sql into two steps
fisrt: create temporary table with subquery
{noformat}
create temporary table tmp as
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10{noformat}
second: use temporary table replace subquery
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a INNER JOIN tmp b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
the result
{noformat}
4.52054794520548745 1150
4.4498902706656915 750 1150
4.476190476190476 904 1150
4.280748663101604 905 1150
3.7314890154597236 10941150
4.189 12361150
4.2104377104377112561150
3.747422680412371 12791150
4.0739348370927315 20641150
4.125390450691656 29971150{noformat}
Why the results of two executions are different?
was:
execute sql
{noformat}
SELECT AVG(a.rate), a.movieid, MAX(b.userid)
FROM t_rating a
INNER JOIN (
SELECT a.movieid AS movieid, a.userid AS userid, a.rate
FROM t_rating a
JOIN (
SELECT COUNT(*) AS rate_times, a.userid AS
userid
FROM t_rating a
JOIN (
SELECT *
FROM t_user
WHERE sex = 'F'
) b
ON a.userid = b.userid
GROUP BY a.userid
ORDER BY rate_times DESC
LIMIT 0, 1
) b
ON a.userid = b.userid
ORDER BY a.rate DESC
LIMIT 0, 10
) b
ON a.movieid = b.movieid
GROUP BY a.movieid{noformat}
The result is below.But this is not the result I want
{noformat}
4.063136456211812 162 1150
4.476190476190476 904 1150
4.249370277078086 951 1150
4.1416791604197912301150
3.6464646464646466 19661150
4.163043478260869 23301150
3.7039473684210527 31631150
4.387453874538745 33071150
4.047363717605005 36711150
3.8265682656826567 36751150{noformat}
So I divided sql into two steps
[jira] [Commented] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095026#comment-17095026
]
Hive QA commented on HIVE-23124:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 14 unchanged -
1 fixed = 14 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22004/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22004/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch, HIVE-23124.2.patch,
> HIVE-23124.3.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23108) Cleanup HiveBaseResultSet.java
[
https://issues.apache.org/jira/browse/HIVE-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095020#comment-17095020
]
Hive QA commented on HIVE-23108:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001466/HIVE-23108.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17187 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_insert_partitioned]
(batchId=98)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=80)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[7]
(batchId=142)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22003/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22003/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22003/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001466 - PreCommit-HIVE-Build
> Cleanup HiveBaseResultSet.java
> --
>
> Key: HIVE-23108
> URL: https://issues.apache.org/jira/browse/HIVE-23108
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23108.1.patch, HIVE-23108.2.patch,
> HIVE-23108.3.patch, HIVE-23108.4.patch
>
>
> * Unify the code (there are several implementations of the same thing)
> * Added better error messages
> * In some cases, the code was throwing RuntimeExceptions which is against
> the JDBC Spec
> * Make findColumn a bit more streamlined
> * Remove non-javadoc comments
> * Add {{@Override}} annotations where appropriate
> * Formatting changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23294) Remove sync bottleneck in TezConfigurationFactory
[ https://issues.apache.org/jira/browse/HIVE-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095005#comment-17095005 ] Rajesh Balamohan commented on HIVE-23294: - Tried out .2 in the cluster. Haven't seen major CPU burn with reflection and also removes the locking issue. +1. > Remove sync bottleneck in TezConfigurationFactory > - > > Key: HIVE-23294 > URL: https://issues.apache.org/jira/browse/HIVE-23294 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23294.1.patch, HIVE-23294.2.patch, Screenshot > 2020-04-24 at 1.53.20 PM.png > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezConfigurationFactory.java#L53] > [https://github.com/apache/hadoop/blob/master/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L1628] > It ends up locking for property names in the config. For short running > queries with concurrency, this is an issue. > > !Screenshot 2020-04-24 at 1.53.20 PM.png|width=1086,height=459! > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23108) Cleanup HiveBaseResultSet.java
[
https://issues.apache.org/jira/browse/HIVE-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094991#comment-17094991
]
Hive QA commented on HIVE-23108:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} jdbc: The patch generated 1 new + 0 unchanged - 23
fixed = 1 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} jdbc generated 0 new + 14 unchanged - 2 fixed = 14
total (was 16) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22003/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22003/yetus/diff-checkstyle-jdbc.txt
|
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22003/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cleanup HiveBaseResultSet.java
> --
>
> Key: HIVE-23108
> URL: https://issues.apache.org/jira/browse/HIVE-23108
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23108.1.patch, HIVE-23108.2.patch,
> HIVE-23108.3.patch, HIVE-23108.4.patch
>
>
> * Unify the code (there are several implementations of the same thing)
> * Added better error messages
> * In some cases, the code was throwing RuntimeExceptions which is against
> the JDBC Spec
> * Make findColumn a bit more streamlined
> * Remove non-javadoc comments
> * Add {{@Override}} annotations where appropriate
> * Formatting changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23311) Fix ValidTxnManager regression
[
https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094978#comment-17094978
]
Hive QA commented on HIVE-23311:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001463/HIVE-23311.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17178 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22002/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22002/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22002/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001463 - PreCommit-HIVE-Build
> Fix ValidTxnManager regression
> --
>
> Key: HIVE-23311
> URL: https://issues.apache.org/jira/browse/HIVE-23311
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23311.1.patch
>
>
> During query execution if there are only shared lock tables, the txnList in
> the driverContext should always be considered valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23311) Fix ValidTxnManager regression
[
https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094962#comment-17094962
]
Hive QA commented on HIVE-23311:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22002/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22002/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix ValidTxnManager regression
> --
>
> Key: HIVE-23311
> URL: https://issues.apache.org/jira/browse/HIVE-23311
> Project: Hive
> Issue Type: Bug
> Components: Locking
>Reporter: Peter Varga
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23311.1.patch
>
>
> During query execution if there are only shared lock tables, the txnList in
> the driverContext should always be considered valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094956#comment-17094956
]
Hive QA commented on HIVE-22769:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 89 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 5 new + 36 unchanged - 0 fixed
= 41 total (was 36) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 2 new + 78 unchanged
- 1 fixed = 80 total (was 79) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
3s{color} | {color:red} root: The patch generated 7 new + 114 unchanged - 1
fixed = 121 total (was 115) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
50s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} patch/llap-server cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 16m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22001/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus/diff-checkstyle-llap-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus/diff-checkstyle-root.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus/patch-findbugs-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus/patch-findbugs-llap-server.txt
|
| modules | C: ql llap-server . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22001/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Incorrect query results and query failure during split generation for
>
[jira] [Commented] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094953#comment-17094953
]
Hive QA commented on HIVE-22769:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001456/HIVE-22769.01.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 17182 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[compressed_skip_header_footer_aggr]
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[skip_header_footer_aggr]
(batchId=55)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[skip_header_footer_proj]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[skiphf_aggr]
(batchId=101)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testCombinationInputFormatWithAcid
(batchId=261)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorization
(batchId=261)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid
(batchId=261)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithBuckets
(batchId=261)
org.apache.hadoop.hive.ql.io.orc.TestRandomAccessHiveInputFormat.testOrcSplitOnS3A
(batchId=261)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22001/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22001/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22001/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001456 - PreCommit-HIVE-Build
> Incorrect query results and query failure during split generation for
> compressed text files
> ---
>
> Key: HIVE-22769
> URL: https://issues.apache.org/jira/browse/HIVE-22769
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22769.01.patch, testcase1.csv.bz2, testcase2.csv.bz2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Hive Query produces incorrect results when data is in text format and
> compressed and for certain data the query fails during split generation.
> This behavior is seen when skip.header.line.count and skip.footer.line.count
> are set for table.
> Case 1: Select count/aggregate query produces Incorrect row counts/displays
> all rows (when hive.fetch.task.conversion=none)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
> 3. Run count(*) on table.
> {code}
> > select * from testcase1;
> INFO : Compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string,
> comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.07 seconds
> INFO : Executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Completed executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.007 seconds
> INFO : OK
> +---+-+
> | testcase1.id | testcase1.name |
> +---+-+
> | 2 | 2019-12-31 |
> +---+-+
> 1 row selected (0.111 seconds)
> > select count(*) from testcase1
> INFO : Compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO :
[jira] [Updated] (HIVE-23261) Check whether encryption is enabled in the cluster before moving files
[ https://issues.apache.org/jira/browse/HIVE-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23261: Attachment: HIVE-23261.3.patch Status: Patch Available (was: Open) > Check whether encryption is enabled in the cluster before moving files > -- > > Key: HIVE-23261 > URL: https://issues.apache.org/jira/browse/HIVE-23261 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Minor > Attachments: HIVE-23261.1.patch, HIVE-23261.2.patch, > HIVE-23261.3.patch > > > Similar to HIVE-23212, there is an unwanted check of encryption paths during > file move operation. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4546] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23261) Check whether encryption is enabled in the cluster before moving files
[ https://issues.apache.org/jira/browse/HIVE-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23261: Status: Open (was: Patch Available) > Check whether encryption is enabled in the cluster before moving files > -- > > Key: HIVE-23261 > URL: https://issues.apache.org/jira/browse/HIVE-23261 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Ramesh Kumar Thangarajan >Priority: Minor > Attachments: HIVE-23261.1.patch, HIVE-23261.2.patch > > > Similar to HIVE-23212, there is an unwanted check of encryption paths during > file move operation. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4546] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094924#comment-17094924
]
Hive QA commented on HIVE-23089:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001453/HIVE-23089.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17178 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadata_only_queries_with_filters]
(batchId=106)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22000/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22000/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22000/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001453 - PreCommit-HIVE-Build
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch, HIVE-23089.2.patch,
> HIVE-23089.3.patch, HIVE-23089.4.patch, HIVE-23089.5.patch, HIVE-23089.6.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues
[ https://issues.apache.org/jira/browse/HIVE-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23316: -- Description: Database managed location was recently introduced, but no tests were added to cover it. also the following issues were fixed: * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a separate path as it should, as in DDL each command type have their own Analyzer, Desc, and Operation class * in case of setting the LOCATION or the MANAGEDLOCATION the location was not getting qualified as in the CREATE DATABASE command * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked if this modification makes the two the same * some minor checkstyle issues were fixed as well Also the DDL documentation was not modified. was: Database managed location was recently introduced, but no tests were added to cover it. also the following issues were fixed: * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a separate path as it should, as in DDL each command type have their own Analyzer, Desc, and Operation class * in case of setting the LOCATION or the MANAGEDLOCATION the location was not getting qualified as in the CREATE DATABASE command * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked if this modification makes the two the same * some minor checkstyle issues were fixed as well > Add tests to cover database managed location related DDL and fix minor issues > - > > Key: HIVE-23316 > URL: https://issues.apache.org/jira/browse/HIVE-23316 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23316.01.patch > > > Database managed location was recently introduced, but no tests were added to > cover it. also the following issues were fixed: > * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a > separate path as it should, as in DDL each command type have their own > Analyzer, Desc, and Operation class > * in case of setting the LOCATION or the MANAGEDLOCATION the location was > not getting qualified as in the CREATE DATABASE command > * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked > if this modification makes the two the same > * some minor checkstyle issues were fixed as well > Also the DDL documentation was not modified. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094911#comment-17094911
]
Hive QA commented on HIVE-23089:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 450 unchanged - 10
fixed = 450 total (was 460) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22000/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22000/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch, HIVE-23089.2.patch,
> HIVE-23089.3.patch, HIVE-23089.4.patch, HIVE-23089.5.patch, HIVE-23089.6.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
>
[jira] [Updated] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues
[ https://issues.apache.org/jira/browse/HIVE-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23316: -- Status: Patch Available (was: Open) > Add tests to cover database managed location related DDL and fix minor issues > - > > Key: HIVE-23316 > URL: https://issues.apache.org/jira/browse/HIVE-23316 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23316.01.patch > > > Database managed location was recently introduced, but no tests were added to > cover it. also the following issues were fixed: > * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a > separate path as it should, as in DDL each command type have their own > Analyzer, Desc, and Operation class > * in case of setting the LOCATION or the MANAGEDLOCATION the location was > not getting qualified as in the CREATE DATABASE command > * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked > if this modification makes the two the same > * some minor checkstyle issues were fixed as well -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues
[ https://issues.apache.org/jira/browse/HIVE-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23316: -- Attachment: HIVE-23316.01.patch > Add tests to cover database managed location related DDL and fix minor issues > - > > Key: HIVE-23316 > URL: https://issues.apache.org/jira/browse/HIVE-23316 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Attachments: HIVE-23316.01.patch > > > Database managed location was recently introduced, but no tests were added to > cover it. also the following issues were fixed: > * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a > separate path as it should, as in DDL each command type have their own > Analyzer, Desc, and Operation class > * in case of setting the LOCATION or the MANAGEDLOCATION the location was > not getting qualified as in the CREATE DATABASE command > * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked > if this modification makes the two the same > * some minor checkstyle issues were fixed as well -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23316) Add tests to cover database managed location related DDL and fix minor issues
[ https://issues.apache.org/jira/browse/HIVE-23316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-23316: - > Add tests to cover database managed location related DDL and fix minor issues > - > > Key: HIVE-23316 > URL: https://issues.apache.org/jira/browse/HIVE-23316 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > > Database managed location was recently introduced, but no tests were added to > cover it. also the following issues were fixed: > * ALTER DATABASE ... SET MANAGEDLOCATION ... commands were not handled in a > separate path as it should, as in DDL each command type have their own > Analyzer, Desc, and Operation class > * in case of setting the LOCATION or the MANAGEDLOCATION the location was > not getting qualified as in the CREATE DATABASE command > * in case of setting the LOCATION or the MANAGEDLOCATION it was not checked > if this modification makes the two the same > * some minor checkstyle issues were fixed as well -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?focusedWorklogId=428413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428413
]
ASF GitHub Bot logged work on HIVE-22769:
-
Author: ASF GitHub Bot
Created on: 28/Apr/20 22:28
Start Date: 28/Apr/20 22:28
Worklog Time Spent: 10m
Work Description: mustafaiman commented on a change in pull request #998:
URL: https://github.com/apache/hive/pull/998#discussion_r416947329
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/PassThruOffsetReader.java
##
@@ -20,23 +20,81 @@
import java.io.IOException;
import
org.apache.hadoop.hive.llap.io.encoded.SerDeEncodedDataReader.ReaderWithOffsets;
+import org.apache.hadoop.hive.ql.exec.FooterBuffer;
+import org.apache.hadoop.hive.ql.exec.Utilities;
import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.RecordReader;
@SuppressWarnings("rawtypes") class PassThruOffsetReader implements
ReaderWithOffsets {
protected final RecordReader sourceReader;
protected final Object key;
protected final Writable value;
+ protected final JobConf jobConf;
+ protected final int skipHeaderCnt;
+ protected final int skipFooterCnt;
+ private transient FooterBuffer footerBuffer;
+ private transient boolean initialized = false;
- PassThruOffsetReader(RecordReader sourceReader) {
+ PassThruOffsetReader(RecordReader sourceReader, JobConf jobConf, int
headerCnt, int footerCnt) {
this.sourceReader = sourceReader;
-key = sourceReader.createKey();
-value = (Writable)sourceReader.createValue();
+this.key = sourceReader.createKey();
+this.value = (Writable)sourceReader.createValue();
+this.jobConf = jobConf;
+this.skipHeaderCnt = headerCnt;
+this.skipFooterCnt = footerCnt;
}
@Override
public boolean next() throws IOException {
-return sourceReader.next(key, value);
+try {
+ boolean opNotEOF = true;
+ /**
+ * Start reading a new file.
+ * If file contains header, skip header lines before reading the records.
+ * If file contains footer, used FooterBuffer to cache and remove footer
+ * records at the end of the file.
+ */
+ if (!initialized) {
+// Skip header lines.
+opNotEOF = Utilities.skipHeader(sourceReader, skipFooterCnt, key,
value);
+
+// Initialize footer buffer.
+if (opNotEOF && skipFooterCnt > 0) {
+ footerBuffer = new FooterBuffer();
+ opNotEOF = footerBuffer.initializeBuffer(jobConf, sourceReader,
skipFooterCnt, (WritableComparable) key, value);
+}
+this.initialized = true;
+ }
+
+ if (opNotEOF && footerBuffer == null) {
+/**
+ * When file doesn't end after skipping header line
+ * and there is NO footer lines, read normally.
+ */
+opNotEOF = sourceReader.next(key, value);
+ }
+
+ if (opNotEOF && footerBuffer != null) {
+/**
+ * When file doesn't end after skipping header line
+ * and there IS footer lines, update footerBuffer
+ */
+opNotEOF = footerBuffer.updateBuffer(jobConf, sourceReader,
(WritableComparable) key, value);
+ }
+
+ if (opNotEOF) {
+// File reached the end
+return true;
+ } else {
+// Done reading
+close();
Review comment:
there were no close calls in this method before. How did closing the
file become the responsibility of this class now?
##
File path:
ql/src/test/org/apache/hadoop/hive/ql/io/TestSkippingTextInputFormat.java
##
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.io;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.plan.MapredWork;
+import org.apache.hadoop.hive.ql.plan.PartitionDesc;
+import
[jira] [Commented] (HIVE-23236) Remove the global lock from acquireLock
[
https://issues.apache.org/jira/browse/HIVE-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094899#comment-17094899
]
Hive QA commented on HIVE-23236:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001442/HIVE-23236.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21999/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21999/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21999/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-04-28 22:16:42.592
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-21999/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-04-28 22:16:42.595
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 83a4018 HIVE-23216: Add new api as replacement of
get_partitions_by_expr to return PartitionSpec instead of Partitions (Vineet
Garg, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 83a4018 HIVE-23216: Add new api as replacement of
get_partitions_by_expr to return PartitionSpec instead of Partitions (Vineet
Garg, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-04-28 22:16:43.719
+ rm -rf ../yetus_PreCommit-HIVE-Build-21999
+ mkdir ../yetus_PreCommit-HIVE-Build-21999
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-21999
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21999/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:2429
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
cleanly.
Going to apply patch with: git apply -p0
error: patch failed:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:2429
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java'
cleanly.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc2783848949461299822.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc2783848949461299822.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc6725427903940188305.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
[jira] [Commented] (HIVE-23305) NullPointerException in LlapTaskSchedulerService addNode due to race condition
[
https://issues.apache.org/jira/browse/HIVE-23305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094898#comment-17094898
]
Hive QA commented on HIVE-23305:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001434/HIVE-23305.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17174 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=80)
org.apache.hive.jdbc.TestJdbcWithMiniLlapRow.testComplexQuery (batchId=215)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21998/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21998/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21998/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001434 - PreCommit-HIVE-Build
> NullPointerException in LlapTaskSchedulerService addNode due to race condition
> --
>
> Key: HIVE-23305
> URL: https://issues.apache.org/jira/browse/HIVE-23305
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23305.1.patch
>
>
> {code:java}
> java.lang.NullPointerException at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.addNode(LlapTaskSchedulerService.java:1575)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.registerAndAddNode(LlapTaskSchedulerService.java:1566)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.access$1800(LlapTaskSchedulerService.java:128)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService$NodeStateChangeListener.onCreate(LlapTaskSchedulerService.java:831)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService$NodeStateChangeListener.onCreate(LlapTaskSchedulerService.java:823)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase$InstanceStateChangeListener.childEvent(ZkRegistryBase.java:612)
> at {code}
> The above exception happens when a node registers too fast, before the active
> activeInstances field was initialized.
>
> The registry is started and the listener is registered before initializing
> activeInstances.
> {code:java}
> registry.start();
> registry.registerStateChangeListener(new NodeStateChangeListener());
> activeInstances = registry.getInstances(); {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Make bucketing version usage more robust
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.33.patch > Make bucketing version usage more robust > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch, > HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch, > HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch, > HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch, > HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch, > HIVE-21304.33.patch, HIVE-21304.33.patch > > Time Spent: 50m > Remaining Estimate: 0h > > * Show Bucketing version for ReduceSinkOp in explain extended plan - this > helps identify what hashing algorithm is being used by by ReduceSinkOp. > * move the actually selected version to the "conf" so that it doesn't get lost > * replace trait related logic with a separate optimizer rule > * do version selection based on a group of operator - this is more reliable > * skip bucketingversion selection for tables with 1 buckets > * prefer to use version 2 if possible > * fix operator creations which didn't set a new conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23305) NullPointerException in LlapTaskSchedulerService addNode due to race condition
[
https://issues.apache.org/jira/browse/HIVE-23305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094886#comment-17094886
]
Hive QA commented on HIVE-23305:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} llap-tez in master has 17 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21998/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: llap-tez U: llap-tez |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21998/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> NullPointerException in LlapTaskSchedulerService addNode due to race condition
> --
>
> Key: HIVE-23305
> URL: https://issues.apache.org/jira/browse/HIVE-23305
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23305.1.patch
>
>
> {code:java}
> java.lang.NullPointerException at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.addNode(LlapTaskSchedulerService.java:1575)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.registerAndAddNode(LlapTaskSchedulerService.java:1566)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.access$1800(LlapTaskSchedulerService.java:128)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService$NodeStateChangeListener.onCreate(LlapTaskSchedulerService.java:831)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService$NodeStateChangeListener.onCreate(LlapTaskSchedulerService.java:823)
> at
> org.apache.hadoop.hive.registry.impl.ZkRegistryBase$InstanceStateChangeListener.childEvent(ZkRegistryBase.java:612)
> at {code}
> The above exception happens when a node registers too fast, before the active
> activeInstances field was initialized.
>
> The registry is started and the listener is registered before initializing
> activeInstances.
> {code:java}
> registry.start();
> registry.registerStateChangeListener(new NodeStateChangeListener());
>
[jira] [Updated] (HIVE-23315) Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED VIEWS
[
https://issues.apache.org/jira/browse/HIVE-23315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23315:
--
Status: Patch Available (was: Open)
> Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED
> VIEWS
> --
>
> Key: HIVE-23315
> URL: https://issues.apache.org/jira/browse/HIVE-23315
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23315.01.patch
>
>
> At the end of each SHOW EXTENDED TABLES; and SHOW MATERIALIZED VIEWS; command
> there is an empty line like this:
> {code:java}
> +++
> | tab_name | table_type |
> +++
> | sample_07 | MANAGED_TABLE |
> | sample_08 | MANAGED_TABLE |
> | web_logs | MANAGED_TABLE |
> | | NULL |
> +++
> {code}
> It should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23315) Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED VIEWS
[
https://issues.apache.org/jira/browse/HIVE-23315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23315:
--
Attachment: HIVE-23315.01.patch
> Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED
> VIEWS
> --
>
> Key: HIVE-23315
> URL: https://issues.apache.org/jira/browse/HIVE-23315
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Attachments: HIVE-23315.01.patch
>
>
> At the end of each SHOW EXTENDED TABLES; and SHOW MATERIALIZED VIEWS; command
> there is an empty line like this:
> {code:java}
> +++
> | tab_name | table_type |
> +++
> | sample_07 | MANAGED_TABLE |
> | sample_08 | MANAGED_TABLE |
> | web_logs | MANAGED_TABLE |
> | | NULL |
> +++
> {code}
> It should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23315) Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED VIEWS
[
https://issues.apache.org/jira/browse/HIVE-23315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely reassigned HIVE-23315:
-
> Remove empty line from the end of SHOW EXTENDED TABLES and SHOW MATERIALIZED
> VIEWS
> --
>
> Key: HIVE-23315
> URL: https://issues.apache.org/jira/browse/HIVE-23315
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> At the end of each SHOW EXTENDED TABLES; and SHOW MATERIALIZED VIEWS; command
> there is an empty line like this:
> {code:java}
> +++
> | tab_name | table_type |
> +++
> | sample_07 | MANAGED_TABLE |
> | sample_08 | MANAGED_TABLE |
> | web_logs | MANAGED_TABLE |
> | | NULL |
> +++
> {code}
> It should be removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: hive-23301.3.patch Status: Patch Available (was: Open) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094878#comment-17094878
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001431/HIVE-21304.33.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17179 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=214)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21997/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21997/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21997/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001431 - PreCommit-HIVE-Build
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch,
> HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch,
> HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch,
> HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch,
> HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch,
> HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch,
> HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch,
> HIVE-21304.21.patch, HIVE-21304.22.patch, HIVE-21304.23.patch,
> HIVE-21304.24.patch, HIVE-21304.25.patch, HIVE-21304.26.patch,
> HIVE-21304.27.patch, HIVE-21304.28.patch, HIVE-21304.29.patch,
> HIVE-21304.30.patch, HIVE-21304.31.patch, HIVE-21304.32.patch,
> HIVE-21304.33.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> * Show Bucketing version for ReduceSinkOp in explain extended plan - this
> helps identify what hashing algorithm is being used by by ReduceSinkOp.
> * move the actually selected version to the "conf" so that it doesn't get lost
> * replace trait related logic with a separate optimizer rule
> * do version selection based on a group of operator - this is more reliable
> * skip bucketingversion selection for tables with 1 buckets
> * prefer to use version 2 if possible
> * fix operator creations which didn't set a new conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: (was: hive-23301.3.patch) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Status: Open (was: Patch Available) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23216) Add new api as replacement of get_partitions_by_expr to return PartitionSpec instead of Partitions
[ https://issues.apache.org/jira/browse/HIVE-23216?focusedWorklogId=428365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428365 ] ASF GitHub Bot logged work on HIVE-23216: - Author: ASF GitHub Bot Created on: 28/Apr/20 20:48 Start Date: 28/Apr/20 20:48 Worklog Time Spent: 10m Work Description: vineetgarg02 commented on pull request #990: URL: https://github.com/apache/hive/pull/990#issuecomment-620846239 This has been pushed to hive/master branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428365) Time Spent: 1h 50m (was: 1h 40m) > Add new api as replacement of get_partitions_by_expr to return PartitionSpec > instead of Partitions > -- > > Key: HIVE-23216 > URL: https://issues.apache.org/jira/browse/HIVE-23216 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23216.1.patch, HIVE-23216.2.patch, > HIVE-23216.3.patch, HIVE-23216.4.patch, HIVE-23216.5.patch, > HIVE-23216.6.patch, HIVE-23216.7.patch, HIVE-23216.8.patch, HIVE-23216.9.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Status: Open (was: Patch Available) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: hive-23301.3.patch Status: Patch Available (was: Open) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: (was: hive-23301.3.patch) > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23301) Optimising privilege synchroniser: UDF for updating privileges
[ https://issues.apache.org/jira/browse/HIVE-23301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simhadri G updated HIVE-23301: -- Attachment: hive-23301.3.patch > Optimising privilege synchroniser: UDF for updating privileges > -- > > Key: HIVE-23301 > URL: https://issues.apache.org/jira/browse/HIVE-23301 > Project: Hive > Issue Type: Improvement > Components: Metastore, UDF >Affects Versions: 3.1.1 >Reporter: Simhadri G >Assignee: Simhadri G >Priority: Minor > Attachments: UDFSplitMapPrivs.patch, UDFSplitMapPrivs.patch, > hive-23301-2.patch, hive-23301.3.patch > > > Privilege synchronizer pulls the list of databases, tables and columns from > the Hive Metastore. For each of these objects it fetches the privilege > information and invokes HMS API to refresh the privilege information in HMS. > The current UDF Maps a bit string to a privilege based on if the privilege > is granted or not. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Make bucketing version usage more robust
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094860#comment-17094860
]
Hive QA commented on HIVE-21304:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
37s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 6 new + 1322 unchanged - 12
fixed = 1328 total (was 1334) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
50s{color} | {color:red} ql generated 1 new + 1528 unchanged - 0 fixed = 1529
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Nullcheck of table_desc at line 8232 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:8232 of value previously dereferenced in
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.createFileSinkDesc(String,
TableDesc, Partition, Path, int, boolean, boolean, boolean, Path,
SemanticAnalyzer$SortBucketRSCtx, DynamicPartitionCtx, ListBucketingCtx,
RowSchema, boolean, Table, Long, boolean, Integer, QB, boolean) At
SemanticAnalyzer.java:[line 8225] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21997/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21997/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21997/yetus/new-findbugs-ql.html
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21997/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make bucketing version usage more robust
>
>
> Key: HIVE-21304
> URL: https://issues.apache.org/jira/browse/HIVE-21304
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>
[jira] [Commented] (HIVE-23280) Trigger compaction with old aborted txns
[
https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094833#comment-17094833
]
Hive QA commented on HIVE-23280:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001445/HIVE-23280.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17179 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21996/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21996/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21996/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001445 - PreCommit-HIVE-Build
> Trigger compaction with old aborted txns
>
>
> Key: HIVE-23280
> URL: https://issues.apache.org/jira/browse/HIVE-23280
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23280.01.patch, HIVE-23280.01.patch,
> HIVE-23280.01.patch, HIVE-23280.01.patch, HIVE-23280.02.patch,
> HIVE-23280.03.patch, HIVE-23280.04.patch
>
>
> When a txn is aborted and the compaction threshold for number of aborted txns
> is not reached then the aborted transaction can remain forever in the RDBMS
> database. This could result in several serious performance degradations:
> - getOpenTxns has to list this aborted txn forever
> - TXN_TO_WRITE_ID table is not cleaned
> We should add a threshold, so after a given time the compaction is started
> anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23280) Trigger compaction with old aborted txns
[
https://issues.apache.org/jira/browse/HIVE-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094821#comment-17094821
]
Hive QA commented on HIVE-23280:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
35s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 106 unchanged - 3 fixed = 108 total (was 109) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 1 fixed = 187 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| |
org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findPotentialCompactions(int,
long, long) passes a nonconstant String to an execute or addBatch method on an
SQL statement At CompactionTxnHandler.java:nonconstant String to an execute or
addBatch method on an SQL statement At CompactionTxnHandler.java:[line 97] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21996/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21996/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21996/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-common common
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21996/yetus.txt |
| Powered by | Apache Yetus
[jira] [Updated] (HIVE-23306) RESET command does not work if there is a config set by System.getProperty
[
https://issues.apache.org/jira/browse/HIVE-23306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23306:
--
Attachment: HIVE-23306.03.patch
> RESET command does not work if there is a config set by System.getProperty
> --
>
> Key: HIVE-23306
> URL: https://issues.apache.org/jira/browse/HIVE-23306
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23306.02.patch, HIVE-23306.03.patch,
> HIVE-23306.patch
>
>
> If a property set is restricted by {{hive.conf.restricted.list}} and the
> setting is set by system property, then {{RESET}} command fails with:
> {code}
> Caused by: java.lang.IllegalArgumentException: Cannot modify
> hive.query.max.length at runtime. It is in the list of parameters that can't
> be modified at runtime or is prefixed by a restricted variable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23310) Add .asf.yaml
[ https://issues.apache.org/jira/browse/HIVE-23310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23310: Status: Patch Available (was: Open) > Add .asf.yaml > - > > Key: HIVE-23310 > URL: https://issues.apache.org/jira/browse/HIVE-23310 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23310.01.patch > > > Earlier we had github pr-s automatically linked to jiras/etc - that feature > suddenly stopped working, because some improvements were made by Infra; so > that we can configure these stuff thru a .asf.yaml file > INFRA-20177 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23310) Add .asf.yaml
[ https://issues.apache.org/jira/browse/HIVE-23310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23310: Attachment: HIVE-23310.01.patch > Add .asf.yaml > - > > Key: HIVE-23310 > URL: https://issues.apache.org/jira/browse/HIVE-23310 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23310.01.patch > > > Earlier we had github pr-s automatically linked to jiras/etc - that feature > suddenly stopped working, because some improvements were made by Infra; so > that we can configure these stuff thru a .asf.yaml file > INFRA-20177 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23265) Duplicate rowsets are returned with Limit and Offset ste
[
https://issues.apache.org/jira/browse/HIVE-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094770#comment-17094770
]
Hive QA commented on HIVE-23265:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001427/HIVE-23265.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17179 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21995/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21995/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21995/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001427 - PreCommit-HIVE-Build
> Duplicate rowsets are returned with Limit and Offset ste
>
>
> Key: HIVE-23265
> URL: https://issues.apache.org/jira/browse/HIVE-23265
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Vectorization
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Chiran Ravani
>Assignee: Attila Magyar
>Priority: Critical
> Attachments: 00_0, HIVE-23265.1.patch, HIVE-23265.2.patch
>
>
> We have a query which produces duplicate results even when there is no
> duplicate records in underlying tables.
> Sample Query
> {code:java}
> select * from orderdatatest_ext order by col1 limit 1000,50
> {code}
> The problem appears when order by clause is used with col1 having non-unique
> rows. Apparently the duplicates are being produced during reducer phase of
> the query.
> set hive.vectorized.execution.reduce.enabled=false does not cause the problem.
> Data in table is as follows.
> {code:java}
> 1,1
> 1,2
> 1,3
> .
> .
> 1,1500
> {code}
> Results with hive.vectorized.execution.reduce.enabled=true
> {code:java}
> +-+-+
> | orderdatatest_ext.col1 | orderdatatest_ext.col2 |
> +-+-+
> | 1 | 1001|
> | 1 | 1002|
> | 1 | 1003|
> | 1 | 1004|
> | 1 | 1005|
> | 1 | 1006|
> | 1 | 1007|
> | 1 | 1008|
> | 1 | 1009|
> | 1 | 1010|
> | 1 | 1011|
> | 1 | 1012|
> | 1 | 1013|
> | 1 | 1014|
> | 1 | 1015|
> | 1 | 1016|
> | 1 | 1017|
> | 1 | 1018|
> | 1 | 1019|
> | 1 | 1020|
> | 1 | 1021|
> | 1 | 1022|
> | 1 | 1023|
> | 1 | 1024|
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> | 1 | 1 |
> |
[jira] [Commented] (HIVE-23265) Duplicate rowsets are returned with Limit and Offset ste
[
https://issues.apache.org/jira/browse/HIVE-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094737#comment-17094737
]
Hive QA commented on HIVE-23265:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
36s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21995/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21995/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Duplicate rowsets are returned with Limit and Offset ste
>
>
> Key: HIVE-23265
> URL: https://issues.apache.org/jira/browse/HIVE-23265
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Vectorization
>Affects Versions: 3.1.0, 3.1.2
>Reporter: Chiran Ravani
>Assignee: Attila Magyar
>Priority: Critical
> Attachments: 00_0, HIVE-23265.1.patch, HIVE-23265.2.patch
>
>
> We have a query which produces duplicate results even when there is no
> duplicate records in underlying tables.
> Sample Query
> {code:java}
> select * from orderdatatest_ext order by col1 limit 1000,50
> {code}
> The problem appears when order by clause is used with col1 having non-unique
> rows. Apparently the duplicates are being produced during reducer phase of
> the query.
> set hive.vectorized.execution.reduce.enabled=false does not cause the problem.
> Data in table is as follows.
> {code:java}
> 1,1
> 1,2
> 1,3
> .
> .
> 1,1500
> {code}
> Results with hive.vectorized.execution.reduce.enabled=true
> {code:java}
> +-+-+
> | orderdatatest_ext.col1 | orderdatatest_ext.col2 |
> +-+-+
> | 1 | 1001|
> | 1
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: HIVE-23293.2.patch > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: (was: HIVE-23293.2.patch) > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23309) Lazy Initialization of Hadoop Shims
[ https://issues.apache.org/jira/browse/HIVE-23309?focusedWorklogId=428294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428294 ] ASF GitHub Bot logged work on HIVE-23309: - Author: ASF GitHub Bot Created on: 28/Apr/20 17:56 Start Date: 28/Apr/20 17:56 Worklog Time Spent: 10m Work Description: aasha opened a new pull request #999: URL: https://github.com/apache/hive/pull/999 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 428294) Remaining Estimate: 0h Time Spent: 10m > Lazy Initialization of Hadoop Shims > --- > > Key: HIVE-23309 > URL: https://issues.apache.org/jira/browse/HIVE-23309 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23309.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Initialize hadoop-shims only if CM is enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23309) Lazy Initialization of Hadoop Shims
[ https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23309: --- Attachment: HIVE-23309.01.patch Status: Patch Available (was: In Progress) > Lazy Initialization of Hadoop Shims > --- > > Key: HIVE-23309 > URL: https://issues.apache.org/jira/browse/HIVE-23309 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23309.01.patch > > > Initialize hadoop-shims only if CM is enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23309) Lazy Initialization of Hadoop Shims
[ https://issues.apache.org/jira/browse/HIVE-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23309 started by Aasha Medhi. -- > Lazy Initialization of Hadoop Shims > --- > > Key: HIVE-23309 > URL: https://issues.apache.org/jira/browse/HIVE-23309 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > > Initialize hadoop-shims only if CM is enabled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: HIVE-23293.2.patch > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: (was: HIVE-23293.2.patch) > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: HIVE-23293.2.patch > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: (was: HIVE-23293.2.patch) > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23306) RESET command does not work if there is a config set by System.getProperty
[
https://issues.apache.org/jira/browse/HIVE-23306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094723#comment-17094723
]
Hive QA commented on HIVE-23306:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 16m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21994/dev-support/hive-personality.sh
|
| git revision | master / 83a4018 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql . itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21994/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> RESET command does not work if there is a config set by System.getProperty
> --
>
> Key: HIVE-23306
> URL: https://issues.apache.org/jira/browse/HIVE-23306
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23306.02.patch, HIVE-23306.patch
>
>
> If a property set is restricted by {{hive.conf.restricted.list}} and the
> setting is set by system property, then {{RESET}} command fails with:
> {code}
> Caused by: java.lang.IllegalArgumentException: Cannot modify
> hive.query.max.length at runtime. It is in the list of parameters that can't
> be modified at runtime or is prefixed by a restricted variable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23306) RESET command does not work if there is a config set by System.getProperty
[
https://issues.apache.org/jira/browse/HIVE-23306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094715#comment-17094715
]
Hive QA commented on HIVE-23306:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001417/HIVE-23306.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17178 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[partition_wise_fileformat15]
(batchId=80)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21994/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21994/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21994/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001417 - PreCommit-HIVE-Build
> RESET command does not work if there is a config set by System.getProperty
> --
>
> Key: HIVE-23306
> URL: https://issues.apache.org/jira/browse/HIVE-23306
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23306.02.patch, HIVE-23306.patch
>
>
> If a property set is restricted by {{hive.conf.restricted.list}} and the
> setting is set by system property, then {{RESET}} command fails with:
> {code}
> Caused by: java.lang.IllegalArgumentException: Cannot modify
> hive.query.max.length at runtime. It is in the list of parameters that can't
> be modified at runtime or is prefixed by a restricted variable
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: HIVE-23293.2.patch > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: (was: HIVE-23293.2.patch) > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23284) Remove dependency on mariadb-java-client
[ https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23284: - Attachment: HIVE-23284.03.patch Status: Patch Available (was: Open) > Remove dependency on mariadb-java-client > > > Key: HIVE-23284 > URL: https://issues.apache.org/jira/browse/HIVE-23284 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch, > HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch, > HIVE-23284.03.patch > > > It has GNU Lesser General Public License which is [Category > X|https://www.apache.org/legal/resolved.html#category-x]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23284) Remove dependency on mariadb-java-client
[ https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23284: - Status: Open (was: Patch Available) > Remove dependency on mariadb-java-client > > > Key: HIVE-23284 > URL: https://issues.apache.org/jira/browse/HIVE-23284 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch, > HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch > > > It has GNU Lesser General Public License which is [Category > X|https://www.apache.org/legal/resolved.html#category-x]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23293) Locks: Implement zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23293: -- Attachment: HIVE-23293.2.patch > Locks: Implement zero-wait readers > --- > > Key: HIVE-23293 > URL: https://issues.apache.org/jira/browse/HIVE-23293 > Project: Hive > Issue Type: Sub-task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23293.1.patch, HIVE-23293.2.patch > > > With a new lock type (EXCL_WRITE) for INSERT_OVERWRITE, SHARED_READ does not > have to wait for any lock - it can fails fast for a pending EXCLUSIVE, > because even if there is an EXCL_WRITE or SHARED_WRITE pending, there's no > semantic reason to wait for them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23284) Remove dependency on mariadb-java-client
[
https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094657#comment-17094657
]
Hive QA commented on HIVE-23284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001412/HIVE-23284.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17154 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21993/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21993/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21993/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001412 - PreCommit-HIVE-Build
> Remove dependency on mariadb-java-client
>
>
> Key: HIVE-23284
> URL: https://issues.apache.org/jira/browse/HIVE-23284
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch,
> HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch
>
>
> It has GNU Lesser General Public License which is [Category
> X|https://www.apache.org/legal/resolved.html#category-x].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094641#comment-17094641 ] Peter Vary commented on HIVE-23311: --- +1 pending tests > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23311.1.patch > > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23048) Use sequences for TXN_ID generation
[ https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094632#comment-17094632 ] Peter Vary commented on HIVE-23048: --- +1 > Use sequences for TXN_ID generation > --- > > Key: HIVE-23048 > URL: https://issues.apache.org/jira/browse/HIVE-23048 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23048.1.patch, HIVE-23048.10.patch, > HIVE-23048.11.patch, HIVE-23048.12.patch, HIVE-23048.13.patch, > HIVE-23048.14.patch, HIVE-23048.2.patch, HIVE-23048.3.patch, > HIVE-23048.4.patch, HIVE-23048.5.patch, HIVE-23048.6.patch, > HIVE-23048.7.patch, HIVE-23048.8.patch, HIVE-23048.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23284) Remove dependency on mariadb-java-client
[
https://issues.apache.org/jira/browse/HIVE-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094610#comment-17094610
]
Hive QA commented on HIVE-23284:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
11s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21993/dev-support/hive-personality.sh
|
| git revision | master / bf9de3c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore standalone-metastore/metastore-server U:
standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21993/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove dependency on mariadb-java-client
>
>
> Key: HIVE-23284
> URL: https://issues.apache.org/jira/browse/HIVE-23284
> Project: Hive
> Issue Type: Improvement
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23284.01.patch, HIVE-23284.01.patch,
> HIVE-23284.01.patch, HIVE-23284.02.patch, HIVE-23284.03.patch
>
>
> It has GNU Lesser General Public License which is [Category
> X|https://www.apache.org/legal/resolved.html#category-x].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23031?focusedWorklogId=428201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428201
]
ASF GitHub Bot logged work on HIVE-23031:
-
Author: ASF GitHub Bot
Created on: 28/Apr/20 15:28
Start Date: 28/Apr/20 15:28
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #988:
URL: https://github.com/apache/hive/pull/988#discussion_r416708106
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -96,19 +97,31 @@ private DataSketchesFunctions() {
return descriptors;
}
+ public SketchFunctionDescriptor getSketchFunction(String className, String
function) {
Review comment:
Yes it would be great - but I would also like to change the function to
be an enum as well - I would like to postpone this to a later patch. I've
opened: HIVE-23313
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 428201)
Time Spent: 2h 20m (was: 2h 10m)
> Add option to enable transparent rewrite of count(distinct) into sketch
> functions
> -
>
> Key: HIVE-23031
> URL: https://issues.apache.org/jira/browse/HIVE-23031
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23031.01.patch, HIVE-23031.02.patch,
> HIVE-23031.03.patch, HIVE-23031.03.patch, HIVE-23031.03.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23216) Add new api as replacement of get_partitions_by_expr to return PartitionSpec instead of Partitions
[ https://issues.apache.org/jira/browse/HIVE-23216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-23216: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for reviewing [~jcamachorodriguez] > Add new api as replacement of get_partitions_by_expr to return PartitionSpec > instead of Partitions > -- > > Key: HIVE-23216 > URL: https://issues.apache.org/jira/browse/HIVE-23216 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23216.1.patch, HIVE-23216.2.patch, > HIVE-23216.3.patch, HIVE-23216.4.patch, HIVE-23216.5.patch, > HIVE-23216.6.patch, HIVE-23216.7.patch, HIVE-23216.8.patch, HIVE-23216.9.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23313) Reorganize sketch functions registration logic
[ https://issues.apache.org/jira/browse/HIVE-23313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23313: --- > Reorganize sketch functions registration logic > -- > > Key: HIVE-23313 > URL: https://issues.apache.org/jira/browse/HIVE-23313 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > * change the procdeure based registration into an enum driven one > * also change the function names into an enum -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23031: Attachment: HIVE-23031.04.patch > Add option to enable transparent rewrite of count(distinct) into sketch > functions > - > > Key: HIVE-23031 > URL: https://issues.apache.org/jira/browse/HIVE-23031 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23031.01.patch, HIVE-23031.02.patch, > HIVE-23031.03.patch, HIVE-23031.03.patch, HIVE-23031.03.patch, > HIVE-23031.04.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23266) Remove QueryWrapper from ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-23266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23266:
--
Attachment: HIVE-23266.5.patch
> Remove QueryWrapper from ObjectStore
>
>
> Key: HIVE-23266
> URL: https://issues.apache.org/jira/browse/HIVE-23266
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23266.1.patch, HIVE-23266.2.patch,
> HIVE-23266.2.patch, HIVE-23266.3.patch, HIVE-23266.4.patch, HIVE-23266.5.patch
>
>
> There is currently a utility called {{QueryWrapper}} that makes a normal
> {{Query}} auto-closable. However, {{Query}} is now in fact already
> auto-closing, so there is no need for this class. In trying to remove it, I
> realized that this wrapper was being passed around in pretty convoluted ways
> and also it was sometimes being created in a {{try-with-resources}} block but
> then never actually used in any way.
> Remove the {{QueryWrapper}} from the class and simplify some of the DB
> interactions.
> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L178
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23048) Use sequences for TXN_ID generation
[
https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094582#comment-17094582
]
Hive QA commented on HIVE-23048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001440/HIVE-23048.14.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17167 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21992/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21992/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21992/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001440 - PreCommit-HIVE-Build
> Use sequences for TXN_ID generation
> ---
>
> Key: HIVE-23048
> URL: https://issues.apache.org/jira/browse/HIVE-23048
> Project: Hive
> Issue Type: Bug
>Reporter: Peter Vary
>Assignee: Peter Varga
>Priority: Major
> Attachments: HIVE-23048.1.patch, HIVE-23048.10.patch,
> HIVE-23048.11.patch, HIVE-23048.12.patch, HIVE-23048.13.patch,
> HIVE-23048.14.patch, HIVE-23048.2.patch, HIVE-23048.3.patch,
> HIVE-23048.4.patch, HIVE-23048.5.patch, HIVE-23048.6.patch,
> HIVE-23048.7.patch, HIVE-23048.8.patch, HIVE-23048.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23031?focusedWorklogId=428191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-428191
]
ASF GitHub Bot logged work on HIVE-23031:
-
Author: ASF GitHub Bot
Created on: 28/Apr/20 14:58
Start Date: 28/Apr/20 14:58
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #988:
URL: https://github.com/apache/hive/pull/988#discussion_r416683591
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java
##
@@ -128,14 +141,26 @@ private void buildCalciteFns() {
OperandTypes.family(),
unionFn);
+
unionSFD.setCalciteFunction(unionFn);
sketchSFD.setCalciteFunction(sketchFn);
+ if (estimateSFD != null) {
+SqlFunction estimateFn = new HiveSqlFunction(estimateSFD.name,
+SqlKind.OTHER_FUNCTION,
+ReturnTypes.explicit(SqlTypeName.DOUBLE),
Review comment:
I've approached this right now by identifying the return type from the
UDF class; we might need to replace that later - but it will work for now...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 428191)
Time Spent: 2h 10m (was: 2h)
> Add option to enable transparent rewrite of count(distinct) into sketch
> functions
> -
>
> Key: HIVE-23031
> URL: https://issues.apache.org/jira/browse/HIVE-23031
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23031.01.patch, HIVE-23031.02.patch,
> HIVE-23031.03.patch, HIVE-23031.03.patch, HIVE-23031.03.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23048) Use sequences for TXN_ID generation
[
https://issues.apache.org/jira/browse/HIVE-23048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094571#comment-17094571
]
Hive QA commented on HIVE-23048:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
31s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
12s{color} | {color:blue} standalone-metastore/metastore-server in master has
187 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} standalone-metastore/metastore-server: The patch
generated 0 new + 560 unchanged - 27 fixed = 560 total (was 587) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} ql: The patch generated 0 new + 838 unchanged - 3
fixed = 838 total (was 841) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
19s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 186 unchanged - 1 fixed = 187 total (was 187) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | Write to static field
org.apache.hadoop.hive.metastore.txn.TxnHandler.openTxnTimeOutMillis from
instance method
org.apache.hadoop.hive.metastore.txn.TxnHandler.setOpenTxnTimeOutMillis(long)
At TxnHandler.java:from instance method
org.apache.hadoop.hive.metastore.txn.TxnHandler.setOpenTxnTimeOutMillis(long)
At TxnHandler.java:[line 894] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21992/dev-support/hive-personality.sh
|
| git revision | master / 256a52a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21992/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-common
standalone-metastore/metastore-server ql U: . |
| Console output |
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23124:
--
Attachment: HIVE-23124.3.patch
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch, HIVE-23124.2.patch,
> HIVE-23124.3.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23108) Cleanup HiveBaseResultSet.java
[
https://issues.apache.org/jira/browse/HIVE-23108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23108:
--
Attachment: HIVE-23108.4.patch
> Cleanup HiveBaseResultSet.java
> --
>
> Key: HIVE-23108
> URL: https://issues.apache.org/jira/browse/HIVE-23108
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23108.1.patch, HIVE-23108.2.patch,
> HIVE-23108.3.patch, HIVE-23108.4.patch
>
>
> * Unify the code (there are several implementations of the same thing)
> * Added better error messages
> * In some cases, the code was throwing RuntimeExceptions which is against
> the JDBC Spec
> * Make findColumn a bit more streamlined
> * Remove non-javadoc comments
> * Add {{@Override}} annotations where appropriate
> * Formatting changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23311: --- Attachment: HIVE-23311.1.patch > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23311.1.patch > > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23311: --- Attachment: (was: HIVE-23311.1.patch) > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23192) "default" database locationUri should be external warehouse root.
[ https://issues.apache.org/jira/browse/HIVE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23192: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > "default" database locationUri should be external warehouse root. > - > > Key: HIVE-23192 > URL: https://issues.apache.org/jira/browse/HIVE-23192 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23192.patch, HIVE-23192.patch, HIVE-23192.patch > > > When creating the default database, the database locationUri should be set to > external warehouse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23260) Add support for unmodified_metadata capability
[ https://issues.apache.org/jira/browse/HIVE-23260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-23260: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Fix has been committed to master. Thanks for the review [~samuelan] > Add support for unmodified_metadata capability > -- > > Key: HIVE-23260 > URL: https://issues.apache.org/jira/browse/HIVE-23260 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23260.patch > > > Currently, the translator removes bucketing info for tables for clients that > do not possess the HIVEBUCKET2 capability. While this is desirable, some > clients that have write access to these tables can turn around overwrite the > metadata thus corrupting original bucketing info. > So adding support for a capability for client that are capable of > interpreting the original metadata would prevent such corruption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23192) "default" database locationUri should be external warehouse root.
[ https://issues.apache.org/jira/browse/HIVE-23192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094552#comment-17094552 ] Naveen Gangam commented on HIVE-23192: -- Fix has been pushed to master for 4.0.0. Thanks for the review [~samuelan] > "default" database locationUri should be external warehouse root. > - > > Key: HIVE-23192 > URL: https://issues.apache.org/jira/browse/HIVE-23192 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-23192.patch, HIVE-23192.patch, HIVE-23192.patch > > > When creating the default database, the database locationUri should be set to > external warehouse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094549#comment-17094549 ] Peter Vary commented on HIVE-23311: --- [~pvargacl]: Maybe use https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html#emptySet(), so we do not create non-necessary objects. > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23311.1.patch > > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23312) Update last event ID more efficiently during incremental run
[ https://issues.apache.org/jira/browse/HIVE-23312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA reassigned HIVE-23312: - > Update last event ID more efficiently during incremental run > > > Key: HIVE-23312 > URL: https://issues.apache.org/jira/browse/HIVE-23312 > Project: Hive > Issue Type: Improvement >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > > Should be done for a batch of events and/or in asynchronous way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23117) Review of HiveStatement Class
[
https://issues.apache.org/jira/browse/HIVE-23117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094523#comment-17094523
]
Hive QA commented on HIVE-23117:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001363/HIVE-23117.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21991/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21991/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21991/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-04-28 13:54:50.002
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-21991/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-04-28 13:54:50.005
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 256a52a HIVE-23201: Improve logging in locking (Marton Bod via
Denys Kuzmenko)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 256a52a HIVE-23201: Improve logging in locking (Marton Bod via
Denys Kuzmenko)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-04-28 13:54:50.823
+ rm -rf ../yetus_PreCommit-HIVE-Build-21991
+ mkdir ../yetus_PreCommit-HIVE-Build-21991
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-21991
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-21991/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
fatal: unrecognized input
Trying to apply the patch with -p1
fatal: unrecognized input
Trying to apply the patch with -p2
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-21991
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001363 - PreCommit-HIVE-Build
> Review of HiveStatement Class
> -
>
> Key: HIVE-23117
> URL: https://issues.apache.org/jira/browse/HIVE-23117
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23117.1.patch, HIVE-23117.2.patch,
> HIVE-23117.2.patch
>
>
> * Remove unused instance variable(s)
> * Remove non-JavaDoc comments
> * Make inPlaceUpdateStream Optional (and remove NO-OP class) (inconsistent
> behavior with 'null' values)
> * {{getQueryTimeout()}} returns incorrect value
> * Unify and improve Exception messages
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23261) Check whether encryption is enabled in the cluster before moving files
[
https://issues.apache.org/jira/browse/HIVE-23261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094520#comment-17094520
]
Hive QA commented on HIVE-23261:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13001366/HIVE-23261.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17158 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestScheduledReplicationScenarios.testAcidTablesReplLoadBootstrapIncr
(batchId=205)
org.apache.hive.jdbc.TestJdbcWithMiniLlapVectorArrowBatch.testLlapInputFormatEndToEndWithMultipleBatches
(batchId=215)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21990/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21990/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21990/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13001366 - PreCommit-HIVE-Build
> Check whether encryption is enabled in the cluster before moving files
> --
>
> Key: HIVE-23261
> URL: https://issues.apache.org/jira/browse/HIVE-23261
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Minor
> Attachments: HIVE-23261.1.patch, HIVE-23261.2.patch
>
>
> Similar to HIVE-23212, there is an unwanted check of encryption paths during
> file move operation.
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L4546]
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23311: --- Status: Patch Available (was: In Progress) > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23311.1.patch > > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga updated HIVE-23311: --- Attachment: HIVE-23311.1.patch > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > Attachments: HIVE-23311.1.patch > > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23311 started by Peter Varga. -- > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23311) Fix ValidTxnManager regression
[ https://issues.apache.org/jira/browse/HIVE-23311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Varga reassigned HIVE-23311: -- > Fix ValidTxnManager regression > -- > > Key: HIVE-23311 > URL: https://issues.apache.org/jira/browse/HIVE-23311 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Major > > During query execution if there are only shared lock tables, the txnList in > the driverContext should always be considered valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22769 started by Panagiotis Garefalakis.
-
> Incorrect query results and query failure during split generation for
> compressed text files
> ---
>
> Key: HIVE-22769
> URL: https://issues.apache.org/jira/browse/HIVE-22769
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Chiran Ravani
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22769.01.patch, testcase1.csv.bz2, testcase2.csv.bz2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive Query produces incorrect results when data is in text format and
> compressed and for certain data the query fails during split generation.
> This behavior is seen when skip.header.line.count and skip.footer.line.count
> are set for table.
> Case 1: Select count/aggregate query produces Incorrect row counts/displays
> all rows (when hive.fetch.task.conversion=none)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
> 3. Run count(*) on table.
> {code}
> > select * from testcase1;
> INFO : Compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string,
> comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.07 seconds
> INFO : Executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Completed executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.007 seconds
> INFO : OK
> +---+-+
> | testcase1.id | testcase1.name |
> +---+-+
> | 2 | 2019-12-31 |
> +---+-+
> 1 row selected (0.111 seconds)
> > select count(*) from testcase1
> INFO : Compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:bigint, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 0.073 seconds
> INFO : Executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Session is already open
> INFO : Dag name: select count(*) from testcase1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1579811438512_0046)
> .
> .
> .
> INFO : Completed executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 4.228 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 3|
> +--+
> 1 row selected (4.335 seconds)
> {code}
> Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
> java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
> The issue is only seen when there is a space in a field (eg:- "3,2019-12-31
> 01" second column has a space)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
> 3. Run count(*) on table.
> {code}
> 0: > select * from testcase2;
> INFO : Compiling
> command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
> select * from testcase2
> INFO : Semantic Analysis Completed (retrial
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-22769:
--
Status: Patch Available (was: In Progress)
> Incorrect query results and query failure during split generation for
> compressed text files
> ---
>
> Key: HIVE-22769
> URL: https://issues.apache.org/jira/browse/HIVE-22769
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 3.1.0, 3.0.0
>Reporter: Chiran Ravani
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22769.01.patch, testcase1.csv.bz2, testcase2.csv.bz2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive Query produces incorrect results when data is in text format and
> compressed and for certain data the query fails during split generation.
> This behavior is seen when skip.header.line.count and skip.footer.line.count
> are set for table.
> Case 1: Select count/aggregate query produces Incorrect row counts/displays
> all rows (when hive.fetch.task.conversion=none)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
> 3. Run count(*) on table.
> {code}
> > select * from testcase1;
> INFO : Compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string,
> comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.07 seconds
> INFO : Executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Completed executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.007 seconds
> INFO : OK
> +---+-+
> | testcase1.id | testcase1.name |
> +---+-+
> | 2 | 2019-12-31 |
> +---+-+
> 1 row selected (0.111 seconds)
> > select count(*) from testcase1
> INFO : Compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:bigint, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 0.073 seconds
> INFO : Executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Session is already open
> INFO : Dag name: select count(*) from testcase1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1579811438512_0046)
> .
> .
> .
> INFO : Completed executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 4.228 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 3|
> +--+
> 1 row selected (4.335 seconds)
> {code}
> Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
> java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
> The issue is only seen when there is a space in a field (eg:- "3,2019-12-31
> 01" second column has a space)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
> 3. Run count(*) on table.
> {code}
> 0: > select * from testcase2;
> INFO : Compiling
> command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
> select * from testcase2
> INFO : Semantic
[jira] [Updated] (HIVE-22769) Incorrect query results and query failure during split generation for compressed text files
[
https://issues.apache.org/jira/browse/HIVE-22769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-22769:
--
Affects Version/s: 4.0.0
> Incorrect query results and query failure during split generation for
> compressed text files
> ---
>
> Key: HIVE-22769
> URL: https://issues.apache.org/jira/browse/HIVE-22769
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 3.0.0, 3.1.0, 4.0.0
>Reporter: Chiran Ravani
>Assignee: Panagiotis Garefalakis
>Priority: Critical
> Attachments: HIVE-22769.01.patch, testcase1.csv.bz2, testcase2.csv.bz2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hive Query produces incorrect results when data is in text format and
> compressed and for certain data the query fails during split generation.
> This behavior is seen when skip.header.line.count and skip.footer.line.count
> are set for table.
> Case 1: Select count/aggregate query produces Incorrect row counts/displays
> all rows (when hive.fetch.task.conversion=none)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase1`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase1'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase1.csv.bz2 file to /user/hive/testcase1
> 3. Run count(*) on table.
> {code}
> > select * from testcase1;
> INFO : Compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:testcase1.id, type:string,
> comment:null), FieldSchema(name:testcase1.name, type:string, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.07 seconds
> INFO : Executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f):
> select * from testcase1
> INFO : Completed executing
> command(queryId=hive_20200124053854_454b03c1-d4c5-4dba-a2c2-91c09f4b670f);
> Time taken: 0.007 seconds
> INFO : OK
> +---+-+
> | testcase1.id | testcase1.name |
> +---+-+
> | 2 | 2019-12-31 |
> +---+-+
> 1 row selected (0.111 seconds)
> > select count(*) from testcase1
> INFO : Compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0,
> type:bigint, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 0.073 seconds
> INFO : Executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7):
> select count(*) from testcase1
> INFO : Query ID = hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Total jobs = 1
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-1:MAPRED] in serial mode
> INFO : Subscribed to counters: [] for queryId:
> hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7
> INFO : Session is already open
> INFO : Dag name: select count(*) from testcase1 (Stage-1)
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1579811438512_0046)
> .
> .
> .
> INFO : Completed executing
> command(queryId=hive_20200124053645_a7d699b7-c7e1-4d92-8d99-666b0a010ba7);
> Time taken: 4.228 seconds
> INFO : OK
> +--+
> | _c0 |
> +--+
> | 3|
> +--+
> 1 row selected (4.335 seconds)
> {code}
> Case 2: Select count/aggregate query fails with java.lang.ClassCastException:
> java.io.PushbackInputStream cannot be cast to org.apache.hadoop.fs.Seekable
> The issue is only seen when there is a space in a field (eg:- "3,2019-12-31
> 01" second column has a space)
> Steps to reproduce:
> 1. Create table as below
> {code}
> CREATE EXTERNAL TABLE `testcase2`(id int, name string) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.OpenCSVSerde' LOCATION '/user/hive/testcase2'
> TBLPROPERTIES ("skip.header.line.count"="1", "skip.footer.line.count"="1");
> {code}
> 2. Upload attached testcase2.csv.bz2 file to /user/hive/testcase2
> 3. Run count(*) on table.
> {code}
> 0: > select * from testcase2;
> INFO : Compiling
> command(queryId=hive_20200124053159_5d8ce56a-183d-4359-a147-bd470d82e134):
> select * from testcase2
> INFO : Semantic Analysis
