[jira] [Work logged] (HIVE-26131) Incorrect OutputFormat when describing jdbc connector table
[
https://issues.apache.org/jira/browse/HIVE-26131?focusedWorklogId=757733&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757733
]
ASF GitHub Bot logged work on HIVE-26131:
-
Author: ASF GitHub Bot
Created on: 18/Apr/22 02:17
Start Date: 18/Apr/22 02:17
Worklog Time Spent: 10m
Work Description: zhangbutao commented on PR #3200:
URL: https://github.com/apache/hive/pull/3200#issuecomment-1101028919
Gentle ping :)
Issue Time Tracking
---
Worklog Id: (was: 757733)
Time Spent: 0.5h (was: 20m)
> Incorrect OutputFormat when describing jdbc connector table
>
>
> Key: HIVE-26131
> URL: https://issues.apache.org/jira/browse/HIVE-26131
> Project: Hive
> Issue Type: Bug
> Components: JDBC storage handler
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Attachments: image-2022-04-12-13-07-09-647.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Step to repro:
> {code:java}
> CREATE CONNECTOR mysql_qtest
> TYPE 'mysql'
> URL 'jdbc:mysql://localhost:3306/testdb'
> WITH DCPROPERTIES (
> "hive.sql.dbcp.username"="root",
> "hive.sql.dbcp.password"="");
> CREATE REMOTE DATABASE db_mysql USING mysql_qtest with
> DBPROPERTIES("connector.remoteDbName"="testdb");
> describe formatted db_mysql.test;{code}
> You can see incorrect OuptputFormat info:
> !image-2022-04-12-13-07-09-647.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26130) Incorrect matching of external table when validating NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-26130?focusedWorklogId=757734&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757734
]
ASF GitHub Bot logged work on HIVE-26130:
-
Author: ASF GitHub Bot
Created on: 18/Apr/22 02:17
Start Date: 18/Apr/22 02:17
Worklog Time Spent: 10m
Work Description: zhangbutao commented on PR #3199:
URL: https://github.com/apache/hive/pull/3199#issuecomment-1101028959
Gentle ping :)
Issue Time Tracking
---
Worklog Id: (was: 757734)
Time Spent: 40m (was: 0.5h)
> Incorrect matching of external table when validating NOT NULL constraints
> -
>
> Key: HIVE-26130
> URL: https://issues.apache.org/jira/browse/HIVE-26130
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> _AbstractAlterTablePropertiesAnalyzer.validate_ uses incorrect external table
> judgment statement:
> {code:java}
> else if (entry.getKey().equals("external") && entry.getValue().equals("true")
> {code}
> In current hive code, we use hive tblproperties('EXTERNAL'='true' or
> 'EXTERNAL'='TRUE) to validate external table.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26114) jdbc connection hivesrerver2 using dfs command with prefix space will cause exception
[
https://issues.apache.org/jira/browse/HIVE-26114?focusedWorklogId=757726&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757726
]
ASF GitHub Bot logged work on HIVE-26114:
-
Author: ASF GitHub Bot
Created on: 18/Apr/22 00:38
Start Date: 18/Apr/22 00:38
Worklog Time Spent: 10m
Work Description: ming95 commented on PR #3176:
URL: https://github.com/apache/hive/pull/3176#issuecomment-1100983724
@kgyrtkirk
Test unstable, please rebuild CI.
@pvary @nrg4878
I add a UT, please review again.
Thanks ~
Issue Time Tracking
---
Worklog Id: (was: 757726)
Time Spent: 50m (was: 40m)
> jdbc connection hivesrerver2 using dfs command with prefix space will cause
> exception
> -
>
> Key: HIVE-26114
> URL: https://issues.apache.org/jira/browse/HIVE-26114
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.3.8, 3.1.2
>Reporter: shezm
>Assignee: shezm
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.2.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {code:java}
> Connection con =
> DriverManager.getConnection("jdbc:hive2://10.214.35.115:1/");
> Statement stmt = con.createStatement();
> // dfs command with prefix space or "\n"
> ResultSet res = stmt.executeQuery(" dfs -ls /");
> //ResultSet res = stmt.executeQuery("\ndfs -ls /"); {code}
> it will cause exception
> {code:java}
> Exception in thread "main" org.apache.hive.service.cli.HiveSQLException:
> Error while processing statement: null
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:217)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:244)
> at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:375)
> at com.ne.gdc.whitemane.shezm.TestJdbc.main(TestJdbc.java:30)
> Caused by: org.apache.hive.service.cli.HiveSQLException: Error while
> processing statement: null
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
> at
> org.apache.hive.service.cli.operation.HiveCommandOperation.runInternal(HiveCommandOperation.java:118)
> at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
> at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> at com.sun.proxy.$Proxy43.executeStatementAsync(Unknown Source)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:605)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> But when I execute sql with prefix "\n" it works fine
> {code:java}
> ResultSet res = stmt.executeQuery("\n select 1"); {code}
[jira] [Work logged] (HIVE-25912) Drop external table at root of s3 bucket throws NPE
[
https://issues.apache.org/jira/browse/HIVE-25912?focusedWorklogId=757725&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757725
]
ASF GitHub Bot logged work on HIVE-25912:
-
Author: ASF GitHub Bot
Created on: 18/Apr/22 00:21
Start Date: 18/Apr/22 00:21
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #2987:
HIVE-25912: Drop external table throw NPE if the location set to ROOT path
URL: https://github.com/apache/hive/pull/2987
Issue Time Tracking
---
Worklog Id: (was: 757725)
Remaining Estimate: 78h 40m (was: 78h 50m)
Time Spent: 17h 20m (was: 17h 10m)
> Drop external table at root of s3 bucket throws NPE
> ---
>
> Key: HIVE-25912
> URL: https://issues.apache.org/jira/browse/HIVE-25912
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.1.2, 4.0.0
> Environment: Hive version: 3.1.2
>Reporter: Fachuan Bai
>Assignee: Fachuan Bai
>Priority: Major
> Labels: metastore, pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Attachments: hive bugs.png, hive-bug-01.png
>
> Original Estimate: 96h
> Time Spent: 17h 20m
> Remaining Estimate: 78h 40m
>
> *new update:*
> I test the master branch, have the same problem.
> --
> ENV:
> Hive 3.1.2
> HDFS:3.3.1
> enable OpenLDAP and Ranger .
>
> I create the external hive table using this command:
>
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 'hdfs://emr-master-1:8020/';
> {code}
>
> The table was created successfully, but when I drop the table throw the NPE:
>
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> MetaException(message:java.lang.NullPointerException)
> (state=08S01,code=1){code}
>
> The same bug can reproduction on the other object storage file system, such
> as S3 or TOS:
> {code:java}
> CREATE EXTERNAL TABLE `fcbai`(
> `inv_item_sk` int,
> `inv_warehouse_sk` int,
> `inv_quantity_on_hand` int)
> PARTITIONED BY (
> `inv_date_sk` int) STORED AS ORC
> LOCATION
> 's3a://bucketname/'; // 'tos://bucketname/'{code}
>
> I see the source code found:
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java
> {code:java}
> // check if sticky bit is set on the parent dir
> FileStatus parStatus = fs.getFileStatus(path.getParent());
> if (!shims.hasStickyBit(parStatus.getPermission())) {
> // no sticky bit, so write permission on parent dir is sufficient
> // no further checks needed
> return;
> }{code}
>
> because I set the table location to HDFS root path
> (hdfs://emr-master-1:8020/), so the path.getParent() function will be return
> null cause the NPE.
> I think have four solutions to fix the bug:
> # modify the create table function, if the location is root dir return
> create table fail.
> # modify the FileUtils.checkDeletePermission function, check the
> path.getParent(), if it is null, the function return, drop successfully.
> # modify the RangerHiveAuthorizer.checkPrivileges function of the hive
> ranger plugin(in ranger rep), if the location is root dir return create table
> fail.
> # modify the HDFS Path object, if the URI is root dir, path.getParent()
> return not null.
> I recommend the first or second method, any suggestion for me? thx.
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25929) Let secret config properties to be propagated to Tez
[ https://issues.apache.org/jira/browse/HIVE-25929?focusedWorklogId=757724&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757724 ] ASF GitHub Bot logged work on HIVE-25929: - Author: ASF GitHub Bot Created on: 18/Apr/22 00:21 Start Date: 18/Apr/22 00:21 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on PR #3019: URL: https://github.com/apache/hive/pull/3019#issuecomment-1100978183 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. Issue Time Tracking --- Worklog Id: (was: 757724) Time Spent: 1.5h (was: 1h 20m) > Let secret config properties to be propagated to Tez > > > Key: HIVE-25929 > URL: https://issues.apache.org/jira/browse/HIVE-25929 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > History in chronological order: > HIVE-10508: removed some passwords from config that's propagated to execution > engines > HIVE-9013: introduced hive.conf.hidden.list, which is used instead of the > hardcoded list in HIVE-10508 > the problem with HIVE-9013 is it's about to introduce a common method for > removing sensitive data from Configuration, which absolutely makes sense in > most of the cases (set command showing sensitive data), but can cause issues > e.g. while using non-secure cloud connectors on a cluster, where instead of > the hadoop credential provider API (which is considered the secure and proper > way), passwords/secrets appear in the Configuration object (like: > "fs.azure.account.oauth2.client.secret") > 2 possible solutions: > 1. introduce a new property like: "hive.conf.hidden.list.exec.engines" -> > which defaults to "hive.conf.hidden.list" (configurable, but maybe just more > confusing to users, having a new config property which should be understood > and maintained on a cluster) > 2. simply revert DAGUtils to use to old stripHivePasswordDetails introduced > by HIVE-10508 (convenient, less confusing for users, but cannot be configured) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26147) OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid file
[
https://issues.apache.org/jira/browse/HIVE-26147?focusedWorklogId=757702&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757702
]
ASF GitHub Bot logged work on HIVE-26147:
-
Author: ASF GitHub Bot
Created on: 17/Apr/22 18:22
Start Date: 17/Apr/22 18:22
Worklog Time Spent: 10m
Work Description: asolimando opened a new pull request, #3219:
URL: https://github.com/apache/hive/pull/3219
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Issue Time Tracking
---
Worklog Id: (was: 757702)
Remaining Estimate: 0h
Time Spent: 10m
> OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid
> file
> --
>
> Key: HIVE-26147
> URL: https://issues.apache.org/jira/browse/HIVE-26147
> Project: Hive
> Issue Type: Bug
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When _hive.acid.key.index_ is missing for an acid ORC file
> _OrcRawRecordMerger_ throws as follows:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.discoverKeyBounds(OrcRawRecordMerger.java:795)
> ~[hive-exec-4.0.0-alpha-2-SNAPS
> HOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1053)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.
> 0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:2096)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-a
> lpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1991)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4
> .0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:769)
> ~[hive-exec-4.0.0-alpha
> -2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-
> alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-
> SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:719)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNA
> PSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:671)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:233)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:489)
> ~[hive-service-4.0.0-alpha-2-SNAPSHOT.jar:
> 4.0.0-alpha-2-SNAPSHOT]
> ... 24 more
> {noformat}
> For this situation to happen, the ORC file must have more than one stripe,
> and the offset of the element to seek should either locate it beyond the
> first stripe (but before the last one), or in the first one if not the last
> one, as the code shows:
> {code:java}
> if (firstStripe != 0) {
> minKey = keyIndex[firstStripe - 1];
> }
> if (!isTail) {
> maxKey = keyIndex[firstStripe + stripeCount - 1];
> }
> {code}
> However, in the context of the detection of the original issue, the NPE was
> triggered even by a simple "select *" over a table with ORC files missing the
> _hive.acid.key.index_ metadata information, but it was never failing for ORC
> files with a single stripe. The file was generated after a major compaction
> of acid and non-acid data.
> If the "select *" is not triggering the NPE, either pick the values of the
> row obtained with "select * from $table limit 1", or try to select based on
> different values trying to get into the sought situation with a filter like
> this:
> {code:sql}
> select * from $table where c = $value
> {code}
> _OrcRawRecordMerger_ should simply leave a
[jira] [Updated] (HIVE-26147) OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid file
[
https://issues.apache.org/jira/browse/HIVE-26147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26147:
--
Labels: pull-request-available (was: )
> OrcRawRecordMerger throws NPE when hive.acid.key.index is missing for an acid
> file
> --
>
> Key: HIVE-26147
> URL: https://issues.apache.org/jira/browse/HIVE-26147
> Project: Hive
> Issue Type: Bug
> Components: ORC, Transactions
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When _hive.acid.key.index_ is missing for an acid ORC file
> _OrcRawRecordMerger_ throws as follows:
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.discoverKeyBounds(OrcRawRecordMerger.java:795)
> ~[hive-exec-4.0.0-alpha-2-SNAPS
> HOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:1053)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.
> 0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:2096)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-a
> lpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1991)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4
> .0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:769)
> ~[hive-exec-4.0.0-alpha
> -2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:335)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-
> alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:529)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-
> SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:150)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.Driver.getFetchingTableResults(Driver.java:719)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNA
> PSHOT]
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:671)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:233)
> ~[hive-exec-4.0.0-alpha-2-SNAPSHOT.jar:4.0.0-alpha
> -2-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:489)
> ~[hive-service-4.0.0-alpha-2-SNAPSHOT.jar:
> 4.0.0-alpha-2-SNAPSHOT]
> ... 24 more
> {noformat}
> For this situation to happen, the ORC file must have more than one stripe,
> and the offset of the element to seek should either locate it beyond the
> first stripe (but before the last one), or in the first one if not the last
> one, as the code shows:
> {code:java}
> if (firstStripe != 0) {
> minKey = keyIndex[firstStripe - 1];
> }
> if (!isTail) {
> maxKey = keyIndex[firstStripe + stripeCount - 1];
> }
> {code}
> However, in the context of the detection of the original issue, the NPE was
> triggered even by a simple "select *" over a table with ORC files missing the
> _hive.acid.key.index_ metadata information, but it was never failing for ORC
> files with a single stripe. The file was generated after a major compaction
> of acid and non-acid data.
> If the "select *" is not triggering the NPE, either pick the values of the
> row obtained with "select * from $table limit 1", or try to select based on
> different values trying to get into the sought situation with a filter like
> this:
> {code:sql}
> select * from $table where c = $value
> {code}
> _OrcRawRecordMerger_ should simply leave as "null" the min and max keys when
> the _hive.acid.key.index_ metadata is missing.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26148) Keep MetaStoreFilterHook interface compatibility after introducing catalogs
[
https://issues.apache.org/jira/browse/HIVE-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26148:
--
Labels: pull-request-available (was: )
> Keep MetaStoreFilterHook interface compatibility after introducing catalogs
> ---
>
> Key: HIVE-26148
> URL: https://issues.apache.org/jira/browse/HIVE-26148
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive 3.0 introduce catalog concept, when we upgrade hive dependency version
> from 2.3 to 3.x, we found some interfaces of *MetaStoreFilterHook* are not
> compatible:
> {code:bash}
> git show ba8a99e115 --
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> {code}
> {code:bash}
> ---
> a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> +++
> b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
>/**
> * Filter given list of tables
> - * @param dbName
> - * @param tableList
> + * @param catName catalog name
> + * @param dbName database name
> + * @param tableList list of table returned by the metastore
> * @return List of filtered table names
> */
> - public List filterTableNames(String dbName, List
> tableList) throws MetaException;
> + List filterTableNames(String catName, String dbName, List
> tableList)
> + throws MetaException;
> {code}
> We can remain the previous interfaces and use the default catalog to
> implement.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26148) Keep MetaStoreFilterHook interface compatibility after introducing catalogs
[
https://issues.apache.org/jira/browse/HIVE-26148?focusedWorklogId=757690&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-757690
]
ASF GitHub Bot logged work on HIVE-26148:
-
Author: ASF GitHub Bot
Created on: 17/Apr/22 16:47
Start Date: 17/Apr/22 16:47
Worklog Time Spent: 10m
Work Description: wecharyu opened a new pull request, #3218:
URL: https://github.com/apache/hive/pull/3218
### What changes were proposed in this pull request?
Add some interfaces of `MetaStoreFilterHook` removed while introducing
catalogs to Hive 3.
### Why are the changes needed?
To keep interfaces compacibility.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No additional tests.
Issue Time Tracking
---
Worklog Id: (was: 757690)
Remaining Estimate: 0h
Time Spent: 10m
> Keep MetaStoreFilterHook interface compatibility after introducing catalogs
> ---
>
> Key: HIVE-26148
> URL: https://issues.apache.org/jira/browse/HIVE-26148
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hive 3.0 introduce catalog concept, when we upgrade hive dependency version
> from 2.3 to 3.x, we found some interfaces of *MetaStoreFilterHook* are not
> compatible:
> {code:bash}
> git show ba8a99e115 --
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> {code}
> {code:bash}
> ---
> a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> +++
> b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
>/**
> * Filter given list of tables
> - * @param dbName
> - * @param tableList
> + * @param catName catalog name
> + * @param dbName database name
> + * @param tableList list of table returned by the metastore
> * @return List of filtered table names
> */
> - public List filterTableNames(String dbName, List
> tableList) throws MetaException;
> + List filterTableNames(String catName, String dbName, List
> tableList)
> + throws MetaException;
> {code}
> We can remain the previous interfaces and use the default catalog to
> implement.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-26148) Keep MetaStoreFilterHook interface compatibility after introducing catalogs
[
https://issues.apache.org/jira/browse/HIVE-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wechar reassigned HIVE-26148:
-
> Keep MetaStoreFilterHook interface compatibility after introducing catalogs
> ---
>
> Key: HIVE-26148
> URL: https://issues.apache.org/jira/browse/HIVE-26148
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Wechar
>Assignee: Wechar
>Priority: Minor
> Fix For: 4.0.0-alpha-1
>
>
> Hive 3.0 introduce catalog concept, when we upgrade hive dependency version
> from 2.3 to 3.x, we found some interfaces of *MetaStoreFilterHook* are not
> compatible:
> {code:bash}
> git show ba8a99e115 --
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> {code}
> {code:bash}
> ---
> a/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
> +++
> b/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreFilterHook.java
>/**
> * Filter given list of tables
> - * @param dbName
> - * @param tableList
> + * @param catName catalog name
> + * @param dbName database name
> + * @param tableList list of table returned by the metastore
> * @return List of filtered table names
> */
> - public List filterTableNames(String dbName, List
> tableList) throws MetaException;
> + List filterTableNames(String catName, String dbName, List
> tableList)
> + throws MetaException;
> {code}
> We can remain the previous interfaces and use the default catalog to
> implement.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-14870) OracleStore: RawStore implementation optimized for Oracle
[ https://issues.apache.org/jira/browse/HIVE-14870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17523298#comment-17523298 ] Sujith Chacko commented on HIVE-14870: -- cc [~sershe] [~cdrome] - Do we have any update on this JIRA? Looks an interesting improvement. > OracleStore: RawStore implementation optimized for Oracle > - > > Key: HIVE-14870 > URL: https://issues.apache.org/jira/browse/HIVE-14870 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Chris Drome >Assignee: Chris Drome >Priority: Major > Attachments: HIVE-14870.patch, OracleStoreDesignProposal.pdf, > schema-oraclestore.sql > > > The attached document is a proposal for a RawStore implementation which is > optimized for Oracle and replaces DataNucleus. The document outlines schema > changes, OracleStore implementation details, and performance tests against > ObjectStore, ObjectStore+DirectSQL, and OracleStore. -- This message was sent by Atlassian Jira (v8.20.1#820001)
