[jira] [Updated] (HIVE-27835) java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for DruidStorageHandler

2023-10-31 Thread Basapuram Kumar (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Basapuram Kumar updated HIVE-27835:
---
Description: 
Hello Team,

I am trying to use Druid storage handler to create a table.

Steps to repro

Create table

 
{code:java}
CREATE TABLE emp_zlib_3110_2038 (
`__time` TIMESTAMP,
 id int,
 name string,
 age int,
 gender string)
 STORED AS ORC
 TBLPROPERTIES ('transactional'='true',"orc.compress"="ZLIB");
 {code}
insert data
{code:java}
insert into emp_zlib_3110_2038 values(CURRENT_TIMESTAMP(),10,'basa',30,'M'); 
{code}
Create a DruidStorageHandler table
{code:java}
CREATE TABLE druid_hive_table_3110_2038_02
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
TBLPROPERTIES ("druid.datasource" = "druid_hive_table_3110_2038_02")
AS
select `__time`, id,name,age,gender from emp_zlib_3110_2038 ; {code}
Error:
{code:java}
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1698753624381_0021)
--
        
VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  
KILLED
--
Map 1 .. container     SUCCEEDED      1          1        0        0    
   0       0
Reducer 2 ...    container       RUNNING      2          1        0        1    
   4       0
--
VERTICES: 01/02  [=>>-] 66%   ELAPSED TIME: 37.91 s
--

ERROR : Status: FailedERROR : Vertex failed, vertexName=Reducer 2, 
vertexId=vertex_1698753624381_0021_3_01, diagnostics=[Task failed, 
taskId=task_1698753624381_0021_3_01_00, diagnostics=[TaskAttempt 0 failed, 
info=[Error: Error while running task ( failure ) : 
attempt_1698753624381_0021_3_01_00_0:java.lang.RuntimeException: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row 
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:348)
 
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276)   
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
 
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
   
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
   
at java.security.AccessController.doPrivileged(Native Method)   
at javax.security.auth.Subject.doAs(Subject.java:422)   
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
 
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)

at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)

at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)  
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
  
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
   
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
   
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 
at java.lang.Thread.run(Thread.java:750)Caused by: java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
processing row
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:311)

at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
 
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
 
... 16 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive 
Runtime Error while processing row 
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:385)

at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:301)

... 18 more

Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory  
at 
org.apache.hive.druid.com.google.common.base.Throwables.propagate(Throwables.java:241)
   
at 
org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:209)
   
at

[jira] [Updated] (HIVE-27835) java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for DruidStorageHandler

2023-10-31 Thread Basapuram Kumar (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Basapuram Kumar updated HIVE-27835:
---
Summary: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for 
DruidStorageHandler  (was: java.lang.NoClassDefFoundError: 
net/jpountz/lz4/LZ4Factory for DruidHandler)

> java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for 
> DruidStorageHandler
> --
>
> Key: HIVE-27835
> URL: https://issues.apache.org/jira/browse/HIVE-27835
> Project: Hive
>  Issue Type: Task
>  Components: Hive
>Affects Versions: 4.0.0-alpha-1
> Environment: hive-4-alpha-1
> Druid-27.0.0
>Reporter: Basapuram Kumar
>Priority: Major
>
> Hello Team,
> I am trying to use Druid storage handler to create a table.
> Steps to repro
> Create table
>  
> {code:java}
> CREATE TABLE emp_zlib_3110_2038 (
> `__time` TIMESTAMP,
>  id int,
>  name string,
>  age int,
>  gender string)
>  STORED AS ORC
>  TBLPROPERTIES ('transactional'='true',"orc.compress"="ZLIB");
>  {code}
> insert data
> {code:java}
> insert into emp_zlib_3110_2038 values(CURRENT_TIMESTAMP(),10,'basa',30,'M'); 
> {code}
> Create a DruidStorageHandler table
> {code:java}
> CREATE TABLE druid_hive_table_3110_2038_02
> STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
> TBLPROPERTIES ("druid.datasource" = "druid_hive_table_3110_2038_02")
> AS
> select `__time`, id,name,age,gender from emp_zlib_3110_2038 ; {code}
> Error:
> {code:java}
> INFO  : Status: Running (Executing on YARN cluster with App id 
> application_1698753624381_0021)
> --
>         VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  
> FAILED  
> KILLED--Map
>  1 .. container     SUCCEEDED      1          1        0        0     
>   0       0Reducer 2 ...    container       RUNNING      2          1        
> 0        1       4       
> 0--VERTICES:
>  01/02  [=>>-] 66%   ELAPSED TIME: 37.91 
> s--
> ERROR : Status: FailedERROR : Vertex failed, vertexName=Reducer 2, 
> vertexId=vertex_1698753624381_0021_3_01, diagnostics=[Task failed, 
> taskId=task_1698753624381_0021_3_01_00, diagnostics=[TaskAttempt 0 
> failed, info=[Error: Error while running task ( failure ) : 
> attempt_1698753624381_0021_3_01_00_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:348)
>  at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276)   
> at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
>  at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
>at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
>at java.security.AccessController.doPrivileged(Native Method)   at 
> javax.security.auth.Subject.doAs(Subject.java:422)   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
>  at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
> at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)  
> at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
>   at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
>at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:750)Caused by: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing rowat 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:311)
> at 
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
>  at 
>

[jira] [Created] (HIVE-27835) java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory for DruidHandler

2023-10-31 Thread Basapuram Kumar (Jira) 
Basapuram Kumar created HIVE-27835:
--

 Summary: java.lang.NoClassDefFoundError: 
net/jpountz/lz4/LZ4Factory for DruidHandler
 Key: HIVE-27835
 URL: https://issues.apache.org/jira/browse/HIVE-27835
 Project: Hive
  Issue Type: Task
  Components: Hive
Affects Versions: 4.0.0-alpha-1
 Environment: hive-4-alpha-1

Druid-27.0.0
Reporter: Basapuram Kumar


Hello Team,

I am trying to use Druid storage handler to create a table.

Steps to repro

Create table

 
{code:java}
CREATE TABLE emp_zlib_3110_2038 (
`__time` TIMESTAMP,
 id int,
 name string,
 age int,
 gender string)
 STORED AS ORC
 TBLPROPERTIES ('transactional'='true',"orc.compress"="ZLIB");
 {code}
insert data
{code:java}
insert into emp_zlib_3110_2038 values(CURRENT_TIMESTAMP(),10,'basa',30,'M'); 
{code}
Create a DruidStorageHandler table
{code:java}
CREATE TABLE druid_hive_table_3110_2038_02
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'
TBLPROPERTIES ("druid.datasource" = "druid_hive_table_3110_2038_02")
AS
select `__time`, id,name,age,gender from emp_zlib_3110_2038 ; {code}
Error:
{code:java}
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1698753624381_0021)
--
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  
KILLED--Map
 1 .. container     SUCCEEDED      1          1        0        0       
0       0Reducer 2 ...    container       RUNNING      2          1        0    
    1       4       
0--VERTICES:
 01/02  [=>>-] 66%   ELAPSED TIME: 37.91 
s--

ERROR : Status: FailedERROR : Vertex failed, vertexName=Reducer 2, 
vertexId=vertex_1698753624381_0021_3_01, diagnostics=[Task failed, 
taskId=task_1698753624381_0021_3_01_00, diagnostics=[TaskAttempt 0 failed, 
info=[Error: Error while running task ( failure ) : 
attempt_1698753624381_0021_3_01_00_0:java.lang.RuntimeException: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:348)
 at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:276)   at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
 at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
   at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
   at java.security.AccessController.doPrivileged(Native Method)   at 
javax.security.auth.Subject.doAs(Subject.java:422)   at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
 at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)  at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
  at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
   at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
   at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:750)Caused by: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing rowat 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:311)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:291)
 at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:293)
 ... 16 moreCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:385)
at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:301)
... 18 more

Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory  at

[jira] [Created] (HIVE-27834) [Postgres] Use schema names instead of db names

2023-10-31 Thread Naveen Gangam (Jira) 
Naveen Gangam created HIVE-27834:


 Summary: [Postgres] Use schema names instead of db names
 Key: HIVE-27834
 URL: https://issues.apache.org/jira/browse/HIVE-27834
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


The Hiveserver2 side support for JDBC storage handler uses a select query to 
fetch the table/column metadata. something like this.
select * from  limit 1; --> where the tbl_name is 
schemaName.tableName where schemaName is the value if hive.sql.schema on the 
table.

https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L125-L129
https://github.com/apache/hive/blob/master/jdbc-handler/src/main/java/org/apache/hive/storage/jdbc/dao/GenericJdbcDatabaseAccessor.java#L557-L563

So natively in Postgres, the following happens from PSQL (same is true for 
oracle where database and schemas are different, on MySQL seems to treat them 
as one and same)

{noformat}
hive_hms_testing=> select * from public."TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from "TXNS" limit 1;
 TXN_ID | TXN_STATE | TXN_STARTED | TXN_LAST_HEARTBEAT | TXN_USER | TXN_HOST | 
TXN_AGENT_INFO | TXN_META_INFO | TXN_HEARTBEAT_COUNT | TXN_TYPE 
+---+-++--+--++---+-+--
(0 rows)

hive_hms_testing=> select * from hive_hms_testing."TXNS" limit 1;
ERROR:  relation "hive_hms_testing.TXNS" does not exist
LINE 1: select * from hive_hms_testing."TXNS" limit 1;
  ^
hive_hms_testing=> 
{noformat}

so schemaname cannot be the name of the database. if a schema name is not 
specified, then all tables in the database, across all schemas are listed. But 
if user wants to limit to a certain schema, they have to use the schemaname in 
"connector.remoteDbName" which then needs to be used as "hive.sql.schema" for 
the table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26192) JDBC data connector queries occur exception at cbo stage

2023-10-31 Thread Naveen Gangam (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781386#comment-17781386
 ] 

Naveen Gangam commented on HIVE-26192:
--

[~zhangbutao] exactly. Also needs a null check when inserting this property. I 
have the fix I have been testing for a couple of days against other DBs. Should 
be done shortly. I can create a PR. Thanks again

> JDBC data connector queries  occur exception at cbo stage
> -
>
> Key: HIVE-26192
> URL: https://issues.apache.org/jira/browse/HIVE-26192
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> If you do a select query qtest with jdbc data connector, you will  see 
> exception at cbo stage:
> {code:java}
> [ERROR] Failures:
> [ERROR]   TestMiniLlapCliDriver.testCliDriver:62 Client execution failed with 
> error code = 4
> running
> select * from country
> fname=dataconnector_mysql.qSee ./ql/target/tmp/log/hive.log or 
> ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports 
> or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
>  org.apache.hadoop.hive.ql.parse.SemanticException: Table qtestDB.country was 
> not found in the database
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3078)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5048)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1665)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1605)
>         at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
>         at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
>         at 
> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
>         at 
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1357)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:567)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12587)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:121)
>         at 
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:227)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:200)
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:126)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:421)
>         at 
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:352)
>         at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:727)
>         at 
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:697)
>         at 
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:114)
>         at 
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
>         at 
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver(TestMiniLlapCliDriver.java:62)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>         at 
>

[jira] [Updated] (HIVE-27833) Hive Acid Replication Support for Dell Powerscale

2023-10-31 Thread Harshal Patel (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harshal Patel updated HIVE-27833:
-
Status: Patch Available  (was: In Progress)

> Hive Acid Replication Support for Dell Powerscale
> -
>
> Key: HIVE-27833
> URL: https://issues.apache.org/jira/browse/HIVE-27833
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Harshal Patel
>Assignee: Harshal Patel
>Priority: Major
>  Labels: pull-request-available
>
> Hive Acid replication on Dell Powerscale Isilon fails because Isilon only 
> supports getXAttrs on /.reserved/raw paths



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27833) Hive Acid Replication Support for Dell Powerscale

2023-10-31 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-27833:
--
Labels: pull-request-available  (was: )

> Hive Acid Replication Support for Dell Powerscale
> -
>
> Key: HIVE-27833
> URL: https://issues.apache.org/jira/browse/HIVE-27833
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Harshal Patel
>Assignee: Harshal Patel
>Priority: Major
>  Labels: pull-request-available
>
> Hive Acid replication on Dell Powerscale Isilon fails because Isilon only 
> supports getXAttrs on /.reserved/raw paths



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27833) Hive Acid Replication Support for Dell Powerscale

2023-10-31 Thread Harshal Patel (Jira) 
Harshal Patel created HIVE-27833:


 Summary: Hive Acid Replication Support for Dell Powerscale
 Key: HIVE-27833
 URL: https://issues.apache.org/jira/browse/HIVE-27833
 Project: Hive
  Issue Type: Bug
  Components: repl
Reporter: Harshal Patel
Assignee: Harshal Patel


Hive Acid replication on Dell Powerscale Isilon fails because Isilon only 
supports getXAttrs on /.reserved/raw paths



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work started] (HIVE-27833) Hive Acid Replication Support for Dell Powerscale

2023-10-31 Thread Harshal Patel (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-27833 started by Harshal Patel.

> Hive Acid Replication Support for Dell Powerscale
> -
>
> Key: HIVE-27833
> URL: https://issues.apache.org/jira/browse/HIVE-27833
> Project: Hive
>  Issue Type: Bug
>  Components: repl
>Reporter: Harshal Patel
>Assignee: Harshal Patel
>Priority: Major
>
> Hive Acid replication on Dell Powerscale Isilon fails because Isilon only 
> supports getXAttrs on /.reserved/raw paths



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27598) Enhance alter table compact to work for partitioned tables without specifying any partition

2023-10-31 Thread Taraka Rama Rao Lethavadla (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taraka Rama Rao Lethavadla resolved HIVE-27598.
---
Fix Version/s: 4.0.0-beta-1
   Resolution: Fixed

Thanks for review & merge [~dkuzmenko] , [~veghlaci05]  

> Enhance alter table compact to work for partitioned tables without specifying 
> any partition
> ---
>
> Key: HIVE-27598
> URL: https://issues.apache.org/jira/browse/HIVE-27598
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-beta-1
>
>
> *current implementation:* the below command will throw error
>  
> {noformat}
> alter table PARTITIONED_T compact 'minor';{noformat}
> for a partitioned table
> {code:java}
> You must specify a partition to compact for partitioned tables{code}
> {*}Problem{*}: One of the problem could be ,let's say a table contains 1000s 
> of partitions and for some issue like files permission automatic compaction 
> will mark all or few partitions as failed and never pick for compaction. 
> After we see performance getting degraded, we find out that there is a 
> problem in compaction due to some issue.
> Now to solve it, we have two approaches, adjust failure thresholds and 
> restart HMS to pick the changes which is not going to work for someone who 
> cannot restart HMS
> And the second approach is to execute alter table mentioning each partition
> {*}Solution{*}:
> In this Jira we are planning to allow alter table command for a partitioned 
> table without specifying any partition name.
> Proposed solution is to list all the partitions and find partitions that are 
> eligible to get compacted and put an entry for it in compaction queue



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27831) Set hive.cbo.fallback.strategy to NEVER by default

2023-10-31 Thread Stamatis Zampetakis (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781316#comment-17781316
 ] 

Stamatis Zampetakis commented on HIVE-27831:


Running the precommit tests with CBO fallback disabled leads to ~100 failures 
that can be grouped in the following categories. For each category, we include 
the file name (fname) from one representative failing test case along with the 
SQL query and the exception.
h3. Union type not supported

fname=annotate_stats_select.q
{code:sql}
explain select CREATE_UNION(0, "hello") from alltypes_orc
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Union 
type is not supported
{noformat}
h3. TABLESAMPLE not supported

fname=archive_excludeHadoop20.q
{code:sql}
SELECT key FROM harbucket TABLESAMPLE(BUCKET 1 OUT OF 10) SORT BY key
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Table 
Sample specified for harbucket. Currently we don't support Table Sample clauses 
in CBO, turn off cbo for queries on tableSamples.
{noformat}
h3. Ambiguous column references

fname=ambiguous_col.q
{code:sql}
explain select * from (select a.key, a.* from (select * from src) a join 
(select * from src1) b on (a.key = b.key)) t 
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Cannot 
add column to RR: a.key => _col1: string due to duplication, see previous 
warnings
{noformat}
h3. Filter expression with non-boolean return type

fname=annotate_stats_filter.q
{code:sql}
explain select * from loc_orc where 'foo' 
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Filter 
expression with non-boolean return type.
{noformat}
h3. SELECT alias in HAVING clause not supported

fname=limit_pushdown_negative.q
{code:sql}
explain select value, sum(key) as sum from src group by value having sum > 100 
limit 20
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: 
Encountered Select alias 'sum' in having clause 'sum > 100' This non standard 
behavior is not supported with cbo on. Turn off cbo for these queries.
{noformat}
h3. Unexpected rexnode

fname=nested_column_pruning.q
{code:sql}
EXPLAIN
SELECT count(s1.f6), s5.f16.f18.f19
FROM nested_tbl_1_n1
GROUP BY s5.f16.f18.f19 
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: 
Unexpected rexnode : org.apache.calcite.rex.RexFieldAccess
{noformat}
fname=udaf_ngrams.q
{code:sql}
SELECT ngrams(sentences(lower(contents)), 1, 100, 1000).estfrequency FROM kafka 
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: 
Unexpected rexnode : org.apache.calcite.rex.RexInputRef
{noformat}
h3. UNIQUE JOIN not supported

fname=explainuser_2.q
{code:sql}
EXPLAIN FROM UNIQUEJOIN PRESERVE src a_n19 (a_n19.key), PRESERVE src1 b_n15 
(b_n15.key), PRESERVE srcpart c_n4 (c_n4.key) SELECT a_n19.key, b_n15.key, 
c_n4.key 
{code}
{noformat}
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: UNIQUE 
JOIN is currently not supported in CBO, turn off cbo to use UNIQUE JOIN.
{noformat}
h3. DirectSQL exception during partition pruning

fname=materialized_view_authorization_sqlstd.q
{code:sql}
explain select * from db1.testmvtable where year=2020 
{code}
{noformat}
java.lang.RuntimeException: org.apache.hadoop.hive.ql.parse.SemanticException: 
MetaException(message:See previous errors; Error executing SQL query "select 
"PARTITIONS"."PART_ID" from "PARTITIONS"  inner join "TBLS" on 
"PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ?   inner 
join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID"  and "DBS"."NAME" = ? inner 
join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" = 
"PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where "DBS"."CTLG_NAME" 
= ?  and (((case when "FILTER0"."PART_KEY_VAL" <> ? and "TBLS"."TBL_NAME" = ? 
and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and "FILTER0"."PART_ID" = 
"PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 then 
cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))".Failed to 
execute [select "PARTITIONS"."PART_ID" from "PARTITIONS"  inner join "TBLS" on 
"PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID" and "TBLS"."TBL_NAME" = ?   inner 
join "DBS" on "TBLS"."DB_ID" = "DBS"."DB_ID"  and "DBS"."NAME" = ? inner 
join "PARTITION_KEY_VALS" "FILTER0" on "FILTER0"."PART_ID" = 
"PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 where "DBS"."CTLG_NAME" 
= ?  and (((case when "FILTER0"."PART_KEY_VAL" <> ? and "TBLS"."TBL_NAME" = ? 
and "DBS"."NAME" = ? and "DBS"."CTLG_NAME" = ? and "FILTER0"."PART_ID" = 
"PARTITIONS"."PART_ID" and "FILTER0"."INTEGER_IDX" = 0 then 
cast("FILTER0"."PART_KEY_VAL" as decimal(21,0)) else null end) = ?))] with 
parameters [testmvtable, db1, hive, __HIVE_DEFAULT_PARTITION__, testmvtable, 
db1, hive, 2020])

[jira] [Commented] (HIVE-26192) JDBC data connector queries occur exception at cbo stage

2023-10-31 Thread zhangbutao (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-26192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781230#comment-17781230
 ] 

zhangbutao commented on HIVE-26192:
---

[~ngangam] Thanks for letting me this issue.

If i understand correctlly, we should change code as follows when the jdbc 
connector has different meaning between schema and database. e.g, postgres and 
oracle.  getCatalogName() can keep be null as for PG the database name must be 
specified in jdbc url, e.g. {*}jdbc:postgresql://localhost:5432/testpgdb{*}, so 
the value in getCatalogName() is no need any more and also it has no effect for 
the PG connection.

And users can  select a certain schema if they use the schemaname from property 
"connector.remoteDbName". I have tested this change locally, it works as 
expected.
{code:java}
diff --git 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/PostgreSQLConnectorProvider.java
 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/PostgreSQLConnectorProvider.java
index b79bee452d..79a505e6a9 100644
--- 
a/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/PostgreSQLConnectorProvider.java
+++ 
b/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/PostgreSQLConnectorProvider.java
@@ -36,11 +36,11 @@ public PostgreSQLConnectorProvider(String dbName, 
DataConnector dataConn) {
   }   @Override protected String getCatalogName() {
-    return scoped_db;
+    return null;
   }   @Override protected String getDatabaseName() {
-    return null;
+    return scoped_db;
   }
 {code}
Do I understand your question correctly? If we come to an agreement about this 
issue, i can submit a PR to fix it. Thanks.

> JDBC data connector queries  occur exception at cbo stage
> -
>
> Key: HIVE-26192
> URL: https://issues.apache.org/jira/browse/HIVE-26192
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> If you do a select query qtest with jdbc data connector, you will  see 
> exception at cbo stage:
> {code:java}
> [ERROR] Failures:
> [ERROR]   TestMiniLlapCliDriver.testCliDriver:62 Client execution failed with 
> error code = 4
> running
> select * from country
> fname=dataconnector_mysql.qSee ./ql/target/tmp/log/hive.log or 
> ./itests/qtest/target/tmp/log/hive.log, or check ./ql/target/surefire-reports 
> or ./itests/qtest/target/surefire-reports/ for specific test cases logs.
>  org.apache.hadoop.hive.ql.parse.SemanticException: Table qtestDB.country was 
> not found in the database
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genTableLogicalPlan(CalcitePlanner.java:3078)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:5048)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1665)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1605)
>         at 
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
>         at 
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
>         at 
> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
>         at 
> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1357)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:567)
>         at 
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12587)
>         at 
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
>         at 
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:317)
>         at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
>         at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:106)
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:500)
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:452)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:416)
>         at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:410)
>         at 
>

[jira] [Updated] (HIVE-27662) Incorrect parsing of nested complex types containing map during vectorized text processing

2023-10-31 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-27662:
--
Labels: pull-request-available  (was: )

> Incorrect parsing of nested complex types containing map during vectorized 
> text processing
> --
>
> Key: HIVE-27662
> URL: https://issues.apache.org/jira/browse/HIVE-27662
> Project: Hive
>  Issue Type: Bug
>  Components: Vectorization
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
>  Labels: pull-request-available
>
> When reading a text table with vectorization on and 
> hive.fetch.task.conversion as none, wrong parsing of delimiter is happening 
> in nested complex types containing map. For example, if a columns schema is 
> like: map then \u0004 char is coming in 
> the output. Here is a example:
>  
> Sample q file:
>  
> {code:java}
> set hive.fetch.task.conversion=none;
> set hive.vectorized.execution.enabled=true;
> create EXTERNAL table `table4` as
> select
>   'bob' as name,
>   map(
>       "Map_Key1",
>         named_struct(
>             'Id',
>             'Id_Value1',
>             'Name',
>             'Name_Value1'
>         ),
>       "Map_Key2",
>         named_struct(
>             'Id',
>             'Id_Value2',
>             'Name',
>             'Name_Value2'
>         )
>   ) as testmarks;
> select * from table4;
> set hive.vectorized.execution.enabled=false;
> select * from table4;
> {code}
> Output of 1st select statement:
> {code:java}
> bob·    
> {"Map_Key1":{"id":"Id_Value1\u0004Name_Value1","name":null},"Map_Key2":{"id":"Id_Value2\u0004Name_Value2","name":null}}{code}
> Output of 2nd select statement:
> {code:java}
> bob·    
> {"Map_Key1":{"id":"Id_Value1","name":"Name_Value1"},"Map_Key2":{"id":"Id_Value2","name":"Name_Value2"}}{code}
>  
> MAP Complex type is not handling the scenario where it contains a nested 
> complex type like STRUCT, ARRAY, UNION.
>  
> *To reproduce this issue:*
> *mvn test -Dtest=TestCliDriver -Pitests -Dqfile=`qfile_name`-pl itests/qtest 
> -Dtest.output.overwrite*



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-27708) Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws ClassCastException

2023-10-31 Thread Sankar Hariappan (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781223#comment-17781223
 ] 

Sankar Hariappan commented on HIVE-27708:
-

Merged the PR to branch-3. Thanks [~kamalsharma] for the contribution!

> Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws 
> ClassCastException
> 
>
> Key: HIVE-27708
> URL: https://issues.apache.org/jira/browse/HIVE-27708
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.1.3
>Reporter: Kamal Sharma
>Assignee: Kamal Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (HIVE-27708) Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws ClassCastException

2023-10-31 Thread Sankar Hariappan (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-27708:

Affects Version/s: 3.1.3

> Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws 
> ClassCastException
> 
>
> Key: HIVE-27708
> URL: https://issues.apache.org/jira/browse/HIVE-27708
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.1.3
>Reporter: Kamal Sharma
>Assignee: Kamal Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-27708) Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws ClassCastException

2023-10-31 Thread Sankar Hariappan (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan resolved HIVE-27708.
-
Resolution: Fixed

> Backport of HIVE-25299 : HIVE-21104: PTF with nested structure throws 
> ClassCastException
> 
>
> Key: HIVE-27708
> URL: https://issues.apache.org/jira/browse/HIVE-27708
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Kamal Sharma
>Assignee: Kamal Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)