[jira] [Created] (HIVE-26402) HiveSchemaTool does not honor metastore-site.xml

2022-07-18 Thread Zhihua Deng (Jira) 
Zhihua Deng created HIVE-26402:
--

 Summary: HiveSchemaTool does not honor metastore-site.xml
 Key: HIVE-26402
 URL: https://issues.apache.org/jira/browse/HIVE-26402
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Zhihua Deng


When using following scripts for initializing metastore schema,

 
{code:java}
export HIVE_CONF_DIR='/path/to/metastore_conf'
./bin/schematool -dbType mysql -initSchema{code}
the schematool command will be failed though we have a valid metastore-site.xml 
under the config path, it tries to init the default embeded db.
{noformat}
Metastore connection URL:     jdbc:derby:;databaseName=metastore_db;create=true
Metastore connection Driver :     org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User:     APP
Initializing the schema to: 4.0.0-alpha-2{noformat}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26403) Iceberg CTAS fails if source table schema contains decimal column

2022-07-18 Thread Jira 
László Pintér created HIVE-26403:


 Summary: Iceberg CTAS fails if source table schema contains 
decimal column
 Key: HIVE-26403
 URL: https://issues.apache.org/jira/browse/HIVE-26403
 Project: Hive
  Issue Type: Bug
Reporter: László Pintér
Assignee: László Pintér


Iceberg CTAS fails if the source table's schema contains a decimal column where 
the scale == precision. 

IcebergDecimalObjectInspector contains a precondition that validates if the 
scale < precision. 
By definition, the scale must be less than or equal to the precision. 





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26404) HMS memory leak when compaction cleaner fails to remove obsolete files

2022-07-18 Thread Stamatis Zampetakis (Jira) 
Stamatis Zampetakis created HIVE-26404:
--

 Summary: HMS memory leak when compaction cleaner fails to remove 
obsolete files
 Key: HIVE-26404
 URL: https://issues.apache.org/jira/browse/HIVE-26404
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 4.0.0-alpha-1
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis


While investigating an issue where HMS becomes unresponsive we noticed a lot of 
failed attempts from the compaction Cleaner thread to remove obsolete 
directories with exceptions similar to the one below.
{noformat}
2022-06-16 05:48:24,819 ERROR org.apache.hadoop.hive.ql.txn.compactor.Cleaner: 
[Cleaner-executor-thread-0]: Caught exception when cleaning, unable to complete 
cleaning of 
id:4410976,dbname:my_database,tableName:my_table,partName:day=20220502,state:,type:MAJOR,enqueueTime:0,start:0,properties:null,runAs:some_user,tooManyAborts:false,hasOldAbort:false,highestWriteId:187502,errorMessage:null
 java.io.IOException: Not enough history available for (187502,x).  Oldest 
available base: 
hdfs://nameservice1/warehouse/tablespace/managed/hive/my_database.db/my_table/day=20220502/base_0188687_v4297872
at 
org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1432)
at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.removeFiles(Cleaner.java:261)
at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.access$000(Cleaner.java:71)
at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner$1.run(Cleaner.java:203)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.clean(Cleaner.java:200)
at 
org.apache.hadoop.hive.ql.txn.compactor.Cleaner.lambda$run$0(Cleaner.java:105)
at 
org.apache.hadoop.hive.ql.txn.compactor.CompactorUtil$ThrowingRunnable.lambda$unchecked$0(CompactorUtil.java:54)
at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
In addition the logs contained a large number of long JVM pauses as shown below 
and the HMS (RSZ) memory kept increasing at rate of 90MB per hour.
{noformat}
2022-06-16 16:17:17,805 WARN  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 34346ms
2022-06-16 16:17:21,497 INFO  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 1690ms
2022-06-16 16:17:57,696 WARN  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 34697ms
2022-06-16 16:18:01,326 INFO  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 1628ms
2022-06-16 16:18:37,280 WARN  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 34453ms
2022-06-16 16:18:40,927 INFO  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 1646ms
2022-06-16 16:19:16,929 WARN  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 33997ms
2022-06-16 16:19:20,572 INFO  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 1637ms
2022-06-16 16:20:01,643 WARN  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor$Monitor@5b022296]: 
Detected pause in JVM or host machine (eg GC): pause of approximately 39329ms
2022-06-16 16:20:05,572 INFO  
org.apache.hadoop.hive.metastore.metrics.JvmPauseMonitor: 
[org.apache.hadoop.h

[jira] [Created] (HIVE-26405) Hive - 'show create database' does not contain managedlocation information.

2022-07-18 Thread Hongdan Zhu (Jira) 
Hongdan Zhu created HIVE-26405:
--

 Summary: Hive - 'show create database' does not contain 
managedlocation information.
 Key: HIVE-26405
 URL: https://issues.apache.org/jira/browse/HIVE-26405
 Project: Hive
  Issue Type: Improvement
Reporter: Hongdan Zhu


'show create database' does not contain managedlocation information.

STR:
1. create database using MANAGEDLOCATION clause.
2. run 'show create database' command.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26406) Hive create query is not working when odd number of column name have apostrophe(')

2022-07-18 Thread Hetashvi (Jira) 
Hetashvi created HIVE-26406:
---

 Summary: Hive create query is not working when odd number of 
column name have apostrophe(')
 Key: HIVE-26406
 URL: https://issues.apache.org/jira/browse/HIVE-26406
 Project: Hive
  Issue Type: Bug
  Components: CLI
Affects Versions: 2.3.6
Reporter: Hetashvi


Hive create table is not working when column name have apostrophe('). This 
happens only when the numbers of columns are odd.  
{code:java}
hive> create table `test_table` (`i'd` int, `i'd1` int); 
OK
Time taken: 1.662 seconds
hive> describe `test_table`;
OK
i'd                     int                                         
i'd1                    int                                         
Time taken: 0.129 seconds, Fetched: 2 row(s)
hive> create table `test_table1` (`i'd` int); 
FAILED: ParseException line 1:38 extraneous input ';' expecting EOF near ''
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26407) Do not collect statistics if the compaction fails

2022-07-18 Thread Zsolt Miskolczi (Jira) 
Zsolt Miskolczi created HIVE-26407:
--

 Summary: Do not collect statistics if the compaction fails
 Key: HIVE-26407
 URL: https://issues.apache.org/jira/browse/HIVE-26407
 Project: Hive
  Issue Type: Test
  Components: Hive
Reporter: Zsolt Miskolczi


It can still compute statistics, even if compaction fails.
if (computeStats) \{
  StatsUpdater.gatherStats(ci, conf, runJobAsSelf(ci.runAs) ? ci.runAs : 
t1.getOwner(),
  CompactorUtil.getCompactorJobQueueName(conf, ci, t1));
}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26408) Vectorization: Fix deallocation of scratch columns, don't reuse a child ConstantVectorExpression as an output

2022-07-18 Thread Jira 
László Bodor created HIVE-26408:
---

 Summary: Vectorization: Fix deallocation of scratch columns, don't 
reuse a child ConstantVectorExpression as an output
 Key: HIVE-26408
 URL: https://issues.apache.org/jira/browse/HIVE-26408
 Project: Hive
  Issue Type: Bug
Reporter: László Bodor
Assignee: László Bodor






--
This message was sent by Atlassian Jira
(v8.20.10#820010)