Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

[Rajesh Balamohan]

Congratulations Simhadri. :)

~Rajesh.B

On Fri, Apr 19, 2024 at 2:02 AM Aman Sinha  wrote:

> Congrats Simhadri !
>
> On Thu, Apr 18, 2024 at 12:25 PM Naveen Gangam
>  wrote:
>
>> Congrats Simhadri. Looking forward to many more contributions in the
>> future.
>>
>> On Thu, Apr 18, 2024 at 12:25 PM Sai Hemanth Gantasala
>>  wrote:
>>
>>> Congratulations Simhadri 🎉 well deserved
>>>
>>> On Thu, Apr 18, 2024 at 8:41 AM Pau Tallada  wrote:
>>>
 Congratulations

 Missatge de Alessandro Solimando  del
 dia dj., 18 d’abr. 2024 a les 17:40:

> Great news, Simhadri, very well deserved!
>
> On Thu, 18 Apr 2024 at 15:07, Simhadri G 
> wrote:
>
>> Thanks everyone!
>> I really appreciate it, it means a lot to me :)
>> The Apache Hive project and its community have truly inspired me .
>> I'm grateful for the chance to contribute to such a remarkable project.
>>
>> Thanks!
>> Simhadri Govindappa
>>
>> On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan
>>  wrote:
>>
>>> Congrats Simhadri!
>>>
>>>
>>>
>>> -Sankar
>>>
>>>
>>>
>>> *From:* Butao Zhang 
>>> *Sent:* Thursday, April 18, 2024 5:39 PM
>>> *To:* u...@hive.apache.org; dev 
>>> *Subject:* [EXTERNAL] Re: [ANNOUNCE] New Committer: Simhadri
>>> Govindappa
>>>
>>>
>>>
>>> You don't often get email from butaozha...@163.com. Learn why this
>>> is important 
>>>
>>> Congratulations Simhadri !!!
>>>
>>>
>>>
>>> Thanks.
>>>
>>>
>>> --
>>>
>>> *发件人**:* user-return-28075-butaozhang1=163@hive.apache.org <
>>> user-return-28075-butaozhang1=163@hive.apache.org> 代表 Ayush
>>> Saxena 
>>> *发送时间**:* 星期四, 四月 18, 2024 7:50 下午
>>> *收件人**:* dev ; u...@hive.apache.org <
>>> u...@hive.apache.org>
>>> *主题**:* [ANNOUNCE] New Committer: Simhadri Govindappa
>>>
>>>
>>>
>>> Hi All,
>>>
>>> Apache Hive's Project Management Committee (PMC) has invited
>>> Simhadri Govindappa to become a committer, and we are pleased to 
>>> announce
>>> that he has accepted.
>>>
>>>
>>>
>>> Please join me in congratulating him, Congratulations Simhadri,
>>> Welcome aboard!!!
>>>
>>>
>>>
>>> -Ayush Saxena
>>>
>>> (On behalf of Apache Hive PMC)
>>>
>>

 --
 --
 Pau Tallada Crespí
 Departament de Serveis
 Port d'Informació Científica (PIC)
 Tel: +34 93 170 2729
 --

[jira] [Created] (HIVE-27188) Explore usage of FilterApi.in(C column, Set values) in Parquet instead of nested OR

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27188:
---

 Summary: Explore usage of FilterApi.in(C column, Set values) in 
Parquet instead of nested OR
 Key: HIVE-27188
 URL: https://issues.apache.org/jira/browse/HIVE-27188
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Following query can throw stackoverflow exception with "Xss256K".

Currently it generates nested OR filter

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/FilterPredicateLeafBuilder.java#L43-L52]

Instead, need to explore the possibility of using {color:#de350b}FilterApi.in(C 
column, Set values) {color:#172b4d}in parquet{color}{color}

 
{noformat}
drop table if exists test;

create external table test (i int) stored as parquet;

insert into test values (1),(2),(3);

select count(*) from test where i in (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 
13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 
53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 
73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 
93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 
110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 
126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 
142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 
158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 
174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 
190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 
206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 
222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 
238, 239, 240, 241, 242, 243);

 {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27184) Add class name profiling option in ProfileServlet

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27184:
---

 Summary: Add class name profiling option in ProfileServlet
 Key: HIVE-27184
 URL: https://issues.apache.org/jira/browse/HIVE-27184
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


With async-profiler "-e classame.method", it is possible to profile specific 
events. Currently profileServlet supports events like cpu, alloc, lock etc. It 
will be good to enhance to support method name profiling as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27183) Iceberg: Table information is loaded multiple times

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27183:
---

 Summary: Iceberg: Table information is loaded multiple times
 Key: HIVE-27183
 URL: https://issues.apache.org/jira/browse/HIVE-27183
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


HMS::getTable invokes "HiveIcebergMetaHook::postGetTable" which internally 
loads iceberg table again.

If this isn't needed or needed only for show-create-table, do not load the 
table again.
{noformat}
    at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
    - parking to wait for  <0x00066f84eef0> (a 
java.util.concurrent.CompletableFuture$Signaller)
    at 
java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
    at 
java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.18/CompletableFuture.java:1796)
    at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.18/ForkJoinPool.java:3128)
    at 
java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.18/CompletableFuture.java:1823)
    at 
java.util.concurrent.CompletableFuture.get(java.base@11.0.18/CompletableFuture.java:1998)
    at org.apache.hadoop.util.functional.FutureIO.awaitFuture(FutureIO.java:77)
    at 
org.apache.iceberg.hadoop.HadoopInputFile.newStream(HadoopInputFile.java:196)
    at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:263)
    at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:177)
    at 
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$609/0x000840e18040.apply(Unknown
 Source)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:191)
    at 
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$610/0x000840e18440.run(Unknown
 Source)
    at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
    at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:191)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
    at 
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:153)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)
    at 
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)
    at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:115)
    at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:105)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil.lambda$getTable$1(IcebergTableUtil.java:99)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil$$Lambda$552/0x000840d59840.apply(Unknown
 Source)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil.lambda$getTable$4(IcebergTableUtil.java:111)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil$$Lambda$557/0x000840d58c40.get(Unknown
 Source)
    at java.util.Optional.orElseGet(java.base@11.0.18/Optional.java:369)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:108)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:69)
    at 
org.apache.iceberg.mr.hive.IcebergTableUtil.getTable(IcebergTableUtil.java:73)
    at 
org.apache.iceberg.mr.hive.HiveIcebergMetaHook.postGetTable(HiveIcebergMetaHook.java:931)
    at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.executePostGetTableHook(HiveMetaStoreClient.java:2638)
    at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:2624)
    at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTable(SessionHiveMetaStoreClient.java:267)
    at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
    at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
    at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:216)
    at com.sun.proxy.$Proxy56.getTable(Unknown Source)
    at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source)
    at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.inv

[jira] [Created] (HIVE-27159) Filters are not pushed down for decimal format in Parquet

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27159:
---

 Summary: Filters are not pushed down for decimal format in Parquet
 Key: HIVE-27159
 URL: https://issues.apache.org/jira/browse/HIVE-27159
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Decimal filters are not created and pushed down in parquet readers. This causes 
latency delays and unwanted row processing in query execution. 

It throws exception in runtime and processes more rows. 

E.g Q13.

{noformat}

Parquet: (Map 1)

INFO  : Task Execution Summary
INFO  : 
--
INFO  :   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   
INPUT_RECORDS   OUTPUT_RECORDS
INFO  : 
--
INFO  :  Map 1  31254.00  0  0 
549,181,950  133
INFO  :  Map 3  0.00  0  0  
73,049  365
INFO  :  Map 4   2027.00  0  0   
6,000,0001,689,919
INFO  :  Map 5  0.00  0  0   
7,2001,440
INFO  :  Map 6517.00  0  0   
1,920,800  493,920
INFO  :  Map 7  0.00  0  0   
1,0021,002
INFO  :  Reducer 2  18716.00  0  0 
1330
INFO  : 
--

ORC:


INFO  : Task Execution Summary
INFO  : 
--
INFO  :   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   
INPUT_RECORDS   OUTPUT_RECORDS
INFO  : 
--
INFO  :  Map 1   6556.00  0  0 
267,146,063  152
INFO  :  Map 3  0.00  0  0  
10,000  365
INFO  :  Map 4   2014.00  0  0   
6,000,0001,689,919
INFO  :  Map 5  0.00  0  0   
7,2001,440
INFO  :  Map 6504.00  0  0   
1,920,800  493,920
INFO  :  Reducer 2   3159.00  0  0 
1520
INFO  : 
--

{noformat}




{noformat}
 Map 1
Map Operator Tree:
TableScan
  alias: store_sales
  filterExpr: (ss_hdemo_sk is not null and ss_addr_sk is not 
null and ss_cdemo_sk is not null and ss_store_sk is not null and 
((ss_sales_price >= 100) or (ss_sales_price <= 150) or (ss_sales_price >= 50) 
or (ss_sales_price <= 100) or (ss_sales_price >= 150) or (ss_sales_price <= 
200)) and ((ss_net_profit >= 100) or (ss_net_profit <= 200) or (ss_net_profit 
>= 150) or (ss_net_profit <= 300) or (ss_net_profit >= 50) or (ss_net_profit <= 
250))) (type: boolean)
  probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_112_container, 
bigKeyColName:ss_hdemo_sk, smallTablePos:1, keyRatio:5.042575832290721E-6
  Statistics: Num rows: 2750380056 Data size: 1321831086472 
Basic stats: COMPLETE Column stats: COMPLETE
  Filter Operator
predicate: (ss_hdemo_sk is not null and ss_addr_sk is not 
null and ss_cdemo_sk is not null and ss_store_sk is not null and 
((ss_sales_price >= 100) or (ss_sales_price <= 150) or (ss_sales_price >= 50) 
or (ss_sales_price <= 100) or (ss_sales_price >= 150) or (ss_sales_price <= 
200)) and ((ss_net_profit >= 100) or (ss_net_profit <= 200) or (ss_net_profit 
>= 150) or (ss_net_profit <= 300) or (ss_net_profit >= 50) or (ss_net_profit <= 
250))) (type: boolean)
Statistics: Num rows: 2500252205 Data size: 1201619783884 
Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
  expressions: ss_cdemo_sk (type: bigint), ss_hdemo_sk 
(type: bigint), ss_addr_sk (type: bigint), ss_store_sk (type: bigint), 
ss_quantity (type: int), ss_ext_sales_price (type: decimal(7,2)), 
ss_ext_wholesale_cost (type: decimal(7,2)), ss_sold_date_sk (type: bigint), 
ss_net_profit BETWEEN 100 AND 200 (type: boolean), ss_net_profit BETWEEN 150 
AND 300 (type: boolean), ss_net_profit BETWEEN 50 AND 250 (type: boolean), 
ss_sales_price BETWEEN 100 AND 150 (type: boolean), ss_sales_price BETWEEN 50 
AN

[jira] [Created] (HIVE-27144) Alter table partitions need not DBNotificationListener for external tables

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27144:
---

 Summary: Alter table partitions need not DBNotificationListener 
for external tables
 Key: HIVE-27144
 URL: https://issues.apache.org/jira/browse/HIVE-27144
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


DBNotificationListener for external tables may not be needed. 

Even for "analyze table blah compute statistics for columns" for external 
partitioned tables, it invokes DBNotificationListener for all partitions. 


{noformat}
at org.datanucleus.store.query.Query.execute(Query.java:1726)
  at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:374)
  at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216)
  at 
org.apache.hadoop.hive.metastore.ObjectStore.addNotificationEvent(ObjectStore.java:11774)
  at jdk.internal.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
  at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
  at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
  at com.sun.proxy.$Proxy33.addNotificationEvent(Unknown Source)
  at 
org.apache.hive.hcatalog.listener.DbNotificationListener.process(DbNotificationListener.java:1308)
  at 
org.apache.hive.hcatalog.listener.DbNotificationListener.onAlterPartition(DbNotificationListener.java:458)
  at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier$14.notify(MetaStoreListenerNotifier.java:161)
  at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:328)
  at 
org.apache.hadoop.hive.metastore.MetaStoreListenerNotifier.notifyEvent(MetaStoreListenerNotifier.java:390)
  at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:863)
  at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:6253)
  at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:6201)
  at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.18/Native 
Method)
  at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.18/NativeMethodAccessorImpl.java:62)
  at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
  at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:160)
  at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:121)
  at com.sun.proxy.$Proxy34.alter_partitions_req(Unknown Source)
  at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:21532)
  at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:21511)
  at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
  at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
  at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:652)
  at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:647)
  at java.security.AccessController.doPrivileged(java.base@11.0.18/Native 
Method)
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27119) Iceberg: Delete from table generates lot of files

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27119:
---

 Summary: Iceberg: Delete from table generates lot of files
 Key: HIVE-27119
 URL: https://issues.apache.org/jira/browse/HIVE-27119
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


With "delete" it generates lot of files due to the way data is sent to the 
reducers. Files per partition is impacted by the number of reduce tasks.

One way could be to explicitly control the number of reducers; Creating this 
ticket to have a long term fix.
 
{noformat}
 explain delete from store_Sales where ss_customer_sk % 10 = 0;
INFO  : Compiling 
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b): 
explain delete from store_Sales where ss_customer_sk % 10 = 0
INFO  : No Stats for tpcds_1000_iceberg_mor_v4@store_sales, Columns: 
ss_sold_time_sk, ss_cdemo_sk, ss_promo_sk, ss_ext_discount_amt, 
ss_ext_sales_price, ss_net_profit, ss_addr_sk, ss_ticket_number, 
ss_wholesale_cost, ss_item_sk, ss_ext_list_price, ss_sold_date_sk, ss_store_sk, 
ss_coupon_amt, ss_quantity, ss_list_price, ss_sales_price, ss_customer_sk, 
ss_ext_wholesale_cost, ss_net_paid, ss_ext_tax, ss_hdemo_sk, ss_net_paid_inc_tax
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:Explain, 
type:string, comment:null)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b); Time 
taken: 0.704 seconds
INFO  : Executing 
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b): 
explain delete from store_Sales where ss_customer_sk % 10 = 0
INFO  : Starting task [Stage-4:EXPLAIN] in serial mode
INFO  : Completed executing 
command(queryId=hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b); Time 
taken: 0.005 seconds
INFO  : OK
Explain
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-2 depends on stages: Stage-1
  Stage-0 depends on stages: Stage-2
  Stage-3 depends on stages: Stage-0

STAGE PLANS:
  Stage: Stage-1
Tez
  DagId: hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b:377
  Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
  DagName: hive_20230303021031_855dd644-8f67-482d-98d7-e9f70b56ae0b:377
  Vertices:
Map 1
Map Operator Tree:
TableScan
  alias: store_sales
  filterExpr: ((ss_customer_sk % 10) = 0) (type: boolean)
  Statistics: Num rows: 2755519629 Data size: 3643899155232 
Basic stats: COMPLETE Column stats: NONE
  Filter Operator
predicate: ((ss_customer_sk % 10) = 0) (type: boolean)
Statistics: Num rows: 1377759814 Data size: 1821949576954 
Basic stats: COMPLETE Column stats: NONE
Select Operator
  expressions: PARTITION__SPEC__ID (type: int), 
PARTITION__HASH (type: bigint), FILE__PATH (type: string), ROW__POSITION (type: 
bigint), ss_sold_time_sk (type: int), ss_item_sk (type: int), ss_customer_sk 
(type: int), ss_cdemo_sk (type: int), ss_hdemo_sk (type: int), ss_addr_sk 
(type: int), ss_store_sk (type: int), ss_promo_sk (type: int), ss_ticket_number 
(type: bigint), ss_quantity (type: int), ss_wholesale_cost (type: 
decimal(7,2)), ss_list_price (type: decimal(7,2)), ss_sales_price (type: 
decimal(7,2)), ss_ext_discount_amt (type: decimal(7,2)), ss_ext_sales_price 
(type: decimal(7,2)), ss_ext_wholesale_cost (type: decimal(7,2)), 
ss_ext_list_price (type: decimal(7,2)), ss_ext_tax (type: decimal(7,2)), 
ss_coupon_amt (type: decimal(7,2)), ss_net_paid (type: decimal(7,2)), 
ss_net_paid_inc_tax (type: decimal(7,2)), ss_net_profit (type: decimal(7,2)), 
ss_sold_date_sk (type: int)
  outputColumnNames: _col0, _col1, _col2, _col3, _col4, 
_col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, 
_col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, 
_col25, _col26
  Statistics: Num rows: 1377759814 Data size: 1821949576954 
Basic stats: COMPLETE Column stats: NONE
  Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: 
bigint), _col2 (type: string), _col3 (type: bigint)
null sort order: 
sort order: 
Statistics: Num rows: 1377759814 Data size: 
1821949576954 Basic stats: COMPLETE Column stats: NONE
value expressions: _col4 (type: int), _col5 (type: 
int), _col6 (type: int), _col7 (type: int), _col8 (type: int), _col9 (type: 
int), _col10 (type: int), _col11 (type: int), _col12 (type: bigint), _col13 
(type: int), _col14 (type: decimal(7,2)), _col15 (type: decimal(7,2)), _col16 
(type: decimal(

[jira] [Created] (HIVE-27099) Iceberg: select count(*) from table queries all data

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27099:
---

 Summary: Iceberg: select count(*) from table queries all data
 Key: HIVE-27099
 URL: https://issues.apache.org/jira/browse/HIVE-27099
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


select count is scanning all data. Though it has complete basic stats, it 
launched tez job which wasn't needed. Second issue is, it ended up scanning 
ENTIRE 148 GB dataset which is completely not required. It should have got the 
data from parq files itself. Ideal situation is getting entire records from 
manifest itself.

Data is stored in parquet format in external tables. This may be broken for 
parquet, as for ORC it is able to read less data (footer info). 

1. Consider fixing count( * ) for parq
2. Check if it is possible to read stats from iceberg manifests after #1.


{noformat}

explain select count(*) from store_sales;

Explain
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
  DagId: hive_20230223031934_2abeb3b9-8c18-4ff7-a8f9-df7368010189:5
  Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
  DagName: hive_20230223031934_2abeb3b9-8c18-4ff7-a8f9-df7368010189:5
  Vertices:
Map 1
Map Operator Tree:
TableScan
  alias: store_sales
  Statistics: Num rows: 2879966589 Data size: 195666988943 
Basic stats: COMPLETE Column stats: COMPLETE
  Select Operator
Statistics: Num rows: 2879966589 Data size: 195666988943 
Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
  aggregations: count()
  minReductionHashAggr: 0.5
  mode: hash
  outputColumnNames: _col0
  Statistics: Num rows: 1 Data size: 8 Basic stats: 
COMPLETE Column stats: COMPLETE
  Reduce Output Operator
null sort order:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: 
COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint)
Execution mode: vectorized
Reducer 2
Execution mode: vectorized
Reduce Operator Tree:
  Group By Operator
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: COMPLETE
File Output Operator
  compressed: false
  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE 
Column stats: COMPLETE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink

53 rows selected (1.454 seconds)

0: jdbc:hive2://ve0:218> select count(*) from store_sales;
INFO  : Query ID = hive_20230223031940_9ff5d61d-1fe2-4476-a561-7820e4a3a5f8
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Subscribed to counters: [] for queryId: 
hive_20230223031940_9ff5d61d-1fe2-4476-a561-7820e4a3a5f8
INFO  : Session is already open
INFO  : Dag name: select count(*) from store_sales (Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1676286357243_0061)

--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED
--
Map 1 .. container SUCCEEDED76776700
   0   0
Reducer 2 .. container SUCCEEDED  1  100
   0   0
--
VERTICES: 02/02  [==>>] 100%  ELAPSED TIME: 54.94 s
--
INFO  : Status: DAG finished successfully in 54.85 seconds
INFO  :
INFO  : Query Execution Summary
INFO  : 
--
INFO  : OPERATIONDURATION
INFO  : 
--
INFO

Re: Proposal to deprecate Hive on Spark from branch-3

[Rajesh Balamohan]

+1 on removing Hive on Spark in branch-3.

It was not done earlier since it was removing a feature in the branch. But
if there is enough consensus, we should consider removing it.

~Rajesh.B

On Wed, Feb 22, 2023 at 12:48 PM Aman Raj 
wrote:

> Hi team,
>
> We have been trying to fix Hive on Spark test failures for a long time. As
> of now, branch-3 has less than 12 test failures (whose fix have not been
> identified). 8 of them are related to Hive on Spark. I had mailed about the
> failures in my previous mail threads. Thanks to Vihang for working on them
> as well. But we have not been able to identify the root cause till now.
> These fixes can be tracked in the following tickets : [HIVE-27087] Fix
> TestMiniSparkOnYarnCliDriver test failures on branch-3 - ASF JIRA (
> apache.org) and
> [HIVE-26940] Backport of HIVE-19882 : Fix QTestUtil session lifecycle - ASF
> JIRA (apache.org)
>
> Until we have a green branch-3, we cannot go ahead to push new features
> for the Hive-3.2.0 release. This is kind of a blocker for this release.
> Already bringing the test fixes to the current state took more than 2
> months.
>
> I wanted to bring up a proposal to deprecate Hive on Spark from branch-3
> altogether. This would ensure that branch-3 is aligned with the master as
> done in https://issues.apache.org/jira/browse/HIVE-26134. Just wanted to
> have a vote on this in parallel working on the test fixes. If we have the
> approval from the community, we can deprecate it altogether.
>
> Please feel free to suggest any concerns or suggestions you have. Also, I
> welcome any possible fix suggestion for the test failures.
>
> Thanks,
> Aman.
>

[jira] [Created] (HIVE-27084) Iceberg: Stats are not populated correctly during query compilation

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27084:
---

 Summary: Iceberg: Stats are not populated correctly during query 
compilation
 Key: HIVE-27084
 URL: https://issues.apache.org/jira/browse/HIVE-27084
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


- Table stats are not properly used/computed during query compilation phase.
 - Here is an example. Check out the query with the filter which give more data 
than the regular query

This is just an example, real world queries can have bad query plans due to this

{{10470974584 with filter, vs 303658262936 without filter}}

{noformat}
explain select count(*) from store_sales where ss_sold_date_sk=2450822 and 
ss_wholesale_cost > 0.0

Explain
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
  DagId: hive_20230216065808_80d68e3f-3a6b-422b-9265-50bc707ae3c6:48
  Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
  DagName: hive_20230216065808_80d68e3f-3a6b-422b-9265-50bc707ae3c6:48
  Vertices:
Map 1
Map Operator Tree:
TableScan
  alias: store_sales
  filterExpr: ((ss_sold_date_sk = 2450822) and 
(ss_wholesale_cost > 0)) (type: boolean)
  Statistics: Num rows: 2755519629 Data size: 303658262936 
Basic stats: COMPLETE Column stats: NONE
  Filter Operator
predicate: ((ss_sold_date_sk = 2450822) and 
(ss_wholesale_cost > 0)) (type: boolean)
Statistics: Num rows: 5 Data size: 550 Basic stats: 
COMPLETE Column stats: NONE
Select Operator
  Statistics: Num rows: 5 Data size: 550 Basic stats: 
COMPLETE Column stats: NONE
  Group By Operator
aggregations: count()
minReductionHashAggr: 0.99
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 124 Basic stats: 
COMPLETE Column stats: NONE
Reduce Output Operator
  null sort order:
  sort order:
  Statistics: Num rows: 1 Data size: 124 Basic stats: 
COMPLETE Column stats: NONE
  value expressions: _col0 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs (cache only)
Reducer 2
Execution mode: vectorized, llap
Reduce Operator Tree:
  Group By Operator
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 124 Basic stats: COMPLETE 
Column stats: NONE
File Output Operator
  compressed: false
  Statistics: Num rows: 1 Data size: 124 Basic stats: COMPLETE 
Column stats: NONE
  table:
  input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
  output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
ListSink

58 rows selected (0.73 seconds)



explain select count(*) from store_sales where ss_sold_date_sk=2450822
INFO  : Starting task [Stage-3:EXPLAIN] in serial mode
INFO  : Completed executing 
command(queryId=hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7); Time 
taken: 0.061 seconds
INFO  : OK
Explain
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
  DagId: hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7:49
  Edges:
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
  DagName: hive_20230216065813_e51482a2-1c9a-41a7-b1b3-9aec2fba9ba7:49
  Vertices:
Map 1
Map Operator Tree:
TableScan
  alias: store_sales
  filterExpr: (ss_sold_date_sk = 2450822) (type: boolean)
  Statistics: Num rows: 2755519629 Data size: 10470974584 Basic 
stats: COMPLETE Column stats: NONE
  Filter Operator
predicate: (ss_sold_date_sk = 2450822) (type: boolean)
Statistics: Num rows: 5 Data size: 18 Basic stats: COMPLETE 
Column stats: NONE
Select Operator
  Statistics: Num rows: 5 Data size: 18 Basic stats: 
COMPLETE Column stats: NONE
  Group By Operator
aggregations: count()
minReductio

[jira] [Created] (HIVE-27050) Iceberg: MOR: Restrict reducer extrapolation to contain number of small files being created

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27050:
---

 Summary: Iceberg: MOR: Restrict reducer extrapolation to contain 
number of small files being created
 Key: HIVE-27050
 URL: https://issues.apache.org/jira/browse/HIVE-27050
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


Scenario:
 # Create a simple table in iceberg (MOR mode). e.g store_sales_delete_1
 # Insert some data into it. 
 # Run an update statement as follows
 ## "update  store_sales_delete_1 set ss_sold_time_sk=699060 where 
ss_sold_time_sk=69906"

Hive estimates the number of reducers as "1". But due to 
"hive.tez.max.partition.factor" which defaults to "2.0", it will double the 
number of reducers.

To put in perspective, it will create very small positional delete files 
spreading across different reducers. This will cause problems during reading, 
as all files should be opened for reading.

 
 # When iceberg MOR tables are involved in update/delete/merges, disable 
"hive.tez.max.partition.factor"; or set it to "1.0" irrespective of the user 
setting;
 # Have explicit logs for easier debugging; User shouldn't be confused on why 
the setting is not taking into effect.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27049) Iceberg: Provide current snapshot version in show-create-table

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27049:
---

 Summary: Iceberg: Provide current snapshot version in 
show-create-table
 Key: HIVE-27049
 URL: https://issues.apache.org/jira/browse/HIVE-27049
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


It will be helpful to show "current snapshot" id in "show create table" 
statement. This will help in easier debugging. Otherwise, user has to 
explicitly query the metadata or read the JSON file to get this info.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27014) Iceberg: getSplits/planTasks should filter out relevant folders instead of scanning entire table

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27014:
---

 Summary: Iceberg: getSplits/planTasks should filter out relevant 
folders instead of scanning entire table
 Key: HIVE-27014
 URL: https://issues.apache.org/jira/browse/HIVE-27014
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


With dynamic partition pruning, only relevant folders in fact tables are 
scanned.

In tez, DynamicPartitionPruner will set the relevant filters.In iceberg, these 
filters are applied after "Table:planTasks()" is invoked in iceberg. This 
forces entire table metadata to be scanned and then throw off the unwanted 
partitions. 

This makes split computation expensive (e.g for store_sales, it has to look at 
all 1800+ partitions and throw off unwanted partitions).

For short running queries, it takes 3-5+ seconds for split computation. 
Creating this ticket as a placeholder to make use of the relevant filters from 
DPP.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27013) Provide an option to enable iceberg manifest caching via table properties

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27013:
---

 Summary: Provide an option to enable iceberg manifest caching via 
table properties
 Key: HIVE-27013
 URL: https://issues.apache.org/jira/browse/HIVE-27013
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


{color:#22}I tried the following thinking that it would work with iceberg 
manifest caching; but it didn't.{color}
{color:#22}{noformat}{color}
{color:#22}alter table store_sales set 
tblproperties('io.manifest.cac{color}{color:#22}he-enabled'='true'); 
\{noformat}{color}
{color:#22}Creating this ticket as a placeholder to fix the same.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27010) Reduce compilation time

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27010:
---

 Summary: Reduce compilation time
 Key: HIVE-27010
 URL: https://issues.apache.org/jira/browse/HIVE-27010
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Context: Post HIVE-24645, compilation time for queries has increased.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27005) Iceberg: Col stats are not used in queries

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27005:
---

 Summary: Iceberg: Col stats are not used in queries
 Key: HIVE-27005
 URL: https://issues.apache.org/jira/browse/HIVE-27005
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan
 Attachments: col_stats.txt

1. Though, insert-queries compute colstats during runtime, they are not 
persisted in HMS during final call. 

2. Due to #1, col stats are not available during runtime for hive queries. This 
includes col stats, NDV etc. So unless users explicitly run "analyse table" 
statements, queries can be have suboptimal plans.

E.g 
[col_stats.txt{^}!https://jira.cloudera.com/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.cloudera.com/secure/attachment/658390/658390_col_stats.txt](note
 that there is no col stats being used)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27003) Iceberg: Vectorization missed out for update/delete due to virtual columns

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-27003:
---

 Summary: Iceberg: Vectorization missed out for update/delete due 
to virtual columns
 Key: HIVE-27003
 URL: https://issues.apache.org/jira/browse/HIVE-27003
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan
 Attachments: delete_iceberg_vect.txt, update_iceberg_vect.txt

Vectorization is missed out during table scan due to the addition of virtual 
columns during scans. I will attach the plan details here with.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26997) Iceberg: Vectorization gets disabled at runtime in merge-into statements

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26997:
---

 Summary: Iceberg: Vectorization gets disabled at runtime in 
merge-into statements
 Key: HIVE-26997
 URL: https://issues.apache.org/jira/browse/HIVE-26997
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan
 Attachments: explain_merge_into.txt

*Query:*

Think of "ssv" table as a table containing trickle feed data in the following 
query. "store_sales_delete_1" is the destination table.

 
{noformat}
MERGE INTO tpcds_1000_iceberg_mor_v4.store_sales_delete_1 t USING 
tpcds_1000_update.ssv s ON (t.ss_item_sk = s.ss_item_sk
                                                                                
              AND t.ss_customer_sk=s.ss_customer_sk
                                                                                
              AND t.ss_sold_date_sk = "2451181"
                                                                                
              AND ((Floor((s.ss_item_sk) / 1000) * 1000) BETWEEN 1000 AND 2000)
                                                                                
              AND s.ss_ext_discount_amt < 0.0) WHEN matched
AND t.ss_ext_discount_amt IS NULL THEN
UPDATE
SET ss_ext_discount_amt = 0.0 WHEN NOT matched THEN
INSERT (ss_sold_time_sk,
        ss_item_sk,
        ss_customer_sk,
        ss_cdemo_sk,
        ss_hdemo_sk,
        ss_addr_sk,
        ss_store_sk,
        ss_promo_sk,
        ss_ticket_number,
        ss_quantity,
        ss_wholesale_cost,
        ss_list_price,
        ss_sales_price,
        ss_ext_discount_amt,
        ss_ext_sales_price,
        ss_ext_wholesale_cost,
        ss_ext_list_price,
        ss_ext_tax,
        ss_coupon_amt,
        ss_net_paid,
        ss_net_paid_inc_tax,
        ss_net_profit,
        ss_sold_date_sk)
VALUES (s.ss_sold_time_sk,
        s.ss_item_sk,
        s.ss_customer_sk,
        s.ss_cdemo_sk,
        s.ss_hdemo_sk,
        s.ss_addr_sk,
        s.ss_store_sk,
        s.ss_promo_sk,
        s.ss_ticket_number,
        s.ss_quantity,
        s.ss_wholesale_cost,
        s.ss_list_price,
        s.ss_sales_price,
        s.ss_ext_discount_amt,
        s.ss_ext_sales_price,
        s.ss_ext_wholesale_cost,
        s.ss_ext_list_price,
        s.ss_ext_tax,
        s.ss_coupon_amt,
        s.ss_net_paid,
        s.ss_net_paid_inc_tax,
        s.ss_net_profit,
        "2451181")

 {noformat}
 

 

*Issue:*
 # Map phase is not getting vectorized due to "PARTITION_{_}SPEC{_}_ID" column

{noformat}
Map notVectorizedReason: Select expression for SELECT operator: Virtual column 
PARTITION__SPEC__ID is not supported {noformat}
 

2. "Reducer 2" stage isn't vectorized. 
{noformat}
Reduce notVectorizedReason: exception: java.lang.RuntimeException: Full Outer 
Small Table Key Mapping duplicate column 0 in ordered column map {0=(value 
column: 30, type info: int), 1=(value column: 31, type info: int)} when adding 
value column 53, type into int stack trace: 
org.apache.hadoop.hive.ql.exec.vector.VectorColumnOrderedMap.add(VectorColumnOrderedMap.java:102),
 
org.apache.hadoop.hive.ql.exec.vector.VectorColumnSourceMapping.add(VectorColumnSourceMapping.java:41),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.canSpecializeMapJoin(Vectorizer.java:3865),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperator(Vectorizer.java:5246),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChild(Vectorizer.java:988),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.doProcessChildren(Vectorizer.java:874),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.validateAndVectorizeOperatorTree(Vectorizer.java:841),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer.access$2400(Vectorizer.java:251),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2298),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceOperators(Vectorizer.java:2246),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.validateAndVectorizeReduceWork(Vectorizer.java:2224),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.convertReduceWork(Vectorizer.java:2206),
 
org.apache.hadoop.hive.ql.optimizer.physical.Vectorizer$VectorizationDispatcher.dispatch(Vectorizer.java:1038),
 
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111),
 org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180), 
... {noformat}
 

I have attached the explain plan for this, which has details on this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26978) Stale "Runtime stats" causes poor query planning

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26978:
---

 Summary: Stale "Runtime stats" causes poor query planning
 Key: HIVE-26978
 URL: https://issues.apache.org/jira/browse/HIVE-26978
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2023-01-24 at 10.23.16 AM.png

* Runtime stats can be stored in hiveserver or in metastore via 
"hive.query.reexecution.stats.persist.scope".
 * Though the table is dropped and recreated, it ends up showing old stats via 
"RUNTIME" stats. Here is an example (note that the table is empty, but gets 
datasize and numRows from RUNTIME stats)
 * This causes suboptimal plan for "MERGE INTO" queries by creating CUSTOM_EDGE 
instead of broadcast edge.

!Screenshot 2023-01-24 at 10.23.16 AM.png|width=2053,height=753!

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26975) MERGE: Wrong reducer estimate causing smaller files to be created

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26975:
---

 Summary: MERGE: Wrong reducer estimate causing smaller files to be 
created
 Key: HIVE-26975
 URL: https://issues.apache.org/jira/browse/HIVE-26975
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


* "Merge into" estimates wrong number of reducers causing more number of small 
files to be created.* e.g 400+ files in 3+ MB file each.*
 * This can be reproduced by writing data into "store_sales" table in iceberg 
format via another source table (using merge-into).
 ** e.g  Running this few times will create wrong number of reduce tasks 
causing lot of small files to be created in iceberg table.

{noformat}
MERGE INTO store_sales_t t

using ssv s

ON ( t.ss_item_sk = s.ss_item_sk

 AND t.ss_customer_sk = s.ss_customer_sk

 AND t.ss_sold_date_sk = "2451181"

 AND ( ( Floor(( s.ss_item_sk ) / 1000) * 1000 ) BETWEEN 1000 AND 2000 )

 AND s.ss_ext_discount_amt < 0.0 )

WHEN matched AND t.ss_ext_discount_amt IS NULL THEN

  UPDATE SET ss_ext_discount_amt = 0.0

WHEN NOT matched THEN

  INSERT ( ss_sold_time_sk,

   ss_item_sk,

   ss_customer_sk,

   ss_cdemo_sk,

   ss_hdemo_sk,

   ss_addr_sk,

   ss_store_sk,

   ss_promo_sk,

   ss_ticket_number,

   ss_quantity,

   ss_wholesale_cost,

   ss_list_price,

   ss_sales_price,

   ss_ext_discount_amt,

   ss_ext_sales_price,

   ss_ext_wholesale_cost,

   ss_ext_list_price,

   ss_ext_tax,

   ss_coupon_amt,

   ss_net_paid,

   ss_net_paid_inc_tax,

   ss_net_profit,

   ss_sold_date_sk )

  VALUES ( s.ss_sold_time_sk,

   s.ss_item_sk,

   s.ss_customer_sk,

   s.ss_cdemo_sk,

   s.ss_hdemo_sk,

   s.ss_addr_sk,

   s.ss_store_sk,

   s.ss_promo_sk,

   s.ss_ticket_number,

   s.ss_quantity,

   s.ss_wholesale_cost,

   s.ss_list_price,

   s.ss_sales_price,

   s.ss_ext_discount_amt,

   s.ss_ext_sales_price,

   s.ss_ext_wholesale_cost,

   s.ss_ext_list_price,

   s.ss_ext_tax,

   s.ss_coupon_amt,

   s.ss_net_paid,

   s.ss_net_paid_inc_tax,

   s.ss_net_profit,

   "2451181") 

{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26974) CTL from iceberg table should copy partition fields correctly

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26974:
---

 Summary: CTL from iceberg table should copy partition fields 
correctly
 Key: HIVE-26974
 URL: https://issues.apache.org/jira/browse/HIVE-26974
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


# Create iceberg table. Ensure it to have a partition field.
 # run "create external table like x"
 # Created table in #2 misses out on creating relevant partition field.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26951) Setting details in PositionDeleteInfo takes up lot of CPU cycles

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26951:
---

 Summary: Setting details in PositionDeleteInfo takes up lot of CPU 
cycles
 Key: HIVE-26951
 URL: https://issues.apache.org/jira/browse/HIVE-26951
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2023-01-17 at 11.29.29 AM.png, Screenshot 
2023-01-17 at 11.29.36 AM.png

!Screenshot 2023-01-17 at 11.29.29 AM.png|width=898,height=532!

 

 

!Screenshot 2023-01-17 at 11.29.36 AM.png|width=1000,height=591!

 

This was observed with merge-into statements.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26950) (CTLT) Create external table like V2 table is not preserving table properties

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26950:
---

 Summary: (CTLT) Create external table like V2 table is not 
preserving table properties
 Key: HIVE-26950
 URL: https://issues.apache.org/jira/browse/HIVE-26950
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


# Create an external iceberg V2 table. e.g t1
 # "create external table t2 like t1" <--- This ends up creating V1 table and 
"format-version=2" is not retained and "'format'='iceberg/parquet'" is also not 
retained.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26944) FileSinkOperator shouldn't check for compactiontable for every row being processed

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26944:
---

 Summary: FileSinkOperator shouldn't check for compactiontable for 
every row being processed
 Key: HIVE-26944
 URL: https://issues.apache.org/jira/browse/HIVE-26944
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2023-01-16 at 10.32.24 AM.png





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26928) LlapIoImpl::getParquetFooterBuffersFromCache throws exception when metadata cache is disabled

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26928:
---

 Summary: LlapIoImpl::getParquetFooterBuffersFromCache throws 
exception when metadata cache is disabled
 Key: HIVE-26928
 URL: https://issues.apache.org/jira/browse/HIVE-26928
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


When metadata / LLAP cache is disabled, "iceberg + parquet" throws the 
following error.

It should check for "metadatacache" correctly or fix it in LlapIoImpl.

 
{noformat}

Caused by: java.lang.NullPointerException: Metadata cache must not be null
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:897)
    at 
org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl.getParquetFooterBuffersFromCache(LlapIoImpl.java:467)
    at 
org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.parquetRecordReader(HiveVectorizedReader.java:227)
    at 
org.apache.iceberg.mr.hive.vector.HiveVectorizedReader.reader(HiveVectorizedReader.java:162)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
    at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at 
org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:65)
    at 
org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:77)
    at 
org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:196)
    at 
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.openVectorized(IcebergInputFormat.java:331)
    at 
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.open(IcebergInputFormat.java:377)
    at 
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextTask(IcebergInputFormat.java:270)
    at 
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.initialize(IcebergInputFormat.java:266)
    at 
org.apache.iceberg.mr.mapred.AbstractMapredIcebergRecordReader.(AbstractMapredIcebergRecordReader.java:40)
    at 
org.apache.iceberg.mr.hive.vector.HiveIcebergVectorizedRecordReader.(HiveIcebergVectorizedRecordReader.java:41)
 {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26927) Iceberg: Add support for set_current_snapshotid

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26927:
---

 Summary: Iceberg: Add support for set_current_snapshotid
 Key: HIVE-26927
 URL: https://issues.apache.org/jira/browse/HIVE-26927
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


Currently, hive supports "rollback" feature. Once rolledback,  it is not 
possible to move from older snapshot to newer snapshot.

It ends up throwing 
{color:#0747a6}"org.apache.iceberg.exceptions.ValidationException: Cannot roll 
back to snapshot, not an ancestor of the current state:" {color}error.

It will be good to support "set_current_snapshot" function to move to different 
snapshot ids.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26917) Upgrade parquet to 1.12.3

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26917:
---

 Summary: Upgrade parquet to 1.12.3
 Key: HIVE-26917
 URL: https://issues.apache.org/jira/browse/HIVE-26917
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26913) HiveVectorizedReader::parquetRecordReader should reuse footer information

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26913:
---

 Summary: HiveVectorizedReader::parquetRecordReader should reuse 
footer information
 Key: HIVE-26913
 URL: https://issues.apache.org/jira/browse/HIVE-26913
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan


HiveVectorizedReader::parquetRecordReader should reuse details of parquet 
footer, instead of reading it again.

 

It reads parquet footer here:

[https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java#L230-L232]

Again it reads the footer here for constructing vectorized recordreader

[https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/vector/HiveVectorizedReader.java#L249]

 

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java#L50]

 

Check the codepath of 
VectorizedParquetRecordReader::setupMetadataAndParquetSplit

[https://github.com/apache/hive/blob/6b0139188aba6a95808c8d1bec63a651ec9e4bdc/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L180]

 

It should be possible to share "ParquetMetadata" in 
VectorizedParuqetRecordReader.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26874) Iceberg: Positional delete files are not cached

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26874:
---

 Summary: Iceberg: Positional delete files are not cached 
 Key: HIVE-26874
 URL: https://issues.apache.org/jira/browse/HIVE-26874
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


With iceberg v2 (MOR mode), "positional delete" files are not cached causing 
runtime delays.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26714) Iceberg delete files are read twice during query processing causing delays

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26714:
---

 Summary: Iceberg delete files are read twice during query 
processing causing delays
 Key: HIVE-26714
 URL: https://issues.apache.org/jira/browse/HIVE-26714
 Project: Hive
  Issue Type: Improvement
  Components: Iceberg integration
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2022-11-08 at 9.37.17 PM.png

Delete positions are read twice in query processing causing delays in runtime.

!Screenshot 2022-11-08 at 9.37.17 PM.png|width=707,height=629!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26699:
---

 Summary: Iceberg: S3 fadvise can hurt JSON parsing significantly 
in DWX
 Key: HIVE-26699
 URL: https://issues.apache.org/jira/browse/HIVE-26699
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Hive reads JSON metadata information (TableMetadataParser::read()) multiple 
times; E.g during query compilation, AM split computation, stats computation, 
during commits  etc.

 

With large JSON files (due to multiple inserts), it takes a lot longer time 
with S3 FS with "fs.s3a.experimental.input.fadvise" set to "random". (e.g in 
the order of 10x).To be on safer side, it will be good to set this to "normal" 
mode in configs, when reading iceberg tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26686) Iceberg: Having lot of snapshots impacts runtime due to multiple loads of the table

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26686:
---

 Summary: Iceberg: Having lot of snapshots impacts runtime due to 
multiple loads of the table
 Key: HIVE-26686
 URL: https://issues.apache.org/jira/browse/HIVE-26686
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


When large number of snpashots are present in manifest file, it adversely 
impacts the runtime of the queries. (e.g 15 mts trickle feed).

Having more snapshots will slow down runtime in 2 additional places.

1. At the time of populating statistics, it tries to load the table details 
again. i.e refresh table invocation
2. At the time of hive metastore hook (HiveIcebergMetaHook::doPreAlterTable), 
during pre alter table.

Need to check if entire table information along with snapshot details are 
needed for this.

{noformat}
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:437)
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:261)
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218)
    at 
org.apache.hive.iceberg.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3251)
    at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:264)
    at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:177)
    at 
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$685/0x000840e1b440.apply(Unknown
 Source)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:191)
    at 
org.apache.iceberg.BaseMetastoreTableOperations$$Lambda$686/0x000840e1a840.run(Unknown
 Source)
    at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
    at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:191)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
    at 
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:153)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)
    at 
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)
    at 
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)
    at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:116)
    at org.apache.iceberg.mr.Catalogs.loadTable(Catalogs.java:106)
    at 
org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.getBasicStatistics(HiveIcebergStorageHandler.java:309)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsTask$BasicStatsProcessor.(BasicStatsTask.java:138)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsTask.aggregateStats(BasicStatsTask.java:301)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsTask.process(BasicStatsTask.java:108)
    at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
    at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
    at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:360)
    at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:333)
    at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:250)
    at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:111)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:806)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:540)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:534)
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
    at 
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:232)
    at 
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:89)
    at 
org.apache.hive.service.cli.operation.SQLOperation

Re: TPCDS query degrade with hive-3.1.2 because of wrong estimation for reducers

[Rajesh Balamohan]

Based on the plan, filtered output in map-1 had mis-estimates and also
groupby operators have large misestimates.

This is causing the number of reducers to be estimated as "4" which is less
for this query. Due to the partition factor of tez, it ends up with 8
reducer slots at runtime for hive 3.x.

Here are a few tickets which can help, but note that it is *very risky* to
backport pointed patches for stats and CBO without complete context. This
may have an adverse impact on other queries.

https://issues.apache.org/jira/browse/HIVE-23684
https://issues.apache.org/jira/browse/HIVE-20432
https://issues.apache.org/jira/browse/HIVE-23485

On Sun, Oct 2, 2022 at 1:56 PM Battula, Brahma Reddy 
wrote:

> + Attaching the hs2 logs also.
>
>
>
> *From: *"Battula, Brahma Reddy" 
> *Date: *Sunday, 2 October 2022 at 2:16 AM
> *To: *"u...@hive.apache.org" 
> *Subject: *TPCDS query degrade with hive-3.1.2 because of wrong
> estimation for reducers
>
>
>
> Hi All,
>
>
>
> We’ve ran TPCDS queries against hive-3.1.2 and trunk(little older
> version). (Attached files suffix “a” is trunk and “v” is 3.1.2)
>
>
>
> The query execution time is higher in hive-3.1.2 as number of the reducers
> estimated is less (8) as compared to trunk version where it’s 46.
>
>
>
> All the hive/tez/Yarn configs are same in both clusters. Even h/w
> resources are same. And query planner is also same.
>
>
>
> *The stats in reduce sink phase are not look same.*
>
>
>
> *HIVE_TRUNK_CODE* - 2022-09-26T05:58:23,786 INFO
> [07243354-f941-419d-8908-45009762e67d HiveServer2-Handler-Pool:
> Thread-168]: optimizer.ConvertJoinMapJoin (:()) - Join input#1;
> onlineDataSize:   9628; Statistics: Num rows:  359 Data size: 4308  Basic
> stats: COMPLETE Column stats: COMPLETE
>
> *HIVE_3.1.2_CODE* - 2022-09-27T03:39:45,116 INFO
> [2fd1493c-f1a0-4874-acac-58f28e9c21ea HiveServer2-Handler-Pool:
> Thread-134]: optimizer.ConvertJoinMapJoin (:()) - Join input#1;
> onlineDataSize: 325856; Statistics: Num rows: 8116 Data size: 97392 Basic
> stats: COMPLETE Column stats: COMPLETE
>
>
>
> Any idea how the reducers getting underestimated.?
>
>
>
>
>
>
>
>
>

[jira] [Created] (HIVE-26540) Iceberg: Select queries after update/delete become expensive in reading contents

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26540:
---

 Summary: Iceberg: Select queries after update/delete become  
expensive in reading contents
 Key: HIVE-26540
 URL: https://issues.apache.org/jira/browse/HIVE-26540
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


- Create basic date_dim table in tpcds. Store it in iceberg v2 format
- Update few 1000 records couple of times
- Run a simple select query {{select count ( * ) from date_dim_ice where d_qoy 
= 11 and d_dom=2 and d_fy_week_seq=3;}}

This takes 8-18 seconds where ACID takes 1.5 seconds.

Basic issue is that, it reads files multiple times (i.e both data and delete 
files).

Lines of interest:

IcebergInputFormat.java

{noformat}
   InternalRecordWrapper wrapper = new 
InternalRecordWrapper(readSchema.asStruct());
Evaluator filter = new Evaluator(readSchema.asStruct(), residual, 
caseSensitive);
return CloseableIterable.filter(iter, record -> 
filter.eval(wrapper.wrap((StructLike) record)));
{noformat}



{noformat}
   case GENERIC:
  DeleteFilter deletes = new GenericDeleteFilter(table.io(), 
currentTask, table.schema(), readSchema);
  Schema requiredSchema = deletes.requiredSchema();
  return deletes.filter(openGeneric(currentTask, requiredSchema));
{noformat}

These get evaluated for each row in the data file, causing delay.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26532) Remove logger from critical path in VectorMapJoinInnerLongOperator::processBatch

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26532:
---

 Summary: Remove logger from critical path in 
VectorMapJoinInnerLongOperator::processBatch
 Key: HIVE-26532
 URL: https://issues.apache.org/jira/browse/HIVE-26532
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2022-09-12 at 10.03.43 AM.png

!Screenshot 2022-09-12 at 10.03.43 AM.png|width=895,height=872!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26529) Fix VectorizedSupport support for DECIMAL_64 in HiveIcebergInputFormat

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26529:
---

 Summary: Fix VectorizedSupport support for  DECIMAL_64 in 
HiveIcebergInputFormat
 Key: HIVE-26529
 URL: https://issues.apache.org/jira/browse/HIVE-26529
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


For supporting vectored reads in parquet, DECIMAL_64 support in ORC has been 
disabled in HiveIcebergInputFormat. This causes regressions in queries.

[https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergInputFormat.java#L182]

It will be good to restore DECIMAL_64 support in iceberg input format.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26520) Improve dynamic partition pruning operator when subqueries are involved

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26520:
---

 Summary: Improve dynamic partition pruning operator when 
subqueries are involved
 Key: HIVE-26520
 URL: https://issues.apache.org/jira/browse/HIVE-26520
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan
 Attachments: q58_test.pdf

Dynamic partition pruning operator sends entire date_dim table and due to this, 
entire catalog_sales data is scanned causing huge IO and decoding cost.

If dynamic partition pruning operator was created after the "date_dim" subquery 
has been evaluated, it would have saved huge IO cost. E.g It would have just 
taken 6-7 partition scans instead of 1800+ partitions.

 

Consider the following simplified query as example

{noformat}

select count(*) from (select i_item_id item_id
,sum(cs_ext_sales_price) cs_item_rev
  from catalog_sales
  ,item
  ,date_dim
 where cs_item_sk = i_item_sk
  and  d_date in (select d_date
  from date_dim
  where d_week_seq = (select d_week_seq 
  from date_dim
  where d_date = '1998-02-21'))
  and  cs_sold_date_sk = d_date_sk
 group by i_item_id) a;
 

CBO PLAN:
HiveAggregate(group=[{}], agg#0=[count()])
  HiveProject(i_item_id=[$0])
HiveAggregate(group=[{4}])
  HiveSemiJoin(condition=[=($6, $7)], joinType=[semi])
HiveJoin(condition=[=($2, $5)], joinType=[inner])
  HiveJoin(condition=[=($0, $3)], joinType=[inner])
HiveProject(cs_item_sk=[$14], cs_ext_sales_price=[$22], 
cs_sold_date_sk=[$33])
  HiveFilter(condition=[AND(IS NOT NULL($33), IS NOT NULL($14))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, 
catalog_sales]], table:alias=[catalog_sales])
HiveProject(i_item_sk=[$0], i_item_id=[$1])
  HiveFilter(condition=[IS NOT NULL($0)])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, 
item]], table:alias=[item])
  HiveProject(d_date_sk=[$0], d_date=[$2])
HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($0))])
  HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, 
date_dim]], table:alias=[date_dim])
HiveProject(d_date=[$0])
  HiveJoin(condition=[=($1, $3)], joinType=[inner])
HiveJoin(condition=[true], joinType=[inner])
  HiveProject(d_date=[$2], d_week_seq=[$4])
HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($4))])
  
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, date_dim]], 
table:alias=[date_dim])
  HiveProject(cnt=[$0])
HiveFilter(condition=[<=(sq_count_check($0), 1)])
  HiveProject(cnt=[$0])
HiveAggregate(group=[{}], cnt=[COUNT()])
  HiveFilter(condition=[=($2, 1998-02-21)])

HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, date_dim]], 
table:alias=[date_dim])
HiveProject(d_week_seq=[$4])
  HiveFilter(condition=[AND(=($2, 1998-02-21), IS NOT NULL($4))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1_external, 
date_dim]], table:alias=[date_dim])
{noformat}
 
I will attach the formatted plan for reference as well. If planner generated 
the dynamic partition pruning event after "date_dim" got evaluated in "Map 7", 
it would be been very efficient.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: FYI: MetaStore running out of threads

[Rajesh Balamohan]

>> In hive, FileUtils.checkFileAccessWithImpersonation can be fixed to use
create UGI once to reduce the impact (suspecting this will have 50%
impact).

Looked closely at the method impl for
"FileUtils.checkFileAccessWithImpersonation". It doesn't make 2
connections; 50% impact may not be relevant here.

On Thu, Sep 1, 2022 at 4:48 AM Rajesh Balamohan 
wrote:

>
> W.r.t to connection reuse issues, LLAP had a similar issue (not in HMS)
> https://issues.apache.org/jira/browse/HIVE-16020. It was making a
> connection in every task and UGI had to be persisted in the QueryInfo level
> to reduce the impact.
>
> In hive, FileUtils.checkFileAccessWithImpersonation can be fixed to use
> create UGI once to reduce the impact (suspecting this will have 50%
> impact).
>
>
> https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L418
>
> https://github.com/apache/hive/blob/d06957f254e026e719f30027d161264be43386b0/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L461
>
> May have to explore whether a local cache with expiry in FileUtils can
> help reduce the impact further.
>
> ~Rajesh.B
>
>
> On Thu, Sep 1, 2022 at 1:24 AM Owen O'Malley 
> wrote:
>
>> We're using HMS with Storage-Based Authorization and have been having
>> trouble with the HMS running out of threads. Looking at the jstack & code,
>> it appears to that the problem is that RPC's ConnectionId is using UGI's
>> equal/hash, which uses the Subject's Object equals/hash. Proxy user UGI's
>> always create a new Subject and thus are always unique.
>>
>> This leads to the HMS creating too many threads. I've created a jira in
>> Hadoop. https://issues.apache.org/jira/browse/HADOOP-18434
>>
>> Thanks,
>>Owen
>>
>

Re: FYI: MetaStore running out of threads

[Rajesh Balamohan]

W.r.t to connection reuse issues, LLAP had a similar issue (not in HMS)
https://issues.apache.org/jira/browse/HIVE-16020. It was making a
connection in every task and UGI had to be persisted in the QueryInfo level
to reduce the impact.

In hive, FileUtils.checkFileAccessWithImpersonation can be fixed to use
create UGI once to reduce the impact (suspecting this will have 50%
impact).

https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L418
https://github.com/apache/hive/blob/d06957f254e026e719f30027d161264be43386b0/common/src/java/org/apache/hadoop/hive/common/FileUtils.java#L461

May have to explore whether a local cache with expiry in FileUtils can help
reduce the impact further.

~Rajesh.B


On Thu, Sep 1, 2022 at 1:24 AM Owen O'Malley  wrote:

> We're using HMS with Storage-Based Authorization and have been having
> trouble with the HMS running out of threads. Looking at the jstack & code,
> it appears to that the problem is that RPC's ConnectionId is using UGI's
> equal/hash, which uses the Subject's Object equals/hash. Proxy user UGI's
> always create a new Subject and thus are always unique.
>
> This leads to the HMS creating too many threads. I've created a jira in
> Hadoop. https://issues.apache.org/jira/browse/HADOOP-18434
>
> Thanks,
>Owen
>

[jira] [Created] (HIVE-26507) Iceberg: In place metadata generation may not work for certain datatypes

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26507:
---

 Summary: Iceberg: In place metadata generation may not work for 
certain datatypes
 Key: HIVE-26507
 URL: https://issues.apache.org/jira/browse/HIVE-26507
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


"alter table" statements can be used for generating iceberg metadata 
information (i.e for converting external tables  -> iceberg tables).

As part of this process, it also converts certain datatypes to iceberg 
compatible types (e.g char -> string). "iceberg.mr.schema.auto.conversion" 
enables this conversion.

This could cause certain issues at runtime. Here is an example
{noformat}

Before conversion:
==
-- external table
select count(*) from customer_demographics where cd_gender = 'F' and 
cd_marital_status = 'U' and cd_education_status = '2 yr Degree';

27440

after conversion:
=
-- iceberg table
select count(*) from customer_demographics where cd_gender = 'F' and 
cd_marital_status = 'U' and cd_education_status = '2 yr Degree';

0

select count(*) from customer_demographics where cd_gender = 'F' and 
cd_marital_status = 'U' and trim(cd_education_status) = '2 yr Degree';

27440
 {noformat}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26496) FetchOperator scans delete_delta folders multiple times causing slowness

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26496:
---

 Summary: FetchOperator scans delete_delta folders multiple times 
causing slowness
 Key: HIVE-26496
 URL: https://issues.apache.org/jira/browse/HIVE-26496
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Rajesh Balamohan


FetchOperator scans way too many number of files/directories than needed.

For e.g here is a layout of a table which had set of updates and deletes. There 
are set of "delta" and "delete_delta" folders which are created.
{noformat}
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/base_001
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_002_002_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_003_003_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_004_004_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_005_005_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_006_006_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_007_007_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_008_008_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_009_009_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_010_010_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_011_011_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_012_012_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_013_013_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_014_014_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_015_015_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_016_016_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_017_017_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_018_018_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_019_019_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_020_020_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_021_021_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delete_delta_022_022_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_002_002_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_003_003_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_004_004_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_005_005_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_006_006_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_007_007_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_008_008_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_009_009_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_010_010_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_011_011_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_012_012_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_013_013_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_014_014_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_015_015_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_016_016_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_017_017_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_018_018_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_019_019_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_020_020_
s3a://bucket-name/warehouse/tablespace/managed/hive/test.db/date_dim/delta_021_021_

{noformat}
 

When user runs *{color:#0747a6}{{select * from date_dim}}{color}* from beeline, 
FetchOperator tries to compute splits in "date_dim". This "base

[jira] [Created] (HIVE-26491) Iceberg: Drop table should purge the data for V2 tables

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26491:
---

 Summary: Iceberg: Drop table should purge the data for V2 tables 
 Key: HIVE-26491
 URL: https://issues.apache.org/jira/browse/HIVE-26491
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


# create external table stored by iceberg in orc format. Convert this to 
iceberg v2 table format via alter table statements. This should ideally have 
set "'external.table.purge'='true'" property by default which is missing for V2 
tables.
 # insert data into it
 # Drop the table.  This drops the metadata information, but retains actual 
data. 

Set "'external.table.purge'='true'" as default for iceberg (if it hasn't been 
set yet).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26490) Iceberg: Residual expression is constructed for the task from multiple places causing CPU burn

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26490:
---

 Summary: Iceberg: Residual expression is constructed for the task 
from multiple places causing CPU burn
 Key: HIVE-26490
 URL: https://issues.apache.org/jira/browse/HIVE-26490
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2022-08-22 at 12.58.47 PM.jpg

"HiveIcebergInputFormat.residualForTask(task, job)" is invoked from multiple 
places causing CPU burn.

!Screenshot 2022-08-22 at 12.58.47 PM.jpg|width=918,height=932!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26194) Unable to interrupt query in the middle of long compilation

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26194:
---

 Summary: Unable to interrupt query in the middle of long 
compilation
 Key: HIVE-26194
 URL: https://issues.apache.org/jira/browse/HIVE-26194
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Rajesh Balamohan



*Issue:*
 * Certain queries can take lot longer time to compile, depending on the number 
of interactions with HMS.
 * When user tries to cancel such queries in the middle of compilation, it 
doesn't work. It interrupts the process only when the entire compilation phase 
is complete.
 * Example is given below (Q66 at 10 TB TPCDS)

 
{noformat}

. . . . . . . . . . . . . . . . . . . . . . .>,d_year
. . . . . . . . . . . . . . . . . . . . . . .>  )
. . . . . . . . . . . . . . . . . . . . . . .>  ) x
. . . . . . . . . . . . . . . . . . . . . . .>  group by
. . . . . . . . . . . . . . . . . . . . . . .> w_warehouse_name
. . . . . . . . . . . . . . . . . . . . . . .>,w_warehouse_sq_ft
. . . . . . . . . . . . . . . . . . . . . . .>,w_city
. . . . . . . . . . . . . . . . . . . . . . .>,w_county
. . . . . . . . . . . . . . . . . . . . . . .>,w_state
. . . . . . . . . . . . . . . . . . . . . . .>,w_country
. . . . . . . . . . . . . . . . . . . . . . .>,ship_carriers
. . . . . . . . . . . . . . . . . . . . . . .>,year
. . . . . . . . . . . . . . . . . . . . . . .>  order by w_warehouse_name
. . . . . . . . . . . . . . . . . . . . . . .>  limit 100;
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.

...
...
...

,w_city
,w_county
,w_state
,w_country
,ship_carriers
,year
order by w_warehouse_name
limit 100
INFO  : Semantic Analysis Completed (retrial = false)
ERROR : FAILED: command has been interrupted: after analyzing query.
INFO  : Compiling 
command(queryId=hive_20220502040541_14c76b6f-f6d2-4ab3-ad82-522f17ede63a) has 
been interrupted after 32.872 seconds <<<<<<<<<<< Notice that 
it interrupted only after entire compilation is done at 32 seconds.
Error: Query was cancelled. Illegal Operation state transition from CANCELED to 
ERROR (state=01000,code=0)

 {noformat}

This becomes an issue in busy cluster.

Interrupt handling should be fixed in compilation phase.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HIVE-26185) Need support for metadataonly operations with iceberg (e.g select distinct on partition column)

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26185:
---

 Summary: Need support for metadataonly operations with iceberg 
(e.g select distinct on partition column)
 Key: HIVE-26185
 URL: https://issues.apache.org/jira/browse/HIVE-26185
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Rajesh Balamohan


{noformat}
select distinct ss_sold_date_sk from store_sales
{noformat}

This query scans 1800+ rows in hive acid. But takes ages to process with 
NullScanOptimiser during compilation phase 
(https://issues.apache.org/jira/browse/HIVE-24262)

{noformat}
Hive ACID

INFO  : Executing 
command(queryId=hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14): 
select distinct ss_sold_date_sk from store_sales
INFO  : Compute 'ndembla-test2' is active.
INFO  : Query ID = hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Subscribed to counters: [] for queryId: 
hive_20220427233926_282bc9d8-220c-4a09-928d-411601c2ef14
INFO  : Tez session hasn't been created yet. Opening session
INFO  : Dag name: select distinct ss_sold_date_s...store_sales (Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1651102345385_)

INFO  : Status: DAG finished successfully in 1.81 seconds
INFO  : DAG ID: dag_1651102345385__5
INFO  :
INFO  : Query Execution Summary
INFO  : 
--
INFO  : OPERATIONDURATION
INFO  : 
--
INFO  : Compile Query  55.47s
INFO  : Prepare Plan2.32s
INFO  : Get Query Coordinator (AM)  0.13s
INFO  : Submit Plan 0.03s
INFO  : Start DAG   0.09s
INFO  : Run DAG 1.80s
INFO  : 
--
INFO  :
INFO  : Task Execution Summary
INFO  : 
--
INFO  :   VERTICES  DURATION(ms)   CPU_TIME(ms)GC_TIME(ms)   
INPUT_RECORDS   OUTPUT_RECORDS
INFO  : 
--
INFO  :  Map 1   1009.00  0  0   
1,8241,824
INFO  :  Reducer 2  0.00  0  0   
1,8240
INFO  : 
--
INFO  :

{noformat}




However, same query scans *2.8 Billion records.* in iceberg format. This can be 
fixed.

{noformat}

INFO  : Executing 
command(queryId=hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72): 
select distinct ss_sold_date_sk from store_sales
INFO  : Compute 'ndembla-test2' is active.
INFO  : Query ID = hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Subscribed to counters: [] for queryId: 
hive_20220427233519_cddc6dd1-95a3-4f0e-afa5-e11e9dc5fa72
INFO  : Tez session hasn't been created yet. Opening session
INFO  : Dag name: select distinct ss_sold_date_s...store_sales (Stage-1)
INFO  : Status: Running (Executing on YARN cluster with App id 
application_1651102345385_)

--
VERTICES  MODESTATUS  TOTAL  COMPLETED  RUNNING  PENDING  
FAILED  KILLED
--
Map 1 ..  llap SUCCEEDED   7141   714100
   0   0
Reducer 2 ..  llap SUCCEEDED  2  200
   0   0
--
VERTICES: 02/02  [==>>] 100%  ELAPSED TIME: 18.48 s
--
INFO  : Status: DAG finished successfully in 17.97 seconds
INFO  : DAG ID: dag_1651102345385__4
INFO  :
INFO  : Query Execution Summary
INFO  : 
--
INFO  : OPERATIONDURATION
INFO  : 
--
INFO  : Compile Query   1.81s
INFO  : Prepare Plan   

[jira] [Created] (HIVE-26181) Add details on the number of partitions/entries in dynamic partition pruning

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26181:
---

 Summary: Add details on the number of partitions/entries in 
dynamic partition pruning
 Key: HIVE-26181
 URL: https://issues.apache.org/jira/browse/HIVE-26181
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


Related ticket: HIVE-26008

It will be good to print details on the number of partition pruning entries for 
debugging and for understanding the eff* of the query.





--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HIVE-26128) Enabling dynamic runtime filtering in iceberg tables throws exception at runtime

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26128:
---

 Summary: Enabling dynamic runtime filtering in iceberg tables 
throws exception at runtime
 Key: HIVE-26128
 URL: https://issues.apache.org/jira/browse/HIVE-26128
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


E.g TPCDS Q2 at 10 TB scale throws the following error when run with 
"hive.disable.unsafe.external.table.operations=false". Iceberg tables were 
created as external tables and setting 
"hive.disable.unsafe.external.table.operations=false" will enable it to have 
dynamic runtime filtering; but throws  the following error at runtime


{noformat}
]Vertex failed, vertexName=Map 6, vertexId=vertex_1649658279052__1_03, 
diagnostics=[Vertex vertex_1649658279052__1_03 [Map 6] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: date_dim initializer failed, 
vertex=vertex_1649658279052__1_03 [Map 6], 
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at 
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translateLeaf(HiveIcebergFilterFactory.java:114)
at 
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:86)
at 
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.translate(HiveIcebergFilterFactory.java:80)
at 
org.apache.iceberg.mr.hive.HiveIcebergFilterFactory.generateFilterExpression(HiveIcebergFilterFactory.java:59)
at 
org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getSplits(HiveIcebergInputFormat.java:92)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:592)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:900)
at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:274)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$runInitializer$3(RootInputInitializerManager.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializer(RootInputInitializerManager.java:192)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.runInitializerAndProcessResult(RootInputInitializerManager.java:173)
at 
org.apache.tez.dag.app.dag.RootInputInitializerManager.lambda$createAndStartInitializing$2(RootInputInitializerManager.java:167)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
]Vertex killed, vertexName=Map 13, vertexId=vertex_1649658279052__1_07, 
diagnostics=[Vertex received Kill in INITED state., Vertex 
vertex_1649658279052__1_07 [Map 13] killed/failed due 
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 10, 
vertexId=vertex_1649658279052__1_06, diagnostics=[Vertex received Kill in 
INITED state., Vertex vertex_1649658279052__1_06 [Map 10] killed/failed due 
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 5, 
vertexId=vertex_1649658279052__1_04, diagnostics=[Vertex received Kill in 
INITED state., Vertex vertex_1649658279052__1_04 [Map 5] killed/failed due 
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 4, 
vertexId=vertex_1649658279052__1_11, diagnostics=[Vertex received Kill in 
NEW state., Vertex vertex_1649658279052__1_11 [Reducer 4] killed/failed due 
to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3, 
vertexId=vertex_1649658279052__1_10, diagnostics=[Vertex received Kill in 
INITED state., Vertex vertex_1649658279052__1_10 [Reducer 3] killed/failed 
due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 12, 
vertexId=vertex_1649658279052__1_09, diagnostics=[Vertex received Kill in 
INITED state., Vertex vertex_1649658279052__1_09 [Reducer 12] killed/failed 
due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Map 1, 
vertexId=vertex_1649658279052__1_08, diagnostics=[Vertex received Kill in 
INITED state., Vertex vertex_1649658279052__1_08 [Map 

[jira] [Created] (HIVE-26115) Parquet footer is read 3 times when reading iceberg data

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26115:
---

 Summary: Parquet footer is read 3 times when reading iceberg data
 Key: HIVE-26115
 URL: https://issues.apache.org/jira/browse/HIVE-26115
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2022-04-05 at 10.08.27 AM.png, Screenshot 
2022-04-05 at 10.08.35 AM.png, Screenshot 2022-04-05 at 10.08.50 AM.png, 
Screenshot 2022-04-05 at 10.09.03 AM.png

!Screenshot 2022-04-05 at 10.08.27 AM.png|width=627,height=331!

Here is the breakup of 3 footer reads per file.

!Screenshot 2022-04-05 at 10.08.35 AM.png|width=1109,height=500! 

 

 

!Screenshot 2022-04-05 at 10.08.50 AM.png|width=1067,height=447! 

 

 

!Screenshot 2022-04-05 at 10.09.03 AM.png|width=827,height=303!

 

HIVE-25827 already talks about the initial 2 footer reads per file.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26110) bulk insert into partitioned table creates lots of files in iceberg

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26110:
---

 Summary: bulk insert into partitioned table creates lots of files 
in iceberg
 Key: HIVE-26110
 URL: https://issues.apache.org/jira/browse/HIVE-26110
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


For e.g, create web_returns table in tpcds in iceberg format and try to copy 
over data from regular table. More like "insert into web_returns_iceberg as 
select * from web_returns".

This inserts the data correctly, however there are lot of files present in each 
partition. IMO, dynamic sort optimisation isn't working fine and this causes 
records not to be grouped in the final phase.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26091) Support DecimalFilterPredicateLeafBuilder for parquet

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26091:
---

 Summary: Support DecimalFilterPredicateLeafBuilder for parquet
 Key: HIVE-26091
 URL: https://issues.apache.org/jira/browse/HIVE-26091
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java#L41

It will nice to have DecimalFilterPredicateLeafBuilder. This will help in 
supporting SARG pushdowns.

{noformat}
2022-03-30 08:59:50,040 [ERROR] [TezChild] 
|read.ParquetFilterPredicateConverter|: fail to build predicate filter leaf 
with errorsorg.apache.hadoop.hive.ql.metadata.HiveException: Conversion to 
Parquet FilterPredicate not supported for DECIMAL
org.apache.hadoop.hive.ql.metadata.HiveException: Conversion to Parquet 
FilterPredicate not supported for DECIMAL
at 
org.apache.hadoop.hive.ql.io.parquet.LeafFilterFactory.getLeafFilterBuilderByType(LeafFilterFactory.java:223)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.buildFilterPredicateFromPredicateLeaf(ParquetFilterPredicateConverter.java:130)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:111)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:97)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:71)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.translate(ParquetFilterPredicateConverter.java:88)
at 
org.apache.hadoop.hive.ql.io.parquet.read.ParquetFilterPredicateConverter.toFilterPredicate(ParquetFilterPredicateConverter.java:57)
at 
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.setFilter(ParquetRecordReaderBase.java:184)
at 
org.apache.hadoop.hive.ql.io.parquet.ParquetRecordReaderBase.getSplit(ParquetRecordReaderBase.java:124)
at 
org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.(VectorizedParquetRecordReader.java:158)
at 
org.apache.hadoop.hive.ql.io.parquet.VectorizedParquetInputFormat.getRecordReader(VectorizedParquetInputFormat.java:50)
at 
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:87)
at 
org.apache.hadoop.hive.ql.io.RecordReaderWrapper.create(RecordReaderWrapper.java:72)
at 
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:429)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:203)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:152)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:282)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:265)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
 {noformat}



--
This message was sent by Atlassian Jira

[jira] [Created] (HIVE-26072) Enable vectorization for stats gathering (tablescan op)

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26072:
---

 Summary: Enable vectorization for stats gathering (tablescan op)
 Key: HIVE-26072
 URL: https://issues.apache.org/jira/browse/HIVE-26072
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Rajesh Balamohan


https://issues.apache.org/jira/browse/HIVE-24510 enabled vectorization for 
compute_bit_vector. 

But tablescan operator for stats gathering is disabled by default.

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java#L2577]

Need to enable vectorization for this. This can significantly reduce runtimes 
for analyze statements for large tables.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26035) Move to directsql for ObjectStore::addPartitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26035:
---

 Summary: Move to directsql for ObjectStore::addPartitions
 Key: HIVE-26035
 URL: https://issues.apache.org/jira/browse/HIVE-26035
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


Currently {{addPartitions}} uses datanuclues and is super slow for large number 
of partitions. It will be good to move to direct sql. Lots of repeated SQLs can 
be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26013) Parquet predicate filters are not properly propogated to task configs at runtime

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26013:
---

 Summary: Parquet predicate filters are not properly propogated to 
task configs at runtime
 Key: HIVE-26013
 URL: https://issues.apache.org/jira/browse/HIVE-26013
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


Hive ParquetRecordReader sets the predicate filter in the config for parquet 
libs to read.

Ref: 
[https://github.com/apache/hive/blob/master/ql%2Fsrc%2Fjava%2Forg%2Fapache%2Fhadoop%2Fhive%2Fql%2Fio%2Fparquet%2FParquetRecordReaderBase.java#L188]
{code:java}
 ParquetInputFormat.setFilterPredicate(conf, p);
{code}
This internally sets {color:#FF}"parquet.private.read.filter.predicate" 
{color}variable in config.

Ref: 
[https://github.com/apache/parquet-mr/blob/master/parquet-hadoop%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fparquet%2Fhadoop%2FParquetInputFormat.java#L231]

Config set in compilation phase isn't visible at runtime for the tasks. This 
causes filters to be lost and tasks run with excessive IO.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26008) Dynamic partition pruning not sending right partitions with subqueries

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-26008:
---

 Summary: Dynamic partition pruning not sending right partitions 
with subqueries
 Key: HIVE-26008
 URL: https://issues.apache.org/jira/browse/HIVE-26008
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


DPP isn't working fine when there are subqueries involved. Here is an example 
query (q83). 

Note that "date_dim" has another query involved. Due to this, DPP operator ends 
up sending entire "date_dim" to the fact tables. 

Because of this, data scanned for fact tables are way higher and query runtime 
is increased.

For context, on a very small cluster, this query ran for 265 seconds and with 
the rewritten query it finished in 11 seconds!. Fact table scan was 10MB vs 10 
GB.

{noformat}
HiveJoin(condition=[=($2, $5)], joinType=[inner])
HiveJoin(condition=[=($0, $3)], joinType=[inner])
  HiveProject(cr_item_sk=[$1], cr_return_quantity=[$16], 
cr_returned_date_sk=[$26])
HiveFilter(condition=[AND(IS NOT NULL($26), IS NOT 
NULL($1))])
  HiveTableScan(table=[[tpcds_bin_partitioned_orc_1, 
catalog_returns]], table:alias=[catalog_returns])
  HiveProject(i_item_sk=[$0], i_item_id=[$1])
HiveFilter(condition=[AND(IS NOT NULL($1), IS NOT 
NULL($0))])
  HiveTableScan(table=[[tpcds_bin_partitioned_orc_1, 
item]], table:alias=[item])
HiveProject(d_date_sk=[$0], d_date=[$2])
  HiveFilter(condition=[AND(IS NOT NULL($2), IS NOT NULL($0))])
HiveTableScan(table=[[tpcds_bin_partitioned_orc_1, 
date_dim]], table:alias=[date_dim])
  HiveProject(d_date=[$0])
HiveSemiJoin(condition=[=($1, $2)], joinType=[semi])
  HiveProject(d_date=[$2], d_week_seq=[$4])
HiveFilter(condition=[AND(IS NOT NULL($4), IS NOT 
NULL($2))])
  HiveTableScan(table=[[tpcds_bin_partitioned_orc_1, 
date_dim]], table:alias=[date_dim])
  HiveProject(d_week_seq=[$4])
HiveFilter(condition=[AND(IN($2, 1998-01-02:DATE, 
1998-10-15:DATE, 1998-11-10:DATE), IS NOT NULL($4))])
  HiveTableScan(table=[[tpcds_bin_partitioned_orc_1, 
date_dim]], table:alias=[date_dim])
{noformat}

*Original Query & Plan: *

{noformat}
explain cbo with sr_items as
(select i_item_id item_id,
sum(sr_return_quantity) sr_item_qty
from store_returns,
item,
date_dim
where sr_item_sk = i_item_sk
and   d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and   sr_returned_date_sk   = d_date_sk
group by i_item_id),
cr_items as
(select i_item_id item_id,
sum(cr_return_quantity) cr_item_qty
from catalog_returns,
item,
date_dim
where cr_item_sk = i_item_sk
and   d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and   cr_returned_date_sk   = d_date_sk
group by i_item_id),
wr_items as
(select i_item_id item_id,
sum(wr_return_quantity) wr_item_qty
from web_returns,
item,
date_dim
where wr_item_sk = i_item_sk
and   d_datein
(select d_date
from date_dim
where d_week_seq in
(select d_week_seq
from date_dim
where d_date in ('1998-01-02','1998-10-15','1998-11-10')))
and   wr_returned_date_sk   = d_date_sk
group by i_item_id)
select  sr_items.item_id
,sr_item_qty
,sr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 sr_dev
,cr_item_qty
,cr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 cr_dev
,wr_item_qty
,wr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 wr_dev
,(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 average
from sr_items
,cr_items
,wr_items
where sr_items.item_id=cr_items.item_id
and sr_items.item_id=wr_items.item_id
order by sr_items.item_id
,sr_item_qty
limit 100
INFO  : Starting task [Stage-3:EXPLAIN] in serial mode
INFO  : Completed executing 
command(queryId=hive_20220307055109_88ad0cbd-bd40-45bc-92ae-ab15fa6b1da4); Time 
taken: 0.973 seconds
INFO  : OK
Explain
CBO PLAN:
HiveSortLimit(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC], fetch=[100])
  HiveProject(item_id=[$0], sr_item_qty=[$4], sr_dev=[*(/(/($5, CAST(+(+($4, 
$1), $7)):DOUBLE), 3), 100)], cr_item_qty=[$1], cr_dev=[*(/(/($2, CAST(+(+($4, 
$1), $7)):DOUBLE), 3), 100)], wr_item_qty=[$7], wr_dev=[*(/(/($8, CAST(+(+($4, 
$1), $7)):DOUBLE), 3), 100)], average=[/(CAST(+(+($4, $1), $7)):DECIMAL(19, 0), 
3:DECIMAL(1, 0))])
HiveJoin(condition=[=($0, $6)], joinType=[inner])
  HiveJoin(condition=[=($3, $0)], joinType=[inner])
HiveProject($f0=[

[jira] [Created] (HIVE-25981) Avoid checking for archived parts in analyze table

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25981:
---

 Summary: Avoid checking for archived parts in analyze table
 Key: HIVE-25981
 URL: https://issues.apache.org/jira/browse/HIVE-25981
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


Analyze table on large partitioned table is expensive due to unwanted checks on 
archived data.

 
{noformat}
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3908)
    - locked <0x0003d4c4c070> (a 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler)
    at com.sun.proxy.$Proxy56.listPartitionsWithAuthInfo(Unknown Source)
    at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:3845)
    at 
org.apache.hadoop.hive.ql.exec.ArchiveUtils.conflictingArchiveNameOrNull(ArchiveUtils.java:299)
    at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.validate(SemanticAnalyzer.java:13579)
    at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:241)
    at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:196)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:615)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:561)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:555)
    at 
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
    at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
    at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
    at org.apache.hive.service.cli.operation.Operation.run(Operation.java:285)  
 {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25958) Optimise BasicStatsNoJobTask

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25958:
---

 Summary: Optimise BasicStatsNoJobTask
 Key: HIVE-25958
 URL: https://issues.apache.org/jira/browse/HIVE-25958
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


When there are large number of files are present, it takes lot of time for 
analyzing table (for stats) takes lot longer time especially on cloud 
platforms. Each file is read in sequential fashion for computing stats, which 
can be optimized.

 
{code:java}
    at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:293)
    at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:506)
    - locked <0x000642995b10> (a org.apache.hadoop.fs.s3a.S3AInputStream)
    at 
org.apache.hadoop.fs.s3a.S3AInputStream.readFully(S3AInputStream.java:775)
    - locked <0x000642995b10> (a org.apache.hadoop.fs.s3a.S3AInputStream)
    at 
org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:116)
    at 
org.apache.orc.impl.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:574)
    at 
org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readFileData(RecordReaderUtils.java:282)
    at 
org.apache.orc.impl.RecordReaderImpl.readAllDataStreams(RecordReaderImpl.java:1172)
    at 
org.apache.orc.impl.RecordReaderImpl.readStripe(RecordReaderImpl.java:1128)
    at 
org.apache.orc.impl.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1281)
    at 
org.apache.orc.impl.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1316)
    at org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:302)
    at 
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:68)
    at 
org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:83)
    at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:367)
    at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.(OrcInputFormat.java:276)
    at 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:2027)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask$FooterStatCollector.run(BasicStatsNoJobTask.java:235)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
"HiveServer2-Background-Pool: Thread-5161" #5161 prio=5 os_prio=0 
tid=0x7f271217d800 nid=0x21b7 waiting on condition [0x7f26fce88000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0006bee1b3a0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at 
java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.shutdownAndAwaitTermination(BasicStatsNoJobTask.java:426)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.aggregateStats(BasicStatsNoJobTask.java:338)
    at 
org.apache.hadoop.hive.ql.stats.BasicStatsNoJobTask.process(BasicStatsNoJobTask.java:121)
    at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
    at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
    at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
    at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
    at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:250) {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25927) Fix DataWritableReadSupport

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25927:
---

 Summary: Fix DataWritableReadSupport 
 Key: HIVE-25927
 URL: https://issues.apache.org/jira/browse/HIVE-25927
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2022-02-04 at 4.57.22 AM.png

!Screenshot 2022-02-04 at 4.57.22 AM.png|width=530,height=406!

Takes n^2 ops to match columns.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25913) Dynamic Partition Pruning Operator: Not working in iceberg tables

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25913:
---

 Summary: Dynamic Partition Pruning Operator: Not working in 
iceberg tables
 Key: HIVE-25913
 URL: https://issues.apache.org/jira/browse/HIVE-25913
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Notice "Dynamic Partitioning Event Operator" missing in Map 3 in iceberg tables.

This causes heavy IO in iceberg tables leading to perf degradation.
{noformat}
ACID table
==

explain select count(*) from store_sales, date_dim  where d_month_seq between 
1212 and 1212+11 and ss_store_sk is not null and ss_sold_date_sk=d_date_sk;


Explain
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1STAGE PLANS:
  Stage: Stage-1
    Tez
      DagId: hive_20220131032425_be2fab7f-7943-4aa1-bbdd-289139ea0f90:17
      Edges:
        Map 1 <- Map 3 (BROADCAST_EDGE)
        Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE)
      DagName: hive_20220131032425_be2fab7f-7943-4aa1-bbdd-289139ea0f90:17
      Vertices:
        Map 1
            Map Operator Tree:
                TableScan
                  alias: store_sales
                  filterExpr: ss_store_sk is not null (type: boolean)
                  Statistics: Num rows: 27503885621 Data size: 434880571744 
Basic stats: COMPLETE Column stats: COMPLETE
                  Filter Operator
                    predicate: ss_store_sk is not null (type: boolean)
                    Statistics: Num rows: 26856185846 Data size: 424639398832 
Basic stats: COMPLETE Column stats: COMPLETE
                    Select Operator
                      expressions: ss_sold_date_sk (type: bigint)
                      outputColumnNames: _col0
                      Statistics: Num rows: 26856185846 Data size: 214849486768 
Basic stats: COMPLETE Column stats: COMPLETE
                      Map Join Operator
                        condition map:
                             Inner Join 0 to 1
                        keys:
                          0 _col0 (type: bigint)
                          1 _col0 (type: bigint)
                        input vertices:
                          1 Map 3
                        Statistics: Num rows: 5279977323 Data size: 42239818584 
Basic stats: COMPLETE Column stats: COMPLETE
                        Group By Operator
                          aggregations: count()
                          minReductionHashAggr: 0.99
                          mode: hash
                          outputColumnNames: _col0
                          Statistics: Num rows: 1 Data size: 8 Basic stats: 
COMPLETE Column stats: COMPLETE
                          Reduce Output Operator
                            null sort order:
                            sort order:
                            Statistics: Num rows: 1 Data size: 8 Basic stats: 
COMPLETE Column stats: COMPLETE
                            value expressions: _col0 (type: bigint)
            Execution mode: vectorized, llap
            LLAP IO: may be used (ACID table)
        Map 3
            Map Operator Tree:
                TableScan
                  alias: date_dim
                  filterExpr: (d_month_seq BETWEEN 1212 AND 1223 and d_date_sk 
is not null) (type: boolean)
                  Statistics: Num rows: 73049 Data size: 876588 Basic stats: 
COMPLETE Column stats: COMPLETE
                  Filter Operator
                    predicate: (d_month_seq BETWEEN 1212 AND 1223 and d_date_sk 
is not null) (type: boolean)
                    Statistics: Num rows: 359 Data size: 4308 Basic stats: 
COMPLETE Column stats: COMPLETE
                    Select Operator
                      expressions: d_date_sk (type: bigint)
                      outputColumnNames: _col0
                      Statistics: Num rows: 359 Data size: 2872 Basic stats: 
COMPLETE Column stats: COMPLETE
                      Reduce Output Operator
                        key expressions: _col0 (type: bigint)
                        null sort order: a
                        sort order: +
                        Map-reduce partition columns: _col0 (type: bigint)
                        Statistics: Num rows: 359 Data size: 2872 Basic stats: 
COMPLETE Column stats: COMPLETE
                      Select Operator
                        expressions: _col0 (type: bigint)
                        outputColumnNames: _col0
                        Statistics: Num rows: 359 Data size: 2872 Basic stats: 
COMPLETE Column stats: COMPLETE
                        Group By Operator
                          keys: _col0 (type: bigint)
                          minReductionHashAggr: 0.5013927
                          mode: hash
                          outputColumnNames: _col0
                          Statistics: Num rows: 179 Data size: 1432 Basic 
stats: COMPLETE Column stats: COMPLETE
                          Dynamic Partitionin

[jira] [Created] (HIVE-25845) Support ColumnIndexes for Parq files

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25845:
---

 Summary: Support ColumnIndexes for Parq files
 Key: HIVE-25845
 URL: https://issues.apache.org/jira/browse/HIVE-25845
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


https://issues.apache.org/jira/browse/PARQUET-1201

 

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L271-L273]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25827) Parquet file footer is read multiple times, when multiple splits are created in same file

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25827:
---

 Summary: Parquet file footer is read multiple times, when multiple 
splits are created in same file
 Key: HIVE-25827
 URL: https://issues.apache.org/jira/browse/HIVE-25827
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: image-2021-12-21-03-19-38-577.png

With large files, it is possible that multiple splits are created in the same 
file. With current codebase, "ParquetRecordReaderBase" ends up reading file 
footer for each split. 

It can be optimized not to read footer information multiple times for the same 
file.

 

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L160]

 

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L91]

 

 

!image-2021-12-21-03-19-38-577.png|width=1363,height=1256!

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25483:
---

 Summary: TxnHandler::acquireLock should close the DB conn to avoid 
connection leaks
 Key: HIVE-25483
 URL: https://issues.apache.org/jira/browse/HIVE-25483
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


TxnHandler::acquireLock should close DB connection on exiting the function.

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688]

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726]

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25482) Add option to enable connectionLeak detection for Hikari datasource

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25482:
---

 Summary: Add option to enable connectionLeak detection for Hikari 
datasource
 Key: HIVE-25482
 URL: https://issues.apache.org/jira/browse/HIVE-25482
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


There are corner cases where we observed connection leaks to DB.

 

It will be good to add an option to provide connection leak timeout parameter 
in HikariCPDataSourceProvider.

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java#L69]

e.g following should help Hikari to warn about connection leak, when a 
connection is not returned to the pool for 1 hour.
{noformat}
config.setLeakDetectionThreshold(3600*1000); {noformat}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25414) Optimise Hive::addWriteNotificationLog: Reduce FS call per notification

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25414:
---

 Summary: Optimise Hive::addWriteNotificationLog: Reduce FS call 
per notification
 Key: HIVE-25414
 URL: https://issues.apache.org/jira/browse/HIVE-25414
 Project: Hive
  Issue Type: Sub-task
Reporter: Rajesh Balamohan


AddWriteNotification is slow due to FS interactions (i.e to get the set of 
insert file information). This can be avoided as FileStatus can be passed 
instead of Path from parent methods.

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3572]
 

[https://github.com/apache/hive/blob/96b39cd5190f0cfadb677e3f3b7ead1d663921b2/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3620]

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25413) Optimise ObjectStore::alterPartitions to reduce DB calls for getPartitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25413:
---

 Summary: Optimise ObjectStore::alterPartitions to reduce DB calls 
for getPartitions
 Key: HIVE-25413
 URL: https://issues.apache.org/jira/browse/HIVE-25413
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L5121]

For dynamic load partitions (e.g when 1000s of parts being updated), it takes 
lot of time to fetch one partition at a time. It would be good to have  
"List getMPartitions" to reduce number of interactions with DB.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25391) Turn off priv synchronizer by default

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25391:
---

 Summary: Turn off priv synchronizer by default
 Key: HIVE-25391
 URL: https://issues.apache.org/jira/browse/HIVE-25391
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


In large deployments, good set of scale issues were identified with priv 
synchronizer. It will be good to turn it off by default, and let the users 
enable it on need basis.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25366) Reduce number of Table calls in updatePartitonColStatsInternal

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25366:
---

 Summary: Reduce number of Table calls in 
updatePartitonColStatsInternal
 Key: HIVE-25366
 URL: https://issues.apache.org/jira/browse/HIVE-25366
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


For every partition, table details are reloaded again which is completely 
wasteful. It will be good to pass the Table details fetched once for all 
partitions.

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java#L1091]

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L9342]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25324) Add option to disable PartitionManagementTask

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25324:
---

 Summary: Add option to disable PartitionManagementTask
 Key: HIVE-25324
 URL: https://issues.apache.org/jira/browse/HIVE-25324
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan


When large number of tables (e.g 2000) and databases are present, 
PartitionManagementTask scans all tables and partitions causing pressure on HMS.

Currently there is no way to disable PartitionManagementTask as well. Round 
about option is to provide pattern via 
"metastore.partition.management.database.pattern / 
metastore.partition.management.table.pattern".

It will be good to provide an option to disable it completely.{color:#807d6e}
{color}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25178) Reduce number of getPartition calls during loadDynamicPartitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25178:
---

 Summary: Reduce number of getPartition calls during 
loadDynamicPartitions
 Key: HIVE-25178
 URL: https://issues.apache.org/jira/browse/HIVE-25178
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Rajesh Balamohan


When dynamic partitions are loaded, Hive::loadDynamicPartition loads all 
partitions from HMS causing heavy load on it. This becomes worse when large 
number of partitions are present in tables.

Only relevant partitions being loaded in dynamic partitions can be queried from 
HMS for partition existence.

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2958]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25077) Direct SQL to fetch column privileges in refreshPrivileges may be broken in postgres

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25077:
---

 Summary: Direct SQL to fetch column privileges in 
refreshPrivileges may be broken in postgres
 Key: HIVE-25077
 URL: https://issues.apache.org/jira/browse/HIVE-25077
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


HIVE-22512 tried to fix direct-sql for col privileges.

 

However, "GRANT_OPTION" field in "TBL_COL_PRIVS" is marked as smallint in 
postgres. In code, it is retrieved as boolean.

Ref: 
[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1533]

 
{code:java}
boolean grantOption = 
MetastoreDirectSqlUtils.extractSqlBoolean(privLine[grantOptionIndex]);
{code}
 

[https://github.com/apache/hive/blob/048336bd0c21163920557a60c88135b1d5b42d3d/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L530]

 

MetastoreDirectSqlUtils::extractSqlBoolean should handle integers to support 
directSQL in postgres.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25075) Hive::loadPartitionInternal establishes HMS connection for every partition for external tables

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25075:
---

 Summary: Hive::loadPartitionInternal establishes HMS connection 
for every partition for external tables
 Key: HIVE-25075
 URL: https://issues.apache.org/jira/browse/HIVE-25075
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2522

{code}
boolean needRecycle = !tbl.isTemporary()
  && 
ReplChangeManager.shouldEnableCm(Hive.get().getDatabase(tbl.getDbName()), 
tbl.getTTable());
{code}

Hive.get() breaks the current connection with HMS. Due to this, for external 
table partition loads, it establishes HMS connection for partition.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25073) Optimise HiveAlterHandler::alterPartitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25073:
---

 Summary: Optimise HiveAlterHandler::alterPartitions
 Key: HIVE-25073
 URL: https://issues.apache.org/jira/browse/HIVE-25073
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan


Table details are populated again and again for each partition, which can be 
avoided.

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L5892

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L808



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25072) Optimise ObjectStore::alterPartitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25072:
---

 Summary: Optimise ObjectStore::alterPartitions
 Key: HIVE-25072
 URL: https://issues.apache.org/jira/browse/HIVE-25072
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan


Avoid fetching table details for every partition in the table.

Ref:

 
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L5104

https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L4986



Following stacktrace may be relevant for apache master as well.
{noformat}

at org.datanucleus.store.query.Query.executeWithArray(Query.java:1744)
at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:368)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:255)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2113)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2152)
at 
org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4951)
at 
org.apache.hadoop.hive.metastore.ObjectStore.alterPartitions(ObjectStore.java:5057)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.alterPartitions(Unknown Source)
at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:798)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:5695)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_req(HiveMetaStore.java:5647)
at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.alter_partitions_req(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18557)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$alter_partitions_req.getResult(ThriftHiveMetastore.java:18541)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
at 
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)

{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25032) Optimise PartitionManagementTask

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25032:
---

 Summary: Optimise PartitionManagementTask
 Key: HIVE-25032
 URL: https://issues.apache.org/jira/browse/HIVE-25032
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


When large number of tables and dbs are present, it takes lot of time to 
iterate over all tables in PartitionManagementTask.

If table parameters are populated in TableMeta, it will cut down lots of 
interactions with HMS in PartitionManagementTask.

 

Code Ref:

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L126]

 

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L132]

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25014) Optimise ObjectStore::updateTableColumnStatistics

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25014:
---

 Summary: Optimise ObjectStore::updateTableColumnStatistics
 Key: HIVE-25014
 URL: https://issues.apache.org/jira/browse/HIVE-25014
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


"ObjectStore::updateTableColumnStatistics" fetches table level details multiple 
times. Need to consider reducing the getTable lookups to avoid pressure on HMS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24960) HMS: Fix initial delay of housekeeping threads in HMS

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24960:
---

 Summary: HMS: Fix initial delay of housekeeping threads in HMS
 Key: HIVE-24960
 URL: https://issues.apache.org/jira/browse/HIVE-24960
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan


HMS starts house keeping threads in background. However, its initial delay is 
set to the frequency mentioned in MetastoreTaskThread. 

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L761]

For e.g, "hive.hook.proto.events.clean.freq" defaults to "1D" and the cleaner 
starts its initial iteration after a day. For frequently stopped/started 
clusters in cloud, this will not be ideal.

It will be good to introduce small random delay for the initial scheduling of 
these threads.
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24943) Initiator: Short circuit compaction checks on tables/partitions when DB isn't eligible for compaction

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24943:
---

 Summary: Initiator: Short circuit compaction checks on 
tables/partitions when DB isn't eligible for compaction
 Key: HIVE-24943
 URL: https://issues.apache.org/jira/browse/HIVE-24943
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


When the entire database isn't qualified for compaction due to replication, 
initiator could potentially skip the entire table/partition checks to save HMS 
resources. Currently it runs in tight loop for each table/partition for the 
database which isn't eligible for compaction.

E.g Note that for "delete_orc_10" database repl parameters are enabled. So 
until "ReplUtils.isFirstIncPending(database.getParameters());", it ends up 
considering that entire database as "not chosen for compaction".

Ref: 
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L469]

[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L469]

 
{noformat}
++
|   createtab_stmt   |
++
| CREATE TABLE `delete_orc_10.test_con_s3_part_1`(   |
|   `d_date_id` string,  |
|   `d_date` date,   |
|   `d_dom` int, |
|   `d_year` int)|
| PARTITIONED BY (   |
|   `d_date_sk` bigint)  |
| ROW FORMAT SERDE   |
|   'org.apache.hadoop.hive.ql.io.orc.OrcSerde'  |
| STORED AS INPUTFORMAT  |
|   'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'  |
| OUTPUTFORMAT   |
|   'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION   |
|   's3a://blah/delete_orc_10.db/test_con_s3_part_1' |
| TBLPROPERTIES (|
|   'bucketing_version'='2', |
|   'transactional'='true',  |
|   'transactional_properties'='default',|
|   'transient_lastDdlTime'='1610926861')|
++




++
|   createdb_stmt|
++
| CREATE DATABASE `delete_orc_10`|
| LOCATION   |
|   's3a://blah/delete_orc_10.db' |
| WITH DBPROPERTIES (|
|   
'hive.repl.ckpt.key'='s3a://blha/loc/dHBjZHNfYmluX3BhcnRpdGlvbmVkX29yY18xMDAwMA==/7f1c5529-e17a-4750-bf16-a9861c9589dc/hive',
  |
|   'hive.repl.first.inc.pending'='true',|
|   'repl.last.id'='95653')  |
++

{noformat}
{noformat}
<14>1 2021-03-25T23:42:50.095Z 
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local 
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060 
class="compactor.CompactorThread" level="INFO" thread="Thread-12"] Compaction 
is disabled for database delete_orc_10
<14>1 2021-03-25T23:42:50.095Z 
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local 
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060 
class="compactor.Initiator" level="INFO" thread="Thread-12"] Checking to see if 
we should compact delete_orc_10.test_con_s3_part_1.d_date_sk=2447933
<14>1 2021-03-25T23:42:50.118Z 
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local 
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060 
class="compactor.CompactorThread" level="INFO" thread="Thread-12"] Compaction 
is disabled for database delete_orc_10
<14>1 2021-03-25T23:42:50.118Z 
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local 
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060 
class="compactor.Initiator" level="INFO" thread="Thread-12"] Checking to see if 
we should compact delete_orc_10.test_con_s3_part_debug.d_date_sk=2471951
<14>1 2021-03-25T23:42:50.142Z 
metastore-0.metastore-service.warehouse-1616689513-66lb.svc.cluster.local 
metastore 1 e6cad49f-5d36-4a73-8da1-083e205976ee [mdc@18060 
class="compactor.CompactorThread" leve

[jira] [Created] (HIVE-24937) Fix sync bottleneck in SyslogParser

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24937:
---

 Summary: Fix sync bottleneck in SyslogParser
 Key: HIVE-24937
 URL: https://issues.apache.org/jira/browse/HIVE-24937
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: sample-thread-dump.txt

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java#L255
 

Sample threadump attached.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24892) Replace getContentSummary::getLength with listStatus(recursive) for blobstores

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24892:
---

 Summary: Replace getContentSummary::getLength with 
listStatus(recursive) for blobstores
 Key: HIVE-24892
 URL: https://issues.apache.org/jira/browse/HIVE-24892
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStats.java#L219

For blobstores, getContentSummary() is super slow. It would be good to replace 
this with "fs.listFiles(path, true)".




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [EXTERNAL] [Announce] New committer : Panos Garefalakis

[Rajesh Balamohan]

Congrats Panos. :)

~Rajesh.B

On Tue, Mar 9, 2021 at 9:55 AM Prasanth Jayachandran
 wrote:

> Congratulations Panos!
>
> Thanks
>
> Prasanth
>
>
> 
> From: Sankar Hariappan 
> Sent: Tuesday, March 9, 2021 9:12:27 AM
> To: dev@hive.apache.org 
> Subject: RE: [EXTERNAL] [Announce] New committer : Panos Garefalakis
>
> Congrats Panos! Welcome to Hive Committers club!😊
>
> Thanks,
> Sankar
>
> -Original Message-
> From: Ashutosh Chauhan 
> Sent: 09 March 2021 01:00
> To: dev@hive.apache.org
> Subject: [EXTERNAL] [Announce] New committer : Panos Garefalakis
>
> Apache Hive's Project Management Committee (PMC) has invited Panos
> Garefalakis to become a committer, and we are pleased to announce that he
> has accepted.
>
> Panos welcome, thank you for your contributions, and we look forward to
> your further interactions with the community!
>
> Thanks,
> Ashutosh
>

Re: [EXTERNAL] [Announce] New committer : Naresh PR

[Rajesh Balamohan]

Congrats Naresh. :)

~Rajesh.B

On Tue, Mar 9, 2021 at 9:55 AM Prasanth Jayachandran
 wrote:

> Congratulations Naresh!
>
> Thanks
>
> Prasanth
>
>
> 
> From: Sankar Hariappan 
> Sent: Tuesday, March 9, 2021 9:11:11 AM
> To: dev@hive.apache.org 
> Subject: RE: [EXTERNAL] [Announce] New committer : Naresh PR
>
> Congrats Naresh! Welcome to Hive Committers club! 😊
>
> Thanks,
> Sankar
>
> -Original Message-
> From: Ashutosh Chauhan 
> Sent: 09 March 2021 01:01
> To: dev@hive.apache.org
> Subject: [EXTERNAL] [Announce] New committer : Naresh PR
>
> Apache Hive's Project Management Committee (PMC) has invited Naresh PR to
> become a committer, and we are pleased to announce that he has accepted.
>
> Naresh welcome, thank you for your contributions, and we look forward to
> your further interactions with the community!
>
> Thanks,
> Ashutosh
>

[jira] [Created] (HIVE-24862) Fix race condition causing NPE during dynamic partition loading

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24862:
---

 Summary: Fix race condition causing NPE during dynamic partition 
loading
 Key: HIVE-24862
 URL: https://issues.apache.org/jira/browse/HIVE-24862
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Following properties default to 15 threads.
{noformat}
hive.load.dynamic.partitions.thread
hive.mv.files.thread  
{noformat}
During loadDynamicPartitions, it ends ups initializing {{newFiles}} without 
synchronization (HIVE-20661, HIVE-24738). 
 
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2871]

This causes race condition when dynamic partition thread internally makes use 
of {{hive.mv.files.threads}} in copyFiles/replaceFiles. 
 This causes "NPE" during retrieval in {{addInsertFileInformation()}}.

 

e.g stacktrace
{noformat}
Caused by: java.lang.NullPointerException
  at org.apache.hadoop.fs.FileSystem.fixRelativePart(FileSystem.java:2734)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.fixRelativePart(DistributedFileSystem.java:3396)
  at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1740)
  at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1740)
  at 
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
  at org.apache.hadoop.hive.ql.metadata.Hive.fireInsertEvent(Hive.java:3540)
  at 
org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2414)
  at 
org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24838) Reduce FS creation in Warehouse::getDnsPath for object stores

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24838:
---

 Summary: Reduce FS creation in Warehouse::getDnsPath for object 
stores
 Key: HIVE-24838
 URL: https://issues.apache.org/jira/browse/HIVE-24838
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2021-03-02 at 11.09.01 AM.png

[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java#L143]

 

Warehouse::getDnsPath gets invoked from multiple places (e.g getDatabase() 
etc). In certain cases like dynamic partition loads, lot of calls FS 
instantiation calls can be avoided for object stores.

It would be good to check for BlobStorages and if so, it should be possible to 
avoid FS creation.

[https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/BlobStorageUtils.java#L33]

 

!Screenshot 2021-03-02 at 11.09.01 AM.png|width=372,height=296!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24806) Compactor: Initiator should lazy evaluate findUserToRunAs()

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24806:
---

 Summary: Compactor: Initiator should lazy evaluate 
findUserToRunAs()
 Key: HIVE-24806
 URL: https://issues.apache.org/jira/browse/HIVE-24806
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L232

{noformat}
cache.putIfAbsent(fullTableName, findUserToRunAs(sd.getLocation(), t));
{noformat}

This ends up evaluating findUserToRunAs() everytime, and looks up from 
FileSystem on every call (thousands of times in large database).

This can be lazy initialized instead (e.g computeIfAbsent);



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24805) Compactor: Initiator shouldn't fetch table details again and again for partitioned tables

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24805:
---

 Summary: Compactor: Initiator shouldn't fetch table details again 
and again for partitioned tables
 Key: HIVE-24805
 URL: https://issues.apache.org/jira/browse/HIVE-24805
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


Initiator shouldn't be fetch table details for all its partitions. When there 
are large number of databases/tables, it takes lot of time for Initiator to 
complete its initial iteration and load on DB also goes higher.


https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L129

https://github.com/apache/hive/blob/64bb52316f19426ebea0087ee15e282cbde1d852/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L456

For all the following partitions, table details would be the same. However, it 
ends up fetching table details from HMS again and again.

{noformat}
2021-02-22 08:13:16,106 INFO  
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see 
if we should compact 
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451899
2021-02-22 08:13:16,124 INFO  
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see 
if we should compact 
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2451830
2021-02-22 08:13:16,140 INFO  
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see 
if we should compact 
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452586
2021-02-22 08:13:16,149 INFO  
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see 
if we should compact 
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452698
2021-02-22 08:13:16,158 INFO  
org.apache.hadoop.hive.ql.txn.compactor.Initiator: [Thread-11]: Checking to see 
if we should compact 
tpcds_bin_partitioned_orc_1000.store_returns_tmp2.sr_returned_date_sk=2452063
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24790) Batch column stats updates to HMS

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24790:
---

 Summary: Batch column stats updates to HMS
 Key: HIVE-24790
 URL: https://issues.apache.org/jira/browse/HIVE-24790
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


When large number of partitions are inserted/updated, it would be good to batch 
column statistics updates to HMS.

Currently, HS2 ends up throwing read timeout exception when updating HMS.

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L180

 {noformat}
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_252]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:171) 
~[?:1.8.0_252]
at java.net.SocketInputStream.read(SocketInputStream.java:141) 
~[?:1.8.0_252]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 
~[?:1.8.0_252]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) 
~[?:1.8.0_252]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) 
~[?:1.8.0_252]
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
 ~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) 
~[hive-exec-3.1]
at 
org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:374) 
~[hive-exec-3.1]
at 
org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:451) 
~[hive-exec-3.1]
at 
org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:433) 
~[hive-exec-3.1]
at 
org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
 ~[hive-exec-3.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) 
~[hive-exec-3.1]
at 
org.apache.hadoop.hive.metastore.security.TFilterTransport.readAll(TFilterTransport.java:62)
 ~[hive-exec-3.1]
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) 
~[hive-exec-3.1]
at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) 
~[hive-exec-3.1]
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
 ~[hive-exec-3.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) 
~[hive-exec-3.1]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:4561)
 ~[hive-exec-3.1]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:4548)
 ~[hive-exec-3.1]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:2496)
 ~[hive-exec-3.1]
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.setPartitionColumnStatistics(SessionHiveMetaStoreClient.java:515)
 ~[hive-exec-3.1]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
 ~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source) 
~[?:?]
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3431)
 ~[hive-exec-3.1]
at com.sun.proxy.$Proxy60.setPartitionColumnStatistics(Unknown Source) 
~[?:?]
at 
org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:5213)
 ~[hive-exec-3.1]
at 
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:192)
 ~[hive-exec-3.1]
at 
org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:87)
 ~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107) 
~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) 
~[hive-exec-3.1]
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) 
~[hive-exec-3.1]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) 
~[hive-exec-3.1

[jira] [Created] (HIVE-24776) Reduce HMS DB calls during stats updates

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24776:
---

 Summary: Reduce HMS DB calls during stats updates
 Key: HIVE-24776
 URL: https://issues.apache.org/jira/browse/HIVE-24776
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


 When adding large number of partitions (100s/1000s) in a table, it ends up 
making lots of getTable calls which are not needed.

Lines mentioned below may vary slightly in apache-master. 

{noformat}

at 
org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:620)
at 
org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:637)
at 
org.datanucleus.api.jdo.JDOPersistenceManager.retrieve(JDOPersistenceManager.java:646)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2112)
at 
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2150)
at 
org.apache.hadoop.hive.metastore.ObjectStore.ensureGetMTable(ObjectStore.java:4578)
at 
org.apache.hadoop.hive.metastore.ObjectStore.ensureGetTable(ObjectStore.java:4588)
at 
org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:9264)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
at com.sun.proxy.$Proxy27.updatePartitionColumnStatistics(Unknown 
Source)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStatsInternal(HiveMetaStore.java:6679)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartColumnStatsWithMerge(HiveMetaStore.java:8655)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:8592)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at 
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.set_aggr_stats_for(Unknown Source)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19060)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:19044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
 {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24774) Reduce FS listing during dynamic partition loading

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24774:
---

 Summary: Reduce FS listing during dynamic partition loading
 Key: HIVE-24774
 URL: https://issues.apache.org/jira/browse/HIVE-24774
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan




When loading large number of partitions in cloud storage, notification log 
takes lot longer time to list newly added files.

It would be good to explore if FileStatus can be reused from 
Hive::listFilesCreatedByQuery or from copyFiles

{noformat}

at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.isDirectory(S3AFileSystem.java:4171)
at 
org.apache.hadoop.hive.ql.metadata.Hive.addInsertFileInformation(Hive.java:3566)
at 
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3519)
at 
org.apache.hadoop.hive.ql.metadata.Hive.addWriteNotificationLog(Hive.java:3504)
at 
org.apache.hadoop.hive.ql.metadata.Hive.loadDynamicPartitions(Hive.java:2984)
at 
org.apache.hadoop.hive.ql.exec.MoveTask.handleDynParts(MoveTask.java:562)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:440)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:730)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:490)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:484)

{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24764) insert overwrite on a partition resets row count stats in other partitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24764:
---

 Summary: insert overwrite on a partition resets row count stats in 
other partitions
 Key: HIVE-24764
 URL: https://issues.apache.org/jira/browse/HIVE-24764
 Project: Hive
  Issue Type: Bug
Reporter: Rajesh Balamohan


After insert overwrite on a partition, stats on other partitions are messed up. 
Subsequent queries end up with plans with PARTIAL stats. In certain cases, this 
leads to suboptimal query plans.

{noformat}

drop table if exists test_stats;
drop table if exists test_stats_2;

create table test_stats(i int, j bigint);
create table test_stats_2(i int) partitioned by (j bigint);

insert into test_stats values (1, 1), (2, 2), (3, 3), (4, 4), (5, NULL);

-- select * from test_stats;
1   1
2   2
3   3
4   4
5   


insert overwrite table test_stats_2 partition(j)  select i, j from test_stats 
where j is not null;


-- After executing this statement, stat gets messed up.
insert overwrite table test_stats_2 partition(j)  select i, j from test_stats 
where j is null;

-- select * from test_stats_2;
1   1
2   2
3   3
4   4
5   


-- This would return "PARTIAL" stats instead of "COMPLETE"
explain select i, count(*) as c from test_stats_2 group by i order by c desc 
limit 10;


STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
Tez
  DagId: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
  Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
  DagName: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
  Vertices:
Map 1 
Map Operator Tree:
TableScan
  alias: test_stats_2
  Statistics: Num rows: 125 Data size: 500 Basic stats: PARTIAL 
Column stats: COMPLETE
  Select Operator
expressions: i (type: int)
outputColumnNames: i
Statistics: Num rows: 125 Data size: 500 Basic stats: 
PARTIAL Column stats: COMPLETE
Group By Operator
  aggregations: count()
  keys: i (type: int)
  minReductionHashAggr: 0.99
  mode: hash
  outputColumnNames: _col0, _col1
  Statistics: Num rows: 125 Data size: 1500 Basic stats: 
PARTIAL Column stats: COMPLETE
  Reduce Output Operator
key expressions: _col0 (type: int)
null sort order: a
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 125 Data size: 1500 Basic stats: 
PARTIAL Column stats: COMPLETE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: may be used (ACID table)
Reducer 2 
Execution mode: vectorized, llap
Reduce Operator Tree:
  Group By Operator
aggregations: count(VALUE._col0)
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
Top N Key Operator
  sort order: -
  keys: _col1 (type: bigint)
  null sort order: a
  Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
  top n: 10
  Reduce Output Operator
key expressions: _col1 (type: bigint)
null sort order: a
sort order: -
Statistics: Num rows: 62 Data size: 744 Basic stats: 
PARTIAL Column stats: COMPLETE
TopN Hash Memory Usage: 0.04
value expressions: _col0 (type: int)
Reducer 3 
Execution mode: vectorized, llap
Reduce Operator Tree:
  Select Operator
expressions: VALUE._col0 (type: int), KEY.reducesinkkey0 (type: 
bigint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
Limit
  Number of rows: 10
  Statistics: Num rows: 10 Data size: 120 Basic stats: PARTIAL 
Column stats: COMPLETE
  File Output Operator
compressed: false
Statistics: Num rows: 10 Data size: 120 Basic stats: 
PARTIAL Column stats: COMPLETE
table:
input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat

Re: Welcome Adam to Hive PMC

[Rajesh Balamohan]

Congratulations Adam. :)

~Rajesh.B

On Tue, Feb 2, 2021 at 1:19 AM Peter Vary 
wrote:

> Congratulations!
>
> > On Feb 1, 2021, at 17:47, László Bodor 
> wrote:
> >
> > Congrats, very well-deserved!
> >
> > Regards,
> > Laszlo Bodor
> >
> > Ashutosh Chauhan  ezt írta (időpont: 2021. febr.
> 1.,
> > H, 17:11):
> >
> >> Hi all,
> >>
> >> It's an honor to announce that Apache Hive PMC has recently voted to
> invite
> >> Adam Szita as a new Hive PMC member. Adam is a long time Hive
> contributor
> >> and committer, and has made significant contributions in Hive. Please
> join
> >> me in congratulating him and looking forward to a bigger role that he
> will
> >> play in the Apache Hive project.
> >>
> >> Thanks,
> >> Ashutosh
> >>
>
>

Re: [Announce] New committer : Mustafa Iman

[Rajesh Balamohan]

Congratulations Mustafa. :)

~Rajesh.B

On Tue, Feb 2, 2021 at 1:19 AM Peter Vary 
wrote:

> Congratulations!
>
> > On Feb 1, 2021, at 17:45, László Bodor 
> wrote:
> >
> > Congrats, well-deserved!
> >
> > Regards,
> > Laszlo Bodor
> >
> > Ashutosh Chauhan  ezt írta (időpont: 2021. febr.
> 1.,
> > H, 17:15):
> >
> >> Apache Hive's Project Management Committee (PMC) has invited Mustafa
> Iman
> >> to become a committer, and we are pleased to announce that he has
> accepted.
> >>
> >> Mustafa welcome, thank you for your contributions, and we look forward
> to
> >> your further interactions with the community!
> >>
> >> Thanks,
> >> Ashutosh
> >>
>
>

Re: [Announce] New committer : Mustafa Iman

[Rajesh Balamohan]

Congratulations Mustafa. :)

~Rajesh.B

On Tue, Feb 2, 2021 at 1:19 AM Peter Vary 
wrote:

> Congratulations!
>
> > On Feb 1, 2021, at 17:45, László Bodor 
> wrote:
> >
> > Congrats, well-deserved!
> >
> > Regards,
> > Laszlo Bodor
> >
> > Ashutosh Chauhan  ezt írta (időpont: 2021. febr.
> 1.,
> > H, 17:15):
> >
> >> Apache Hive's Project Management Committee (PMC) has invited Mustafa
> Iman
> >> to become a committer, and we are pleased to announce that he has
> accepted.
> >>
> >> Mustafa welcome, thank you for your contributions, and we look forward
> to
> >> your further interactions with the community!
> >>
> >> Thanks,
> >> Ashutosh
> >>
>
>

-- 
~Rajesh.B

Re: [Announce] New committer : Peter Varga

[Rajesh Balamohan]

Congratulations Peter Varga. :)

~Rajesh.B

On Tue, Feb 2, 2021 at 1:19 AM Peter Vary 
wrote:

> Congratulations!
>
> > On Feb 1, 2021, at 17:45, László Bodor 
> wrote:
> >
> > Congrats, well-deserved!
> >
> > Regards,
> > Laszlo Bodor
> >
> >
> > Ashutosh Chauhan  ezt írta (időpont: 2021. febr.
> 1.,
> > H, 17:10):
> >
> >> Apache Hive's Project Management Committee (PMC) has invited Peter
> Varga to
> >> become a committer, and we are pleased to announce that he has accepted.
> >>
> >> Peter welcome, thank you for your contributions, and we look forward to
> >> your further interactions with the community!
> >>
> >> Thanks,
> >> Ashutosh
> >>
>
>

[jira] [Created] (HIVE-24710) PTFRowContainer could be reading more number of blocks than needed

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24710:
---

 Summary: PTFRowContainer could be reading more number of blocks 
than needed
 Key: HIVE-24710
 URL: https://issues.apache.org/jira/browse/HIVE-24710
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


PTFRowContainer could be reading the same block repeatedly for the first block. 
Default block size is around 25000. For the first 25000 rowIdx, it would read 
the block repeatedly due to ("rowIdx < currentReadBlockStartRow ") condition.

{noformat}
 public Row getAt(int rowIdx) throws HiveException {
int blockSize = getBlockSize();
if ( rowIdx < currentReadBlockStartRow || rowIdx >= 
currentReadBlockStartRow + blockSize ) {
  readBlock(getBlockNum(rowIdx));
}
return getReadBlockRow(rowIdx - currentReadBlockStartRow);
  }
{noformat} 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24695) Clean up session resources, if TezSession is unable to start

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24695:
---

 Summary: Clean up session resources, if TezSession is unable to 
start
 Key: HIVE-24695
 URL: https://issues.apache.org/jira/browse/HIVE-24695
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


There are cases when TezSessionState would not be able to start. (e.g resource 
constraints on YARN queues). 

However by this time, session directories & certain resources are localized. 
(e.g, hive-exec jars are stored in  
hdfs:///tmp/hive/hive/_tez_session_dir/*/hive-exec*.jar). When tezClient is not 
started, it does not clear up the resources. This leaks ~100MB data in HDFS per 
failure. 





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24663) Batch process in ColStatsProcessor

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24663:
---

 Summary: Batch process in ColStatsProcessor
 Key: HIVE-24663
 URL: https://issues.apache.org/jira/browse/HIVE-24663
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


When large number of partitions (>20K) are processed, ColStatsProcessor runs 
into DB issues. 

{{ db.setPartitionColumnStatistics(request);}} gets stuck for hours together 
and in some cases postgres stops processing. 

It would be good to introduce small batches for stats gathering in 
ColStatsProcessor instead of bulk update.

Ref: 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L181

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java#L199





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24662) Append txn_id to table path location

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24662:
---

 Summary: Append txn_id to table path location
 Key: HIVE-24662
 URL: https://issues.apache.org/jira/browse/HIVE-24662
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


It would be good to append txn_id in the path during table creation. This would 
help later in optimizing for drop/create table scenario (as drop can purge the 
data later and create would point to new location automatically). It would also 
help in having undrop table feature later.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24649) Optimise Hive::addWriteNotificationLog for large data inserts

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24649:
---

 Summary: Optimise Hive::addWriteNotificationLog for large data 
inserts
 Key: HIVE-24649
 URL: https://issues.apache.org/jira/browse/HIVE-24649
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Rajesh Balamohan


When loading dynamic partition with large dataset, it spends lot of time in 
"Hive::loadDynamicPartitions --> addWriteNotificationLog".

Though it is for same for same table, it ends up loading table and partition 
details for every partition and writes to notification log.

Also, "Partition" details may be already present in {{PartitionDetails}} object 
in {{Hive::loadDynamicPartitions}}. This is unnecessarily recomputed again in 
{{HiveMetaStore::add_write_notification_log}}

 
Lines of interest:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3028
https://github.com/apache/hive/blob/89073a94354f0cc14ec4ae0a43e05aae29276b4d/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L8500
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24596) Explain ddl for debugging

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24596:
---

 Summary: Explain ddl for debugging
 Key: HIVE-24596
 URL: https://issues.apache.org/jira/browse/HIVE-24596
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


For debugging query issues, basic details like table schema, statistics, 
partition details, query plans are needed.

It would be good to have "explain ddl" support, which can generate these 
details. This can help in recreating the schema and planner issues without 
sample data.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24546) Avoid unwanted cloud storage call during dynamic partition load

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24546:
---

 Summary: Avoid unwanted cloud storage call during dynamic 
partition load
 Key: HIVE-24546
 URL: https://issues.apache.org/jira/browse/HIVE-24546
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan



{code:java}

 private void createDpDirCheckSrc(final Path dpStagingPath, final Path 
dpFinalPath) throws IOException {
if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
  fs.mkdirs(dpStagingPath);
  // move task will create dp final path
  if (reporter != null) {
reporter.incrCounter(counterGroup, 
Operator.HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, 1);
  }
}
  }

 {code}
 

 
{noformat}
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:370)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1960)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3164)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2899)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1723)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4157)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createDpDir(FileSinkOperator.java:948)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.updateDPCounters(FileSinkOperator.java:916)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:849)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:814)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:1200)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:1324)
at 
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1036)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
at 
org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
 {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24537) Optimise locking in LlapTaskSchedulerService

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24537:
---

 Summary: Optimise locking in LlapTaskSchedulerService
 Key: HIVE-24537
 URL: https://issues.apache.org/jira/browse/HIVE-24537
 Project: Hive
  Issue Type: Improvement
  Components: llap
Reporter: Rajesh Balamohan
 Attachments: Screenshot 2020-12-15 at 11.41.49 AM.png

1. Read lock should suffice for "notifyStarted()".
2. Locking in "allocateTask()" can be optimised. 
3. Optimize preemptTasks() & preemptTasksFromMap(). This would help in reducing 
the codepath with writeLock. Currently, it iterates through all tasks.

 

  !Screenshot 2020-12-15 at 11.41.49 AM.png|width=847,height=446!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)