[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, uses a
static noop EventSequence as the argument. We can replace it in future
works, e.g. expose execution timeline of a slow processing on an HMS
event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for a REFRESH statement on an unloaded table
(IncompleteTable):
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Example timeline for a global INVALIDATE METADATA:
Catalog Server Operation: 301.618ms
   - Got catalog version write lock: 9.908ms (9.908ms)
   - Got Metastore client: 9.922ms (14.013us)
   - Got database list: 11.396ms (1.473ms)
   - Loaded functions of default: 44.919ms (33.523ms)
   - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms)
   - Loaded functions of functional: 50.846ms (3.321ms)
   - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms)
   - Loaded functions of functional_avro: 54.861ms (2.281ms)
   - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms 
(928.120us)
   ...
   - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms)
   - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms 
(784.725us)
   - Updated catalog cache: 300.366ms (78.045us)
   - Finished resetMetadata request: 301.618ms (1.251ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Reviewed-o

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 16: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 21 Mar 2024 16:04:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 16:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10402/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 21 Mar 2024 10:57:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 16: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 21 Mar 2024 10:57:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 15: Code-Review+2

Nice feature! LGTM!


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 21 Mar 2024 10:35:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 15: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 20 Mar 2024 08:18:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10390/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 20 Mar 2024 03:16:54 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 15:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/15576/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 20 Mar 2024 01:19:33 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 14:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/15575/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 14
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 20 Mar 2024 01:12:00 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 15:

(1 comment)

Thanks for the review! Resolved the conflicts.

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1514
PS12, Line 1514: params.getAlter_type());
> How about "Iceberg operations are prepared for commit"?
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 20 Mar 2024 00:56:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#15).

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, uses a
static noop EventSequence as the argument. We can replace it in future
works, e.g. expose execution timeline of a slow processing on an HMS
event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for a REFRESH statement on an unloaded table
(IncompleteTable):
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Example timeline for a global INVALIDATE METADATA:
Catalog Server Operation: 301.618ms
   - Got catalog version write lock: 9.908ms (9.908ms)
   - Got Metastore client: 9.922ms (14.013us)
   - Got database list: 11.396ms (1.473ms)
   - Loaded functions of default: 44.919ms (33.523ms)
   - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms)
   - Loaded functions of functional: 50.846ms (3.321ms)
   - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms)
   - Loaded functions of functional_avro: 54.861ms (2.281ms)
   - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms 
(928.120us)
   ...
   - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms)
   - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms 
(784.725us)
   - Updated catalog cache: 300.366ms (78.045us)
   - Finished resetMetadata request: 301.618ms (1.251ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - R

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#14).

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, uses a
static noop EventSequence as the argument. We can replace it in future
works, e.g. expose execution timeline of a slow processing on an HMS
event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for a REFRESH statement on an unloaded table
(IncompleteTable):
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Example timeline for a global INVALIDATE METADATA:
Catalog Server Operation: 301.618ms
   - Got catalog version write lock: 9.908ms (9.908ms)
   - Got Metastore client: 9.922ms (14.013us)
   - Got database list: 11.396ms (1.473ms)
   - Loaded functions of default: 44.919ms (33.523ms)
   - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms)
   - Loaded functions of functional: 50.846ms (3.321ms)
   - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms)
   - Loaded functions of functional_avro: 54.861ms (2.281ms)
   - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms 
(928.120us)
   ...
   - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms)
   - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms 
(784.725us)
   - Updated catalog cache: 300.366ms (78.045us)
   - Finished resetMetadata request: 301.618ms (1.251ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - R

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 13: Code-Review+2

(4 comments)

Great work! LGTM!

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG@31
PS12, Line 31: Example
> The REFRESH command is executed when the table is in unloaded state (Incomp
That's clearer, thanks.


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc@125
PS12, Line 125: catalog_profile_ = 
make_unique(exec_response_->profile);
> It's possible that we get a very long timeline for global INVALIDATE METADA
I see, thanks for the explanation!


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2014
PS12, Line 2014:  MetastoreShim.m
> "table metadata" might be misunderstood as the HMS table object which conta
Ah, I see, now I'm fine with either.


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1514
PS12, Line 1514: }
> Maybe we can use a better name unless the above codes won't invoke any exte
How about "Iceberg operations are prepared for commit"?



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 03 Jan 2024 19:09:42 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 13:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14849/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Tue, 02 Jan 2024 15:16:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 12:

(12 comments)

Thanks for the feedbacks!

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG@31
PS12, Line 31: unloaded
> reloaded?
The REFRESH command is executed when the table is in unloaded state 
(IncompleteTable). Updated the sentence.


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc@125
PS12, Line 125: catalog_profile_ = 
make_unique(exec_response_->profile);
> Would it make sense to do a sanity check that we didn't get an awful lot of
It's possible that we get a very long timeline for global INVALIDATE METADATA 
commands, e.g.

Catalog Server Operation: 301.618ms
   - Got catalog version write lock: 9.908ms (9.908ms)
   - Got Metastore client: 9.922ms (14.013us)
   - Got database list: 11.396ms (1.473ms)
   - Loaded functions of default: 44.919ms (33.523ms)
   - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms)
   - Loaded functions of functional: 50.846ms (3.321ms)
   - Loaded TableMeta of 101 tables in database functional: 52.580ms 
(1.734ms)
   - Loaded functions of functional_avro: 54.861ms (2.281ms)
   - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms 
(928.120us)
   - Loaded functions of functional_avro_def: 58.193ms (2.403ms)
   - Loaded TableMeta of 34 tables in database functional_avro_def: 
59.433ms (1.240ms)
   - Loaded functions of functional_avro_gzip: 61.639ms (2.206ms)
   - Loaded TableMeta of 42 tables in database functional_avro_gzip: 
62.655ms (1.015ms)
   - Loaded functions of functional_avro_snap: 64.879ms (2.224ms)
   - Loaded TableMeta of 55 tables in database functional_avro_snap: 
66.150ms (1.270ms)
   ...

This patch adds events for loading table and function list for each db. So we 
can see which db have most of the tables and takes the longest time to load 
(admin can blacklist it).


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc@1596
PS12, Line 1596: catalog_timeline.name
> nit: could be 'timeline_name'
Done


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc@1773
PS12, Line 1773:   if (catalog_op_executor_->catalog_profile() != nullptr) {
   : for (const TEventSequence& catalog_timeline :
   : 
catalog_op_executor_->catalog_profile()->event_sequences) {
   :   
summary_profile_->AddEventSequence(catalog_timeline.name, catalog_timeline);
   : }
   :   }
> nit: same as L711-L716. Could be moved to a member function.
Done


http://gerrit.cloudera.org:8080/#/c/20491/12/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/20491/12/common/thrift/CatalogService.thrift@296
PS12, Line 296: DDL
> Maybe DDL/DML? E.g. this will also contain information about Iceberg DML op
Done


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2014
PS12, Line 2014: Loaded TableMeta
> nit: I found the camelcase word a bit weird here, so maybe 'Loaded table me
"table metadata" might be misunderstood as the HMS table object which contains 
the schema, properties, etc. What about "Loaded 100 table names of database 
db1" ? By default, TableMeta just contains the table name. When catalogd is 
started with --pull_table_types_and_comments=true, TableMeta will contain the 
table type and comment. But it's off by default.


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2312
PS12, Line 2312: versionLock_.readLock().lock();
   : catalogTimeline.markEvent(GOT_CATALOG_VERSION_READ_LOCK);
> Would it make sense to create a method that encapsulates these two?
Done


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2347
PS12, Line 2347: tbl.readLock().lock();
   : catalogTimeline.markEvent(GOT_TABLE_READ_LOCK);
> Would it make sense to create a method that encapsulates these two?
Done


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/or

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#13).

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, uses a
static noop EventSequence as the argument. We can replace it in future
works, e.g. expose execution timeline of a slow processing on an HMS
event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for a REFRESH statement on an unloaded table
(IncompleteTable):
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Example timeline for a global INVALIDATE METADATA:
Catalog Server Operation: 301.618ms
   - Got catalog version write lock: 9.908ms (9.908ms)
   - Got Metastore client: 9.922ms (14.013us)
   - Got database list: 11.396ms (1.473ms)
   - Loaded functions of default: 44.919ms (33.523ms)
   - Loaded TableMeta of 82 tables in database default: 47.524ms (2.604ms)
   - Loaded functions of functional: 50.846ms (3.321ms)
   - Loaded TableMeta of 101 tables in database functional: 52.580ms (1.734ms)
   - Loaded functions of functional_avro: 54.861ms (2.281ms)
   - Loaded TableMeta of 35 tables in database functional_avro: 55.789ms 
(928.120us)
   ...
   - Loaded functions of tpch_text_gzip: 299.503ms (1.710ms)
   - Loaded TableMeta of 8 tables in database tpch_text_gzip: 300.288ms 
(784.725us)
   - Updated catalog cache: 300.366ms (78.045us)
   - Finished resetMetadata request: 301.618ms (1.251ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - R

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 12:

(13 comments)

Great work, Quanlong! Left a couple of comments, but looks really good!

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/20491/12//COMMIT_MSG@31
PS12, Line 31: unloaded
reloaded?


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/exec/catalog-op-executor.cc@125
PS12, Line 125: catalog_profile_ = 
make_unique(exec_response_->profile);
Would it make sense to do a sanity check that we didn't get an awful lot of 
timeline events, e.g. events should be less than 100?


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc@1596
PS12, Line 1596: catalog_timeline.name
nit: could be 'timeline_name'


http://gerrit.cloudera.org:8080/#/c/20491/12/be/src/service/client-request-state.cc@1773
PS12, Line 1773:   if (catalog_op_executor_->catalog_profile() != nullptr) {
   : for (const TEventSequence& catalog_timeline :
   : 
catalog_op_executor_->catalog_profile()->event_sequences) {
   :   
summary_profile_->AddEventSequence(catalog_timeline.name, catalog_timeline);
   : }
   :   }
nit: same as L711-L716. Could be moved to a member function.


http://gerrit.cloudera.org:8080/#/c/20491/12/common/thrift/CatalogService.thrift
File common/thrift/CatalogService.thrift:

http://gerrit.cloudera.org:8080/#/c/20491/12/common/thrift/CatalogService.thrift@296
PS12, Line 296: DDL
Maybe DDL/DML? E.g. this will also contain information about Iceberg DML 
operations.


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2014
PS12, Line 2014: Loaded TableMeta
nit: I found the camelcase word a bit weird here, so maybe 'Loaded table 
metadata'?


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2312
PS12, Line 2312: versionLock_.readLock().lock();
   : catalogTimeline.markEvent(GOT_CATALOG_VERSION_READ_LOCK);
Would it make sense to create a method that encapsulates these two?


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2347
PS12, Line 2347: tbl.readLock().lock();
   : catalogTimeline.markEvent(GOT_TABLE_READ_LOCK);
Would it make sense to create a method that encapsulates these two?


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/HBaseTable.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/HBaseTable.java@115
PS12, Line 115: hbase
nit: HBase?


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/KuduTable.java
File fe/src/main/java/org/apache/impala/catalog/KuduTable.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/catalog/KuduTable.java@324
PS12, Line 324: kudu
nit: Kudu


http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7109
PS9, Line 7109:   // always responsible for aborting transactions when 
queries hit errors.
> Sure. update.getIceberg_operation() returns a TIcebergOperationParam which
Thank you!


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1514
PS12, Line 1514: catalogTimeline.markEvent("Updated table in Iceberg");
This might be misleading, as we don't update the table until we commit. So I 
think it's enough to only keep the "Committed Iceberg transaction".

Also, 
https://gerrit.cloudera.org/#/c/20823/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
 will get rid of 'needsTxn' anyway.


http://gerrit.cloudera.org:8080/#/c/20491/12/fe/src/main/java/org/apache/impala/util/E

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 12:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14740/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 15 Dec 2023 13:25:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 12:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20491/11/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/20491/11/be/src/service/client-request-state.cc@854
PS11, Line 854: lock_guard l(lock_);
> In ExecDdlRequestImpl() and UpdateTableAndColumnStats(), AddEventSequence()
Oops, we should put this before UpdateQueryStatus(). Nice catch!


http://gerrit.cloudera.org:8080/#/c/20491/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2690
PS11, Line 2690:   int kudu_table_reserve_seconds, EventSequence 
catalogTimeline)
> line too long (93 > 90)
Done



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 15 Dec 2023 12:57:53 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Zihao Ye, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#12).

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTab

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Zihao Ye (Code Review)]

Zihao Ye has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 11:

(1 comment)

Overall LGTM, this will be helpful for analyzing DDL performance and issues, I 
just have one small question.

http://gerrit.cloudera.org:8080/#/c/20491/11/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/20491/11/be/src/service/client-request-state.cc@854
PS11, Line 854:   summary_profile_->AddEventSequence(catalog_timeline.name, 
catalog_timeline);
In ExecDdlRequestImpl() and UpdateTableAndColumnStats(), AddEventSequence() is 
placed before UpdateQueryStatus().  Why is it placed after here?  Could this 
result in the loss of some events in case of an error?



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zihao Ye 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 15 Dec 2023 10:29:18 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14735/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 15 Dec 2023 08:14:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#11).

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..

IMPALA-12443: Add catalog timeline for all DDL profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M 

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all DDL profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all DDL profiles
..


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20491/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2690
PS11, Line 2690:   int kudu_table_reserve_seconds, EventSequence 
catalogTimeline) throws ImpalaException {
line too long (93 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Fri, 15 Dec 2023 07:46:58 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 10:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14684/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Tue, 12 Dec 2023 23:55:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 10:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@405
PS9, Line 405: GroupedContentFiles icebergFiles = 
IcebergUtil.getIcebergFiles(this,
 : new ArrayList<>(), /*timeTravelSpec=*/null);
> I think it'd be worth to mark an event after this since the planFiles() cal
Done


http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7109
PS9, Line 7109: catalogTimeline.markEvent("Executed Iceberg operation " 
+
> Can we include update.getIceberg_operation() in the event?
Sure. update.getIceberg_operation() returns a TIcebergOperationParam which 
contains too many details. I use update.getIceberg_operation().getOperation() 
instead.



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Tue, 12 Dec 2023 23:23:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Zoltan Borok-Nagy, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#10).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M 

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 9:

(2 comments)

Thanks for working on this, it will be extremely useful.
Looked at the Iceberg-related parts.

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@405
PS9, Line 405: GroupedContentFiles icebergFiles = 
IcebergUtil.getIcebergFiles(this,
 : new ArrayList<>(), /*timeTravelSpec=*/null);
I think it'd be worth to mark an event after this since the planFiles() call in 
getIcebergFiles() can take a significant time.


http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/9/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7109
PS9, Line 7109: catalogTimeline.markEvent("Executed Iceberg operation");
Can we include update.getIceberg_operation() in the event?



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Tue, 12 Dec 2023 20:26:09 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14662/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 12 Dec 2023 08:39:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 8:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14661/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 12 Dec 2023 08:23:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#9).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/cat

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 7:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14660/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 12 Dec 2023 08:15:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14659/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 12 Dec 2023 08:09:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#8).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/cat

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#7).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#6).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14653/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 12 Dec 2023 02:56:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#5).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. So we can better debug
slow DDLs with the catalog timeline in profiles.

Tried to add some constant strings for widely used events, e.g. "Fetched
table from Metastore". Didn't do so for events that only occurs once.

Most of the catalog methods now have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Example timeline for "REFRESH tpcds.store_sales" when the table is
unloaded:
Catalog Server Operation: 2s300ms
   - Got catalog version read lock: 26.407us (26.407us)
   - Start loading table: 314.663us (288.256us)
   - Got Metastore client: 629.599us (314.936us)
   - Fetched table from Metastore: 7.248ms (6.618ms)
   - Loaded table schema: 27.947ms (20.699ms)
   - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms)
   - Got access level: 1s514ms (588.314us)
   - Created partition builders: 2s103ms (588.270ms)
   - Start loading file metadata: 2s103ms (49.760us)
   - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms)
   - Async loaded table: 2s289ms (6.931ms)
   - Loaded table from scratch: 2s289ms (72.038us)
   - Got table read lock: 2s289ms (2.289us)
   - Finished resetMetadata request: 2s300ms (10.188ms)

Example timeline for an INSERT statement:
Catalog Server Operation: 178.120ms
   - Got catalog version read lock: 4.238us (4.238us)
   - Got catalog version write lock and table write lock: 52.768us (48.530us)
   - Got Metastore client: 15.768ms (15.715ms)
   - Fired Metastore events: 156.650ms (140.882ms)
   - Got Metastore client: 163.317ms (6.666ms)
   - Fetched table from Metastore: 166.561ms (3.244ms)
   - Start refreshing file metadata: 167.961ms (1.399ms)
   - Loaded file metadata for 24 partitions: 177.679ms (9.717ms)
   - Reloaded table metadata: 178.021ms (342.261us)
   - Finished updateCatalog request: 178.120ms (98.929us)

Example timeline for a "COMPUTE STATS tpcds_parquet.store_sales":
Catalog Server Operation: 6s737ms
   - Got catalog version read lock: 19.971us (19.971us)
   - Got catalog version write lock and table write lock: 50.255us (30.284us)
   - Got Metastore client: 171.819us (121.564us)
   - Updated column stats: 25.560ms (25.388ms)
   - Got Metastore client: 69.298ms (43.738ms)
   - Altered 500 partitions in Metastore: 1s894ms (1s825ms)
   - Altered 1000 partitions in Metastore: 3s558ms (1s664ms)
   - Altered 1500 partitions in Metastore: 5s144ms (1s586ms)
   - Altered 1824 partitions in Metastore: 6s205ms (1s060ms)
   - Got Metastore client: 6s205ms (329.481us)
   - Altered table in Metastore: 6s216ms (11.073ms)
   - Got Metastore client: 6s216ms (13.377us)
   - Fetched table from Metastore: 6s219ms (2.419ms)
   - Loaded table schema: 6s223ms (4.130ms)
   - Got current Metastore event id 19017: 6s639ms (415.690ms)
   - Start loading file metadata: 6s639ms (9.591us)
   - Loaded file metadata for 1824 partitions: 6s729ms (90.196ms)
   - Reloaded table metadata: 6s735ms (5.865ms)
   - DDL finished: 6s737ms (2.255ms)

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.
 - Ran CORE tests

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14617/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Fri, 08 Dec 2023 02:35:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/20491

to look at the new patch set (#4).

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This is a follow-up work of IMPALA-12024 where we add the catalog
timeline for CreateTable statements. Using the same mechanism, this
patch adds catalog timeline for all DDL/DML profiles, including
REFRESH and INSERT.

The goal is to add timeline markers after each step that could be
blocked, e.g. acquiring locks, external RPCs. Tried to add some constant
strings for widely used events, e.g. "Fetched table from Metastore".
Didn't do so for events that only occurs once.

Most of the catalog methods have a new argument for tracking the
execution timeline. To avoid adding null checks everywhere, for code
paths that don't need a catalog profile, e.g. EventProcessor, creates an
unused catalogTimeline as the argument. We can use them in future works,
e.g. expose execution timeline of a slow processing on an HMS event.

This patch also removes some unused overloads of HdfsTable#load() and
HdfsTable#reloadPartitionsFromNames().

Tests:
 - Add e2e test to verify the catalog timeline in some DDLs.

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IncompleteTable.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/main/java/org/apache/impala/catalog/MaterializedViewHdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/TableLoader.java
M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/EventSequence.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java
M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
M tests/query_test/test_observability.py
33 files changed, 924 insertions(+), 603 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20491/4
--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/14025/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 19 Sep 2023 11:11:15 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Quanlong Huang (Code Review)]

Quanlong Huang has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/20491


Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..

IMPALA-12443: Add catalog timeline for all ddl profiles

This patch adds catalog timeline for all DDL profiles.

Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
---
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogdTableInvalidator.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/CatalogdTableInvalidatorTest.java
M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java
9 files changed, 445 insertions(+), 315 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20491/1
--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 

[Impala-ASF-CR] IMPALA-12443: Add catalog timeline for all ddl profiles

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/20491 )

Change subject: IMPALA-12443: Add catalog timeline for all ddl profiles
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20491/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/20491/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1165
PS1, Line 1165:   fileFormatParams.getFile_format(), 
numUpdatedPartitions, catalogTimeline);
line too long (92 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/20491
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifbceefaeb24c66eb1a064c449d6f56077ea347c5
Gerrit-Change-Number: 20491
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 19 Sep 2023 10:45:53 +
Gerrit-HasComments: Yes