[
https://issues.apache.org/jira/browse/IMPALA-12870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-12870:
------------------------------------
Epic Link: IMPALA-12877
> Tag TableLoader logs with query ids
> -----------------------------------
>
> Key: IMPALA-12870
> URL: https://issues.apache.org/jira/browse/IMPALA-12870
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
>
> In local-catalog mode, the reason of table metadata loading is logged as
> "needed by coordinator". It'd be helpful to tag the logs with the query ids.
> So we can confirm which query actually triggers metadata loading on the table.
> Here are the catalogd logs for a REFRESH statement in local-catalog mode. The
> first part is loading the metadata of the table triggered by the planning
> phase in the coordinator. No query ids are tagged in this part. The second
> part starts with the resetMetadata request. Logs are tagged with query ids.
> That's what we want.
> {noformat}
> I0305 09:05:32.802394 11245 TableLoadingMgr.java:72] Loading metadata for
> table: functional.alltypestiny
> I0305 09:05:32.802747 11336 TableLoader.java:76] Loading metadata for:
> functional.alltypestiny (needed by coordinator)
> I0305 09:05:32.803429 11245 TableLoadingMgr.java:74] Remaining items in
> queue: 0. Loads in progress: 1
> I0305 09:05:32.831334 11336 Table.java:244] createEventId_ for table:
> functional.alltypestiny set to: -1
> I0305 09:05:32.831614 11336 HdfsTable.java:1251] Loading metadata for table
> definition and all partition(s) of functional.alltypestiny (needed by
> coordinator)
> I0305 09:05:32.846228 11336 HdfsTable.java:1886] Loaded 13 columns from HMS.
> Actual columns: 13
> I0305 09:05:32.867285 11336 HdfsTable.java:3065] Load Valid Write Id List
> Done. Time taken: 8.479us
> I0305 09:05:32.867355 11336 HdfsTable.java:1291] Fetching partition metadata
> from the Metastore: functional.alltypestiny
> I0305 09:05:32.870157 11336 MetaStoreUtil.java:190] Fetching 4 partitions
> for: functional.alltypestiny using partition batch size: 1000
> I0305 09:05:32.887723 11336 MetaStoreUtil.java:208] Fetched 4/4 partitions
> for table functional.alltypestiny
> I0305 09:05:32.887784 11336 HdfsTable.java:1298] Fetched partition metadata
> from the Metastore: functional.alltypestiny
> I0305 09:05:32.893668 11336 ParallelFileMetadataLoader.java:224] Loading file
> and block metadata for 4 paths for table functional.alltypestiny using a
> thread pool of size 4
> I0305 09:05:32.898553 11336 HdfsTable.java:821] Loaded file and block
> metadata for functional.alltypestiny partitions: year=2009/month=1,
> year=2009/month=2, year=2009/month=3, and 1 others. Time taken: 5.429ms
> I0305 09:05:32.899248 11336 Table.java:1104] last refreshed event id for
> table: functional.alltypestiny set to: -1
> I0305 09:05:32.899305 11336 TableLoader.java:176] Loaded metadata for:
> functional.alltypestiny (96ms)
> I0305 09:05:32.930579 11245 JniUtil.java:166]
> c740fb0d001885d9:6a12143000000000] resetMetadata request: REFRESH TABLE
> functional.alltypestiny issued by quanlong
> I0305 09:05:32.955250 11245 CatalogServiceCatalog.java:2642]
> c740fb0d001885d9:6a12143000000000] Refreshing table metadata:
> functional.alltypestiny
> I0305 09:05:32.963577 11245 HdfsTable.java:1251]
> c740fb0d001885d9:6a12143000000000] Reloading metadata for table definition
> and all partition(s) of functional.alltypestiny (REFRESH TABLE
> functional.alltypestiny issued by quanlong)
> I0305 09:05:32.966183 11245 HdfsTable.java:1886]
> c740fb0d001885d9:6a12143000000000] Loaded 13 columns from HMS. Actual
> columns: 13
> I0305 09:05:32.975786 11245 HdfsTable.java:3065]
> c740fb0d001885d9:6a12143000000000] Load Valid Write Id List Done. Time taken:
> 7.288us
> I0305 09:05:32.981313 11245 ParallelFileMetadataLoader.java:224]
> c740fb0d001885d9:6a12143000000000] Refreshing file and block metadata for 4
> paths for table functional.alltypestiny using a thread pool of size 4
> I0305 09:05:32.985464 11245 HdfsTable.java:821]
> c740fb0d001885d9:6a12143000000000] Loaded file and block metadata for
> functional.alltypestiny partitions: year=2009/month=1, year=2009/month=2,
> year=2009/month=3, and 1 others. Time taken: 5.839ms
> I0305 09:05:32.985622 11245 HdfsTable.java:1289]
> c740fb0d001885d9:6a12143000000000] Incrementally loaded table metadata for:
> functional.alltypestiny
> I0305 09:05:32.985924 11245 CatalogServiceCatalog.java:2703]
> c740fb0d001885d9:6a12143000000000] Refreshed table metadata:
> functional.alltypestiny
> I0305 09:05:32.985971 11245 Table.java:1104]
> c740fb0d001885d9:6a12143000000000] last refreshed event id for table:
> functional.alltypestiny set to: -1
> I0305 09:05:32.988221 11245 JniUtil.java:177]
> c740fb0d001885d9:6a12143000000000] Finished resetMetadata request: REFRESH
> TABLE functional.alltypestiny issued by quanlong. Time spent: 58ms{noformat}
> CC [~fangyurao]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
