[
https://issues.apache.org/jira/browse/IMPALA-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858405#comment-16858405
]
Todd Lipcon commented on IMPALA-8606:
-------------------------------------
Seems we need to add some interface like FeCatalog.getTableIfCached() which
returns the table object if it's already resident, or otherwise avoids doing
any round trips.
That said, the user visible behavior here ends up a bit goofy -- stuff like
comments are silently missing for unloaded tables. On catalog v1 that was
already the case, but given v1 was much more eager about caching, it would be
less likely to be visible. For v2 people are more likely to notice. Any ideas?
> GET_TABLES performance in local catalog mode
> --------------------------------------------
>
> Key: IMPALA-8606
> URL: https://issues.apache.org/jira/browse/IMPALA-8606
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.2.0
> Reporter: Balazs Jeszenszky
> Assignee: Todd Lipcon
> Priority: Critical
>
> With local catalog mode enabled, GET_TABLES JDBC requests will return more
> than the always available table information. Any request for more metadata
> about a table will trigger a full load of that table on the catalogd side,
> meaning that GET_TABLES triggers the load of the entire catalog. Also, as far
> as I can see, the requests for more metadata are made one table at a time.
> Once the tables are loaded, the coordinator needs 3 roundtrips to the catalog
> to fetch all the details about a single table. My test case had around 57k
> tables, 1700 DBs, and ~120k partitions.
> GET_TABLES on a cold catalog takes 18 minutes. With a warm catalog, but cold
> impalad, it still takes ~70 seconds.
> Many tools use GET_TABLES to populate dropdowns, etc. so this is bad for both
> end user experience and catalog memory usage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
