pvary commented on PR #17317:
URL: https://github.com/apache/iceberg/pull/17317#issuecomment-5057621678
> **1. OOM** — Theoretical so far; we haven't hit it in production. The old
path loads a full `Table` object per table into client memory via
`getTableObjectsByName`, so memory grows unbounded with namespace size — that's
the risk.
Thanks for clarifying
> **2. Timeout** — Covered in #17312: a namespace with ~46,000 tables still
timed out even at `hive.metastore.client.socket.timeout=900s`. The timeout is
raised on the HMS client side, since serializing/transferring that many full
`Table` objects exceeds the socket timeout. With the new server-side filter,
the same call returns in about 5 seconds.
Based on the description of #17312, the timeout is in the
`getTableObjectsByName` call.
The issue is that we send too many tables in a single call, and HMS is not
able to respond in a timely manner.
> **3. New config** — `list-iceberg-tables-legacy` is just a defensive
fallback. The old path has run stably in production for a long time, so the
flag lets users revert if the new server-side-filter path hits an unexpected
issue. It's off by default.
Do we expect any issues with the new strategy? Seems like a quite nice
optimisation, without any obvious user facing drawback. Also we prefer as few
configs as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
