[
https://issues.apache.org/jira/browse/HIVE-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated HIVE-30196:
-------------------------------
Description:
h3. Problem
{\{get_partitions_with_specs}} (thrift RPC introduced by HIVE-20306) skipped
the \{{hive.metastore.limit.partition.request}} guard that every regular
partition listing RPC enforces (\{{get_partitions}},
\{{get_partitions_by_names}}, \{{get_partitions_with_auth}},
\{{get_partitions_by_filter}}, \{{get_partitions_by_expr}},
\{{get_partitions_ps}}, \{{get_partitions_ps_with_auth}}). The request has no
client-side max parameter, so one call made the metastore materialize all
matching partitions server-side.
h3. Fix
* Fail fast *before* the partition fetch, using the same cheap count queries
the regular listing RPCs use, mapped per filter mode:
** filter spec unset -> \{{rs.getNumPartitionsByFilter(cat, db, tbl, "")}}
** BY_NAMES -> the requested partition names count (as
\{{get_partitions_by_names}} does)
** BY_VALUES -> \{{rs.getNumPartitionsByPs}} (same
\{{MetaStoreUtils.makePartNameMatcher}} predicate as the fetch, both SQL and
JDO paths)
** BY_EXPR -> \{{rs.getNumPartitionsByFilter}} on the joined expressions (same
\{{PartFilterExprUtil}} grammar the projection path parses)
* Keep a check on the fetched result size as the final backstop (covers the
count/fetch TOCTOU window and unknown filter modes)
* Add \{{get_partitions_with_specs}} to the \{{LIMIT_PARTITION_REQUEST}} config
description
Authorization ordering is preserved: \{{firePreEvent}} (pre-event listeners)
still runs before the limit checks.
h3. Behavior changes (only when the operator configures the limit; default -1
is unchanged)
* Requests matching more partitions than the limit now throw \{{MetaException}}
(same message/config as the regular RPCs)
* BY_NAMES uses the strict requested-count semantics, mirroring
\{{get_partitions_by_names}} (requesting more names than the limit fails even
if fewer exist)
* no-filter / BY_VALUES / BY_EXPR requests pay one extra count query per
request; BY_NAMES is free
h3. Tests
Added to \{{TestGetPartitionsUsingProjectionAndFilterSpecs}} (limit=100, table
with 101 partitions): exceed (no filter), at-limit boundary (exactly 100
succeeds), BY_NAMES exceed, BY_NAMES requested-count exceed (150 requested / 50
existing — pins the strict semantics), BY_VALUES exceed, BY_EXPR exceed, and a
JDO-path variant (DirectSQL disabled). All 32 tests plus the related suites
(\{{TestRemoteHiveMetaStore}}, \{{TestEmbeddedHiveMetaStore}},
\{{TestHmsServerAuthorization}}) pass.
h3. Follow-ups
* Push the limit down into
\{{MetaStoreDirectSql.getPartitionIdsViaSqlFilter(max)}} (the projection path
currently passes \{{null}}) to bound the ID query with limit+1 in one round
trip and remove the extra count query + TOCTOU window.
* Related: HIVE-30195 — the same RPC also skips the table-level filter hook
check (\{{checkDbAndTableFilters}}); same class of gap.
> get_partitions_with_specs skips hive.metastore.limit.partition.request
> enforcement performed by all regular partition listing RPCs
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30196
> URL: https://issues.apache.org/jira/browse/HIVE-30196
> Project: Hive
> Issue Type: Sub-task
> Reporter: Yuming Wang
> Priority: Major
>
> h3. Problem
> {\{get_partitions_with_specs}} (thrift RPC introduced by HIVE-20306) skipped
> the \{{hive.metastore.limit.partition.request}} guard that every regular
> partition listing RPC enforces (\{{get_partitions}},
> \{{get_partitions_by_names}}, \{{get_partitions_with_auth}},
> \{{get_partitions_by_filter}}, \{{get_partitions_by_expr}},
> \{{get_partitions_ps}}, \{{get_partitions_ps_with_auth}}). The request has no
> client-side max parameter, so one call made the metastore materialize all
> matching partitions server-side.
> h3. Fix
> * Fail fast *before* the partition fetch, using the same cheap count queries
> the regular listing RPCs use, mapped per filter mode:
> ** filter spec unset -> \{{rs.getNumPartitionsByFilter(cat, db, tbl, "")}}
> ** BY_NAMES -> the requested partition names count (as
> \{{get_partitions_by_names}} does)
> ** BY_VALUES -> \{{rs.getNumPartitionsByPs}} (same
> \{{MetaStoreUtils.makePartNameMatcher}} predicate as the fetch, both SQL and
> JDO paths)
> ** BY_EXPR -> \{{rs.getNumPartitionsByFilter}} on the joined expressions
> (same \{{PartFilterExprUtil}} grammar the projection path parses)
> * Keep a check on the fetched result size as the final backstop (covers the
> count/fetch TOCTOU window and unknown filter modes)
> * Add \{{get_partitions_with_specs}} to the \{{LIMIT_PARTITION_REQUEST}}
> config description
> Authorization ordering is preserved: \{{firePreEvent}} (pre-event listeners)
> still runs before the limit checks.
> h3. Behavior changes (only when the operator configures the limit; default -1
> is unchanged)
> * Requests matching more partitions than the limit now throw
> \{{MetaException}} (same message/config as the regular RPCs)
> * BY_NAMES uses the strict requested-count semantics, mirroring
> \{{get_partitions_by_names}} (requesting more names than the limit fails even
> if fewer exist)
> * no-filter / BY_VALUES / BY_EXPR requests pay one extra count query per
> request; BY_NAMES is free
> h3. Tests
> Added to \{{TestGetPartitionsUsingProjectionAndFilterSpecs}} (limit=100,
> table with 101 partitions): exceed (no filter), at-limit boundary (exactly
> 100 succeeds), BY_NAMES exceed, BY_NAMES requested-count exceed (150
> requested / 50 existing — pins the strict semantics), BY_VALUES exceed,
> BY_EXPR exceed, and a JDO-path variant (DirectSQL disabled). All 32 tests
> plus the related suites (\{{TestRemoteHiveMetaStore}},
> \{{TestEmbeddedHiveMetaStore}}, \{{TestHmsServerAuthorization}}) pass.
> h3. Follow-ups
> * Push the limit down into
> \{{MetaStoreDirectSql.getPartitionIdsViaSqlFilter(max)}} (the projection path
> currently passes \{{null}}) to bound the ID query with limit+1 in one round
> trip and remove the extra count query + TOCTOU window.
> * Related: HIVE-30195 — the same RPC also skips the table-level filter hook
> check (\{{checkDbAndTableFilters}}); same class of gap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
