[
https://issues.apache.org/jira/browse/IMPALA-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107597#comment-18107597
]
ASF subversion and git services commented on IMPALA-15229:
----------------------------------------------------------
Commit 892f15d128ab082224db5d1349c0d8cea57848b4 in impala's branch
refs/heads/master from Yubi Lee
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=892f15d12 ]
IMPALA-15229: Derive the EC Policy column from file metadata, not RPCs
SHOW TABLE STATS, SHOW PARTITIONS and SHOW FILES report an "EC Policy"
column that is populated by FileSystemUtil.getErasureCodingPolicy(),
which issues one NameNode RPC per path, uncached, from a serial loop.
The statements therefore cost O(#partitions) or O(#files) round trips
to populate a display-only column: on a table with ~105k partitions a
single SHOW TABLE STATS produced ~105k serial getErasureCodingPolicy
RPCs, and NameNode audit logs showed 952k such calls in a five minute
window, tripping RPC-spike alerts on clusters that do not use erasure
coding at all (where every call returns NONE).
The information is already available for free: the directory listing
that loads the file metadata returns HdfsFileStatus objects that carry
the erasure coding policy of each file. This change records the policy
id (a single byte) in the file descriptor at load time and derives the
EC Policy column from it, eliminating the per-path RPCs. The recorded
value also describes how the data is actually stored, rather than which
policy the directory would assign to new files.
Details:
- FbFileDesc gains 'ec_policy_id: byte = 0 (id: 8)'. 0 is the id of
the replication policy, i.e. "not erasure-coded or unknown". The
field is additive with an explicit id, so catalog objects remain
compatible in both directions and rolling upgrades keep working:
descriptors written by older versions read as 0, which makes the new
code fall back to the old per-path lookup for erasure-coded files.
'is_ec' is untouched and remains what scheduling relies on.
- FileDescriptor records the id extracted from the FileStatus (a purely
local call, no RPC) and exposes getEcPolicyId().
- Policy ids resolve to names locally through
SystemErasureCodingPolicies for the built-in policies. Names of
user-defined policies are fetched with a single
getAllErasureCodingPolicies() call per filesystem and cached. The
cache is refreshed once when an id missing from it is looked up, so
policies added to the NameNode while Impala is running still resolve
without falling back to per-path lookups.
- SHOW TABLE STATS / SHOW PARTITIONS aggregate the descriptors of the
partition: NONE if no file is erasure-coded, the policy name if all
files share one policy, and the new label MIXED otherwise, including
when an erasure-coded file with an unknown policy id is mixed with
plain files. Partitions with no files fall back to the previous
per-path lookup, as do partitions where every file is erasure-coded
but the policy id of some is unknown (older metadata, or filesystems
like Ozone whose listing does not carry the policy).
- SHOW FILES resolves the policy from each file's descriptor directly.
- DESCRIBE FORMATTED and the Iceberg table-level row keep the existing
lookup, which is one RPC per statement. The per-file lookups in the
Paimon SHOW FILES path iterate Paimon file handles that have no
Impala file descriptor and are left for a follow-up.
Behaviour change: for a partition whose directory has an erasure coding
policy set but whose files are not erasure-coded (or the reverse), the
column now describes the files instead of the directory attribute, and
mixed contents report MIXED.
Testing:
- FileSystemUtilTest covers the policy id extraction from plain and
HDFS file statuses and the local resolution of system policy names.
- FeFsTableTest covers the aggregation of a partition's descriptors:
NONE, a single policy resolved locally, and the MIXED combinations,
including erasure-coded files with unknown policy ids.
- Existing EC end-to-end tests exercise the single-policy path on an
erasure-coded minicluster via $ERASURECODE_POLICY.
Change-Id: I35fc413c4a1f4730baab55406b13f5541f379707
Reviewed-on: http://gerrit.cloudera.org:8080/24642
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> SHOW TABLE STATS / SHOW PARTITIONS / SHOW FILES issue one NameNode RPC
> perpartition/file for the display-only EC Policy column
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15229
> URL: https://issues.apache.org/jira/browse/IMPALA-15229
> Project: IMPALA
> Issue Type: Improvement
> Reporter: YUBI LEE
> Priority: Major
>
> {{FileSystemUtil.getErasureCodingPolicy(Path)}} issues one
> {{getErasureCodingPolicy}} RPC per
> path, uncached, from a serial loop, called once per partition by SHOW TABLE
> STATS / SHOW
> PARTITIONS and once per file by SHOW FILES — only to fill a display-only
> column. Nothing
> in planning, scheduling, or the read path consumes it
> ({{FileDescriptor.isEc}}, sourced from
> the listing, covers that separately).
> Production impact (non-EC HDFS cluster): one SHOW TABLE STATS on a
> ~105k-partition table
> generated ~105k serial NameNode RPCs; audit logs showed ~952k calls in a
> 5-minute window,
> tripping RPC-spike alerts and adding tens of seconds of statement latency.
> Every call
> returned NONE.
> The listing that loads file metadata already carries the EC policy in
> {{HdfsFileStatus}}.
> Record the policy id in the file descriptor at load time and derive the
> column from it;
> resolve ids to names locally (system policies) or with one cached
> {{getAllErasureCodingPolicies()}} call per filesystem. Fall back to the
> per-path lookup only
> for empty partitions and metadata written before the upgrade.
> Alternatives considered:
> - Removing the column introduced by IMPALA-9487 altogether. If the community
> feels the
> column does not carry its weight, that is also a valid resolution — but it
> changes the
> output schema of shipped statements that tools and tests parse, so it would
> need a
> deprecation discussion. Deriving the value from metadata we already have
> keeps the
> feature at roughly zero cost, which seemed like the less disruptive fix.
> - A Hadoop configuration switch to skip the lookups (in the style of
> impala.preload-block-locations-for-scheduling). This works as a mitigation
> but leaves
> the default behaviour expensive and drops the information where it is
> disabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
