Siyao Meng created HDDS-16772:
---------------------------------
Summary: Default ACLs of new keys and directories come from the
applying OM's local group mapping, so OMs can store different ACLs
Key: HDDS-16772
URL: https://issues.apache.org/jira/browse/HDDS-16772
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-7-resolve-default-acls-in-preexecute.patch,
TestBugCR7HaFollowerReplayGroupMapping.java
h3. Mechanism
{{OMKeyRequest.getAclsForDir}} and {{getAclsForKey}} run inside
{{validateAndUpdateCache}} and call
{{OzoneAclUtil.getDefaultAclList(createUGIForApi())}}, which resolves the
creator's primary group through the applying OM's own Hadoop {{Groups}}
service. The Raft log entry carries only the user name ({{preExecute}} puts no
ACLs or groups in the request).
{{OMClientRequest}} warns against exactly this ("Do NOT bring external
dependencies into this method, doing so could potentially cause divergence in
OM DB states in HA"). The lookup came in with HDDS-11656.
h3. Trigger
A user creates a key, file, directory or multipart upload while the OMs resolve
that user's primary group differently when they apply the entry. A transient
LDAP or SSSD failure on one OM host is enough (a failed lookup silently drops
the group ACL); a membership change while one OM is down or lagging is another
way. The helpers are reached from the key, file and directory create requests
and from multipart initiate and complete, so LEGACY and OBS buckets are
affected as well as FSO.
h3. Impact
The same log entry stores different default group ACL rows in the directory /
file tables of different OMs, permanently. {{getAcl}} output and, under the
native authorizer, access decisions change depending on which OM serves the
request. The lookup also runs on the apply thread under the bucket lock, so a
slow group lookup stalls the state machine.
h3. Reproduction
PASS: [^TestBugCR7HaFollowerReplayGroupMapping.java]
({{hadoop-ozone/integration-test}}), 3 OM mini cluster, follower stopped and
restarted, with the creator's primary group changed between applies. The test
changes the group mapping inside one shared JVM, which stands in for hosts that
resolve groups differently.
h3. Scope
This covers default ACLs derived from local group mapping only. ACL checks and
bucket link resolution inside apply are already tracked by HDDS-13634 and
HDDS-15460 and are not part of this report. This could be a subtask of
HDDS-13634.
h3. Suggested fix
Resolve the default ACLs once in {{preExecute}} on the leader and carry the
result in the request. {{OMVolumeCreateRequest}} ({{:165}}) and
{{OMBucketCreateRequest}} ({{:341}}) call the same helper inside apply and have
the same problem, confirmed from code; they need their own carrier field.
h3. Patch
[^CR-7-resolve-default-acls-in-preexecute.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
c99d291e863.
{{KeyArgs}} gets {{repeated OzoneAclInfo defaultAcls = 25}}. A new helper,
{{OMKeyRequest.setDefaultAcls}}, clears whatever the client sent, resolves the
default ACLs for the requesting user on the leader and stores them in the
request. It is called from the five {{preExecute}} methods that feed
{{getAclsForKey}} / {{getAclsForDir}} ({{OMKeyCreateRequest}},
{{OMFileCreateRequest}}, {{OMDirectoryCreateRequest}},
{{S3InitiateMultipartUploadRequest}}, {{S3MultipartUploadCompleteRequest}}; the
FSO subclasses inherit them). Apply uses the list from the request and falls
back to the old local lookup when it is empty, which is what an entry from a
leader without this change looks like, so no layout feature is needed.
Review point: apply now trusts {{defaultAcls}}, so any future request type that
reuses these helpers must call {{setDefaultAcls}} in its {{preExecute}}. Volume
and bucket create are not covered by this patch.
Covered by {{testDefaultAclsResolvedInPreExecute}} in the existing
{{TestOMDirectoryCreateRequestWithFSO}} and {{TestOMKeyCreateRequest}}
(inherited by the FSO variant): the user maps to {{group1}} at {{preExecute}}
and {{group2}} at apply, and a client supplied {{defaultAcls}} entry must not
be stored. Without the change they fail with {{["user1", "group2"]}} where
{{["user1", "group1"]}} is expected. All OM request unit suites (1307 tests)
and {{TestOzoneNonHAWithNativeACL}} pass, checkstyle is clean. The proto
backwards compatibility check was not run.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
