[
https://issues.apache.org/jira/browse/HADOOP-20041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135432#comment-18135432
]
Konstantin Bereznyakov commented on HADOOP-20041:
-------------------------------------------------
One more execution path, relevant when the S3A shared async scheduler from
HADOOP-19862 is enabled
({{fs.s3a.aws.s3.async.client.shared.threadpool.enabled}}; HADOOP-20044
proposes making it the default).
Measured locally with SDK 2.35.4 and JDK 17 (not part of the attached
reproducer), replacing each S3A shared scheduler with a recording pool:
* The S3 async scheduler runs the start of each delayed attempt, such as a
retry: its signer and {{beforeTransmission}} interceptor calls ran on
{{s3a-s3-async-scheduler}}.
* It also runs the work that follows a timed-out request. After a part copy
timed out, the SDK's multipart cleanup issued an {{AbortMultipartUpload}} on
that thread, including the credentials-provider lookup, the request-side
interceptor calls and the signer.
* Those calls ran as the user who started the pool thread, not the requesting
user. Signing keys stayed the requester's.
* The S3 sync, STS and KMS schedulers ran only timeout tasks; no extension code
ran on them, including after retries and fired timeouts.
This affects only custom credential providers, signers or execution
interceptors that read the current Hadoop user; S3A's own implementations read
it only when they are created. A fix that binds the filesystem owner to the
extension objects, rather than relying on thread context, would cover this path
as well.
> S3A: AWS SDK extension callbacks can observe another filesystem user's UGI
> --------------------------------------------------------------------------
>
> Key: HADOOP-20041
> URL: https://issues.apache.org/jira/browse/HADOOP-20041
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 3.5.0
> Reporter: Konstantin Bereznyakov
> Priority: Major
> Attachments: HADOOP-20041-manual-trunk-a2d306c65a8.txt,
> ReleaseRetryIdentityProbe.java.txt, pom.xml.txt, run.sh.txt
>
>
> h2. Background
> During review of HADOOP-19862 ([apache/hadoop PR
> 8426|https://github.com/apache/hadoop/pull/8426]), [~stevel] has [raised the
> concern about shared UGI subjects on shared-user
> systems|https://github.com/apache/hadoop/pull/8426#issuecomment-4802130168].
> That discussion concerned the proposed shared scheduled executors for AWS SDK
> clients and whether to enable them by default.
> Subsequent investigation demonstrated related ambient-UGI behaviour on the
> SDK's existing Netty event-loop and completion threads, independently of
> those shared scheduled executors. It reproduces in released Hadoop 3.5.0,
> which does not contain HADOOP-19862, and on current trunk with all four
> shared-scheduler flags disabled.
> This report follows up on that review concern. It is not reporting a
> regression introduced by HADOOP-19862. Enabling the shared schedulers adds a
> separate execution path requiring consideration; keeping them disabled does
> not eliminate the existing behaviour described here.
> h2. Summary
> S3A extension code invoked by the AWS SDK can observe the UGI of another
> filesystem user when running on SDK Netty event-loop or completion threads.
> This was reproduced with Hadoop 3.5.0 and freshly built trunk at
> 6980beafb9ed7d672c0a61bd4eec1a3246db1ae1, using AWS SDK 2.35.4.
> h2. Reproduction
> The standalone Maven reproduction uses a local HTTP S3 stand-in and synthetic
> credentials. No AWS account or real credentials are required.
> * Create separate Alice and Bob UGIs.
> * Under Alice.doAs, create enough separate S3A filesystem instances and
> perform asynchronous copies to start the SDK's shared Netty event-loop
> threads. Keep those filesystem instances open.
> * Under Bob.doAs, create another filesystem instance and rename an object.
> The local endpoint returns one 503 response so that the SDK also exercises a
> retry.
> * Record the executing thread and UserGroupInformation.getCurrentUser() in
> an execution interceptor, a custom signer delegating to AwsS3V4Signer, and a
> credentials provider.
> * Repeat with the classic input stream and a 15 MiB object, using a 5 MiB
> multipart threshold and part size, so the copy uses multipart follow-up
> requests.
> * As a control, repeat the same scenarios but close Alice's filesystem
> instances before starting Bob's operation.
> The credentials provider returns a distinct, configuration-bound synthetic
> access key for each filesystem. The endpoint records the access key used to
> sign every request.
> h2. Observed behaviour
> With Alice's filesystem instances still open, Bob's callbacks on
> aws-java-sdk-NettyEventLoop threads observe Alice's UGI. Callbacks on Bob's
> per-client sdk-async-response completion threads also observe Alice.
> For the multipart copy, the completion threads execute 10
> credentials-provider lookups and 5 signer calls under Alice's ambient UGI.
> ||Hadoop version||JDK||Alice's filesystems remain open||Alice's filesystems
> closed before Bob starts||
> |3.5.0|17, 21|Alice on Netty/completion callbacks|Bob|
> |3.5.0|25|Login user on SDK worker callbacks|Login user|
> |Trunk 6980beafb9e|17, 21, 25|Alice on Netty/completion callbacks|Bob|
> The caller-thread callbacks run as Bob. On trunk with the shared schedulers
> disabled, per-client retry signing also runs as Bob.
> Every recorded Bob request is signed with Bob's synthetic access key: 10
> requests in the non-multipart scenario and 15 in the multipart scenario. The
> demonstrated mismatch is the ambient UGI seen by extension code, not use of
> Alice's signing credentials.
> h2. Thread-context mechanism
> The SDK uses a JVM-wide shared Netty event-loop group. Its threads can retain
> the context inherited when they were created.
> Although the SDK completion executor is per client, its workers are created
> lazily and can be created from an event-loop thread carrying another user's
> context. A per-client completion executor therefore does not ensure that its
> worker threads carry that client's filesystem-owner UGI.
> HADOOP-19906 accounts for the JDK 25 difference above: trunk restores Subject
> inheritance on newer JDKs, while released 3.5.0 falls back to the login user
> in these worker-thread calls.
> h2. Expected behaviour and impact
> An extension selecting per-filesystem or per-user state should not
> inadvertently select another filesystem user's state because an unrelated
> client started the SDK threads first.
> This matters for extensions that consult the current UGI. Hadoop's
> ITestCustomSigner.CustomSigner.sign() itself uses
> UserGroupInformation.getCurrentUser() to look up bucket/user-specific state,
> so this is an existing S3A extension pattern.
> If preserving the filesystem owner's UGI is not the intended callback
> contract, the integration needs a supported way for such extensions to
> identify the filesystem owner independently of the executing thread.
> h2. Scope
> * The issue is reproducible without enabling the shared scheduled executors.
> * Both non-multipart and classic-stream multipart copies were exercised.
> * Fresh trunk verification covered four scenarios on each of JDK 17, 21 and
> 25.
> * No incorrect AWS signing key, unauthorized access or data corruption was
> demonstrated.
> * The standalone reproduction establishes the UGI observations above.
> Separate earlier trunk characterizations observed context-class-loader and
> CallerContext carryover, but those are not measured by this standalone
> reproduction.
> * These observations do not establish an independent defect in Netty or
> UserGroupInformation itself.
> * No concrete patch is proposed yet. The intended context contract and the
> appropriate boundary for supplying filesystem-owner context to S3A extensions
> may need discussion.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
