In addition, the Spark DT framework does not work correctly in Java 25 (especially Spark on K8s mode) due JDK 22+ Subject propagation change. HADOOP-19668[1] partially solves the issue but does not fully meet Spark requirements, I proposed a simple approach HADOOP-19906[2][3] as an alternative, hoping to get reviews by folks who know UGI stuff well.
[1] https://issues.apache.org/jira/browse/HADOOP-19668 [2] https://issues.apache.org/jira/browse/HADOOP-19906 [3] https://github.com/apache/hadoop/pull/8522 Thanks, Cheng Pan > On Jun 23, 2026, at 10:48, Cheng Pan <[email protected]> wrote: > > I think this SPIP functionality has overlaps with [1] - both SPIPs solve > application-level credentials, and I’d like to know if you have plan to > extend this SPIP to support session-level credentials in the future. As a > reference, Kousuke Saruta and I had some high-level discussions on this in > [2]. > > [1] https://lists.apache.org/thread/cy2wqt9xtt5fh0qwmsfx9ht762cqpqs1 > [2] > https://docs.google.com/document/d/1usJKncCPMiyFUg7aIdpZ0HQsklXIHow_sU_6dfFMjN0/edit?disco=AAAB88lRrIA > > Thanks, > Cheng Pan > > > >> On Jun 23, 2026, at 02:52, Steve Loughran <[email protected]> wrote: >> >> yeah, hadoop dt interface doesn't say anything about kerberos being >> required, it is just that spark doesn't ask for dt unless security is >> enabled. s3a and abfs connectors will, if delegation tokens are enabled for >> them, happily issue their tokens >> >> HadoopDelegationTokenProvider *does not require kerberos*. it is just that >> spark doesn't ask filesystems for tokens without it. >> >> You can see this with the fetchdt command of cloudstore >> https://github.com/steveloughran/cloudstore >> https://github.com/steveloughran/cloudstore/blob/main/src/site/markdown/fetchdt.md >> >> point it an FS and it'll ask for them; >> https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/delegation_tokens.html >> >> azure can do the same with fs.azure.enable.delegation.token and a token >> type, such as oauth. >> >> for example, s3a set to ask for session tokens in an extra restricted role >> <property> >> <name>fs.s3a.delegation.token.role.arn</name> >> <value>${ARN-restricted}</value> >> </property> >> >> <property> >> <name>fs.s3a.delegation.token.binding</name> >> >> <value>org.apache.hadoop.fs.s3a.auth.delegation.SessionTokenBinding</value> >> </property> >> >> >> then you can use cloudstore (which is soon to have its first asf release, >> but currently needs to be downloaded from >> https://github.com/steveloughran/cloudstore ) >> >> > bin/hadoop jar $CLOUDSTORE fetchdt out.tokens s3a://stevel-london/ >> >> Collecting tokens for 1 filesystem to to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> 2026-06-22 19:39:30,652 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:<init>(84)) - Starting: Fetching tokens for >> s3a://stevel-london/ >> 2026-06-22 19:39:32,204 [main] INFO delegation.S3ADelegationTokens >> (DurationInfo.java:<init>(77)) - Starting: Creating New Delegation Token >> 2026-06-22 19:39:32,282 [main] INFO auth.STSClientFactory >> (STSClientFactory.java:lambda$requestSessionCredentials$0(227)) - Requesting >> Amazon STS Session credentials >> 2026-06-22 19:39:32,936 [main] INFO delegation.S3ADelegationTokens >> (S3ADelegationTokens.java:noteTokenCreated(443)) - Created S3A Delegation >> Token: Kind: S3ADelegationToken/Session, Service: s3a://stevel-london, >> Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; >> uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; >> encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on >> VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time >> 2026-06-22T18:39:32.212268Z.}; session credentials, expiry >> 2026-06-23T06:39:32Z; (valid)) >> 2026-06-22 19:39:32,938 [main] INFO delegation.S3ADelegationTokens >> (DurationInfo.java:close(98)) - Creating New Delegation Token: duration >> 0:00.732s >> Fetched token: Kind: S3ADelegationToken/Session, Service: >> s3a://stevel-london, Ident: (S3ATokenIdentifier{S3ADelegationToken/Session; >> uri=s3a://stevel-london; timestamp=1782153572890; renewer=stevel; >> encryption=SSE-KMS; 7eef8fbd-7a09-4a7e-a9ee-79a631c9f469; Created on >> VXM63P4JG2/192.168.50.99 <http://192.168.50.99/> at time >> 2026-06-22T18:39:32.212268Z.}; session credentials, expiry >> 2026-06-23T06:39:32Z; (valid)) >> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:close(190)) - Duration of Fetching tokens for >> s3a://stevel-london/: 00:00:02.290 >> 2026-06-22 19:39:32,941 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:<init>(84)) - Starting: Saving 1 token to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> 2026-06-22 19:39:33,233 [main] INFO commands.FetchTokens >> (StoreDurationInfo.java:close(190)) - Duration of Saving 1 token to >> file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens: 00:00:00.292 >> Saved 1 token to file:/Users/stevel/Projects/Releases/hadoop-3.5.0/out.tokens >> >> Token issued; no kerberos around and the file now has session credentials >> valid for 12h and fs encryption settings included. >> >> Accordingly, I'm going to suggest a different design >> >> Don't bother with a new subclass >> Simply add a switch to enable token collection even if kerberos is off >> For a test, add a subclass of file:// with a new url and see if you can >> issue tokens off it. More rigorously, add a switch enabling it to blow up >> during creation/unmarshalling to test spark reslience >> The hard part here is actually implementing any test DT implementation; if >> you can avoid that your life is better. >> >> On Thu, 18 Jun 2026 at 11:55, Peter Toth <[email protected] >> <mailto:[email protected]>> wrote: >>> Hi Parth, >>> >>> Thanks for the SPIP and for answering our questions. >>> Does anyone have any other questions or points they'd like to discuss? >>> >>> Peter >>> >>> On Thu, Jun 4, 2026 at 10:02 PM Parth Chandra <[email protected] >>> <mailto:[email protected]>> wrote: >>>> Hi all, >>>> I've a proposal to enhance the current mechanism to distribute >>>> delegation tokens and other secure tokens. >>>> >>>> The summary is that the current mechanism is gated behind Kerberos even >>>> though the actual distribution does not require Kerberos except where the >>>> tokens themselves are Kerberos tokens. Cloud environments may not have a >>>> Kerberos setup and this creates an unnecessary setup step that users may >>>> have to perform. The current implementation of >>>> KafkaDelegationTokenProvider illustrates this. The implementation does not >>>> require Kerberos, yet it has to pass the Kerberos gates. >>>> >>>> The proposal then is to allow a second path that does not require the >>>> Kerberos gates unless the provider indicates that it be required. the >>>> design has minimal change to the existing code and is fully backward >>>> compatible. >>>> >>>> The proposal and corresponding JIRA are in [1], [2] >>>> >>>> I'd greatly appreciate it if committers can take some time to review and >>>> provide feedback >>>> >>>> Thanks >>>> >>>> Parth >>>> [1] >>>> https://docs.google.com/document/d/1PPqAoJAj48MdjMJNc7DlytXi745z-imFpVaFDnt18Xg/edit?tab=t.0#heading=h.21tncge82jbl >>>> [2] https://issues.apache.org/jira/browse/SPARK-57252 >
