[
https://issues.apache.org/jira/browse/KNOX-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18056262#comment-18056262
]
Emil Kleszcz commented on KNOX-3231:
------------------------------------
Hi [~smolnar],
I tested Spark 4 + HS256 as suggested.
Setup:
* Spark 4.0 + Java 17
* _org.apache.spark.ui.JWSFilter_ enabled with base64url HS256 secret
* Generated HS256 JWT using the decoded HMAC key
{code:java}
curl https://host:18080/ -> 403
curl -H "Authorization: Bearer <valid token>" -> 200{code}
So the Spark 4 _JWSFilter_ works correctly and accepts properly signed HS256
tokens.
However, in our deployment, Knox does not reverse-proxy the Spark History UI.
nginx returns a 301 redirect to _:18080_ instead. Because requests then bypass
Knox, no Authorization header is injected, and the JWSFilter always rejects the
browser request.
For context, we previously tried to _proxy_ Spark HS UI through Knox, but ran
into persistent UI breakage (static assets / rewrite quirks and CORS/302 issues
on the _/api/v1/applications_ XHR). As a workaround, we switched to an nginx
"redirect-to-service-port" pattern (similar to HDFS meta-redirect), which is
why Knox currently does a 301 to _:18080_ instead of reverse proxying. Ref.
[reported issue in Cloudera
forum|https://community.cloudera.com/t5/Support-Questions/Spark-History-CORS-header-Access-Control-Allow-Origin/m-p/391343]
My conclusion:
* Spark side: works
* Knox + redirect model: incompatible with Bearer/JWT protection
For now, we will secure the HS using Knox-only exposure + FW rules. JWT-based
protection would require true proxying rather than redirect.
Thanks for pointing me to the HS256 support.
> Spark 4 History Server cannot be protected by Knox JWT after migration to
> jakarta.servlet
> -----------------------------------------------------------------------------------------
>
> Key: KNOX-3231
> URL: https://issues.apache.org/jira/browse/KNOX-3231
> Project: Apache Knox
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Emil Kleszcz
> Priority: Minor
>
> After upgrading to Apache Spark 4, the Spark History Server can no longer be
> secured using the Knox-based JWT authentication mechanism that worked with
> Spark 3.
> Root cause:
> - Spark 4 migrated from javax.servlet to jakarta.servlet
> - The Hadoop AuthenticationFilter used previously via Knox depends on
> javax.servlet
> - This makes the filter incompatible and prevents the History Server from
> starting
> - Spark 4 introduces org.apache.spark.ui.JWSFilter, but it requires HS256 JWT
> tokens
> - Knox currently injects Spark 3–style JWTs that are not compatible with
> JWSFilter
> Impact:
> - Spark 4 HS cannot currently be protected via Knox using JWT
> - Only two insecure or partial workarounds exist:
> - Run the HS without any UI authentication filter
> - Rely solely on network/firewall protection + Knox proxying
> This means:
> - Spark 4 HS is functionally working
> - Secure UI authentication via Knox is currently broken
> Environment:
> - Apache Spark: 4.0.1
> - Java: 17
> - Knox-proxied Spark HS
> - Previously working with Spark 3 using Hadoop AuthenticationFilter + Knox JWT
> Expected behaviour:
> - Knox should be able to protect the Spark 4 HSUI using a supported
> authentication mechanism (either via HS256-compatible tokens or an
> alternative)
> Open questions:
> - Is this a known incompatibility between Knox and Spark 4?
> - Is HS256 token generation for Spark 4 JWSFilter supported or planned in
> Knox?
> - Is there a recommended authentication pattern for securing the Spark 4 HS
> behind Knox?
> My workaround in production:
> - Knox proxying + strict firewall rules blocking direct access to the HS UI
> ports
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
