[
https://issues.apache.org/jira/browse/KNOX-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18044634#comment-18044634
]
Sandor Molnar commented on KNOX-3231:
-------------------------------------
Hi [~ekleszcz] !
Let me try to answer your open questions:
{quote} Is this a known incompatibility between Knox and Spark 4? {quote}
It is now :) I'm afraid nobody reported that issue before. Thanks for pointing
this out.
{quote}Is HS256 token generation for Spark 4 JWSFilter supported or planned in
Knox?{quote}
We added the ability to switch to HMAC signatures in Knox-generated tokens.
That change is available since 1.6.0. See details in KNOX-2527 (and testing in
the attached PR).
{quote}Is there a recommended authentication pattern for securing the Spark 4
HS behind Knox?{quote}
I guess you want to test out the above-referenced HS256 token signature
configuration together with Spark 4 HS. If that's working, we can see it as the
recommended way.
Thanks for filing this JIRA and we are waiting for your thoughts.
> Spark 4 History Server cannot be protected by Knox JWT after migration to
> jakarta.servlet
> -----------------------------------------------------------------------------------------
>
> Key: KNOX-3231
> URL: https://issues.apache.org/jira/browse/KNOX-3231
> Project: Apache Knox
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Emil Kleszcz
> Priority: Minor
>
> After upgrading to Apache Spark 4, the Spark History Server can no longer be
> secured using the Knox-based JWT authentication mechanism that worked with
> Spark 3.
> Root cause:
> - Spark 4 migrated from javax.servlet to jakarta.servlet
> - The Hadoop AuthenticationFilter used previously via Knox depends on
> javax.servlet
> - This makes the filter incompatible and prevents the History Server from
> starting
> - Spark 4 introduces org.apache.spark.ui.JWSFilter, but it requires HS256 JWT
> tokens
> - Knox currently injects Spark 3–style JWTs that are not compatible with
> JWSFilter
> Impact:
> - Spark 4 HS cannot currently be protected via Knox using JWT
> - Only two insecure or partial workarounds exist:
> - Run the HS without any UI authentication filter
> - Rely solely on network/firewall protection + Knox proxying
> This means:
> - Spark 4 HS is functionally working
> - Secure UI authentication via Knox is currently broken
> Environment:
> - Apache Spark: 4.0.1
> - Java: 17
> - Knox-proxied Spark HS
> - Previously working with Spark 3 using Hadoop AuthenticationFilter + Knox JWT
> Expected behaviour:
> - Knox should be able to protect the Spark 4 HSUI using a supported
> authentication mechanism (either via HS256-compatible tokens or an
> alternative)
> Open questions:
> - Is this a known incompatibility between Knox and Spark 4?
> - Is HS256 token generation for Spark 4 JWSFilter supported or planned in
> Knox?
> - Is there a recommended authentication pattern for securing the Spark 4 HS
> behind Knox?
> My workaround in production:
> - Knox proxying + strict firewall rules blocking direct access to the HS UI
> ports
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
