You need to use the spark.ui.filters setting on the history server https://spark.apache.org/docs/latest/configuration.html#spark-ui:
spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.type=org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.authentication.provider.url=https:// <knox-hostname>:8443/gateway/knoxsso/api/v1/websso ...etc On Thu, Jul 11, 2024 at 4:18 PM Thomas Mauran <thomas.mau...@etu.umontpellier.fr.invalid> wrote: > Hello, > I am sending this email to the mailing list, to get your help on a problem > that I can't seem to resolve myself. > > I am trying to secure Spark history ui running with Yarn as master using > Apache Knox. > > From the Knox configuration point of view I managed to secure the Spark > service, if I go on https://:8443/gateway/default/spark3history I have to > login using SSO then I get redirected to spark history server web ui which > works as expected. > > But if I directly access Spark without getting logged in I don't get > redirected to Knox login page which is what I would like to have, same as > HDFS and YarnUI. > > From what I see in Spark documentation the webui needs to be protected > using the filter system. I can' t seem to find a filter to protect my Spark > history UI using Knox, I protected both HDFS and Yarn by adding this in > core-site.xml which works fine. > > <property> > <name>hadoop.http.authentication.type</name> > > <value>org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler</value></property> > <property> > <name>hadoop.http.authentication.authentication.provider.url</name> > > <value>https://<knox-hostname>:8443/gateway/knoxsso/api/v1/websso</value></property> > > > <property> > <name>hadoop.http.authentication.public.key.pem</name> > <value><token></value></property> > > Adding those properties allowed me to get redirected to knox host page > when I didn' t login yet. > > I am wondering if you knew how to secure Spark history UI to have the same > behavior. > > Do you know what configuration I am missing to redirect it back to the > Knox gateway login page from the Spark history UI as for the other services > where the JWT token is passed and used for keeping the user session ? > > I tried to play with the filters especially > org.apache.hadoop.security.authentication.server.AuthenticationFilter but > didn' t manage to get anything working, so I don' t even know if this is > the right way to do. > > Thanks for your answer > > -- Adam Binford
