Hello, I am writing this email to get your help on an issue with my Apache knox configuration.
I am facing a problem on both YarnUI and Sparkhistory UI where I have to write myself rewrite rules for static files like css or js ones. For example when trying to access https://<knox>:8443/gateway/default/yarn/ I get 404 errors on the following files: gateway-audit.log 8:17 ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/yarn.css|unavailable|Request method: GET 24/07/11 15:18:17 ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/yarn.css|success| 24/07/11 15:18:17 ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/yarn.css|success|Response status: 404 24/07/11 15:18:17 ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|unavailable|Request method: GET 24/07/11 15:18:17 ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|success| 24/07/11 15:18:17 ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|success|Response status: 404 24/07/11 15:18:17 ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|unavailable|Request method: GET 24/07/11 15:18:17 ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|success| 24/07/11 15:18:17 ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|success|Response status: 404 24/07/11 15:18:17 ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|unavailable|Request method: GET 24/07/11 15:18:17 ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|success| 24/07/11 15:18:17 ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|success|Response status: 404 24/07/11 15:18:17 ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|unavailable|Request method: GET 24/07/11 15:18:17 ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|success| 24/07/11 15:18:17 ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|unavailable|Request method: GET 24/07/11 15:18:17 ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|success|Response status: 404 24/07/11 15:18:17 ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|success| 24/07/11 15:18:17 ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|success|Response status: 404 24/07/11 15:18:17 ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit| < ip-address > |KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|unavailable|Request method: GET 24/07/11 15:18:17 ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit| < ip-address > |KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|success| 24/07/11 15:18:17 ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit| < ip-address > |KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|success|Response status: 404 ... This the page I get without JS or CSS with the following message This page will not function without javascript enabled. Please enable javascript on your browser. I found a post with the same issue [ https://community.cloudera.com/t5/Support-Questions/Knox-wrong-mapping-help-will-be-appriciated/td-p/144055 | here ] but after replacing my rewrite.xml with the IBM one I still had the same problems To fix this issue I had to change the /static rule the following way. FROM < rule dir = "OUT" name = "YARNUI/yarn/outbound/static" pattern = "/static/{**}" > < rewrite template = "{$frontendUrl[url]}/static/{**}" /> </ rule > TO < rule dir = "OUT" name = "YARNUI/yarn/outbound/static" pattern = "/static/{**}" > < rewrite template = "{$serviceUrl[YARNUI]}/static/{**}" /> </ rule > Also doing that doesn’t feel right since all the links on the Yarn UI are still broken for example clicking on the application link in the left nav bar redirects me to https://<knox-host>:8443/default/yarn/cluster/apps (without the gateway in front for some reason) where I get a 404 error. The same problem seems to happen to Spark History UI. When trying to access to https://<knox>:8443/gateway/default/spark3history the only thing with a response 200 is the html document of the sparkui page but every other ressources gets a 404 error. I added the following rule in data/services/spark3historyui/3.0.0/rewrite.xml < rule dir = "OUT" name = "SPARKHISTORYUI/outbound/static" pattern = "/static/{**}" > < rewrite template = "{$serviceUrl[SPARKHISTORYUI]}/static/{**}" /> </ rule > which fixes the css but I am still having issues with the jquery called made to get the json of all the jobs. Here is the output of gateway.log 2024-07-11 15:44:12,879 DEBUG knox.gateway (PortMappingHelperHandler.java:handleDefaultTopologyMapping(150)) - Default topology forward from /api/v1/applications to /gateway/default/api/v1/applications 2024-07-11 15:44:12,880 568624ae-69ea-4574-b162-3faa22c9d85e DEBUG knox.gateway (GatewayFilter.java:doFilter(126)) - Received request: GET /api/v1/applications 2024-07-11 15:44:13,356 TRACE gateway.access (AccessHandler.java:log(49)) - |||194.12.154.214|GET|https:// < knox-host > :8443/api/v1/applications?limit=2147483647&status=completed|-1|404|0|477 I’m pretty sure that the problems on those 2 services are linked and that I’m missing something in my configuration but I can’ t tell what at all. To give additional informations Here is my default topology and my gateway-site.xml I replaced my host <knox-host> for privacy issues here but I am using a real host in those files. < topology > < gateway > < provider > < role > webappsec </ role > < name > WebAppSec </ name > < enabled > true </ enabled > </ provider > < provider > < role > hostmap </ role > < name > static </ name > < enabled > false </ enabled > < param > < name > localhost </ name > < value > sandbox,sandbox.hortonworks.com </ value > </ param > </ provider > < provider > < role > identity-assertion </ role > < name > Default </ name > < enabled > false </ enabled > </ provider > < provider > < role > federation </ role > < name > SSOCookieProvider </ name > < enabled > true </ enabled > < param > < name > sso.authentication.provider.url </ name > < value > https:// < knox-host > :8443/gateway/knoxsso/api/v1/websso </ value > </ param > </ provider > </ gateway > < service > < role > KNOX </ role > </ service > < service > < role > HDFSUI </ role > < version > 2.7.0 </ version > < url > https:// < host > :50070 </ url > </ service > < service > < role > NAMENODE </ role > < url > https://localhost:8020 </ url > < param > < name > webhdfs-redirect </ name > < value > https:// < host > :8443/gateway/default/webhdfs/v1 </ value > </ param > </ service > < service > < role > WEBHDFS </ role > < url > https:// < host > :50070/webhdfs </ url > </ service > < service > < role > YARNUI </ role > < version > 2.7.0 </ version > < url > https:// < host > :8088 </ url > < param > < name > webyarn-redirect </ name > < value > https:// < host > :8443/gateway/default/webhdfs/v1 </ value > </ param > </ service > < service > < role > YARN </ role > < url > https:// < host > :8088/ </ url > </ service > < service > < role > HBASEUI </ role > < url > https:// < host > :60010 </ url > </ service > < service > < role > SPARK3HISTORYUI </ role > < version > 3.0.0 </ version > < url > https:// < host > :18080 </ url > </ service > < application > < name > admin-ui </ name > </ application > </ topology > gateway-site.xml <?xml version="1.0" encoding="UTF-8"?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration> <property> <name>gateway.service.alias.impl</name> <value>org.apache.knox.gateway.services.security.impl.RemoteAliasService</value> </property> <property> <name>default.app.topology.name</name> <value>default</value> </property> <property> <name>gateway.port</name> <value>8443</value> <description>The HTTP port for the Gateway.</description> </property> <property> <name>gateway.path</name> <value>gateway</value> <description>The default context path for the gateway.</description> </property> <property> <name>gateway.gateway.conf.dir</name> <value>deployments</value> <description>The directory within GATEWAY_HOME that contains gateway topology files and deployments.</description> </property> <property> <name>gateway.keystore.cert.algorithm</name> <value>SHA256withRSA</value> </property> <property> <name>gateway.hadoop.kerberos.secured</name> <value>false</value> <description>Boolean flag indicating whether the Hadoop cluster protected by Gateway is secured with Kerberos</description> </property> <property> <name>java.security.krb5.conf</name> <value>/etc/knox/conf/krb5.conf</value> <description>Absolute path to krb5.conf file</description> </property> <property> <name>java.security.auth.login.config</name> <value>/etc/knox/conf/krb5JAASLogin.conf</value> <description>Absolute path to JAAS login config file</description> </property> <property> <name>sun.security.krb5.debug</name> <value>false</value> <description>Boolean flag indicating whether to enable debug messages for krb5 authentication</description> </property> <!-- @since 0.10 Websocket configs --> <property> <name>gateway.websocket.feature.enabled</name> <value>true</value> <description>Enable/Disable websocket feature.</description> </property> <property> <name>gateway.scope.cookies.feature.enabled</name> <value>true</value> <description>Enable/Disable cookie scoping feature.</description> </property> <property> <name>gateway.cluster.config.monitor.ambari.enabled</name> <value>false</value> <description>Enable/disable Ambari cluster configuration monitoring.</description> </property> <property> <name>gateway.cluster.config.monitor.ambari.interval</name> <value>60</value> <description>The interval (in seconds) for polling Ambari for cluster configuration changes.</description> </property> <!-- @since 2.0.0 WebShell configs --> <!-- must have websocket enabled to use webshell --> <property> <name>gateway.webshell.feature.enabled</name> <value>false</value> <description>Enable/Disable webshell feature.</description> </property> <property> <name>gateway.webshell.max.concurrent.sessions</name> <value>20</value> <description>Maximum number of total concurrent webshell sessions</description> </property> <property> <name>gateway.webshell.audit.logging.enabled</name> <value>false</value> <description>[Experimental Feature] Enable/Disable webshell command audit logging. NOTE: Turning this on might log secrets that might be part of command line arguments, please consider this before turning this on.</description> </property> <property> <name>gateway.webshell.read.buffer.size</name> <value>1024</value> <description>Web Shell buffer size for reading</description> </property> <!-- @since 2.0.0 websocket JWT validation configs --> <property> <name>gateway.websocket.JWT.validation.feature.enabled</name> <value>true</value> <description>Enable/Disable websocket JWT validation at websocket layer.</description> </property> <!-- @since 1.5.0 homepage logout --> <property> <name>knox.homepage.logout.enabled</name> <value>true</value> <description>Enable/disable logout from the Knox Homepage.</description> </property> <!-- @since 1.6.0 token management related properties --> <property> <name>gateway.knox.token.eviction.grace.period</name> <value>0</value> <description>A duration (in seconds) beyond a token’s expiration to wait before evicting its state. This configuration only applies when server-managed token state is enabled either in gateway-site or at the topology level.</description> </property> <!-- Knox Admin related config --> <property> <name>gateway.knox.admin.groups</name> <value>admin</value> </property> <!-- DEMO LDAP config for Hadoop Group Provider --> <property> <name>gateway.group.config.hadoop.security.group.mapping</name> <value>org.apache.hadoop.security.LdapGroupsMapping</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.bind.user</name> <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.bind.password</name> <value>guest-password</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.url</name> <value>ldap://localhost:33389</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.base</name> <value></value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.filter.user</name> <value>(&(|(objectclass=person)(objectclass=applicationProcess))(cn={0}))</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.filter.group</name> <value>(objectclass=groupOfNames)</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.attr.member</name> <value>member</value> </property> <property> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.attr.group.name</name> <value>cn</value> </property> <property> <name>gateway.dispatch.whitelist.services</name> <value>DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,YARNUI,SPARK3HISTORYUI,knoxauth</value> <description>The comma-delimited list of service roles for which the gateway.dispatch.whitelist should be applied.</description> </property> <property> <name>gateway.dispatch.whitelist</name> <value>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$|^https:\/\/<host>:8443.*$|^https:\/\/<host>:50070.*$|^https:\/\/<host>:60010.*$|^https:\/\/<host>:8088.*$|^https:\/\/<host>:18080.*$</value> <!--<value>^https?:\/\/.*$</value>--> <!--<value>DEFAULT</value> --> <description>The whitelist to be applied for dispatches associated with the service roles specified by gateway.dispatch.whitelist.services. If the value is DEFAULT, a domain-based whitelist will be derived from the Knox host.</description> </property> <property> <name>gateway.frontend.url</name> <value>https://<host>:8443/</value> </property> <property> <name>gateway.xforwarded.enabled</name> <value>true</value> </property> <property> <name>gateway.server.header.enabled</name> <value>true</value> </property> <property> <name>gateway.xforwarded.header.context.append.servicename</name> <value>LIVYSERVER</value> <description>Add service name to x-forward-context header for the list of services defined above.</description> </property> </configuration> My spark version I think that there is a better way to achieve this without having to do all the manual workarounds ? Thanks for your help
