It should have worked :( rewrite rules are contributed by the component teams (spark and yarn in this case) since we cannot keep up with all the UI changes for these components. Perhaps you can try asking on Spark or Yarn mailing list someone might have had some luck there.
On Mon, Jul 15, 2024 at 2:09 AM thomas.mau...@etu.umontpellier.fr < thomas.mau...@etu.umontpellier.fr> wrote: > > Hello Sandeep, > Thank you very much for your answer I am gonna try your method to debug > that. But I am wondering, shouldn’t yarnui and sparky I work out of the box > ? I am using yarn 3.3.6 and sparkui 3.5.1 ? Because having to add some new > rules seems odd. > > Thank you for your answer, > Thomas > > Le 15 juil. 2024 à 00:23, Sandeep Moré <moresand...@gmail.com> a écrit : > > > Hello Thomas, > Rewrite rules are tricky to troubleshoot. The way I narrow down the > problem is by turning on debug log in knox. > These are the instructions on turning on DEBUG logging for Knox > https://knox.apache.org/books/knox-2-0-0/user-guide.html#Logging > > The way I go about debugging is by isolating a resource file (CSS or JS) > and then just using that file to tweak rewrite rules instead of focusing on > the entire page. > > Hopefully it works good luck. > > > On Thu, Jul 11, 2024 at 3:14 PM Thomas Mauran < > thomas.mau...@etu.umontpellier.fr> wrote: > >> Hello, I am writing this email to get your help on an issue with my >> Apache knox configuration. >> >> I am facing a problem on both YarnUI and Sparkhistory UI where I have to >> write myself rewrite rules for static files like css or js ones. For >> example when trying to access https://<knox>:8443/gateway/default/yarn/ >> I get 404 errors on the following files: >> >> *gateway-audit.log* >> >> 8:17 >> ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/yarn.css|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/yarn.css|success|24/07/11 >> 15:18:17 >> ||aa26e33e-e97b-4ff9-a977-79f5e9643ae3|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/yarn.css|success|Response >> status: 40424/07/11 15:18:17 >> ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|success|24/07/11 >> 15:18:17 >> ||475f6c06-c90d-4d14-ae39-9d81d5a51fee|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/themes-1.9.1/base/jquery-ui.css|success|Response >> status: 40424/07/11 15:18:17 >> ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|success|24/07/11 >> 15:18:17 >> ||0ea69ed6-8f4c-4cd1-86ca-ba8d1d30f505|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/jquery-3.6.0.min.js|success|Response >> status: 40424/07/11 15:18:17 >> ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|success|24/07/11 >> 15:18:17 >> ||9e5f2aeb-cff2-42cc-ae98-858e994e4214|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/yarn.dt.plugins.js|success|Response >> status: 40424/07/11 15:18:17 >> ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|success|24/07/11 >> 15:18:17 >> ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||91eb4324-6b52-41ab-8473-6aebd0fec591|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/dt-1.10.18/css/jquery.dataTables.css|success|Response >> status: 40424/07/11 15:18:17 >> ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|success|24/07/11 >> 15:18:17 >> ||5c26cfad-9397-406a-bc66-dfa668933d9a|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/dt-sorting/natural.js|success|Response >> status: 40424/07/11 15:18:17 >> ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit|<ip-address>|KNOX||||access|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|unavailable|Request >> method: GET24/07/11 15:18:17 >> ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit|<ip-address>|KNOX|user|||authentication|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|success|24/07/11 >> 15:18:17 >> ||3aaa7592-bf99-4c84-814b-7a985a27454c|audit|<ip-address>|KNOX|user|||access|uri|/gateway/default/default/yarn/static/jquery/jquery-ui-1.13.2.custom.min.js|success|Response >> status: 404 >> >> ... >> >> >> >> This the page I get without JS or CSS with the following message This >> page will not function without javascript enabled. Please enable javascript >> on your browser. >> >> I found a post with the same issue here >> <https://community.cloudera.com/t5/Support-Questions/Knox-wrong-mapping-help-will-be-appriciated/td-p/144055> >> but after replacing my rewrite.xml with the IBM one I still had the same >> problems >> >> To fix this issue I had to change the /static rule the following way. >> >> *FROM* >> >> <rule dir="OUT" name="YARNUI/yarn/outbound/static" pattern="/static/{**}"> >> <rewrite template="{$frontendUrl[url]}/static/{**}"/></rule> >> >> *TO* >> >> <rule dir="OUT" name="YARNUI/yarn/outbound/static" pattern="/static/{**}"> >> <rewrite template="{$serviceUrl[YARNUI]}/static/{**}"/></rule> >> >> Also doing that doesn’t feel right since all the links on the Yarn UI are >> still broken for example clicking on the application link in the left nav >> bar redirects me to >> https://<knox-host>:8443/default/yarn/cluster/apps >> >> (without the gateway in front for some reason) where I get a 404 error. >> >> The same problem seems to happen to Spark History UI. When trying to >> access to https://<knox>:8443/gateway/default/spark3history the only >> thing with a response 200 is the html document of the sparkui page but >> every other ressources gets a 404 error. >> >> I added the following rule in >> data/services/spark3historyui/3.0.0/rewrite.xml >> >> <rule dir="OUT" name="SPARKHISTORYUI/outbound/static" >> pattern="/static/{**}"> >> <rewrite template="{$serviceUrl[SPARKHISTORYUI]}/static/{**}"/> >> </rule> >> >> which fixes the css but I am still having issues with the jquery called >> made to get the json of all the jobs. >> >> Here is the output of gateway.log >> >> 2024-07-11 15:44:12,879 DEBUG knox.gateway >> (PortMappingHelperHandler.java:handleDefaultTopologyMapping(150)) - Default >> topology forward from /api/v1/applications to >> /gateway/default/api/v1/applications2024-07-11 15:44:12,880 >> 568624ae-69ea-4574-b162-3faa22c9d85e DEBUG knox.gateway >> (GatewayFilter.java:doFilter(126)) - Received request: GET >> /api/v1/applications2024-07-11 15:44:13,356 TRACE gateway.access >> (AccessHandler.java:log(49)) - >> |||194.12.154.214|GET|https://<knox-host>:8443/api/v1/applications?limit=2147483647&status=completed|-1|404|0|477 >> >> I’m pretty sure that the problems on those 2 services are linked and that >> I’m missing something in my configuration but I can’ t tell what at all. To >> give additional informations Here is my default topology and my >> gateway-site.xml >> >> I replaced my host <knox-host> for privacy issues here but I am using a >> real host in those files. >> >> <topology> >> <gateway> >> <provider> >> <role>webappsec</role> >> <name>WebAppSec</name> >> <enabled>true</enabled> >> </provider> >> <provider> >> <role>hostmap</role> >> <name>static</name> >> <enabled>false</enabled> >> >> <param><name>localhost</name><value>sandbox,sandbox.hortonworks.com</value></param> >> </provider> >> >> <provider> >> <role>identity-assertion</role> >> <name>Default</name> >> <enabled>false</enabled> >> </provider> >> >> <provider> >> <role>federation</role> >> <name>SSOCookieProvider</name> >> <enabled>true</enabled> >> <param> >> <name>sso.authentication.provider.url</name> >> >> <value>https://<knox-host>:8443/gateway/knoxsso/api/v1/websso</value> >> </param> >> </provider></gateway> >> <service> >> <role>KNOX</role> >> </service> >> <service> >> <role>HDFSUI</role> >> <version>2.7.0</version> >> <url>https://<host>:50070</url> >> </service> >> <service> >> <role>NAMENODE</role> >> <url>https://localhost:8020</url> >> <param> >> <name>webhdfs-redirect</name> >> <value>https://<host>:8443/gateway/default/webhdfs/v1</value> >> </param> >> </service><service> >> <role>WEBHDFS</role> >> <url>https://<host>:50070/webhdfs</url></service> >> >> <service> >> <role>YARNUI</role> >> <version>2.7.0</version> >> <url>https://<host>:8088</url> >> <param> >> <name>webyarn-redirect</name> >> <value>https://<host>:8443/gateway/default/webhdfs/v1</value> >> </param></service><service> >> <role>YARN</role> >> <url>https://<host>:8088/</url></service> >> <service> >> <role>HBASEUI</role> >> <url>https://<host>:60010</url> >> </service> >> >> <service> >> <role>SPARK3HISTORYUI</role> >> <version>3.0.0</version> >> <url>https://<host>:18080</url> >> </service> >> >> >> <application> >> <name>admin-ui</name> >> </application></topology> >> >> *gateway-site.xml* >> >> <?xml version="1.0" encoding="UTF-8"?> >> <!-- >> Licensed to the Apache Software Foundation (ASF) under one >> or more contributor license agreements. See the NOTICE file >> distributed with this work for additional information >> regarding copyright ownership. The ASF licenses this file >> to you under the Apache License, Version 2.0 (the >> "License"); you may not use this file except in compliance >> with the License. You may obtain a copy of the License at >> http://www.apache.org/licenses/LICENSE-2.0 >> >> Unless required by applicable law or agreed to in writing, software >> distributed under the License is distributed on an "AS IS" BASIS, >> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> See the License for the specific language governing permissions and >> limitations under the License. >> --> >> <configuration> >> >> <property> >> <name>gateway.service.alias.impl</name> >> >> <value>org.apache.knox.gateway.services.security.impl.RemoteAliasService</value> >> </property> >> >> <property> >> <name>default.app.topology.name</name> >> <value>default</value> >> >> </property> >> >> <property> >> <name>gateway.port</name> >> <value>8443</value> >> <description>The HTTP port for the Gateway.</description> >> </property> >> >> <property> >> <name>gateway.path</name> >> <value>gateway</value> >> <description>The default context path for the gateway.</description> >> </property> >> >> <property> >> <name>gateway.gateway.conf.dir</name> >> <value>deployments</value> >> <description>The directory within GATEWAY_HOME that contains gateway >> topology files and deployments.</description> >> </property> >> <property> >> <name>gateway.keystore.cert.algorithm</name> >> <value>SHA256withRSA</value> >> </property> >> >> <property> >> <name>gateway.hadoop.kerberos.secured</name> >> <value>false</value> >> <description>Boolean flag indicating whether the Hadoop cluster >> protected by Gateway is secured with Kerberos</description> >> </property> >> >> <property> >> <name>java.security.krb5.conf</name> >> <value>/etc/knox/conf/krb5.conf</value> >> <description>Absolute path to krb5.conf file</description> >> </property> >> >> <property> >> <name>java.security.auth.login.config</name> >> <value>/etc/knox/conf/krb5JAASLogin.conf</value> >> <description>Absolute path to JAAS login config file</description> >> </property> >> >> <property> >> <name>sun.security.krb5.debug</name> >> <value>false</value> >> <description>Boolean flag indicating whether to enable debug >> messages for krb5 authentication</description> >> </property> >> >> <!-- @since 0.10 Websocket configs --> >> <property> >> <name>gateway.websocket.feature.enabled</name> >> <value>true</value> >> <description>Enable/Disable websocket feature.</description> >> </property> >> >> <property> >> <name>gateway.scope.cookies.feature.enabled</name> >> <value>true</value> >> <description>Enable/Disable cookie scoping feature.</description> >> </property> >> >> <property> >> <name>gateway.cluster.config.monitor.ambari.enabled</name> >> <value>false</value> >> <description>Enable/disable Ambari cluster configuration >> monitoring.</description> >> </property> >> >> <property> >> <name>gateway.cluster.config.monitor.ambari.interval</name> >> <value>60</value> >> <description>The interval (in seconds) for polling Ambari for >> cluster configuration changes.</description> >> </property> >> <!-- @since 2.0.0 WebShell configs --> >> <!-- must have websocket enabled to use webshell --> >> <property> >> <name>gateway.webshell.feature.enabled</name> >> <value>false</value> >> <description>Enable/Disable webshell feature.</description> >> </property> >> <property> >> <name>gateway.webshell.max.concurrent.sessions</name> >> <value>20</value> >> <description>Maximum number of total concurrent webshell >> sessions</description> >> </property> >> <property> >> <name>gateway.webshell.audit.logging.enabled</name> >> <value>false</value> >> <description>[Experimental Feature] Enable/Disable webshell command >> audit logging. >> NOTE: Turning this on might log secrets that might be part of >> command line arguments, please consider this before turning this >> on.</description> >> </property> >> <property> >> <name>gateway.webshell.read.buffer.size</name> >> <value>1024</value> >> <description>Web Shell buffer size for reading</description> >> </property> >> >> <!-- @since 2.0.0 websocket JWT validation configs --> >> <property> >> <name>gateway.websocket.JWT.validation.feature.enabled</name> >> <value>true</value> >> <description>Enable/Disable websocket JWT validation at websocket >> layer.</description> >> </property> >> >> <!-- @since 1.5.0 homepage logout --> >> <property> >> <name>knox.homepage.logout.enabled</name> >> <value>true</value> >> <description>Enable/disable logout from the Knox >> Homepage.</description> >> </property> >> >> <!-- @since 1.6.0 token management related properties --> >> <property> >> <name>gateway.knox.token.eviction.grace.period</name> >> <value>0</value> >> <description>A duration (in seconds) beyond a token’s expiration to >> wait before evicting its state. This configuration only applies when >> server-managed token state is enabled either in gateway-site or at the >> topology level.</description> >> </property> >> >> <!-- Knox Admin related config --> >> <property> >> <name>gateway.knox.admin.groups</name> >> <value>admin</value> >> </property> >> >> <!-- DEMO LDAP config for Hadoop Group Provider --> >> <property> >> <name>gateway.group.config.hadoop.security.group.mapping</name> >> <value>org.apache.hadoop.security.LdapGroupsMapping</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.bind.user</name> >> <value>uid=guest,ou=people,dc=hadoop,dc=apache,dc=org</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.bind.password</name> >> <value>guest-password</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.url</name> >> <value>ldap://localhost:33389</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.base</name> >> <value></value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.filter.user</name> >> >> <value>(&(|(objectclass=person)(objectclass=applicationProcess))(cn={0}))</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.filter.group</name> >> <value>(objectclass=groupOfNames)</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.attr.member</name> >> <value>member</value> >> </property> >> <property> >> >> <name>gateway.group.config.hadoop.security.group.mapping.ldap.search.attr.group.name</name> >> <value>cn</value> >> </property> >> <property> >> <name>gateway.dispatch.whitelist.services</name> >> >> <value>DATANODE,HBASEUI,HDFSUI,JOBHISTORYUI,NODEUI,YARNUI,SPARK3HISTORYUI,knoxauth</value> >> <description>The comma-delimited list of service roles for which the >> gateway.dispatch.whitelist should be applied.</description> >> </property> >> <property> >> <name>gateway.dispatch.whitelist</name> >> >> >> <value>^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1):[0-9].*$|^https:\/\/<host>:8443.*$|^https:\/\/<host>:50070.*$|^https:\/\/<host>:60010.*$|^https:\/\/<host>:8088.*$|^https:\/\/<host>:18080.*$</value> >> >> >> <!--<value>^https?:\/\/.*$</value>--> >> <!--<value>DEFAULT</value> --> >> <description>The whitelist to be applied for dispatches >> associated with the service roles specified by >> gateway.dispatch.whitelist.services. >> If the value is DEFAULT, a domain-based whitelist will be derived from >> the Knox host.</description> >> </property> >> <property> >> <name>gateway.frontend.url</name> >> <value>https://<host>:8443/</value> >> </property> >> >> <property> >> <name>gateway.xforwarded.enabled</name> >> <value>true</value> >> </property> >> <property> >> <name>gateway.server.header.enabled</name> >> <value>true</value> >> </property> >> >> <property> >> <name>gateway.xforwarded.header.context.append.servicename</name> >> <value>LIVYSERVER</value> >> <description>Add service name to x-forward-context header for the >> list of services defined above.</description> >> </property> >> >> </configuration> >> >> >> My spark version >> >> I think that there is a better way to achieve this without having to do >> all the manual workarounds ? >> >> Thanks for your help >> >
