Added: knox/trunk/books/0.14.0/service_yarn.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/service_yarn.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/service_yarn.md (added) +++ knox/trunk/books/0.14.0/service_yarn.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,124 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### Yarn ### + +Knox provides gateway functionality for the REST APIs of the ResourceManager. The ResourceManager REST APIs allow the +user to get information about the cluster - status on the cluster, metrics on the cluster, scheduler information, +information about nodes in the cluster, and information about applications on the cluster. Also as of Hadoop version +2.5.0, the user can submit a new application as well as kill it (or get state) using the 'Writable' APIs. + +The docs for this can be found here + +http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html + +To enable this functionality, a topology file needs to have the following configuration: + + <service> + <role>RESOURCEMANAGER</role> + <url>http://<hostname>:<port>/ws</url> + </service> + +The default resource manager http port is 8088. If it is configured to some other port, that configuration can be +found in `yarn-site.xml` under the property `yarn.resourcemanager.webapp.address`. + +#### Yarn URL Mapping #### + +For Yarn URLs, the mapping of Knox Gateway accessible URLs to direct Yarn URLs is the following. + +| ------- | ------------------------------------------------------------------------------------- | +| Gateway | `https://{gateway-host}:{gateway-port}/{gateway-path}/{cluster-name}/resourcemanager` | +| Cluster | `http://{yarn-host}:{yarn-port}/ws}` | + + +#### Yarn Examples via cURL + +Some of the various calls that can be made and examples using curl are listed below. + + # 0. Getting cluster info + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster' + + # 1. Getting cluster metrics + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/metrics' + + To get the same information in an xml format + + curl -ikv -u guest:guest-password -H Accept:application/xml -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/metrics' + + # 2. Getting scheduler information + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/scheduler' + + # 3. Getting all the applications listed and their information + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps' + + # 4. Getting applications statistics + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/appstatistics' + + Also query params can be used as below to filter the results + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/appstatistics?states=accepted,running,finished&applicationTypes=mapreduce' + + # 5. To get a specific application (please note, replace the application id with a real value) + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}' + + # 6. To get the attempts made for a particular application + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/appattempts' + + # 7. To get information about the various nodes + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/nodes' + + Also to get a specific node, use an id obtained in the response from above (the node id is scrambled) and issue the following + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/nodes/{node_id}' + + # 8. To create a new Application + + curl -ikv -u guest:guest-password -X POST 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/new-application' + + An application id is returned from the request above and this can be used to submit an application. + + # 9. To submit an application, put together a request containing the application id received in the above response (please refer to Yarn REST + API documentation). + + curl -ikv -u guest:guest-password -T request.json -H Content-Type:application/json -X POST 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps' + + Here the request is saved in a file called request.json + + #10. To get application state + + curl -ikv -u guest:guest-password -X GET 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/state' + + curl -ikv -u guest:guest-password -H Content-Type:application/json -X PUT -T state-killed.json 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/application_1409008107556_0007/state' + + # 11. To kill an application that is running issue the below command with the application id of the application that is to be killed. + The contents of the state-killed.json file are : + + { + "state":"KILLED" + } + + + curl -ikv -u guest:guest-password -H Content-Type:application/json -X PUT -T state-killed.json 'https://localhost:8443/gateway/sandbox/resourcemanager/v1/cluster/apps/{application_id}/state' +
Added: knox/trunk/books/0.14.0/websocket-support.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/websocket-support.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/websocket-support.md (added) +++ knox/trunk/books/0.14.0/websocket-support.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,76 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +## Websocket Support ## + +### Introduction + +Websocket is a communication protocol that allows full duplex communication over a single TCP connection. +Knox provides out-of-the-box support for websocket protocol, currently only text messages are supported. + +### Configuration ### + +By default websocket functionality is disabled, it can be easily enabled by changing the 'gateway.websocket.feature.enabled' property to 'true' in <KNOX-HOME>/conf/gateway-site.xml file. + + <property> + <name>gateway.websocket.feature.enabled</name> + <value>true</value> + <description>Enable/Disable websocket feature.</description> + </property> + +Service and rewrite rules need to changed accordingly to match the appropriate websocket context. + +### Example ### + +In the following sample configuration we assume that the backend websocket URL is ws://myhost:9999/ws. And 'gateway.websocket.feature.enabled' property is set to 'true' as shown above. + +#### rewrite #### + +Example code snippet from <KNOX-HOME>/data/services/{myservice}/{version}/rewrite.xml where myservice = websocket and version = 0.6.0 + + <rules> + <rule dir="IN" name="WEBSOCKET/ws/inbound" pattern="*://*:*/**/ws"> + <rewrite template="{$serviceUrl[WEBSOCKET]}/ws"/> + </rule> + </rules> + +#### service #### + +Example code snippet from <KNOX-HOME>/data/services/{myservice}/{version}/service.xml where myservice = websocket and version = 0.6.0 + + <service role="WEBSOCKET" name="websocket" version="0.6.0"> + <policies> + <policy role="webappsec"/> + <policy role="authentication" name="Anonymous"/> + <policy role="rewrite"/> + <policy role="authorization"/> + </policies> + <routes> + <route path="/ws"> + <rewrite apply="WEBSOCKET/ws/inbound" to="request.url"/> + </route> + </routes> + </service> + +#### topology #### + +Finally, update the topology file at <KNOX-HOME>/conf/{topology}.xml with the backend service url + + <service> + <role>WEBSOCKET</role> + <url>ws://myhost:9999/ws</url> + </service> Added: knox/trunk/books/0.14.0/x-forwarded-headers.md URL: http://svn.apache.org/viewvc/knox/trunk/books/0.14.0/x-forwarded-headers.md?rev=1809165&view=auto ============================================================================== --- knox/trunk/books/0.14.0/x-forwarded-headers.md (added) +++ knox/trunk/books/0.14.0/x-forwarded-headers.md Thu Sep 21 13:57:57 2017 @@ -0,0 +1,76 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +---> + +### X-Forwarded-* Headers Support ### +Out-of-the-box Knox provides support for some X-Forwarded-* headers through the use of a Servlet Filter. Specifically the +headers handled/populated by Knox are: + +* X-Forwarded-For +* X-Forwarded-Proto +* X-Forwarded-Port +* X-Forwarded-Host +* X-Forwarded-Server +* X-Forwarded-Context + +If this functionality can be turned off by a configuration setting in the file gateway-site.xml and redeploying the +necessary topology/topologies. + +The setting is (under the 'configuration' tag) : + + <property> + <name>gateway.xforwarded.enabled</name> + <value>false</value> + </property> + +If this setting is absent, the default behavior is that the X-Forwarded-* header support is on or in other words, +'gateway.xforwarded.enabled' is set to 'true' by default. + + +#### Header population #### + +The following are the various rules for population of these headers: + +##### X-Forwarded-For ##### + +This header represents a list of client IP addresses. If the header is already present Knox adds a comma separated value +to the list. The value added is the client's IP address as Knox sees it. This value is added to the end of the list. + +##### X-Forwarded-Proto ##### + +The protocol used in the client request. If this header is passed into Knox its value is maintained, otherwise Knox will +populate the header with the value 'https' if the request is a secure one or 'http' otherwise. + +##### X-Forwarded-Port ##### + +The port used in the client request. If this header is passed into Knox its value is maintained, otherwise Knox will +populate the header with the value of the port that the request was made coming into Knox. + +##### X-Forwarded-Host ##### + +Represents the original host requested by the client in the Host HTTP request header. The value passed into Knox is maintained +by Knox. If no value is present, Knox populates the header with the value of the HTTP Host header. + +##### X-Forwarded-Server ##### + +The hostname of the server Knox is running on. + +##### X-Forwarded-Context ##### + +This header value contains the context path of the request to Knox. + + +