http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_ops_config.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_ops_config.txt b/docs/latest/_sources/admin/traffic_ops_config.txt index 90c0c1c..b17d57e 100644 --- a/docs/latest/_sources/admin/traffic_ops_config.txt +++ b/docs/latest/_sources/admin/traffic_ops_config.txt @@ -29,7 +29,7 @@ Content Delivery Networks Profile Parameters ====================== -Many of the settings for the different servers in a Traffic Control CDN are controlled by parameters in the parameter view of Traffic Ops. Parameters are grouped in profiles and profiles are assigned to a server. For a typical cache there are hundreds of configuration settings to apply. The Traffic Ops parameter view contains the defined settings. To make life easier, Traffic Ops allows for duplication, comparison, import and export of Profiles. Traffic Ops also has a "Global profile" - the parameters in this profile are going to be applied to all servers in the Traffic Ops instance, or apply to Traffic Ops themselves. These parameters are: +Many of the settings for the different servers in a Traffic Control CDN are controlled by parameters in the parameter view of Traffic Ops. Parameters are grouped in profiles and profiles are assigned to a server or a deliveryservice. For a typical cache there are hundreds of configuration settings to apply. The Traffic Ops parameter view contains the defined settings. To make life easier, Traffic Ops allows for duplication, comparison, import and export of Profiles. Traffic Ops also has a "Global profile" - the parameters in this profile are going to be applied to all servers in the Traffic Ops instance, or apply to Traffic Ops themselves. These parameters are: .. index:: @@ -40,6 +40,8 @@ Many of the settings for the different servers in a Traffic Control CDN are cont +==========================+===============+=======================================================================================================================================+ | tm.url | global | The URL where this Traffic Ops instance is being served from. | +--------------------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------+ +| tm.cache.url | global | Not required. The URL where the Traffic Ops Config file cache instance is being served from. Requires Traffic Ops 2.1 and above. | ++--------------------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------+ | tm.toolname | global | The name of the Traffic Ops tool. Usually "Traffic Ops". Used in the About screen and in the comments headers of the files generated. | +--------------------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------+ | tm.infourl | global | This is the "for more information go here" URL, which is visible in the About page. | @@ -90,12 +92,20 @@ Below is a list of cache parameters that are likely to need changes from the def | allow_ip6 | astats.config | This is a comma separated list of IPv6 CIDR blocks that will have access to the astats statistics on the caches. | | | | The Traffic Monitor IP addresses have to be included in this, if they are using IPv6 to monitor the caches. | +--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ -| Drive_Prefix | storage.config | JvD/Jeff to supply blurb | +| Drive_Prefix | storage.config | The device path start of the disks. For example, if you have ``/dev/sda`` through ``/dev/sdf`` set this to ``/dev/sd`` | +--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ -| Drive_Letters | storage.config | JvD/Jeff to supply blurb | +| Drive_Letters | storage.config | The letter part of the disks, in the same example as above set this to ``a,b,c,d,e,f`` | +--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ | purge_allow_ip | ip_allow.config | The IP address range that is allowed to execute the PURGE method on the caches (not related to :ref:`rl-purge`) | +--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ +| coalesce_masklen_v4 | ip_allow.config | The masklen to use when coalescing v4 networks into one line using http://search.cpan.org/~miker/NetAddr-IP-4.078/IP.pm | ++--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ +| coalesce_number_v4 | ip_allow.config | The number to use when coalescing v4 networks into one line using http://search.cpan.org/~miker/NetAddr-IP-4.078/IP.pm | ++--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ +| coalesce_masklen_v6 | ip_allow.config | The masklen to use when coalescing v6 networks into one line using http://search.cpan.org/~miker/NetAddr-IP-4.078/IP.pm | ++--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ +| coalesce_masklen_v6 | ip_allow.config | The masklen to use when coalescing v6 networks into one line using http://search.cpan.org/~miker/NetAddr-IP-4.078/IP.pm | ++--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ | health.threshold.loadavg | rascal.properties | The Unix load average at which Traffic Router will stop sending traffic to this cache | +--------------------------+-------------------+-------------------------------------------------------------------------------------------------------------------------+ | health.threshold.\\ | rascal.properties | The amount of bandwidth that Traffic Router will try to keep available on the cache. | @@ -143,6 +153,11 @@ Content Purge is controlled by the following parameters in the profile of the ca | regex_revalidate | plugin.config | The config to be used for regex_revalidate. | `regex_revalidate <https://docs.trafficserver.apache.org/en/5.3.x/reference/plugins/regex_remap.en.html>`_ | | | | For example: --config regex_revalidate.config | | +----------------------+-------------------------+--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ +| use_reval_pending | global | Configures Traffic Ops to use separate | When this flag is in use ORT will check for a new regex_revalidate.config every 60 seconds in syncds mode during the dispersal timer. This will | +| | | reval_pending flag for each cache. | also allow ORT to be run in revalidate mode, which will check for and clear the reval_pending flag. This can be set to run via cron task. | +| | | | This value is set to 0 by default. Enable with a value of 1. Use of this feature requires Traffic Ops 2.1 and above. | ++----------------------+-------------------------+--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ + Note that the TTL the adminstrator enters in the purge request should be longer than the TTL of the content to ensure the bad content will not be used. If the CDN is serving content of unknown, or unlimited TTL, the administrator should consider using `proxy-config-http-cache-guaranteed-min-lifetime <https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.config.en.html#proxy-config-http-cache-guaranteed-min-lifetime>`_ to limit the maximum time an object can be in the cache before it is considered stale, and set that to the same value as `maxRevalDurationDays` (Note that the former is in seconds and the latter is in days, so convert appropriately).
http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_ops_install.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_ops_install.txt b/docs/latest/_sources/admin/traffic_ops_install.txt index ca49461..d0caaf8 100644 --- a/docs/latest/_sources/admin/traffic_ops_install.txt +++ b/docs/latest/_sources/admin/traffic_ops_install.txt @@ -379,3 +379,106 @@ To upgrade: 2. Enter the following command:``yum upgrade traffic_ops`` 3. See :ref:`rl-ps` to run postinstall. 4. Enter the following command:``service traffic_ops start`` + +Manually Generating and Installing the SSL Certificate +------------------------------------------------------ + +.. Note:: This section is valid for traffic-control 2.0.0 and later. + +Self-signed Certificate (Development) +===================================== + + Example Procedure:: + + $ openssl genrsa -des3 -passout pass:x -out localhost.pass.key 2048 + Generating RSA private key, 2048 bit long modulus + ... + $ openssl rsa -passin pass:x -in localhost.pass.key -out localhost.key + writing RSA key + $ rm localhost.pass.key + + $ openssl req -new -key localhost.key -out localhost.csr + You are about to be asked to enter information that will be incorporated + into your certificate request. + What you are about to enter is what is called a Distinguished Name or a DN. + There are quite a few fields but you can leave some blank + For some fields there will be a default value, + If you enter '.', the field will be left blank. + ----- + Country Name (2 letter code) [XX]:US<enter> + State or Province Name (full name) []:CO<enter> + Locality Name (eg, city) [Default City]:Denver<enter> + Organization Name (eg, company) [Default Company Ltd]: <enter> + Organizational Unit Name (eg, section) []: <enter> + Common Name (eg, your name or your server's hostname) []: <enter> + Email Address []: <enter> + + Please enter the following 'extra' attributes + to be sent with your certificate request + A challenge password []: pass<enter> + An optional company name []: <enter> + $ openssl x509 -req -sha256 -days 365 -in localhost.csr -signkey localhost.key -out localhost.crt + Signature ok + subject=/C=US/ST=CO/L=Denver/O=Default Company Ltd + Getting Private key + $ sudo cp localhost.crt /etc/pki/tls/certs + $ sudo cp localhost.key /etc/pki/tls/private + $ sudo chown trafops:trafops /etc/pki/tls/certs/localhost.crt + $ sudo chown trafops:trafops /etc/pki/tls/private/localhost.key + +Certificate from Certificate Authority (Production) +=================================================== + +.. Note:: You will need to know the appropriate answers when generating the certificate request file `trafficopss.csr` below. + + Example Procedure:: + + $ openssl genrsa -des3 -passout pass:x -out trafficops.pass.key 2048 + Generating RSA private key, 2048 bit long modulus + ... + $ openssl rsa -passin pass:x -in trafficops.pass.key -out trafficops.key + writing RSA key + $ rm localhost.pass.key + + Generate the Certificate Signing Request (CSR) file needed for Certificate Authority (CA) request. + + $ openssl req -new -key trafficops.key -out trafficops.csr + You are about to be asked to enter information that will be incorporated + into your certificate request. + What you are about to enter is what is called a Distinguished Name or a DN. + There are quite a few fields but you can leave some blank + For some fields there will be a default value, + If you enter '.', the field will be left blank. + ----- + Country Name (2 letter code) [XX]: <enter country code> + State or Province Name (full name) []: <enter state or province> + Locality Name (eg, city) [Default City]: <enter locality name> + Organization Name (eg, company) [Default Company Ltd]: <enter organization name> + Organizational Unit Name (eg, section) []: <enter organizational unit name> + Common Name (eg, your name or your server's hostname) []: <enter server's hostname name> + Email Address []: <enter e-mail address> + + Please enter the following 'extra' attributes + to be sent with your certificate request + A challenge password []: <enter challenge password> + An optional company name []: <enter> + $ sudo cp trafficops.key /etc/pki/tls/private + $ sudo chown trafops:trafops /etc/pki/tls/private/trafficops.key + + You must then take the output file trafficops.csr and submit a request to your Certificate Authority (CA). + Once you get approved and receive your trafficops.crt file: + + $ sudo cp trafficops.crt /etc/pki/tls/certs + $ sudo chown trafops:trafops /etc/pki/tls/certs/trafficops.crt + + If necessary, install the CA certificates .pem and .crt in /etc/pki/tls/certs. + + You will need to update the file /opt/traffic_ops/app/conf/cdn.conf with the following changes: + ... + e.g. given trafficops.crt and trafficops.key + 'hypnotoad' => ... + 'listen' => 'https://[::]:443?cert=/etc/pki/tls/certs/trafficops.crt&key=/etc/pki/tls/private/trafficops.key&ca=/etc/pki/tls/certs/localhost.ca&verify=0x00&ciphers=AES128-GCM-SHA256:HIGH:!RC4:!MD5:!aNULL:!EDH:!ED' + ... + + + http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_ops_using.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_ops_using.txt b/docs/latest/_sources/admin/traffic_ops_using.txt index eb3d0ee..c62704c 100644 --- a/docs/latest/_sources/admin/traffic_ops_using.txt +++ b/docs/latest/_sources/admin/traffic_ops_using.txt @@ -122,6 +122,8 @@ The following tabs are available in the menu at the top of the Traffic Ops user +--------------------+-------------------------------------------------------------------------------------------+ | Option | Description | +====================+===========================================================================================+ + | CDNs | Create/Read/Update/Delete CDNs | + +--------------------+-------------------------------------------------------------------------------------------+ | Cache Groups | Create/Read/Update/Delete cache groups | +--------------------+-------------------------------------------------------------------------------------------+ | Users | Create/Read/Update/Delete users | @@ -255,7 +257,7 @@ These are the types of servers that can be managed in Traffic Ops: +---------------+---------------------------------------------+ | ORG | Origin | +---------------+---------------------------------------------+ -| CCR | Comcast Content Router | +| CCR | Traffic Router | +---------------+---------------------------------------------+ | RASCAL | Rascal health polling & reporting | +---------------+---------------------------------------------+ @@ -278,7 +280,7 @@ These are the types of servers that can be managed in Traffic Ops: Bulk Upload Server ++++++++++++++++++ - +TBD Delivery Service @@ -309,6 +311,8 @@ The fields in the Delivery Service view are: | | - 0 use in cache key and hand up to origin -this means each unique query string Is treated as a unique URL. | | | - 1 Do not use in cache key, but pass up to origin - this means a 2 URLs that are the same except for the query string will match, and cache HIT, while the origin still sees original query string in the request. | | | - 2 Drop at edge - this means a 2 URLs that are the same except for the query string will match, and cache HIT, while the origin will not see original query string in the request. | +| | | +| | **Note:** Choosing to drop query strings at the edge will preclude the use of a Regex Remap Expression. See :ref:`rl-regex-remap`. | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Geo Limit? | Some services are intended to be limited by geography. The possible settings are are: | | | | @@ -339,12 +343,7 @@ The fields in the Delivery Service view are: +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Use Multi Site Origin Feature | Enable the Multi Site Origin feature for this delivery service. See :ref:`rl-multi-site-origin` | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| Multi Site Origin Algorithm | - 1 Consistent Hash spreads requests across multiple parents simultaneously based on hash of content URL. | -| | - 2 Strict Round Robin spreads requests across multiple parents simultaneously based on order of requests. | -| | - 3 IP Based Round Robin spreads requests across multiple parents simultaneously based on order of requests, but ensures that requests from the same IP always go to the same parent if available. | -| | - 4 Latched uses only a single parent at any given time and switches to a new parent only if the current parent fails. | -+--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| CCR profile | The Traffic Router profile for this delivery service. See :ref:`rl-ccr-profile`. | +| Profile | The profile for this delivery service. | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Maximum Bits per Second allowed globally | The maximum bits per second this delivery service can serve across all EDGE caches before traffic will be diverted to the bypass destination. For a DNS delivery service, the Bypass Ipv4 or Ipv6 will be used | | | (depending on whether this was a A or AAAA request), and for HTTP delivery services the Bypass FQDN will be used. | @@ -361,6 +360,8 @@ The fields in the Delivery Service view are: | Mid Header Rewrite Rules | Header Rewrite rules to apply for this delivery service at the MID tier. See :ref:`rl-header-rewrite`. [1]_ | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Regex Remap Expression | Regex Remap rule to apply to this delivery service at the Edge tier. See `ATS documentation on regex_remap <https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/regex_remap.en.html>`_. [1]_ | +| | | +| | **Note:** you will not be able to save a Regex Remap Expression if you have Query String Handling set to drop query strings at the edge. See :ref:`rl-regex-remap`. | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Cache URL expression | Cache URL rule to apply to this delivery service. See `ATS documentation on cacheurl <https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/cacheurl.en.html>`_. [1]_ | +--------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -594,14 +595,14 @@ Parameters in the Mid (parent) profile that influence this feature: Multi Site Origin +++++++++++++++++ -.. Note:: The Multi Site Origin feature is based upon a feature n ATS that has yet to be submitted to Traffic Server upstream, until it is, set this to 0, or use the ATS rpm supplied on the trafficcontrol.apache.org website. -Normally, the mid servers are not aware of any redundancy at the origin layer. With Multi Site Origin enabled this changes - Traffic Server (and Traffic Ops) are now made aware of the fact there are multiple origins, and can be configured to do more advanced failover and loadbalancing actions. +.. Note:: The configuration of this feature changed significantly between ATS version 5 and >= 6. Some configuration in Traffic Control is different as well. This documentation assumes ATS 6 or higher. See :ref:`rl-multi-site-origin-qht-ats5` for the ATS version 5.x configuration details. -With This feature enabled, origin servers (or origin server VIP names for a site) are going to be entered as servers in to the Traiffic Ops UI. Server type is "" +Normally, the mid servers are not aware of any redundancy at the origin layer. With Multi Site Origin enabled this changes - Traffic Server (and Traffic Ops) are now made aware of the fact there are multiple origins, and can be configured to do more advanced failover and loadbalancing actions. A prerequisite for MSO to work is that the multiple origin sites serve identical content with identical paths, and both are configured to serve the same origin hostname as is configured in the deliveryservice `Origin Server Base URL` field. See the `Apache Traffic Server docs <https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html>`_ for more information on that cache's implementation. +With This feature enabled, origin servers (or origin server VIP names for a site) are going to be entered as servers in to the Traiffic Ops UI. Server type is "ORG". -Parameters in the Origin profile that influence this feature: +Parameters in the mid profile that influence this feature: +--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ | Name | Filename | Default | Description | @@ -610,27 +611,40 @@ Parameters in the Origin profile that influence this feature: +--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ | CONFIG proxy.config. url_remap.remap_required | records.config | INT 1 | required for parent selection. | +--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_proxy.per_parent_connect_attempts | records.config | INT 5 | maximum of 5 connection attempts per parent (parent.config list) within a transaction. | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_proxy.total_connect_attempts | records.config | INT 10 | maximum of 10 total connection attempts within a transaction. | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_origin.simple_retry_enabled | records.config | INT 1 | enables simple retry. | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_origin.simple_retry_response_codes | records.config | STRING 404 | the response code that invokes simple retry. May be a comman separated list of response codes. | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_origin.dead_server_retry_response_codes | records.config | STRING 503 | the response code that invokes dead server retry. May be a comma separated list of response codes | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. http.parent_origin.dead_server_retry_enabled | records.config | INT 1 | enables dead server retry. | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ -| CONFIG proxy.config. diags.debug.enabled | records.config | INT 1 | enable debugging for testing only | -+--------------------------------------------------------------------------+----------------+------------+----------------------------------------------------------------------------------------------------+ + + +Parameters in the deliveryservice profile that influence this feature: + ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| Name | Filename | Default | Description | ++=============================================+================+=================+=================================================================================================================================+ +| mso.parent_retry | parent.config | \- | Either ``simple_retry``, ``dead_server_retry`` or ``both``. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| mso.algorithm | parent.config | consistent_hash | The algorithm to use. ``consisten_hash``, ``strict``, ``true``, ``false``, or ``latched``. | +| | | | | +| | | | - ``consisten_hash`` - spreads requests across multiple parents simultaneously based on hash of content URL. | +| | | | - ``strict`` - strict Round Robin spreads requests across multiple parents simultaneously based on order of requests. | +| | | | - ``true`` - same as strict, but ensures that requests from the same IP always go to the same parent if available. | +| | | | - ``false`` - uses only a single parent at any given time and switches to a new parent only if the current parent fails. | +| | | | - ``latched`` - same as false, but now, a failed parent will not be retried. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| mso.unavailable_server_retry_response_codes | parent.config | "503" | Quoted, comma separated list of HTTP status codes that count as a unavailable_server_retry_response_code. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| mso.max_unavailable_server_retries | parent.config | 1 | How many times an unavailable server will be retried. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| mso.simple_retry_response_codes | parent.config | "404" | Quoted, comma separated list of HTTP status codes that count as a simple retry response code. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ +| mso.max_simple_retries | parent.config | 1 | How many times a simple retry will be done. | ++---------------------------------------------+----------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------+ + + see :ref:`rl-multi-site-origin-qht` for a *quick how to* on this feature. .. _rl-ccr-profile: -CCR Profile or Traffic Router Profile -+++++++++++++++++++++++++++++++++++++ +Traffic Router Profile +++++++++++++++++++++++ +-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Name | Config_file | Description | @@ -655,8 +669,6 @@ CCR Profile or Traffic Router Profile +-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | coveragezone.polling.url | CRConfig.json | The location (URL) to retrieve the coverage zone map file in XML format from. | +-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ -| domain_name | CRConfig.json | The top level domain of this Traffic Router instance. | -+-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | tld.soa.expire | CRConfig.json | The value for the expire field the Traffic Router DNS Server will respond with on Start of Authority (SOA) records. | +-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | tld.soa.minimum | CRConfig.json | The value for the minimum field the Traffic Router DNS Server will respond with on SOA records. | @@ -719,6 +731,21 @@ CCR Profile or Traffic Router Profile +-----------------------------------------+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ +.. index:: + Regex Remap Expression + +.. _rl-regex-remap: + +Regex Remap Expression +++++++++++++++++++++++ +The regex remap expression allows to to use a regex and resulting match group(s) in order to modify the request URIs that are sent to origin. For example: :: + + ^/original/(.*) http://origin.example.com/remapped/$1 + +.. Note:: If **Query String Handling** is set to ``2 Drop at edge``, then you will not be allowed to save a regex remap expression, as dropping query strings actually relies on a regex remap of its own. However, if there is a need to both drop query strings **and** remap request URIs, this can be accomplished by setting **Query String Handling** to ``1 Do not use in cache key, but pass up to origin``, and then using a custom regex remap expression to do the necessary remapping, while simultaneously dropping query strings. The following example will capture the original request URI up to, but not including, the query string and then forward to a remapped URI: :: + + ^/([^?]*).* http://origin.example.com/remapped/$1 + .. index:: HOST_REGEXP PATH_REGEXP http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_router.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_router.txt b/docs/latest/_sources/admin/traffic_router.txt index 4fe4759..33ba4ee 100644 --- a/docs/latest/_sources/admin/traffic_router.txt +++ b/docs/latest/_sources/admin/traffic_router.txt @@ -47,6 +47,8 @@ The following are requirements to ensure an accurate set up: 6. Start Tomcat: ``sudo service tomcat start``, and test lookups with dig and curl against that server. + To restart, ``sudo service tomcat stop``, kill the traffic router process, and ``sudo service tomcat start`` + Also, crconfig previously recieved will be cached, and needs to be removed manually to actually be reloaded /opt/traffic_router/db/cr-config.json 7. Snapshot CRConfig; See :ref:`rl-snapshot-crconfig` .. Note:: Once the CRConfig is snapshotted, live traffic will be sent to the new Traffic Routers provided that their status is set to ``ONLINE``. http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_server.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_server.txt b/docs/latest/_sources/admin/traffic_server.txt index 1b99ce0..9283708 100644 --- a/docs/latest/_sources/admin/traffic_server.txt +++ b/docs/latest/_sources/admin/traffic_server.txt @@ -18,26 +18,53 @@ Traffic Server Administration ***************************** Installing Traffic Server ========================= -1. Select **Servers** in the Traffic Ops web interface. -2. Scroll to the bottom of the page and click **Add Server**. -3. Complete the *Required Info*: section. -4. Click **Submit**. -5. Click **Save**. -.. 6. Click **Online Server**. -.. 7. From the Set status of this machine to ONLINE? screen, click **OK**. +#. Get the Traffic Server RPM and the astats RPM. + + Sample command: :: + + wget http://traffic-control-cdn.net/downloads/1.7.0/RELEASE-1.7.0/trafficserver-5.3.2-759.ee14bbe.el6.x86_64.rpm + wget http://traffic-control-cdn.net/downloads/1.6.1/RELEASE-1.6.1/astats_over_http-1.2-8.el6.x86_64.rpm + + (astats was not released as part of 1.7, so in this examples 1.6.1 was used) +#. Install Traffic Server and astats: :: + + sudo yum -y install trafficserver-*.rpm astats_over_http*.rpm + +#. Add the server using the Traffic Ops web interface: + + #. Select **Servers**. + #. Scroll to the bottom of the page and click **Add Server**. + #. Complete the "Required Info:" section: + * Set 'Interface Name' to the name of the interface from which traffic server delivers content. + * Set 'Type' to 'MID' or 'EDGE'. + + #. Click **Submit**. + #. Click **Save**. + #. Click **Online Server**. + #. Verify that the server status is now listed as **Reported** + +#. Install the ORT script and run it in 'badass' mode to create the initial configuration, see :ref:`reference-traffic-ops-ort` + +#. Start the service: ``sudo service trafficserver start`` + +#. Configure traffic server to start automatically: ``sudo chkconfig trafficserver on`` + +#. Verify that the installation is good: + + #. Make sure that the service is running: ``sudo service trafficserver status`` + + #. Assuming a traffic monitor is already installed, browse to it, and verify that the traffic server appears in the "Cahce States" table, in white. + .. _reference-traffic-ops-ort: Configuring Traffic Server ========================== All of the Traffic Server application configuration files are generated by Traffic Ops and installed by way of the traffic_ops_ort.pl script. +The traffic_ops_ort.pl should be installed on all caches (by puppet or other non Traffic Ops means), usually in /opt/ort. It is used to do the initial install of the config files when the cache is being deployed, and to keep the config files up to date when the cache is already in service. The usage message of the script is shown below: :: - -**traffic_ops_ort.pl** - The traffic_ops_ort.pl should be installed on all caches (by puppet or other non Traffic Ops means), usually in /opt/ort. It is used to do initial install of the config files when the cache is being deployed, and to keep the config files up to date when the cache is already in service. The usage message of the script is shown below: :: - - $ sudo ./traffic_ops_ort.pl syncds warn https://to.cdn.kabletown.net + $ sudo /opt/ort/traffic_ops_ort.pl Thu May 26 15:52:11 UTC 2016 ====-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==== Usage: ./traffic_ops_ort.pl <Mode> <Log_Level> <Traffic_Ops_URL> <Traffic_Ops_Login> [optional flags] @@ -59,13 +86,58 @@ All of the Traffic Server application configuration files are generated by Traff ====-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==== $ +Installing the ORT script +-------------------------- + +#. The ORT script is not a part of the traffic server distribution. In this sample session, we get it manually from github: :: + + sudo mkdir /opt/ort + sudo wget -P /opt/ort https://raw.githubusercontent.com/apache/incubator-trafficcontrol/1.7.x/traffic_ops/bin/traffic_ops_ort.pl + sudo chmod +x /opt/ort/traffic_ops_ort.pl + +#. Install modules required by ORT: ``sudo yum -y install perl-JSON perl-Crypt-SSLeay`` + +#. For initial configuration or when major changes (like a Profile change) need to be made, run the script in "badass mode". All required rpm packages + will be installed, all Traffic Server config files will be fetched and installed, and (if needed) the Traffic Server application will be restarted. + + Example run below: :: + + $ sudo /opt/ort/traffic_ops_ort.pl --dispersion=0 badass warn https://ops.$tcDomain admin:admin123 + + .. Note:: First run gives a lot of state errors that are expected. The "badass" mode fixes these issue s. Run it a second time, this should be cleaner. + Also, note that many ERROR messages emitted by ORT are actually information messages. Do not panic. + +#. Create a cron entry for running ort in 'syncds' mode every 15 minutes. + This makes traffic control check periodically if 'Queue Updates' was run on Traffic Ops, and it so, get the updated configuration. + + Run ``sudo crontab -e`` and add the following line :: + + */15 * * * * /opt/ort/traffic_ops_ort.pl syncds warn https://traffops.kabletown.net admin:password --login_dispersion=30 --dispersion=180 > /tmp/ort/syncds.log 2>&1 + + Changing ``https://traffops.kabletown.net``, ``admin``, and ``password`` to your CDN URL and credentials. + + .. Note:: By default, running ort on an edge traffic server waits for it's parent (mid) servers to download their configuration before + it downloads it's own configuration. Because of this, scheduling ort for running every 15 minutes (with 5 minutes default dispersion) means + that it might take up to ~35 minutes for a "Queue Updates" operation to affect all traffic servers. To customize this dispersion time, use + the command line option --dispersion=x where x is the number of seconds for the dispersion period. Servers will select a random number from + within this dispersion period to being pulling down configuration files from Traffic Ops. Another option, --login_dispersion=x can be used. + This option creates a dispersion period after the job begins during which ORT will wait before logging in and checking Traffic Ops for updates + to the server. This defaults to 0. If use_reval_pending, a.k.a. Rapid Revalidate is enabled, edges will NOT wait for their parents to download + their configuration before downloading their own. + + .. Note:: In 'syncds' mode, the ort script updates only configurations that might be changed as part of normal operations, such as: + + * Delivery Services + * SSL certificates + * Traffic Monitor IP addresses + * Logging configuration + * Revalidation requests (By default. If Rapid Revalidate is enabled, this will only be checked by using a separate revalidate command in ORT.) - For initial configuration or when major changes (like a Profile change) need to be made, run the script in "badass mode". All required rpm packages will be installed, all Traffic Server config files will be fetched and installed, and (if needed) the Traffic Server application will be restarted. Example run below: :: - run here +#. If Rapid Revalidate is enabled in Traffic Ops, create a second cron job for revalidation checks. ORT will not check revalidation files if Rapid Revalidate + is enabled. This setting allows for a separate check to be performed every 60 seconds to verify if a revalidation update has been made. - For "every day changes" such as adding deliveryservices or changing records.config parameters caches should run the script in "syncds" mode out of cron. Example crontab entry: :: + Run ``sudo crontab -e`` and add the following line :: - */15 * * * * /opt/ort/traffic_ops_ort.pl syncds warn https://traffops.kabletown.net admin:password > /tmp/ort/syncds.log 2>&1 + */1 * * * * /opt/ort/traffic_ops_ort.pl revalidate warn https://traffops.kabletown.net admin:password --login_dispersion=30 > /tmp/ort/syncds.log 2>&1 - .. Note:: <disclaimer on what is "hot changeable" here> http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/admin/traffic_stats.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/admin/traffic_stats.txt b/docs/latest/_sources/admin/traffic_stats.txt index df720ae..2355202 100644 --- a/docs/latest/_sources/admin/traffic_stats.txt +++ b/docs/latest/_sources/admin/traffic_stats.txt @@ -24,9 +24,10 @@ Installation **Installing Traffic Stats:** - - Download the Traffic Stats RPM from the traffic control `downloads <https://trafficcontrol.apache.org/downloads/index.html>`_ page. - - Copy the Traffic Stats RPM to your server - - sudo rpm -ivh <traffic_stats rpm> + - See the `downloads <https://trafficcontrol.apache.org/downloads/index.html>`_ page for Traffic Control to get the lastest release. + - Follow our build `intructions <https://github.com/apache/incubator-trafficcontrol/tree/master/build>`_ to generate an RPM. + - Copy the RPM to your server + - perform the following command: ``sudo rpm -ivh <traffic_stats rpm>`` **Installing InfluxDB:** @@ -128,30 +129,30 @@ InfluxDb Tools Under the Traffic Stats source directory there is a directory called influxdb_tools. These tools are meant to be used as one-off scripts to help a user quickly get new databases and continuous queries setup in influxdb. They are specific for traffic stats and are not meant to be generic to influxdb. Below is an brief description of each script along with how to use it. -**create_ts_databases** +**create/create_ts_databases.go** This script creates all `databases <https://docs.influxdata.com/influxdb/latest/concepts/key_concepts/#database>`_, `retention policies <https://docs.influxdata.com/influxdb/latest/concepts/key_concepts/#retention-policy>`_, and `continuous queries <https://docs.influxdata.com/influxdb/v0.11/query_language/continuous_queries/>`_ required by traffic stats. **How to use create_ts_databases:** Pre-Requisites: - 1. Go 1.4 or later + 1. Go 1.7 or later 2. configured $GOPATH (e.g. export GOPATH=~/go) Using create_ts_databases.go - 1. go get github.com/influxdata/influxdb + 1. go to the traffic_stats/influxdb_tools/create directory - 2. go build create_ts_databases.go + 2. build it by running ``go build create_ts_databases.go`` or simply ``go build`` 3. Run it: - - ./create_ts_databases -help + - ``./create_ts_databases -help`` or ``./create -help`` - optional flags: - url - The influxdb url and port - replication - The number of nodes in the cluster - user - The user to use - password - The password to use - - example: ./create_ts_databases -url=localhost:8086 -replication=3 -user=joe -password=mysecret + - example: ``./create_ts_databases -url=localhost:8086 -replication=3 -user=joe -password=mysecret`` or ``./create -url=localhost:8086 -replication=3 -user=joe -password=mysecret`` **sync_ts_databases** This script is used to sync one influxdb environment to another. Only data from continuous queries is synced as it is downsampled data and much smaller in size than syncing raw data. Possible use cases are syncing from Production to Development or Syncing a new cluster once brought online. @@ -160,28 +161,28 @@ They are specific for traffic stats and are not meant to be generic to influxdb. Pre-Requisites: - 1. Go 1.4 or later + 1. Go 1.7 or later 2. configured $GOPATH (e.g. export GOPATH=~/go) Using sync_ts_databases.go: - 1. go get github.com/influxdata/influxdb + 1. go to the traffic_stats/influxdb_tools/create directory - 2. go build sync_ts_databases.go + 2. build it by running ``go build sync_ts_databases.go`` or simply ``go build`` 3. Run it - - ./sync_ts_databases -help + - ``./sync_ts_databases -help`` or ``./sync -help`` - required flags: - - sourceUrl - The URL of the source database - - targetUrl - The URL of the target database + - source-url - The URL of the source database + - target-url - The URL of the target database -optional flags: - database - The database to sync (default = sync all databases) - days - Days in the past to sync (default = sync all data) - - sourceUser - The user of the source database - - sourcePass - The password for the source database - - targetUser - The user of the target database - - targetPass - The password for the target database + - source-user - The user of the source database + - source-pass - The password for the source database + - target-user - The user of the target database + - target-pass - The password for the target database - - example: ./sync_ts_databases -sourceUrl=http://influxdb-production-01.kabletown.net:8086 -targetUrl=http://influxdb-dev-01.kabletown.net:8086 -database=cache_stats -days=7 -sourceUser=joe sourcePass=mysecret + - example: `./sync -source-url=http://idb-01.foo.net:8086 -target-url=http://idb-01.foo.net:8086 -database=cache_stats -days=7 -source-user=admin source-pass=mysecret` http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/development/building.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/development/building.txt b/docs/latest/_sources/development/building.txt new file mode 100644 index 0000000..87e68e9 --- /dev/null +++ b/docs/latest/_sources/development/building.txt @@ -0,0 +1,94 @@ +.. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. +.. + +.. _dev-building: + +Building Traffic Control +======================== + + +Build using pkg +--------------- + +This is the easiest way to build all the components of Traffic Control; +all requirements are automatically loaded into the image used to build +each component. + +Requirements +~~~~~~~~~~~~ + +- ``docker`` (https://docs.docker.com/engine/installation/) +- ``docker-compose`` (https://docs.docker.com/compose/install/) + (optional, but recommended) + +If ``docker-compose`` is not available, the ``pkg`` script will +automatically download and run it in a container. This is noticeably +slower than running it natively. + +Usage +~~~~~ + +:: + + $ ./pkg -? + Usage: ./pkg [options] [projects] + -q Quiet mode. Supresses output. + -v Verbose mode. Lists all build output. + -l List available projects. + + If no projects are listed, all projects will be packaged. + Valid projects: + - traffic_portal_build + - traffic_router_build + - traffic_monitor_build + - source + - traffic_ops_build + - traffic_stats_build + + +If any project names are provided on the command line, only those will be built. +Otherwise, all projects are built. + +All artifacts (rpms, logs, source tar ball) are copied to ``dist`` at the top level of the +``incubator-trafficcontrol`` directory. + +Example +~~~~~~~ + +:: + + $ ./pkg -v source traffic_ops_build + Building source. + Building traffic_ops_build. + +Build using docker-compose +-------------------------- + +If the ``pkg`` script fails, ``docker-compose`` can still be used directly. + +Usage +~~~~~ + +:: + + $ docker-compose -f ./infrastructure/docker/build/docker-compose.yml down -v + $ docker-compose -f ./infrastructure/docker/build/docker-compose.yml up --build source traffic_ops_build + $ ls -1 dist/ + build-traffic_ops.log + traffic_ops-2.1.0-6396.07033d6d.el7.src.rpm + traffic_ops-2.1.0-6396.07033d6d.el7.x86_64.rpm + traffic_ops_ort-2.1.0-6396.07033d6d.el7.src.rpm + traffic_ops_ort-2.1.0-6396.07033d6d.el7.x86_64.rpm + trafficcontrol-incubating-2.1.0.tar.gz http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/development/index.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/development/index.txt b/docs/latest/_sources/development/index.txt index 851e463..ecabe1d 100644 --- a/docs/latest/_sources/development/index.txt +++ b/docs/latest/_sources/development/index.txt @@ -20,10 +20,12 @@ Use this guide to start developing applications that consume the Traffic Control .. toctree:: :maxdepth: 2 + building traffic_ops traffic_portal traffic_router traffic_monitor + traffic_monitor_golang traffic_stats traffic_server http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/development/traffic_monitor_golang.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/development/traffic_monitor_golang.txt b/docs/latest/_sources/development/traffic_monitor_golang.txt new file mode 100644 index 0000000..118a623 --- /dev/null +++ b/docs/latest/_sources/development/traffic_monitor_golang.txt @@ -0,0 +1,295 @@ +.. +.. +.. Licensed under the Apache License, Version 2.0 (the "License"); +.. you may not use this file except in compliance with the License. +.. You may obtain a copy of the License at +.. +.. http://www.apache.org/licenses/LICENSE-2.0 +.. +.. Unless required by applicable law or agreed to in writing, software +.. distributed under the License is distributed on an "AS IS" BASIS, +.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +.. See the License for the specific language governing permissions and +.. limitations under the License. +.. + +Traffic Monitor Golang +********************** +Introduction +============ +The next major version of Traffic Monitor has been completely rewritten in Golang. Currently, this version is functionally equivalent, and should be considered "beta." It is recommended that new CDN deployments continue to use the existing Java version, until the new version is completely moved over in the source and binary distributions. However, developers and administrators are encouraged to test the Golang version, to prepare for operational differences and look for bugs. + +Traffic Monitor is an HTTP service application that monitors caches, provides health state information to Traffic Router, and collects statistics for use in tools such as Traffic Ops and Traffic Stats. The health state provided by Traffic Monitor is used by Traffic Router to control which caches are available on the CDN. + +Software Requirements +===================== +To work on Traffic Monitor you need a \*nix (MacOS and Linux are most commonly used) environment that has the following installed: + +* Golang + +Project Tree Overview +===================================== + +* ``traffic_control/traffic_monitor/`` - base directory for Traffic Monitor. + +* ``cache/`` - Handler for processing cache results. +* ``config/`` - Application configuration; in-memory objects from ``traffic_monitor.cfg``. +* ``crconfig/`` - struct for deserlializing the CRConfig from JSON. +* ``deliveryservice/`` - aggregates delivery service data from cache results. +* ``deliveryservicedata/`` - deliveryservice structs. This exists separate from ``deliveryservice`` to avoid circular dependencies. +* ``enum/`` - enumerations and name alias types. +* ``health/`` - functions for calculating cache health, and creating health event objects. +* ``manager/`` - manager goroutines (microthreads). + * ``health.go`` - Health request manager. Processes health results, from the health poller -> fetcher -> manager. The health poll is the "heartbeat" containing a small amount of stats, primarily to determine whether a cache is reachable as quickly as possible. Data is aggregated and inserted into shared threadsafe objects. + * ``manager.go`` - Contains ``Start`` function to start all pollers, handlers, and managers. + * ``monitorconfig.go`` - Monitor config manager. Gets data from the monitor config poller, which polls Traffic Ops for changes to which caches are monitored and how. + * ``opsconfig.go`` - Ops config manager. Gets data from the ops config poller, which polls Traffic Ops for changes to monitoring settings. + * ``peer.go`` - Peer manager. Gets data from the peer poller -> fetcher -> handler and aggregates it into the shared threadsafe objects. + * ``stat.go`` - Stat request manager. Processes stat results, from the stat poller -> fetcher -> manager. The stat poll is the large statistics poll, containing all stats (such as HTTP codes, transactions, delivery service statistics, and more). Data is aggregated and inserted into shared threadsafe objects. + * ``statecombiner.go`` - Manager for combining local and peer states, into a single combined states threadsafe object, for serving the CrStates endpoint. +* ``datareq/`` - HTTP routing, which has threadsafe health and stat objects populated by stat and health managers. +* ``peer/`` - Manager for getting and populating peer data from other Traffic Monitors +* ``srvhttp/`` - HTTP service. Given a map of endpoint functions, which are lambda closures containing aggregated data objects. +* ``static/`` - Web GUI HTML and javascript files +* ``threadsafe/`` - Threadsafe objects for storing aggregated data needed by multiple goroutines (typically the aggregator and HTTP server) +* ``trafficopsdata/`` - Struct for fetching and storing Traffic Ops data needed from the CRConfig. This is primarily mappings, such as delivery service servers, and server types. +* ``trafficopswrapper/`` - Threadsafe wrapper around the Traffic Ops client. The client used to not be threadsafe, however, it mostly (possibly entirely) is now. But, the wrapper also serves to overwrite the Traffic Ops ``monitoring.json`` values, which are live, with snapshotted CRConfig values. + +Architecture +============ +At the highest level, Traffic Monitor polls caches, aggregates their data and availability, and serves it at HTTP JSON endpoints. + +In the code, the data flows thru microthread (goroutine) pipelines. All stages of the pipeline are independent running microthreads:sup:`1`. The pipelines are: + +* **stat poll** - polls caches for all statistics data. This should be a slower poll, which gets a lot of data. +* **health poll** - polls caches for a tiny amount of data, typically system information. This poll is designed to be a heartbeat, determining quickly whether the cache is reachable. Since it's a small amount of data, it should poll more frequently. +* **peer poll** - polls Traffic Monitor peers for their availability data, and aggregates it with its own availability results and that of all other peers. +* **monitor config** - polls Traffic Ops for the list of Traffic Monitors and their info. +* **ops config** - polls for changes to the ops config file ``traffic_ops.cfg``, and sends updates to other pollers when the config file has changed. + + * The ops config manager also updates the shared Traffic Ops client, since it's the actor which becomes notified of config changes requiring a new client. + + * The ops config manager also manages, creates, and recreates the HTTP server, since ops config changes necessitate restarting the HTTP server. + +All microthreads in the pipeline are started by ``manager/manager.go:Start()``. + +:: + + -------------------- -------------------- -------------------- + | ops config poller |-->| ops config handler |-->| ops config manager |-->-restart HTTP server------------------------- + ------------------- -------------------- -------------------- | | + -->-ops config change subscriber------------- | + | | | + -->-Traffic Ops client change subscriber-- | | + | | | + ------------------------------------------------------------------------------------------------------------- | | + | | | + | ------------------------------------------------------------------------------------------------------------ | + | | | + \/ \/ | + ----------------------- ------------------------ | + | monitor config poller |-->| monitor config manager |-->-stat subscriber-------- ----------------------- + ----------------------- ------------------------ | | | + |->-health subscriber--- | \/ _ + | | | ------------- _( )._ + -->-peer subscriber-- | | | HTTP server |->-HTTP request-> (____)_) + | | | ------------- + ----------------------------------------------------------------------------- | | ^ + | | | | + | ----------------------------------------------------------------------------- | ------------------------ + | | | | + | | ----------------------------------------------------------------------------- | + | | | ^ + | | | ------------- -------------- -------------- -------------- ----------------------- + | | -->| stat poller |-->| stat fetcher |-->| stat handler |-->| stat manager |->--------set shared data->| shared data | + | | ------------- | -------------- -------------- | -------------- ----------------------- + | | | -------------- -------------- | | events | + | | |->| stat fetcher |-->| stat handler |-| | toData | + | | | -------------- -------------- | | errorCount | + | | ... ... | healthIteration | + | | | fetchCount | + | | --------------- ---------------- ---------------- ---------------- | localStates | + | ---->| health poller |-->| health fetcher |-->| health handler |-->| health manager |->-set shared data->| toSession | + | --------------- | ---------------- ---------------- | ---------------- | peerStates | + | | ---------------- ---------------- | | monitorConfig | + | |->| health fetcher |-->| health handler |-| | combinedStates | + | | ---------------- ---------------- | | statInfoHistory | + | ... ... | statResultHistory | + | | statMaxKbpses | + | ------------- -------------- -------------- -------------- | lastKbpsStats | + ------>| peer poller |-->| peer fetcher |-->| peer handler |-->| peer manager |->----------set shared data->| dsStats | + ------------- | -------------- -------------- | -------------- | localCacheStatus | + | -------------- -------------- | | lastHealthDurations | + |->| peer fetcher |-->| peer handler |-| | healthHistory | + | -------------- -------------- | ----------------------- + ... ... + +:sup:`1`Technically, some stages which are one-to-one simply call the next stage as a function. For example, the Fetcher calls the Handler as a function in the same microthread. But this isn't architecturally significant. + + +Stat Pipeline +------------- + +:: + + --------- --------- --------- --------- + | poller |-->| fetcher |-->| handler |-->| manager | + -------- | --------- --------- | --------- + | --------- --------- | + |->| fetcher |-->| handler |-| + | --------- --------- | + ... ... + +* **poller** - ``common/poller/poller.go:HttpPoller.Poll()``. Listens for config changes (from the ops config manager), and starts its own internal microthreads, one for each cache to poll. These internal microthreads call the Fetcher at each cache's poll interval. + +* **fetcher** - ``common/fetcher/fetcher.go:HttpFetcher.Fetch()``. Fetches the given URL, and passes the returned data to the Handler, along with any errors. + + +* **handler** - ``traffic_monitor/cache/cache.go:Handler.Handle()``. Takes the given result and does all data computation possible with the single result. Currently, this computation primarily involves processing the denormalized ATS data into Go structs, and processing System data into OutBytes, Kbps, etc. Precomputed data is then passed to its result channel, which is picked up by the Manager. + +* **manager** - ``traffic_monitor/manager/stat.go:StartStatHistoryManager()``. Takes preprocessed results, and aggregates them. Aggregated results are then placed in shared data structures. The major data aggregated are delivery service statistics, and cache availability data. See :ref:`Aggregated Stat Data` and :ref:`Aggregated Availability Data`. + + +Health Pipeline +--------------- + +:: + + --------- --------- --------- --------- + | poller |-->| fetcher |-->| handler |-->| manager | + -------- | --------- --------- | --------- + | --------- --------- | + |->| fetcher |-->| handler |-| + | --------- --------- | + ... ... + +* **poller** - ``common/poller/poller.go:HttpPoller.Poll()``. Same poller type as the Stat Poller pipeline, with a different handler object. + +* **fetcher** - ``common/fetcher/fetcher.go:HttpFetcher.Fetch()``. Same fetcher type as the Stat Poller pipeline, with a different handler object. + +* **handler** - ``traffic_monitor/cache/cache.go:Handler.Handle()``. Same handler type as the Stat Poller pipeline, but constructed with a flag to not precompute. The health endpoint is of the same form as the stat endpoint, but doesn't return all stat data. So, it doesn't precompute like the Stat Handler, but only processes the system data, and passes the processed result to its result channel, which is picked up by the Manager. + +* **manager** - ``traffic_monitor/manager/health.go:StartHealthResultManager()``. Takes preprocessed results, and aggregates them. For the Health pipeline, only health availability data is aggregated. Aggregated results are then placed in shared data structures (lastHealthDurationsThreadsafe, lastHealthEndTimes, etc). See :ref:`Aggregated Availability Data`. + + +Peer Pipeline +------------- + +:: + + --------- --------- --------- --------- + | poller |-->| fetcher |-->| handler |-->| manager | + -------- | --------- --------- | --------- + | --------- --------- | + |->| fetcher |-->| handler |-| + | --------- --------- | + ... ... + +* **poller** - ``common/poller/poller.go:HttpPoller.Poll()``. Same poller type as the Stat and Health Poller pipelines, with a different handler object. Its config changes come from the Monitor Config Manager, and it starts an internal microthread for each peer to poll. + +* **fetcher** - ``common/fetcher/fetcher.go:HttpFetcher.Fetch()``. Same fetcher type as the Stat and Health Poller pipeline, with a different handler object. + +* **handler** - ``traffic_monitor/cache/peer.go:Handler.Handle()``. Decodes the JSON result into an object, and without further processing passes to its result channel, which is picked up by the Manager. + +* **manager** - ``traffic_monitor/manager/peer.go:StartPeerManager()``. Takes JSON peer Traffic Monitor results, and aggregates them. The availability of the Peer Traffic Monitor itself, as well as all cache availability from the given peer result, is stored in the shared ``peerStates`` object. Results are then aggregated via a call to the ``combineState()`` lambda, which signals the State Combiner microthread (which stores the combined availability in the shared object ``combinedStates``; See :ref:`State Combiner`). + + +Monitor Config Pipeline +----------------------- + +:: + + --------- --------- + | poller |-->| manager |--> stat subscriber (Stat pipeline Poller) + -------- --------- | + |-> health subscriber (Health pipeline Poller) + | + --> peer subscriber (Peer pipeline Poller) + +* **poller** - ``common/poller/poller.go:MonitorConfigPoller.Poll()``. The Monitor Config poller, on its interval, polls Traffic Ops for the Monitor configuration, and writes the polled value to its result channel, which is read by the Manager. + +* **manager** - ``traffic_monitor/manager/monitorconfig.go:StartMonitorConfigManager()``. Listens for results from the poller, and processes them. Cache changes are written to channels read by the Health, Stat, and Peer pollers. In the Shared Data objects, this also sets the list of new delivery services and removes ones which no longer exist, and sets the list of peer Traffic Monitors. + + +Ops Config Pipeline +------------------- +:: + + --------- --------- --------- + | poller |-->| handler |-->| manager |--> ops config change subscriber (Monitor Config Poller) + -------- --------- --------- | + --> Traffic ops client change subscriber (Monitor Config Poller) + +* **poller** - ``common/poller/poller.go:FilePoller.Poll()``. Polls for changes to the Traffic Ops config file ``traffic_ops.cfg``, and writes the changed config to its result channel, which is read by the Handler. + +* **handler** - ``common/handler/handler.go:OpsConfigFileHandler.Listen()``. Takes the given raw config, unmarshalls the JSON into an object, and writes the object to its channel, which is read by the Manager, along with any error. + +* **manager** - ``traffic_monitor/manager/monitorconfig.go:StartMonitorConfigManager()``. Listens for new configs, and processes them. When a new config is received, a new HTTP dispatch map is created via ``traffic_monitor/datareq/datareq.go:MakeDispatchMap()``, and the HTTP server is restarted with the new dispatch map. The Traffic Ops client is also recreated, and stored in its shared data object. The Ops Config change subscribers and Traffic Ops Client change subscribers (the Monitor Config poller) are also passed the new ops config and new Traffic Ops client. + + +Events +------ +The ``events`` shared data object is passed to each pipeline microthread which needs to signal events. Most of them do. Events are then logged, and visible in the UI as well as an HTTP JSON endpoint. Most events are caches becoming available or unavailable, but include other things such as peer availability changes. + + +State Combiner +-------------- +The State Combiner is a microthread started in ``traffic_monitor/manager/manager.go:Start()`` via ``traffic_monitor/manager/statecombiner.go:StartStateCombiner()``, which listens for signals to combine states. It should be signaled by any pipeline which updates the local or peer availability shared data objects, ``localStates`` and ``peerStates``. It holds the threadsafe shared data objects for local states and peer states, so no data is passed or returned, only a signal. + +When a signal is received, it combines the local and peer states optimistically. That is, if a cache is marked available locally or by any peer, that cache is marked available in the combined states. There exists a variable to combine pessimistically, which may be set at compile time (it's unusual for a CDN to operate well with pessimistic cache availability). Combined data is stored in the threadsafe shared data object ``combinedStates``. + + +Aggregated Stat Data +-------------------- +The Stat pipeline Manager is responsible for aggregating stats from all caches, into delivery services statistics. This is done via a call to ``traffic_monitor/deliveryservice/stat.go:CreateStats()``. + + +Aggregated Availability Data +---------------------------- +Both the Stat and Health pipelines aggregate availability data received from caches. This is done via a call to ``traffic_monitor/deliveryservice/health.go:CalcAvailability()`` followed by a call to ``combineState()``. The ``CalcAvailability`` function calculates the availability of each cache from the result of polling it, that is, local availability. The ``combineState()`` function is a lambda passed to the Manager, which signals the State Combiner microthread, which will combine the local and peer Traffic Monitor availability data, and insert it into the shared data ``combinedStates`` object. + + +HTTP Data Requests +------------------ +Data is provided to HTTP requests via the threadsafe shared data objects (see :ref:`Shared Data`). These objects are closed in lambdas created via ``traffic_monitor/datareq/datareq.go:MakeDispatchMap()``. This is called by the Ops Config Manager when it recreates the HTTP server. + +Each HTTP endpoint is mapped to a function which closes around the shared data objects it needs, and takes the request data it needs (such as query parameters). Each endpoint function resides in its own file in ``traffic_monitor/datareq/``. Because each Go HTTP routing function must be a ``http.HandlerFunc``, wrapper functions take the endpoint functions and return ``http.HandlerFunc`` functions which call them, and which are stored in the dispatch map, to be registered with the HTTP server. + + +Shared Data +----------- +Processed and aggregated data must be shared between the end of the stat and health processing pipelines, and HTTP requests. The CSP paradigm of idiomatic Go does not work efficiently with storing and sharing state. While not idiomatic Go, shared mutexed data structures are faster and simpler than CSP manager microthreads for each data object. + +Traffic Monitor has many threadsafe shared data types and objects. All shared data objects can be seen in ``manager/manager.go:Start()``, where they are created and passed to the various pipeline stage microthreads that need them. Their respective types all include the word ``Threadsafe``, and can be found in ``traffic_monitor/threadsafe/`` as well as, for dependency reasons, various appropriate directories. + +Currently, all Threadsafe shared data types use mutexes. In the future, these could be changed to lock-free or wait-free structures, if the performance needs outweighed the readability and correctness costs. They could also easily be changed to internally be manager microthreads and channels, if being idiomatic were deemed more important than readability or performance. + + + +Formatting Conventions +=========================== +Go code should be formatted with ``gofmt``. See also ``CONTRIBUTING.md``. + +Installing The Developer Environment +==================================== +To install the Traffic Monitor Developer environment: + +1. Install `go` version 1.7 or greater, from https://golang.org/doc/install and https://golang.org/doc/code.html +2. Clone the traffic_control repository using Git, into ``$GOPATH/src/github.com/apache/incubator-trafficcontrol`` +3. Change directories into ``$GOPATH/src/github.com/apache/incubator-trafficcontrol/traffic_monitor_golang/traffic_monitor`` +4. Run ``./build.sh`` + +Test Cases +========== +Tests can be executed by running ``go test ./...`` at the root of the ``traffic_monitor_golang`` project. + +API +=== + +:ref:`reference-tm-api` + +.. toctree:: + :hidden: + :maxdepth: 1 + + traffic_monitor/traffic_monitor_api http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/development/traffic_ops.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/development/traffic_ops.txt b/docs/latest/_sources/development/traffic_ops.txt index 9e30148..b18bc26 100644 --- a/docs/latest/_sources/development/traffic_ops.txt +++ b/docs/latest/_sources/development/traffic_ops.txt @@ -597,22 +597,28 @@ API 1.1 Reference traffic_ops_api/v11/type traffic_ops_api/v11/user -API 1.2 Reference +API 1.2 Reference ----------------- .. toctree:: :maxdepth: 1 + traffic_ops_api/v12/api_capability traffic_ops_api/v12/asn traffic_ops_api/v12/cachegroup + traffic_ops_api/v12/cachegroup_parameter traffic_ops_api/v12/cache_stats + traffic_ops_api/v12/capability traffic_ops_api/v12/cdn traffic_ops_api/v12/changelog + traffic_ops_api/v12/configfiles_ats traffic_ops_api/v12/deliveryservice + traffic_ops_api/v12/deliveryservice_regex traffic_ops_api/v12/deliveryservice_stats traffic_ops_api/v12/division traffic_ops_api/v12/federation traffic_ops_api/v12/hwinfo + traffic_ops_api/v12/job traffic_ops_api/v12/parameter traffic_ops_api/v12/phys_location traffic_ops_api/v12/profile @@ -623,7 +629,9 @@ API 1.2 Reference traffic_ops_api/v12/server traffic_ops_api/v12/static_dns traffic_ops_api/v12/status + traffic_ops_api/v12/steering_target traffic_ops_api/v12/system + traffic_ops_api/v12/tenant traffic_ops_api/v12/to_extension traffic_ops_api/v12/type traffic_ops_api/v12/user http://git-wip-us.apache.org/repos/asf/incubator-trafficcontrol-website/blob/42d25dc5/docs/latest/_sources/development/traffic_ops_api/routes.txt ---------------------------------------------------------------------- diff --git a/docs/latest/_sources/development/traffic_ops_api/routes.txt b/docs/latest/_sources/development/traffic_ops_api/routes.txt index ec90963..1c849b3 100644 --- a/docs/latest/_sources/development/traffic_ops_api/routes.txt +++ b/docs/latest/_sources/development/traffic_ops_api/routes.txt @@ -114,3 +114,5 @@ API Routes +------------------------------------+----------------------------------------------------+----------------------------------------------------+ | /datauser/orderby/:field | :ref:`to-api-v11-users-route` | :ref:`to-api-v12-users-route` | +------------------------------------+----------------------------------------------------+----------------------------------------------------+ +| *Not Implemented* | *Not Implemented* | :ref:`to-api-v12-configfiles_ats-route` | ++------------------------------------+----------------------------------------------------+----------------------------------------------------+
