This is an automated email from the ASF dual-hosted git repository. bneradt pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/trafficserver.git
The following commit(s) were added to refs/heads/master by this push: new 075dcda4ae Core implementation of IP categories for ip_allow.yaml (#11004) 075dcda4ae is described below commit 075dcda4ae01338b155846917db6035fd0788d5a Author: Brian Neradt <brian.ner...@gmail.com> AuthorDate: Mon Feb 5 17:49:25 2024 -0600 Core implementation of IP categories for ip_allow.yaml (#11004) Adds the ip_category feature for ip_allow.yaml so that users can specify arbitrary IP descriptions like the following: ```yaml ip_allow: - apply: in ip_category: ACME_INTERNAL action: allow methods: - GET - HEAD - POST - PUSH - DELETE ``` IP categories are defined via an `ip_categories` root node or through a file configured via proxy.config.cache.ip_categories.filename. --- configs/ip_allow.schema.json | 27 +- doc/admin-guide/files/ip_allow.yaml.en.rst | 68 ++++- doc/admin-guide/files/records.yaml.en.rst | 18 ++ doc/admin-guide/files/remap.config.en.rst | 17 +- include/proxy/IPAllow.h | 77 ++++- include/proxy/http/remap/AclFiltering.h | 33 ++- include/proxy/http/remap/RemapConfig.h | 14 +- include/tscore/Filenames.h | 1 + src/mgmt/config/AddConfigFilesHere.cc | 1 + src/proxy/IPAllow.cc | 248 ++++++++++++++-- src/proxy/http/remap/AclFiltering.cc | 20 ++ src/proxy/http/remap/RemapConfig.cc | 41 ++- src/proxy/http/remap/UrlRewrite.cc | 19 +- src/records/RecordsConfig.cc | 2 + src/traffic_server/traffic_server.cc | 4 + tests/gold_tests/ip_allow/ip_category.test.py | 326 +++++++++++++++++++++ .../replays/https_categories_all.replay.yaml | 94 ++++++ .../replays/https_categories_external.replay.yaml | 92 ++++++ .../https_categories_external_remap.replay.yaml | 92 ++++++ .../replays/https_categories_internal.replay.yaml | 106 +++++++ .../replays/https_categories_server.replay.yaml | 94 ++++++ 21 files changed, 1340 insertions(+), 54 deletions(-) diff --git a/configs/ip_allow.schema.json b/configs/ip_allow.schema.json index 6c9a8853d4..07564ceff1 100644 --- a/configs/ip_allow.schema.json +++ b/configs/ip_allow.schema.json @@ -22,6 +22,10 @@ "description": "A range of IP addresses in a single family.", "type": "string" }, + "category": { + "description": "An IP category representing a set of IP ranges.", + "type": "string" + }, "action": { "description": "Enforcement action.", "type": "string", @@ -68,6 +72,20 @@ } ] }, + "ip_categories": { + "oneOf": [ + { + "$ref": "#/definitions/category" + }, + { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/definitions/category" + } + } + ] + }, "action": { "$ref": "#/definitions/action" }, @@ -75,7 +93,14 @@ "$ref": "#/definitions/methods" } }, - "required": [ "apply", "ip_addrs", "action" ] + "oneOf": [ + { + "required": [ "apply", "ip_addrs", "action" ] + }, + { + "required": [ "apply", "ip_categories", "action" ] + } + ] } } } diff --git a/doc/admin-guide/files/ip_allow.yaml.en.rst b/doc/admin-guide/files/ip_allow.yaml.en.rst index db2e58dfa4..83b8458b9a 100644 --- a/doc/admin-guide/files/ip_allow.yaml.en.rst +++ b/doc/admin-guide/files/ip_allow.yaml.en.rst @@ -26,7 +26,7 @@ The :file:`ip_allow.yaml` file controls client access to |TS| and |TS| connectio This control is specified via rules. Each rule has: * A direction (inbound or out). -* A range of IP address to which the rule applies. +* A range of IP addresses or an IP category to which the rule applies. * An action, either accept or deny. * A list of HTTP methods. @@ -82,7 +82,21 @@ The keys in a rule are: ``ip_addrs`` IP addresses to match for the rule to be applied. This can be either an address range or an - array of address ranges. This is a required key. + array of address ranges. Either this or ``ip_categories`` are required keys for a rule. + +``ip_categories`` + A user defined string identifying a category of IP addresses relevant to a particular network. + For example, ``ACME_INTERNAL`` might represent the set of IP addresses for hosts within a + company's network. ``ACME_EXTERNAL`` might represet hosts belonging to the company's network, but + which are outside the company's firewall. ``ACME_ALL`` could be used to represent the set of both + of these categories. Multiple categories can be specified as an array of strings. + + The set of IP ranges belonging to each category is specified via the separate ``ip_categories`` + root level node. The :file:`ip_allow.yaml` parser also supports supplying the IP categories via + an external file specified with the :ts:cv:`proxy.config.cache.ip_categories.filename` + configuration. + + Either this or ``ip_addrs`` are required keys for a rule. ``action`` The action, which must be ``allow`` or ``deny``. This is a required key. @@ -93,8 +107,9 @@ The keys in a rule are: keyword "ALL" means all methods, making the specification of any other method redundant. All methods comparisons are case insensitive. This is an optional key. -An IP address range can be specified in several ways. A range is always IPv4 or IPv6, it is not -allowed to have a range that contains addresses from different IP address families. +An IP address range for ``ip_addrs`` or ``ip_categories`` can be specified in several ways. A range +is always IPv4 or IPv6, it is not allowed to have a range that contains addresses from different IP +address families. * A single address, which specifies a range of size 1, e.g. "127.0.0.1". * A minimum and maximum address separated by a dash, e.g. "10.1.0.0-10.1.255.255". @@ -131,7 +146,7 @@ enables all methods for all outbound connections. Examples ======== -The following example enables all clients access.:: +The following example enables all clients access:: apply: in ip_addrs: 0.0.0.0-255.255.255.255 @@ -222,7 +237,7 @@ This will match the IP address for the target servers on the outbound connection method is ``GET`` or ``HEAD`` the connection will be allowed, otherwise the connection will be denied. -As a final example, here is the default configuration in compact form:: +For the purposes of illustration, here is the default configuration in compact form:: ip_allow: [ { apply: in, ip_addrs: 127.0.0.1, action: allow }, @@ -231,6 +246,47 @@ As a final example, here is the default configuration in compact form:: { apply: in, ip_addrs: "::/0", action: deny, methods: [ PURGE, PUSH, DELETE, TRACE ] } ] +The following example demonstrates how to use ``ip_categories``. In this example, the +``ip_categories`` is ``ACME_INTERNAL`` which is presumably associated with trusted internal IP +addresses and thus are allowed to ``POST`` and ``DELETE`` resources. + +Note this example demonstrates that it is OK to mix ``ip_categories`` and ``ip_addrs`` rules in a +single :file:`ip_allow.yaml` file. In this case all other IPv4 addresses not matched on +``ACME_INTERNAL`` match on ``0/0`` and can only perform ``GET`` and ``HEAD`` requests:: + + - apply: in + ip_categories: ACME_INTERNAL + action: allow + methods: + - GET + - HEAD + - POST + - DELETE + - apply: in + ip_addrs: 0/0 + action: allow + methods: + - GET + - HEAD + +The set of IP addresses associated with ``ACME_INTERNAL`` can be specified +using the ``ip_categories`` node like so:: + + ip_categories: + - name: ACME_INTERNAL + ip_addrs: + - 10.0.0.0/8 + - 172.16.0.0/20 + - 192.168.1.0/24 + + ip_allow: + - apply: in + # ... + +The ``ip_categories`` node will generally be at the start of the :file:`ip_allow.yaml` file. +Alternatively, the same content with the ``ip_categories`` root node can exist in a separate file +specified with the :ts:cv:`proxy.config.cache.ip_categories.filename` configuration. + .. note:: For ATS 9.0, this file is (almost) backwards compatible. If the first line is a single '#' diff --git a/doc/admin-guide/files/records.yaml.en.rst b/doc/admin-guide/files/records.yaml.en.rst index 68a312aad2..afb7437562 100644 --- a/doc/admin-guide/files/records.yaml.en.rst +++ b/doc/admin-guide/files/records.yaml.en.rst @@ -2090,6 +2090,24 @@ Security this value via a :ref:`host_sni_policy<override-host-sni-policy>` attribute. +IP Allow +======== + +.. ts:cv:: CONFIG proxy.config.cache.ip_allow.filename STRING ip_allow.yaml + :reloadable: + + Set the file path for the IP allow configuration file. For details of the use + of this file, see :file:`ip_allow.yaml`. If this is a relative path, |TS| + loads it relative to the ``SYSCONFDIR`` directory. + +.. ts:cv:: CONFIG proxy.config.cache.ip_categories.filename STRING NULL + :reloadable: + + Set the file path for the IP allow categories definition file. For details of + the use of this file, see :file:`ip_allow.yaml`. If this is a relative path, + |TS| loads it relative to the ``SYSCONFDIR`` directory. + + Cache Control ============= diff --git a/doc/admin-guide/files/remap.config.en.rst b/doc/admin-guide/files/remap.config.en.rst index 407e4fde03..b1c7d0e1db 100644 --- a/doc/admin-guide/files/remap.config.en.rst +++ b/doc/admin-guide/files/remap.config.en.rst @@ -428,6 +428,13 @@ This will pass "1" and "2" to plugin1.so and "3" to plugin2.so .. _remap-config-named-filters: +NextHop Selection Strategies +============================ + +You may configure Nexthop or Parent hierarchical caching rules by remap using the +**@strategy** tag. See :doc:`../configuration/hierarchical-caching.en` and :doc:`strategies.yaml.en` +for configuration details and examples. + Acl Filters =========== @@ -455,10 +462,13 @@ Examples map http://foo.example.com/ http://foo.example.com/ @action=allow @src_ip=127.0.0.1 @method=post @method=get @method=head + map http://foo.example.com/ http://foo.example.com/ @action=allow @src_ip_category=ACME_INTERNAL @method=post @method=get @method=head + Note that these Acl filters will return a 403 response if the resource is restricted. The difference between ``@src_ip`` and ``@in_ip`` is that the ``@src_ip`` is the client ip and the ``in_ip`` is the ip address the client is connecting to (the incoming address). +``@src_ip_category`` functions like ``ip_category`` described in :file:`ip_allow.yaml`. Named Filters ============= @@ -516,13 +526,6 @@ would be :: Note this entirely disables IP Allow checks for those remap rules. -NextHop Selection Strategies -============================ - -You may configure Nexthop or Parent hierarchical caching rules by remap using the -**@strategy** tag. See :doc:`../configuration/hierarchical-caching.en` and :doc:`strategies.yaml.en` -for configuration details and examples. - Including Additional Remap Files ================================ diff --git a/include/proxy/IPAllow.h b/include/proxy/IPAllow.h index 3c332e2ce8..8dc5545e65 100644 --- a/include/proxy/IPAllow.h +++ b/include/proxy/IPAllow.h @@ -32,7 +32,6 @@ #include <string> #include <string_view> -#include <vector> #include "proxy/hdrs/HTTP.h" #include "iocore/eventsystem/ConfigProcessor.h" @@ -89,6 +88,7 @@ public: using self_type = IpAllow; ///< Self reference type. using scoped_config = ConfigProcessor::scoped_config<self_type, self_type>; using IpMap = swoc::IPSpace<Record const *>; + using IpCategories = std::unordered_map<std::string, swoc::IPSpace<bool>>; // indicator for whether we should be checking the acl record for src ip or dest ip enum match_key_t { SRC_ADDR, DST_ADDR }; @@ -102,8 +102,38 @@ public: static constexpr swoc::TextView OPT_METHOD{"method"}; static constexpr swoc::TextView OPT_METHOD_ALL{"all"}; + /* + * A YAML configuration file looks something like this: + * + * ip_categories: + * - name: ACME_INTERNAL + * ip_addrs: + * - 10.0.0.0/8 + * - 172.16.0.0/20 + * - 192.168.1.0/24 + * + * ip_allow: + * - apply: in + * ip_categories: ACME_INTERNAL + * action: allow + * methods: + * - GET + * - HEAD + * - POST + * - apply: in + * ip_addrs: 127.0.0.1 + * action: allow + * methods: ALL + * + */ static const inline std::string YAML_TAG_ROOT{"ip_allow"}; + + static const inline std::string YAML_TAG_CATEGORY_ROOT{"ip_categories"}; + static const inline std::string YAML_TAG_CATEGORY_NAME{"name"}; + static const inline std::string YAML_TAG_CATEGORY_IP_ADDRS{"ip_addrs"}; + static const inline std::string YAML_TAG_IP_ADDRS{"ip_addrs"}; + static const inline std::string YAML_TAG_IP_CATEGORIES{"ip_categories"}; static const inline std::string YAML_TAG_APPLY{"apply"}; static const inline std::string YAML_VALUE_APPLY_IN{"in"}; static const inline std::string YAML_VALUE_APPLY_OUT{"out"}; @@ -163,7 +193,7 @@ public: IpAllow *_config{nullptr}; ///< The backing configuration. }; - explicit IpAllow(const char *config_var); + explicit IpAllow(const char *ip_allow_config_var, const char *categories_config_var); void Print() const; @@ -201,6 +231,25 @@ public: const swoc::file::path &get_config_file() const; + /** + * Check if an IP category contains a specific IP address. + * + * @param category The IP category to check. + * @param addr The IP address to check against the category. + * @return True if the category contains the address, false otherwise. + */ + static bool ip_category_contains_addr(std::string const &category, swoc::IPAddr const &addr); + + /** Indicate whether ip_allow.yaml has no rules associated with it. + * + * If there are no rules, then all traffic will be blocked. This is used + * during ATS configuration to verify that the user has provided a usable + * ip_allow.yaml file. + * + * @return True if there are no rules in ip_allow.yaml, false otherwise. + */ + static bool has_no_rules(); + private: static size_t configid; ///< Configuration ID for update management. static const Record ALLOW_ALL_RECORD; ///< Static record that allows all access. @@ -209,17 +258,26 @@ private: void DebugMap(IpMap const &map) const; swoc::Errata BuildTable(); - swoc::Errata YAMLBuildTable(const std::string &); + swoc::Errata YAMLBuildTable(const std::string &content); swoc::Errata YAMLLoadEntry(const YAML::Node &); swoc::Errata YAMLLoadIPAddrRange(const YAML::Node &, IpMap *map, Record const *mark); + swoc::Errata YAMLLoadIPCategory(const YAML::Node &, IpMap *map, Record const *mark); swoc::Errata YAMLLoadMethod(const YAML::Node &node, Record &rec); - /// Copy @a src to the local arena and review a view of the copy. + swoc::Errata BuildCategories(); + swoc::Errata YAMLBuildCategories(const std::string &content); + swoc::Errata YAMLLoadCategoryRoot(const YAML::Node &); + swoc::Errata YAMLLoadCategoryDefinition(const YAML::Node &); + swoc::Errata YAMLLoadCategoryIpRange(const YAML::Node &, swoc::IPSpace<bool> &space); + + /// Copy @a src to the local arena and return a view of the copy. swoc::TextView localize(swoc::TextView src); - swoc::file::path config_file; ///< Path to configuration file. + swoc::file::path ip_allow_config_file; ///< Path to ip_allow configuration file. + swoc::file::path ip_categories_config_file; ///< Path to ip_allow configuration file. IpMap _src_map; IpMap _dst_map; + IpCategories ip_category_map; ///< Map of IP categories to IP spaces. /// Storage for records. swoc::MemArena _arena; @@ -364,5 +422,12 @@ IpAllow::makeAllowAllACL() -> ACL inline const swoc::file::path & IpAllow::get_config_file() const { - return config_file; + return ip_allow_config_file; +} + +inline bool +IpAllow::has_no_rules() +{ + auto const *self = IpAllow::acquire(); + return self->_src_map.count() == 0 && self->_dst_map.count() == 0; } diff --git a/include/proxy/http/remap/AclFiltering.h b/include/proxy/http/remap/AclFiltering.h index 53d2a91e63..a19287b107 100644 --- a/include/proxy/http/remap/AclFiltering.h +++ b/include/proxy/http/remap/AclFiltering.h @@ -25,8 +25,11 @@ #include "tscore/ink_inet.h" -#include <string> +#include "swoc/IPAddr.h" + #include <set> +#include <string> +#include <string_view> #include <vector> // =============================================================================== @@ -51,13 +54,35 @@ struct src_ip_info_t { /// @return @c true if @a ip is inside @a this range. bool - contains(IpEndpoint const &ip) + contains(IpEndpoint const &ip) const { IpAddr addr{ip}; return addr.cmp(start) >= 0 && addr.cmp(end) <= 0; } }; +struct src_ip_category_info_t { + std::string category; ///< The IP category for this remap rule. + bool invert = false; ///< Should we "invert" the meaning of these IP categories ("not in categories") + + void + reset() + { + category.clear(); + invert = false; + } + + /// @return @c true if @a ip is inside @a this categories. + bool + contains(IpEndpoint const &ip) const + { + return ask_ip_allow_about_category(category, swoc::IPAddr{ip}); + } + +private: + bool ask_ip_allow_about_category(std::string const &category, swoc::IPAddr const &addr) const; +}; + /** * **/ @@ -71,6 +96,7 @@ public: char *filter_name = nullptr; // optional filter name unsigned int allow_flag : 1, // action allow deny src_ip_valid : 1, // src_ip range valid + src_ip_category_valid : 1, // src_ip range valid in_ip_valid : 1, active_queue_flag : 1, // filter is in active state (used by .useflt directive) internal : 1; // filter internal HTTP requests @@ -90,6 +116,9 @@ public: int src_ip_cnt; // how many valid src_ip rules we have src_ip_info_t src_ip_array[ACL_FILTER_MAX_SRC_IP]; + int src_ip_category_cnt = 0; // how many valid src_ip rules we have + src_ip_category_info_t src_ip_category_array[ACL_FILTER_MAX_SRC_IP]; + // in_ip int in_ip_cnt; // how many valid dest_ip rules we have src_ip_info_t in_ip_array[ACL_FILTER_MAX_IN_IP]; diff --git a/include/proxy/http/remap/RemapConfig.h b/include/proxy/http/remap/RemapConfig.h index 2cca432f8a..4eac75d9de 100644 --- a/include/proxy/http/remap/RemapConfig.h +++ b/include/proxy/http/remap/RemapConfig.h @@ -35,13 +35,15 @@ class UrlRewrite; #define REMAP_OPTFLG_PPARAM 0x0004u /* "pparam=" option (per remap plugin option) */ #define REMAP_OPTFLG_METHOD 0x0008u /* "method=" option (used for ACL filtering) */ #define REMAP_OPTFLG_SRC_IP 0x0010u /* "src_ip=" option (used for ACL filtering) */ -#define REMAP_OPTFLG_ACTION 0x0020u /* "action=" option (used for ACL filtering) */ -#define REMAP_OPTFLG_INTERNAL 0x0040u /* only allow internal requests to hit this remap */ -#define REMAP_OPTFLG_IN_IP 0x0080u /* "in_ip=" option (used for ACL filtering)*/ -#define REMAP_OPTFLG_STRATEGY 0x0100u /* "strategy=" the name of the nexthop selection strategy */ +#define REMAP_OPTFLG_SRC_IP_CATEGORY 0x0020u /* "src_ip_category=" option (used for ACL filtering) */ +#define REMAP_OPTFLG_ACTION 0x0040u /* "action=" option (used for ACL filtering) */ +#define REMAP_OPTFLG_INTERNAL 0x0080u /* only allow internal requests to hit this remap */ +#define REMAP_OPTFLG_IN_IP 0x0100u /* "in_ip=" option (used for ACL filtering)*/ +#define REMAP_OPTFLG_STRATEGY 0x0200u /* "strategy=" the name of the nexthop selection strategy */ #define REMAP_OPTFLG_MAP_ID 0x0800u /* associate a map ID with this rule */ -#define REMAP_OPTFLG_INVERT 0x80000000u /* "invert" the rule (for src_ip at least) */ -#define REMAP_OPTFLG_ALL_FILTERS (REMAP_OPTFLG_METHOD | REMAP_OPTFLG_SRC_IP | REMAP_OPTFLG_ACTION | REMAP_OPTFLG_INTERNAL) +#define REMAP_OPTFLG_INVERT 0x80000000u /* "invert" the rule (for src_ip and src_ip_category at least) */ +#define REMAP_OPTFLG_ALL_FILTERS \ + (REMAP_OPTFLG_METHOD | REMAP_OPTFLG_SRC_IP | REMAP_OPTFLG_SRC_IP_CATEGORY | REMAP_OPTFLG_ACTION | REMAP_OPTFLG_INTERNAL) struct BUILD_TABLE_INFO { BUILD_TABLE_INFO(); diff --git a/include/tscore/Filenames.h b/include/tscore/Filenames.h index b6ff616cb8..90ae0485df 100644 --- a/include/tscore/Filenames.h +++ b/include/tscore/Filenames.h @@ -34,6 +34,7 @@ namespace filename constexpr const char *LOGGING = "logging.yaml"; constexpr const char *CACHE = "cache.config"; constexpr const char *IP_ALLOW = "ip_allow.yaml"; + constexpr const char *IP_CATEGORIES = "ip_categories.yaml"; constexpr const char *HOSTING = "hosting.config"; constexpr const char *SOCKS = "socks.config"; constexpr const char *PARENT = "parent.config"; diff --git a/src/mgmt/config/AddConfigFilesHere.cc b/src/mgmt/config/AddConfigFilesHere.cc index 9e9412d995..9e3fa3ef2d 100644 --- a/src/mgmt/config/AddConfigFilesHere.cc +++ b/src/mgmt/config/AddConfigFilesHere.cc @@ -70,6 +70,7 @@ initializeRegistry() registerFile(ts::filename::RECORDS, ts::filename::RECORDS, NOT_REQUIRED); registerFile("proxy.config.cache.control.filename", ts::filename::CACHE, NOT_REQUIRED); registerFile("proxy.config.cache.ip_allow.filename", ts::filename::IP_ALLOW, NOT_REQUIRED); + registerFile("proxy.config.cache.ip_categories.filename", ts::filename::IP_CATEGORIES, NOT_REQUIRED); registerFile("proxy.config.http.parent_proxy.file", ts::filename::PARENT, NOT_REQUIRED); registerFile("proxy.config.url_remap.filename", ts::filename::REMAP, NOT_REQUIRED); registerFile("", ts::filename::VOLUME, NOT_REQUIRED); diff --git a/src/proxy/IPAllow.cc b/src/proxy/IPAllow.cc index 1e134d897b..7c8c6141e8 100644 --- a/src/proxy/IPAllow.cc +++ b/src/proxy/IPAllow.cc @@ -24,15 +24,16 @@ limitations under the License. */ -#include <sstream> - #include "proxy/IPAllow.h" +#include "records/RecCore.h" +#include "swoc/Errata.h" +#include "swoc/TextView.h" #include "tscore/Filenames.h" +#include "tscore/ink_memory.h" #include "tsutil/ts_errata.h" #include "swoc/Vectray.h" #include "swoc/BufferWriter.h" -#include "swoc/bwf_std.h" #include "swoc/bwf_ex.h" #include "swoc/bwf_ip.h" @@ -91,12 +92,14 @@ IpAllow::startup() ipAllowUpdate = new ConfigUpdateHandler<IpAllow>(); ipAllowUpdate->attach("proxy.config.cache.ip_allow.filename"); + ipAllowUpdate->attach("proxy.config.cache.ip_categories.filename"); reconfigure(); ConfigInfo *config = configProcessor.get(configid); if (config == nullptr) { - configid = configProcessor.set(configid, new self_type("proxy.config.cache.ip_allow.filename")); + configid = configProcessor.set( + configid, new self_type("proxy.config.cache.ip_allow.filename", "proxy.config.cache.ip_categories.filename")); Warning("%s not loaded; All IP Addresses will be blocked.", ts::filename::IP_ALLOW); } } @@ -108,17 +111,24 @@ IpAllow::reconfigure() Note("%s loading ...", ts::filename::IP_ALLOW); - new_table = new self_type("proxy.config.cache.ip_allow.filename"); - auto errata = new_table->BuildTable(); - if (!errata.is_ok()) { + new_table = new self_type("proxy.config.cache.ip_allow.filename", "proxy.config.cache.ip_categories.filename"); + // IP rules need categories, so load them first (if they exist). + if (auto errata = new_table->BuildCategories(); !errata.is_ok()) { + std::string text; + swoc::bwprint(text, "{} failed to load\n{}", new_table->ip_categories_config_file, errata); + Error("%s", text.c_str()); + delete new_table; + return; + } + if (auto errata = new_table->BuildTable(); !errata.is_ok()) { std::string text; swoc::bwprint(text, "{} failed to load\n{}", ts::filename::IP_ALLOW, errata); Error("%s", text.c_str()); delete new_table; - } else { - configid = configProcessor.set(configid, new_table); - Note("%s finished loading", ts::filename::IP_ALLOW); + return; } + configid = configProcessor.set(configid, new_table); + Note("%s finished loading", ts::filename::IP_ALLOW); } IpAllow * @@ -139,6 +149,20 @@ IpAllow::release() configProcessor.release(configid, this); } +bool +IpAllow::ip_category_contains_addr(std::string const &category, swoc::IPAddr const &addr) +{ + self_type *self = acquire(); + auto const spot = self->ip_category_map.find(category); + if (spot == self->ip_category_map.end()) { + return false; + } + auto const &space = spot->second; + bool const found = space.find(addr) != space.end(); + self->release(); + return found; +} + IpAllow::ACL IpAllow::match(swoc::IPAddr const &addr, match_key_t key) { @@ -170,7 +194,14 @@ IpAllow::match(swoc::IPAddr const &addr, match_key_t key) // End API functions // -IpAllow::IpAllow(const char *config_var) : config_file(ats_scoped_str(RecConfigReadConfigPath(config_var)).get()) {} +IpAllow::IpAllow(const char *ip_allow_config_var, const char *ip_categories_config_var) + : ip_allow_config_file(ats_scoped_str(RecConfigReadConfigPath(ip_allow_config_var)).get()) +{ + std::string const path = RecConfigReadConfigPath(ip_categories_config_var); + if (!path.empty()) { + ip_categories_config_file = ats_scoped_str(path).get(); + } +} BufferWriter & bwformat(BufferWriter &w, Spec const &spec, IpAllow::IpMap const &map) @@ -232,7 +263,7 @@ IpAllow::BuildTable() ink_assert(_src_map.count() == 0 && _dst_map.count() == 0); std::error_code ec; - std::string content{swoc::file::load(config_file, ec)}; + std::string content{swoc::file::load(ip_allow_config_file, ec)}; swoc::Errata errata; if (ec.value() == 0) { try { @@ -314,9 +345,8 @@ IpAllow::YAMLLoadIPAddrRange(const YAML::Node &node, IpMap *map, IpAllow::Record return swoc::Errata(ERRATA_ERROR, "{} Expected IP address range at {}, found non-literal.", this, node.Mark()); } - swoc::TextView debug(node.Scalar()); - (void)debug; - if (swoc::IPRange r; r.load(node.Scalar())) { + swoc::TextView ip_range(node.Scalar()); + if (swoc::IPRange r; r.load(ip_range)) { map->fill(r, record); } else { return swoc::Errata(ERRATA_ERROR, "{} {} - '{}' is not a valid range.", this, node.Mark(), node.Scalar()); @@ -324,6 +354,23 @@ IpAllow::YAMLLoadIPAddrRange(const YAML::Node &node, IpMap *map, IpAllow::Record return {}; } +swoc::Errata +IpAllow::YAMLLoadIPCategory(const YAML::Node &node, IpMap *map, IpAllow::Record const *record) +{ + if (!node.IsScalar()) { + return swoc::Errata(ERRATA_ERROR, "{} Expected IP address category at {}, found non-literal.", this, node.Mark()); + } + std::string const &category(node.Scalar()); + if (auto spot = ip_category_map.find(category); spot != ip_category_map.end()) { + for (auto const &range : spot->second) { + map->fill(range.range_view(), record); + } + } else { + return swoc::Errata(ERRATA_ERROR, "{} {} - '{}' is not category with a defined range.", this, node.Mark(), category); + } + return {}; +} + swoc::Errata IpAllow::YAMLLoadEntry(const YAML::Node &entry) { @@ -374,6 +421,10 @@ IpAllow::YAMLLoadEntry(const YAML::Node &entry) return swoc::Errata(ERRATA_ERROR, "{} {} - item ignored, required '{}' key not found.", this, entry.Mark(), YAML_TAG_ACTION); } + if (entry[YAML_TAG_IP_ADDRS] && entry[YAML_TAG_IP_CATEGORIES]) { + return swoc::Errata(ERRATA_ERROR, "{} {} - '{}' and '{}' cannot both be used in the same rule.", this, entry.Mark(), + YAML_TAG_IP_ADDRS, YAML_TAG_IP_CATEGORIES); + } if (YAML::Node addr_node = entry[YAML_TAG_IP_ADDRS]; addr_node) { bool marked_p = false; if (addr_node.IsSequence()) { @@ -396,8 +447,31 @@ IpAllow::YAMLLoadEntry(const YAML::Node &entry) if (!marked_p) { return swoc::Errata(ERRATA_ERROR, "No valid addresses for rule at {}", node.Mark()); } + } else if (YAML::Node category_node = entry[YAML_TAG_IP_CATEGORIES]; category_node) { + bool marked_p = false; + if (category_node.IsSequence()) { + for (auto const &n : category_node) { + if (auto errata = this->YAMLLoadIPCategory(n, map, record); errata.is_ok()) { + marked_p = true; + } else { + errata.note(R"(In record at {})", entry.Mark()); + return errata; + } + } + } else { + if (auto errata = this->YAMLLoadIPCategory(category_node, map, record); errata.is_ok()) { + marked_p = true; + } else { + errata.note(R"(In record at {})", entry.Mark()); + return errata; + } + } + if (!marked_p) { + return swoc::Errata(ERRATA_ERROR, "No valid IP category for rule at {}", node.Mark()); + } } else { - return swoc::Errata(ERRATA_ERROR, "{} {} - item ignored, required '{}' key not found.", this, entry.Mark(), YAML_TAG_IP_ADDRS); + return swoc::Errata(ERRATA_ERROR, "{} {} - item ignored, required '{}' or '{}' key not found.", this, entry.Mark(), + YAML_TAG_IP_ADDRS, YAML_TAG_IP_CATEGORIES); } if (auto methodNode = entry[YAML_TAG_METHODS]) { @@ -426,19 +500,149 @@ IpAllow::YAMLBuildTable(std::string const &content) return swoc::Errata("{} - top level object was not a map. All IP Addresses will be blocked", this); } - YAML::Node data{root[YAML_TAG_ROOT.data()]}; - if (!data) { + // IP categories are optional. Load them if specified. Note that the rules, + // if they use categories, depend upon the categories being defined. So the + // categories have to be processed first before the rules are. + YAML::Node categories{root[YAML_TAG_CATEGORY_ROOT.data()]}; + if (auto errata = this->YAMLLoadCategoryRoot(categories); !errata.is_ok()) { + return errata; + } + + // Now load the IPAllow rules. + YAML::Node rules{root[YAML_TAG_ROOT.data()]}; + if (!rules) { return swoc::Errata("{} - root tag '{}' not found. All IP Addresses will be blocked", this, YAML_TAG_ROOT); - } else if (data.IsSequence()) { - for (auto const &entry : data) { + } else if (rules.IsSequence()) { + for (auto const &entry : rules) { if (auto errata = this->YAMLLoadEntry(entry); !errata.is_ok()) { return errata; } } - } else if (data.IsMap()) { - return this->YAMLLoadEntry(data); // singleton, just load it. + } else if (rules.IsMap()) { + return this->YAMLLoadEntry(rules); // singleton, just load it. } else { return swoc::Errata("{} - root tag '{}' is not an map or sequence. All IP Addresses will be blocked", this, YAML_TAG_ROOT); } return {}; } + +swoc::Errata +IpAllow::BuildCategories() +{ + std::error_code ec; + if (ip_categories_config_file.empty()) { + return {}; + } + + Note("%s loading categores file %s ...", ts::filename::IP_ALLOW, ip_categories_config_file.c_str()); + std::string content{swoc::file::load(ip_categories_config_file, ec)}; + swoc::Errata errata; + if (ec.value() == 0) { + try { + errata = this->YAMLBuildCategories(content); + } catch (std::exception &ex) { + return swoc::Errata(ec, ERRATA_ERROR, "{} - Invalid IP Categories {} content: {}", this, ip_categories_config_file, + ex.what()); + } + if (!errata.is_ok()) { + errata.note("While parsing ip categories file: {}", ip_categories_config_file); + return errata; + } + } else { + return swoc::Errata(ERRATA_ERROR, "{} Failed to load {}", this, ec); + } + Note("%s done loading categores file %s ...", ts::filename::IP_ALLOW, ip_categories_config_file.c_str()); + return {}; +} + +swoc::Errata +IpAllow::YAMLBuildCategories(std::string const &content) +{ + YAML::Node root{YAML::Load(content)}; + YAML::Node categories{root[YAML_TAG_CATEGORY_ROOT.data()]}; + if (auto errata = this->YAMLLoadCategoryRoot(categories); !errata.is_ok()) { + return errata; + } + return {}; +} + +swoc::Errata +IpAllow::YAMLLoadCategoryRoot(const YAML::Node &categories) +{ + if (categories) { + if (!categories.IsSequence()) { + return swoc::Errata("{} - '{}' tag must be a sequence of maps. All IP Addresses will be blocked", this, + YAML_TAG_CATEGORY_ROOT); + } + for (auto const &category : categories) { + if (!category.IsMap()) { + return swoc::Errata("{} - '{}' tag must be a sequence of maps. All IP Addresses will be blocked", this, + YAML_TAG_CATEGORY_ROOT); + } + if (auto errata = this->YAMLLoadCategoryDefinition(category); !errata.is_ok()) { + return errata; + } + } + } + return {}; +} + +swoc::Errata +IpAllow::YAMLLoadCategoryDefinition(const YAML::Node &entry) +{ + /* Parse this into ip_category_map: + * + * - name: <category name> + * ip_addrs: + * - <ip range> + * - <ip range> + * - <ip range> + */ + if (!entry.IsMap()) { + return swoc::Errata(ERRATA_ERROR, "{} {} - Category definition must be a map.", this, entry.Mark()); + } + + if (auto name_node = entry[YAML_TAG_CATEGORY_NAME]; name_node) { + if (!name_node.IsScalar()) { + return swoc::Errata(ERRATA_ERROR, "{} {} - Category name must be a string.", this, name_node.Mark()); + } + std::string const &name(name_node.Scalar()); + if (auto ip_addrs_node = entry[YAML_TAG_CATEGORY_IP_ADDRS]; ip_addrs_node) { + if (ip_addrs_node.IsSequence()) { + for (auto const &ip_addr_node : ip_addrs_node) { + if (auto errata = this->YAMLLoadCategoryIpRange(ip_addr_node, ip_category_map[name]); !errata.is_ok()) { + errata.note(R"(In category definition at {})", entry.Mark()); + return errata; + } + } + } else { + if (auto errata = this->YAMLLoadCategoryIpRange(ip_addrs_node, ip_category_map[name]); !errata.is_ok()) { + errata.note(R"(In category definition at {})", entry.Mark()); + return errata; + } + } + } else { // No ip_addrs. + return swoc::Errata(ERRATA_ERROR, "{} {} - IP Addresses must be specified.", this, entry.Mark()); + } + } else { // No name + return swoc::Errata(ERRATA_ERROR, "{} {} - Category name must be specified.", this, entry.Mark()); + } + return {}; +} + +swoc::Errata +IpAllow::YAMLLoadCategoryIpRange(const YAML::Node &node, swoc::IPSpace<bool> &space) +{ + if (!node.IsScalar()) { + return swoc::Errata(ERRATA_ERROR, "{} Expected IP address range for category at {}, found non-literal.", this, node.Mark()); + } + + swoc::TextView ip_range(node.Scalar()); + if (swoc::IPRange r; r.load(ip_range)) { + space.fill(r, true); + } else { + return swoc::Errata(ERRATA_ERROR, "{} {} - '{}' is not a valid range.", this, node.Mark(), node.Scalar()); + } + + return {}; +} diff --git a/src/proxy/http/remap/AclFiltering.cc b/src/proxy/http/remap/AclFiltering.cc index 739d11612f..259972768c 100644 --- a/src/proxy/http/remap/AclFiltering.cc +++ b/src/proxy/http/remap/AclFiltering.cc @@ -22,7 +22,19 @@ */ #include "proxy/http/remap/AclFiltering.h" + #include "proxy/hdrs/HTTP.h" +#include "proxy/IPAllow.h" + +// =============================================================================== +// src_ip_category_info_t +// =============================================================================== + +bool +src_ip_category_info_t::ask_ip_allow_about_category(std::string const &category, swoc::IPAddr const &addr) const +{ + return IpAllow::ip_category_contains_addr(category, addr); +} // =============================================================================== // acl_filter_rule @@ -43,6 +55,9 @@ acl_filter_rule::reset() for (i = (src_ip_cnt = 0); i < ACL_FILTER_MAX_SRC_IP; i++) { src_ip_array[i].reset(); } + for (i = (src_ip_category_cnt = 0); i < ACL_FILTER_MAX_SRC_IP; i++) { + src_ip_category_array[i].reset(); + } src_ip_valid = 0; for (i = (in_ip_cnt = 0); i < ACL_FILTER_MAX_IN_IP; i++) { in_ip_array[i].reset(); @@ -114,6 +129,11 @@ acl_filter_rule::print() printf("%s - %s, ", src_ip_array[i].start.toString(b1, sizeof(b1)), src_ip_array[i].end.toString(b2, sizeof(b2))); } printf("\n"); + printf("src_ip_category_cnt=%d\n", src_ip_category_cnt); + for (i = 0; i < src_ip_category_cnt; i++) { + printf("%s, ", src_ip_category_array[i].category.c_str()); + } + printf("\n"); printf("in_ip_cnt=%d\n", in_ip_cnt); for (i = 0; i < in_ip_cnt; i++) { ip_text_buffer b1, b2; diff --git a/src/proxy/http/remap/RemapConfig.cc b/src/proxy/http/remap/RemapConfig.cc index e76dd7dca6..2736802a14 100644 --- a/src/proxy/http/remap/RemapConfig.cc +++ b/src/proxy/http/remap/RemapConfig.cc @@ -423,7 +423,6 @@ const char * remap_validate_filter_args(acl_filter_rule **rule_pp, const char **argv, int argc, char *errStrBuf, size_t errStrBufSize) { acl_filter_rule *rule; - src_ip_info_t *ipi; int i, j; bool new_rule_flg = false; @@ -505,7 +504,7 @@ remap_validate_filter_args(acl_filter_rule **rule_pp, const char **argv, int arg } return (const char *)errStrBuf; } - ipi = &rule->src_ip_array[rule->src_ip_cnt]; + src_ip_info_t *ipi = &rule->src_ip_array[rule->src_ip_cnt]; if (ul & REMAP_OPTFLG_INVERT) { ipi->invert = true; } @@ -532,6 +531,34 @@ remap_validate_filter_args(acl_filter_rule **rule_pp, const char **argv, int arg } } + if (ul & REMAP_OPTFLG_SRC_IP_CATEGORY) { /* "src_ip_category=" option */ + if (rule->src_ip_category_cnt >= ACL_FILTER_MAX_SRC_IP) { + Debug("url_rewrite", "[validate_filter_args] Too many \"src_ip_category=\" filters"); + snprintf(errStrBuf, errStrBufSize, "Defined more than %d \"src_ip_category=\" filters!", ACL_FILTER_MAX_SRC_IP); + errStrBuf[errStrBufSize - 1] = 0; + if (new_rule_flg) { + delete rule; + *rule_pp = nullptr; + } + return (const char *)errStrBuf; + } + src_ip_category_info_t *ipi = &rule->src_ip_category_array[rule->src_ip_category_cnt]; + if (ul & REMAP_OPTFLG_INVERT) { + ipi->invert = true; + } + for (j = 0; j < rule->src_ip_category_cnt; j++) { + if (rule->src_ip_category_array[j].category == ipi->category) { + ipi->reset(); + ipi = nullptr; + break; /* we have the same src_ip_category in the list */ + } + } + if (ipi) { + rule->src_ip_category_cnt++; + rule->src_ip_category_valid = 1; + } + } + if (ul & REMAP_OPTFLG_IN_IP) { /* "dest_ip=" option */ if (rule->in_ip_cnt >= ACL_FILTER_MAX_IN_IP) { Debug("url_rewrite", "[validate_filter_args] Too many \"in_ip=\" filters"); @@ -543,7 +570,7 @@ remap_validate_filter_args(acl_filter_rule **rule_pp, const char **argv, int arg } return (const char *)errStrBuf; } - ipi = &rule->in_ip_array[rule->in_ip_cnt]; + src_ip_info_t *ipi = &rule->in_ip_array[rule->in_ip_cnt]; if (ul & REMAP_OPTFLG_INVERT) { ipi->invert = true; } @@ -648,6 +675,14 @@ remap_check_option(const char **argv, int argc, unsigned long findmode, int *_re *argptr = &argv[i][8]; } ret_flags |= (REMAP_OPTFLG_SRC_IP | REMAP_OPTFLG_INVERT); + } else if (!strncasecmp(argv[i], "src_ip_category=~", 8)) { + if ((findmode & REMAP_OPTFLG_SRC_IP_CATEGORY) != 0) { + idx = i; + } + if (argptr) { + *argptr = &argv[i][17]; + } + ret_flags |= (REMAP_OPTFLG_SRC_IP_CATEGORY | REMAP_OPTFLG_INVERT); } else if (!strncasecmp(argv[i], "src_ip=", 7)) { if ((findmode & REMAP_OPTFLG_SRC_IP) != 0) { idx = i; diff --git a/src/proxy/http/remap/UrlRewrite.cc b/src/proxy/http/remap/UrlRewrite.cc index 79645ae19c..925b1b0e21 100644 --- a/src/proxy/http/remap/UrlRewrite.cc +++ b/src/proxy/http/remap/UrlRewrite.cc @@ -444,8 +444,25 @@ UrlRewrite::PerformACLFiltering(HttpTransact::State *s, url_mapping *map) } } + if (match && rp->src_ip_category_valid) { + Debug("url_rewrite", "match was true and we have specified an src_ip_category field"); + match = false; + for (int j = 0; j < rp->src_ip_category_cnt && !match; j++) { + bool in_category = rp->src_ip_category_array[j].contains(s->client_info.src_addr); + if (rp->src_ip_category_array[j].invert) { + if (!in_category) { + match = true; + } + } else { + if (in_category) { + match = true; + } + } + } + } + if (match && rp->in_ip_valid) { - Debug("url_rewrite", "match was true and we have specified a in_ip field"); + Debug("url_rewrite", "match was true and we have specified an in_ip field"); match = false; for (int j = 0; j < rp->in_ip_cnt && !match; j++) { IpEndpoint incoming_addr; diff --git a/src/records/RecordsConfig.cc b/src/records/RecordsConfig.cc index ece1c6656c..e8a7abc8ff 100644 --- a/src/records/RecordsConfig.cc +++ b/src/records/RecordsConfig.cc @@ -795,6 +795,8 @@ static const RecordElement RecordsConfig[] = , {RECT_CONFIG, "proxy.config.cache.ip_allow.filename", RECD_STRING, ts::filename::IP_ALLOW, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , + {RECT_CONFIG, "proxy.config.cache.ip_categories.filename", RECD_STRING, "", RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} + , {RECT_CONFIG, "proxy.config.cache.hosting_filename", RECD_STRING, ts::filename::HOSTING, RECU_DYNAMIC, RR_NULL, RECC_NULL, nullptr, RECA_NULL} , {RECT_CONFIG, "proxy.config.cache.volume_filename", RECD_STRING, ts::filename::VOLUME, RECU_RESTART_TS, RR_NULL, RECC_NULL, nullptr, RECA_NULL} diff --git a/src/traffic_server/traffic_server.cc b/src/traffic_server/traffic_server.cc index f91b6a13e9..83614b1c47 100644 --- a/src/traffic_server/traffic_server.cc +++ b/src/traffic_server/traffic_server.cc @@ -2188,6 +2188,10 @@ main(int /* argc ATS_UNUSED */, const char **argv) pluginInitCheck.notify_one(); } + if (IpAllow::has_no_rules()) { + Error("No ip_allow.yaml entries found. All requests will be denied!"); + } + SSLConfigParams::init_ssl_ctx_cb = init_ssl_ctx_callback; SSLConfigParams::load_ssl_file_cb = load_ssl_file_callback; sslNetProcessor.start(-1, stacksize); diff --git a/tests/gold_tests/ip_allow/ip_category.test.py b/tests/gold_tests/ip_allow/ip_category.test.py new file mode 100644 index 0000000000..e5044ecc62 --- /dev/null +++ b/tests/gold_tests/ip_allow/ip_category.test.py @@ -0,0 +1,326 @@ +''' +Verify IP allow ip_category behavior. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import re + +Test.Summary = ''' +Verify IP allow ip_category behavior. +''' + + +class CategoryFile: + """Encapsulate the various ip_category.yaml contents.""" + + contents: list['CategoryFile'] = [] + parent_directory: str = Test.RunDirectory + _index: int = 0 + + def __init__(self, content: str): + """Initialize the object. + + :param content: The content of the ip_category.yaml file. + """ + self._content = content + self._index = len(CategoryFile.contents) + self._filename = os.path.join(CategoryFile.parent_directory, f'categories{self._index}.yaml') + CategoryFile.contents.append(self) + + def _write(self): + with open(self._filename, 'w') as f: + f.write(self._content) + + def get_path(self): + return self._filename + + @classmethod + def write_all(cls): + for content in cls.contents: + content._write() + + +localhost_is_internal_and_external = CategoryFile( + ''' +ip_categories: + - name: ACME_INTERNAL + ip_addrs: 127.0.0.1 + - name: ACME_EXTERNAL + ip_addrs: 127.0.0.1 + - name: ACME_ALL + ip_addrs: 127.0.0.1 + - name: ALL + ip_addrs: 127.0.0.1 +''') + +localhost_is_external = CategoryFile( + ''' +ip_categories: + - name: ACME_INTERNAL + ip_addrs: 1.2.3.4 + - name: ACME_REMAP_EXTERNAL + ip_addrs: 127.0.0.1 + - name: ACME_EXTERNAL + ip_addrs: 127.0.0.1 + - name: ACME_ALL + ip_addrs: + - 1.2.3.4 + - 127.0.0.1 + - name: ALL + ip_addrs: + - 1.2.3.4 + - 127.0.0.1 +''') + +localhost_is_neither = CategoryFile( + ''' +ip_categories: + - name: ACME_INTERNAL + ip_addrs: 1.2.3.4 + - name: ACME_EXTERNAL + ip_addrs: 1.2.3.4 + - name: ACME_ALL + ip_addrs: 1.2.3.4 + - name: ALL + ip_addrs: + - 1.2.3.4 + - 127.0.0.1 +''') + +# Keep this below the above content instantiations. +CategoryFile.write_all() + + +class Test_ip_category: + """Configure a test to verify ip_category behavior.""" + + _client_counter: int = 0 + _ts_is_started: bool = False + _reload_server_is_started: bool = False + _server_is_started: bool = False + _reload_counter: int = 0 + + _ts: 'TestProcess' = None + _server: 'TestProcess' = None + _reload_server: 'TestProcess' = None + + _categories_filename: str = f'{Test.RunDirectory}/categories.yaml' + _category_files_are_written: bool = False + + _server_replay = 'replays/https_categories_server.replay.yaml' + + def __init__( + self, name: str, replay_file: str, ip_allow_config: str, ip_category_config: 'CategoryFile', acl_configuration: str, + expected_responses: list[int]): + """Initialize the test. + + :param name: The name of the test. + :param replay_file: The replay file to be used. + :param ip_allow_config: The ip_allow configuration to be used. + :param ip_category_config: The ip_category.yaml configuration to be used. + :param acl_configuration: The ACL configuration to be used. + :param expect_responses: The in-order expected responses from the proxy. + """ + self._replay_file = replay_file + self._ip_allow_config = ip_allow_config + self._acl_configuration = acl_configuration + self._expected_responses = expected_responses + + self._update_categories_file(ip_category_config) + self._update_remap_with_acl() + + self._configure_server() + self._configure_traffic_server() + + tr = Test.AddTestRun(name) + self._configure_client(tr) + + def _update_remap_with_acl(self) -> None: + """Update the remap.config file with the ACL configuration.""" + if Test_ip_category._ts: + if self._acl_configuration: + tr = Test.AddTestRun(f"remap.config file update with acl: {self._acl_configuration}") + p = tr.Processes.Default + destination = os.path.join(Test_ip_category._ts.Variables.CONFIGDIR, 'remap.config') + common = f'map / http://127.0.0.1:{Test_ip_category._server.Variables.http_port} ' + p.Command = f'echo {common} {self._acl_configuration} > {destination}; cat {destination}' + p.ReturnCode = 0 + else: + tr = Test.AddTestRun(f"remap.config file update with no acl") + p = tr.Processes.Default + destination = os.path.join(Test_ip_category._ts.Variables.CONFIGDIR, 'remap.config') + common = f'map / http://127.0.0.1:{Test_ip_category._server.Variables.http_port} ' + p.Command = f'echo {common} > {destination}; cat {destination}' + p.ReturnCode = 0 + + def _update_categories_file(self, category_content: 'CategoryFile') -> None: + """Update the categories file. + + :param category_content: The content of the categories file. + """ + tr = Test.AddTestRun(f"Categories file update: {category_content.get_path()}") + p = tr.Processes.Default + destination = Test_ip_category._categories_filename + p.Command = f'cp {category_content.get_path()} {destination}; cat {destination}; ls -ltr {destination}' + p.ReturnCode = 0 + + def _configure_server(self) -> None: + """Configure the server.""" + if Test_ip_category._server: + # All test runs share a single server instance. + return + server = Test.MakeVerifierServerProcess(f"server", self._server_replay) + Test_ip_category._server = server + + def _configure_traffic_server(self) -> None: + """Configure Traffic Server.""" + + if Test_ip_category._ts: + # All test runs share a single Traffic Server instance. + + # Reload the ip_allow.yaml file. + ts = Test_ip_category._ts + tr = Test.AddTestRun(f"Reload the configuration file.") + Test_ip_category._reload_counter += 1 + p = tr.Processes.Process(f"reload-{Test_ip_category._reload_counter}") + # The sleep is added to give time for the reload to happen. + p.Command = 'traffic_ctl config reload; sleep 30' + p.Env = ts.Env + # Killing the sleep can result in a -2 return code. + p.ReturnCode = Any(0, -2) + p.Ready = When.FileContains( + ts.Disk.diags_log.Name, "ip_allow.yaml finished loading", 1 + Test_ip_category._reload_counter) + p.Timeout = 20 + tr.StillRunningAfter = ts + tr.Processes.Default.StartBefore(p) + tr.Processes.Default.Command = 'echo "waiting upon traffic server to reload"' + tr.TimeOut = 20 + + return + ts = Test.MakeATSProcess("ts", enable_cache=False, enable_tls=True) + Test_ip_category._ts = ts + + ts.addDefaultSSLFiles() + ts.Disk.ssl_multicert_config.AddLine('dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key') + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'http|ip_allow', + 'proxy.config.cache.ip_categories.filename': Test_ip_category._categories_filename, + 'proxy.config.http.push_method_enabled': 1, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.quic.no_activity_timeout_in': 0, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + 'proxy.config.ssl.client.verify.server.policy': 'PERMISSIVE', + 'proxy.config.http.connect_ports': Test_ip_category._server.Variables.http_port, + }) + + ts.Disk.remap_config.AddLine( + f'map / http://127.0.0.1:{Test_ip_category._server.Variables.http_port} {self._acl_configuration}') + ts.Disk.ip_allow_yaml.AddLines(self._ip_allow_config.split("\n")) + + def _configure_client(self, tr: 'TestRun') -> None: + """Run the test. + + :param tr: The TestRun object to associate the client process with. + """ + + if not Test_ip_category._server_is_started: + tr.Processes.Default.StartBefore(Test_ip_category._server) + Test_ip_category._server_is_started = True + if not Test_ip_category._ts_is_started: + tr.Processes.Default.StartBefore(Test_ip_category._ts) + Test_ip_category._ts_is_started = True + + p = tr.AddVerifierClientProcess( + f'client-{Test_ip_category._client_counter}', self._replay_file, https_ports=[Test_ip_category._ts.Variables.ssl_port]) + Test_ip_category._client_counter += 1 + + if self._expected_responses: + codes = [str(code) for code in self._expected_responses] + p.Streams.stdout += Testers.ContainsExpression( + '.*'.join(codes), "Verifying the expected order of responses", reflags=re.DOTALL | re.MULTILINE) + else: + # If there are no expected responses, expect the Warning about the rejected ip. + self._ts.Disk.diags_log.Content += Testers.ContainsExpression( + "client '127.0.0.1' prohibited by ip-allow policy", "Verify the client rejection warning message.") + + # Also, the client will complain about the broken connections. + p.ReturnCode = 1 + + +IP_ALLOW_CONTENT = f''' +ip_allow: + - apply: in + ip_categories: ACME_INTERNAL + action: allow + methods: + - GET + - HEAD + - POST + - PUSH + - apply: in + ip_categories: ACME_EXTERNAL + action: allow + methods: + - GET + - HEAD + - apply: in + ip_categories: ACME_ALL + action: allow + methods: + - HEAD + - apply: in + ip_categories: ALL + action: deny +''' + +test_ip_allow_optional_methods = Test_ip_category( + "IP Category: INTERNAL", + replay_file='replays/https_categories_internal.replay.yaml', + ip_allow_config=IP_ALLOW_CONTENT, + ip_category_config=localhost_is_internal_and_external, + acl_configuration='', + expected_responses=[200, 200, 400, 403]) + +test_ip_allow_optional_methods = Test_ip_category( + "IP Category: EXTERNAL", + replay_file='replays/https_categories_external.replay.yaml', + ip_allow_config=IP_ALLOW_CONTENT, + ip_category_config=localhost_is_external, + acl_configuration='', + expected_responses=[200, 403, 403]) + +# Because all requests are outright rejected for 127.0.0.1, ATS will +# reject all incoming transactions and not even give a 403 response. +test_ip_allow_optional_methods = Test_ip_category( + "IP Category: ALL", + replay_file='replays/https_categories_all.replay.yaml', + ip_allow_config=IP_ALLOW_CONTENT, + ip_category_config=localhost_is_neither, + acl_configuration='', + expected_responses=None) + +# Deny GET as well via remap.config ACL. +test_ip_allow_optional_methods = Test_ip_category( + "IP Category: INTERNAL", + replay_file='replays/https_categories_external_remap.replay.yaml', + ip_allow_config=IP_ALLOW_CONTENT, + ip_category_config=localhost_is_external, + acl_configuration='@action=deny @src_ip_category=ACME_REMAP_EXTERNAL @method=GET', + expected_responses=[403, 403, 403]) diff --git a/tests/gold_tests/ip_allow/replays/https_categories_all.replay.yaml b/tests/gold_tests/ip_allow/replays/https_categories_all.replay.yaml new file mode 100644 index 0000000000..9c41fae78c --- /dev/null +++ b/tests/gold_tests/ip_allow/replays/https_categories_all.replay.yaml @@ -0,0 +1,94 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The replay file executes various HTTP requests to verify the ip_allow policy +# applies by default to all methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: tls + sni: test_sni + transactions: + + # GET rejected + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS confirmed that the PUSH was successful, which it does + # with a 201 response. + proxy-response: + status: 403 diff --git a/tests/gold_tests/ip_allow/replays/https_categories_external.replay.yaml b/tests/gold_tests/ip_allow/replays/https_categories_external.replay.yaml new file mode 100644 index 0000000000..54b67f5b97 --- /dev/null +++ b/tests/gold_tests/ip_allow/replays/https_categories_external.replay.yaml @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The replay file executes various HTTP requests to verify the ip_allow policy +# applies by default to all methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: tls + sni: test_sni + transactions: + + # GET allowed + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 diff --git a/tests/gold_tests/ip_allow/replays/https_categories_external_remap.replay.yaml b/tests/gold_tests/ip_allow/replays/https_categories_external_remap.replay.yaml new file mode 100644 index 0000000000..16ca64ddf7 --- /dev/null +++ b/tests/gold_tests/ip_allow/replays/https_categories_external_remap.replay.yaml @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The replay file executes various HTTP requests to verify the ip_allow policy +# applies by default to all methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: tls + sni: test_sni + transactions: + + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + # Even GET is now blocked via remap ACL. + proxy-response: + status: 403 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS rejected the PUSH. + proxy-response: + status: 403 diff --git a/tests/gold_tests/ip_allow/replays/https_categories_internal.replay.yaml b/tests/gold_tests/ip_allow/replays/https_categories_internal.replay.yaml new file mode 100644 index 0000000000..c8178be60e --- /dev/null +++ b/tests/gold_tests/ip_allow/replays/https_categories_internal.replay.yaml @@ -0,0 +1,106 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Assume everything is allowed by default. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: tls + sni: test_sni + transactions: + + # GET allowed + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + <<: *standard_response + + proxy-response: + status: 200 + + # POST allowed + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + <<: *standard_response + + proxy-response: + status: 200 + + # PUSH allowed + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Cacching is off, but a 400 still indicates that ATS processed it. + proxy-response: + status: 400 + + # DELETE is not allowed even for internal. + - client-request: + method: "DELETE" + version: "1.1" + url: /test/ip_allow/test_delete + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, delete ] + - [ X-Request, delete ] + + <<: *standard_response + + proxy-response: + status: 403 diff --git a/tests/gold_tests/ip_allow/replays/https_categories_server.replay.yaml b/tests/gold_tests/ip_allow/replays/https_categories_server.replay.yaml new file mode 100644 index 0000000000..9c41fae78c --- /dev/null +++ b/tests/gold_tests/ip_allow/replays/https_categories_server.replay.yaml @@ -0,0 +1,94 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The replay file executes various HTTP requests to verify the ip_allow policy +# applies by default to all methods. + +meta: + version: "1.0" + + blocks: + - standard_response: &standard_response + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, 20 ] + +sessions: +- protocol: + - name: http + version: 1 + - name: tls + sni: test_sni + transactions: + + # GET rejected + - client-request: + method: "GET" + version: "1.1" + url: /test/ip_allow/test_get + headers: + fields: + - [ Content-Length, 0 ] + - [ uuid, get ] + - [ X-Request, get ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # POST rejected + - client-request: + method: "POST" + version: "1.1" + url: /test/ip_allow/test_post + headers: + fields: + - [Content-Length, 10] + - [ uuid, post ] + - [ X-Request, post ] + + # Shouldn't be used. + <<: *standard_response + + proxy-response: + status: 403 + + # PUSH rejected + - client-request: + method: "PUSH" + version: "1.1" + url: /test/ip_allow/test_push + headers: + fields: + - [ Host, example.com ] + - [ uuid, push ] + - [ X-Request, push ] + - [ Content-Length, 113 ] + content: + encoding: plain + data: "HTTP/1.1 200 OK\nServer: ATS/10.0.0\nAccept-Ranges: bytes\nContent-Length: 6\nCache-Control: public,max-age=2\n\nCACHED" + + <<: *standard_response + + # Verify that ATS confirmed that the PUSH was successful, which it does + # with a 201 response. + proxy-response: + status: 403