-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Susan!
Thank you for your feedback. Well, i think my conclusion is wrong about 500 concurrent connections against ATS. Because: httpress -n 100000000 -c 500 -t 4 'http://mytestsite.domain/' 4 threads x 500 concurrent connections against ATS does = 2000 concurrent connnects. Anyways, it gets down for me: the symtoms is: under that high load test: ATS starves all available sockets on server. So as result: ATS = is unable to accept new connections, MYSQL, HTTPD (origin) also gets down (mysql server has gone aways etc). Sockets starvation + CPU starvation. In fact when i do stress test from other box (over WAN) any request reaching ATS should get (and gets i can confirm) 403 (ATS's 403) because i wrote simple rule to get always 403 against requests. So i can confirm my concurrent requests never reaches origin (HTTPD). Here is what i see under the stress-test: # traffic_top Disk Used 1.2M Ram Hit 40.8% GET 1.2% 200 53.5% Disk Total 837.8M Fresh 0.3% HEAD 0.2% 206 0.0% Ram Used 99.8K Revalidate 0.0% POST 0.2% 301 0.0% Ram Total 4.3G Cold 98.8% 2xx 53.5% 302 1.3% Lookups 7.2K Changed 0.5% 3xx 45.4% 304 44.0% Writes 0.0 Not Cache 0.0% 4xx 1.1% 404 0.8% Updates 37.0 No Cache 0.0% 5xx 0.0% 502 0.0% Deletes 0.0 Fresh (ms) 44.1G Conn Fail 0.0 100 B 45.2% Read Activ 0.0 Reval (ms) 0.0 Other Err 78.0K 1 KB 16.2% Writes Act 1.0 Cold (ms) 161.7M Abort 22.0 3 KB 1.1% Update Act 0.0 Chang (ms) 29.0G 5 KB 4.9% Entries 42.0 Not (ms) 337.8G 10 KB 7.9% Avg Size 29.3K No (ms) 0.0 1 MB 21.2% DNS Lookup 7.2K DNS Hit 100.0% > 1 MB 0.2% DNS Hits 7.2K DNS Entry 3.0 CLIENT ORIGIN SERVER Requests 7.2K Head Bytes 16.1M Requests 7.2K Head Bytes 1.9M Req/Conn 0.4 Body Bytes 266.4M Req/Conn 1.2 Body Bytes 176.8M New Conn 19.7K Avg Size 39.0K New Conn 5.8K Avg Size 24.8K Curr Conn 921.0 Net (bits) 2.3G Curr Conn 1.0 Net (bits) 1.4G Active Con 144.0 Resp (ms) 96.6 Dynamic KA 5.0 xxxxxxxxxx (r)esponse (q)uit (h)elp (a)bsolute - ------------------------- $ watch -n 2 "netstat -tulpan|grep -i "traffic_ser"|wc -l" - ------------- Every 2.0s: netstat -tulpan|grep -i traffic_ser|wc -l Mon Oct 19 17:35:22 2020 914 - ------------- # free -m total used free shared buff/cache a vailable Mem: 7809 1488 5831 163 490 5919 Swap: 0 0 0 - ---------------- # top -u nobody top - 17:46:09 up 1 day, 10:42, 1 user, load average: 4.18, 2.35, 1.83 Tasks: 155 total, 3 running, 152 sleeping, 0 stopped, 0 zombie %Cpu(s): 95.6 us, 3.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.7 hi, 0.5 si, 0.0 st MiB Mem : 7809.8 total, 5843.2 free, 1473.2 used, 493.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5934.3 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 167102 nobody 20 0 1491900 90612 14648 S 362.1 1.1 31:59.49 [TS_MAIN] 43697 nobody 20 0 329940 19284 8572 S 0.0 0.2 0:35.26 traffic_manager - ------------ # netstat -tulpan|grep -i :80|wc -l 2599 - ------------ Here is my rule: - ------------------ // config_1.conf cond %{READ_REQUEST_HDR_HOOK} cond %{CLIENT-HEADER:CF-Connecting-IP} /MY_.IP._FROM_.WHERE_.I_.DoS/ set-status 403 - ------------------- I'm not using cache mechanism and not doing negative caching etc. This is a simple 403 page of ATS where i hit always. Regarding RAM = allocated memory 4 GB. Here is my records.config (a bit messed up while playing with it :)) $ cat /usr/local/etc/trafficserver/records.config ####################################################################### ####### # *NOTE*: All options covered in this file should be documented in the docs: # # https://docs.trafficserver.apache.org/records.config ####################################################################### ####### ####################################################################### ####### # Thread configurations. Docs: # https://docs.trafficserver.apache.org/records.config#thread-variables ####################################################################### ####### CONFIG proxy.config.exec_thread.autoconfig INT 1 CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.5 CONFIG proxy.config.exec_thread.limit INT 2 CONFIG proxy.config.accept_threads INT 1 CONFIG proxy.config.task_threads INT 2 CONFIG proxy.config.cache.threads_per_disk INT 8 CONFIG proxy.config.exec_thread.affinity INT 1 ####################################################################### ####### # Specify server addresses and ports to bind for HTTP and HTTPS. Docs: # https://docs.trafficserver.apache.org/records.config#proxy.config.http.server_ports ####################################################################### ####### CONFIG proxy.config.http.server_ports STRING 80 8096:ipv6 ####################################################################### ####### # Via: headers. Docs: # https://docs.trafficserver.apache.org/records.config#proxy-config-http-insert-response-via-str ####################################################################### ####### CONFIG proxy.config.http.insert_request_via_str INT 0 CONFIG proxy.config.http.insert_response_via_str INT 0 #CONFIG proxy.config.http.response_via_str 0 #CONFIG proxy.config.http.insert_age_in_response #CONFIG proxy.config.http.response_via_str GoogleWebServer CONFIG proxy.config.http.response_server_enabled INT 0 ####################################################################### ####### # Parent proxy configuration, in addition to these settings also see parent.config. Docs: # https://docs.trafficserver.apache.org/records.config#parent-proxy-configuration # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html ####################################################################### ####### CONFIG proxy.config.http.parent_proxy_routing_enable INT 0 CONFIG proxy.config.http.parent_proxy.retry_time INT 300 CONFIG proxy.config.http.parent_proxy.connect_attempts_timeout INT 30 CONFIG proxy.config.http.forward.proxy_auth_to_parent INT 0 CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1 CONFIG proxy.config.http.cache.ignore_client_no_cache INT 1 ####################################################################### ####### # HTTP connection timeouts (secs). Docs: # https://docs.trafficserver.apache.org/records.config#http-connection-timeouts ####################################################################### ####### CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 20 CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 20 CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 20 CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 20 CONFIG proxy.config.http.transaction_active_timeout_in INT 900 CONFIG proxy.config.http.transaction_active_timeout_out INT 0 CONFIG proxy.config.http.accept_no_activity_timeout INT 120 CONFIG proxy.config.net.default_inactivity_timeout INT 86400 ####################################################################### ####### # Origin server connect attempts. Docs: # https://docs.trafficserver.apache.org/records.config#origin-server-connect-attempts ####################################################################### ####### CONFIG proxy.config.http.connect_attempts_max_retries INT 10 CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 3 CONFIG proxy.config.http.connect_attempts_rr_retries INT 3 CONFIG proxy.config.http.connect_attempts_timeout INT 30 CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800 CONFIG proxy.config.http.down_server.cache_time INT 60 CONFIG proxy.config.http.down_server.abort_threshold INT 10 ####################################################################### ####### # Negative response caching, for redirects and errors. Docs: # https://docs.trafficserver.apache.org/records.config#negative-response-caching ####################################################################### ####### CONFIG proxy.config.http.negative_caching_enabled INT 0 CONFIG proxy.config.http.negative_caching_lifetime INT 1800 ####################################################################### ####### # Proxy users variables. Docs: # https://docs.trafficserver.apache.org/records.config#proxy-user-variables ####################################################################### ####### CONFIG proxy.config.http.insert_client_ip INT 1 CONFIG proxy.config.http.insert_squid_x_forwarded_for INT 1 ####################################################################### ####### # Security. Docs: # https://docs.trafficserver.apache.org/records.config#security ####################################################################### ####### CONFIG proxy.config.http.push_method_enabled INT 0 ####################################################################### ####### # Enable / disable HTTP caching. Useful for testing, but also as an # overridable (per remap) config ####################################################################### ####### CONFIG proxy.config.http.cache.http INT 1 ####################################################################### ####### # Cache control. Docs: # https://docs.trafficserver.apache.org/records.config#cache-control # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/cache.config.en.html ####################################################################### ####### CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 1 CONFIG proxy.config.http.normalize_ae INT 1 CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 1 CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT 1 # https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-when-to-revalidate CONFIG proxy.config.http.cache.when_to_revalidate INT 0 # https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-required-headers CONFIG proxy.config.http.cache.required_headers INT 2 ####################################################################### ####### # Heuristic cache expiration. Docs: # https://docs.trafficserver.apache.org/records.config#heuristic-expiration ####################################################################### ####### CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 3600 CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.10 ####################################################################### ####### # Network. Docs: # https://docs.trafficserver.apache.org/records.config#network ####################################################################### ####### CONFIG proxy.config.net.connections_throttle INT 50000 CONFIG proxy.config.net.max_connections_in INT 50000 CONFIG proxy.config.net.max_connections_active_in INT 50000 ####################################################################### ####### # RAM and disk cache configurations. Docs: # https://docs.trafficserver.apache.org/records.config#ram-cache # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.config.en.html ####################################################################### ####### CONFIG proxy.config.cache.ram_cache.size INT -1 CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-limits-http-max-alts CONFIG proxy.config.cache.limits.http.max_alts INT 5 # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-max-doc-size CONFIG proxy.config.cache.max_doc_size INT 0 CONFIG proxy.config.cache.min_average_object_size INT 8000 ####################################################################### ####### # Logging Config. Docs: # https://docs.trafficserver.apache.org/records.config#logging-configuration # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/logging.yaml.en.html ####################################################################### ####### CONFIG proxy.config.log.logging_enabled INT 3 CONFIG proxy.config.log.max_space_mb_for_logs INT 25000 CONFIG proxy.config.log.max_space_mb_headroom INT 1000 CONFIG proxy.config.log.rolling_enabled INT 1 CONFIG proxy.config.log.rolling_interval_sec INT 86400 CONFIG proxy.config.log.rolling_size_mb INT 10 CONFIG proxy.config.log.auto_delete_rolled_files INT 1 CONFIG proxy.config.log.periodic_tasks_interval INT 5 ####################################################################### ####### # These settings control remapping, and if the proxy allows (open) forward proxy or not. Docs: # https://docs.trafficserver.apache.org/records.config#url-remap-rules # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/remap.config.en.html ####################################################################### ####### CONFIG proxy.config.url_remap.remap_required INT 1 # https://docs.trafficserver.apache.org/records.config#proxy-config-url-remap-pristine-host-hdr CONFIG proxy.config.url_remap.pristine_host_hdr INT 0 # https://docs.trafficserver.apache.org/records.config#reverse-proxy CONFIG proxy.config.reverse_proxy.enabled INT 1 ####################################################################### ####### # SSL Termination. Docs: # https://docs.trafficserver.apache.org/records.config#client-related-configuration # https://docs.trafficserver.apache.org/en/latest/admin-guide/files/ssl_multicert.config.en.html ####################################################################### ####### CONFIG proxy.config.ssl.client.verify.server INT 0 CONFIG proxy.config.ssl.client.CA.cert.filename STRING NULL CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA-AES256- GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM- SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-DSS- AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM- SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA- AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA- AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA- AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS- AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128- SHA:DHE-DSS-AES128- SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3- SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA ####################################################################### ####### # Debugging. Docs: # https://docs.trafficserver.apache.org/records.config#diagnostic-logging-configuration ####################################################################### ####### CONFIG proxy.config.diags.debug.enabled INT 0 CONFIG proxy.config.diags.debug.tags STRING http|dns # ToDo: Undocumented CONFIG proxy.config.dump_mem_info_frequency INT 0 CONFIG proxy.config.http.slow.log.threshold INT 0 CONFIG proxy.config.hostdb.host_file.path STRING /etc/hosts #CONFIG proxy.config.thread.default.stacksize INT 3145728 CONFIG proxy.config.cache.ram_cache.size INT 4G CONFIG proxy.config.net.inactivity_check_frequency INT 80 - ---------------- Cheers, On Mon, 2020-10-19 at 08:29 -0500, Susan Hinrichs wrote: > Sounds like ATS should be able to keep up with that work > load. However, > I've not worked with httpress specifically, so I'm not certain what > kind of > RPS those arguments would correspond to.. > > How did your ATS box fail? Did it crash? Did it just get so slow > that the > requests started timing out? Is your request mapping through to an > origin? Or hitting cache? > > I would start looking at memory usage. If ATS starts swapping, > things go > downhill fast. If you are caching, how much ram cache do you have > configured? If you aren't caching, does you configuration still have > ram > cache enabled? > > Susan > > On Sun, Oct 18, 2020 at 3:11 PM pentester <[email protected]> > wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > Hello Folks! > > > > I need your valuable advise regarding how to fine-tune ATS for > > highly > > loaded server. > > > > The situation: > > Test server: (VPS) CPU(s) 4 (Intel(R) Xeon(R) CPU E5-2630 v4 @ > > 2.20GHz), Total RAM: 8 GB, HDD (SSD) > > OS: Centos 8 > > ATSv: 8.1.0 > > > > Problem: According to ATS documentation and it's default config > > files > > it can handle 30k concurrent requests. That's great! > > > > //records.config > > CONFIG proxy.config.net.connections_throttle INT 30000 > > CONFIG proxy.config.net.max_connections_in INT 30000 > > CONFIG proxy.config.net.max_connections_active_in INT 30000 > > > > > > > > **During the the stress-test: 400-500 concurrent connections > > against > > simple 403 page completely downs ATS. (over WAN) > > > > httpress -n 100000000 -c 500 -t 4 'http://mytestsite.domain/' > > > > > > (?) Question: > > Should i play with settings / fine-tune them (if yes, please let me > > know them) or better to upgrade HW / or any other advise on this > > please. > > > > > > Played a bit with the following settings (increased/decreased > > them). No > > avail. > > > > proxy.config.net.connections_throttle > > proxy.config.net.max_connections_in > > proxy.config.net.max_connections_active_in > > > > > > proxy.config.http.keep_alive_no_activity_timeout_in > > proxy.config.http.keep_alive_no_activity_timeout_out > > proxy.config.http.transaction_no_activity_timeout_in > > proxy.config.http.transaction_no_activity_timeout_out > > > > > > > > Thanks in advance! > > > > > > > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 > > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr > > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P > > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY > > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s > > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c > > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn > > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL > > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF > > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o > > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB > > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 > > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G > > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz > > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp > > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk > > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ > > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW > > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb > > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN > > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n > > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw > > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU > > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd > > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO > > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs > > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ > > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 > > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ > > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn > > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d > > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY > > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn > > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n > > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 > > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc > > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx > > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr > > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 > > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 > > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl > > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl > > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP > > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY > > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E > > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv > > 0/I9kH7m8tCtYKI= > > =vsDs > > - -----END PGP PUBLIC KEY BLOCK----- > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > iQIcBAEBAgAGBQJfjKGcAAoJEMrVQenMmzu+Sa0P/2BIU1biMZj/MM7oVIiNecl8 > > GTfiTl2s+2Df56LA6o5IBKKBcKNJQHKwK2aOd9bhF5HY4uH9hl53Aq4mWcN8L3bY > > 5ugmT9M7aSd5qRJvyWN1E6ryCLzaagdz+vSC63xi9G9slsdOZ7xVIgYE6nVLWOVG > > 48zokAwh7MxjIkogdRCTOQrgehZWh32ugXgQoxXFSozge+jO0nFwJmdB52fNsgEA > > tVENKE5xhYMdvGNTdRoq8RZ0PWJ0/2lhut77rnHWQ9eo/PRlMxEPgByWZkvLVUZ5 > > lJKYpEWFVYrnox3D9bIl49I0bqNqWAVOfVMQs09cMRnSoqJxTPy/TMCOLNInp0mL > > 4J+O65ga690Pap0ZeywSIDIIGy3eC14m9ACf8dW8JP7h75CpGhwZLtvWV/S+nkcG > > +ZW6XNKoU4H5Jbi/PT9OZmYeJu0CsmlxjBRaq2oexlpEBt6pwaN7B347eb1DlCEr > > 8dKUzpeik7PNTwkNCxq+tDHrQdbC1ANJyPN3avKvzmKAiAkn3DK+IPBAY84On0LH > > 270iO96EAgRg53/csSCJJLxSnpBuJs8O1xGy6OFI4xxsDMBygSjo7AzNgu53tNDa > > P6lQynfx+hLyYJu61CUY3yj6HAF5phlONz57ZkPt4BVu2P1RbSeILY7w9sacEDOR > > I8G8prtCJ2SLFJXCvmOj > > =7rQ7 > > -----END PGP SIGNATURE----- > > > > - -- Hörmətlə, XRITDX KİMM (CERT.GOV.AZ) - ------------------------------ - -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v2.0.22 (GNU/Linux) mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv 0/I9kH7m8tCtYKI= =vsDs - -----END PGP PUBLIC KEY BLOCK----- -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJfjbZcAAoJEMrVQenMmzu+0poQAOaUujECxax8x4wvmk11jbw6 k8K80ryCXDmV5gnZj7DEcwpYrNEM49S2DSeHVUckQOngOVz5lsPFuGxdKimIuNbs L7W/1AP11VPeyZOiJjwYZB6/S9xEsUGGxKzCz5dE1hkpQgai4NToFSeOXLOT/8m4 9nQOW2S+cvamTKi0NQV9leqGFr4lkGeFXG623YFsfT/+FsmZN0/aFQfuw3qkvZjE liDOpEdzAEOr3vMrA6A56ijIgC/Z47t7bGwE/nNmcBLcKgh9qCq3dpg7eVyg6C1O 5GpG9tZ5TlvswzvHd/waFoPBCZWEIgtv6qeIOIuMzibua+HEDIEbd8cbQQDzAmyM ib9Y/9Fk/O2vFD7wpJX5epbttvIyGlUftsgDt7CjaaHXaU/kbVLKmkw/gGhPW6iC GJcMosxWUCsDQ7Uh3k8g1BWDNEl2Q6UPYgVokcyDEToac+yXbZkBbJHCYM0tJe7y c22pKdGAJ4LkLZYhZfjQ30FwA/z+6MjGHbUbjBnk/lJ0DAyV0i0t+aL8VYP+CrOM vz6O0IWXW0iK8Bpj6/FpuVMrQRWAKimbnHj2VO8sWQbl3WcQCFK7MA8Td/B2yV+t cOl0M+/eo9eG8aQXU/zq5GPt8sj77qda2XjHPud/GnWXJBtqEKV+GZwB/AQ3yBn7 AsUvUTOnsrAy7FSKduMN =Ckx4 -----END PGP SIGNATURE-----
