-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Susan,
The stress-tests are conducted using the 1 Gbps link. Plus my test machine (where ATS installed) it has unlimited bandwith too. So it's unlikely that i'm overhelming the link IMO. From other hand i can connect via SSH to ATS box. If that was network link overhelming i don't think i should be able to login SSH etc. Thank you once again! Cheers, On Mon, 2020-10-19 at 15:49 -0500, Susan Hinrichs wrote: > One last question. What is the bandwidth of your WAN connection? Is > it > possible that your client requests are overwhelming the network? > > On Mon, Oct 19, 2020 at 12:13 PM pentester <[email protected]> > wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > Dear Susan, once again thank you very much for your valuable > > feedback. I really highly appreciate it. > > > > Well, to be honest this is a test environment just for playing with > > ATS > > (especialy to create env to mitigate layer 7 DDoS attacks etc > > cases.) I > > did nothing to server setup (no pre-configured ports reuse / no > > increased security limits etc. Just default Centos 8 installation) > > > > For the second part: > > "Do your client requests (and sql requests) just timeout after > > bit?" > > > > Well, i'm not sure about timeout but definitely mysqld goes down > > (according to journal files i can see that records too (mysql > > server > > has gone aways etc) > > > > Currently, for prod i did cookie based checking via ATS (human or > > bot) > > but i'm sure it will not help in such cases when someone smashes > > the > > server via httpress over Cloudflare. The only working mitigation / > > solution i think should be Cloudflare + "I'm under attack mode" + > > additionally ATS's cookie based checking something like this: > > > > > > cond %{READ_REQUEST_HDR_HOOK} > > cond %{COOKIE:botorhuman} = "" > > set-status 403 > > > > or rate limiting (without Cloudflare) > > > > > > I'll try to play with port: with "local port range and other port > > reuse > > settings " and increasing / decreasing default security limits too. > > > > Plus for future i'm planning to use ingres shaping / policing? to > > mitigate such cases. Better to try than nothing) > > > > Once again thank you very much! > > > > > > Cheers, > > > > > > > > > > On Mon, 2020-10-19 at 11:43 -0500, Susan Hinrichs wrote: > > > So definitely not a memory problem. > > > > > > Do your client requests (and sql requests) just timeout after a > > > bit? If > > > I'm reading your netstat output correctly, traffic_server just > > > has > > > 2600 > > > sockets open for port 80, which does not seem like too many. > > > > > > It is quite easy in a single client stress test to run out of > > > ephemeral > > > ports from the single client to the server. But that should > > > affect > > > connections to other processes such as SQL. I assume that you've > > > already > > > adjusted the local port range and other port reuse settings on > > > your > > > tests > > > boxes to set up more ephemeral ports. > > > > > > On Mon, Oct 19, 2020 at 10:51 AM pentester <[email protected]> > > > wrote: > > > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > > Hash: SHA1 > > > > > > > > Hello Susan! > > > > > > > > Thank you for your feedback. Well, i think my conclusion is > > > > wrong > > > > about > > > > 500 concurrent connections against ATS. > > > > Because: httpress -n 100000000 -c 500 -t 4 ' > > > > http://mytestsite.domain/' > > > > > > > > 4 threads x 500 concurrent connections against ATS does = 2000 > > > > concurrent connnects. > > > > > > > > Anyways, it gets down for me: the symtoms is: under that high > > > > load > > > > test: ATS starves all available sockets on server. > > > > So as result: ATS = is unable to accept new connections, MYSQL, > > > > HTTPD > > > > (origin) also gets down (mysql server has gone aways etc). > > > > Sockets starvation + CPU starvation. > > > > > > > > In fact when i do stress test from other box (over WAN) any > > > > request > > > > reaching ATS should get (and gets i can confirm) 403 (ATS's > > > > 403) > > > > because i wrote simple rule to get always 403 against requests. > > > > So > > > > i > > > > can confirm my concurrent requests never reaches origin > > > > (HTTPD). > > > > > > > > > > > > Here is what i see under the stress-test: > > > > > > > > # traffic_top > > > > > > > > Disk Used 1.2M Ram > > > > Hit 40.8% GET 1.2% 200 53.5% > > > > Disk Total > > > > 837.8M Fresh 0.3% HEAD 0.2% 206 > > > > 0.0 > > > > % > > > > Ram > > > > Used 99.8K Revalidate 0.0% POST 0.2% 301 > > > > 0.0% > > > > Ram > > > > Total 4.3G Cold 98.8% 2xx 53.5% 302 > > > > 1.3% > > > > Lookups 7.2K Changed 0.5% 3xx 45.4% 3 > > > > 04 > > > > 44.0% > > > > Writes 0.0 Not > > > > Cache 0.0% 4xx 1.1% 404 0.8% > > > > Updates 37.0 No > > > > Cache 0.0% 5xx 0.0% 502 0.0% > > > > Deletes 0.0 Fresh (ms) 44.1G Conn > > > > Fail 0.0 100 > > > > B 45.2% > > > > Read Activ 0.0 Reval (ms) 0.0 Other Err 78.0K 1 > > > > KB 16.2% > > > > Writes Act 1.0 Cold (ms) 161.7M Abort 22.0 3 > > > > KB 1.1% > > > > Update Act 0.0 Chang (ms) 29.0G 5 > > > > KB 4.9% > > > > Entries 42.0 Not > > > > (ms) 337.8G 10 > > > > KB 7.9% > > > > Avg Size 29.3K No (ms) 0.0 1 > > > > MB 21.2% > > > > DNS Lookup 7.2K DNS Hit 100.0% > > > > > 1 > > > > MB 0.2% > > > > DNS Hits 7.2K DNS Entry 3.0 > > > > CLIENT ORIGIN > > > > SERVER > > > > Requests 7.2K Head > > > > Bytes 16.1M Requests 7.2K Head > > > > Bytes 1.9M > > > > Req/Conn 0.4 Body Bytes > > > > 266.4M Req/Conn 1.2 Body > > > > Bytes 176.8M > > > > New Conn 19.7K Avg Size 39.0K New > > > > Conn 5.8K Avg > > > > Size 24.8K > > > > Curr Conn 921.0 Net (bits) 2.3G Curr > > > > Conn 1.0 Net > > > > (bits) 1.4G > > > > Active Con 144.0 Resp (ms) 96.6 > > > > Dynamic KA 5.0 > > > > xxxxxxxxxx (r)esponse (q)uit > > > > (h)elp > > > > (a)bsolute > > > > > > > > > > > > - ------------------------- > > > > > > > > > > > > > > > > $ watch -n 2 "netstat -tulpan|grep -i "traffic_ser"|wc -l" > > > > - ------------- > > > > Every 2.0s: netstat -tulpan|grep -i traffic_ser|wc -l > > > > Mon Oct 19 17:35:22 2020 > > > > > > > > 914 > > > > > > > > - ------------- > > > > > > > > > > > > > > > > > > > > # free -m > > > > total used free shared buff/c > > > > ache > > > > a > > > > vailable > > > > Mem: 7809 1488 5831 163 > > > > 490 > > > > 5919 > > > > Swap: 0 0 0 > > > > > > > > > > > > - ---------------- > > > > > > > > # top -u nobody > > > > top - 17:46:09 up 1 day, 10:42, 1 user, load average: 4.18, > > > > 2.35, > > > > 1.83 > > > > Tasks: 155 total, 3 running, 152 sleeping, 0 stopped, 0 > > > > zombie > > > > %Cpu(s): 95.6 us, 3.2 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.7 > > > > hi, 0.5 > > > > si, 0.0 st > > > > MiB Mem : 7809.8 total, 5843.2 free, 1473.2 > > > > used, 493.4 > > > > buff/cache > > > > MiB Swap: 0.0 total, 0.0 free, 0.0 > > > > used. 5934.3 > > > > avail > > > > Mem > > > > > > > > PID USER PR NI VIRT RES SHR > > > > S %CPU %MEM TIME+ > > > > COMMAND > > > > > > > > 167102 nobody 20 0 1491900 90612 14648 S > > > > 362.1 1.1 31:59.49 > > > > [TS_MAIN] > > > > > > > > 43697 nobody 20 0 329940 19284 8572 > > > > S 0.0 0.2 0:35.26 > > > > traffic_manager > > > > > > > > > > > > > > > > - ------------ > > > > > > > > # netstat -tulpan|grep -i :80|wc -l > > > > 2599 > > > > > > > > > > > > - ------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here is my rule: > > > > > > > > - ------------------ > > > > > > > > // config_1.conf > > > > > > > > cond %{READ_REQUEST_HDR_HOOK} > > > > cond %{CLIENT-HEADER:CF-Connecting-IP} > > > > /MY_.IP._FROM_.WHERE_.I_.DoS/ > > > > set-status 403 > > > > > > > > - ------------------- > > > > > > > > > > > > I'm not using cache mechanism and not doing negative caching > > > > etc. > > > > This > > > > is a simple 403 page of ATS where i hit always. > > > > > > > > Regarding RAM = allocated memory 4 GB. > > > > > > > > > > > > > > > > Here is my records.config (a bit messed up while playing with > > > > it > > > > :)) > > > > > > > > > > > > > > > > $ cat /usr/local/etc/trafficserver/records.config > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # *NOTE*: All options covered in this file should be documented > > > > in > > > > the > > > > docs: > > > > # > > > > # https://docs.trafficserver.apache.org/records.config > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Thread configurations. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#thread-variables > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.exec_thread.autoconfig INT 1 > > > > CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.5 > > > > CONFIG proxy.config.exec_thread.limit INT 2 > > > > CONFIG proxy.config.accept_threads INT 1 > > > > CONFIG proxy.config.task_threads INT 2 > > > > CONFIG proxy.config.cache.threads_per_disk INT 8 > > > > CONFIG proxy.config.exec_thread.affinity INT 1 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Specify server addresses and ports to bind for HTTP and > > > > HTTPS. > > > > Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy.config.http.server_ports > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.server_ports STRING 80 8096:ipv6 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Via: headers. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-insert-response-via-str > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.insert_request_via_str INT 0 > > > > CONFIG proxy.config.http.insert_response_via_str INT 0 > > > > #CONFIG proxy.config.http.response_via_str 0 > > > > #CONFIG proxy.config.http.insert_age_in_response > > > > #CONFIG proxy.config.http.response_via_str GoogleWebServer > > > > CONFIG proxy.config.http.response_server_enabled INT 0 > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Parent proxy configuration, in addition to these settings > > > > also > > > > see > > > > parent.config. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#parent-proxy-configuration > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.parent_proxy_routing_enable INT 0 > > > > CONFIG proxy.config.http.parent_proxy.retry_time INT 300 > > > > CONFIG proxy.config.http.parent_proxy.connect_attempts_timeout > > > > INT > > > > 30 > > > > CONFIG proxy.config.http.forward.proxy_auth_to_parent INT 0 > > > > CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT > > > > 1 > > > > CONFIG proxy.config.http.cache.ignore_client_no_cache INT 1 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # HTTP connection timeouts (secs). Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#http-connection-timeouts > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT > > > > 20 > > > > CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT > > > > 20 > > > > CONFIG proxy.config.http.transaction_no_activity_timeout_in INT > > > > 20 > > > > CONFIG proxy.config.http.transaction_no_activity_timeout_out > > > > INT 20 > > > > CONFIG proxy.config.http.transaction_active_timeout_in INT 900 > > > > CONFIG proxy.config.http.transaction_active_timeout_out INT 0 > > > > CONFIG proxy.config.http.accept_no_activity_timeout INT 120 > > > > CONFIG proxy.config.net.default_inactivity_timeout INT 86400 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Origin server connect attempts. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#origin-server-connect-attempts > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.connect_attempts_max_retries INT 10 > > > > CONFIG > > > > proxy.config.http.connect_attempts_max_retries_dead_server > > > > INT 3 > > > > CONFIG proxy.config.http.connect_attempts_rr_retries INT 3 > > > > CONFIG proxy.config.http.connect_attempts_timeout INT 30 > > > > CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800 > > > > CONFIG proxy.config.http.down_server.cache_time INT 60 > > > > CONFIG proxy.config.http.down_server.abort_threshold INT 10 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Negative response caching, for redirects and errors. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#negative-response-caching > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.negative_caching_enabled INT 0 > > > > CONFIG proxy.config.http.negative_caching_lifetime INT 1800 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Proxy users variables. Docs: > > > > # > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-user-variables > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.insert_client_ip INT 1 > > > > CONFIG proxy.config.http.insert_squid_x_forwarded_for INT 1 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Security. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#security > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.push_method_enabled INT 0 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Enable / disable HTTP caching. Useful for testing, but also > > > > as an > > > > # overridable (per remap) config > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.cache.http INT 1 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Cache control. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#cache-control > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/cache.config.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 1 > > > > CONFIG proxy.config.http.normalize_ae INT 1 > > > > CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 1 > > > > CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT > > > > 1 > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-when-to-revalidate > > > > CONFIG proxy.config.http.cache.when_to_revalidate INT 0 > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-required-headers > > > > CONFIG proxy.config.http.cache.required_headers INT 2 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Heuristic cache expiration. Docs: > > > > # > > > > > > > > https://docs.trafficserver.apache.org/records.config#heuristic-expiration > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 3600 > > > > CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400 > > > > CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.10 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Network. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#network > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.net.connections_throttle INT 50000 > > > > CONFIG proxy.config.net.max_connections_in INT 50000 > > > > CONFIG proxy.config.net.max_connections_active_in INT 50000 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # RAM and disk cache configurations. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#ram-cache > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/storage.config.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.cache.ram_cache.size INT -1 > > > > CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304 > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-cache-limits-http-max-alts > > > > CONFIG proxy.config.cache.limits.http.max_alts INT 5 > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-cache-max-doc-size > > > > CONFIG proxy.config.cache.max_doc_size INT 0 > > > > CONFIG proxy.config.cache.min_average_object_size INT 8000 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Logging Config. Docs: > > > > # > > > > > > > > https://docs.trafficserver.apache.org/records.config#logging-configuration > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/logging.yaml.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.log.logging_enabled INT 3 > > > > CONFIG proxy.config.log.max_space_mb_for_logs INT 25000 > > > > CONFIG proxy.config.log.max_space_mb_headroom INT 1000 > > > > CONFIG proxy.config.log.rolling_enabled INT 1 > > > > CONFIG proxy.config.log.rolling_interval_sec INT 86400 > > > > CONFIG proxy.config.log.rolling_size_mb INT 10 > > > > CONFIG proxy.config.log.auto_delete_rolled_files INT 1 > > > > CONFIG proxy.config.log.periodic_tasks_interval INT 5 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # These settings control remapping, and if the proxy allows > > > > (open) > > > > forward proxy or not. Docs: > > > > # > > > > https://docs.trafficserver.apache.org/records.config#url-remap-rules > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/remap.config.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.url_remap.remap_required INT 1 > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#proxy-config-url-remap-pristine-host-hdr > > > > CONFIG proxy.config.url_remap.pristine_host_hdr INT 0 > > > > # > > > > https://docs.trafficserver.apache.org/records.config#reverse-proxy > > > > CONFIG proxy.config.reverse_proxy.enabled INT 1 > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # SSL Termination. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#client-related-configuration > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/en/latest/admin-guide/files/ssl_multicert.config.en.html > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.ssl.client.verify.server INT 0 > > > > CONFIG proxy.config.ssl.client.CA.cert.filename STRING NULL > > > > CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-ECDSA- > > > > AES256- > > > > GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM- > > > > SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM- > > > > SHA384:DHE- > > > > DSS- > > > > AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM- > > > > SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE- > > > > ECDSA- > > > > AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128- > > > > SHA256:ECDHE- > > > > RSA- > > > > AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE- > > > > RSA- > > > > AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE- > > > > DSS- > > > > AES128-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA- > > > > AES128- > > > > SHA:DHE-DSS-AES128- > > > > SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS- > > > > DES- > > > > CBC3- > > > > SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA > > > > > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > # Debugging. Docs: > > > > # > > > > > > > > > > > > https://docs.trafficserver.apache.org/records.config#diagnostic-logging-configuration > > > > ############################################################### > > > > #### > > > > #### > > > > ####### > > > > CONFIG proxy.config.diags.debug.enabled INT 0 > > > > CONFIG proxy.config.diags.debug.tags STRING http|dns > > > > # ToDo: Undocumented > > > > CONFIG proxy.config.dump_mem_info_frequency INT 0 > > > > CONFIG proxy.config.http.slow.log.threshold INT 0 > > > > > > > > CONFIG proxy.config.hostdb.host_file.path STRING /etc/hosts > > > > > > > > > > > > #CONFIG proxy.config.thread.default.stacksize INT 3145728 > > > > > > > > > > > > CONFIG proxy.config.cache.ram_cache.size INT 4G > > > > CONFIG proxy.config.net.inactivity_check_frequency INT 80 > > > > > > > > > > > > > > > > - ---------------- > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > On Mon, 2020-10-19 at 08:29 -0500, Susan Hinrichs wrote: > > > > > Sounds like ATS should be able to keep up with that work > > > > > load. However, > > > > > I've not worked with httpress specifically, so I'm not > > > > > certain > > > > > what > > > > > kind of > > > > > RPS those arguments would correspond to.. > > > > > > > > > > How did your ATS box fail? Did it crash? Did it just get so > > > > > slow > > > > > that the > > > > > requests started timing out? Is your request mapping through > > > > > to > > > > > an > > > > > origin? Or hitting cache? > > > > > > > > > > I would start looking at memory usage. If ATS starts > > > > > swapping, > > > > > things go > > > > > downhill fast. If you are caching, how much ram cache do you > > > > > have > > > > > configured? If you aren't caching, does you configuration > > > > > still > > > > > have > > > > > ram > > > > > cache enabled? > > > > > > > > > > Susan > > > > > > > > > > On Sun, Oct 18, 2020 at 3:11 PM pentester < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > > > > Hash: SHA1 > > > > > > > > > > > > Hello Folks! > > > > > > > > > > > > I need your valuable advise regarding how to fine-tune ATS > > > > > > for > > > > > > highly > > > > > > loaded server. > > > > > > > > > > > > The situation: > > > > > > Test server: (VPS) CPU(s) 4 (Intel(R) Xeon(R) CPU E5-2630 > > > > > > v4 @ > > > > > > 2.20GHz), Total RAM: 8 GB, HDD (SSD) > > > > > > OS: Centos 8 > > > > > > ATSv: 8.1.0 > > > > > > > > > > > > Problem: According to ATS documentation and it's default > > > > > > config > > > > > > files > > > > > > it can handle 30k concurrent requests. That's great! > > > > > > > > > > > > //records.config > > > > > > CONFIG proxy.config.net.connections_throttle INT 30000 > > > > > > CONFIG proxy.config.net.max_connections_in INT 30000 > > > > > > CONFIG proxy.config.net.max_connections_active_in INT 30000 > > > > > > > > > > > > > > > > > > > > > > > > **During the the stress-test: 400-500 concurrent > > > > > > connections > > > > > > against > > > > > > simple 403 page completely downs ATS. (over WAN) > > > > > > > > > > > > httpress -n 100000000 -c 500 -t 4 ' > > > > > > http://mytestsite.domain/' > > > > > > > > > > > > > > > > > > (?) Question: > > > > > > Should i play with settings / fine-tune them (if yes, > > > > > > please > > > > > > let me > > > > > > know them) or better to upgrade HW / or any other advise on > > > > > > this > > > > > > please. > > > > > > > > > > > > > > > > > > Played a bit with the following settings > > > > > > (increased/decreased > > > > > > them). No > > > > > > avail. > > > > > > > > > > > > proxy.config.net.connections_throttle > > > > > > proxy.config.net.max_connections_in > > > > > > proxy.config.net.max_connections_active_in > > > > > > > > > > > > > > > > > > proxy.config.http.keep_alive_no_activity_timeout_in > > > > > > proxy.config.http.keep_alive_no_activity_timeout_out > > > > > > proxy.config.http.transaction_no_activity_timeout_in > > > > > > proxy.config.http.transaction_no_activity_timeout_out > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > > > > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > > > > > > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlN > > > > > > dvfH > > > > > > 5 > > > > > > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhK > > > > > > sEFN > > > > > > r > > > > > > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTG > > > > > > n3i9 > > > > > > P > > > > > > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb > > > > > > /2qr > > > > > > Y > > > > > > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXF > > > > > > cvt6 > > > > > > s > > > > > > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1 > > > > > > Yl63 > > > > > > c > > > > > > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3 > > > > > > p+hF > > > > > > n > > > > > > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzY > > > > > > k+5I > > > > > > L > > > > > > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVX > > > > > > LWFJ > > > > > > F > > > > > > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWj > > > > > > vQD0 > > > > > > o > > > > > > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwA > > > > > > RAQA > > > > > > B > > > > > > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJ > > > > > > evJF > > > > > > 5 > > > > > > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cy > > > > > > bO77 > > > > > > G > > > > > > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmP > > > > > > wJda > > > > > > z > > > > > > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vd > > > > > > RGdW > > > > > > p > > > > > > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQU > > > > > > vSKP > > > > > > k > > > > > > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02 > > > > > > pdUV > > > > > > Z > > > > > > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb > > > > > > 3/eD > > > > > > W > > > > > > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe > > > > > > 4mfg > > > > > > b > > > > > > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZAD > > > > > > xIco > > > > > > N > > > > > > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu > > > > > > +2C3 > > > > > > n > > > > > > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0 > > > > > > hHCW > > > > > > w > > > > > > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+Y > > > > > > EjUs > > > > > > U > > > > > > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQE > > > > > > QANs > > > > > > d > > > > > > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0 > > > > > > TbwH > > > > > > O > > > > > > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8 > > > > > > CMPj > > > > > > s > > > > > > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88t > > > > > > VhWN > > > > > > + > > > > > > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0m > > > > > > r1Sj > > > > > > 2 > > > > > > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1l > > > > > > cvRh > > > > > > Z > > > > > > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinR > > > > > > Rmuj > > > > > > n > > > > > > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7 > > > > > > Pwa6 > > > > > > d > > > > > > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl3 > > > > > > 9xSc > > > > > > Y > > > > > > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6Vpw > > > > > > N66n > > > > > > n > > > > > > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/Ty > > > > > > rY0/ > > > > > > n > > > > > > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8 > > > > > > FAl6 > > > > > > 8 > > > > > > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmb > > > > > > JNWl > > > > > > c > > > > > > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcR > > > > > > udbA > > > > > > x > > > > > > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReL > > > > > > KbtW > > > > > > r > > > > > > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnW > > > > > > PMmG > > > > > > 7 > > > > > > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIs > > > > > > GaSx > > > > > > 9 > > > > > > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfV > > > > > > PYYO > > > > > > l > > > > > > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf2 > > > > > > 5hVJ > > > > > > l > > > > > > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lO > > > > > > G6uI > > > > > > P > > > > > > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8 > > > > > > cYFU > > > > > > Y > > > > > > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMP > > > > > > HsQ9 > > > > > > E > > > > > > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPD > > > > > > OVTb > > > > > > v > > > > > > 0/I9kH7m8tCtYKI= > > > > > > =vsDs > > > > > > - -----END PGP PUBLIC KEY BLOCK----- > > > > > > -----BEGIN PGP SIGNATURE----- > > > > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > > > > > > > iQIcBAEBAgAGBQJfjKGcAAoJEMrVQenMmzu+Sa0P/2BIU1biMZj/MM7oVIi > > > > > > Necl > > > > > > 8 > > > > > > GTfiTl2s+2Df56LA6o5IBKKBcKNJQHKwK2aOd9bhF5HY4uH9hl53Aq4mWcN > > > > > > 8L3b > > > > > > Y > > > > > > 5ugmT9M7aSd5qRJvyWN1E6ryCLzaagdz+vSC63xi9G9slsdOZ7xVIgYE6nV > > > > > > LWOV > > > > > > G > > > > > > 48zokAwh7MxjIkogdRCTOQrgehZWh32ugXgQoxXFSozge+jO0nFwJmdB52f > > > > > > NsgE > > > > > > A > > > > > > tVENKE5xhYMdvGNTdRoq8RZ0PWJ0/2lhut77rnHWQ9eo/PRlMxEPgByWZkv > > > > > > LVUZ > > > > > > 5 > > > > > > lJKYpEWFVYrnox3D9bIl49I0bqNqWAVOfVMQs09cMRnSoqJxTPy/TMCOLNI > > > > > > np0m > > > > > > L > > > > > > 4J+O65ga690Pap0ZeywSIDIIGy3eC14m9ACf8dW8JP7h75CpGhwZLtvWV/S > > > > > > +nkc > > > > > > G > > > > > > +ZW6XNKoU4H5Jbi/PT9OZmYeJu0CsmlxjBRaq2oexlpEBt6pwaN7B347eb1 > > > > > > DlCE > > > > > > r > > > > > > 8dKUzpeik7PNTwkNCxq+tDHrQdbC1ANJyPN3avKvzmKAiAkn3DK+IPBAY84 > > > > > > On0L > > > > > > H > > > > > > 270iO96EAgRg53/csSCJJLxSnpBuJs8O1xGy6OFI4xxsDMBygSjo7AzNgu5 > > > > > > 3tND > > > > > > a > > > > > > P6lQynfx+hLyYJu61CUY3yj6HAF5phlONz57ZkPt4BVu2P1RbSeILY7w9sa > > > > > > cEDO > > > > > > R > > > > > > I8G8prtCJ2SLFJXCvmOj > > > > > > =7rQ7 > > > > > > -----END PGP SIGNATURE----- > > > > > > > > > > > > > > > > > > > > - -- > > > > > > > > > > > > Hörmətlə, > > > > > > > > XRITDX KİMM (CERT.GOV.AZ) > > > > > > > > - ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH > > > > 5 > > > > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFN > > > > r > > > > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9 > > > > P > > > > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qr > > > > Y > > > > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6 > > > > s > > > > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63 > > > > c > > > > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hF > > > > n > > > > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5I > > > > L > > > > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJ > > > > F > > > > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0 > > > > o > > > > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQA > > > > B > > > > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF > > > > 5 > > > > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77 > > > > G > > > > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJda > > > > z > > > > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdW > > > > p > > > > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKP > > > > k > > > > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUV > > > > Z > > > > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eD > > > > W > > > > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfg > > > > b > > > > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIco > > > > N > > > > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3 > > > > n > > > > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCW > > > > w > > > > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUs > > > > U > > > > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANs > > > > d > > > > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwH > > > > O > > > > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPj > > > > s > > > > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN > > > > + > > > > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj > > > > 2 > > > > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRh > > > > Z > > > > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmuj > > > > n > > > > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6 > > > > d > > > > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xSc > > > > Y > > > > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66n > > > > n > > > > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/ > > > > n > > > > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl6 > > > > 8 > > > > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWl > > > > c > > > > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbA > > > > x > > > > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtW > > > > r > > > > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG > > > > 7 > > > > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx > > > > 9 > > > > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYO > > > > l > > > > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJ > > > > l > > > > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uI > > > > P > > > > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFU > > > > Y > > > > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9 > > > > E > > > > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTb > > > > v > > > > 0/I9kH7m8tCtYKI= > > > > =vsDs > > > > - -----END PGP PUBLIC KEY BLOCK----- > > > > -----BEGIN PGP SIGNATURE----- > > > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > > > > > iQIcBAEBAgAGBQJfjbZcAAoJEMrVQenMmzu+0poQAOaUujECxax8x4wvmk11jbw > > > > 6 > > > > k8K80ryCXDmV5gnZj7DEcwpYrNEM49S2DSeHVUckQOngOVz5lsPFuGxdKimIuNb > > > > s > > > > L7W/1AP11VPeyZOiJjwYZB6/S9xEsUGGxKzCz5dE1hkpQgai4NToFSeOXLOT/8m > > > > 4 > > > > 9nQOW2S+cvamTKi0NQV9leqGFr4lkGeFXG623YFsfT/+FsmZN0/aFQfuw3qkvZj > > > > E > > > > liDOpEdzAEOr3vMrA6A56ijIgC/Z47t7bGwE/nNmcBLcKgh9qCq3dpg7eVyg6C1 > > > > O > > > > 5GpG9tZ5TlvswzvHd/waFoPBCZWEIgtv6qeIOIuMzibua+HEDIEbd8cbQQDzAmy > > > > M > > > > ib9Y/9Fk/O2vFD7wpJX5epbttvIyGlUftsgDt7CjaaHXaU/kbVLKmkw/gGhPW6i > > > > C > > > > GJcMosxWUCsDQ7Uh3k8g1BWDNEl2Q6UPYgVokcyDEToac+yXbZkBbJHCYM0tJe7 > > > > y > > > > c22pKdGAJ4LkLZYhZfjQ30FwA/z+6MjGHbUbjBnk/lJ0DAyV0i0t+aL8VYP+CrO > > > > M > > > > vz6O0IWXW0iK8Bpj6/FpuVMrQRWAKimbnHj2VO8sWQbl3WcQCFK7MA8Td/B2yV+ > > > > t > > > > cOl0M+/eo9eG8aQXU/zq5GPt8sj77qda2XjHPud/GnWXJBtqEKV+GZwB/AQ3yBn > > > > 7 > > > > AsUvUTOnsrAy7FSKduMN > > > > =Ckx4 > > > > -----END PGP SIGNATURE----- > > > > > > > > > > > > - -- > > > > > > Hörmətlə, > > > > XRITDX KİMM (CERT.GOV.AZ) > > > > - ------------------------------ > > > > > > > > > > > > - -----BEGIN PGP PUBLIC KEY BLOCK----- > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 > > c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr > > hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P > > tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY > > maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s > > 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c > > gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn > > 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL > > zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF > > va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o > > 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB > > tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 > > AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G > > Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz > > GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp > > p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk > > Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ > > WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW > > RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb > > AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN > > YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n > > hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw > > bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU > > WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd > > iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO > > 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs > > 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ > > /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 > > YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ > > 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn > > Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d > > FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY > > k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn > > sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n > > qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 > > kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc > > gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx > > sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr > > 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 > > oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 > > eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl > > DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl > > DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP > > U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY > > a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E > > zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv > > 0/I9kH7m8tCtYKI= > > =vsDs > > - -----END PGP PUBLIC KEY BLOCK----- > > -----BEGIN PGP SIGNATURE----- > > Version: GnuPG v2.0.22 (GNU/Linux) > > > > iQIcBAEBAgAGBQJfjcluAAoJEMrVQenMmzu+KxUP/jPLYEDTrndWgnzYXJ0pDqkv > > KWCMb16C8Noaq6dIR5qswVE37cNDZxoEYaKcbuSMj9LNFEbl/JfIyRyRYnQVbuhE > > nFiDNlBbqwJTjL7Valy3sYg6raLQCCE2y+PYsyfr4VMEKTADzUyxg+ZYufi5vMGV > > wWmdzJDvCwD0WS6hs0fvu8ZfzYHh5Ufxi+avkyS78PZZ4FBgBOdxopj9VVAP08Ek > > D45z3obzRqzgIev9sfq18hf00MUuNtDuu723RFpBix9pNwM4WRhsP3d2zWwp0wWY > > g3UiagwiGKiZF9kJ53kDEehni1BPcof4qakJ382uOkDWv0zQ9N5a9JP3Kzj+5knz > > TDNCyxyHqyvGfVHDvvkRPg2KeEg5Pm1nx/OuSCy462xiaZ9Ar8jqF8qaSsb4iTGl > > DOKuhhKPbhemFpRXjqvjpJ5e20BPaycuRvyxm9+LMJT0FJqhLIg1R2Mkc5uRpg2p > > c1iH1VFjjpExXujkMVNBw6wkT0ZHzJuB6z/xSqZnIyjbxDB7rA9i8lphEPiTiCnz > > GGQGJkynYkNoJLsFZ9ZNCOKdX5XPp0ZHJgs7tuGG57311WtC5p4CKXJP9XqsVls0 > > ZkMxOGayCWtZfxEGB7gB6GSu1nmcuF7wZw4pjZZAPoHYFU5gW6NwogGR1taZgZ0Y > > a8db872ruM9NXfIxsHyW > > =yf14 > > -----END PGP SIGNATURE----- > > > > > > - -- Hörmətlə, XRITDX KİMM (CERT.GOV.AZ) - ------------------------------ - -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v2.0.22 (GNU/Linux) mQINBF68kXkBEAD9fDwkcQqbbgbAa7/Oy07r2Lt2SoYbQFAlK8ECa+25QlNdvfH5 c/EBrvb5hyE5D33AZdv9Dee40OTmJGQQc1M6jDb8iOlZiNaDc3ZcfZzGhhKsEFNr hWPnaC+58Z0dTjpG03dRRWFs1Xrn1r2HTqhq5rG5UddnPT3pwFjQBuSxJTGn3i9P tZKl7WPgmfXgGXKDit+tAEjqPRFgiMkJYqsDJZFd+Xoon8Bi/6LK1asxykb/2qrY maEaWkJ7x5nzio/5pne0/OVZtExZCK3439ibEI0ppzNHcCEhua1Kk3SzQXFcvt6s 9QssaC/N/njm07eEJ7GyvV6w9NtJRgQZ0Bf8uYFEkDgdtHnkVnCr/YsS3f1Yl63c gOPMPxYaZrUjYWrJEnm2V5XxdoVYs4s8gxY6jEQ5oWQmMpkPhOekWMkOw+3p+hFn 2nyZhdb8KkvWa34Lif3fNfNoKhk0PWD+rwG9BwqTKxEAUVmxHvy74F1hvzYk+5IL zCDzwGTpCQ/xgIdnjfHdel0tOOIgCnXCsyh/sFuB4XsoVeFdnrws2nVZQVXLWFJF va8WpzVLWr2twHO6Z3pB10JbQcFEPBnYlCO8pcZvydLwcrROG9G2PrUBHWjvQD0o 2kpDC89byGpMVM1cx8AWq7sDtjYeW6vrcFJaHarymG1DzAFUjYO9UhMgNwARAQAB tB9wZW50ZXN0ZXIgPHBlbnRlc3RAY2VydC5nb3YuYXo+iQI/BBMBAgApBQJevJF5 AhsDBQkB4TOABwsJCAcDAgEGFQgCCQoLBBYCAwECHgECF4AACgkQytVB6cybO77G Qg//StzzAgIHZHuDMh94Kh/qOSAxynumTnhZ6VZU0/59CzQUu285cLpZUmPwJdaz GmS795NwZVnbla3bKm1HR7d2LwOB24rSgoCPgbTEqMxDzm2KbdOnnSi98vdRGdWp p3nS7T6oZ2+6CPY8Pg+hvD8LfhwdcUmVNhDENrZ0agSEBmdj+At+7jOBbQUvSKPk Revw/kdQeo+SZKOE2gs/MDJsaSFv4lhyTy6jKbPFHbk5C+ZJ32BPDL3IZ02pdUVZ WKBnX5d3z9EHliKpMyZHQu2998t9wz447ab9W7AIxhuD+vqjrQYRLFk2oKb3/eDW RPDRIfCk2py9+e9j38cnPhtsmXO5eYoXa4mbBwMPieCqRfN3Msna7USbPbe4mfgb AIdrJCYj2TrYs6ErdRXpb4I60iXZlTWXYdQlVYLLdn2Sb6lqfgREhryLZADxIcoN YwYSJE3NsULKlhPub4wOMCIIaQk7GLz02ZcFGORTtnFKHcg91GqcNCj7SGu+2C3n hpPNJVDnbgU4XToMmKgALFpG82JXEuQYUOH8/SB+qlrBMCwoSuLSO2Tu5l0hHCWw bp8al8yIMHCfqAvrCntGE+rpJKHvO7BhzL+tvvA3HJn4loKhMq9J8v+gI+YEjUsU WpEOmusARUiBVi4MlUxoNQH8yS7zMNXkFoaQOf+yWH3BCSi5Ag0EXryReQEQANsd iZZW8vt4STelg0D/PW4sRYUgm0gQgp/V6PlOiwlJ+x7DRCdEtlzLnr/wyB0TbwHO 7hxAhAlAHjgTAbDYAuz2Scm9OiiNiZpJqL/ka9gsVVczXgZzgWg9cuFtHc8CMPjs 0D7yqc6ANQeVJLCqJ/+4jyQzNc5VZPm+tJ6hnTLcEaok+9dPnB8/9cVo88tVhWN+ /9jvGAxB/rcfY+eQEBwxmOk9w74Vs0Fj1pIcsZYrWounYG4Dh9/O+xw3E0mr1Sj2 YJmpTWRrUqLqxeGm5lptn9HBTA6qecJrX9urh7djXUbcRZcptBpaHfYvO1lcvRhZ 6ndpNAnknp/6FaCCqWKwKkAr3TtB7uJbxz+xJAi+fU2ZOkyNeBlavzmZinRRmujn Rdkqgtks0qUMQVdbNju/A36O9uVZyEm+lkMFNNq+Q0dehRLsUjYJJywCjB7Pwa6d FmrRGhTxd9J1qxkzCue941TNYJkPfVAuhyuAAO85TXi3fF20QdxC4IgIzl39xScY k/gTzIbDUFlKn+BIC5O2QSLpof8r0qhkZIXA3D/6HBMpT9IemI2BeOf6VpwN66nn sJqk/3t8naS5nxyZ8lQe85JOyxane5KT7mRK+x9vyBixlnWh3UhRbMP8/TyrY0/n qySu0VOrQn2FlssL13EWvsi45rY13h0i2UcEI641ABEBAAGJAiUEGAECAA8FAl68 kXkCGwwFCQHhM4AACgkQytVB6cybO76hSRAA2v5Wf1C9yJBCr/USYRcgsmbJNWlc gm4agVBorFxBlr3+D3numYNNfm20PenYku/UBgMK5bCr67S2+0xo97RnDcRudbAx sV49/1xVPjjwj5HCY0EFS34oKKoGMp57ll4ewbAdBwgz8j6ld72Mg05ZReLKbtWr 9h12VaO1JrGC+XzCTWJc1pDWB5Q3Sv2UENpjr203jIlE44O7qepfaP+3bnWPMmG7 oQlmuggx/k3nbLPUiCcYLg5yiYBmTQHTUnB4v5L2nT3IEG2DbuEqWbwewIsGaSx9 eGG2gkRtJNpRSS6+sWMFKOW7RR2YgDQoNm5XOwgZhoMYwXs5fD0ul5kFVfVPYYOl DrDIdRjr3L3gSQoWg+3ZlO6Tlwtva8QlDjUdY93cPykMgtv0iovNiJe3Lf25hVJl DMsQNjtmhsHugjT+VuIk9Zw2+SCSKFWcxeUs3p+nmhNlx+NIIE4zJUEu9lOG6uIP U/lGOu/SX53o0eoBsofPfJqILOlweFEvMnL/+wOy534eaPe1d5HhPm37h/8cYFUY a2bV4gCxZjAPyQpLsOI9EOVJJLxF36MIKrZ44OyRxix1DNCQu2S3oG6ljMPHsQ9E zmWOvETxGfgppc8A1smmoBMiGnNnaZVbo4v7TE+9x337yDVUiYYriV3RMPDOVTbv 0/I9kH7m8tCtYKI= =vsDs - -----END PGP PUBLIC KEY BLOCK----- -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iQIcBAEBAgAGBQJfjqDaAAoJEMrVQenMmzu+4UoQAIdQtsvECxjntTLcE5Rt+WCt n+LBO6iJVFW9H8gu8WCAyKXldL27vQ75qZXNx9AuVTES/Uq2pPeJNKTZ/HCOUqCa 0Q8Rmqj935g67/UUEvbfVpOFqMz8gf8kFmnm0YRxqMnjhktwhmHhCqzERINaOqAd 6COst5vzOLbQdOiY4DQ8NM8PwzhIXjoVDQ+XRBfnLcucHH77YS0KmfHxHr7hEKI0 Q2ATF439Dm2VZEb2WwRV4Rsl86n8or/WwLLgTIIUD64Vpm2zywWPdOLBZnIjgMv0 aG++waFO4C3+f+jxKXosD9gQ3+0UgmT+FsQ43qwDBMLgX+fwDKtJ3JiiqmT/967G 8e3L7xuZR+kqd38c7pV61FCtN4Q7nPJMlo/VFIPQYnft1ko2VSaGrrPl3A/IYtTu pfEkSIO5NXSuJVt3YF7hCgE4XW0+GK4rguyeW21/vidM8LUtEEzDjS6Y+DYWCS9g i68bgTypahwwyUJ3Ms8c+0KlYs00/hHL+MOxMY0OQougTpVNnoIOWeQ5kFUWosHd D6V99uNBW1/XwbcrMS66MmiGvk6ti27SDPnXIxw99aDNldLWQ+Man/qSLCY9q2Ux P4EUfC6lKID6a6n0hJeM1tqtK+bdzKNaNgftQLKlb+22GUwu2ZUrwJkfg0wFAVg6 bELc35LhltOieyBjWMYN =bvax -----END PGP SIGNATURE-----
