Re: [go-nuts] Re: HTTP client timeout
On Fri, May 5, 2023 at 4:21 PM envee wrote: > > Thanks Jan. > My interface speed is 20 Gb/s > The payload size is about 2.8Kb. > So at 9000 Txn/s this works out to 9000 x 3 ≅27MB/s ≅ 200 Mb/s (This matches > the so (socket out) value shown in my atop command output) > I guess the interface speed is sufficient ? Yes it is. Next question: what's the speed of the slowest segment en route? ;-) -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-Xw3bTGhhg0DZem%2BHMg433%2Bm4wvrmNk6AuaGZXasYhwLA%40mail.gmail.com.
Re: [go-nuts] Re: HTTP client timeout
I mean you can use curl to simulate the 1000’s of clients (depending on protocol dependencies) > On May 5, 2023, at 9:23 AM, envee wrote: > > Hi Robert, Yes I have tried HTTP request via curl. That yields a response in > about 200ms. > >> On Friday, 5 May 2023 at 22:56:58 UTC+10 Robert Engels wrote: >> Have you tried using curl as the client in order to narrow down the problem? >> On May 5, 2023, at 7:45 AM, envee wrote: >>> An update. >> >>> The VM on which I am running has 40 vCPUs. >>> I changed my GOMAXPROCS to a lower value of 8. >>> And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 >>> server. >>> With this setup, I can now achieve about 9000 Txns per second. >>> However, if I go any higher than that, my client starts throwing the >>> "context deadline exceeded" errors. I am sure that the responses are >>> received by the client well within the timeout of 7 seconds. >>> When I observe the CPU utilization of my client process, it only uses about >>> 300% vCPU i.e. 3 out of 40 vCPUs. >>> Still, I don't understand why I cannot achieve higher throughput than 9000 >>> per second. >>> Here is a sample output from the atop utility. As seen below, only 257% >>> shows as CPU utilization. >>> >>> Is this because goroutines are not getting context switched onto a logical >>> processor or thread. I thought if there is so much CPU available, then >>> context switching should be very fast and not an issue at all. >>> Thus leading to timeouts ? >>> >>> ATOP - myserver 2023/05/05 22:16:19 >>>yP >>>10s elapsed >>> PRC | sys6.63s | user 21.56s | | #proc543 | #trun >>> 4 | #tslpi 639 | | #tslpu 0 | #zombie0 | clones >>> 11 | | no procacct | >>> CPU | sys 61% | user213% | irq 4% | idle 3724% | wait >>> 0% | steal 0% | guest 0% | | ipc notavail | cycl >>> unknown | curf 3.00GHz | curscal ?% | >>> CPL | avg12.28 | avg52.40 | | avg15 2.45 | >>>| | csw 656705 | intr 515996 | | >>>| numcpu40 | | >>> MEM | tot 503.7G | free 468.6G | cache 26.9G | dirty 0.1M | buff >>> 1.0G | slab1.4G | slrec 1.2G | shmem 4.1G | shrss 117.0M | vmbal >>> 0.0M | hptot 0.0M | hpuse 0.0M | >>> SWP | tot 4.0G | free4.0G | | | >>>| | | | | >>>| vmcom 6.0G | vmlim 255.8G | >>> LVM | g_vd0-lv_var | busy 0% | read 0 | | write >>> 158 | KiB/r 0 | KiB/w 4 | MBr/s0.0 | MBw/s0.1 | >>> | avq 5.00 | avio 0.01 ms | >>> DSK | sda | busy 0% | read 0 | | write >>> 122 | KiB/r 0 | KiB/w 5 | MBr/s0.0 | MBw/s0.1 | >>> | avq 2.00 | avio 0.01 ms | >>> NET | transport| tcpi 181070 | tcpo 289884 | udpi 18 | udpo >>> 18 | tcpao 0 | tcppo 26 | tcprs556 | tcpie 0 | tcpor >>> 1 | udpnp 0 | udpie 0 | >>> NET | network | ipi 181092 | ipo 201839 | | ipfrw >>> 0 | deliv 181091 | | | | >>>| icmpi 0 | icmpo 0 | >>> NET | bond1 0% | pcki 181057 | pcko 290688 | sp 20 Gbps | si 16 >>> Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro >>>0 | drpi 0 | drpo 0 | >>> NET | net2 0% | pcki 181056 | pcko 290688 | sp 20 Gbps | si 16 >>> Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro >>>0 | drpi 0 | drpo 0 | >>> >>> PID TID SYSCPUUSRCPUVGROWRGROW >>> RUID EUID STEXCTHR S >>> CPUNRCPU CMD 1/1 >>> 40753 -4.08s21.52s 0K -11.7M >>> mtxmtx-- - 15 S >>> 20 257% 5gclient >>> 40753 407530.00s 0.00s 0K -11.7M >>> mtxmtx-- - 1 S >>> 20 0% 5gclient >>> 40753 407540.00s 0.00s 0K -11.7M >>> mtxmtx-- - 1 S >>> 5 0% 5gclient >>> 40753 407550.00s 0.00s 0K -11.7M >>> mtxmtx-- - 1 S >>> 3 0% 5gclient >>> 40753 407560.00s 0.00s 0K
Re: [go-nuts] Re: HTTP client timeout
Hi Robert, Yes I have tried HTTP request via curl. That yields a response in about 200ms. On Friday, 5 May 2023 at 22:56:58 UTC+10 Robert Engels wrote: > Have you tried using curl as the client in order to narrow down the > problem? > > On May 5, 2023, at 7:45 AM, envee wrote: > > An update. > > The VM on which I am running has 40 vCPUs. > I changed my GOMAXPROCS to a lower value of 8. > And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 > server. > With this setup, I can now achieve about 9000 Txns per second. > However, if I go any higher than that, my client starts throwing the > "context deadline exceeded" errors. I am sure that the responses are > received by the client well within the timeout of 7 seconds. > When I observe the CPU utilization of my client process, it only uses > about 300% vCPU i.e. 3 out of 40 vCPUs. > Still, I don't understand why I cannot achieve higher throughput than 9000 > per second. > Here is a sample output from the *atop* utility. As seen below, only 257% > shows as CPU utilization. > > Is this because goroutines are not getting context switched onto a logical > processor or thread. I thought if there is so much CPU available, then > context switching should be very fast and not an issue at all. > Thus leading to timeouts ? > > ATOP - myserver 2023/05/05 22:16:19 > yP > 10s elapsed > PRC | sys6.63s | user 21.56s | | #proc543 | #trun >4 | #tslpi 639 | | #tslpu 0 | #zombie0 | clones >11 | | no procacct | > CPU | sys 61% | user213% | irq 4% | idle 3724% | wait > 0% | steal 0% | guest 0% | | ipc notavail | cycl > unknown | curf 3.00GHz | curscal ?% | > CPL | avg12.28 | avg52.40 | | avg15 2.45 | > | | csw 656705 | intr 515996 | | > | numcpu40 | | > MEM | tot 503.7G | free 468.6G | cache 26.9G | dirty 0.1M | buff > 1.0G | slab1.4G | slrec 1.2G | shmem 4.1G | shrss 117.0M | vmbal > 0.0M | hptot 0.0M | hpuse 0.0M | > SWP | tot 4.0G | free4.0G | | | > | | | | | > | vmcom 6.0G | vmlim 255.8G | > LVM | g_vd0-lv_var | busy 0% | read 0 | | write > 158 | KiB/r 0 | KiB/w 4 | MBr/s0.0 | MBw/s0.1 | > | avq 5.00 | avio 0.01 ms | > DSK | sda | busy 0% | read 0 | | write > 122 | KiB/r 0 | KiB/w 5 | MBr/s0.0 | MBw/s0.1 | > | avq 2.00 | avio 0.01 ms | > NET | transport| tcpi 181070 | tcpo 289884 | udpi 18 | udpo > 18 | tcpao 0 | tcppo 26 | tcprs556 | tcpie 0 | tcpor >1 | udpnp 0 | udpie 0 | > NET | network | ipi 181092 | ipo 201839 | | ipfrw >0 | deliv 181091 | | | | > | icmpi 0 | icmpo 0 | > NET | bond1 0% | pcki 181057 | pcko 290688 | sp 20 Gbps | si 16 > Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro > 0 | drpi 0 | drpo 0 | > NET | net2 0% | pcki 181056 | pcko 290688 | sp 20 Gbps | si 16 > Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro > 0 | drpi 0 | drpo 0 | > > PID TID SYSCPUUSRCPUVGROWRGROW >RUID EUID STEXCTHR S > CPUNRCPU CMD 1/1 > 40753 -4.08s21.52s 0K -11.7M >mtxmtx-- - 15 S > 20 *257% 5gclient* > 40753 407530.00s 0.00s 0K -11.7M >mtxmtx-- - 1 S > 20 0% 5gclient > 40753 407540.00s 0.00s 0K -11.7M >mtxmtx-- - 1 S > 5 0% 5gclient > 40753 407550.00s 0.00s 0K -11.7M >mtxmtx-- - 1 S > 3 0% 5gclient > 40753 407560.00s 0.00s 0K -11.7M >mtxmtx-- - 1 S > 21 0% 5gclient > 40753 407570.45s 2.35s 0K -11.7M >mtxmtx-- - 1 S > 628%
Re: [go-nuts] Re: HTTP client timeout
Thanks Jan. My interface speed is 20 Gb/s The payload size is about 2.8Kb. So at 9000 Txn/s this works out to 9000 x 3 ≅27MB/s ≅ 200 Mb/s (This matches the so (socket out) value shown in my atop command output) I guess the interface speed is sufficient ? On Friday, 5 May 2023 at 23:01:22 UTC+10 Jan Mercl wrote: > On Fri, May 5, 2023 at 2:45 PM envee wrote: > > > Still, I don't understand why I cannot achieve higher throughput than > 9000 per second. > > What is the max bitrate of the network interface used in the > measurement and what is the size of the payload that has to get > through for every connection? > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/352cdcdc-7a0f-43f4-82da-4cf6d8e4fba0n%40googlegroups.com.
Re: [go-nuts] Re: HTTP client timeout
On Fri, May 5, 2023 at 2:45 PM envee wrote: > Still, I don't understand why I cannot achieve higher throughput than 9000 > per second. What is the max bitrate of the network interface used in the measurement and what is the size of the payload that has to get through for every connection? -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-Wavm9vJ6WBQ%3DtAgRwiquDgXMaMj%2B17BLT6%3DF32Jh7Tbw%40mail.gmail.com.
Re: [go-nuts] Re: HTTP client timeout
Have you tried using curl as the client in order to narrow down the problem? > On May 5, 2023, at 7:45 AM, envee wrote: > > An update. > The VM on which I am running has 40 vCPUs. > I changed my GOMAXPROCS to a lower value of 8. > And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 > server. > With this setup, I can now achieve about 9000 Txns per second. > However, if I go any higher than that, my client starts throwing the "context > deadline exceeded" errors. I am sure that the responses are received by the > client well within the timeout of 7 seconds. > When I observe the CPU utilization of my client process, it only uses about > 300% vCPU i.e. 3 out of 40 vCPUs. > Still, I don't understand why I cannot achieve higher throughput than 9000 > per second. > Here is a sample output from the atop utility. As seen below, only 257% shows > as CPU utilization. > > Is this because goroutines are not getting context switched onto a logical > processor or thread. I thought if there is so much CPU available, then > context switching should be very fast and not an issue at all. > Thus leading to timeouts ? > > ATOP - myserver 2023/05/05 22:16:19 > yP >10s elapsed > PRC | sys6.63s | user 21.56s | | #proc543 | #trun > 4 | #tslpi 639 | | #tslpu 0 | #zombie0 | clones11 > | | no procacct | > CPU | sys 61% | user213% | irq 4% | idle 3724% | wait > 0% | steal 0% | guest 0% | | ipc notavail | cycl > unknown | curf 3.00GHz | curscal ?% | > CPL | avg12.28 | avg52.40 | | avg15 2.45 | > | | csw 656705 | intr 515996 | | > | numcpu40 | | > MEM | tot 503.7G | free 468.6G | cache 26.9G | dirty 0.1M | buff > 1.0G | slab1.4G | slrec 1.2G | shmem 4.1G | shrss 117.0M | vmbal > 0.0M | hptot 0.0M | hpuse 0.0M | > SWP | tot 4.0G | free4.0G | | | > | | | | | > | vmcom 6.0G | vmlim 255.8G | > LVM | g_vd0-lv_var | busy 0% | read 0 | | write > 158 | KiB/r 0 | KiB/w 4 | MBr/s0.0 | MBw/s0.1 | > | avq 5.00 | avio 0.01 ms | > DSK | sda | busy 0% | read 0 | | write > 122 | KiB/r 0 | KiB/w 5 | MBr/s0.0 | MBw/s0.1 | > | avq 2.00 | avio 0.01 ms | > NET | transport| tcpi 181070 | tcpo 289884 | udpi 18 | udpo > 18 | tcpao 0 | tcppo 26 | tcprs556 | tcpie 0 | tcpor > 1 | udpnp 0 | udpie 0 | > NET | network | ipi 181092 | ipo 201839 | | ipfrw > 0 | deliv 181091 | | | | > | icmpi 0 | icmpo 0 | > NET | bond1 0% | pcki 181057 | pcko 290688 | sp 20 Gbps | si 16 > Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro > 0 | drpi 0 | drpo 0 | > NET | net2 0% | pcki 181056 | pcko 290688 | sp 20 Gbps | si 16 > Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro > 0 | drpi 0 | drpo 0 | > > PID TID SYSCPUUSRCPUVGROWRGROW > RUID EUID STEXCTHR SCPUNR > CPU CMD 1/1 > 40753 -4.08s21.52s 0K -11.7M > mtxmtx-- - 15 S 20 > 257% 5gclient > 40753 407530.00s 0.00s 0K -11.7M > mtxmtx-- - 1 S 20 > 0% 5gclient > 40753 407540.00s 0.00s 0K -11.7M > mtxmtx-- - 1 S5 > 0% 5gclient > 40753 407550.00s 0.00s 0K -11.7M > mtxmtx-- - 1 S3 > 0% 5gclient > 40753 407560.00s 0.00s 0K -11.7M > mtxmtx-- - 1 S 21 > 0% 5gclient > 40753 407570.45s 2.35s 0K -11.7M > mtxmtx-- - 1 S6 > 28% 5gclient > 40753 407580.44s 2.31s 0K -11.7M > mtxmtx-- - 1
[go-nuts] Re: HTTP client timeout
An update. The VM on which I am running has 40 vCPUs. I changed my GOMAXPROCS to a lower value of 8. And also changed my client to use 90 HTTP/2 connections towards the HTTP/2 server. With this setup, I can now achieve about 9000 Txns per second. However, if I go any higher than that, my client starts throwing the "context deadline exceeded" errors. I am sure that the responses are received by the client well within the timeout of 7 seconds. When I observe the CPU utilization of my client process, it only uses about 300% vCPU i.e. 3 out of 40 vCPUs. Still, I don't understand why I cannot achieve higher throughput than 9000 per second. Here is a sample output from the *atop* utility. As seen below, only 257% shows as CPU utilization. Is this because goroutines are not getting context switched onto a logical processor or thread. I thought if there is so much CPU available, then context switching should be very fast and not an issue at all. Thus leading to timeouts ? ATOP - myserver 2023/05/05 22:16:19 yP 10s elapsed PRC | sys6.63s | user 21.56s | | #proc543 | #trun 4 | #tslpi 639 | | #tslpu 0 | #zombie0 | clones 11 | | no procacct | CPU | sys 61% | user213% | irq 4% | idle 3724% | wait 0% | steal 0% | guest 0% | | ipc notavail | cycl unknown | curf 3.00GHz | curscal ?% | CPL | avg12.28 | avg52.40 | | avg15 2.45 | | | csw 656705 | intr 515996 | | | numcpu40 | | MEM | tot 503.7G | free 468.6G | cache 26.9G | dirty 0.1M | buff 1.0G | slab1.4G | slrec 1.2G | shmem 4.1G | shrss 117.0M | vmbal 0.0M | hptot 0.0M | hpuse 0.0M | SWP | tot 4.0G | free4.0G | | | | | | | | | vmcom 6.0G | vmlim 255.8G | LVM | g_vd0-lv_var | busy 0% | read 0 | | write 158 | KiB/r 0 | KiB/w 4 | MBr/s0.0 | MBw/s0.1 | | avq 5.00 | avio 0.01 ms | DSK | sda | busy 0% | read 0 | | write 122 | KiB/r 0 | KiB/w 5 | MBr/s0.0 | MBw/s0.1 | | avq 2.00 | avio 0.01 ms | NET | transport| tcpi 181070 | tcpo 289884 | udpi 18 | udpo 18 | tcpao 0 | tcppo 26 | tcprs556 | tcpie 0 | tcpor 1 | udpnp 0 | udpie 0 | NET | network | ipi 181092 | ipo 201839 | | ipfrw 0 | deliv 181091 | | | | | icmpi 0 | icmpo 0 | NET | bond1 0% | pcki 181057 | pcko 290688 | sp 20 Gbps | si 16 Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 | NET | net2 0% | pcki 181056 | pcko 290688 | sp 20 Gbps | si 16 Mbps | so 190 Mbps | coll 0 | mlti 0 | erri 0 | erro 0 | drpi 0 | drpo 0 | PID TID SYSCPUUSRCPUVGROWRGROW RUID EUID STEXCTHR S CPUNRCPU CMD 1/1 40753 -4.08s21.52s 0K -11.7M mtxmtx-- - 15 S 20 *257% 5gclient* 40753 407530.00s 0.00s 0K -11.7M mtxmtx-- - 1 S 20 0% 5gclient 40753 407540.00s 0.00s 0K -11.7M mtxmtx-- - 1 S 5 0% 5gclient 40753 407550.00s 0.00s 0K -11.7M mtxmtx-- - 1 S 3 0% 5gclient 40753 407560.00s 0.00s 0K -11.7M mtxmtx-- - 1 S 21 0% 5gclient 40753 407570.45s 2.35s 0K -11.7M mtxmtx-- - 1 S 628% 5gclient 40753 407580.44s 2.31s 0K -11.7M mtxmtx-- - 1 S 2028% 5gclient 40753 407590.44s 2.52s 0K -11.7M mtxmtx-- - 1 S 030% 5gclient 40753 407600.36s
