If there is a way to determine: vCPU thread utilization numbers over a period of time, such as a few hours
or which processes are consuming the most CPU top always indicates that the server is consuming the most CPU. Now i am begining to wonder if 8 vCPU threads really are capable of running 6 high intensity threads or only 4 such threads? Dont know Also tried to utilize pthread_setschedparam() explicitly on some of the threads, it made no difference to the performance. But if we do it on more than 1-2 threads then it hangs the whole system. This is primarily a matter of CPU scheduling, and if we restirct context switching on even 2 critical threads we have a win. regards On Thursday, September 26, 2024 at 09:56:04 AM PDT, amit sehas <cu...@yahoo.com> wrote: Belos is the lscpu that was requested, it appears to suggest an 8 vCPU thread setup ... if am reading it correctly: $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz BIOS Model name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 Stepping: 7 BogoMIPS: 4999.99 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cl flush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc re p_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclm ulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_t imer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpci d_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx5 12f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xs aveopt xsavec xgetbv1 xsaves ida arat pku ospke Virtualization features: Hypervisor vendor: KVM Virtualization type: full Caches (sum of all): L1d: 128 KiB (4 instances) L1i: 128 KiB (4 instances) L2: 4 MiB (4 instances) L3: 35.8 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-7 Vulnerabilities: Gather data sampling: Unknown: Dependent on hypervisor status Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Mitigation; PTE Inversion Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unkn own Meltdown: Mitigation; PTI Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unkn own Reg file data sampling: Not affected Retbleed: Vulnerable Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affec ted; BHI Retpoline Srbds: Not affected Tsx async abort: Not affected On Thursday, September 26, 2024 at 05:32:52 AM PDT, amit sehas <cu...@yahoo.com> wrote: Simply reordering the launch of different threads brings back a lot of the lost performance, this is a clear evidence that some CPU threads are more predisposed to context switches than the others. This is a thread scheduling issue at the CPU level as we have expected. In a previous exchange someone has suggested that utilizing rte_thread_set_priority to RTE_THREAD_PRIORITY_REALTIME_CRITICAL is not a good idea we should be able to prioritize some threads over the other threads ... since we are utilizing rte_eal_remote_launch, one would think that such a functonality should be a part of the library ... any ideas folks? regards On Tuesday, September 24, 2024 at 01:47:05 PM PDT, amit sehas <cu...@yahoo.com> wrote: Thanks for the suggestions, so this is a database server which is doing lots of stuff, not every thread is heavily involved in dpdk packet processing. As a result the guidelines for attaining the most dpdk performance are applicable to only a few threads. In this particular issue we are specificially looking at CPU scheduling of threads that are primarily heavily processing database queries. These threads, from our measurements, are not being uniformly scheduled on the CPU ... This is our primary concern, since we utilized rte_eal_remote_launch to spawn the threads, we are wondering if there are any options in this API that will allow us to more uniformly allocate the CPU to threads that are critical... regards On Tuesday, September 24, 2024 at 09:38:16 AM PDT, Stephen Hemminger <step...@networkplumber.org> wrote: On Tue, 24 Sep 2024 14:40:49 +0000 (UTC) amit sehas <cu...@yahoo.com> wrote: > Thanks for your response, and thanks for your input on the set_priority, > > The best guess we have at this point is that this is not a dpdk performance > issue. This is an issue with some threads running into more context switches > than the others and hence not getting the same slice of the CPU. We are > certain that this is not a dpdk performance issue, the code > is uniformly slow in one thread versus the other and the threads are doing a > very large amount of work including accessing databases. The threads in > question are not really doing packet processing as much as other work. > > So this is certainly not a dpdk performance issue. This is an issue of kernel > threads not being scheduled properly or in the worse case the cores running > on different frequency (which is quite unlikely no the AWS Xeons we are > running this on). > > If you are asking for the dpdk config files to check for dpdk related > performance issue then we are quite certain the issue is not with dpdk > performance ... > On Mon, Sep 23, 2024 at 10:06 PM amit sehas <cu...@yahoo.com> wrote: > > Thanks for your response, i am not sure i understand your question ... we > > have our product that utilizes dpdk ... the commands are just our server > > commands and parameters ... and the lscpu is the hyperthreaded 8 thread > > Xeon instance in AWS ... The rules of getting performance in DPDK: - use DPDK threads (pinned) for datapath - use isolated CPU's for those DPDK threads - do not do any system calls - avoid floating point You can use tracing tools like strace or BPF to see what the thread is doing.
