Additional performance data from an Oracle Cloud Ampere Altra (Neoverse-N1) instance, running Ubuntu 22.04 with the Oracle 6.8 kernel.
I am experiencing severe process-creation overhead on a production ARM64 VM, and have collected profiling data that may be relevant to this bug. **Environment** - Cloud: Oracle Cloud Infrastructure (OCI), VM.Standard.A1.Flex - CPU: 2 vCPUs, ARM Neoverse-N1 - OS: Ubuntu 22.04 LTS, aarch64 - Current kernel: 6.8.0-1061-oracle - Previous kernel: 6.8.0-1062-oracle (rolled back due to severe performance degradation) **1. Performance regression** A benchmark launching 100 instances of `/bin/true` produced: - 6.8.0-1062-oracle: 28.33 seconds - 6.8.0-1061-oracle: 8.24 seconds Rolling back restored application availability, but performance on 1061 remains abnormally slow. Repeating the benchmark on 1061 produced 8.06, 8.51, and 7.49 seconds. **2. Isolating the overhead** For 100 iterations: - Shell builtin (`:`): 0.06 seconds - Fork-only subshell (`( : )`): 7.14 seconds - Fork + exec (`/bin/true`): 7.30 seconds This suggests most of the overhead is related to process creation, page- table management, or process teardown, rather than executable loading. **3. CPU affinity** Repeating the fork-only benchmark with CPU affinity: - CPU 0: 11.68, 11.65, 11.51 seconds (average 11.61 s) - CPU 1: 6.73, 6.70, 7.91 seconds (average 7.11 s) The difference is consistent, although both CPUs are unusually slow. **4. perf profiling** Using `perf record -g -e cpu-clock` and `perf report --no-children`: CPU 0: - ptep_clear_flush: 55.75% - ptep_set_access_flags: 37.11% CPU 1: - ptep_clear_flush: 57.23% - ptep_set_access_flags: 36.44% Thus, approximately 93% of CPU-clock samples are concentrated in these two page-table functions on both CPUs. `perf annotate` places essentially all samples immediately after `dsb ish` barriers in these functions. This is consistent with expensive TLB synchronization, although I recognize that software sampling may attribute stalls to the following instruction rather than the barrier itself. **5. Additional observations** - Kernel configuration contains `CONFIG_ARM64_WORKAROUND_REPEAT_TLBI=y`. - `CONFIG_ARM64_ERRATUM_4118414` was not present in the inspected kernel configuration. - No repeat-TLBI activation message was found in the boot log. - Minor page faults were comparable between CPUs (6,638 vs. 6,438). - No major page faults were recorded. - CPU steal time and disk I/O wait were negligible during earlier measurements. **Questions for the kernel team** 1. Could this be related to the TLB regression addressed by commit `a8f78680ee6b`, even though substantial overhead is already present on 6.8.0-1061-oracle? 2. Is this an indication of a separate ARM64 TLB performance issue affecting the Oracle kernel or OCI Ampere Altra virtualization? 3. Would you prefer that I file a separate bug against the Ubuntu Oracle kernel package for the residual slowdown on 1061? 4. Are there additional targeted diagnostics or kernel configurations that would help distinguish guest-kernel overhead from virtualization-related latency? I can provide the complete perf reports, annotations, and benchmark outputs if useful. I am keeping 6.8.0-1061-oracle pinned pending a verified stable fix, as this is a production instance. Thank you for investigating. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170644 Title: arm64: munmap performance regression To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2170644/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
