Additional performance data from an Oracle Cloud Ampere Altra
(Neoverse-N1) instance, running Ubuntu 22.04 with the Oracle 6.8 kernel.

I am experiencing severe process-creation overhead on a production ARM64
VM, and have collected profiling data that may be relevant to this bug.

**Environment**

- Cloud: Oracle Cloud Infrastructure (OCI), VM.Standard.A1.Flex
- CPU: 2 vCPUs, ARM Neoverse-N1
- OS: Ubuntu 22.04 LTS, aarch64
- Current kernel: 6.8.0-1061-oracle
- Previous kernel: 6.8.0-1062-oracle (rolled back due to severe performance 
degradation)

**1. Performance regression**

A benchmark launching 100 instances of `/bin/true` produced:

- 6.8.0-1062-oracle: 28.33 seconds
- 6.8.0-1061-oracle: 8.24 seconds

Rolling back restored application availability, but performance on 1061
remains abnormally slow.

Repeating the benchmark on 1061 produced 8.06, 8.51, and 7.49 seconds.

**2. Isolating the overhead**

For 100 iterations:

- Shell builtin (`:`): 0.06 seconds
- Fork-only subshell (`( : )`): 7.14 seconds
- Fork + exec (`/bin/true`): 7.30 seconds

This suggests most of the overhead is related to process creation, page-
table management, or process teardown, rather than executable loading.

**3. CPU affinity**

Repeating the fork-only benchmark with CPU affinity:

- CPU 0: 11.68, 11.65, 11.51 seconds (average 11.61 s)
- CPU 1: 6.73, 6.70, 7.91 seconds (average 7.11 s)

The difference is consistent, although both CPUs are unusually slow.

**4. perf profiling**

Using `perf record -g -e cpu-clock` and `perf report --no-children`:

CPU 0:
- ptep_clear_flush: 55.75%
- ptep_set_access_flags: 37.11%

CPU 1:
- ptep_clear_flush: 57.23%
- ptep_set_access_flags: 36.44%

Thus, approximately 93% of CPU-clock samples are concentrated in these
two page-table functions on both CPUs.

`perf annotate` places essentially all samples immediately after `dsb
ish` barriers in these functions. This is consistent with expensive TLB
synchronization, although I recognize that software sampling may
attribute stalls to the following instruction rather than the barrier
itself.

**5. Additional observations**

- Kernel configuration contains `CONFIG_ARM64_WORKAROUND_REPEAT_TLBI=y`.
- `CONFIG_ARM64_ERRATUM_4118414` was not present in the inspected kernel 
configuration.
- No repeat-TLBI activation message was found in the boot log.
- Minor page faults were comparable between CPUs (6,638 vs. 6,438).
- No major page faults were recorded.
- CPU steal time and disk I/O wait were negligible during earlier measurements.

**Questions for the kernel team**

1. Could this be related to the TLB regression addressed by commit 
`a8f78680ee6b`, even though substantial overhead is already present on 
6.8.0-1061-oracle?
2. Is this an indication of a separate ARM64 TLB performance issue affecting 
the Oracle kernel or OCI Ampere Altra virtualization?
3. Would you prefer that I file a separate bug against the Ubuntu Oracle kernel 
package for the residual slowdown on 1061?
4. Are there additional targeted diagnostics or kernel configurations that 
would help distinguish guest-kernel overhead from virtualization-related 
latency?

I can provide the complete perf reports, annotations, and benchmark
outputs if useful.

I am keeping 6.8.0-1061-oracle pinned pending a verified stable fix, as
this is a production instance.

Thank you for investigating.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2170644

Title:
  arm64: munmap performance regression

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2170644/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to