** Description changed:

  [ Impact ]
  
   * Since openscap 1.2.17-0.1ubuntu7.22.04.3 (LP: #2116751), Jammy USG
  compliance scans on machines with large directory trees (~100k+ files
- under scanned paths) print thousands of spurious `E: probe_file: Failed
- to check available memory` errors despite ample free memory.
+ under scanned paths) can print thousands of spurious `E: probe_file:
+ Failed to check available memory` errors despite ample free memory.
  
   * Additionally, rules that often collect more than 1000 items, like
  `xccdf_org.ssgproject.content_rule_no_files_unowned_by_user` and
  `xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned` are
  incorrectly reported as failures when the spurious errors begin, leading
  to inaccurate test results.
  
   * Each spurious error leaks the collected item, so `probe_file`'s
  memory usage grows until it hits the memory usage limit introduced in
  LP: #2116751, printing `W: probe_file: Memory usage ratio limit
- reached!` or getting OOMed.
+ reached!` and aborting early or getting OOMed.
  
   * The fix corrects this with two upstream commits:
     - 12f9c02a612bb "Reset errno before call to strtoll": correctly initialize 
errno to 0 in the success case so that a single error doesn't propagate to 
future collected files
     - 83592087efc58 "Fix leak of item in probe_item_collect": Correctly free 
the collected items in the error case, following up on LP: #2116751.
  
  [ Test Plan ]
  
   * The bug is probabilistic, scaling with the number of files scanned,
  so a deterministic reproducer is not feasible. A script that performs
  the steps below is attached to the bug.
  
     0. Set up a clean jammy system
  
     1. Install the current jammy package and content:
          apt-cache policy libopenscap8   # 1.2.17-0.1ubuntu7.22.04.3
          sudo apt install libopenscap8 scap-security-guide
        (alternatively install usg + usg-benchmarks for the usg path)
  
     2. Create a large directory tree:
          sudo bash -c 'mkdir -p /srv/manyfiles && cd /srv/manyfiles && \
                        for i in $(seq 1 200000); do : > f$i; done'
  
     3. Evaluate the two known-affected rules directly:
          sudo oscap xccdf eval --profile '(all)' \
            --rule xccdf_org.ssgproject.content_rule_no_files_unowned_by_user \
            --rule 
xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned \
            --results /tmp/oscap-results.xml \
            /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml \
            2>&1 | tee /tmp/scan.log
        Or run the full USG script:
          sudo usg audit cis_level1_workstation
  
     4a. Failure output (unpatched):
        - /tmp/scan.log contains hundreds of "E: probe_file: Failed to check 
available memory" lines
        - the two rules show "Result fail" (or error) even on systems where 
they should pass per the rule definition
        - observe probe_file memory usage growing drastically or a "W: 
probe_file: Memory usage ratio limit reached!" message
  
     4b. Success output (patched):
        - at most a handful "Failed to check available memory" lines, genuine 
errors are what LP: #2116751 was trying to catch
        - the two rules report their genuine result; i.e. passes on a compliant 
system per the rule definition
        - probe_file memory usage stays reasonable (less than 20+ GiB, no OOMs).
  
  [ Where problems could occur ]
  
   * 12f9c02 (initializing the errno to 0) is strictly a small correctness
  improvement. If the change were wrong, misbehavior would manifest either
  as false "memory limit reached" errors (as is the current case) or
  genuine failures becoming hidden as successes. The change itself has
  been shipped upstream since 1.3.7; Noble and later already contain it.
  
   * 8359208 (freeing the items on the error paths) could result in a
  double free / use-after-free in a probe_* process, printing errors like
  "W: oscap: Can't receive message", "Result error", or a segfault. This
  change is upstream as of 1.3.9, so it is also already present in Noble
  and later.
  
   * Both changes touch a code path that is only active when a single rule
  collects more than 1000 items (the memcheck threshold from LP:
  #2116751), so the majority of rules are unaffected.
  
  [ Other Info ]
  
   * These patches affect /usr/bin/oscap and the probe_* executables,
  which are in the libopenscap8 binary package in Jammy.
  
   * Only Jammy is affected: Noble and later ship openscap 1.3.9+ which
  already contain both commits
  
   * Follow-ups not included in this SRU:
     - 3ce4482466 "Improve counting collected items": replaces 
SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the 
exact number of items doesn't really matter when triggering the memory 
safegaurd. This change is too large to have the same functional impact as just 
initializing the errno to 0 as is done in 12f9c02.
     - b20fca6e0d and f2461c7498: hard cap of 10000 collected items per object 
plus the OSCAP_PROBE_MAX_COLLECTED_ITEMS environment variable. This is a new 
feature, so not backporting.
     - ded3d58cd6: OSCAP_PROBE_MEMORY_USAGE_RATIO environment variable to tune 
the memory guard (discussed in LP: #2116751 comment #14).
  
   * References:
     - LP: #2116751 (SRU that introduced the memory guard and message spam: 
d/p/lp2116751-lower-memory-limits.patch)
     - LP: #2056775 (--debug masks the symptoms)
     - Errno fix: https://github.com/OpenSCAP/openscap/commit/12f9c02a612bb
     - Item leak fix: https://github.com/OpenSCAP/openscap/commit/83592087efc58
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1858
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1867
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/2345

** Description changed:

  [ Impact ]
  
   * Since openscap 1.2.17-0.1ubuntu7.22.04.3 (LP: #2116751), Jammy USG
  compliance scans on machines with large directory trees (~100k+ files
  under scanned paths) can print thousands of spurious `E: probe_file:
  Failed to check available memory` errors despite ample free memory.
  
   * Additionally, rules that often collect more than 1000 items, like
  `xccdf_org.ssgproject.content_rule_no_files_unowned_by_user` and
  `xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned` are
  incorrectly reported as failures when the spurious errors begin, leading
  to inaccurate test results.
  
   * Each spurious error leaks the collected item, so `probe_file`'s
  memory usage grows until it hits the memory usage limit introduced in
  LP: #2116751, printing `W: probe_file: Memory usage ratio limit
  reached!` and aborting early or getting OOMed.
  
   * The fix corrects this with two upstream commits:
     - 12f9c02a612bb "Reset errno before call to strtoll": correctly initialize 
errno to 0 in the success case so that a single error doesn't propagate to 
future collected files
     - 83592087efc58 "Fix leak of item in probe_item_collect": Correctly free 
the collected items in the error case, following up on LP: #2116751.
  
  [ Test Plan ]
  
   * The bug is probabilistic, scaling with the number of files scanned,
- so a deterministic reproducer is not feasible. A script that performs
- the steps below is attached to the bug.
+ so a deterministic reproducer is not feasible. A genuine error is likely
+ to trigger the spurious errors, but introducing a genuine error is also
+ difficult in a test plan. A script that performs the steps below is
+ attached to the bug.
  
     0. Set up a clean jammy system
  
     1. Install the current jammy package and content:
          apt-cache policy libopenscap8   # 1.2.17-0.1ubuntu7.22.04.3
          sudo apt install libopenscap8 scap-security-guide
        (alternatively install usg + usg-benchmarks for the usg path)
  
     2. Create a large directory tree:
          sudo bash -c 'mkdir -p /srv/manyfiles && cd /srv/manyfiles && \
                        for i in $(seq 1 200000); do : > f$i; done'
  
     3. Evaluate the two known-affected rules directly:
          sudo oscap xccdf eval --profile '(all)' \
            --rule xccdf_org.ssgproject.content_rule_no_files_unowned_by_user \
            --rule 
xccdf_org.ssgproject.content_rule_file_permissions_ungroupowned \
            --results /tmp/oscap-results.xml \
            /usr/share/xml/scap/ssg/content/ssg-ubuntu2204-ds.xml \
            2>&1 | tee /tmp/scan.log
        Or run the full USG script:
          sudo usg audit cis_level1_workstation
  
     4a. Failure output (unpatched):
        - /tmp/scan.log contains hundreds of "E: probe_file: Failed to check 
available memory" lines
        - the two rules show "Result fail" (or error) even on systems where 
they should pass per the rule definition
        - observe probe_file memory usage growing drastically or a "W: 
probe_file: Memory usage ratio limit reached!" message
  
     4b. Success output (patched):
-       - at most a handful "Failed to check available memory" lines, genuine 
errors are what LP: #2116751 was trying to catch
+       - at most a handful of "Failed to check available memory" 
lines--genuine errors are what LP: #2116751 was trying to catch
        - the two rules report their genuine result; i.e. passes on a compliant 
system per the rule definition
        - probe_file memory usage stays reasonable (less than 20+ GiB, no OOMs).
  
  [ Where problems could occur ]
  
   * 12f9c02 (initializing the errno to 0) is strictly a small correctness
  improvement. If the change were wrong, misbehavior would manifest either
  as false "memory limit reached" errors (as is the current case) or
  genuine failures becoming hidden as successes. The change itself has
  been shipped upstream since 1.3.7; Noble and later already contain it.
  
   * 8359208 (freeing the items on the error paths) could result in a
  double free / use-after-free in a probe_* process, printing errors like
  "W: oscap: Can't receive message", "Result error", or a segfault. This
  change is upstream as of 1.3.9, so it is also already present in Noble
  and later.
  
   * Both changes touch a code path that is only active when a single rule
  collects more than 1000 items (the memcheck threshold from LP:
  #2116751), so the majority of rules are unaffected.
  
  [ Other Info ]
  
   * These patches affect /usr/bin/oscap and the probe_* executables,
  which are in the libopenscap8 binary package in Jammy.
  
   * Only Jammy is affected: Noble and later ship openscap 1.3.9+ which
  already contain both commits
  
   * Follow-ups not included in this SRU:
-    - 3ce4482466 "Improve counting collected items": replaces 
SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the 
exact number of items doesn't really matter when triggering the memory 
safegaurd. This change is too large to have the same functional impact as just 
initializing the errno to 0 as is done in 12f9c02.
+    - 3ce4482466 "Improve counting collected items": replaces 
SEXP_list_length() with an explicit counter in probe_ctx. Not necessary, as the 
exact number of items doesn't really matter when triggering the memory 
safegaurd. This change is too large to backport when it has the same functional 
impact as just initializing the errno to 0 as is done in 12f9c02.
     - b20fca6e0d and f2461c7498: hard cap of 10000 collected items per object 
plus the OSCAP_PROBE_MAX_COLLECTED_ITEMS environment variable. This is a new 
feature, so not backporting.
     - ded3d58cd6: OSCAP_PROBE_MEMORY_USAGE_RATIO environment variable to tune 
the memory guard (discussed in LP: #2116751 comment #14).
  
   * References:
     - LP: #2116751 (SRU that introduced the memory guard and message spam: 
d/p/lp2116751-lower-memory-limits.patch)
     - LP: #2056775 (--debug masks the symptoms)
     - Errno fix: https://github.com/OpenSCAP/openscap/commit/12f9c02a612bb
     - Item leak fix: https://github.com/OpenSCAP/openscap/commit/83592087efc58
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1858
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/1867
     - Upstream bug report: https://github.com/OpenSCAP/openscap/issues/2345

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2175424

Title:
  openscap 1.2.17-0.1ubuntu7.22.04.3:  spurious rule failures on large
  filesystems

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openscap/+bug/2175424/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to