Hi all, Over the past few weeks, I have been systematically investigating Gerrit CI false negatives: workflows that failed and later passed on the same patch set without any code change.
During this period, several CI and HST fixes from multiple contributors were merged. An updated analysis performed on September 14, covering the two complete weeks of August 23–29 and September 7–13, found that the confirmed false-negative rate fell from 32/253 conclusive gerrit-verify workflows (12.6%) to 9/209 (4.3%), an approximately 66% reduction. The latest figure remains provisional because some recent failures may not yet have received an unchanged recheck. At signature level, the tracked QUIC, stale QEMU namespace, VCL HTTP POST and exact worker-barrier failures were no longer observed on patch sets containing the corresponding fixes. The following patches remain open and are currently Verified +1. The occurrence counts are not additive, as some fixes address different stages of the same failure cascade: - 46634 — preserve timeout diagnostics across retries. Diagnostic-only; no prevented failure claimed. Code-Review +1. https://gerrit.fd.io/r/c/vpp/+/46634 - 46636 — avoid repeated packet-generator barriers. CaptureTimeoutError appeared in 39 of the initial 65 confirmed false-negative runs, although it was often a secondary symptom and cannot always be attributed to this mechanism. Code-Review +1. https://gerrit.fd.io/r/c/vpp/+/46636 - 46640 — tolerate delayed BFD observations. One complete false-negative workflow is demonstrably covered. https://gerrit.fd.io/r/c/vpp/+/46640 - 46651 — keep deleted interface node names unique. The same Stats abort was observed in two confirmed CI occurrences. https://gerrit.fd.io/r/c/vpp/+/46651 - 46657 — tolerate unchanged translated NAT ports. One confirmed CI workflow failed on this false assertion in all three attempts. Code-Review +1. https://gerrit.fd.io/r/c/vpp/+/46657 - 46719 and 46720 — avoid iperf pre-exec deadlocks and make the QEMU network lock recoverable. Together, they address four confirmed red CI jobs involving 89 timed-out suites. https://gerrit.fd.io/r/c/vpp/+/46719 https://gerrit.fd.io/r/c/vpp/+/46720 - 46723 — wake HSI drains when ownership changes. One confirmed false-negative run contained the exact HSI signature addressed. https://gerrit.fd.io/r/c/vpp/+/46723 - 46780 — extend the BFD peer-detection window. The targeted timeout cascade appeared in six of seven BFD-red Maketest jobs, across five Gerrit changes. https://gerrit.fd.io/r/c/vpp/+/46780 Using only cases directly attributable within the initial 65-run sample, the pending patches cover at least five confirmed false-negative runs, or 7.7%. The additional Stats, NAT and newer BFD occurrences are reported separately because they fall outside that sample or cannot be added without double counting. All nine pending changes are Verified +1, have no unresolved comments, and three already have Code-Review +1. Reviews and merges from the maintainers of the affected components would now be greatly appreciated. Best regards, Jerome
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#27181): https://lists.fd.io/g/vpp-dev/message/27181 Mute This Topic: https://lists.fd.io/mt/121242935/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
