[
https://issues.apache.org/jira/browse/SOLR-18609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18609:
----------------------------------
Labels: pull-request-available (was: )
> Packaging BATS tests accept failures they should catch: harden the assertions
> -----------------------------------------------------------------------------
>
> Key: SOLR-18609
> URL: https://issues.apache.org/jira/browse/SOLR-18609
> Project: Solr
> Issue Type: Test
> Reporter: Nick Shanin
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A review of the packaging BATS suites ({{solr/packaging/test}}) found
> assertions that pass when the operation they test has failed. The suites stay
> green while the behavior they guard is broken.
> h3. What the review found
> * Post tests that never verified the posted documents arrived, including an
> authenticated post aimed at the wrong collection name.
> * A log batch whose sends failed could still pass after an empty commit; the
> check did not require any indexed records.
> * Failed ZooKeeper reads were accepted as proof that a child node was
> deleted (seven negative existence checks).
> * Degraded collection health was accepted: shard statuses were read without
> requiring command success and the collection-level healthy status.
> * A failed stop was accepted as a graceful stop, and a stale export output
> file could pass as a fresh result.
> * TLS failure checks matched so generically that a plain connection refusal
> satisfied them; a failed curl was accepted as a compression check.
> h3. Control evidence
> With an HTTP 503 injected into a log batch add, the current authenticated
> post test passes 1/1 while the follow-up probes show zero documents indexed;
> the hardened test rejects the same injection. The hardened assertions require
> command success, the specific failure diagnostic where one exists, and a
> positive check that the expected data or state is present.
> h3. Prepared branch
> Branch:
> [solr-bats-assertion-hardening-submit|https://github.com/nick-boss-tech/solr/tree/solr-bats-assertion-hardening-submit]
> at {{4d5d876d946a4f5ac174a7f68e16f4dd9b5f9f68}}, 14 files under
> {{solr/packaging/test}} (13 existing files and the new
> {{test_bats_assertions.bats}} with nine helper regression cases). Test files
> only; no production, launcher, or build changes.
> Diff: [base to
> branch|https://github.com/nick-boss-tech/solr/compare/0d6b2e32f598d772aed094038022a6a588fdfc5a...4d5d876d946a4f5ac174a7f68e16f4dd9b5f9f68]
> This generalizes the one-line fix merged under SOLR-18565 (PR #5193) and the
> assertion direction of the author's own BATS work; it is filed separately
> because its scope is the whole suite, not the single test_ssl case.
> h3. Verification and limits
> Proven on Linux: the Picocli-selected cases pass 18/18 on JDK 21 and on JDK
> 25; the batch-failure controls above pass (old test accepts the injection,
> hardened test rejects it); tidy is clean and {{:solr:packaging:check -x
> test}} passes.
> Not yet proven, stated plainly: the legacy TLS legs show connection-refused
> failures in the sandbox they ran in, the client-hostname policy control needs
> a host with more tmpfs headroom than any host available here, and
> {{test_rolling_upgrade.bats}} needs Docker, which is not available here. An
> upstream CI run is the natural place to settle those three.
> 🤖 _AI text above_ 🤖 _(reviewed by Nick Shanin)_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
