roaldm153 opened a new issue, #1888:
URL: https://github.com/apache/cloudberry/issues/1888
### Apache Cloudberry version
2.1.0
### What happened
The regression test contrib/pax_storage/sql/pax_inject_fault.sql is flaky
and fails non-deterministically.
The test deliberately injects a panic fault (orc_writer_write_tuple) and
then executes an INSERT statement to trigger it. Because the fault type is
panic, it crashes the segment postmaster. When the segment crashes, it
asynchronously outputs system crash notices (DETAIL, HINT, WARNING).
The .out expected file only expects a single line: ERROR: fault triggered
.... Due to timing variations (race condition), the QD sometimes receives only
the ERROR message (test passes), and sometimes it receives the ERROR along with
the crash notices (test fails with a diff).
Here is the exact diff of the failure:
```
insert into t_insert select generate_series(1,10);
+DETAIL: The postmaster has commanded this server process to roll back the
current
+ transaction and exit, because another server process exited
abnormally
+ and possibly corrupted shared memory.
ERROR: fault triggered, fault name:'orc_writer_write_tuple' fault
type:'panic'
+HINT: In a moment you should be able to reconnect to the database and
repeat your command.
+WARNING: terminating connection because of crash of another server process
```
In CI, it often fails non-deterministically in profiles like
pax-ic-deb-good-opt-off. Furthermore, this crash can cause a cascading failure
where the immediately following test (e.g., transactions) fails because it
catches the aftermath of the segment crash.
### What you think should happen instead
resulting crash messages should be deterministic
### How to reproduce
Run the pax_storage regression tests, specifically the pax_inject_fault test.
Because this is a timing-related race condition, you may need to run the
test in a loop or under specific load/CI configurations to catch the moment
when the segment manages to send the DETAIL/HINT/WARNING messages before the QD
finishes reading the response.
### Operating System
rockylinux9
### Anything else
on testing
### Are you willing to submit PR?
- [x] Yes, I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/cloudberry/blob/main/CODE_OF_CONDUCT.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
