On Thursday, 9 February 2017 at 17:41:09 UTC, Jack Stouffer wrote:
On Wednesday, 8 February 2017 at 21:05:45 UTC, Jack Stouffer
wrote:
...
Still can't find the root cause. I'm also unable to recreate
the problem locally using the same commands as the doc builder.
We currently have nine PRs in the pipe ready to be merged once
this error is nailed down. If anyone could lend a hand here, it
would be very helpful.
Apologies for that. I made the documentation tester mandatory a
while ago, so extended downtime like this is unacceptable.
In the interest of public disclosure, here is the timeline and
problems encountered:
- In response to some complaints about forum performance, I
investigated sources of high I/O on the server, and identified
the documentation tester as a major culprit. On 2017-02-06, I
moved the working directory to a tmpfs (/dev/shm), which resulted
in a dramatic improvement of I/O operations:
https://dump.thecybershadow.net/d41c095b6a0dcdb7b827499a487b7c65/16%3A42%3A10-upload.png
- I've begun receiving reports on the autotester malfunctioning.
In the process of debugging this problem, I've discovered a
second problem: some files on the tmpfs would periodically
disappear. This is what caused intermittent "file not found"
errors.
- After some trial and error, I've identified the source of the
second problem (an unusual systemd behaviour). I've adjusted the
server configuration on 2017-02-09 to disable the behaviour.
- However, the first problem persisted (which manifested as
compilation errors in the 2.073.0 version of Phobos). Finally,
yesterday (2017-02-11) with some experimentation I've discovered
that the root problem was a latent DMD bug which manifested only
when the Phobos source files were being passed to it in a certain
order, which happened to be the file iteration order on tmpfs.
Details in the pull request:
https://github.com/dlang/dlang.org/pull/1568
- Now that the PR is merged, master and stable are green again.
I accept that this shouldn't have taken a week to fix, and the
initial change in question (tmpfs move) would have been better
done in a test environment. FWIW, in parallel I've been working
on a full-disk backup strategy to prepare for having one of the
server's HDDs replaced. (We already have backups of critical
data, but rebuilding from backups and reinstalling the system
would result in downtime that can be avoided. The HDDs are
already in RAID1 configuration, so the full disk backup is a
precaution.)
