Hello bearophile,
Do you know fuzzy logic? One of the purposes of fuzzy logic is to
design control systems (that can be used for washing machines,
cameras, missiles, etc) that work and fail gracefully. They don't work
in two binary ways perfect/totallywrong. A graceful failure may have
avoided the Ariane to crash and go boom.
Today people are studying software systems based on fuzzy logic,
neural networks, support vector machines, and more, that are designed
to keep working despite some small problems and faults.
But this still assumes some degree of reliability of the code doing the fuzzy
logic. If I had to guess, I'd expect that the systems you mention are designed
to function under external faults (some expected input vanishes or some other
component in a distributed system fails). It would be almost impossible to
make a program that can work correctly once it has had an internal fault.
Once that has happened, I think Walter is correct and the only thing to do
is shut down. In the auto pilot case, this could amount to kill off the current
auto pilot process and boot up a very simple fly-straight-and-level program
to take over while the pilot reacts to a nice loud klaxon.
