Hi all - I want to start to get some feedback on a skunkworks project I did recently.
Impala has long had 'debug options' which are small scripts that can inject a limited class of failures into query execution, mostly at the exec node scope (e.g. delaying Prepare(), returning an error during Open() etc). These are useful, but have a number of limitations: * Extending them is hard, because of the lack of regular language * The set of available actions is limited to waiting forever, or immediately cancelling. * Actions can't be composed or sequenced, or executed with some probability. * Actions are only applicable to exec nodes, and generalising them is hard. * Only one debug action may be applied per-query. I have wanted to make improvements here for a long time, not least because testing any RPC changes is hard, and am about done with the first cut that addresses all of the above issues and more. I'm calling them 'debug rules', and they have the following features: 1. Regular rule structure so that the language may be extended without touching a parser. 2. JSON-based serialisation so that rules may be human readable and (somewhat) writeable. 3. Python builder classes to make complex rule construction easy. 4. Extensible set of actions, which now includes setting a CLI flag at runtime, logging a message, failing with a probability, sleeping, and setting RPC timeouts. 5. Actions may have arguments which may themselves be functions. Functions may access their environment in a limited way (so, for example, they may log their fragment ID). 6. Actions may be applied at any point in Impalad, not just in exec nodes. 7. Disabled completely in release builds. The full commit is the HEAD here: https://github.com/ henryr/impala/tree/debug-rules. But I'd like some general feedback on the 'user experience', and with that in mind I'd like to point you to four locations: 1. Introductory documentation here: https://github.com/ henryr/Impala/blob/debug-rules/be/src/runtime/debug-rules.h#L25 <https://github.com/henryr/Impala/blob/debug-rules/be/src/runtime/debug-rules.h#L25> 2. A set of tests for failure modes after IMPALA-1599, which parallelised fragment start-up: https://github.com/ henryr/Impala/blob/debug-rules/tests/rpc/test_1599.py <https://github.com/henryr/Impala/blob/debug-rules/tests/rpc/test_1599.py> 3. A randomized test that introduces random delays to a subset of phases in PHJ, PHA, scans and xchg operators: https://github.com/henryr/ Impala/blob/debug-rules/tests/rpc/test_exec_node_injection.py <https://github.com/henryr/Impala/blob/debug-rules/tests/rpc/test_exec_node_injection.py> 4. A port of the existing test_rpc_timeouts.py test (that confirms the RPC timeout mechanism that Juan wrote is working correctly) that doesn't need to be a custom cluster test - debug rules do all the parameter and flag setting: https://github.com/henryr/Impala/blob/debug- rules/tests/rpc/test_timeouts.py <https://github.com/henryr/Impala/blob/debug-rules/tests/rpc/test_timeouts.py> I like the last one particularly as it shaves more than 60s off the execution time of that test. The changes themselves are very large (~2000 lines and growing), and aren't remotely ready for review yet (you'll see half commented code all over the place) - but I'd like feedback on whether this is useful, would help with any testing scenarios you haven't been able to address yet, and if the abstractions and test APIs seem about right. Let me know what you think - Henry
