Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6172#discussion_r195992920
  
    --- Diff: flink-end-to-end-tests/README.md ---
    @@ -33,6 +33,53 @@ $ FLINK_DIR=<flink dir> 
flink-end-to-end-tests/test-scripts/test_batch_wordcount
     
     ## Writing Tests
     
    -Have a look at test_batch_wordcount.sh for a very basic test and
    -test_streaming_kafka010.sh for a more involved example. Whenever possible, 
try
    -to put new functionality in common.sh so that it can be reused by other 
tests.
    +### Examples
    +Have a look at `test_batch_wordcount.sh` for a very basic test and
    +`test_streaming_kafka010.sh` for a more involved example. Whenever 
possible, try
    +to put new functionality in `common.sh` so that it can be reused by other 
tests.
    +
    +### Adding a test case
    +In order to add a new test case you need a new line to either 
`test-scripts/run-nightly-tests.sh` and / or 
`test-scripts/run-pre-commit-tests.sh`
    +
    +Adding a new test case generally follows the following pattern
    +
    +```sh
    +run_test "simple end-to-end test" "$END_TO_END_DIR/test-scripts/simple.sh 
arg1 arg2"
    +```
    +
    +_Note: If you want to parameterize your tests please do so by adding 
multiple test cases with parameters as arguments to the nightly / pre-commit 
test suites. This allows the test runner to do a cleanup in between each 
individual test and also to fail those tests individually._
    +
    +### Passing your test
    +A test is considered to have passed if it:
    +- has exit code 0
    +- there are no non-empty .out files (nothing was written to stdout / 
stderr by your flink program)
    +- there are no exceptions in the log files
    +- there are no errors in the log files
    +
    +### Failing your test
    +A test is considered to have failed if it:
    +- exited with non-zero exit code
    +- has non-empty *.out files (something was written to stdout / stderr by 
your flink program)
    +- there are errors in the log files
    +- there are exceptions in the log files
    +
    +_There is a whitelist for exceptions and errors that do not lead to 
failure, they can be found in the `check_logs_for_errors` and 
`check_logs_for_exceptions` in `test-scripts/common.sh`_
    +
    +Please note that a previously supported pattern where you could assign a 
value the global variable `PASS` to have your tests fail **is not supported 
anymore**
    +
    +### Cleanup
    +The test runner performs a cleanup after each test case, which includes:
    +- Stopping the cluster
    +- Killing all task and job managers
    +- Reverting config to default (if changed before)
    +- Cleaning up log and temp directories
    +
    +If your test case should do some *additional* cleanup, a common pattern is 
to trap a `test_cleanup` function to `EXIT` in your test case like so:
    --- End diff --
    
    I would explicitly state that this typically consists of shutting down 
external systems used by the e2e test, e.g. Kafka / Elasticsearch.


---

Reply via email to