Big +1 for your proposal Robert. I think the decision to use bash for our e2e tests already fired back on us several times. Hence, I believe it is time for a serious effort to get away from it.
The e2e test framework can be found here [1]. When using the e2e test framework, it should be possible to use Flink's programmatic tools to submit a job (e.g. the RestClusterClient). For killing K8s processes one could write a wrapper for kubectl which can be called from the Java code. I think we already have it for some other cli tools [2]. Alternatively, one could use Fabric8 to communicate with the K8s cluster. [1] https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common [2] https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/CommandLineWrapper.java Cheers, Till On Wed, Nov 18, 2020 at 7:42 AM Yang Wang <danrtsey...@gmail.com> wrote: > Thanks for starting this discussion. > > In general, I agree with you that a java-based testing framework is better > than the bash-based. It will > help a lot for the commons and utilities. > > Since I am trying to add a new bash-based Kubernetes HA test, I have some > quick questions. > * I am not sure where the java-based framework is. Do you mean > "flink-jepsen" module or sth else? > * Maybe it will be harder to run a cli command(e.g. flink run / > run-application) to submit a Flink job in the java-based framework. > * It will be harder to inject some operations. For example, kill the > JobManager in Kubernetes. Currently, I > am trying to use "kubectl exec" to do this. > > > Best, > Yang > > Robert Metzger <rmetz...@apache.org> 于2020年11月17日周二 下午11:36写道: > > > Hi all, > > > > Since we are currently testing the 1.12 release, and potentially adding > > more automated e2e tests, I would like to bring up our end to end tests > for > > discussion. > > > > Some time ago, we introduced a Java-based testing framework, with the > idea > > of replacing the current bash-based end to end tests. > > Since the introduction of the java-based framework, more bash tests were > > actually added, making a future migration even harder. > > > > *For that reason, I would like to propose that we are stopping to add any > > new bash end to end tests to Flink. All new end to end tests must be > > written in Java and rely on the existing testing framework.* > > > > For the 1.13 release, I'm trying to find some time to revisit potential > > improvements for the existing java e2e framework (such as using Docker > > images everywhere), as well as a migration plan for the existing bash > > tests. We have a large number of bash e2e tests that are just > parameterized > > differently. If we would start migrating them to Java, we could move a > > larger proportion of tests over to the new Java framework, and tackle the > > more involved bash tests later (kerberized yarn, kubernetes, ...). > > > > Let me know what you think! > > > > Best, > > Robert > > > > > > PS: If you are wondering why I'm bringing this up now: I'm spending > quite a > > lot of time trying to figure out really hard to debug issues with our > bash > > testing infra. > > Also, it is very difficult to introduce something generic for all tests > > (such as a test-timeout, using docker as the preferred deployment method > > etc.) since the tests often don't share common tooling. > > Speaking about tooling: there are a lot of utilities everywhere, > sometimes > > duplicated, with different features / stability etc. > > I believe bash is not the right tool for a project this size (in terms of > > developers and lines of code) > > >