Big +1 for java-based e2e tests. Currently PyFlink related tests each take ~15minutes in bash e2e tests because we are using a secured YARN cluster which is the only convenient way of starting a YARN cluster in the bash e2e tests. I think if we migrate these tests to the java-based testing framework, we will start a Yarn Cluster more conveniently, which will greatly reduce our testing time.
Best, Xingbo Rui Li <lirui.fu...@gmail.com> 于2020年11月19日周四 上午10:47写道： > Big +1 to java-based e2e tests. It'll be much easier to write/debug these > tests. > > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xbjt...@gmail.com> wrote: > > > +1 to stop using bash scripts, > > and I also have experienced the bash scripts that is really hard to > > maintain and debug, thanks @Robert for the great work again. > > > > I think testcontainers is a nice candidate. > > > > Best, > > Leonard > > > > > 在 2020年11月18日，19:46，Aljoscha Krettek <aljos...@apache.org> 写道： > > > > > > +1 > > > > > > And I want to second Arvid's mention of testcontainers [1]. > > > > > > [1] https://www.testcontainers.org/ > > > > > > On 18.11.20 10:43, Yang Wang wrote: > > >> Thanks till and Jark for sharing the information. > > >> I am also +1 for this proposal and glad to wire the new introduced K8s > > HA > > >> e2e tests to java based framework. > > >> Best, > > >> Yang > > >> Jark Wu <imj...@gmail.com> 于2020年11月18日周三 下午5:23写道： > > >>> +1 to use the Java-based testing framework and +1 for using docker > > images > > >>> in the future. > > >>> IIUC, the Java-based testing framework refers to the > > >>> `flink-end-to-end-tests-common` module. > > >>> The java-based framework helped us a lot when debugging the unstable > > e2e > > >>> tests. > > >>> > > >>> Best, > > >>> Jark > > >>> > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <danrtsey...@gmail.com> > wrote: > > >>> > > >>>> Thanks for starting this discussion. > > >>>> > > >>>> In general, I agree with you that a java-based testing framework is > > >>> better > > >>>> than the bash-based. It will > > >>>> help a lot for the commons and utilities. > > >>>> > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have > > some > > >>>> quick questions. > > >>>> * I am not sure where the java-based framework is. Do you mean > > >>>> "flink-jepsen" module or sth else? > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run / > > >>>> run-application) to submit a Flink job in the java-based framework. > > >>>> * It will be harder to inject some operations. For example, kill the > > >>>> JobManager in Kubernetes. Currently, I > > >>>> am trying to use "kubectl exec" to do this. > > >>>> > > >>>> > > >>>> Best, > > >>>> Yang > > >>>> > > >>>> Robert Metzger <rmetz...@apache.org> 于2020年11月17日周二 下午11:36写道： > > >>>> > > >>>>> Hi all, > > >>>>> > > >>>>> Since we are currently testing the 1.12 release, and potentially > > adding > > >>>>> more automated e2e tests, I would like to bring up our end to end > > tests > > >>>> for > > >>>>> discussion. > > >>>>> > > >>>>> Some time ago, we introduced a Java-based testing framework, with > the > > >>>> idea > > >>>>> of replacing the current bash-based end to end tests. > > >>>>> Since the introduction of the java-based framework, more bash tests > > >>> were > > >>>>> actually added, making a future migration even harder. > > >>>>> > > >>>>> *For that reason, I would like to propose that we are stopping to > add > > >>> any > > >>>>> new bash end to end tests to Flink. All new end to end tests must > be > > >>>>> written in Java and rely on the existing testing framework.* > > >>>>> > > >>>>> For the 1.13 release, I'm trying to find some time to revisit > > potential > > >>>>> improvements for the existing java e2e framework (such as using > > Docker > > >>>>> images everywhere), as well as a migration plan for the existing > bash > > >>>>> tests. We have a large number of bash e2e tests that are just > > >>>> parameterized > > >>>>> differently. If we would start migrating them to Java, we could > move > > a > > >>>>> larger proportion of tests over to the new Java framework, and > tackle > > >>> the > > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...). > > >>>>> > > >>>>> Let me know what you think! > > >>>>> > > >>>>> Best, > > >>>>> Robert > > >>>>> > > >>>>> > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending > > >>>> quite a > > >>>>> lot of time trying to figure out really hard to debug issues with > our > > >>>> bash > > >>>>> testing infra. > > >>>>> Also, it is very difficult to introduce something generic for all > > tests > > >>>>> (such as a test-timeout, using docker as the preferred deployment > > >>> method > > >>>>> etc.) since the tests often don't share common tooling. > > >>>>> Speaking about tooling: there are a lot of utilities everywhere, > > >>>> sometimes > > >>>>> duplicated, with different features / stability etc. > > >>>>> I believe bash is not the right tool for a project this size (in > > terms > > >>> of > > >>>>> developers and lines of code) > > >>>>> > > >>>> > > >>> > > > > > > > > > -- > Best regards! > Rui Li >
