+1 using Java for implementing e2e testing. I have already seem many bash tests reuse logics written in Java from example modules. However creating and maintaining the bash script to extend upon these example cases for testing is kind of painful.
Maybe we can start extending the logics for e2e tests. Giving Flink is mainly written in Java, it should be an easier path comparing with Python. We can also probably use Scala for table API e2e test consider Scala is also used extensively in table API. -- Rong On Tue, Mar 27, 2018, 8:53 AM Till Rohrmann <trohrm...@apache.org> wrote: > +1 for Timo's proposal. Bash scripts can be difficult to maintain and more > complex logic can be hard to express with bash. Thus, implementing logic in > Java or Python is a good idea and will help to reuse code. I would also > favour Java at the moment, because the rest of Flink is also written in > Java. > > Cheers, > Till > > On Tue, Mar 27, 2018 at 3:54 PM, zhangminglei <18717838...@163.com> wrote: > > > At this moment, +1 from my side for maintaining bash scripts. > > > > Mingleizhang > > > > > 在 2018年3月27日，下午9:42，Piotr Nowojski <pi...@data-artisans.com> 写道： > > > > > > +1 > > > > > > I would personally go with Python, but I see a reason to Kostas’s > > arguments in favour of Java. Regardless, based on my experience with > > maintaining bash scripts, both of those options (Java/python) are much > > better in the long run. > > > > > > Piotrek > > > > > >> On 27 Mar 2018, at 15:38, Kostas Kloudas <k.klou...@data-artisans.com > > > > wrote: > > >> > > >> Hi Timo, > > >> > > >> Thanks for opening this. > > >> > > >> I agree that bash is not the best tool for all the reasons that you > > mention plus: > > >> 1) it is difficult to write re-usable code > > >> 2) there are a lot of ways to express the same thing and difficult to > > build “best practices” (as it should be > > >> in a community project) resulting in a difficulty also to make it a > > real community effort. > > >> > > >> Now on the with what to replace bash, I agree that Java is the best > > option. Flink is mainly written in Java > > >> and most of its community is already familiar with it. In addition, > > there are a lot of utilities already built > > >> that can help reduce the boilerplate code that, for example, check > logs. > > >> > > >> The only alternative I can think of it Python, but given that Flink is > > a mainly Java project. > > >> I would vouch for Java. > > >> > > >> Cheers, > > >> Kostas > > >> > > >>> On Mar 27, 2018, at 2:00 PM, Timo Walther <twal...@apache.org> > wrote: > > >>> > > >>> Hi everyone, > > >>> > > >>> after reviewing a bunch of end-to-end tests, I'm wondering if we > > should really continue implementing everything in bash scripts. Wouldn't > it > > be nicer to implement them in Java code that just calls the interfaces of > > Flink (e.g. "./bin/flink run" or REST API)? > > >>> > > >>> Here are some thoughts why I think we should consider that: > > >>> > > >>> - Implement the tests in a more object oriented way: We could have > > classes such as Cluster, LogAnalyzer, WatchdogProcess, and other utility > > classes that can make writing tests easier. This also ensure > > maintainability in the future because all tools for proper Java coding > > style, comments etc. are already in place. > > >>> > > >>> - Exception handling and type safety: We could catch excpetion more > > easily (e.g. if a REST request fails or a process is not there) > > >>> > > >>> - Debuggability: We could run end-to-end tests in the IDE for > > debugging and set breakpoints etc. > > >>> > > >>> > > >>> What do you think? > > >>> > > >>> > > >>> Regards, > > >>> > > >>> Timo > > >>> > > >> > > > > > > >
