[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846924#comment-17846924 ] Matthias Pohl commented on FLINK-20392: --- Sure, sounds reasonable. Feel free to update it. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846873#comment-17846873 ] Lorenzo Affetti commented on FLINK-20392: - [~mapohl] [~m.orazow] thanks for your wonderful comments. The solution lies in between as many times happen :) Sounds perfect to me. I will head to addressing your last comments [~mapohl] I'll also review your PR [~m.orazow] Thank you for the fruitful conversation. One last thing: [~mapohl] do you think we should add my previous comment to the umbrella description, maybe with some rephrasing? So that a future contributor does not need to read the entire discussion? > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846857#comment-17846857 ] Muhammet Orazov commented on FLINK-20392: - [~mapohl] yes, I agree I think it is good balance, let's keep both options. We initially was opting to keep them consistent but we can come back to it again if that is issue again. Migrating them into dockerized tests will add some overhead since each suite will create (2-5) containers, wait until they are ready and terminate at the end. At some point later we can invest some time to group tests that can reuse the Flink cluster, testcontainers have the `reuse` option. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846838#comment-17846838 ] Matthias Pohl commented on FLINK-20392: --- This discussion feels similar to our efforts around migrating to JUnit5 and assertj as the standard JUnit tests. It costed (and is still costing) quite a bit of resources with the risk of missing things when reviewing the tests. That is why I still see value in just keeping both options around. That requires less resources and we're not losing much. The pros and cons are still a good guideline for developers to decide on which technology to use if they are planning to create a new e2e test in Java. WDYT? > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[
https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846609#comment-17846609
]
Muhammet Orazov commented on FLINK-20392:
-
Thanks [~lorenzo.affetti] ,
{quote}MiniClusterExtension:
- PRO: the code is very clean: no need to deploy separate JAR for the Flink
app, as that is coded in the test and the Flink cluster is registered in the
current test environment
{quote}
I would see this also extra work as CON. Most of the e2e are already have so
called TestProgram, with MiniCluster approach these functionality will have to
extracted. I think this is extra effort, instead we could submit those test
programs to FlinkContainers and validate the expected outcome(s).
> Migrating bash e2e tests to Java/Docker
> ---
>
> Key: FLINK-20392
> URL: https://issues.apache.org/jira/browse/FLINK-20392
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure, Tests
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> starter
>
> This Jira issue serves as an umbrella ticket for single e2e test migration
> tasks. This should enable us to migrate all bash-based e2e tests step-by-step.
> The goal is to utilize the e2e test framework (see
> [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]).
> Ideally, the test should use Docker containers as much as possible
> disconnect the execution from the environment. A good source to achieve that
> is [testcontainers.org|https://www.testcontainers.org/].
> The related ML discussion is [Stop adding new bash-based e2e tests to
> Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846540#comment-17846540 ] Matthias Pohl commented on FLINK-20392: --- Thanks for the write-up. I'm just wondering whether we gain anything from only allowing one of the two approaches. What about allowing both options? > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846535#comment-17846535 ] Lorenzo Affetti commented on FLINK-20392: - [~m.orazow] and I found ourselves working on two issues under this umbrella: - https://issues.apache.org/jira/browse/FLINK-20398 (batch sql) - https://issues.apache.org/jira/browse/FLINK-20400 (streaming sql) And contributed with this 2 PRs: - [https://github.com/apache/flink/pull/24471] (batch sql) - [https://github.com/apache/flink/pull/24776] (streaming sql) The "batch" PR makes use of the MiniClusterExtension while the "streaming" PR makes use of TestContainers and, specifically, FlinkContainers. We think that the underlying purpose here (apart from porting tests from bash to Java) is to: - reflect their bash behavior - adopt a homogeneous solution for (hopefully) every test porting. [~m.orazow] and I had an offline discussion mainly focused on the nature of these end-to-end tests: at the best of our understanding, these tests are meant to test the end-to-end functionalities of some "close-to-reality" Flink cluster. Indeed, their current behavior (in bash) consists of starting a local Flink cluster and issuing `flink run` commands against it. Moreover, some test (e.g.: flink-end-to-end-tests/test-scripts/test_queryable_state_restart_tm.sh) also test failure by killing a random TaskManager (as a side note, this umbrella does not have an issue per existing e2e test, we volunteer to sink up its state in the near future). After an offline discussion, Muhammet and I believe that _the TestContainers approach is the one that fits most the idea behind this umbrella issue._ We elicit here below the PROs and CONs of both approaches. TestContainers : - PRO: the Flink cluster is as close as possible to reality (full TCP/IP stack involved, separate processes) - PRO: uses the Flink binaries in the Flink dist as the bash version does - PRO: handlers to control the cluster available -> a random TM can be killed to check the failure state - CON: running tests requires Docker images to be built (time required) MiniClusterExtension: - PRO: the code is very clean: no need to deploy separate JAR for the Flink app, as that is coded in the test and the Flink cluster is registered in the current test environment - PRO: no further build steps required apart from java build (fast run) - CON: missing handlers for TM kill (could be implemented by exposing MiniClusterResource from the extension) - CON: less "close-to-reality" as each service runs in a separate thread (not process), however an in-depth analysis of the implementation of the MiniClusterResource should be conducted to understand how much this is close to a Flink cluster. *Wrapping up, we do think that if we want to mimic as much as possible the current behavior of tests, TestContainers is the way to go at the cost of the the Docker-build time of containers in CI.* We would like to gather at least 2 people's agreement on one of the 2 solution without a strong opposing position from somebody else in order to edit our PRs to match and to continue with this massive port. As they are active in the thread and experts on the topic, I suggest [~mapohl] and [~jark] to help us in this. Thank you guys for all the support and helpful reviews as of now. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823485#comment-17823485 ] Matthias Pohl commented on FLINK-20392: --- Relying on MiniCluster from the Flink side should work as well. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[
https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823242#comment-17823242
]
Lorenzo Affetti commented on FLINK-20392:
-
{quote}I'm not able to follow what you mean here. Both linked PRs are still
accessible (including its code change). Or do you mean that the code of the PR
is not present in the main repo anymore?
{quote}
Yeah, sorry, I meant that that code was moved/refactored (y)
But thank you for the clarification.
My goal here was to clarify which part of tests should rely on test containers.
For example, it could be that Flink runs in the same thread of the tests and
the dependencies (for example, Kafka) run in containers.
As far as I understood, also Flink should run in containers (y)
I was adding references to the code that enables that at the current state of
the repo.
> Migrating bash e2e tests to Java/Docker
> ---
>
> Key: FLINK-20392
> URL: https://issues.apache.org/jira/browse/FLINK-20392
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure, Tests
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> starter
>
> This Jira issue serves as an umbrella ticket for single e2e test migration
> tasks. This should enable us to migrate all bash-based e2e tests step-by-step.
> The goal is to utilize the e2e test framework (see
> [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]).
> Ideally, the test should use Docker containers as much as possible
> disconnect the execution from the environment. A good source to achieve that
> is [testcontainers.org|https://www.testcontainers.org/].
> The related ML discussion is [Stop adding new bash-based e2e tests to
> Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[
https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823193#comment-17823193
]
Matthias Pohl commented on FLINK-20392:
---
The goal is to move away from bash-based e2e tests. We created the issue here
to have this goal being documented in Jira (so that new contributors can pick
up subtasks). The most likely solution would be to rely on TestContainers, I
guess.
{quote}
As the code of the PR linked above
{quote}
I'm not able to follow what you mean here. Both linked PRs are still accessible
(including its code change). Or do you mean that the code of the PR is not
present in the main repo anymore?
> Migrating bash e2e tests to Java/Docker
> ---
>
> Key: FLINK-20392
> URL: https://issues.apache.org/jira/browse/FLINK-20392
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure, Tests
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> starter
>
> This Jira issue serves as an umbrella ticket for single e2e test migration
> tasks. This should enable us to migrate all bash-based e2e tests step-by-step.
> The goal is to utilize the e2e test framework (see
> [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]).
> Ideally, the test should use Docker containers as much as possible
> disconnect the execution from the environment. A good source to achieve that
> is [testcontainers.org|https://www.testcontainers.org/].
> The related ML discussion is [Stop adding new bash-based e2e tests to
> Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823188#comment-17823188 ] Lorenzo Affetti commented on FLINK-20392: - [~mapohl] [~jark] In order to better understand the current status of this issue and of the codebase as well nowadays: As the code of the PR linked above ([https://github.com/apache/flink/pull/14085]) does not exist anymore (I think the major PR to move it was this one: [https://github.com/apache/flink/pull/17892),] I will proceed following [this example|[https://github.com/apache/flink/blob/4595762f974f03bdf5a4c3d42e211c3a24604bd4/flink-end-to-end-tests/flink-sql-client-test/src/test/java/SqlClientITCase.java]], that uses [FlinkContainers|[https://github.com/apache/flink/blob/d6c7eee8243b4fe3e593698f250643534dc79cb5/flink-test-utils-parent/flink-connector-test-utils/src/main/java/org/apache/flink/connector/testframe/container/FlinkContainers.java]]. I think the goal of these IT tests is to actually spawn a Flink cluster via testcontainers (as `FlinkContainers` does) and required dependencies as well (e.g.: Kafka, etc.) and run the tests. Do you have any feedback on this? Thank you > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822606#comment-17822606 ] Lorenzo Affetti commented on FLINK-20392: - Hello, I would like to take https://issues.apache.org/jira/browse/FLINK-20398 I see there is no previous PR merged about this. So, we just go test-by-test case and see what we get? As far as I understood the outcome would be to eliminate the bash-side of the equation and just go with Java and use as much Docker as possible (y) > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Minor > Labels: auto-deprioritized-major, auto-deprioritized-minor, > starter > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[
https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17561002#comment-17561002
]
Yang Wang commented on FLINK-20392:
---
I am not aware of other widely used java based e2e framework for Kubernetes.
If we start the minikube by script before hand or use something else(e.g.
[https://www.kubermatic.com/blog/2019-03-12-running-kubernetes-in-the-ci-pipeline-/]),
then migrating the k8s e2e tests will be possible. Because we could use
{{CliFrontend}} for the application submission and {{KubernetesSessionCli}} for
the session cluster starting. The fabric8 kubernetes-client could be used to
replace kubectl to verify the result, get the k8s resources status, or check
the logs.
> Migrating bash e2e tests to Java/Docker
> ---
>
> Key: FLINK-20392
> URL: https://issues.apache.org/jira/browse/FLINK-20392
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure, Tests
>Reporter: Matthias Pohl
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> This Jira issue serves as an umbrella ticket for single e2e test migration
> tasks. This should enable us to migrate all bash-based e2e tests step-by-step.
> The goal is to utilize the e2e test framework (see
> [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]).
> Ideally, the test should use Docker containers as much as possible
> disconnect the execution from the environment. A good source to achieve that
> is [testcontainers.org|https://www.testcontainers.org/].
> The related ML discussion is [Stop adding new bash-based e2e tests to
> Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560895#comment-17560895 ] Martijn Visser commented on FLINK-20392: [~wangyang0918] I think we're open for anything. Are there already best practices when it comes to running these type of tests that we could adopt? Is everyone still using Bash based tests? > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Technical Debt > Components: Test Infrastructure, Tests >Reporter: Matthias Pohl >Priority: Major > Labels: auto-deprioritized-major, auto-deprioritized-minor > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[
https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560793#comment-17560793
]
Yang Wang commented on FLINK-20392:
---
Thanks [~martijnvisser] for adding the K8s related e2e tests migration. I still
have the concern as before. Do we expect the minikube already running before
the java based tests or we need to start/stop the minikube via java
implementation? Another thing is how to submit the Flink application especially
for the native K8s integration. Currently, we are using CLI command {{{}flink
run-application{}}}. Do we need to use the java interface(e.g.
{{{}ApplicationDeployer){}}}? Then I am afraid the e2e tests does not cover the
real use case because most users are using the CLI.
> Migrating bash e2e tests to Java/Docker
> ---
>
> Key: FLINK-20392
> URL: https://issues.apache.org/jira/browse/FLINK-20392
> Project: Flink
> Issue Type: Technical Debt
> Components: Test Infrastructure, Tests
>Reporter: Matthias Pohl
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> This Jira issue serves as an umbrella ticket for single e2e test migration
> tasks. This should enable us to migrate all bash-based e2e tests step-by-step.
> The goal is to utilize the e2e test framework (see
> [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]).
> Ideally, the test should use Docker containers as much as possible
> disconnect the execution from the environment. A good source to achieve that
> is [testcontainers.org|https://www.testcontainers.org/].
> The related ML discussion is [Stop adding new bash-based e2e tests to
> Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240748#comment-17240748 ] Matthias commented on FLINK-20392: -- Thanks [~fly_in_gis] for bringing this up. Useful features missing in the Java-based testing framework should be added, I guess. We should see it as a joint effort to improve the framework. The way new features need be implemented can be discussed as part of the migration efforts when looking at the individual e2e tests. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Test > Components: Test Infrastructure, Tests >Reporter: Matthias >Priority: Major > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240549#comment-17240549 ] Yang Wang commented on FLINK-20392: --- [~mapohl] Thanks for creating this umbrella ticket to track all the migration. It is really a good start point. For the Junit based tests, I think it will be great if we use the testcontainer to run. However, for some e2e tests which depends on some external system(e.g. minikube), I am not sure how to migrate them to Java based framework. Could we expect that the external system is already running there before running the e2e test? Maybe it could be started by script or azure pipeline. An alternative is we wrap the start/stop into the Java code and call it for each test. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Test > Components: Test Infrastructure, Tests >Reporter: Matthias >Priority: Major > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239716#comment-17239716 ] Dawid Wysakowicz commented on FLINK-20392: -- Just to add to the FLINK-19997, I set up a FlinkContainer that runs a Flink cluster within a docker container with a subset of functionalities provided by the FlinkResource. It can be a good base to move the FlinkResource onto testcontainers. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Test > Components: Test Infrastructure, Tests >Reporter: Matthias >Priority: Major > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20392) Migrating bash e2e tests to Java/Docker
[ https://issues.apache.org/jira/browse/FLINK-20392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239714#comment-17239714 ] Jark Wu commented on FLINK-20392: - As a reference, [~dwysakowicz] has checked in a good example that uses testcontainers to startup Dockers for Kafka and Schema Registery in FLINK-19997. > Migrating bash e2e tests to Java/Docker > --- > > Key: FLINK-20392 > URL: https://issues.apache.org/jira/browse/FLINK-20392 > Project: Flink > Issue Type: Test > Components: Test Infrastructure, Tests >Reporter: Matthias >Priority: Major > > This Jira issue serves as an umbrella ticket for single e2e test migration > tasks. This should enable us to migrate all bash-based e2e tests step-by-step. > The goal is to utilize the e2e test framework (see > [flink-end-to-end-tests-common|https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common]). > Ideally, the test should use Docker containers as much as possible > disconnect the execution from the environment. A good source to achieve that > is [testcontainers.org|https://www.testcontainers.org/]. > The related ML discussion is [Stop adding new bash-based e2e tests to > Flink|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Stop-adding-new-bash-based-e2e-tests-to-Flink-td46607.html]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
