[
https://issues.apache.org/jira/browse/FLINK-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925457#comment-16925457
]
Zhu Zhu edited comment on FLINK-13953 at 9/9/19 9:04 AM:
---------------------------------------------------------
Hi [~gjy], I did some investigation on how to get different MiniCluster test
methods to run multiple times for different schedulers, so that we can do all
the tests in one maven tests pass. (the
[prototype|[https://github.com/tillrohrmann/flink/tree/introduceSchedulerSwitch]]
requires one more tests pass to run tests for schedulerNG tests)
1. The parameterized tests mechanism of JUnit 4.12(currently used) only
supports the same parameters for all tests in the same class.
2. The parameterized tests mechanism of JUnit 5 can support to execute a single
test method multiple times with different parameters. However, I'm not sure if
anything would break if we'd like to upgrade it.
3. I also found a lib
[JUnitParams|[https://github.com/Pragmatists/JUnitParams]] which supports
method level test parameters for JUnit 4.12. However, the project has no new
commit for almost 1 year. So it might have been out of maintenance.
One concern to use parameterized tests is that we may need to change the
existing MiniCluster tests to generate and handle scheduler parameters, even
though the tests target for verifying nothing related with schedulers. And
later anyone developing an IT case needs to tag and parameterize it as well.
This is a bit weird and is hard to guarantee as some IT cases are out of the
control of runtime.
Considering a long term solution to verify different schedulers, I think the
[prototype|[https://github.com/tillrohrmann/flink/tree/introduceSchedulerSwitch]]
from [~till.rohrmann] would be better.
With it we can run MiniCluster tests for different schedulers with different
profiles.
For per-commit tests maybe we can test the default scheduler("legacy" currently
and "ng" later) only. And other schedulers can be verified with other scheduled
travis tests or in manual ways.
A category named "MiniClusterTest" (rather than "AlsoRunWithSchedulerNG") can
be used to tag the all MiniCluster tests so that it still works when we switch
the default scheduler to "ng".
What do you think?
was (Author: zhuzh):
Hi [~gjy], I did some investigation on how to get different MiniCluster test
methods to run multiple times for different schedulers, so that we can do all
the tests in one maven tests pass. (the
[prototype|[https://github.com/tillrohrmann/flink/tree/introduceSchedulerSwitch]]
requires one more tests pass to run tests for schedulerNG tests)
1. The parameterized tests mechanism of JUnit 4.12(currently used) only
supports the same parameters for all tests in the same class.
2. The parameterized tests mechanism of JUnit 5 can support to execute a single
test method multiple times with different parameters. However, I'm not sure if
anything would broke if we'd like to upgrade it.
3. I also found a lib
[JUnitParams|[https://github.com/Pragmatists/JUnitParams]] which supports
method level test parameters for JUnit 4.12. However, the project has no new
commit for almost 1 year. So I'm not sure if it's a good idea to use it as it
might have been out of maintenance.
One concern to use parameterized tests is that we may need to change the
existing MiniCluster tests to generate, parse and handle the parameters, even
though the tests target for verifying nothing related with schedulers. And
later anyone developing an IT case needs to tag and parameterize it as well.
This is a bit weird and is hard to guarantee as some IT cases are out of the
control of runtime.
Considering a long term solution to verify different schedulers, I think the
[prototype|[https://github.com/tillrohrmann/flink/tree/introduceSchedulerSwitch]]
from [~till.rohrmann] would be better.
With it we can run MiniCluster tests for different schedulers with different
profiles.
For per-commit tests maybe we can test the default scheduler("legacy" currently
and "ng" later) only. And other schedulers can be verified with other scheduled
travis tests and in manual ways.
A category named "MiniClusterTest" (rather than "AlsoRunWithSchedulerNG") can
be used to tag the all MiniCluster tests so that it still works when we switch
the default scheduler to "ng".
What do you think?
> Facilitate enabling new Scheduler in MiniCluster Tests
> ------------------------------------------------------
>
> Key: FLINK-13953
> URL: https://issues.apache.org/jira/browse/FLINK-13953
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination, Tests
> Reporter: Gary Yao
> Assignee: Zhu Zhu
> Priority: Major
>
> Currently, tests using the {{MiniCluster}} use the legacy scheduler by
> default. Once the new scheduler is implemented, we should run tests with the
> new scheduler enabled. However, it is not expected that all tests will pass
> immediately. Therefore, it should be possible to enable the new scheduler for
> a subset of tests.
> *Acceptance Criteria*
> * Subset of tests using {{MiniCluster}} can be run on a per-commit basis (on
> Travis) against new scheduler and also legacy scheduler
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
