[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443688#comment-16443688 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5665 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16442595#comment-16442595 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5665 merging > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441020#comment-16441020 ] ASF GitHub Bot commented on FLINK-8703: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5665 I think this looks good to go then. 👍 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440774#comment-16440774 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5665 @aljoscha I've addressed your comments and rebased the branch. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439564#comment-16439564
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r181777925
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -217,12 +231,12 @@ public void getTaskManagerLogAndStdoutFiles() {
@Test
public void getConfiguration() {
try {
- String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + port + "/jobmanager/config");
+ String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + CLUSTER.getWebUIPort() +
"/jobmanager/config");
Map conf =
WebMonitorUtils.fromKeyValueJsonArray(config);
assertEquals(
-
cluster.configuration().getString("taskmanager.numberOfTaskSlots", null),
-
conf.get(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS));
+
CLUSTER_CONFIGURATION.getString(ConfigConstants.LOCAL_START_WEBSERVER, null),
--- End diff --
Yeah, that's what I meant, the previous code was a bit strange
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439561#comment-16439561
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r181776571
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -217,12 +231,12 @@ public void getTaskManagerLogAndStdoutFiles() {
@Test
public void getConfiguration() {
try {
- String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + port + "/jobmanager/config");
+ String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + CLUSTER.getWebUIPort() +
"/jobmanager/config");
Map conf =
WebMonitorUtils.fromKeyValueJsonArray(config);
assertEquals(
-
cluster.configuration().getString("taskmanager.numberOfTaskSlots", null),
-
conf.get(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS));
+
CLUSTER_CONFIGURATION.getString(ConfigConstants.LOCAL_START_WEBSERVER, null),
--- End diff --
I wanted an option for which the configured value is different from the
default. The default for `numberOfTaskSlots` is one, so the test would've
passed even if the handler returned a vanilla configuration.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439554#comment-16439554
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r181775179
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -127,14 +137,18 @@ public void testResponseHeaders() throws Exception {
Assert.assertEquals("application/json; charset=UTF-8",
taskManagerConnection.getContentType());
// check headers in case of an error
- URL notFoundJobUrl = new URL("http://localhost:"; + port +
"/jobs/dontexist");
+ URL notFoundJobUrl = new URL("http://localhost:"; +
CLUSTER.getWebUIPort() + "/jobs/dontexist");
HttpURLConnection notFoundJobConnection = (HttpURLConnection)
notFoundJobUrl.openConnection();
notFoundJobConnection.setConnectTimeout(10);
notFoundJobConnection.connect();
if (notFoundJobConnection.getResponseCode() >= 400) {
// we don't set the content-encoding header
Assert.assertNull(notFoundJobConnection.getContentEncoding());
- Assert.assertEquals("text/plain; charset=UTF-8",
notFoundJobConnection.getContentType());
+ if (Objects.equals("flip6",
System.getProperty("codebase"))) {
--- End diff --
this must be updated to "new"
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439519#comment-16439519 ] ASF GitHub Bot commented on FLINK-8703: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5665#discussion_r181756207 --- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/testutils/HttpTestClient.java --- @@ -185,6 +185,25 @@ public void sendDeleteRequest(String path, FiniteDuration timeout) throws Timeou sendRequest(getRequest, timeout); } + /** +* Sends a simple PATH request to the given path. You only specify the $path part of +* http://$host:$host/$path. +* +* @param path The $path to DELETE (http://$host:$host/$path) --- End diff -- copy error? `DELETE` > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439518#comment-16439518 ] ASF GitHub Bot commented on FLINK-8703: --- Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/5665#discussion_r181756063 --- Diff: flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/testutils/HttpTestClient.java --- @@ -185,6 +185,25 @@ public void sendDeleteRequest(String path, FiniteDuration timeout) throws Timeou sendRequest(getRequest, timeout); } + /** +* Sends a simple PATH request to the given path. You only specify the $path part of --- End diff -- typo: PATH -> PATCH? > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439516#comment-16439516
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r181755757
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -242,32 +256,44 @@ public void testStop() throws Exception {
final JobGraph jobGraph = new JobGraph("Stoppable streaming
test job", sender);
final JobID jid = jobGraph.getJobID();
- cluster.submitJobDetached(jobGraph);
+ ClusterClient clusterClient = CLUSTER.getClusterClient();
+ clusterClient.setDetached(true);
+ clusterClient.submitJob(jobGraph,
WebFrontendITCase.class.getClassLoader());
// wait for job to show up
- while (cluster.getCurrentlyRunningJobsJava().isEmpty()) {
+ while (getRunningJobs(CLUSTER.getClusterClient()).isEmpty()) {
Thread.sleep(10);
}
final FiniteDuration testTimeout = new FiniteDuration(2,
TimeUnit.MINUTES);
final Deadline deadline = testTimeout.fromNow();
- while (!cluster.getCurrentlyRunningJobsJava().isEmpty()) {
- try (HttpTestClient client = new
HttpTestClient("localhost", port)) {
- // Request the file from the web server
- client.sendDeleteRequest("/jobs/" + jid +
"/stop", deadline.timeLeft());
- HttpTestClient.SimpleHttpResponse response =
client.getNextResponse(deadline.timeLeft());
-
- assertEquals(HttpResponseStatus.OK,
response.getStatus());
- assertEquals("application/json; charset=UTF-8",
response.getType());
- assertEquals("{}", response.getContent());
+ while (!getRunningJobs(CLUSTER.getClusterClient()).isEmpty()) {
+ try (HttpTestClient client = new
HttpTestClient("localhost", CLUSTER.getWebUIPort())) {
+ if
(Objects.equals(MiniClusterResource.FLIP6_CODEBASE,
System.getProperty(MiniClusterResource.CODEBASE_KEY))) {
+ // Request the file from the web server
--- End diff --
This comment is/was outdated, it seems.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16439513#comment-16439513
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r181755304
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -217,12 +231,12 @@ public void getTaskManagerLogAndStdoutFiles() {
@Test
public void getConfiguration() {
try {
- String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + port + "/jobmanager/config");
+ String config =
TestBaseUtils.getFromHTTP("http://localhost:"; + CLUSTER.getWebUIPort() +
"/jobmanager/config");
Map conf =
WebMonitorUtils.fromKeyValueJsonArray(config);
assertEquals(
-
cluster.configuration().getString("taskmanager.numberOfTaskSlots", null),
-
conf.get(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS));
+
CLUSTER_CONFIGURATION.getString(ConfigConstants.LOCAL_START_WEBSERVER, null),
--- End diff --
what was up here? 😅
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428311#comment-16428311 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5669 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428152#comment-16428152 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5669 merging. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426402#comment-16426402
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r179332025
--- Diff:
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java
---
@@ -1063,23 +1078,27 @@ public void runCancelingOnEmptyInputTest() throws
Exception {
final AtomicReference error = new
AtomicReference<>();
+ final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
--- End diff --
As you wish, I can open a follow up since it's a trivial fixup.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425522#comment-16425522 ] Chesnay Schepler commented on FLINK-8703: - Partially addressed in: master: 20bda911067dc2e2503e4447e498e2ff8731dada 1.5: 20d7af77005f831f1cadf3769e0a9a059b9f37d6 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425412#comment-16425412 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5664 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423778#comment-16423778 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5669 I've rebased and updated the PR. The `JobManagerCommunicationUtils` were reworked into a `ClusterCommunicationUtils`. All contained methods were now unused. We are also using `Iterables.getOnlyElement` now, > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423774#comment-16423774
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r178769136
--- Diff:
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java
---
@@ -1063,23 +1078,27 @@ public void runCancelingOnEmptyInputTest() throws
Exception {
final AtomicReference error = new
AtomicReference<>();
+ final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
--- End diff --
That would go further than porting the test; I'd rather not do that in this
PR.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423706#comment-16423706 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5664 merging. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417622#comment-16417622
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5664#discussion_r177801072
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/cancelling/JoinCancelingITCase.java
---
@@ -30,19 +30,18 @@
import org.apache.flink.test.util.InfiniteIntegerTupleInputFormat;
import org.apache.flink.test.util.UniformIntTupleGeneratorInputFormat;
+import org.junit.Ignore;
+import org.junit.Test;
+
/**
* Test job cancellation from within a JoinFunction.
*/
+@Ignore("Takes too long.")
--- End diff --
No, I also didn't check how long it actually takes.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417621#comment-16417621
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5664#discussion_r177800863
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/cancelling/CancelingTestBase.java
---
@@ -18,160 +18,111 @@
package org.apache.flink.test.cancelling;
+import org.apache.flink.api.common.JobSubmissionResult;
import org.apache.flink.api.common.Plan;
+import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.configuration.AkkaOptions;
-import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.CoreOptions;
import org.apache.flink.configuration.TaskManagerOptions;
import org.apache.flink.optimizer.DataStatistics;
import org.apache.flink.optimizer.Optimizer;
import org.apache.flink.optimizer.plan.OptimizedPlan;
import org.apache.flink.optimizer.plantranslate.JobGraphGenerator;
-import org.apache.flink.runtime.instance.ActorGateway;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.runtime.jobgraph.JobStatus;
-import org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster;
import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testutils.JobManagerActorTestUtils;
+import org.apache.flink.test.util.MiniClusterResource;
import org.apache.flink.util.TestLogger;
-import org.apache.hadoop.fs.FileSystem;
-import org.junit.After;
import org.junit.Assert;
-import org.junit.Before;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.junit.ClassRule;
import java.util.concurrent.TimeUnit;
-import scala.concurrent.Await;
-import scala.concurrent.Future;
+import scala.concurrent.duration.Deadline;
import scala.concurrent.duration.FiniteDuration;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancelJob;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancellationFailure;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancellationSuccess;
-
/**
* Base class for testing job cancellation.
*/
public abstract class CancelingTestBase extends TestLogger {
- private static final Logger LOG =
LoggerFactory.getLogger(CancelingTestBase.class);
-
private static final int MINIMUM_HEAP_SIZE_MB = 192;
- /**
-* Defines the number of seconds after which an issued cancel request
is expected to have taken effect (i.e. the job
-* is canceled), starting from the point in time when the cancel
request is issued.
-*/
- private static final int DEFAULT_CANCEL_FINISHED_INTERVAL = 10 * 1000;
-
- private static final int DEFAULT_TASK_MANAGER_NUM_SLOTS = 1;
+ protected static final int PARALLELISM = 4;
//
- protected LocalFlinkMiniCluster executor;
-
- protected int taskManagerNumSlots = DEFAULT_TASK_MANAGER_NUM_SLOTS;
+ @ClassRule
+ public static final MiniClusterResource CLUSTER = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ getConfiguration(),
+ 2,
--- End diff --
Possibly, i just copied the existing setup.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417571#comment-16417571
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5664#discussion_r177794734
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/cancelling/CancelingTestBase.java
---
@@ -18,160 +18,111 @@
package org.apache.flink.test.cancelling;
+import org.apache.flink.api.common.JobSubmissionResult;
import org.apache.flink.api.common.Plan;
+import org.apache.flink.client.program.ClusterClient;
import org.apache.flink.configuration.AkkaOptions;
-import org.apache.flink.configuration.ConfigConstants;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.CoreOptions;
import org.apache.flink.configuration.TaskManagerOptions;
import org.apache.flink.optimizer.DataStatistics;
import org.apache.flink.optimizer.Optimizer;
import org.apache.flink.optimizer.plan.OptimizedPlan;
import org.apache.flink.optimizer.plantranslate.JobGraphGenerator;
-import org.apache.flink.runtime.instance.ActorGateway;
import org.apache.flink.runtime.jobgraph.JobGraph;
import org.apache.flink.runtime.jobgraph.JobStatus;
-import org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster;
import org.apache.flink.runtime.testingUtils.TestingUtils;
-import org.apache.flink.runtime.testutils.JobManagerActorTestUtils;
+import org.apache.flink.test.util.MiniClusterResource;
import org.apache.flink.util.TestLogger;
-import org.apache.hadoop.fs.FileSystem;
-import org.junit.After;
import org.junit.Assert;
-import org.junit.Before;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+import org.junit.ClassRule;
import java.util.concurrent.TimeUnit;
-import scala.concurrent.Await;
-import scala.concurrent.Future;
+import scala.concurrent.duration.Deadline;
import scala.concurrent.duration.FiniteDuration;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancelJob;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancellationFailure;
-import static
org.apache.flink.runtime.messages.JobManagerMessages.CancellationSuccess;
-
/**
* Base class for testing job cancellation.
*/
public abstract class CancelingTestBase extends TestLogger {
- private static final Logger LOG =
LoggerFactory.getLogger(CancelingTestBase.class);
-
private static final int MINIMUM_HEAP_SIZE_MB = 192;
- /**
-* Defines the number of seconds after which an issued cancel request
is expected to have taken effect (i.e. the job
-* is canceled), starting from the point in time when the cancel
request is issued.
-*/
- private static final int DEFAULT_CANCEL_FINISHED_INTERVAL = 10 * 1000;
-
- private static final int DEFAULT_TASK_MANAGER_NUM_SLOTS = 1;
+ protected static final int PARALLELISM = 4;
//
- protected LocalFlinkMiniCluster executor;
-
- protected int taskManagerNumSlots = DEFAULT_TASK_MANAGER_NUM_SLOTS;
+ @ClassRule
+ public static final MiniClusterResource CLUSTER = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ getConfiguration(),
+ 2,
--- End diff --
Why do we start a mini cluster with 2 TMs and 4 slots per TM? Wouldn't a
single TM be sufficient?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417570#comment-16417570
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5664#discussion_r177794320
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/cancelling/JoinCancelingITCase.java
---
@@ -30,19 +30,18 @@
import org.apache.flink.test.util.InfiniteIntegerTupleInputFormat;
import org.apache.flink.test.util.UniformIntTupleGeneratorInputFormat;
+import org.junit.Ignore;
+import org.junit.Test;
+
/**
* Test job cancellation from within a JoinFunction.
*/
+@Ignore("Takes too long.")
--- End diff --
Do you know what's taking so long?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417489#comment-16417489 ] ASF GitHub Bot commented on FLINK-8703: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/5664 Interesting, there is a aborted test run with a core dump. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417316#comment-16417316
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r177747320
--- Diff:
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java
---
@@ -1063,23 +1078,27 @@ public void runCancelingOnEmptyInputTest() throws
Exception {
final AtomicReference error = new
AtomicReference<>();
+ final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
--- End diff --
Please unify and extract common code between `runCancelingOnFullInputTest`
and `runCancelingOnEmptyInputTest` for example into:
```
private void runCancelingTest(boolean emptyInput) throws Exception {
final String topic = emptyInput ? "cancelingOnEmptyInputTopic"
: "cancelingOnFullTopic";
final int parallelism = 3;
createTestTopic(topic, parallelism, 1);
// launch a producer thread
DataGenerators.InfiniteStringsGenerator generator =
new
DataGenerators.InfiniteStringsGenerator(kafkaServer, topic);
if (!emptyInput) {
generator.start();
}
// launch a consumer asynchronously
final AtomicReference jobError = new
AtomicReference<>();
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(parallelism);
env.enableCheckpointing(100);
env.getConfig().disableSysoutLogging();
Properties props = new Properties();
props.putAll(standardProps);
props.putAll(secureProps);
FlinkKafkaConsumerBase source =
kafkaServer.getConsumer(topic, new SimpleStringSchema(), props);
env.addSource(source).addSink(new DiscardingSink());
JobGraph jobGraph =
StreamingJobGraphGenerator.createJobGraph(env.getStreamGraph());
final JobID jobId = jobGraph.getJobID();
final Runnable jobRunner = new Runnable() {
@Override
public void run() {
try {
client.setDetached(false);
client.submitJob(jobGraph,
KafkaConsumerTestBase.class.getClassLoader());
}
catch (Throwable t) {
jobError.set(t);
}
}
};
Thread runnerThread = new Thread(jobRunner, "program runner
thread");
runnerThread.start();
// wait a bit before canceling
Thread.sleep(2000);
Throwable failueCause = jobError.get();
if (failueCause != null) {
failueCause.printStackTrace();
Assert.fail("Test failed prematurely with: " +
failueCause.getMessage());
}
// cancel
client.cancel(jobId);
// wait for the program to be done and validate that we failed
with the right exception
runnerThread.join();
assertEquals(JobStatus.CANCELED,
client.getJobStatus(jobId).get());
if (generator.isAlive()) {
generator.shutdown();
generator.join();
}
else if (!emptyInput) {
Throwable t = generator.getError();
if (t != null) {
t.printStackTrace();
fail("Generator failed: " + t.getMessage());
} else {
fail("Generator failed with no exception");
}
}
deleteTestTopic(topic);
}
```
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417317#comment-16417317
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r177742687
--- Diff:
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java
---
@@ -244,7 +255,7 @@ public void run() {
while (System.nanoTime() < deadline);
// cancel the job & wait for the job to finish
-
JobManagerCommunicationUtils.cancelCurrentJob(flink.getLeaderGateway(timeout));
+ client.cancel(getRunningJobs(client).get(0));
--- End diff --
`Iterables.getOnlyElement(getRunningJobs(client))`?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16417318#comment-16417318
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r177743725
--- Diff:
flink-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaConsumerTestBase.java
---
@@ -132,8 +140,11 @@
* the same mini cluster. Otherwise, missing slots may happen.
*/
@Before
- public void ensureNoJobIsLingering() throws Exception {
-
JobManagerCommunicationUtils.waitUntilNoJobIsRunning(flink.getLeaderGateway(timeout));
+ public void setClientAndEnsureNoJobIsLingering() throws Exception {
+ client = flink.getClusterClient();
+ while (!getRunningJobs(client).isEmpty()){
--- End diff --
```
while (!getRunningJobs(client).isEmpty()){
Thread.sleep(50);
}
```
This is being copied pasted couple of times. Please extract to common
method. Maybe to an equivalent of `JobManagerCommunicationUtils` or even to
some `TestingClusterClient`?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406792#comment-16406792 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5701 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406391#comment-16406391 ] ASF GitHub Bot commented on FLINK-8703: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/5701 Sure @zentol. Won't squash them. Thanks for your work. Merging once Travis gave green light. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406321#comment-16406321 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5701 @tillrohrmann done. I wouldn't squash the savepoint port commit with the accumulator changes though. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406302#comment-16406302 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5701 let me quickly address the `getCOnfigurationSafe` comment and then let's merge them together. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406289#comment-16406289 ] ASF GitHub Bot commented on FLINK-8703: --- Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/5701 What about the other commit @zentol? Do you want me to commit them separately? I guess it would be easiest to commit everything together. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406287#comment-16406287
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175755400
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
// Submit the job
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
- JobSubmissionResult jobSubmissionResult =
cluster.submitJobDetached(jobGraph);
-
- StandaloneClusterClient clusterClient = new
StandaloneClusterClient(cluster.configuration());
- JobListeningContext jobListeningContext =
clusterClient.connectToJob(jobSubmissionResult.getJobID());
+ JobSubmissionResult jobSubmissionResult =
client.submitJob(jobGraph, SavepointMigrationTestBase.class.getClassLoader());
boolean done = false;
while (DEADLINE.hasTimeLeft()) {
// try and get a job result, this will fail if the job
already failed. Use this
// to get out of this loop
JobID jobId = jobSubmissionResult.getJobID();
- FiniteDuration timeout = FiniteDuration.apply(5,
TimeUnit.SECONDS);
try {
+ CompletableFuture jobStatusFuture =
client.getJobStatus(jobSubmissionResult.getJobID());
- Future future = clusterClient
- .getJobManagerGateway()
-
.ask(JobManagerMessages.getRequestJobStatus(jobSubmissionResult.getJobID()),
timeout);
-
- Object result = Await.result(future, timeout);
+ JobStatus jobStatus = jobStatusFuture.get(5,
TimeUnit.SECONDS);
- if (result instanceof
JobManagerMessages.CurrentJobStatus) {
- if
(((JobManagerMessages.CurrentJobStatus) result).status() == JobStatus.FAILED) {
- Object jobResult = Await.result(
-
jobListeningContext.getJobResultFuture(),
-
Duration.apply(5, TimeUnit.SECONDS));
- fail("Job failed: " +
jobResult);
- }
- }
+ assertNotEquals(JobStatus.FAILED, jobStatus);
} catch (Exception e) {
fail("Could not connect to job: " + e);
}
Thread.sleep(100);
--- End diff --
True, you're right. Sorry I didn't look closely enough at the test case.
Forget my comments here :-)
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406270#comment-16406270
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175751892
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
// Submit the job
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
- JobSubmissionResult jobSubmissionResult =
cluster.submitJobDetached(jobGraph);
-
- StandaloneClusterClient clusterClient = new
StandaloneClusterClient(cluster.configuration());
- JobListeningContext jobListeningContext =
clusterClient.connectToJob(jobSubmissionResult.getJobID());
+ JobSubmissionResult jobSubmissionResult =
client.submitJob(jobGraph, SavepointMigrationTestBase.class.getClassLoader());
boolean done = false;
while (DEADLINE.hasTimeLeft()) {
// try and get a job result, this will fail if the job
already failed. Use this
// to get out of this loop
JobID jobId = jobSubmissionResult.getJobID();
- FiniteDuration timeout = FiniteDuration.apply(5,
TimeUnit.SECONDS);
try {
+ CompletableFuture jobStatusFuture =
client.getJobStatus(jobSubmissionResult.getJobID());
- Future future = clusterClient
- .getJobManagerGateway()
-
.ask(JobManagerMessages.getRequestJobStatus(jobSubmissionResult.getJobID()),
timeout);
-
- Object result = Await.result(future, timeout);
+ JobStatus jobStatus = jobStatusFuture.get(5,
TimeUnit.SECONDS);
- if (result instanceof
JobManagerMessages.CurrentJobStatus) {
- if
(((JobManagerMessages.CurrentJobStatus) result).status() == JobStatus.FAILED) {
- Object jobResult = Await.result(
-
jobListeningContext.getJobResultFuture(),
-
Duration.apply(5, TimeUnit.SECONDS));
- fail("Job failed: " +
jobResult);
- }
- }
+ assertNotEquals(JobStatus.FAILED, jobStatus);
} catch (Exception e) {
fail("Could not connect to job: " + e);
}
Thread.sleep(100);
--- End diff --
If the job reaches `JobStatus.FAILED` the test fails.
`assertNotEquals(JobStatus.FAILED, jobStatus);`
We're polling the accumulators in a loop so long as the job is not failed
and the deadline wasn't reached yet.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406253#comment-16406253
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175747950
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
// Submit the job
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
- JobSubmissionResult jobSubmissionResult =
cluster.submitJobDetached(jobGraph);
-
- StandaloneClusterClient clusterClient = new
StandaloneClusterClient(cluster.configuration());
- JobListeningContext jobListeningContext =
clusterClient.connectToJob(jobSubmissionResult.getJobID());
+ JobSubmissionResult jobSubmissionResult =
client.submitJob(jobGraph, SavepointMigrationTestBase.class.getClassLoader());
boolean done = false;
while (DEADLINE.hasTimeLeft()) {
// try and get a job result, this will fail if the job
already failed. Use this
// to get out of this loop
JobID jobId = jobSubmissionResult.getJobID();
- FiniteDuration timeout = FiniteDuration.apply(5,
TimeUnit.SECONDS);
try {
+ CompletableFuture jobStatusFuture =
client.getJobStatus(jobSubmissionResult.getJobID());
- Future future = clusterClient
- .getJobManagerGateway()
-
.ask(JobManagerMessages.getRequestJobStatus(jobSubmissionResult.getJobID()),
timeout);
-
- Object result = Await.result(future, timeout);
+ JobStatus jobStatus = jobStatusFuture.get(5,
TimeUnit.SECONDS);
- if (result instanceof
JobManagerMessages.CurrentJobStatus) {
- if
(((JobManagerMessages.CurrentJobStatus) result).status() == JobStatus.FAILED) {
- Object jobResult = Await.result(
-
jobListeningContext.getJobResultFuture(),
-
Duration.apply(5, TimeUnit.SECONDS));
- fail("Job failed: " +
jobResult);
- }
- }
+ assertNotEquals(JobStatus.FAILED, jobStatus);
} catch (Exception e) {
fail("Could not connect to job: " + e);
}
Thread.sleep(100);
--- End diff --
How so? Don't we call it once after the job has reached the
`JobStatus.FAILED` test? Maybe it is actually not needed and can be removed.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406252#comment-16406252
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175747586
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/AccumulatorReport.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.runtime.accumulators.AccumulatorSnapshot;
+
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A report about the current values of all accumulators of the
TaskExecutor for a given job.
+ */
+public class AccumulatorReport implements Serializable,
Iterable {
--- End diff --
I think `SlotReport` is wrongly implementing this interface.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406195#comment-16406195
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175735031
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
// Submit the job
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
- JobSubmissionResult jobSubmissionResult =
cluster.submitJobDetached(jobGraph);
-
- StandaloneClusterClient clusterClient = new
StandaloneClusterClient(cluster.configuration());
- JobListeningContext jobListeningContext =
clusterClient.connectToJob(jobSubmissionResult.getJobID());
+ JobSubmissionResult jobSubmissionResult =
client.submitJob(jobGraph, SavepointMigrationTestBase.class.getClassLoader());
boolean done = false;
while (DEADLINE.hasTimeLeft()) {
// try and get a job result, this will fail if the job
already failed. Use this
// to get out of this loop
JobID jobId = jobSubmissionResult.getJobID();
- FiniteDuration timeout = FiniteDuration.apply(5,
TimeUnit.SECONDS);
try {
+ CompletableFuture jobStatusFuture =
client.getJobStatus(jobSubmissionResult.getJobID());
- Future future = clusterClient
- .getJobManagerGateway()
-
.ask(JobManagerMessages.getRequestJobStatus(jobSubmissionResult.getJobID()),
timeout);
-
- Object result = Await.result(future, timeout);
+ JobStatus jobStatus = jobStatusFuture.get(5,
TimeUnit.SECONDS);
- if (result instanceof
JobManagerMessages.CurrentJobStatus) {
- if
(((JobManagerMessages.CurrentJobStatus) result).status() == JobStatus.FAILED) {
- Object jobResult = Await.result(
-
jobListeningContext.getJobResultFuture(),
-
Duration.apply(5, TimeUnit.SECONDS));
- fail("Job failed: " +
jobResult);
- }
- }
+ assertNotEquals(JobStatus.FAILED, jobStatus);
} catch (Exception e) {
fail("Could not connect to job: " + e);
}
Thread.sleep(100);
--- End diff --
probably meant to prevent the test from spamming the cluster with
accumulator requests.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406196#comment-16406196 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5701 @tillrohrmann I've addressed your comments regarding the accumulator commit. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406184#comment-16406184
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175733627
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/AccumulatorReport.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.runtime.accumulators.AccumulatorSnapshot;
+
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A report about the current values of all accumulators of the
TaskExecutor for a given job.
+ */
+public class AccumulatorReport implements Serializable,
Iterable {
+ private final List accumulatorSnapshots;
--- End diff --
sure
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406187#comment-16406187
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175733781
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/AccumulatorReport.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.runtime.accumulators.AccumulatorSnapshot;
+
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A report about the current values of all accumulators of the
TaskExecutor for a given job.
+ */
+public class AccumulatorReport implements Serializable,
Iterable {
--- End diff --
I can change it. I followed the design of `SlotReport` which also
implements iterable even though it just holds a collection.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406175#comment-16406175
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175731427
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
// Submit the job
JobGraph jobGraph = env.getStreamGraph().getJobGraph();
jobGraph.setSavepointRestoreSettings(SavepointRestoreSettings.forPath(savepointPath));
- JobSubmissionResult jobSubmissionResult =
cluster.submitJobDetached(jobGraph);
-
- StandaloneClusterClient clusterClient = new
StandaloneClusterClient(cluster.configuration());
- JobListeningContext jobListeningContext =
clusterClient.connectToJob(jobSubmissionResult.getJobID());
+ JobSubmissionResult jobSubmissionResult =
client.submitJob(jobGraph, SavepointMigrationTestBase.class.getClassLoader());
boolean done = false;
while (DEADLINE.hasTimeLeft()) {
// try and get a job result, this will fail if the job
already failed. Use this
// to get out of this loop
JobID jobId = jobSubmissionResult.getJobID();
- FiniteDuration timeout = FiniteDuration.apply(5,
TimeUnit.SECONDS);
try {
+ CompletableFuture jobStatusFuture =
client.getJobStatus(jobSubmissionResult.getJobID());
- Future future = clusterClient
- .getJobManagerGateway()
-
.ask(JobManagerMessages.getRequestJobStatus(jobSubmissionResult.getJobID()),
timeout);
-
- Object result = Await.result(future, timeout);
+ JobStatus jobStatus = jobStatusFuture.get(5,
TimeUnit.SECONDS);
- if (result instanceof
JobManagerMessages.CurrentJobStatus) {
- if
(((JobManagerMessages.CurrentJobStatus) result).status() == JobStatus.FAILED) {
- Object jobResult = Await.result(
-
jobListeningContext.getJobResultFuture(),
-
Duration.apply(5, TimeUnit.SECONDS));
- fail("Job failed: " +
jobResult);
- }
- }
+ assertNotEquals(JobStatus.FAILED, jobStatus);
} catch (Exception e) {
fail("Could not connect to job: " + e);
}
Thread.sleep(100);
--- End diff --
For what do we need this sleep?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406174#comment-16406174
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175731028
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -87,17 +89,24 @@ protected static String getResourceFilename(String
filename) {
return resource.getFile();
}
- @Before
- public void setup() throws Exception {
+ private Configuration getConfigurationSafe() {
--- End diff --
I think we can get rid of this method by moving the `miniClusterResource`
initialization to the constructor which says that it can throw an `Exception`.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406173#comment-16406173
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175731826
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SavepointMigrationTestBase.java
---
@@ -194,49 +187,35 @@ protected final void restoreAndExecute(
String savepointPath,
Tuple2... expectedAccumulators) throws
Exception {
- // Retrieve the job manager
- Await.result(cluster.leaderGateway().future(),
DEADLINE.timeLeft());
+ ClusterClient client =
miniClusterResource.getClusterClient();
+ client.setDetached(true);
--- End diff --
As a side note which is out of scope for this issue. I think we should
`deprecate` `ClusterClient#setDetached`. It should not be an attribute of the
client but more of how you submit a job.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406162#comment-16406162
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175727728
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/AccumulatorReport.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.runtime.accumulators.AccumulatorSnapshot;
+
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A report about the current values of all accumulators of the
TaskExecutor for a given job.
+ */
+public class AccumulatorReport implements Serializable,
Iterable {
+ private final List accumulatorSnapshots;
--- End diff --
This could also be a `Collection` right?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406161#comment-16406161
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175730158
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/AccumulatorReport.java
---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.runtime.accumulators.AccumulatorSnapshot;
+
+import java.io.Serializable;
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A report about the current values of all accumulators of the
TaskExecutor for a given job.
+ */
+public class AccumulatorReport implements Serializable,
Iterable {
--- End diff --
I think it would be better to not implement `Iterable`. `AccumulatorReport`
is a value class which contains a `Collection` of `AccumulatorSnapshot`. Thus,
it should be as simple as this. By letting this class implement the `Iterable`
interface it basically means that `AccumulatorReport` can be used in places
where an `Iterable` is usable. I think this should not be the case for this
class. My concern is that we only let it implement this interface to cut a
corner when iterating over the list of `AccumulatorSnapshots`.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406160#comment-16406160
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/5701#discussion_r175729168
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
---
@@ -1515,8 +1516,22 @@ public void reportPayload(ResourceID resourceID,
Void payload) {
}
@Override
- public CompletableFuture retrievePayload(ResourceID
resourceID) {
- return CompletableFuture.completedFuture(null);
+ public CompletableFuture
retrievePayload(ResourceID resourceID) {
--- End diff --
Let's add `validateRunsInMainThread` as a first statement. That way we
enforce that this method really runs in the main thread context.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400010#comment-16400010 ] Chesnay Schepler commented on FLINK-8703: - Partially addressed in: master: aa86a86252881d5320e658b6f1315de7a62fac73 12cb09bd4d46da1979e3584622db584b0a316596 1e51e4369b8807489c415a7d8aae5e5cf4f66dfb 1.5: ba43d6bc586bb47c30c2b70f33cadf038f0323ef 44f7533db0d0b0791a463d68e28b76fb7622fb88 2c850d14c9c11e421ae832e0ea62004f9ef27426 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1646#comment-1646 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5667 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1645#comment-1645 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5668 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1644#comment-1644 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5666 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398739#comment-16398739 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5666 merging > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398738#comment-16398738 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5667 merging > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398735#comment-16398735 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5668 merging > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398667#comment-16398667 ] ASF GitHub Bot commented on FLINK-8703: --- Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/5667 LGTM 👍 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398666#comment-16398666 ] ASF GitHub Bot commented on FLINK-8703: --- Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/5668 LGTM 👍 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398660#comment-16398660 ] ASF GitHub Bot commented on FLINK-8703: --- Github user StefanRRichter commented on the issue: https://github.com/apache/flink/pull/5666 LGTM 👍 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395021#comment-16395021
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r173743909
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -66,40 +73,44 @@
private static final int NUM_TASK_MANAGERS = 2;
private static final int NUM_SLOTS = 4;
- private static LocalFlinkMiniCluster cluster;
+ private static final Configuration CLUSTER_CONFIGURATION =
getClusterConfiguration();
- private static int port = -1;
+ @ClassRule
+ public static final MiniClusterResource CLUSTER = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ CLUSTER_CONFIGURATION,
+ NUM_TASK_MANAGERS,
+ NUM_SLOTS),
+ true
+ );
- @BeforeClass
- public static void initialize() throws Exception {
+ private static Configuration getClusterConfiguration() {
Configuration config = new Configuration();
- config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
NUM_TASK_MANAGERS);
- config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS,
NUM_SLOTS);
- config.setLong(TaskManagerOptions.MANAGED_MEMORY_SIZE, 12L);
- config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);
-
- File logDir = File.createTempFile("TestBaseUtils-logdir", null);
- assertTrue("Unable to delete temp file", logDir.delete());
- assertTrue("Unable to create temp directory", logDir.mkdir());
- File logFile = new File(logDir, "jobmanager.log");
- File outFile = new File(logDir, "jobmanager.out");
-
- Files.createFile(logFile.toPath());
- Files.createFile(outFile.toPath());
+ try {
+ File logDir =
File.createTempFile("TestBaseUtils-logdir", null);
--- End diff --
Making it a rule is a bit tricky since you end up with a dependency between
rules as the cluster is also one.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392905#comment-16392905
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r173455497
--- Diff:
flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java
---
@@ -81,12 +81,6 @@ public void testCancelingEmptyTopic() throws Exception {
public void testCancelingFullTopic() throws Exception {
runCancelingOnFullInputTest();
}
-
--- End diff --
ah right, forgot to comment that. I found these test to be rather..._odd_.
They check behavior when not enough slots are available, but in the old code
afaik this fails before the client even submits the job, and in flip6 this
stalls as we never check whether enough slots are available (i guess with the
underlying assumption that we would just allocate more TMs until we have
enough).
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392804#comment-16392804
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5669#discussion_r173437603
--- Diff:
flink-connectors/flink-connector-kafka-0.10/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka010ITCase.java
---
@@ -81,12 +81,6 @@ public void testCancelingEmptyTopic() throws Exception {
public void testCancelingFullTopic() throws Exception {
runCancelingOnFullInputTest();
}
-
--- End diff --
Why are those removed?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392798#comment-16392798
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5665#discussion_r173436756
--- Diff:
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/WebFrontendITCase.java
---
@@ -66,40 +73,44 @@
private static final int NUM_TASK_MANAGERS = 2;
private static final int NUM_SLOTS = 4;
- private static LocalFlinkMiniCluster cluster;
+ private static final Configuration CLUSTER_CONFIGURATION =
getClusterConfiguration();
- private static int port = -1;
+ @ClassRule
+ public static final MiniClusterResource CLUSTER = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ CLUSTER_CONFIGURATION,
+ NUM_TASK_MANAGERS,
+ NUM_SLOTS),
+ true
+ );
- @BeforeClass
- public static void initialize() throws Exception {
+ private static Configuration getClusterConfiguration() {
Configuration config = new Configuration();
- config.setInteger(ConfigConstants.LOCAL_NUMBER_TASK_MANAGER,
NUM_TASK_MANAGERS);
- config.setInteger(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS,
NUM_SLOTS);
- config.setLong(TaskManagerOptions.MANAGED_MEMORY_SIZE, 12L);
- config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true);
-
- File logDir = File.createTempFile("TestBaseUtils-logdir", null);
- assertTrue("Unable to delete temp file", logDir.delete());
- assertTrue("Unable to create temp directory", logDir.mkdir());
- File logFile = new File(logDir, "jobmanager.log");
- File outFile = new File(logDir, "jobmanager.out");
-
- Files.createFile(logFile.toPath());
- Files.createFile(outFile.toPath());
+ try {
+ File logDir =
File.createTempFile("TestBaseUtils-logdir", null);
--- End diff --
This could use the `TemporaryFolder` `@Rule`, couldn't it?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392629#comment-16392629 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5669 [FLINK-8703][tests] Port KafkaTestBase to MiniClusterResource ## What is the purpose of the change Ports the `KafkaTestBase` and extending classes to use `MiniClusterResource`. ## Verifying this change Run all tests extending `KafkaTestBase` with `flip6` profile enabled/disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_kafkaB Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5669.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5669 commit 545b4dcccf79ce1b8bb530bae77ab4c2b6e85351 Author: zentol Date: 2018-03-07T12:38:03Z Remove Kafka testFailOnDeploy test commit 3cdfd906cdefa85fe36b6717f41e831ca7e5ea72 Author: zentol Date: 2018-03-07T12:39:25Z [FLINK-8703][tests] Port KafkaTestBase to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392625#comment-16392625 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5668 [FLINK-8703][tests] Port StreamingScalabilityAndLatency to MiniCluste… ## What is the purpose of the change Ports the `StreamingScalabilityAndLatency` to use `MiniClusterResource`. ## Verifying this change Run `StreamingScalabilityAndLatency` with `flip6` profile enabled/disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_stream Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5668.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5668 commit 08f56f6e954a37c9135bac891ad0bb6a31dcc8b0 Author: zentol Date: 2018-02-27T14:21:50Z [FLINK-8703][tests] Port StreamingScalabilityAndLatency to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392623#comment-16392623 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5667 [FLINK-8703][tests] Port NotSoMiniClusterIterations to MiniClusterResource ## What is the purpose of the change Ports the `NotSoMiniClusterIterations` to use `MiniClusterResource`. ## Verifying this change Run `NotSoMiniClusterIterations` with `flip6` profile enabled/disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_iter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5667.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5667 commit 858b2e4a31ad186b718c529a70757fda428a92ef Author: zentol Date: 2018-02-27T14:19:50Z [FLINK-8703][tests] Port NotSoMiniClusterIterations to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392621#comment-16392621 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5666 [FLINK-8703][tests] Port KafkaShortRetentionTestBase to MiniClusterRe… ## What is the purpose of the change Ports the `KafkaShortRetentionTestBase` to use `MiniClusterResource`. ## Verifying this change Run `KafkaShortRetentionTestBase` with `flip6` profile enabled/disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_kafke_short Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5666.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5666 commit 2aa3b9cdb39f812d40fb0be88265d1b36b631661 Author: zentol Date: 2018-02-27T10:11:59Z [FLINK-8703][tests] Port KafkaShortRetentionTestBase to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392620#comment-16392620 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5665 [FLINK-8703][tests] Port WebFrontendITCase to MiniClusterResource ## What is the purpose of the change Ports the `WebFrontendITCase` to use `MiniClusterResource`. ## Brief change log * implement prerequisite `MiniClusterClient#listJobs()` * modify `MiniClusterResource` to expose WebUI/REST port * make `MiniClusterResource#CODEBASE_KEY/FLIP6_CODEBASE` public to support version dependent test behavior (not pretty but the alternative would be a full copy of the test) * port WebFrontendITCase ## Verifying this change Run `WebFrontendITCase` with `flip6` profile enabled/disabled. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_web Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5665.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5665 commit 41c5737545897506470f531394269b94ebe2ef12 Author: zentol Date: 2018-03-07T10:05:12Z [FLINK-8811][tests] Implement MiniClusterClient#listJobs commit 98f366238cddcef38fff08b14764c9120dbcccea Author: zentol Date: 2018-03-07T10:14:20Z [FLINK-8703][tests] Expose WebUI port commit cb903f3681c808a3e5011211af6b7d174f86a23e Author: zentol Date: 2018-03-07T10:14:46Z [FLINK-8703][tests] Port WebFrontendITCase to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392612#comment-16392612
]
ASF GitHub Bot commented on FLINK-8703:
---
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/5664
[FLINK-8703][tests] Port CancelingTestBase to MiniClusterResource
## What is the purpose of the change
Ports the {{CancelingTestBase}} to use {{MiniClusterResource}}.
## Verifying this change
Run `MapCancelingTestBase` with `flip6` profile enabled/disabled.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 8703_canceling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5664.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5664
commit 57338df4819b2324f7ede2b131f81d83bc9096b2
Author: zentol
Date: 2018-02-26T14:36:37Z
[FLINK-8703][tests] Port CancelingTestBase to MiniClusterResource
commit 22d4a2f02c256eb41a1684a5766a1dd53dc9351d
Author: zentol
Date: 2018-02-28T12:43:42Z
[hotfix][tests] Properly disable JoinCancelingITCase
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386404#comment-16386404 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol closed the pull request at: https://github.com/apache/flink/pull/5636 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386031#comment-16386031 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5636 [FLINK-8703][tests] Port CancelingTestBase to MiniClusterResource ## What is the purpose of the change The `CancelingTestBase` now uses the `MiniClusterResource` and can be run against both legacy and flip6 clusters. ## Brief change log * Do not use singleActorSystem in LocalFlinkMiniCluster as this rendered the returned client to be unusable * port `CancelingTestBase` * properly disable JoinCancelingITCase ## Verifying this change Run `MapCancelingITCase` with flip6 profile on/off. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703_canceling Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5636.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5636 commit 24d95a00164f4b93ff30a237680cf4772855d7fc Author: zentol Date: 2018-03-05T12:45:33Z [hotfix][tests] Do not use singleActorSystem in LocalFlinkMiniCluster Using a singleActorSystem rendered the returned client unusable. commit 3fad83426a6356dee7966cf9e55d0de40b3bf6da Author: zentol Date: 2018-02-26T14:36:37Z [FLINK-8703][tests] Port CancelingTestBase to MiniClusterResource commit 07c77df577e62bcbfc4aeac3e5220151768319dd Author: zentol Date: 2018-02-28T12:43:42Z [hotfix][tests] Properly disable JoinCancelingITCase > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377472#comment-16377472 ] Chesnay Schepler commented on FLINK-8703: - second batch merged to master in 3e056b34f7be817e0d0eee612b6ae44891e33501 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377469#comment-16377469 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5542 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374154#comment-16374154 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5542 yes i think so, but I'll revert the `BigUserProgramJobSubmitITCase` port. It's not testing the actual codepath for job submission (which i guess would be the `RestClusterClient`) and thus provides no value. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374104#comment-16374104 ] ASF GitHub Bot commented on FLINK-8703: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5542 This is now good to go? > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371707#comment-16371707
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5542#discussion_r169714628
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BigUserProgramJobSubmitITCase.java
---
@@ -43,40 +40,27 @@
/**
* Integration test that verifies that a user program with a big(ger)
payload is successfully
* submitted and run.
+ *
+ * This test fails for non-flip6 cluster due to job submission payload
being too large, see [FLINK-7285].
*/
-@Ignore("Fails on job submission payload being too large - [FLINK-7285]")
+@Category(Flip6.class)
public class BigUserProgramJobSubmitITCase extends TestLogger {
//
// The mini cluster that is shared across tests
//
- private static final int DEFAULT_PARALLELISM = 1;
-
- private static LocalFlinkMiniCluster cluster;
-
- private static final Logger LOG =
LoggerFactory.getLogger(BigUserProgramJobSubmitITCase.class);
+ @ClassRule
+ public static final MiniClusterResource MINI_CLUSTER_RESOURCE = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ new Configuration(),
+ 1,
+ 1));
//
// Cluster setup & teardown
//
- @BeforeClass
- public static void setup() throws Exception {
- // make sure we do not use a singleActorSystem for the tests
- // (therefore, we cannot simply inherit from
StreamingMultipleProgramsTestBase)
--- End diff --
maybe it does succeed exactly because of that. If everything runs in the
same actorSystem the messagesa ren't subject to the `akka.framesize` limit.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371704#comment-16371704 ] Chesnay Schepler commented on FLINK-8703: - first batch merged to master in cb08aca3585bde903bc879b2009c7abdcd225019. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371703#comment-16371703 ] ASF GitHub Bot commented on FLINK-8703: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/5535 > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371461#comment-16371461
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5542#discussion_r169648970
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BigUserProgramJobSubmitITCase.java
---
@@ -43,40 +40,27 @@
/**
* Integration test that verifies that a user program with a big(ger)
payload is successfully
* submitted and run.
+ *
+ * This test fails for non-flip6 cluster due to job submission payload
being too large, see [FLINK-7285].
*/
-@Ignore("Fails on job submission payload being too large - [FLINK-7285]")
+@Category(Flip6.class)
public class BigUserProgramJobSubmitITCase extends TestLogger {
//
// The mini cluster that is shared across tests
//
- private static final int DEFAULT_PARALLELISM = 1;
-
- private static LocalFlinkMiniCluster cluster;
-
- private static final Logger LOG =
LoggerFactory.getLogger(BigUserProgramJobSubmitITCase.class);
+ @ClassRule
+ public static final MiniClusterResource MINI_CLUSTER_RESOURCE = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ new Configuration(),
+ 1,
+ 1));
//
// Cluster setup & teardown
//
- @BeforeClass
- public static void setup() throws Exception {
- // make sure we do not use a singleActorSystem for the tests
- // (therefore, we cannot simply inherit from
StreamingMultipleProgramsTestBase)
--- End diff --
Doesn't the `MiniClusterResource` always start the cluster with a single
actor system? I'm wondering why this test succeeds then.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371455#comment-16371455
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5542#discussion_r169647902
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BigUserProgramJobSubmitITCase.java
---
@@ -43,40 +40,27 @@
/**
* Integration test that verifies that a user program with a big(ger)
payload is successfully
* submitted and run.
+ *
+ * This test fails for non-flip6 cluster due to job submission payload
being too large, see [FLINK-7285].
*/
-@Ignore("Fails on job submission payload being too large - [FLINK-7285]")
+@Category(Flip6.class)
--- End diff --
Ah, and you're marking it as `Flip6` because it doesn't succeed with the
legacy components?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371344#comment-16371344 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5542 Together with #5535 this should cover all usages of `LocalFlinkMiniCluster` that we can port in isolation. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371338#comment-16371338
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5542#discussion_r169623190
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/BigUserProgramJobSubmitITCase.java
---
@@ -43,40 +40,27 @@
/**
* Integration test that verifies that a user program with a big(ger)
payload is successfully
* submitted and run.
+ *
+ * This test fails for non-flip6 cluster due to job submission payload
being too large, see [FLINK-7285].
*/
-@Ignore("Fails on job submission payload being too large - [FLINK-7285]")
+@Category(Flip6.class)
--- End diff --
this test actually succeeds with Flip6 so i figured we may just as well
enable it.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371336#comment-16371336 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5542 [FLINK-8703][tests] Migrate tests from LocalFlinkMiniCluster to MiniClusterResource ## What is the purpose of the change This PR contains the second batch of test ports from `LocalFlinkMiniCluster` to `MiniClusterResource`. No port required changes to `MiniClusterResource`. Each test has it's own commit and is self-contained. If a test was not explicitly creating a configuration and the number of taskmanagers/slots matched the settings in `AbstractTestBase` the test was modified to extend that. Otherwise, a separate `MiniClusterResource` `@ClassRule`/`@Rule` was created. ## Verifying this change This change is covered by existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703b Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5542.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5542 commit 038dc7ef91159dc8099b35c74164dd7331e3c5fa Author: zentol Date: 2018-02-20T17:02:32Z [FLINK-8703][tests] Port CustomDistributionITCase to MiniClusterResource commit b7e52de03ad19f61a6c2cc07b40a115e14ae2ca0 Author: zentol Date: 2018-02-20T17:02:51Z [FLINK-8703][tests] Port BigUserProgramJobSubmitITCase to MiniClusterResource commit 4d9b92b97384b5cffed6fe95c0463ba8113584ef Author: zentol Date: 2018-02-20T17:03:08Z [FLINK-8703][tests] Port scala FlinkTestBase to MiniClusterResource commit 9268219bf1069d4d70f1a662993ee5db41df1e2d Author: zentol Date: 2018-02-20T17:05:28Z [FLINK-8703][tests] Port ScalaStreamingMultipleProgramsTestBase to MiniClusterResource commit cb099561877517babb2a6d248b9dc34098701d78 Author: zentol Date: 2018-02-20T17:24:17Z [FLINK-8703][tests] Port AbstractEventTimeWindowCheckpointingITCase to MiniClusterResource commit 70a9ad0e38a3673038096721e35b92f24b7f1798 Author: zentol Date: 2018-02-20T17:33:22Z [FLINK-8703][tests] Port IPv6HostnamesITCase to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371333#comment-16371333 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5535 will merge this, as is, once travis gives a green light. The commits have the rather nice property that they don't require changes to other classes (like the MiniClusterResource); wouldn't want to destroy that by making parallelism protected. As for deriving the parallelism, we could do it but there are in general numerous things we could clean up in these tests. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371185#comment-16371185
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169589778
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/SuccessAfterNetworkBuffersFailureITCase.java
---
@@ -50,22 +50,27 @@
public class SuccessAfterNetworkBuffersFailureITCase extends TestLogger {
private static final int PARALLELISM = 16;
+
+ @ClassRule
+ public static final MiniClusterResource MINI_CLUSTER_RESOURCE = new
MiniClusterResource(
+ new MiniClusterResource.MiniClusterResourceConfiguration(
+ getConfiguration(),
+ 2,
--- End diff --
This could be derived from `PARALLELISM`
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371183#comment-16371183
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169589443
--- Diff:
flink-libraries/flink-gelly/src/test/java/org/apache/flink/graph/test/operations/DegreesWithExceptionITCase.java
---
@@ -38,29 +32,10 @@
* Test expected errors for {@link Graph#inDegrees()},
* {@link Graph#outDegrees()}, and {@link Graph#getDegrees()}.
*/
-public class DegreesWithExceptionITCase extends TestLogger {
+public class DegreesWithExceptionITCase extends AbstractTestBase {
private static final int PARALLELISM = 4;
--- End diff --
I think this could be replaced in all the tests by making
`DEFAULT_PARALLELISM` in `AbstractTestBase` protected and using that instead.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371124#comment-16371124 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5535 I'm removing `AccumulatorErrorITCase` from the port due to FLINK-8721. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371102#comment-16371102 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5535 Thanks @zentol. Yea. I agree with you. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371093#comment-16371093 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5535 And we also seems lost refactor ```WebFrontendITCase``` which still use ```LocalFlinkMiniCluster```. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371095#comment-16371095 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5535 yes, as mentioned in PR the description this does not port all tests. There are several tests that cannot be ported in isolation as they access specific methods on the `LocalFlinkMiniCluster` or communicate directly with components via akka. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371091#comment-16371091 ] ASF GitHub Bot commented on FLINK-8703: --- Github user zhangminglei commented on the issue: https://github.com/apache/flink/pull/5535 JobManagerHACheckpointRecoveryITCase still use ```LocalFlinkMiniCluster``` as it's default one. And we seem to be lost refactor it. And I dont think it is a very easy work since in this ITCase, it use ```LocalFlinkMiniCluster.getLeaderGateway```. And in the new pattern of ```AbstractTestBase```. I dont know how to get that. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370205#comment-16370205
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169370975
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/MiscellaneousIssuesITCase.java
---
@@ -134,7 +106,8 @@ public void testAccumulatorsAfterNoOp() {
final String accName = "test_accumulator";
try {
- env.setParallelism(6);
+ ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(4);
--- End diff --
yeah, that's probably the better option.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370196#comment-16370196
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169369295
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/MiscellaneousIssuesITCase.java
---
@@ -134,7 +106,8 @@ public void testAccumulatorsAfterNoOp() {
final String accName = "test_accumulator";
try {
- env.setParallelism(6);
+ ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(4);
--- End diff --
Why not start a mini cluster with 2 TMs and 3 slots as before?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370125#comment-16370125
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169346590
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/experimental/CollectITCase.java
---
@@ -38,33 +35,23 @@
* This experimental class is relocated from flink-streaming-contrib.
Please see package-info.java
* for more information.
*/
-public class CollectITCase extends TestLogger {
+public class CollectITCase extends AbstractTestBase {
--- End diff --
Thanks @zentol.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370117#comment-16370117
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169341512
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/experimental/CollectITCase.java
---
@@ -38,33 +35,23 @@
* This experimental class is relocated from flink-streaming-contrib.
Please see package-info.java
* for more information.
*/
-public class CollectITCase extends TestLogger {
+public class CollectITCase extends AbstractTestBase {
--- End diff --
Please have a look at the javadocs for AbstractTestBase.
The logs should still be printed as `AbstractTestBase` extends
`TestBaseUtils` which extends `TestLogger`.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370083#comment-16370083
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169328894
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/experimental/CollectITCase.java
---
@@ -38,33 +35,23 @@
* This experimental class is relocated from flink-streaming-contrib.
Please see package-info.java
* for more information.
*/
-public class CollectITCase extends TestLogger {
+public class CollectITCase extends AbstractTestBase {
--- End diff --
One thing I found the difference from them , If we extends AbstractTestBase
class. No logs will print to console.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370076#comment-16370076
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169327476
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/streaming/experimental/CollectITCase.java
---
@@ -38,33 +35,23 @@
* This experimental class is relocated from flink-streaming-contrib.
Please see package-info.java
* for more information.
*/
-public class CollectITCase extends TestLogger {
+public class CollectITCase extends AbstractTestBase {
--- End diff --
It seems that we do not use anything in AbstractTestBase class. Why do we
still extend this class ?
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370033#comment-16370033
]
ASF GitHub Bot commented on FLINK-8703:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5535#discussion_r169312549
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/misc/MiscellaneousIssuesITCase.java
---
@@ -134,7 +106,8 @@ public void testAccumulatorsAfterNoOp() {
final String accName = "test_accumulator";
try {
- env.setParallelism(6);
+ ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+ env.setParallelism(4);
--- End diff --
i had to reduce the parallelism to 4 as it failed with 6 due to not enough
slots.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16370029#comment-16370029 ] ASF GitHub Bot commented on FLINK-8703: --- GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5535 [FLINK-8703][tests] Migrate tests from LocalFlinkMiniCluster to MiniClusterResource ## What is the purpose of the change This PR contains the first batch of test ports from `LocalFlinkMiniCluster` to `MiniClusterResource`. No port required changes to `MiniClusterResource`. Each test has it's own commit. If a test was not explicitly creating a configuration and the number of taskmanagers/slots matched the settings in `AbstractTestBase` the test was modified to extend that. Otherwise, a separate `MiniClusterResource` `@ClassRule` was created. ## Verifying this change This change is covered by existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 8703 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5535.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5535 commit 2b4168c932304d0a855d330cfe902b361cf7b174 Author: zentol Date: 2018-02-20T12:41:45Z [FLINK-8703][tests] Port CollectITCase to AbstractTestBase commit f00f69ca1eb347d4d235cb24bb1d49656548cc98 Author: zentol Date: 2018-02-20T12:42:01Z [FLINK-8703][tests] Port MiscellaneuousIssuesITCase to AbstractTestBase commit e6b8751b982e3b80c74b7e93ef26e60cd9059b37 Author: zentol Date: 2018-02-20T12:43:01Z [FLINK-8703][tests] Port StreamFaultToleranceTestBase to MiniClusterResource commit a2163be631695f78da7f40cebae1c62a3016fe17 Author: zentol Date: 2018-02-20T12:43:17Z [FLINK-8703][tests] Port KeyedStateCheckpointITCase to MiniClusterResource commit f85d9577df4002e98c34606984ee69573a78a6f7 Author: zentol Date: 2018-02-20T12:43:34Z [FLINK-8703][tests] Port AccumulatorErrorITCase to AbstractTestBase commit 06dfdb7c75298c1b501599a25e7fdae93de92c5f Author: zentol Date: 2018-02-20T12:46:10Z [FLINK-8703][tests] Port FastFailuresITCase to MiniClusterResource commit 12e29e0fec70824415216ee738853d45b204fc03 Author: zentol Date: 2018-02-20T12:58:40Z [FLINK-8703][tests] Port SimpleRecoveryITCaseBase to MiniClusterResource > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Assignee: Chesnay Schepler >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369812#comment-16369812 ] mingleizhang commented on FLINK-8703: - You are welcome, I just need a check for what I think in my mind. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[ https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369811#comment-16369811 ] Aljoscha Krettek commented on FLINK-8703: - [~mingleizhang] Yes, this is what I meant. I'm sorry for the lack of description but I put a bit more text in the parent issue: FLINK-8700. > Migrate tests from LocalFlinkMiniCluster to MiniClusterResource > --- > > Key: FLINK-8703 > URL: https://issues.apache.org/jira/browse/FLINK-8703 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Aljoscha Krettek >Priority: Blocker > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-8703) Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
[
https://issues.apache.org/jira/browse/FLINK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369803#comment-16369803
]
mingleizhang commented on FLINK-8703:
-
Hi, [~aljoscha] I would like to know, what you mean in this jira is making the
original ITCases that used {{LocalFlinkMiniCluster }} change to the new Cluster
{{MiniClusterResource}} ? Thanks.
> Migrate tests from LocalFlinkMiniCluster to MiniClusterResource
> ---
>
> Key: FLINK-8703
> URL: https://issues.apache.org/jira/browse/FLINK-8703
> Project: Flink
> Issue Type: Sub-task
> Components: Tests
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.5.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
