[GitHub] zentol commented on a change in pull request #6415: [FLINK-8974] Run all-round DataSet job with failures in HA mode
zentol commented on a change in pull request #6415: [FLINK-8974] Run all-round
DataSet job with failures in HA mode
URL: https://github.com/apache/flink/pull/6415#discussion_r205116733
##
File path:
flink-end-to-end-tests/flink-dataset-allround-test/src/main/java/org/apache/flink/batch/tests/DataSetAllroundTestProgram.java
##
@@ -66,14 +61,21 @@ public static void main(String[] args) throws Exception {
ParameterTool params = ParameterTool.fromArgs(args);
int loadFactor =
Integer.parseInt(params.getRequired("loadFactor"));
String outputPath = params.getRequired("outputPath");
+ boolean infinite = params.getBoolean("infinite", false);
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
int numKeys = loadFactor * 128 * 1024;
- DataSet> x1Keys = env.createInput(new
Generator(numKeys, 1)).setParallelism(4);
+ DataSet> x1Keys;
DataSet> x2Keys = env.createInput(new
Generator(numKeys * 32, 2)).setParallelism(4);
DataSet> x8Keys = env.createInput(new
Generator(numKeys, 8)).setParallelism(4);
+ if (infinite) {
+ x1Keys =
env.createInput(Generator.infinite()).setParallelism(4).filter(t -> t.f1 >= 0);
Review comment:
that shouldn't be possible due to back-pressure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] zentol commented on a change in pull request #6415: [FLINK-8974] Run all-round DataSet job with failures in HA mode
zentol commented on a change in pull request #6415: [FLINK-8974] Run all-round
DataSet job with failures in HA mode
URL: https://github.com/apache/flink/pull/6415#discussion_r205116908
##
File path:
flink-end-to-end-tests/flink-dataset-allround-test/src/main/java/org/apache/flink/batch/tests/DataSetAllroundTestProgram.java
##
@@ -66,14 +61,21 @@ public static void main(String[] args) throws Exception {
ParameterTool params = ParameterTool.fromArgs(args);
int loadFactor =
Integer.parseInt(params.getRequired("loadFactor"));
String outputPath = params.getRequired("outputPath");
+ boolean infinite = params.getBoolean("infinite", false);
ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
int numKeys = loadFactor * 128 * 1024;
- DataSet> x1Keys = env.createInput(new
Generator(numKeys, 1)).setParallelism(4);
+ DataSet> x1Keys;
DataSet> x2Keys = env.createInput(new
Generator(numKeys * 32, 2)).setParallelism(4);
DataSet> x8Keys = env.createInput(new
Generator(numKeys, 8)).setParallelism(4);
+ if (infinite) {
+ x1Keys =
env.createInput(Generator.infinite()).setParallelism(4).filter(t -> t.f1 >= 0);
Review comment:
IIRC the deployment of operators in the batch API is also dependent on the
passing of data.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
