[flink-web] branch asf-site updated (8730f94 -> 357b4c6)
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git. from 8730f94 [FLINK-7391] Normalize release entries in Downloads page new 127e99b Remove completed item (removal of Hadoop convenience builds) from roadmap. new 357b4c6 Rebuild website The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: content/roadmap.html| 5 - content/zh/roadmap.html | 5 - roadmap.md | 4 roadmap.zh.md | 4 4 files changed, 18 deletions(-)
[flink-web] 02/02: Rebuild website
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit 357b4c6ac576f97133c068f5259dd01bb8c7f8af Author: Fabian Hueske AuthorDate: Tue Apr 30 16:32:07 2019 +0200 Rebuild website --- content/roadmap.html| 5 - content/zh/roadmap.html | 5 - 2 files changed, 10 deletions(-) diff --git a/content/roadmap.html b/content/roadmap.html index b3760f8..adba88f 100644 --- a/content/roadmap.html +++ b/content/roadmap.html @@ -346,11 +346,6 @@ metastore and Hive UDF support https://issues.apache.org/jira/browse/FL -We are changing the build setup to not bundle Hadoop by default, but rather offer pre-packaged Hadoop -libraries for the use with Yarn, HDFS, etc. as convenience downloads -https://issues.apache.org/jira/browse/FLINK-11266";>FLINK-11266. - - The Flink code base is being updates to support Java 9, 10, and 11 https://issues.apache.org/jira/browse/FLINK-8033";>FLINK-8033, https://issues.apache.org/jira/browse/FLINK-10725";>FLINK-10725. diff --git a/content/zh/roadmap.html b/content/zh/roadmap.html index 26dba2c..22efd3a 100644 --- a/content/zh/roadmap.html +++ b/content/zh/roadmap.html @@ -344,11 +344,6 @@ metastore and Hive UDF support https://issues.apache.org/jira/browse/FL -We are changing the build setup to not bundle Hadoop by default, but rather offer pre-packaged Hadoop -libraries for the use with Yarn, HDFS, etc. as convenience downloads -https://issues.apache.org/jira/browse/FLINK-11266";>FLINK-11266. - - The Flink code base is being updates to support Java 9, 10, and 11 https://issues.apache.org/jira/browse/FLINK-8033";>FLINK-8033, https://issues.apache.org/jira/browse/FLINK-10725";>FLINK-10725.
[flink-web] 01/02: Remove completed item (removal of Hadoop convenience builds) from roadmap.
This is an automated email from the ASF dual-hosted git repository. fhueske pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit 127e99bfd4300c4ae272a611c9ddcb1076553478 Author: Fabian Hueske AuthorDate: Mon Apr 29 17:10:11 2019 +0200 Remove completed item (removal of Hadoop convenience builds) from roadmap. This closes #203. --- roadmap.md| 4 roadmap.zh.md | 4 2 files changed, 8 deletions(-) diff --git a/roadmap.md b/roadmap.md index 7fdd3cd..2ef95c2 100644 --- a/roadmap.md +++ b/roadmap.md @@ -160,10 +160,6 @@ Support for additional connectors and formats is a continuous process. # Miscellaneous - - We are changing the build setup to not bundle Hadoop by default, but rather offer pre-packaged Hadoop -libraries for the use with Yarn, HDFS, etc. as convenience downloads -[FLINK-11266](https://issues.apache.org/jira/browse/FLINK-11266). - - The Flink code base is being updates to support Java 9, 10, and 11 [FLINK-8033](https://issues.apache.org/jira/browse/FLINK-8033), [FLINK-10725](https://issues.apache.org/jira/browse/FLINK-10725). diff --git a/roadmap.zh.md b/roadmap.zh.md index 40d0edb..6508c6e 100644 --- a/roadmap.zh.md +++ b/roadmap.zh.md @@ -160,10 +160,6 @@ Support for additional connectors and formats is a continuous process. # Miscellaneous - - We are changing the build setup to not bundle Hadoop by default, but rather offer pre-packaged Hadoop -libraries for the use with Yarn, HDFS, etc. as convenience downloads -[FLINK-11266](https://issues.apache.org/jira/browse/FLINK-11266). - - The Flink code base is being updates to support Java 9, 10, and 11 [FLINK-8033](https://issues.apache.org/jira/browse/FLINK-8033), [FLINK-10725](https://issues.apache.org/jira/browse/FLINK-10725).
[flink] branch master updated: [FLINK-12203] Refactor ResultPartitionManager to break tie with Task
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new b62db93 [FLINK-12203] Refactor ResultPartitionManager to break tie with Task b62db93 is described below commit b62db93bf63cb3bb34dd03d611a779d9e3fc61ac Author: Andrey Zagrebin AuthorDate: Thu Apr 18 15:26:24 2019 +0200 [FLINK-12203] Refactor ResultPartitionManager to break tie with Task At the moment, we have ResultPartitionManager.releasePartitionsProducedBy which uses indexing by task in network environment. These methods are eventually used only by Task which already knows its partitions so Task can use ResultPartition.fail(cause) and TaskExecutor.failPartition could directly use NetworkEnviroment.releasePartitions(Collection). This also requires that JM Execution sends produced partition ids instead of just ExecutionAttemptID. This closes #8210. --- .../flink/runtime/executiongraph/Execution.java| 20 ++--- .../runtime/io/network/NetworkEnvironment.java | 13 ++ .../io/network/partition/ResultPartition.java | 2 +- .../network/partition/ResultPartitionManager.java | 50 ++ .../jobmanager/slots/TaskManagerGateway.java | 8 ++-- .../runtime/jobmaster/RpcTaskManagerGateway.java | 6 ++- .../flink/runtime/taskexecutor/TaskExecutor.java | 7 ++- .../runtime/taskexecutor/TaskExecutorGateway.java | 8 ++-- .../utils/SimpleAckingTaskManagerGateway.java | 5 ++- .../taskexecutor/TestingTaskExecutorGateway.java | 4 +- 10 files changed, 65 insertions(+), 58 deletions(-) diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java index 63f3125..e413619 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/Execution.java @@ -37,6 +37,7 @@ import org.apache.flink.runtime.concurrent.FutureUtils; import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor; import org.apache.flink.runtime.execution.ExecutionState; import org.apache.flink.runtime.instance.SlotSharingGroupId; +import org.apache.flink.runtime.io.network.partition.ResultPartitionID; import org.apache.flink.runtime.jobmanager.scheduler.CoLocationConstraint; import org.apache.flink.runtime.jobmanager.scheduler.LocationPreferenceConstraint; import org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException; @@ -688,7 +689,7 @@ public class Execution implements AccessExecution, Archiveable partitions = vertex.getProducedPartitions().values(); + Collection partitionIds = new ArrayList<>(partitions.size()); + for (IntermediateResultPartition partition : partitions) { + partitionIds.add(new ResultPartitionID(partition.getPartitionId(), attemptId)); + } + + if (!partitionIds.isEmpty()) { + // TODO For some tests this could be a problem when querying too early if all resources were released + taskManagerGateway.releasePartitions(partitionIds); + } } } diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java index 98c61a4..0ee8595 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/NetworkEnvironment.java @@ -26,6 +26,7 @@ import org.apache.flink.runtime.io.network.buffer.NetworkBufferPool; import org.apache.flink.runtime.io.network.netty.NettyConfig; import org.apache.flink.runtime.io.network.netty.NettyConnectionManager; import org.apache.flink.runtime.io.network.partition.ResultPartition; +import org.apache.flink.runtime.io.network.partition.ResultPartitionID; import org.apache.flink.runtime.io.network.partition.ResultPartitionManager; import org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate; import org.apache.flink.runtime.taskexecutor.TaskExecutor; @@ -38,6 +39,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.IOException; +import java.util.Collection; import java.util.Optional; import static org.apache.flink.util.Preconditions.checkNotNull; @@ -207,6 +209,17 @@ public class NetworkEnvironment { } } + /** +* Batch release intermediate result partitions. +* +* @param partitionIds partition ids to release +*/ + public void releasePart
[flink] branch master updated: [hotfix] regenerate rest-docs to latest code
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 3c20704 [hotfix] regenerate rest-docs to latest code 3c20704 is described below commit 3c207041c350227743bcbe8e36111657b1dfb371 Author: Yun Tang AuthorDate: Sun Apr 28 23:12:13 2019 +0800 [hotfix] regenerate rest-docs to latest code --- docs/_includes/generated/rest_v1_dispatcher.html | 159 ++- 1 file changed, 157 insertions(+), 2 deletions(-) diff --git a/docs/_includes/generated/rest_v1_dispatcher.html b/docs/_includes/generated/rest_v1_dispatcher.html index 36ffe51..4df103d 100644 --- a/docs/_includes/generated/rest_v1_dispatcher.html +++ b/docs/_includes/generated/rest_v1_dispatcher.html @@ -355,6 +355,96 @@ Using 'curl' you can upload a jar via 'curl -X POST -H "Expect:" -F "jarfile=@pa + /jars/:jarid/plan + + + Verb: GET + Response code: 200 OK + + + Returns the dataflow plan of a job contained in a jar previously uploaded via '/jars/upload'. Program arguments can be passed both via the JSON request (recommended) or query parameters. + + + Path parameters + + + + +jarid - String value that identifies a jar. When uploading the jar a path is returned, where the filename is the ID. This value is equivalent to the `id` field in the list of uploaded jars (/jars). + + + + + Query parameters + + + + +program-args (optional): Deprecated, please use 'programArg' instead. String value that specifies the arguments for the program or plan +programArg (optional): Comma-separated list of program arguments. +entry-class (optional): String value that specifies the fully qualified name of the entry point class. Overrides the class defined in the jar file manifest. +parallelism (optional): Positive integer value that specifies the desired parallelism for the job. + + + + + +Request + + + +{ + "type" : "object", + "id" : "urn:jsonschema:org:apache:flink:runtime:webmonitor:handlers:JarPlanRequestBody", + "properties" : { +"entryClass" : { + "type" : "string" +}, +"programArgs" : { + "type" : "string" +}, +"programArgsList" : { + "type" : "array", + "items" : { +"type" : "string" + } +}, +"parallelism" : { + "type" : "integer" +}, +"jobId" : { + "type" : "any" +} + } +} + + + + + + +Response + + + +{ + "type" : "object", + "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:JobPlanInfo", + "properties" : { +"plan" : { + "type" : "any" +} + } +} + + + + + + + + + /jars/:jarid/run @@ -592,7 +682,7 @@ Using 'curl' you can upload a jar via 'curl -X POST -H "Expect:" -F "jarfile=@pa }, "status" : { "type" : "string", -"enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDING", "SUSPENDED", "RECONCILING" ] +"enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ] } } } @@ -829,7 +919,7 @@ Using 'curl' you can upload a jar via 'curl -X POST -H "Expect:" -F "jarfile=@pa }, "state" : { "type" : "string", - "enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDING", "SUSPENDED", "RECONCILING" ] + "enum" : [ "CREATED", "RUNNING", "FAILING", "FAILED", "CANCELLING", "CANCELED", "FINISHED", "RESTARTING", "SUSPENDED", "RECONCILING" ] }, "start-time" : { "type" : "integer" @@ -2295,6 +2385,71 @@ Using 'curl' you can upload a jar via 'curl -X POST -H "Expect:" -F "jarfile=@pa + /jobs/:jobid/stop-with-savepoint + + + Verb: POST + Response code: 202 Accepted + + + Stops a job with a savepoint. Optionally, it can also emit a MAX_WATERMARK before taking the savepoint to flush out any state waiting for timers to fire. + + + Path parameters + + + + +jobid - 32-character hexadecimal string value that identifies a job. + + + + + +Request + + + +{ + "type" : "object", + "id" : "urn:jsonschema:org:apache:flink:runtime:rest:messages:job:savepoints:stop:StopWithSavepointRequestBody", + "properties" : { +"targetDirectory" : { + "type" : "string" +}, +"endOfEventTime" : { + "type" : "boolean" +
[flink] branch master updated: [FLINK-12187] Bug fixes, Use BufferedWriter in a loop instead of FileWriter
This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new ca8145e [FLINK-12187] Bug fixes, Use BufferedWriter in a loop instead of FileWriter ca8145e is described below commit ca8145e54607aa46b275d09e785e7c1653f0181c Author: bd2019us AuthorDate: Sat Apr 13 23:33:42 2019 -0500 [FLINK-12187] Bug fixes, Use BufferedWriter in a loop instead of FileWriter --- .../flink/examples/java/relational/util/WebLogDataGenerator.java | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java b/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java index f68ece1..7bc7ca9 100644 --- a/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java +++ b/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java @@ -20,6 +20,7 @@ package org.apache.flink.examples.java.relational.util; import org.apache.flink.examples.java.relational.WebLogAnalysis; +import java.io.BufferedWriter; import java.io.FileWriter; import java.io.IOException; import java.util.Calendar; @@ -98,7 +99,7 @@ public class WebLogDataGenerator { Random rand = new Random(Calendar.getInstance().getTimeInMillis()); - try (FileWriter fw = new FileWriter(path)) { + try (BufferedWriter fw = new BufferedWriter(new FileWriter(path))) { for (int i = 0; i < noDocs; i++) { int wordsInDoc = rand.nextInt(40) + 10; @@ -136,7 +137,7 @@ public class WebLogDataGenerator { Random rand = new Random(Calendar.getInstance().getTimeInMillis()); - try (FileWriter fw = new FileWriter(path)) { + try (BufferedWriter fw = new BufferedWriter(new FileWriter(path))) { for (int i = 0; i < noDocs; i++) { // Rank StringBuilder rank = new StringBuilder(rand.nextInt(100) + "|"); @@ -168,7 +169,7 @@ public class WebLogDataGenerator { Random rand = new Random(Calendar.getInstance().getTimeInMillis()); - try (FileWriter fw = new FileWriter(path)) { + try (BufferedWriter fw = new BufferedWriter(new FileWriter(path))) { for (int i = 0; i < noVisits; i++) { int year = 2000 + rand.nextInt(10); // yearFilter 3
[flink] branch release-1.8 updated: [hotfix] Fix compile error from rebase of FLINK-12296]
This is an automated email from the ASF dual-hosted git repository. srichter pushed a commit to branch release-1.8 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.8 by this push: new 4d1605b [hotfix] Fix compile error from rebase of FLINK-12296] 4d1605b is described below commit 4d1605bf52ec03b5e01d0bb950f279a3e6da9471 Author: Stefan Richter AuthorDate: Tue Apr 30 11:26:34 2019 +0200 [hotfix] Fix compile error from rebase of FLINK-12296] --- .../org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flink-tests/src/test/java/org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java b/flink-tests/src/test/java/org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java index 5651929..c3e1ba9 100644 --- a/flink-tests/src/test/java/org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java +++ b/flink-tests/src/test/java/org/apache/flink/test/state/StatefulOperatorChainedTaskTest.java @@ -176,7 +176,7 @@ public class StatefulOperatorChainedTaskTest { testHarness.getTaskStateManager().setWaitForReportLatch(new OneShotLatch()); - while (!streamTask.triggerCheckpoint(checkpointMetaData, CheckpointOptions.forCheckpointWithDefaultLocation(), false)) {} + while (!streamTask.triggerCheckpoint(checkpointMetaData, CheckpointOptions.forCheckpointWithDefaultLocation())) {} testHarness.getTaskStateManager().getWaitForReportLatch().await(); long reportedCheckpointId = testHarness.getTaskStateManager().getReportedCheckpointId();
[flink] branch master updated: [FLINK-12238][hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module
This is an automated email from the ASF dual-hosted git repository. kurt pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 39d8236 [FLINK-12238][hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module 39d8236 is described below commit 39d82368eca3891d27c85dd9bc56db2344ee73ba Author: Bowen L AuthorDate: Tue Apr 30 01:19:39 2019 -0700 [FLINK-12238][hive] Support database related operations in GenericHiveMetastoreCatalog and setup flink-connector-hive module This closes #8205 --- flink-connectors/flink-connector-hive/pom.xml | 429 + .../catalog/hive/GenericHiveMetastoreCatalog.java | 352 + .../hive/GenericHiveMetastoreCatalogUtil.java | 49 +++ .../src/main/resources/META-INF/NOTICE | 26 ++ .../main/resources/META-INF/licenses/LICENSE.antlr | 38 ++ .../hive/GenericHiveMetastoreCatalogTest.java | 83 .../flink/table/catalog/hive/HiveTestUtils.java| 54 +++ .../src/test/resources/hive-site.xml | 42 ++ .../src/test/resources/log4j-test.properties | 24 ++ flink-connectors/pom.xml | 1 + flink-table/flink-table-api-java/pom.xml | 10 + .../table/catalog/GenericCatalogDatabase.java | 2 +- .../table/catalog/GenericInMemoryCatalogTest.java | 248 +++- .../flink/table/catalog/CatalogTestBase.java | 241 14 files changed, 1396 insertions(+), 203 deletions(-) diff --git a/flink-connectors/flink-connector-hive/pom.xml b/flink-connectors/flink-connector-hive/pom.xml new file mode 100644 index 000..cb09934 --- /dev/null +++ b/flink-connectors/flink-connector-hive/pom.xml @@ -0,0 +1,429 @@ + + +http://maven.apache.org/POM/4.0.0"; +xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; +xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + + 4.0.0 + + + org.apache.flink + flink-connectors + 1.9-SNAPSHOT + .. + + + flink-connector-hive_${scala.binary.version} + flink-connector-hive + + jar + + + 2.3.4 + 2.7.2 + + + + + + + + org.apache.flink + flink-table-common + ${project.version} + provided + + + + org.apache.flink + flink-table-api-java + ${project.version} + provided + + + + + + + org.apache.hadoop + hadoop-common + ${hivemetastore.hadoop.version} + provided + + + + org.apache.hadoop + hadoop-mapreduce-client-core + ${hivemetastore.hadoop.version} + provided + + + + + + + org.apache.hive + hive-metastore + ${hive.version} + + + org.apache.hive + hive-shims + + + javolution + javolution + + + com.google.guava + guava + + + com.google.protobuf + protobuf-java + + + org.apache.derby + derby + + + org.apache.hbase + hbase-client + + + commons-lang + commons-lang + + + com.zaxxer + HikariCP + + + javax.jdo +
[flink] branch master updated: [FLINK-12306][Runtime] Change the name of log into LOG
This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new 0632600 [FLINK-12306][Runtime] Change the name of log into LOG 0632600 is described below commit 06326007da509a6dcd3c64c13f7286d0bbcb6179 Author: Zi Li <820972...@qq.com> AuthorDate: Sat Apr 27 10:31:09 2019 +0800 [FLINK-12306][Runtime] Change the name of log into LOG --- .../java/org/apache/flink/runtime/rpc/MainThreadValidatorUtil.java| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flink-runtime/src/main/java/org/apache/flink/runtime/rpc/MainThreadValidatorUtil.java b/flink-runtime/src/main/java/org/apache/flink/runtime/rpc/MainThreadValidatorUtil.java index c2e2b7d..b380e2d 100644 --- a/flink-runtime/src/main/java/org/apache/flink/runtime/rpc/MainThreadValidatorUtil.java +++ b/flink-runtime/src/main/java/org/apache/flink/runtime/rpc/MainThreadValidatorUtil.java @@ -34,7 +34,7 @@ import static org.apache.flink.util.Preconditions.checkNotNull; */ public final class MainThreadValidatorUtil { - private static final Logger log = LoggerFactory.getLogger(MainThreadValidatorUtil.class); + private static final Logger LOG = LoggerFactory.getLogger(MainThreadValidatorUtil.class); private final RpcEndpoint endpoint; @@ -65,7 +65,7 @@ public final class MainThreadValidatorUtil { String violationMsg = "Violation of main thread constraint detected: expected <" + expected + "> but running in <" + actual + ">."; - log.warn(violationMsg, new Exception(violationMsg)); + LOG.warn(violationMsg, new Exception(violationMsg)); return false; }
[flink] branch master updated (0632600 -> f98a7a8)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0632600 [FLINK-12306][Runtime] Change the name of log into LOG new f1b2e9b [hotfix][metrics] Remove legacy/unused code new 0af34bd [hotfix][tests][runtime] Extract utilities for creating InputChannels new f98a7a8 [FLINK-12199][metrics][network] Decouple network metrics from Task The 16461 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../webmonitor/history/HistoryServerTest.java | 7 +- .../flink/runtime/executiongraph/IOMetrics.java| 70 + .../partition/consumer/InputChannelMetrics.java| 73 + .../partition/consumer/LocalInputChannel.java | 5 +- .../partition/consumer/RemoteInputChannel.java | 5 +- .../partition/consumer/SingleInputGate.java| 14 +- .../partition/consumer/UnknownInputChannel.java| 5 +- .../apache/flink/runtime/metrics/MetricNames.java | 11 +- .../runtime/metrics/groups/TaskIOMetricGroup.java | 53 ++- .../rest/handler/job/JobDetailsHandler.java| 4 +- .../rest/handler/job/JobVertexDetailsHandler.java | 4 +- .../handler/job/JobVertexTaskManagersHandler.java | 4 +- .../rest/handler/util/MutableIOMetrics.java| 76 ++ .../job/SubtaskExecutionAttemptDetailsInfo.java| 4 +- .../org/apache/flink/runtime/taskmanager/Task.java | 5 +- .../ExecutionGraphDeploymentTest.java | 6 +- .../runtime/io/network/NetworkEnvironmentTest.java | 3 +- .../netty/PartitionRequestClientHandlerTest.java | 4 +- .../network/partition/InputChannelTestUtils.java | 71 + .../network/partition/InputGateConcurrentTest.java | 28 ++-- .../network/partition/InputGateFairnessTest.java | 33 ++-- .../partition/consumer/LocalInputChannelTest.java | 53 ++- .../partition/consumer/RemoteInputChannelTest.java | 14 +- .../partition/consumer/SingleInputGateTest.java| 24 +-- .../metrics/groups/TaskIOMetricGroupTest.java | 10 +- .../SubtaskCurrentAttemptDetailsHandlerTest.java | 15 +- .../SubtaskExecutionAttemptDetailsHandlerTest.java | 15 +- .../legacy/utils/ArchivedExecutionBuilder.java | 3 +- .../legacy/utils/ArchivedJobGenerationUtils.java | 167 - .../runtime/taskmanager/TaskAsyncCallTest.java | 5 +- .../apache/flink/runtime/taskmanager/TaskTest.java | 5 +- .../StreamNetworkBenchmarkEnvironment.java | 6 +- .../runtime/tasks/SynchronousCheckpointITCase.java | 5 +- 33 files changed, 286 insertions(+), 521 deletions(-) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/InputChannelMetrics.java delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/legacy/utils/ArchivedJobGenerationUtils.java
[flink] branch release-1.8 updated: [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators
This is an automated email from the ASF dual-hosted git repository. srichter pushed a commit to branch release-1.8 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.8 by this push: new 531d727 [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators 531d727 is described below commit 531d727f9b32c310d8d63b253019b8cc4a23a3eb Author: klion26 AuthorDate: Wed Apr 24 04:52:03 2019 +0200 [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators - Change Will change the local data path from `.../local_state_root/allocatio_id/job_id/jobvertext_id_subtask_id/chk_id/rocksdb` to `.../local_state_root/allocatio_id/job_id/jobvertext_id_subtask_id/chk_id/operator_id` When preparing the local directory Flink deletes the local directory for each subtask if it already exists, If more than one stateful operators chained in a single task, they'll share the same local directory path, then the local directory will be deleted unexpectedly, and the we'll get data loss. This closes #8263. (cherry picked from commit ee60846dc588b1a832a497ff9522d7a3a282c350) --- .../CheckpointStreamWithResultProviderTest.java| 3 + .../state/StateSnapshotCompressionTest.java| 2 +- .../ttl/mock/MockKeyedStateBackendBuilder.java | 1 + .../runtime/state/ttl/mock/MockStateBackend.java | 2 +- .../state/RocksDBKeyedStateBackendBuilder.java | 1 + .../snapshot/RocksIncrementalSnapshotStrategy.java | 17 +- .../tasks/OneInputStreamTaskTestHarness.java | 50 +++- .../runtime/tasks/StreamConfigChainer.java | 23 +- .../runtime/tasks/StreamMockEnvironment.java | 8 +- .../runtime/tasks/StreamTaskTestHarness.java | 21 +- .../state/StatefulOperatorChainedTaskTest.java | 260 + 11 files changed, 369 insertions(+), 19 deletions(-) diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java index 2af25d9..57653e2 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java @@ -35,6 +35,9 @@ import java.io.Closeable; import java.io.File; import java.io.IOException; +/** + * Test for CheckpointStreamWithResultProvider. + */ public class CheckpointStreamWithResultProviderTest extends TestLogger { private static TemporaryFolder temporaryFolder; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java index a10be26..de687ff 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java @@ -18,7 +18,6 @@ package org.apache.flink.runtime.state; -import org.apache.commons.io.IOUtils; import org.apache.flink.api.common.ExecutionConfig; import org.apache.flink.api.common.state.ValueStateDescriptor; import org.apache.flink.api.common.typeutils.base.StringSerializer; @@ -34,6 +33,7 @@ import org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory; import org.apache.flink.runtime.state.ttl.TtlTimeProvider; import org.apache.flink.util.TestLogger; +import org.apache.commons.io.IOUtils; import org.junit.Assert; import org.junit.Test; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java index efef923..2196dc9 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java @@ -30,6 +30,7 @@ import org.apache.flink.runtime.state.StreamCompressionDecorator; import org.apache.flink.runtime.state.ttl.TtlTimeProvider; import javax.annotation.Nonnull; + import java.util.Collection; import java.util.HashMap; import java.util.Map; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java index bdf07bf..f50f1b6 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java @@ -35,8 +35,8 @@ import
[flink] branch master updated: [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators
This is an automated email from the ASF dual-hosted git repository. srichter pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/master by this push: new ee60846 [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators ee60846 is described below commit ee60846dc588b1a832a497ff9522d7a3a282c350 Author: klion26 AuthorDate: Wed Apr 24 10:52:03 2019 +0800 [FLINK-12296][StateBackend] Fix local state directory collision with state loss for chained keyed operators - Change Will change the local data path from `.../local_state_root/allocatio_id/job_id/jobvertext_id_subtask_id/chk_id/rocksdb` to `.../local_state_root/allocatio_id/job_id/jobvertext_id_subtask_id/chk_id/operator_id` When preparing the local directory Flink deletes the local directory for each subtask if it already exists, If more than one stateful operators chained in a single task, they'll share the same local directory path, then the local directory will be deleted unexpectedly, and the we'll get data loss. This closes #8263. --- .../CheckpointStreamWithResultProviderTest.java| 3 + .../state/StateSnapshotCompressionTest.java| 2 +- .../ttl/mock/MockKeyedStateBackendBuilder.java | 1 + .../runtime/state/ttl/mock/MockStateBackend.java | 2 +- .../state/RocksDBKeyedStateBackendBuilder.java | 1 + .../snapshot/RocksIncrementalSnapshotStrategy.java | 17 +- .../tasks/OneInputStreamTaskTestHarness.java | 50 +++- .../runtime/tasks/StreamConfigChainer.java | 23 +- .../runtime/tasks/StreamMockEnvironment.java | 8 +- .../runtime/tasks/StreamTaskTestHarness.java | 21 +- .../state/StatefulOperatorChainedTaskTest.java | 260 + 11 files changed, 369 insertions(+), 19 deletions(-) diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java index 2af25d9..57653e2 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/CheckpointStreamWithResultProviderTest.java @@ -35,6 +35,9 @@ import java.io.Closeable; import java.io.File; import java.io.IOException; +/** + * Test for CheckpointStreamWithResultProvider. + */ public class CheckpointStreamWithResultProviderTest extends TestLogger { private static TemporaryFolder temporaryFolder; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java index a10be26..de687ff 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/StateSnapshotCompressionTest.java @@ -18,7 +18,6 @@ package org.apache.flink.runtime.state; -import org.apache.commons.io.IOUtils; import org.apache.flink.api.common.ExecutionConfig; import org.apache.flink.api.common.state.ValueStateDescriptor; import org.apache.flink.api.common.typeutils.base.StringSerializer; @@ -34,6 +33,7 @@ import org.apache.flink.runtime.state.memory.MemCheckpointStreamFactory; import org.apache.flink.runtime.state.ttl.TtlTimeProvider; import org.apache.flink.util.TestLogger; +import org.apache.commons.io.IOUtils; import org.junit.Assert; import org.junit.Test; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java index 3ffe183..8ec9b4d 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockKeyedStateBackendBuilder.java @@ -31,6 +31,7 @@ import org.apache.flink.runtime.state.heap.InternalKeyContextImpl; import org.apache.flink.runtime.state.ttl.TtlTimeProvider; import javax.annotation.Nonnull; + import java.util.Collection; import java.util.HashMap; import java.util.Map; diff --git a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java index bdf07bf..f50f1b6 100644 --- a/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java +++ b/flink-runtime/src/test/java/org/apache/flink/runtime/state/ttl/mock/MockStateBackend.java @@ -35,8 +35,8 @@ import org.apache.flink.runtime.state.CheckpointStorageLocationReference; import org.apache