[flink] branch release-1.11 updated: [FLINK-19619][e2e] Temporarily disable the pubsub test
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch release-1.11
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.11 by this push:
new d0744c9 [FLINK-19619][e2e] Temporarily disable the pubsub test
d0744c9 is described below
commit d0744c9911d4a352a171a3ab40f81ff330f532ff
Author: Robert Metzger
AuthorDate: Wed Oct 14 08:18:23 2020 +0200
[FLINK-19619][e2e] Temporarily disable the pubsub test
---
flink-end-to-end-tests/run-nightly-tests.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/flink-end-to-end-tests/run-nightly-tests.sh
b/flink-end-to-end-tests/run-nightly-tests.sh
index 7ff6524..5b85041 100755
--- a/flink-end-to-end-tests/run-nightly-tests.sh
+++ b/flink-end-to-end-tests/run-nightly-tests.sh
@@ -140,7 +140,8 @@ if [[ ${PROFILE} != *"jdk11"* ]]; then
run_test "Run Mesos multiple submission test"
"$END_TO_END_DIR/test-scripts/test_mesos_multiple_submissions.sh"
# `google/cloud-sdk` docker image doesn't support aarch64
currently.
- run_test "Test PubSub connector with Docker based Google PubSub
Emulator" "$END_TO_END_DIR/test-scripts/test_streaming_gcp_pubsub.sh"
+ # Disabled until
https://issues.apache.org/jira/browse/FLINK-19619 is fixed.
+ #run_test "Test PubSub connector with Docker based Google
PubSub Emulator" "$END_TO_END_DIR/test-scripts/test_streaming_gcp_pubsub.sh"
fi
fi
[flink] branch master updated: [FLINK-19619][e2e] Temporarily disable the pubsub test
This is an automated email from the ASF dual-hosted git repository.
rmetzger pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 6f1eb7f [FLINK-19619][e2e] Temporarily disable the pubsub test
6f1eb7f is described below
commit 6f1eb7f4995ba2934d980fb7af69a365b2f205b7
Author: Robert Metzger
AuthorDate: Wed Oct 14 08:18:23 2020 +0200
[FLINK-19619][e2e] Temporarily disable the pubsub test
---
flink-end-to-end-tests/run-nightly-tests.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/flink-end-to-end-tests/run-nightly-tests.sh
b/flink-end-to-end-tests/run-nightly-tests.sh
index 670d149..9e7c836 100755
--- a/flink-end-to-end-tests/run-nightly-tests.sh
+++ b/flink-end-to-end-tests/run-nightly-tests.sh
@@ -142,7 +142,8 @@ if [[ ${PROFILE} != *"jdk11"* ]]; then
run_test "Run Mesos multiple submission test"
"$END_TO_END_DIR/test-scripts/test_mesos_multiple_submissions.sh"
# `google/cloud-sdk` docker image doesn't support aarch64
currently.
- run_test "Test PubSub connector with Docker based Google PubSub
Emulator" "$END_TO_END_DIR/test-scripts/test_streaming_gcp_pubsub.sh"
+ # Disabled until
https://issues.apache.org/jira/browse/FLINK-19619 is fixed.
+ #run_test "Test PubSub connector with Docker based Google
PubSub Emulator" "$END_TO_END_DIR/test-scripts/test_streaming_gcp_pubsub.sh"
fi
fi
[flink] branch release-1.10 updated: [FLINK-19523][conf] Hide sensitive command-line configurations
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new ffd352e [FLINK-19523][conf] Hide sensitive command-line configurations
ffd352e is described below
commit ffd352e9d0f71a5e43fcc8b36f3b006fa8fb6080
Author: wineandcheeze
AuthorDate: Wed Oct 14 08:36:54 2020 +0200
[FLINK-19523][conf] Hide sensitive command-line configurations
---
.../org/apache/flink/runtime/util/EnvironmentInformation.java | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
index 1a3269b..638ef7f 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
@@ -18,6 +18,8 @@
package org.apache.flink.runtime.util;
+import org.apache.flink.configuration.GlobalConfiguration;
+
import java.io.InputStream;
import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;
@@ -318,7 +320,11 @@ public class EnvironmentInformation {
else {
log.info(" Program Arguments:");
for (String s: commandLineArgs) {
- log.info("" + s);
+ if (GlobalConfiguration.isSensitive(s))
{
+ log.info("" +
GlobalConfiguration.HIDDEN_CONTENT + " (sensitive information)");
+ } else {
+ log.info("" + s);
+ }
}
}
[flink] branch release-1.10 updated: [FLINK-19523][conf] Hide sensitive command-line configurations
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new ffd352e [FLINK-19523][conf] Hide sensitive command-line configurations
ffd352e is described below
commit ffd352e9d0f71a5e43fcc8b36f3b006fa8fb6080
Author: wineandcheeze
AuthorDate: Wed Oct 14 08:36:54 2020 +0200
[FLINK-19523][conf] Hide sensitive command-line configurations
---
.../org/apache/flink/runtime/util/EnvironmentInformation.java | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
index 1a3269b..638ef7f 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
@@ -18,6 +18,8 @@
package org.apache.flink.runtime.util;
+import org.apache.flink.configuration.GlobalConfiguration;
+
import java.io.InputStream;
import java.lang.management.ManagementFactory;
import java.lang.management.RuntimeMXBean;
@@ -318,7 +320,11 @@ public class EnvironmentInformation {
else {
log.info(" Program Arguments:");
for (String s: commandLineArgs) {
- log.info("" + s);
+ if (GlobalConfiguration.isSensitive(s))
{
+ log.info("" +
GlobalConfiguration.HIDDEN_CONTENT + " (sensitive information)");
+ } else {
+ log.info("" + s);
+ }
}
}
[flink] branch release-1.11 updated (8dc13e8 -> bcee07c)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch release-1.11 in repository https://gitbox.apache.org/repos/asf/flink.git. from 8dc13e8 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add bcee07c [FLINK-19523][conf] Hide sensitive command-line configurations No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/runtime/util/EnvironmentInformation.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)
[flink] branch release-1.11 updated (8dc13e8 -> bcee07c)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch release-1.11 in repository https://gitbox.apache.org/repos/asf/flink.git. from 8dc13e8 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add bcee07c [FLINK-19523][conf] Hide sensitive command-line configurations No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/runtime/util/EnvironmentInformation.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)
[flink] branch master updated (e6d7f97 -> 43aa3fd)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog add 43aa3fd [FLINK-19523][conf] Hide sensitive command-line configurations No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/runtime/util/EnvironmentInformation.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)
[flink] branch master updated (e6d7f97 -> 43aa3fd)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog add 43aa3fd [FLINK-19523][conf] Hide sensitive command-line configurations No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/runtime/util/EnvironmentInformation.java | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-)
[flink] branch master updated (b62c13b -> e6d7f97)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 add 24f8806 [FLINK-18999][table-common][table-planner-blink] Add isTemporary flag to table factory context add e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog No new revisions were added by this update. Summary of changes: .../elasticsearch/table/TestContext.java | 5 +++ .../hbase1/HBaseDynamicTableFactoryTest.java | 6 ++-- .../hbase2/HBaseDynamicTableFactoryTest.java | 6 ++-- .../connectors/hive/HiveDynamicTableFactory.java | 19 +++- .../flink/connectors/hive/HiveTableFactory.java| 6 ++-- .../connectors/hive/HiveLookupJoinITCase.java | 2 +- .../connectors/hive/HiveTableFactoryTest.java | 8 ++--- .../table/catalog/hive/HiveCatalogITCase.java | 30 ++ .../jdbc/table/JdbcDynamicTableFactoryTest.java| 6 ++-- .../table/KafkaDynamicTableFactoryTestBase.java| 36 ++ .../confluent/RegistryAvroFormatFactoryTest.java | 6 ++-- .../flink/formats/avro/AvroFormatFactoryTest.java | 6 ++-- .../flink/formats/csv/CsvFormatFactoryTest.java| 6 ++-- .../flink/formats/json/JsonFormatFactoryTest.java | 6 ++-- .../json/canal/CanalJsonFormatFactoryTest.java | 6 ++-- .../debezium/DebeziumJsonFormatFactoryTest.java| 6 ++-- .../json/maxwell/MaxwellJsonFormatFactoryTest.java | 6 ++-- .../client/gateway/local/ExecutionContext.java | 4 +-- .../table/factories/BlackHoleSinkFactoryTest.java | 3 +- .../factories/DataGenTableSourceFactoryTest.java | 3 +- .../table/factories/PrintSinkFactoryTest.java | 3 +- .../flink/table/factories/TableFactoryUtil.java| 12 +--- .../flink/table/factories/DynamicTableFactory.java | 5 +++ .../apache/flink/table/factories/FactoryUtil.java | 26 +++- .../flink/table/factories/TableSinkFactory.java| 5 +++ .../factories/TableSinkFactoryContextImpl.java | 10 +- .../flink/table/factories/TableSourceFactory.java | 5 +++ .../factories/TableSourceFactoryContextImpl.java | 10 +- .../flink/table/factories/FactoryUtilTest.java | 6 ++-- .../table/planner/catalog/CatalogSchemaTable.java | 2 +- .../planner/plan/FlinkCalciteCatalogReader.java| 3 +- .../table/planner/delegation/PlannerBase.scala | 18 +++ .../planner/plan/schema/CatalogSourceTable.scala | 3 +- .../plan/schema/LegacyCatalogSourceTable.scala | 4 +-- .../flink/table/catalog/DatabaseCalciteSchema.java | 8 +++-- .../flink/table/api/internal/TableEnvImpl.scala| 6 ++-- .../apache/flink/table/planner/StreamPlanner.scala | 6 ++-- 37 files changed, 222 insertions(+), 86 deletions(-)
[flink] branch master updated (b62c13b -> e6d7f97)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 add 24f8806 [FLINK-18999][table-common][table-planner-blink] Add isTemporary flag to table factory context add e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog No new revisions were added by this update. Summary of changes: .../elasticsearch/table/TestContext.java | 5 +++ .../hbase1/HBaseDynamicTableFactoryTest.java | 6 ++-- .../hbase2/HBaseDynamicTableFactoryTest.java | 6 ++-- .../connectors/hive/HiveDynamicTableFactory.java | 19 +++- .../flink/connectors/hive/HiveTableFactory.java| 6 ++-- .../connectors/hive/HiveLookupJoinITCase.java | 2 +- .../connectors/hive/HiveTableFactoryTest.java | 8 ++--- .../table/catalog/hive/HiveCatalogITCase.java | 30 ++ .../jdbc/table/JdbcDynamicTableFactoryTest.java| 6 ++-- .../table/KafkaDynamicTableFactoryTestBase.java| 36 ++ .../confluent/RegistryAvroFormatFactoryTest.java | 6 ++-- .../flink/formats/avro/AvroFormatFactoryTest.java | 6 ++-- .../flink/formats/csv/CsvFormatFactoryTest.java| 6 ++-- .../flink/formats/json/JsonFormatFactoryTest.java | 6 ++-- .../json/canal/CanalJsonFormatFactoryTest.java | 6 ++-- .../debezium/DebeziumJsonFormatFactoryTest.java| 6 ++-- .../json/maxwell/MaxwellJsonFormatFactoryTest.java | 6 ++-- .../client/gateway/local/ExecutionContext.java | 4 +-- .../table/factories/BlackHoleSinkFactoryTest.java | 3 +- .../factories/DataGenTableSourceFactoryTest.java | 3 +- .../table/factories/PrintSinkFactoryTest.java | 3 +- .../flink/table/factories/TableFactoryUtil.java| 12 +--- .../flink/table/factories/DynamicTableFactory.java | 5 +++ .../apache/flink/table/factories/FactoryUtil.java | 26 +++- .../flink/table/factories/TableSinkFactory.java| 5 +++ .../factories/TableSinkFactoryContextImpl.java | 10 +- .../flink/table/factories/TableSourceFactory.java | 5 +++ .../factories/TableSourceFactoryContextImpl.java | 10 +- .../flink/table/factories/FactoryUtilTest.java | 6 ++-- .../table/planner/catalog/CatalogSchemaTable.java | 2 +- .../planner/plan/FlinkCalciteCatalogReader.java| 3 +- .../table/planner/delegation/PlannerBase.scala | 18 +++ .../planner/plan/schema/CatalogSourceTable.scala | 3 +- .../plan/schema/LegacyCatalogSourceTable.scala | 4 +-- .../flink/table/catalog/DatabaseCalciteSchema.java | 8 +++-- .../flink/table/api/internal/TableEnvImpl.scala| 6 ++-- .../apache/flink/table/planner/StreamPlanner.scala | 6 ++-- 37 files changed, 222 insertions(+), 86 deletions(-)
[flink] branch master updated (b62c13b -> e6d7f97)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 add 24f8806 [FLINK-18999][table-common][table-planner-blink] Add isTemporary flag to table factory context add e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog No new revisions were added by this update. Summary of changes: .../elasticsearch/table/TestContext.java | 5 +++ .../hbase1/HBaseDynamicTableFactoryTest.java | 6 ++-- .../hbase2/HBaseDynamicTableFactoryTest.java | 6 ++-- .../connectors/hive/HiveDynamicTableFactory.java | 19 +++- .../flink/connectors/hive/HiveTableFactory.java| 6 ++-- .../connectors/hive/HiveLookupJoinITCase.java | 2 +- .../connectors/hive/HiveTableFactoryTest.java | 8 ++--- .../table/catalog/hive/HiveCatalogITCase.java | 30 ++ .../jdbc/table/JdbcDynamicTableFactoryTest.java| 6 ++-- .../table/KafkaDynamicTableFactoryTestBase.java| 36 ++ .../confluent/RegistryAvroFormatFactoryTest.java | 6 ++-- .../flink/formats/avro/AvroFormatFactoryTest.java | 6 ++-- .../flink/formats/csv/CsvFormatFactoryTest.java| 6 ++-- .../flink/formats/json/JsonFormatFactoryTest.java | 6 ++-- .../json/canal/CanalJsonFormatFactoryTest.java | 6 ++-- .../debezium/DebeziumJsonFormatFactoryTest.java| 6 ++-- .../json/maxwell/MaxwellJsonFormatFactoryTest.java | 6 ++-- .../client/gateway/local/ExecutionContext.java | 4 +-- .../table/factories/BlackHoleSinkFactoryTest.java | 3 +- .../factories/DataGenTableSourceFactoryTest.java | 3 +- .../table/factories/PrintSinkFactoryTest.java | 3 +- .../flink/table/factories/TableFactoryUtil.java| 12 +--- .../flink/table/factories/DynamicTableFactory.java | 5 +++ .../apache/flink/table/factories/FactoryUtil.java | 26 +++- .../flink/table/factories/TableSinkFactory.java| 5 +++ .../factories/TableSinkFactoryContextImpl.java | 10 +- .../flink/table/factories/TableSourceFactory.java | 5 +++ .../factories/TableSourceFactoryContextImpl.java | 10 +- .../flink/table/factories/FactoryUtilTest.java | 6 ++-- .../table/planner/catalog/CatalogSchemaTable.java | 2 +- .../planner/plan/FlinkCalciteCatalogReader.java| 3 +- .../table/planner/delegation/PlannerBase.scala | 18 +++ .../planner/plan/schema/CatalogSourceTable.scala | 3 +- .../plan/schema/LegacyCatalogSourceTable.scala | 4 +-- .../flink/table/catalog/DatabaseCalciteSchema.java | 8 +++-- .../flink/table/api/internal/TableEnvImpl.scala| 6 ++-- .../apache/flink/table/planner/StreamPlanner.scala | 6 ++-- 37 files changed, 222 insertions(+), 86 deletions(-)
[flink] branch master updated (b62c13b -> e6d7f97)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 add 24f8806 [FLINK-18999][table-common][table-planner-blink] Add isTemporary flag to table factory context add e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog No new revisions were added by this update. Summary of changes: .../elasticsearch/table/TestContext.java | 5 +++ .../hbase1/HBaseDynamicTableFactoryTest.java | 6 ++-- .../hbase2/HBaseDynamicTableFactoryTest.java | 6 ++-- .../connectors/hive/HiveDynamicTableFactory.java | 19 +++- .../flink/connectors/hive/HiveTableFactory.java| 6 ++-- .../connectors/hive/HiveLookupJoinITCase.java | 2 +- .../connectors/hive/HiveTableFactoryTest.java | 8 ++--- .../table/catalog/hive/HiveCatalogITCase.java | 30 ++ .../jdbc/table/JdbcDynamicTableFactoryTest.java| 6 ++-- .../table/KafkaDynamicTableFactoryTestBase.java| 36 ++ .../confluent/RegistryAvroFormatFactoryTest.java | 6 ++-- .../flink/formats/avro/AvroFormatFactoryTest.java | 6 ++-- .../flink/formats/csv/CsvFormatFactoryTest.java| 6 ++-- .../flink/formats/json/JsonFormatFactoryTest.java | 6 ++-- .../json/canal/CanalJsonFormatFactoryTest.java | 6 ++-- .../debezium/DebeziumJsonFormatFactoryTest.java| 6 ++-- .../json/maxwell/MaxwellJsonFormatFactoryTest.java | 6 ++-- .../client/gateway/local/ExecutionContext.java | 4 +-- .../table/factories/BlackHoleSinkFactoryTest.java | 3 +- .../factories/DataGenTableSourceFactoryTest.java | 3 +- .../table/factories/PrintSinkFactoryTest.java | 3 +- .../flink/table/factories/TableFactoryUtil.java| 12 +--- .../flink/table/factories/DynamicTableFactory.java | 5 +++ .../apache/flink/table/factories/FactoryUtil.java | 26 +++- .../flink/table/factories/TableSinkFactory.java| 5 +++ .../factories/TableSinkFactoryContextImpl.java | 10 +- .../flink/table/factories/TableSourceFactory.java | 5 +++ .../factories/TableSourceFactoryContextImpl.java | 10 +- .../flink/table/factories/FactoryUtilTest.java | 6 ++-- .../table/planner/catalog/CatalogSchemaTable.java | 2 +- .../planner/plan/FlinkCalciteCatalogReader.java| 3 +- .../table/planner/delegation/PlannerBase.scala | 18 +++ .../planner/plan/schema/CatalogSourceTable.scala | 3 +- .../plan/schema/LegacyCatalogSourceTable.scala | 4 +-- .../flink/table/catalog/DatabaseCalciteSchema.java | 8 +++-- .../flink/table/api/internal/TableEnvImpl.scala| 6 ++-- .../apache/flink/table/planner/StreamPlanner.scala | 6 ++-- 37 files changed, 222 insertions(+), 86 deletions(-)
[flink] branch master updated (b62c13b -> e6d7f97)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 add 24f8806 [FLINK-18999][table-common][table-planner-blink] Add isTemporary flag to table factory context add e6d7f97 [FLINK-18999][hive] Temporary generic table doesn't work with HiveCatalog No new revisions were added by this update. Summary of changes: .../elasticsearch/table/TestContext.java | 5 +++ .../hbase1/HBaseDynamicTableFactoryTest.java | 6 ++-- .../hbase2/HBaseDynamicTableFactoryTest.java | 6 ++-- .../connectors/hive/HiveDynamicTableFactory.java | 19 +++- .../flink/connectors/hive/HiveTableFactory.java| 6 ++-- .../connectors/hive/HiveLookupJoinITCase.java | 2 +- .../connectors/hive/HiveTableFactoryTest.java | 8 ++--- .../table/catalog/hive/HiveCatalogITCase.java | 30 ++ .../jdbc/table/JdbcDynamicTableFactoryTest.java| 6 ++-- .../table/KafkaDynamicTableFactoryTestBase.java| 36 ++ .../confluent/RegistryAvroFormatFactoryTest.java | 6 ++-- .../flink/formats/avro/AvroFormatFactoryTest.java | 6 ++-- .../flink/formats/csv/CsvFormatFactoryTest.java| 6 ++-- .../flink/formats/json/JsonFormatFactoryTest.java | 6 ++-- .../json/canal/CanalJsonFormatFactoryTest.java | 6 ++-- .../debezium/DebeziumJsonFormatFactoryTest.java| 6 ++-- .../json/maxwell/MaxwellJsonFormatFactoryTest.java | 6 ++-- .../client/gateway/local/ExecutionContext.java | 4 +-- .../table/factories/BlackHoleSinkFactoryTest.java | 3 +- .../factories/DataGenTableSourceFactoryTest.java | 3 +- .../table/factories/PrintSinkFactoryTest.java | 3 +- .../flink/table/factories/TableFactoryUtil.java| 12 +--- .../flink/table/factories/DynamicTableFactory.java | 5 +++ .../apache/flink/table/factories/FactoryUtil.java | 26 +++- .../flink/table/factories/TableSinkFactory.java| 5 +++ .../factories/TableSinkFactoryContextImpl.java | 10 +- .../flink/table/factories/TableSourceFactory.java | 5 +++ .../factories/TableSourceFactoryContextImpl.java | 10 +- .../flink/table/factories/FactoryUtilTest.java | 6 ++-- .../table/planner/catalog/CatalogSchemaTable.java | 2 +- .../planner/plan/FlinkCalciteCatalogReader.java| 3 +- .../table/planner/delegation/PlannerBase.scala | 18 +++ .../planner/plan/schema/CatalogSourceTable.scala | 3 +- .../plan/schema/LegacyCatalogSourceTable.scala | 4 +-- .../flink/table/catalog/DatabaseCalciteSchema.java | 8 +++-- .../flink/table/api/internal/TableEnvImpl.scala| 6 ++-- .../apache/flink/table/planner/StreamPlanner.scala | 6 ++-- 37 files changed, 222 insertions(+), 86 deletions(-)
[flink] branch master updated (0e79f1f -> b62c13b)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream add b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (0e79f1f -> b62c13b)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream add b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (0e79f1f -> b62c13b)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream add b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (0e79f1f -> b62c13b)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream add b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (0e79f1f -> b62c13b)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream add b62c13b [FLINK-19584][hbase] Not start flush thread when bufferFlushMaxMutations = 1 No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (14b17cf -> 0e79f1f)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 14b17cf [FLINK-19414][file connector] Revert "Extract FileSourceTextLinesITCase to AbstractFileSourceITCase" (#13610) add 0e79f1f [FLINK-19416][python] Support Python datetime object in from_collection of Python DataStream No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/typeinfo.py| 123 - .../datastream/stream_execution_environment.py | 7 +- .../tests/test_stream_execution_environment.py | 51 - .../flink/api/common/python/PythonBridgeUtils.java | 15 +++ 4 files changed, 184 insertions(+), 12 deletions(-)
[flink] branch master updated (e079d5a -> 14b17cf)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures add 14b17cf [FLINK-19414][file connector] Revert "Extract FileSourceTextLinesITCase to AbstractFileSourceITCase" (#13610) No new revisions were added by this update. Summary of changes: flink-connectors/flink-connector-files/pom.xml | 15 -- .../file/src/AbstractFileSourceITCase.java | 263 - .../file/src/FileSourceTextLinesITCase.java| 250 ++-- .../parquet/FileSourceParquetColumnarITCase.java | 81 --- 4 files changed, 236 insertions(+), 373 deletions(-) delete mode 100644 flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/AbstractFileSourceITCase.java delete mode 100644 flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/FileSourceParquetColumnarITCase.java
[flink] branch master updated (e079d5a -> 14b17cf)
This is an automated email from the ASF dual-hosted git repository. lzljs3620320 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures add 14b17cf [FLINK-19414][file connector] Revert "Extract FileSourceTextLinesITCase to AbstractFileSourceITCase" (#13610) No new revisions were added by this update. Summary of changes: flink-connectors/flink-connector-files/pom.xml | 15 -- .../file/src/AbstractFileSourceITCase.java | 263 - .../file/src/FileSourceTextLinesITCase.java| 250 ++-- .../parquet/FileSourceParquetColumnarITCase.java | 81 --- 4 files changed, 236 insertions(+), 373 deletions(-) delete mode 100644 flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/AbstractFileSourceITCase.java delete mode 100644 flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/FileSourceParquetColumnarITCase.java
[flink] branch master updated: [FLINK-19414][file connector] Revert "Extract FileSourceTextLinesITCase to AbstractFileSourceITCase" (#13610)
This is an automated email from the ASF dual-hosted git repository.
lzljs3620320 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 14b17cf [FLINK-19414][file connector] Revert "Extract
FileSourceTextLinesITCase to AbstractFileSourceITCase" (#13610)
14b17cf is described below
commit 14b17cf77bdaa90f19c62288fbcee8c28943de60
Author: Jingsong Lee
AuthorDate: Wed Oct 14 10:09:16 2020 +0800
[FLINK-19414][file connector] Revert "Extract FileSourceTextLinesITCase to
AbstractFileSourceITCase" (#13610)
We should not have a test-jar for the flink-connector-files project,
because the module it is not scala-versioned. Thus we cannot release an
artifact with a dependency on a specific scala version (like the test jar).
We need to have the connector test utils in a different project, or add a
scala version to the module.
---
flink-connectors/flink-connector-files/pom.xml | 15 --
.../file/src/AbstractFileSourceITCase.java | 263 -
.../file/src/FileSourceTextLinesITCase.java| 250 ++--
.../parquet/FileSourceParquetColumnarITCase.java | 81 ---
4 files changed, 236 insertions(+), 373 deletions(-)
diff --git a/flink-connectors/flink-connector-files/pom.xml
b/flink-connectors/flink-connector-files/pom.xml
index 69ec305..e1a5149 100644
--- a/flink-connectors/flink-connector-files/pom.xml
+++ b/flink-connectors/flink-connector-files/pom.xml
@@ -80,19 +80,4 @@ under the License.
-
-
-
- org.apache.maven.plugins
- maven-jar-plugin
-
-
-
- test-jar
-
-
-
-
-
-
diff --git
a/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/AbstractFileSourceITCase.java
b/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/AbstractFileSourceITCase.java
deleted file mode 100644
index 43e0227..000
---
a/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/AbstractFileSourceITCase.java
+++ /dev/null
@@ -1,263 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.connector.file.src;
-
-import org.apache.flink.core.fs.Path;
-import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
-import org.apache.flink.streaming.api.datastream.DataStream;
-import org.apache.flink.streaming.api.datastream.DataStreamUtils;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-import org.apache.flink.test.util.MiniClusterWithClientResource;
-import org.apache.flink.util.TestLogger;
-
-import org.junit.ClassRule;
-import org.junit.Test;
-import org.junit.rules.TemporaryFolder;
-
-import java.io.File;
-import java.io.IOException;
-import java.time.Duration;
-import java.util.Arrays;
-import java.util.List;
-
-import static org.hamcrest.Matchers.equalTo;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
-
-/**
- * MiniCluster-based integration test for the {@link FileSource}.
- */
-public abstract class AbstractFileSourceITCase extends TestLogger {
-
- private static final int PARALLELISM = 4;
-
- @ClassRule
- public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
-
- @ClassRule
- public static final MiniClusterWithClientResource MINI_CLUSTER = new
MiniClusterWithClientResource(
- new MiniClusterResourceConfiguration.Builder()
- .setNumberTaskManagers(1)
- .setNumberSlotsPerTaskManager(PARALLELISM)
- .build());
-
- //
[flink] branch master updated (099e982 -> e079d5a)
This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool add 5a7dbcd [hotfix][test] Deduplicate TaskStateStatsTest code add 0ee3a5f [hotfix][task] Rename SubtaskCheckpointCoordinator#executorService to asyncOperationsThreadPool add b49f08a [hotfix][task] Log when AsyncCheckpointRunnalbe starts executing (with the execution delay) add a27f5a7 [hotfix][test] Remove unnecessary abstraction in CheckpointBarrierAlignerTestBase add caab8f1 [hotfix][test] Simplify CheckpointBarrierAlignerTest add 1bea821 [hotfix][test] Deduplicate code of ValidatingCheckpointHandler add 117b1cf [hotfix][test] Fix method naming in CheckpointBarrierTrackerTest add f86c959 [FLINK-18662][task] Introduce CheckpointMetricsBuilder add fc5f49e [FLINK-18662][task] Calculate alignmentDurationNanos for unaligned checkpoints and CheckpointBarrierTracker add 1d43462c [FLINK-19487][task] Fix calculation of checkpoinStartDelay for single channel CheckpointBarrierAligner add 6e38869 [FLINK-18662][task] Add persisted bytes to CheckpointMetrics add fa5c1a8 [hotfix][test] Use ValidatingCheckpointHandler in AlternatingCheckpointBarrierHandlerTest add adad56b [FLINK-18662][task] Calculate processed bytes during alignment metric add fbd9321 [FLINK-18662][metrics] Expose persisted and processed bytes metrics to the REST and Web UI add 6497965 [FLINK-18662][docs][metrics] Update the metrics documentation to reflect current state. add 6cea96f [hotfix][task] Throw MailboxClosedException if TaskMailbox is closing instead of IllegalStateException add bd3ae7b [hotfix][test] Do not hide original exception in ShuffleCompressionITCase add e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures No new revisions were added by this update. Summary of changes: docs/_includes/generated/rest_v1_dispatcher.html | 58 ++ .../fig/checkpoint_monitoring-history-subtasks.png | Bin 0 -> 332751 bytes docs/fig/checkpoint_monitoring-history.png | Bin 83561 -> 222715 bytes docs/monitoring/checkpoint_monitoring.md | 14 +- docs/monitoring/checkpoint_monitoring.zh.md| 14 +- docs/monitoring/metrics.md | 2 +- docs/monitoring/metrics.zh.md | 2 +- docs/ops/state/large_state_tuning.md | 28 +-- docs/ops/state/large_state_tuning.zh.md| 28 +-- .../java/org/apache/flink/util/Preconditions.java | 19 ++ .../src/test/resources/rest_api_v1.snapshot| 58 ++ .../detail/job-checkpoints-detail.component.html | 4 +- .../job/checkpoints/job-checkpoints.component.html | 12 +- .../subtask/job-checkpoints-subtask.component.html | 12 +- .../checkpoint/AbstractCheckpointStats.java| 10 + .../runtime/checkpoint/CheckpointMetrics.java | 63 --- .../checkpoint/CheckpointMetricsBuilder.java | 118 .../runtime/checkpoint/CheckpointStatsTracker.java | 32 .../checkpoint/CompletedCheckpointStats.java | 20 ++ .../CompletedCheckpointStatsSummary.java | 26 ++- .../runtime/checkpoint/FailedCheckpointStats.java | 20 ++ .../runtime/checkpoint/PendingCheckpoint.java | 2 + .../runtime/checkpoint/PendingCheckpointStats.java | 29 ++- .../runtime/checkpoint/SubtaskStateStats.java | 22 +++ .../flink/runtime/checkpoint/TaskStateStats.java | 20 ++ .../flink/runtime/concurrent/FutureUtils.java | 29 ++- .../runtime/io/network/buffer/LocalBufferPool.java | 3 +- .../network/partition/consumer/UnionInputGate.java | 11 +- .../runtime/jobgraph/tasks/AbstractInvokable.java | 8 +- .../CheckpointingStatisticsHandler.java| 4 +- .../TaskCheckpointStatisticDetailsHandler.java | 6 + .../messages/checkpoints/CheckpointStatistics.java | 51 - .../checkpoints/CheckpointingStatistics.java | 22 ++- .../checkpoints/SubtaskCheckpointStatistics.java | 22 ++- .../checkpoints/TaskCheckpointStatistics.java | 29 ++- ...TaskCheckpointStatisticsWithSubtaskDetails.java | 34 +++- .../checkpoint/CheckpointStatsTrackerTest.java | 18 +- .../CompletedCheckpointStatsSummaryTest.java | 24 ++- .../checkpoint/CompletedCheckpointTest.java| 6 +- .../checkpoint/FailedCheckpointStatsTest.java | 6 + .../checkpoint/PendingCheckpointStatsTest.java | 2 + .../runtime/checkpoint/SubtaskStateStatsTest.java | 51 ++--- .../runtime/checkpoint/TaskStateStatsTest.java | 62 +++--- .../testutils/DummyCheckpointInvokable.java| 4 +- .../checkpoints/CheckpointingStatisticsTest.java | 16 +- .../checkpoints/TaskCheckpointStatisticsTest.java | 2 + ...CheckpointStatisticsWith
[flink] branch master updated (099e982 -> e079d5a)
This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool add 5a7dbcd [hotfix][test] Deduplicate TaskStateStatsTest code add 0ee3a5f [hotfix][task] Rename SubtaskCheckpointCoordinator#executorService to asyncOperationsThreadPool add b49f08a [hotfix][task] Log when AsyncCheckpointRunnalbe starts executing (with the execution delay) add a27f5a7 [hotfix][test] Remove unnecessary abstraction in CheckpointBarrierAlignerTestBase add caab8f1 [hotfix][test] Simplify CheckpointBarrierAlignerTest add 1bea821 [hotfix][test] Deduplicate code of ValidatingCheckpointHandler add 117b1cf [hotfix][test] Fix method naming in CheckpointBarrierTrackerTest add f86c959 [FLINK-18662][task] Introduce CheckpointMetricsBuilder add fc5f49e [FLINK-18662][task] Calculate alignmentDurationNanos for unaligned checkpoints and CheckpointBarrierTracker add 1d43462c [FLINK-19487][task] Fix calculation of checkpoinStartDelay for single channel CheckpointBarrierAligner add 6e38869 [FLINK-18662][task] Add persisted bytes to CheckpointMetrics add fa5c1a8 [hotfix][test] Use ValidatingCheckpointHandler in AlternatingCheckpointBarrierHandlerTest add adad56b [FLINK-18662][task] Calculate processed bytes during alignment metric add fbd9321 [FLINK-18662][metrics] Expose persisted and processed bytes metrics to the REST and Web UI add 6497965 [FLINK-18662][docs][metrics] Update the metrics documentation to reflect current state. add 6cea96f [hotfix][task] Throw MailboxClosedException if TaskMailbox is closing instead of IllegalStateException add bd3ae7b [hotfix][test] Do not hide original exception in ShuffleCompressionITCase add e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures No new revisions were added by this update. Summary of changes: docs/_includes/generated/rest_v1_dispatcher.html | 58 ++ .../fig/checkpoint_monitoring-history-subtasks.png | Bin 0 -> 332751 bytes docs/fig/checkpoint_monitoring-history.png | Bin 83561 -> 222715 bytes docs/monitoring/checkpoint_monitoring.md | 14 +- docs/monitoring/checkpoint_monitoring.zh.md| 14 +- docs/monitoring/metrics.md | 2 +- docs/monitoring/metrics.zh.md | 2 +- docs/ops/state/large_state_tuning.md | 28 +-- docs/ops/state/large_state_tuning.zh.md| 28 +-- .../java/org/apache/flink/util/Preconditions.java | 19 ++ .../src/test/resources/rest_api_v1.snapshot| 58 ++ .../detail/job-checkpoints-detail.component.html | 4 +- .../job/checkpoints/job-checkpoints.component.html | 12 +- .../subtask/job-checkpoints-subtask.component.html | 12 +- .../checkpoint/AbstractCheckpointStats.java| 10 + .../runtime/checkpoint/CheckpointMetrics.java | 63 --- .../checkpoint/CheckpointMetricsBuilder.java | 118 .../runtime/checkpoint/CheckpointStatsTracker.java | 32 .../checkpoint/CompletedCheckpointStats.java | 20 ++ .../CompletedCheckpointStatsSummary.java | 26 ++- .../runtime/checkpoint/FailedCheckpointStats.java | 20 ++ .../runtime/checkpoint/PendingCheckpoint.java | 2 + .../runtime/checkpoint/PendingCheckpointStats.java | 29 ++- .../runtime/checkpoint/SubtaskStateStats.java | 22 +++ .../flink/runtime/checkpoint/TaskStateStats.java | 20 ++ .../flink/runtime/concurrent/FutureUtils.java | 29 ++- .../runtime/io/network/buffer/LocalBufferPool.java | 3 +- .../network/partition/consumer/UnionInputGate.java | 11 +- .../runtime/jobgraph/tasks/AbstractInvokable.java | 8 +- .../CheckpointingStatisticsHandler.java| 4 +- .../TaskCheckpointStatisticDetailsHandler.java | 6 + .../messages/checkpoints/CheckpointStatistics.java | 51 - .../checkpoints/CheckpointingStatistics.java | 22 ++- .../checkpoints/SubtaskCheckpointStatistics.java | 22 ++- .../checkpoints/TaskCheckpointStatistics.java | 29 ++- ...TaskCheckpointStatisticsWithSubtaskDetails.java | 34 +++- .../checkpoint/CheckpointStatsTrackerTest.java | 18 +- .../CompletedCheckpointStatsSummaryTest.java | 24 ++- .../checkpoint/CompletedCheckpointTest.java| 6 +- .../checkpoint/FailedCheckpointStatsTest.java | 6 + .../checkpoint/PendingCheckpointStatsTest.java | 2 + .../runtime/checkpoint/SubtaskStateStatsTest.java | 51 ++--- .../runtime/checkpoint/TaskStateStatsTest.java | 62 +++--- .../testutils/DummyCheckpointInvokable.java| 4 +- .../checkpoints/CheckpointingStatisticsTest.java | 16 +- .../checkpoints/TaskCheckpointStatisticsTest.java | 2 + ...CheckpointStatisticsWith
[flink] branch master updated (099e982 -> e079d5a)
This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool add 5a7dbcd [hotfix][test] Deduplicate TaskStateStatsTest code add 0ee3a5f [hotfix][task] Rename SubtaskCheckpointCoordinator#executorService to asyncOperationsThreadPool add b49f08a [hotfix][task] Log when AsyncCheckpointRunnalbe starts executing (with the execution delay) add a27f5a7 [hotfix][test] Remove unnecessary abstraction in CheckpointBarrierAlignerTestBase add caab8f1 [hotfix][test] Simplify CheckpointBarrierAlignerTest add 1bea821 [hotfix][test] Deduplicate code of ValidatingCheckpointHandler add 117b1cf [hotfix][test] Fix method naming in CheckpointBarrierTrackerTest add f86c959 [FLINK-18662][task] Introduce CheckpointMetricsBuilder add fc5f49e [FLINK-18662][task] Calculate alignmentDurationNanos for unaligned checkpoints and CheckpointBarrierTracker add 1d43462c [FLINK-19487][task] Fix calculation of checkpoinStartDelay for single channel CheckpointBarrierAligner add 6e38869 [FLINK-18662][task] Add persisted bytes to CheckpointMetrics add fa5c1a8 [hotfix][test] Use ValidatingCheckpointHandler in AlternatingCheckpointBarrierHandlerTest add adad56b [FLINK-18662][task] Calculate processed bytes during alignment metric add fbd9321 [FLINK-18662][metrics] Expose persisted and processed bytes metrics to the REST and Web UI add 6497965 [FLINK-18662][docs][metrics] Update the metrics documentation to reflect current state. add 6cea96f [hotfix][task] Throw MailboxClosedException if TaskMailbox is closing instead of IllegalStateException add bd3ae7b [hotfix][test] Do not hide original exception in ShuffleCompressionITCase add e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures No new revisions were added by this update. Summary of changes: docs/_includes/generated/rest_v1_dispatcher.html | 58 ++ .../fig/checkpoint_monitoring-history-subtasks.png | Bin 0 -> 332751 bytes docs/fig/checkpoint_monitoring-history.png | Bin 83561 -> 222715 bytes docs/monitoring/checkpoint_monitoring.md | 14 +- docs/monitoring/checkpoint_monitoring.zh.md| 14 +- docs/monitoring/metrics.md | 2 +- docs/monitoring/metrics.zh.md | 2 +- docs/ops/state/large_state_tuning.md | 28 +-- docs/ops/state/large_state_tuning.zh.md| 28 +-- .../java/org/apache/flink/util/Preconditions.java | 19 ++ .../src/test/resources/rest_api_v1.snapshot| 58 ++ .../detail/job-checkpoints-detail.component.html | 4 +- .../job/checkpoints/job-checkpoints.component.html | 12 +- .../subtask/job-checkpoints-subtask.component.html | 12 +- .../checkpoint/AbstractCheckpointStats.java| 10 + .../runtime/checkpoint/CheckpointMetrics.java | 63 --- .../checkpoint/CheckpointMetricsBuilder.java | 118 .../runtime/checkpoint/CheckpointStatsTracker.java | 32 .../checkpoint/CompletedCheckpointStats.java | 20 ++ .../CompletedCheckpointStatsSummary.java | 26 ++- .../runtime/checkpoint/FailedCheckpointStats.java | 20 ++ .../runtime/checkpoint/PendingCheckpoint.java | 2 + .../runtime/checkpoint/PendingCheckpointStats.java | 29 ++- .../runtime/checkpoint/SubtaskStateStats.java | 22 +++ .../flink/runtime/checkpoint/TaskStateStats.java | 20 ++ .../flink/runtime/concurrent/FutureUtils.java | 29 ++- .../runtime/io/network/buffer/LocalBufferPool.java | 3 +- .../network/partition/consumer/UnionInputGate.java | 11 +- .../runtime/jobgraph/tasks/AbstractInvokable.java | 8 +- .../CheckpointingStatisticsHandler.java| 4 +- .../TaskCheckpointStatisticDetailsHandler.java | 6 + .../messages/checkpoints/CheckpointStatistics.java | 51 - .../checkpoints/CheckpointingStatistics.java | 22 ++- .../checkpoints/SubtaskCheckpointStatistics.java | 22 ++- .../checkpoints/TaskCheckpointStatistics.java | 29 ++- ...TaskCheckpointStatisticsWithSubtaskDetails.java | 34 +++- .../checkpoint/CheckpointStatsTrackerTest.java | 18 +- .../CompletedCheckpointStatsSummaryTest.java | 24 ++- .../checkpoint/CompletedCheckpointTest.java| 6 +- .../checkpoint/FailedCheckpointStatsTest.java | 6 + .../checkpoint/PendingCheckpointStatsTest.java | 2 + .../runtime/checkpoint/SubtaskStateStatsTest.java | 51 ++--- .../runtime/checkpoint/TaskStateStatsTest.java | 62 +++--- .../testutils/DummyCheckpointInvokable.java| 4 +- .../checkpoints/CheckpointingStatisticsTest.java | 16 +- .../checkpoints/TaskCheckpointStatisticsTest.java | 2 + ...CheckpointStatisticsWith
[flink] branch master updated (099e982 -> e079d5a)
This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool add 5a7dbcd [hotfix][test] Deduplicate TaskStateStatsTest code add 0ee3a5f [hotfix][task] Rename SubtaskCheckpointCoordinator#executorService to asyncOperationsThreadPool add b49f08a [hotfix][task] Log when AsyncCheckpointRunnalbe starts executing (with the execution delay) add a27f5a7 [hotfix][test] Remove unnecessary abstraction in CheckpointBarrierAlignerTestBase add caab8f1 [hotfix][test] Simplify CheckpointBarrierAlignerTest add 1bea821 [hotfix][test] Deduplicate code of ValidatingCheckpointHandler add 117b1cf [hotfix][test] Fix method naming in CheckpointBarrierTrackerTest add f86c959 [FLINK-18662][task] Introduce CheckpointMetricsBuilder add fc5f49e [FLINK-18662][task] Calculate alignmentDurationNanos for unaligned checkpoints and CheckpointBarrierTracker add 1d43462c [FLINK-19487][task] Fix calculation of checkpoinStartDelay for single channel CheckpointBarrierAligner add 6e38869 [FLINK-18662][task] Add persisted bytes to CheckpointMetrics add fa5c1a8 [hotfix][test] Use ValidatingCheckpointHandler in AlternatingCheckpointBarrierHandlerTest add adad56b [FLINK-18662][task] Calculate processed bytes during alignment metric add fbd9321 [FLINK-18662][metrics] Expose persisted and processed bytes metrics to the REST and Web UI add 6497965 [FLINK-18662][docs][metrics] Update the metrics documentation to reflect current state. add 6cea96f [hotfix][task] Throw MailboxClosedException if TaskMailbox is closing instead of IllegalStateException add bd3ae7b [hotfix][test] Do not hide original exception in ShuffleCompressionITCase add e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures No new revisions were added by this update. Summary of changes: docs/_includes/generated/rest_v1_dispatcher.html | 58 ++ .../fig/checkpoint_monitoring-history-subtasks.png | Bin 0 -> 332751 bytes docs/fig/checkpoint_monitoring-history.png | Bin 83561 -> 222715 bytes docs/monitoring/checkpoint_monitoring.md | 14 +- docs/monitoring/checkpoint_monitoring.zh.md| 14 +- docs/monitoring/metrics.md | 2 +- docs/monitoring/metrics.zh.md | 2 +- docs/ops/state/large_state_tuning.md | 28 +-- docs/ops/state/large_state_tuning.zh.md| 28 +-- .../java/org/apache/flink/util/Preconditions.java | 19 ++ .../src/test/resources/rest_api_v1.snapshot| 58 ++ .../detail/job-checkpoints-detail.component.html | 4 +- .../job/checkpoints/job-checkpoints.component.html | 12 +- .../subtask/job-checkpoints-subtask.component.html | 12 +- .../checkpoint/AbstractCheckpointStats.java| 10 + .../runtime/checkpoint/CheckpointMetrics.java | 63 --- .../checkpoint/CheckpointMetricsBuilder.java | 118 .../runtime/checkpoint/CheckpointStatsTracker.java | 32 .../checkpoint/CompletedCheckpointStats.java | 20 ++ .../CompletedCheckpointStatsSummary.java | 26 ++- .../runtime/checkpoint/FailedCheckpointStats.java | 20 ++ .../runtime/checkpoint/PendingCheckpoint.java | 2 + .../runtime/checkpoint/PendingCheckpointStats.java | 29 ++- .../runtime/checkpoint/SubtaskStateStats.java | 22 +++ .../flink/runtime/checkpoint/TaskStateStats.java | 20 ++ .../flink/runtime/concurrent/FutureUtils.java | 29 ++- .../runtime/io/network/buffer/LocalBufferPool.java | 3 +- .../network/partition/consumer/UnionInputGate.java | 11 +- .../runtime/jobgraph/tasks/AbstractInvokable.java | 8 +- .../CheckpointingStatisticsHandler.java| 4 +- .../TaskCheckpointStatisticDetailsHandler.java | 6 + .../messages/checkpoints/CheckpointStatistics.java | 51 - .../checkpoints/CheckpointingStatistics.java | 22 ++- .../checkpoints/SubtaskCheckpointStatistics.java | 22 ++- .../checkpoints/TaskCheckpointStatistics.java | 29 ++- ...TaskCheckpointStatisticsWithSubtaskDetails.java | 34 +++- .../checkpoint/CheckpointStatsTrackerTest.java | 18 +- .../CompletedCheckpointStatsSummaryTest.java | 24 ++- .../checkpoint/CompletedCheckpointTest.java| 6 +- .../checkpoint/FailedCheckpointStatsTest.java | 6 + .../checkpoint/PendingCheckpointStatsTest.java | 2 + .../runtime/checkpoint/SubtaskStateStatsTest.java | 51 ++--- .../runtime/checkpoint/TaskStateStatsTest.java | 62 +++--- .../testutils/DummyCheckpointInvokable.java| 4 +- .../checkpoints/CheckpointingStatisticsTest.java | 16 +- .../checkpoints/TaskCheckpointStatisticsTest.java | 2 + ...CheckpointStatisticsWith
[flink] branch master updated (099e982 -> e079d5a)
This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool add 5a7dbcd [hotfix][test] Deduplicate TaskStateStatsTest code add 0ee3a5f [hotfix][task] Rename SubtaskCheckpointCoordinator#executorService to asyncOperationsThreadPool add b49f08a [hotfix][task] Log when AsyncCheckpointRunnalbe starts executing (with the execution delay) add a27f5a7 [hotfix][test] Remove unnecessary abstraction in CheckpointBarrierAlignerTestBase add caab8f1 [hotfix][test] Simplify CheckpointBarrierAlignerTest add 1bea821 [hotfix][test] Deduplicate code of ValidatingCheckpointHandler add 117b1cf [hotfix][test] Fix method naming in CheckpointBarrierTrackerTest add f86c959 [FLINK-18662][task] Introduce CheckpointMetricsBuilder add fc5f49e [FLINK-18662][task] Calculate alignmentDurationNanos for unaligned checkpoints and CheckpointBarrierTracker add 1d43462c [FLINK-19487][task] Fix calculation of checkpoinStartDelay for single channel CheckpointBarrierAligner add 6e38869 [FLINK-18662][task] Add persisted bytes to CheckpointMetrics add fa5c1a8 [hotfix][test] Use ValidatingCheckpointHandler in AlternatingCheckpointBarrierHandlerTest add adad56b [FLINK-18662][task] Calculate processed bytes during alignment metric add fbd9321 [FLINK-18662][metrics] Expose persisted and processed bytes metrics to the REST and Web UI add 6497965 [FLINK-18662][docs][metrics] Update the metrics documentation to reflect current state. add 6cea96f [hotfix][task] Throw MailboxClosedException if TaskMailbox is closing instead of IllegalStateException add bd3ae7b [hotfix][test] Do not hide original exception in ShuffleCompressionITCase add e079d5a [hotfix][task][network] Do not swallow exceptions from actions chained to the availability futures No new revisions were added by this update. Summary of changes: docs/_includes/generated/rest_v1_dispatcher.html | 58 ++ .../fig/checkpoint_monitoring-history-subtasks.png | Bin 0 -> 332751 bytes docs/fig/checkpoint_monitoring-history.png | Bin 83561 -> 222715 bytes docs/monitoring/checkpoint_monitoring.md | 14 +- docs/monitoring/checkpoint_monitoring.zh.md| 14 +- docs/monitoring/metrics.md | 2 +- docs/monitoring/metrics.zh.md | 2 +- docs/ops/state/large_state_tuning.md | 28 +-- docs/ops/state/large_state_tuning.zh.md| 28 +-- .../java/org/apache/flink/util/Preconditions.java | 19 ++ .../src/test/resources/rest_api_v1.snapshot| 58 ++ .../detail/job-checkpoints-detail.component.html | 4 +- .../job/checkpoints/job-checkpoints.component.html | 12 +- .../subtask/job-checkpoints-subtask.component.html | 12 +- .../checkpoint/AbstractCheckpointStats.java| 10 + .../runtime/checkpoint/CheckpointMetrics.java | 63 --- .../checkpoint/CheckpointMetricsBuilder.java | 118 .../runtime/checkpoint/CheckpointStatsTracker.java | 32 .../checkpoint/CompletedCheckpointStats.java | 20 ++ .../CompletedCheckpointStatsSummary.java | 26 ++- .../runtime/checkpoint/FailedCheckpointStats.java | 20 ++ .../runtime/checkpoint/PendingCheckpoint.java | 2 + .../runtime/checkpoint/PendingCheckpointStats.java | 29 ++- .../runtime/checkpoint/SubtaskStateStats.java | 22 +++ .../flink/runtime/checkpoint/TaskStateStats.java | 20 ++ .../flink/runtime/concurrent/FutureUtils.java | 29 ++- .../runtime/io/network/buffer/LocalBufferPool.java | 3 +- .../network/partition/consumer/UnionInputGate.java | 11 +- .../runtime/jobgraph/tasks/AbstractInvokable.java | 8 +- .../CheckpointingStatisticsHandler.java| 4 +- .../TaskCheckpointStatisticDetailsHandler.java | 6 + .../messages/checkpoints/CheckpointStatistics.java | 51 - .../checkpoints/CheckpointingStatistics.java | 22 ++- .../checkpoints/SubtaskCheckpointStatistics.java | 22 ++- .../checkpoints/TaskCheckpointStatistics.java | 29 ++- ...TaskCheckpointStatisticsWithSubtaskDetails.java | 34 +++- .../checkpoint/CheckpointStatsTrackerTest.java | 18 +- .../CompletedCheckpointStatsSummaryTest.java | 24 ++- .../checkpoint/CompletedCheckpointTest.java| 6 +- .../checkpoint/FailedCheckpointStatsTest.java | 6 + .../checkpoint/PendingCheckpointStatsTest.java | 2 + .../runtime/checkpoint/SubtaskStateStatsTest.java | 51 ++--- .../runtime/checkpoint/TaskStateStatsTest.java | 62 +++--- .../testutils/DummyCheckpointInvokable.java| 4 +- .../checkpoints/CheckpointingStatisticsTest.java | 16 +- .../checkpoints/TaskCheckpointStatisticsTest.java | 2 + ...CheckpointStatisticsWith
[flink] branch master updated (3199606 -> 099e982)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 3199606 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool No new revisions were added by this update. Summary of changes: .../jobmaster/slotpool/AllocatedSlotPool.java | 125 .../slotpool/DefaultAllocatedSlotPool.java | 214 ++ .../slotpool/DefaultAllocatedSlotPoolTest.java | 315 + 3 files changed, 654 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/AllocatedSlotPool.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
[flink] branch master updated (3199606 -> 099e982)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 3199606 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool No new revisions were added by this update. Summary of changes: .../jobmaster/slotpool/AllocatedSlotPool.java | 125 .../slotpool/DefaultAllocatedSlotPool.java | 214 ++ .../slotpool/DefaultAllocatedSlotPoolTest.java | 315 + 3 files changed, 654 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/AllocatedSlotPool.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
[flink] branch master updated (3199606 -> 099e982)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 3199606 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool No new revisions were added by this update. Summary of changes: .../jobmaster/slotpool/AllocatedSlotPool.java | 125 .../slotpool/DefaultAllocatedSlotPool.java | 214 ++ .../slotpool/DefaultAllocatedSlotPoolTest.java | 315 + 3 files changed, 654 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/AllocatedSlotPool.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
[flink] branch master updated (3199606 -> 099e982)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 3199606 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment add 099e982 [FLINK-19315][coordination] Add AllocatedSlotPool No new revisions were added by this update. Summary of changes: .../jobmaster/slotpool/AllocatedSlotPool.java | 125 .../slotpool/DefaultAllocatedSlotPool.java | 214 ++ .../slotpool/DefaultAllocatedSlotPoolTest.java | 315 + 3 files changed, 654 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/AllocatedSlotPool.java create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPool.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/DefaultAllocatedSlotPoolTest.java
[flink-web] branch asf-site updated: Fix typo and rebuild website
This is an automated email from the ASF dual-hosted git repository.
tzulitai pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
The following commit(s) were added to refs/heads/asf-site by this push:
new abfce66 Fix typo and rebuild website
abfce66 is described below
commit abfce66d116d6e3f8a238f4a935451737f8b8f76
Author: Tzu-Li (Gordon) Tai
AuthorDate: Tue Oct 13 22:15:48 2020 +0800
Fix typo and rebuild website
---
_posts/2020-10-13-stateful-serverless-internals.md | 2 +-
content/blog/feed.xml | 2 +-
content/news/2020/10/13/stateful-serverless-internals.html | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/_posts/2020-10-13-stateful-serverless-internals.md
b/_posts/2020-10-13-stateful-serverless-internals.md
index c637609..f04681d 100644
--- a/_posts/2020-10-13-stateful-serverless-internals.md
+++ b/_posts/2020-10-13-stateful-serverless-internals.md
@@ -177,7 +177,7 @@ The event is thus routed to partition A `(2)`.
* Once partition B receives the `RequestItem` message, the runtime invokes the
function `(inventory:socks)` in the same
way as described above, and receives a reply with a modification of the state
of the inventory (the number of reserved socks is now increased by 2).
`(inventory:socks)` now also wants to reply reservation of 2 socks for Kim, so
an `ItemReserved("socks", 2)`
-message targeted for `(cart:Kim)` is also included in the response (7), which
will again be routed by the StateFun runtime.
+message targeted for `(cart:Kim)` is also included in the response
`(7)`, which will again be routed by the StateFun runtime.
## Stateful Serverless in the Cloud with FaaS and StateFun
diff --git a/content/blog/feed.xml b/content/blog/feed.xml
index a78dc5f..7667b62 100644
--- a/content/blog/feed.xml
+++ b/content/blog/feed.xml
@@ -190,7 +190,7 @@ and marks (cart:Kim) as
“available”Once partition B receives the RequestItem
message, the runtime invokes the function
(inventory:socks) in the same
way as described above, and receives a reply with a modification of the state
of the inventory (the number of reserved socks is now increased by 2).
(inventory:socks) now also wants to reply reservation
of 2 socks for Kim, so an ItemReserved("socks",
2)
-message targeted for (cart:Kim) is also included in
the response (7), which will again be routed by the StateFun runtime.
+message targeted for (cart:Kim) is also included in
the response (7), which will again
be routed by the StateFun runtime.
Stateful
Serverless in the Cloud with FaaS and StateFun
diff --git a/content/news/2020/10/13/stateful-serverless-internals.html
b/content/news/2020/10/13/stateful-serverless-internals.html
index e0ac47f..e81d72a 100644
--- a/content/news/2020/10/13/stateful-serverless-internals.html
+++ b/content/news/2020/10/13/stateful-serverless-internals.html
@@ -379,7 +379,7 @@ and marks (cart:Kim) as “available”
again for invocation
Once partition B receives the RequestItem message, the
runtime invokes the function (inventory:socks) in the same
way as described above, and receives a reply with a modification of the state
of the inventory (the number of reserved socks is now increased by 2).
(inventory:socks) now also wants to reply reservation of 2 socks
for Kim, so an ItemReserved("socks", 2)
-message targeted for (cart:Kim) is also included in the response
(7), which will again be routed by the StateFun runtime.
+message targeted for (cart:Kim) is also included in the response
(7), which will again be routed by the StateFun
runtime.
Stateful
Serverless in the Cloud with FaaS and StateFun
[flink] branch release-1.10 updated: [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new fb34ce7 [FLINK-17458] Stop
TaskSubmissionTestEnvironment.testingRpcService when closing the environment
fb34ce7 is described below
commit fb34ce741a26fede33c6ae316558c8bd9b10e5d6
Author: Till Rohrmann
AuthorDate: Fri Oct 9 13:01:48 2020 +0200
[FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when
closing the environment
This closes #13569.
---
.../flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
index 1fb941c..5b92b74 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
+++
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
@@ -43,7 +43,6 @@ import
org.apache.flink.runtime.jobmaster.utils.TestingJobMasterGatewayBuilder;
import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
import org.apache.flink.runtime.rpc.RpcService;
-import org.apache.flink.runtime.rpc.RpcUtils;
import org.apache.flink.runtime.rpc.TestingRpcService;
import org.apache.flink.runtime.shuffle.ShuffleEnvironment;
import org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager;
@@ -261,7 +260,7 @@ class TaskSubmissionTestEnvironment implements
AutoCloseable {
@Override
public void close() throws Exception {
- RpcUtils.terminateRpcEndpoint(taskExecutor, timeout);
+ testingRpcService.stopService().join();
timerService.stop();
[flink] branch release-1.10 updated: [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch release-1.10
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.10 by this push:
new fb34ce7 [FLINK-17458] Stop
TaskSubmissionTestEnvironment.testingRpcService when closing the environment
fb34ce7 is described below
commit fb34ce741a26fede33c6ae316558c8bd9b10e5d6
Author: Till Rohrmann
AuthorDate: Fri Oct 9 13:01:48 2020 +0200
[FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when
closing the environment
This closes #13569.
---
.../flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
index 1fb941c..5b92b74 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
+++
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
@@ -43,7 +43,6 @@ import
org.apache.flink.runtime.jobmaster.utils.TestingJobMasterGatewayBuilder;
import org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
import org.apache.flink.runtime.rpc.RpcService;
-import org.apache.flink.runtime.rpc.RpcUtils;
import org.apache.flink.runtime.rpc.TestingRpcService;
import org.apache.flink.runtime.shuffle.ShuffleEnvironment;
import org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager;
@@ -261,7 +260,7 @@ class TaskSubmissionTestEnvironment implements
AutoCloseable {
@Override
public void close() throws Exception {
- RpcUtils.terminateRpcEndpoint(taskExecutor, timeout);
+ testingRpcService.stopService().join();
timerService.stop();
[flink] branch release-1.11 updated: [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch release-1.11
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.11 by this push:
new 8dc13e8 [FLINK-17458] Stop
TaskSubmissionTestEnvironment.testingRpcService when closing the environment
8dc13e8 is described below
commit 8dc13e8c7dab7bd740331a6edd44aa8978b95120
Author: Till Rohrmann
AuthorDate: Fri Oct 9 13:01:48 2020 +0200
[FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when
closing the environment
This closes #13569.
---
.../flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
index d7fcd02..f497927 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
+++
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
@@ -44,7 +44,6 @@ import
org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
import org.apache.flink.runtime.rpc.MainThreadExecutable;
import org.apache.flink.runtime.rpc.RpcService;
-import org.apache.flink.runtime.rpc.RpcUtils;
import org.apache.flink.runtime.rpc.TestingRpcService;
import org.apache.flink.runtime.shuffle.ShuffleEnvironment;
import org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager;
@@ -277,7 +276,7 @@ class TaskSubmissionTestEnvironment implements
AutoCloseable {
@Override
public void close() throws Exception {
- RpcUtils.terminateRpcEndpoint(taskExecutor, timeout);
+ testingRpcService.stopService().join();
timerService.stop();
[flink] branch release-1.11 updated: [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch release-1.11
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-1.11 by this push:
new 8dc13e8 [FLINK-17458] Stop
TaskSubmissionTestEnvironment.testingRpcService when closing the environment
8dc13e8 is described below
commit 8dc13e8c7dab7bd740331a6edd44aa8978b95120
Author: Till Rohrmann
AuthorDate: Fri Oct 9 13:01:48 2020 +0200
[FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when
closing the environment
This closes #13569.
---
.../flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
index d7fcd02..f497927 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
+++
b/flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java
@@ -44,7 +44,6 @@ import
org.apache.flink.runtime.leaderretrieval.SettableLeaderRetrievalService;
import org.apache.flink.runtime.metrics.groups.UnregisteredMetricGroups;
import org.apache.flink.runtime.rpc.MainThreadExecutable;
import org.apache.flink.runtime.rpc.RpcService;
-import org.apache.flink.runtime.rpc.RpcUtils;
import org.apache.flink.runtime.rpc.TestingRpcService;
import org.apache.flink.runtime.shuffle.ShuffleEnvironment;
import org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager;
@@ -277,7 +276,7 @@ class TaskSubmissionTestEnvironment implements
AutoCloseable {
@Override
public void close() throws Exception {
- RpcUtils.terminateRpcEndpoint(taskExecutor, timeout);
+ testingRpcService.stopService().join();
timerService.stop();
[flink] branch master updated (28c1646 -> 3199606)
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 28c1646 [FLINK-19512] Introduce the new sink API add 3199606 [FLINK-17458] Stop TaskSubmissionTestEnvironment.testingRpcService when closing the environment No new revisions were added by this update. Summary of changes: .../flink/runtime/taskexecutor/TaskSubmissionTestEnvironment.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[flink] branch master updated (0c37382 -> 28c1646)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0c37382 [FLINK-19570][tests] Avoid directly creating ExecutionJobVertex and ExecutionVertex via constructors in tests add 28c1646 [FLINK-19512] Introduce the new sink API No new revisions were added by this update. Summary of changes: .../apache/flink/api/connector/sink/Committer.java | 23 -- .../flink/api/connector/sink/GlobalCommitter.java | 65 +++ .../org/apache/flink/api/connector/sink/Sink.java | 95 ++ .../apache/flink/api/connector/sink/Writer.java| 77 ++ 4 files changed, 251 insertions(+), 9 deletions(-) copy flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/split/JdbcParameterValuesProvider.java => flink-core/src/main/java/org/apache/flink/api/connector/sink/Committer.java (55%) create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/GlobalCommitter.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Writer.java
[flink] branch master updated (0c37382 -> 28c1646)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0c37382 [FLINK-19570][tests] Avoid directly creating ExecutionJobVertex and ExecutionVertex via constructors in tests add 28c1646 [FLINK-19512] Introduce the new sink API No new revisions were added by this update. Summary of changes: .../apache/flink/api/connector/sink/Committer.java | 23 -- .../flink/api/connector/sink/GlobalCommitter.java | 65 +++ .../org/apache/flink/api/connector/sink/Sink.java | 95 ++ .../apache/flink/api/connector/sink/Writer.java| 77 ++ 4 files changed, 251 insertions(+), 9 deletions(-) copy flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/split/JdbcParameterValuesProvider.java => flink-core/src/main/java/org/apache/flink/api/connector/sink/Committer.java (55%) create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/GlobalCommitter.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Writer.java
[flink] branch master updated (0c37382 -> 28c1646)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 0c37382 [FLINK-19570][tests] Avoid directly creating ExecutionJobVertex and ExecutionVertex via constructors in tests add 28c1646 [FLINK-19512] Introduce the new sink API No new revisions were added by this update. Summary of changes: .../apache/flink/api/connector/sink/Committer.java | 23 -- .../flink/api/connector/sink/GlobalCommitter.java | 65 +++ .../org/apache/flink/api/connector/sink/Sink.java | 95 ++ .../apache/flink/api/connector/sink/Writer.java| 77 ++ 4 files changed, 251 insertions(+), 9 deletions(-) copy flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/split/JdbcParameterValuesProvider.java => flink-core/src/main/java/org/apache/flink/api/connector/sink/Committer.java (55%) create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/GlobalCommitter.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Sink.java create mode 100644 flink-core/src/main/java/org/apache/flink/api/connector/sink/Writer.java
[flink-web] branch asf-site updated (f26afc9 -> ede0907)
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git. from f26afc9 rebuild website new d00721f Add blog "Stateful Functions Internals: Behind the scenes of Stateful Serverless" new ede0907 Rebuild website The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: _posts/2020-10-13-stateful-serverless-internals.md | 213 ++ content/blog/feed.xml | 354 ++-- content/blog/index.html| 36 +- content/blog/page10/index.html | 37 +- content/blog/page11/index.html | 38 +- content/blog/page12/index.html | 44 +- content/blog/page13/index.html | 45 +- content/blog/page14/index.html | 25 ++ content/blog/page2/index.html | 38 +- content/blog/page3/index.html | 40 +- content/blog/page4/index.html | 38 +- content/blog/page5/index.html | 36 +- content/blog/page6/index.html | 38 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 38 +- content/blog/page9/index.html | 36 +- .../abstract-deployment.png| Bin 0 -> 122292 bytes .../aws-deployment.png | Bin 0 -> 251991 bytes .../protocol-walkthrough-1.png | Bin 0 -> 165851 bytes .../request-reply-protocol.png | Bin 0 -> 149165 bytes .../shopping-cart-overview.png | Bin 0 -> 109615 bytes content/index.html | 8 +- .../2020/10/13/stateful-serverless-internals.html | 468 + content/zh/index.html | 8 +- .../abstract-deployment.png| Bin 0 -> 122292 bytes .../aws-deployment.png | Bin 0 -> 251991 bytes .../protocol-walkthrough-1.png | Bin 0 -> 165851 bytes .../request-reply-protocol.png | Bin 0 -> 149165 bytes .../shopping-cart-overview.png | Bin 0 -> 109615 bytes 29 files changed, 1253 insertions(+), 327 deletions(-) create mode 100644 _posts/2020-10-13-stateful-serverless-internals.md create mode 100644 content/img/blog/2020-10-13-stateful-serverless-internals/abstract-deployment.png create mode 100644 content/img/blog/2020-10-13-stateful-serverless-internals/aws-deployment.png create mode 100644 content/img/blog/2020-10-13-stateful-serverless-internals/protocol-walkthrough-1.png create mode 100644 content/img/blog/2020-10-13-stateful-serverless-internals/request-reply-protocol.png create mode 100644 content/img/blog/2020-10-13-stateful-serverless-internals/shopping-cart-overview.png create mode 100644 content/news/2020/10/13/stateful-serverless-internals.html create mode 100644 img/blog/2020-10-13-stateful-serverless-internals/abstract-deployment.png create mode 100644 img/blog/2020-10-13-stateful-serverless-internals/aws-deployment.png create mode 100644 img/blog/2020-10-13-stateful-serverless-internals/protocol-walkthrough-1.png create mode 100644 img/blog/2020-10-13-stateful-serverless-internals/request-reply-protocol.png create mode 100644 img/blog/2020-10-13-stateful-serverless-internals/shopping-cart-overview.png
[flink-web] 02/02: Rebuild website
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit ede0907fb8f4a7e40a06e4eb9a3931dcd4f7cd83 Author: Tzu-Li (Gordon) Tai AuthorDate: Tue Oct 13 20:55:32 2020 +0800 Rebuild website --- content/blog/feed.xml | 354 ++-- content/blog/index.html| 36 +- content/blog/page10/index.html | 37 +- content/blog/page11/index.html | 38 +- content/blog/page12/index.html | 44 +- content/blog/page13/index.html | 45 +- content/blog/page14/index.html | 25 ++ content/blog/page2/index.html | 38 +- content/blog/page3/index.html | 40 +- content/blog/page4/index.html | 38 +- content/blog/page5/index.html | 36 +- content/blog/page6/index.html | 38 +- content/blog/page7/index.html | 40 +- content/blog/page8/index.html | 38 +- content/blog/page9/index.html | 36 +- .../abstract-deployment.png| Bin 0 -> 122292 bytes .../aws-deployment.png | Bin 0 -> 251991 bytes .../protocol-walkthrough-1.png | Bin 0 -> 165851 bytes .../request-reply-protocol.png | Bin 0 -> 149165 bytes .../shopping-cart-overview.png | Bin 0 -> 109615 bytes content/index.html | 8 +- .../2020/10/13/stateful-serverless-internals.html | 468 + content/zh/index.html | 8 +- 23 files changed, 1040 insertions(+), 327 deletions(-) diff --git a/content/blog/feed.xml b/content/blog/feed.xml index 0da80b8..b50a8e2 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -7,6 +7,231 @@ https://flink.apache.org/blog/feed.xml"; rel="self" type="application/rss+xml" /> +Stateful Functions Internals: Behind the scenes of Stateful Serverless +Stateful Functions (StateFun) simplifies the building of distributed stateful applications by combining the best of two worlds: +the strong messaging and state consistency guarantees of stateful stream processing, and the elasticity and serverless experience of today’s cloud-native architectures and +popular event-driven FaaS platforms. Typical StateFun applications consist of functions deployed behind simple services +using these modern platforms, with a separate StateFun cluster playing the role of an “event-driven database” +that provides consistency and fault-tolerance for the functions’ state and messaging.
+ +But how exactly does StateFun achieve that? How does the StateFun cluster communicate with the functions?
+ +This blog dives deep into the internals of the StateFun runtime. The entire walkthrough is complemented by a +demo application which can be completely deployed on AWS services. +Most significantly, in the demo, the stateful functions are deployed and serviced using AWS Lambda, +a popular FaaS platform among many others. The goal here is to allow readers to have a good grasp of the interaction between +the StateFun runtime and the functions, how that works cohesively to provide a Stateful Serverless experience, and how they can apply +what they’ve learnt to deploy their StateFun applications on other public cloud offerings such as GCP or Microsoft Azure.
+ +++
- Introducing the example: Shopping Cart
+- What happens in the Stateful Functions runtime? +
+- Stateful Serverless
[flink-web] 01/02: Add blog "Stateful Functions Internals: Behind the scenes of Stateful Serverless"
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git commit d00721fcd0f7ae7e9001f020ea07d44398c0ed55 Author: Tzu-Li (Gordon) Tai AuthorDate: Thu Oct 8 20:03:46 2020 +0800 Add blog "Stateful Functions Internals: Behind the scenes of Stateful Serverless" --- _posts/2020-10-13-stateful-serverless-internals.md | 213 + .../abstract-deployment.png| Bin 0 -> 122292 bytes .../aws-deployment.png | Bin 0 -> 251991 bytes .../protocol-walkthrough-1.png | Bin 0 -> 165851 bytes .../request-reply-protocol.png | Bin 0 -> 149165 bytes .../shopping-cart-overview.png | Bin 0 -> 109615 bytes 6 files changed, 213 insertions(+) diff --git a/_posts/2020-10-13-stateful-serverless-internals.md b/_posts/2020-10-13-stateful-serverless-internals.md new file mode 100644 index 000..9270915 --- /dev/null +++ b/_posts/2020-10-13-stateful-serverless-internals.md @@ -0,0 +1,213 @@ +--- +layout: post +title: "Stateful Functions Internals: Behind the scenes of Stateful Serverless" +subtitle: "A look at how Apache Flink Stateful Functions' runtime enables consistent and fault-tolerant stateful serverless applications" +date: 2020-10-13T08:00:00.000Z +categories: news +authors: +- tzulitai: + name: "Tzu-Li (Gordon) Tai" + twitter: "tzulitai" + +excerpt: This blog post dives deep into the internals of the StateFun runtime, taking a look at how it enables consistent and fault-tolerant stateful serverless applications. +--- + +Stateful Functions (StateFun) simplifies the building of distributed stateful applications by combining the best of two worlds: +the strong messaging and state consistency guarantees of stateful stream processing, and the elasticity and serverless experience of today's cloud-native architectures and +popular event-driven FaaS platforms. Typical StateFun applications consist of functions deployed behind simple services +using these modern platforms, with a separate StateFun cluster playing the role of an “[event-driven database](https://flink.apache.org/news/2020/04/07/release-statefun-2.0.0.html)” +that provides consistency and fault-tolerance for the functions' state and messaging. + +But how exactly does StateFun achieve that? How does the StateFun cluster communicate with the functions? + +This blog dives deep into the internals of the StateFun runtime. The entire walkthrough is complemented by a +[demo application](https://github.com/tzulitai/statefun-aws-demo/) which can be completely deployed on AWS services. +Most significantly, in the demo, the stateful functions are deployed and serviced using [AWS Lambda](https://aws.amazon.com/lambda/), +a popular FaaS platform among many others. The goal here is to allow readers to have a good grasp of the interaction between +the StateFun runtime and the functions, how that works cohesively to provide a Stateful Serverless experience, and how they can apply +what they've learnt to deploy their StateFun applications on other public cloud offerings such as GCP or Microsoft Azure. + +{% toc %} + +## Introducing the example: Shopping Cart + + + Note +You can find the full code [here](https://github.com/tzulitai/statefun-aws-demo/blob/master/app/shopping_cart.py), which +uses StateFun's [Python SDK](https://ci.apache.org/projects/flink/flink-statefun-docs-master/sdk/python.html). Alternatively, if you are +unfamiliar with StateFun's API, you can check out this [earlier blog](https://flink.apache.org/2020/08/19/statefun.html) +on modeling applications and stateful entities using [StateFun's programming constructs](https://ci.apache.org/projects/flink/flink-statefun-docs-master/concepts/application-building-blocks.html). + + +Let’s first take a look at a high-level overview of the motivating demo for this blog post: a shopping cart application. +The diagram below covers the functions that build up the application, the state that the functions would keep, and the messages +that flow between them. We’ll be referencing this example throughout the blog post. + + + + + Fig.1: An overly simplified shopping cart application. + + + + +The application consists of two function types: a `cart` function and an `inventory` function. +Each instance of the `cart` function is associated with a single user entity, with its state being the items in the cart +for that user (`ItemsInCart`). In the same way, each instance of the `inventory` function represents a single inventory, +maintaining as state the number of items in stock (`NumInStock`) as well as the number of items reserved across all +user carts (`NumReserved`). Messages can be sent to function instances using their logical addresses, which consists +of the function type and the instance's entity ID, e.g. `(car
[flink-web] branch asf-site updated: Rebuild website
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git The following commit(s) were added to refs/heads/asf-site by this push: new 3e1a78f Rebuild website 3e1a78f is described below commit 3e1a78fc43e3f9d11396aaff4304ae811a26e518 Author: Tzu-Li (Gordon) Tai AuthorDate: Tue Oct 13 21:01:19 2020 +0800 Rebuild website --- _posts/2020-10-13-stateful-serverless-internals.md | 2 +- content/blog/feed.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_posts/2020-10-13-stateful-serverless-internals.md b/_posts/2020-10-13-stateful-serverless-internals.md index 9270915..c637609 100644 --- a/_posts/2020-10-13-stateful-serverless-internals.md +++ b/_posts/2020-10-13-stateful-serverless-internals.md @@ -2,7 +2,7 @@ layout: post title: "Stateful Functions Internals: Behind the scenes of Stateful Serverless" subtitle: "A look at how Apache Flink Stateful Functions' runtime enables consistent and fault-tolerant stateful serverless applications" -date: 2020-10-13T08:00:00.000Z +date: 2020-10-13T00:00:00.000Z categories: news authors: - tzulitai: diff --git a/content/blog/feed.xml b/content/blog/feed.xml index b50a8e2..a78dc5f 100644 --- a/content/blog/feed.xml +++ b/content/blog/feed.xml @@ -226,7 +226,7 @@ StateFun cluster, and an AWS S3 bucket to store the periodic checkpoints. You caIf you’d like to learn more about Stateful Functions, head over to the official documentation, where you can also find more hands-on tutorials to try out yourself!
-Tue, 13 Oct 2020 10:00:00 +0200 +Tue, 13 Oct 2020 02:00:00 +0200 https://flink.apache.org/news/2020/10/13/stateful-serverless-internals.html /news/2020/10/13/stateful-serverless-internals.html
[flink] branch master updated (e8ad578 -> 0c37382)
This is an automated email from the ASF dual-hosted git repository. zhuzh pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e8ad578 [FLINK-19484][python][tests] Set longer timeout for jobmanager pod availability in PyFlink kubernetes application test (#13607) add f2d70c8 [FLINK-19570][tests] Remove preferred location tests from ExecutionTest add 0c37382 [FLINK-19570][tests] Avoid directly creating ExecutionJobVertex and ExecutionVertex via constructors in tests No new revisions were added by this update. Summary of changes: .../runtime/executiongraph/ExecutionJobVertex.java | 22 --- .../runtime/executiongraph/ExecutionVertex.java| 23 +-- .../checkpoint/StateAssignmentOperationTest.java | 20 +- .../executiongraph/AllVerticesIteratorTest.java| 27 +-- .../executiongraph/ExecutionGraphTestUtils.java| 68 ++- .../executiongraph/ExecutionJobVertexTest.java | 17 +- .../runtime/executiongraph/ExecutionTest.java | 59 -- .../executiongraph/ExecutionVertexCancelTest.java | 60 ++ .../ExecutionVertexDeploymentTest.java | 58 ++ .../ExecutionVertexSchedulingTest.java | 16 +- .../IntermediateResultPartitionTest.java | 21 ++- .../executiongraph/TestExecutionVertex.java| 55 -- .../scheduler/ScheduleWithCoLocationHintTest.java | 42 ++--- .../scheduler/SchedulerIsolatedTasksTest.java | 15 +- .../BackPressureRequestCoordinatorTest.java| 206 - 15 files changed, 186 insertions(+), 523 deletions(-) delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/TestExecutionVertex.java
[flink] branch master updated (e8ad578 -> 0c37382)
This is an automated email from the ASF dual-hosted git repository. zhuzh pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e8ad578 [FLINK-19484][python][tests] Set longer timeout for jobmanager pod availability in PyFlink kubernetes application test (#13607) add f2d70c8 [FLINK-19570][tests] Remove preferred location tests from ExecutionTest add 0c37382 [FLINK-19570][tests] Avoid directly creating ExecutionJobVertex and ExecutionVertex via constructors in tests No new revisions were added by this update. Summary of changes: .../runtime/executiongraph/ExecutionJobVertex.java | 22 --- .../runtime/executiongraph/ExecutionVertex.java| 23 +-- .../checkpoint/StateAssignmentOperationTest.java | 20 +- .../executiongraph/AllVerticesIteratorTest.java| 27 +-- .../executiongraph/ExecutionGraphTestUtils.java| 68 ++- .../executiongraph/ExecutionJobVertexTest.java | 17 +- .../runtime/executiongraph/ExecutionTest.java | 59 -- .../executiongraph/ExecutionVertexCancelTest.java | 60 ++ .../ExecutionVertexDeploymentTest.java | 58 ++ .../ExecutionVertexSchedulingTest.java | 16 +- .../IntermediateResultPartitionTest.java | 21 ++- .../executiongraph/TestExecutionVertex.java| 55 -- .../scheduler/ScheduleWithCoLocationHintTest.java | 42 ++--- .../scheduler/SchedulerIsolatedTasksTest.java | 15 +- .../BackPressureRequestCoordinatorTest.java| 206 - 15 files changed, 186 insertions(+), 523 deletions(-) delete mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/TestExecutionVertex.java
[flink] branch master updated (4eba4c3 -> e8ad578)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 4eba4c3 [FLINK-19309][coordination] Add TaskExecutorManager add e8ad578 [FLINK-19484][python][tests] Set longer timeout for jobmanager pod availability in PyFlink kubernetes application test (#13607) No new revisions were added by this update. Summary of changes: .../test-scripts/test_kubernetes_pyflink_application.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (4eba4c3 -> e8ad578)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 4eba4c3 [FLINK-19309][coordination] Add TaskExecutorManager add e8ad578 [FLINK-19484][python][tests] Set longer timeout for jobmanager pod availability in PyFlink kubernetes application test (#13607) No new revisions were added by this update. Summary of changes: .../test-scripts/test_kubernetes_pyflink_application.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (b60c59d -> 4eba4c3)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF add 4eba4c3 [FLINK-19309][coordination] Add TaskExecutorManager No new revisions were added by this update. Summary of changes: .../slotmanager/TaskExecutorManager.java | 447 + .../slotmanager/TaskExecutorManagerBuilder.java| 90 + .../slotmanager/TaskExecutorManagerTest.java | 351 3 files changed, 888 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManager.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerBuilder.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerTest.java
[flink] branch master updated (b60c59d -> 4eba4c3)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF add 4eba4c3 [FLINK-19309][coordination] Add TaskExecutorManager No new revisions were added by this update. Summary of changes: .../slotmanager/TaskExecutorManager.java | 447 + .../slotmanager/TaskExecutorManagerBuilder.java| 90 + .../slotmanager/TaskExecutorManagerTest.java | 351 3 files changed, 888 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManager.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerBuilder.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerTest.java
[flink] branch master updated (b60c59d -> 4eba4c3)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF add 4eba4c3 [FLINK-19309][coordination] Add TaskExecutorManager No new revisions were added by this update. Summary of changes: .../slotmanager/TaskExecutorManager.java | 447 + .../slotmanager/TaskExecutorManagerBuilder.java| 90 + .../slotmanager/TaskExecutorManagerTest.java | 351 3 files changed, 888 insertions(+) create mode 100644 flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManager.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerBuilder.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/resourcemanager/slotmanager/TaskExecutorManagerTest.java
[flink] branch master updated (f6f9cb7 -> b60c59d)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' add b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/state.py | 47 ++- flink-python/pyflink/common/types.py | 59 ++-- flink-python/pyflink/fn_execution/aggregate.py | 129 ++- .../pyflink/fn_execution/beam/beam_coders.py | 29 +- .../fn_execution/beam/beam_operations_fast.pyx | 29 +- .../fn_execution/beam/beam_operations_slow.py | 31 +- .../pyflink/fn_execution/flink_fn_execution_pb2.py | 380 - .../pyflink/fn_execution/operation_utils.py| 16 +- flink-python/pyflink/fn_execution/state_impl.py| 54 ++- .../pyflink/proto/flink-fn-execution.proto | 47 ++- flink-python/pyflink/table/__init__.py | 3 + flink-python/pyflink/table/data_view.py| 100 ++ flink-python/pyflink/table/tests/test_aggregate.py | 42 +++ flink-python/pyflink/table/tests/test_types.py | 10 + flink-python/pyflink/table/types.py| 50 ++- flink-python/pyflink/table/udf.py | 10 +- .../PythonStreamGroupAggregateOperator.java| 14 +- .../python/utils/PythonOperatorUtils.java | 46 ++- .../PythonStreamGroupAggregateOperatorTest.java| 2 + .../functions/python/PythonAggregateFunction.java | 29 +- .../table/planner/typeutils/DataViewUtils.java | 5 +- .../plan/nodes/common/CommonPythonAggregate.scala | 75 +++- .../stream/StreamExecPythonGroupAggregate.scala| 19 +- .../plan/stream/table/PythonAggregateTest.scala| 48 +++ 24 files changed, 1068 insertions(+), 206 deletions(-) create mode 100644 flink-python/pyflink/table/data_view.py
[flink] branch master updated (f6f9cb7 -> b60c59d)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' add b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/state.py | 47 ++- flink-python/pyflink/common/types.py | 59 ++-- flink-python/pyflink/fn_execution/aggregate.py | 129 ++- .../pyflink/fn_execution/beam/beam_coders.py | 29 +- .../fn_execution/beam/beam_operations_fast.pyx | 29 +- .../fn_execution/beam/beam_operations_slow.py | 31 +- .../pyflink/fn_execution/flink_fn_execution_pb2.py | 380 - .../pyflink/fn_execution/operation_utils.py| 16 +- flink-python/pyflink/fn_execution/state_impl.py| 54 ++- .../pyflink/proto/flink-fn-execution.proto | 47 ++- flink-python/pyflink/table/__init__.py | 3 + flink-python/pyflink/table/data_view.py| 100 ++ flink-python/pyflink/table/tests/test_aggregate.py | 42 +++ flink-python/pyflink/table/tests/test_types.py | 10 + flink-python/pyflink/table/types.py| 50 ++- flink-python/pyflink/table/udf.py | 10 +- .../PythonStreamGroupAggregateOperator.java| 14 +- .../python/utils/PythonOperatorUtils.java | 46 ++- .../PythonStreamGroupAggregateOperatorTest.java| 2 + .../functions/python/PythonAggregateFunction.java | 29 +- .../table/planner/typeutils/DataViewUtils.java | 5 +- .../plan/nodes/common/CommonPythonAggregate.scala | 75 +++- .../stream/StreamExecPythonGroupAggregate.scala| 19 +- .../plan/stream/table/PythonAggregateTest.scala| 48 +++ 24 files changed, 1068 insertions(+), 206 deletions(-) create mode 100644 flink-python/pyflink/table/data_view.py
[flink] branch master updated (f6f9cb7 -> b60c59d)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' add b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/state.py | 47 ++- flink-python/pyflink/common/types.py | 59 ++-- flink-python/pyflink/fn_execution/aggregate.py | 129 ++- .../pyflink/fn_execution/beam/beam_coders.py | 29 +- .../fn_execution/beam/beam_operations_fast.pyx | 29 +- .../fn_execution/beam/beam_operations_slow.py | 31 +- .../pyflink/fn_execution/flink_fn_execution_pb2.py | 380 - .../pyflink/fn_execution/operation_utils.py| 16 +- flink-python/pyflink/fn_execution/state_impl.py| 54 ++- .../pyflink/proto/flink-fn-execution.proto | 47 ++- flink-python/pyflink/table/__init__.py | 3 + flink-python/pyflink/table/data_view.py| 100 ++ flink-python/pyflink/table/tests/test_aggregate.py | 42 +++ flink-python/pyflink/table/tests/test_types.py | 10 + flink-python/pyflink/table/types.py| 50 ++- flink-python/pyflink/table/udf.py | 10 +- .../PythonStreamGroupAggregateOperator.java| 14 +- .../python/utils/PythonOperatorUtils.java | 46 ++- .../PythonStreamGroupAggregateOperatorTest.java| 2 + .../functions/python/PythonAggregateFunction.java | 29 +- .../table/planner/typeutils/DataViewUtils.java | 5 +- .../plan/nodes/common/CommonPythonAggregate.scala | 75 +++- .../stream/StreamExecPythonGroupAggregate.scala| 19 +- .../plan/stream/table/PythonAggregateTest.scala| 48 +++ 24 files changed, 1068 insertions(+), 206 deletions(-) create mode 100644 flink-python/pyflink/table/data_view.py
[flink] branch master updated (f6f9cb7 -> b60c59d)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' add b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/state.py | 47 ++- flink-python/pyflink/common/types.py | 59 ++-- flink-python/pyflink/fn_execution/aggregate.py | 129 ++- .../pyflink/fn_execution/beam/beam_coders.py | 29 +- .../fn_execution/beam/beam_operations_fast.pyx | 29 +- .../fn_execution/beam/beam_operations_slow.py | 31 +- .../pyflink/fn_execution/flink_fn_execution_pb2.py | 380 - .../pyflink/fn_execution/operation_utils.py| 16 +- flink-python/pyflink/fn_execution/state_impl.py| 54 ++- .../pyflink/proto/flink-fn-execution.proto | 47 ++- flink-python/pyflink/table/__init__.py | 3 + flink-python/pyflink/table/data_view.py| 100 ++ flink-python/pyflink/table/tests/test_aggregate.py | 42 +++ flink-python/pyflink/table/tests/test_types.py | 10 + flink-python/pyflink/table/types.py| 50 ++- flink-python/pyflink/table/udf.py | 10 +- .../PythonStreamGroupAggregateOperator.java| 14 +- .../python/utils/PythonOperatorUtils.java | 46 ++- .../PythonStreamGroupAggregateOperatorTest.java| 2 + .../functions/python/PythonAggregateFunction.java | 29 +- .../table/planner/typeutils/DataViewUtils.java | 5 +- .../plan/nodes/common/CommonPythonAggregate.scala | 75 +++- .../stream/StreamExecPythonGroupAggregate.scala| 19 +- .../plan/stream/table/PythonAggregateTest.scala| 48 +++ 24 files changed, 1068 insertions(+), 206 deletions(-) create mode 100644 flink-python/pyflink/table/data_view.py
[flink] branch master updated (f6f9cb7 -> b60c59d)
This is an automated email from the ASF dual-hosted git repository. dianfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' add b60c59d [FLINK-19231][python] Support ListState and ListView for Python UDAF No new revisions were added by this update. Summary of changes: flink-python/pyflink/common/state.py | 47 ++- flink-python/pyflink/common/types.py | 59 ++-- flink-python/pyflink/fn_execution/aggregate.py | 129 ++- .../pyflink/fn_execution/beam/beam_coders.py | 29 +- .../fn_execution/beam/beam_operations_fast.pyx | 29 +- .../fn_execution/beam/beam_operations_slow.py | 31 +- .../pyflink/fn_execution/flink_fn_execution_pb2.py | 380 - .../pyflink/fn_execution/operation_utils.py| 16 +- flink-python/pyflink/fn_execution/state_impl.py| 54 ++- .../pyflink/proto/flink-fn-execution.proto | 47 ++- flink-python/pyflink/table/__init__.py | 3 + flink-python/pyflink/table/data_view.py| 100 ++ flink-python/pyflink/table/tests/test_aggregate.py | 42 +++ flink-python/pyflink/table/tests/test_types.py | 10 + flink-python/pyflink/table/types.py| 50 ++- flink-python/pyflink/table/udf.py | 10 +- .../PythonStreamGroupAggregateOperator.java| 14 +- .../python/utils/PythonOperatorUtils.java | 46 ++- .../PythonStreamGroupAggregateOperatorTest.java| 2 + .../functions/python/PythonAggregateFunction.java | 29 +- .../table/planner/typeutils/DataViewUtils.java | 5 +- .../plan/nodes/common/CommonPythonAggregate.scala | 75 +++- .../stream/StreamExecPythonGroupAggregate.scala| 19 +- .../plan/stream/table/PythonAggregateTest.scala| 48 +++ 24 files changed, 1068 insertions(+), 206 deletions(-) create mode 100644 flink-python/pyflink/table/data_view.py
[flink] branch release-1.11 updated: [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink
This is an automated email from the ASF dual-hosted git repository. jark pushed a commit to branch release-1.11 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.11 by this push: new 42320ef [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink 42320ef is described below commit 42320ef205cb6afd98ffc9c2756a4abc7dc042e6 Author: Jark Wu AuthorDate: Tue Oct 13 18:44:55 2020 +0800 [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink This closes #13594 --- .../jdbc/table/JdbcDynamicOutputFormatBuilder.java | 21 ++ .../jdbc/table/UnsignedTypeConversionITCase.java | 81 ++ 2 files changed, 75 insertions(+), 27 deletions(-) diff --git a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java index 3b31d39..d02664c 100644 --- a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java +++ b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java @@ -114,20 +114,6 @@ public class JdbcDynamicOutputFormatBuilder implements Serializable { } } - private static JdbcBatchStatementExecutor createKeyedRowExecutor( - JdbcDialect dialect, - int[] pkFields, - LogicalType[] pkTypes, - String sql, - LogicalType[] logicalTypes) { - final JdbcRowConverter rowConverter = dialect.getRowConverter(RowType.of(pkTypes)); - final Function keyExtractor = createRowKeyExtractor(logicalTypes, pkFields); - return JdbcBatchStatementExecutor.keyed( - sql, - keyExtractor, - (st, record) -> rowConverter.toExternal(keyExtractor.apply(record), st)); - } - private static JdbcBatchStatementExecutor createBufferReduceExecutor( JdbcDmlOptions opt, RuntimeContext ctx, @@ -182,7 +168,12 @@ public class JdbcDynamicOutputFormatBuilder implements Serializable { checkArgument(dmlOptions.getKeyFields().isPresent()); String[] pkNames = Arrays.stream(pkFields).mapToObj(k -> dmlOptions.getFieldNames()[k]).toArray(String[]::new); String deleteSql = dmlOptions.getDialect().getDeleteStatement(dmlOptions.getTableName(), pkNames); - return createKeyedRowExecutor(dmlOptions.getDialect(), pkFields, pkTypes, deleteSql, fieldTypes); + JdbcRowConverter rowConverter = dmlOptions.getDialect().getRowConverter(RowType.of(pkTypes)); + Function keyExtractor = createRowKeyExtractor(fieldTypes, pkFields); + return JdbcBatchStatementExecutor.keyed( + deleteSql, + keyExtractor, + (st, key) -> rowConverter.toExternal(key, st)); } private static Function createRowKeyExtractor(LogicalType[] logicalTypes, int[] pkFields) { diff --git a/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java b/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java index 906d5a0..3553432 100644 --- a/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java +++ b/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java @@ -20,9 +20,13 @@ package org.apache.flink.connector.jdbc.table; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.DataTypes; +import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.TableEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.table.planner.factories.TestValuesTableFactory; import org.apache.flink.table.planner.runtime.utils.TableEnvUtil; +import org.apache.flink.table.planner.runtime.utils.TestData; import org.apache.flink.test.util.AbstractTestBase; import org.apache.flink.types.Row; @@ -42,11 +46,13 @@ import java.sql.DriverManager; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; +import java.sql.Statement; import java.util.Collections; import java.util.Iterator; import java.util.List; imp
[flink] branch release-1.11 updated: [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink
This is an automated email from the ASF dual-hosted git repository. jark pushed a commit to branch release-1.11 in repository https://gitbox.apache.org/repos/asf/flink.git The following commit(s) were added to refs/heads/release-1.11 by this push: new 42320ef [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink 42320ef is described below commit 42320ef205cb6afd98ffc9c2756a4abc7dc042e6 Author: Jark Wu AuthorDate: Tue Oct 13 18:44:55 2020 +0800 [FLINK-19423][jdbc] Fix ArrayIndexOutOfBoundsException when executing DELETE statement in JDBC upsert sink This closes #13594 --- .../jdbc/table/JdbcDynamicOutputFormatBuilder.java | 21 ++ .../jdbc/table/UnsignedTypeConversionITCase.java | 81 ++ 2 files changed, 75 insertions(+), 27 deletions(-) diff --git a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java index 3b31d39..d02664c 100644 --- a/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java +++ b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicOutputFormatBuilder.java @@ -114,20 +114,6 @@ public class JdbcDynamicOutputFormatBuilder implements Serializable { } } - private static JdbcBatchStatementExecutor createKeyedRowExecutor( - JdbcDialect dialect, - int[] pkFields, - LogicalType[] pkTypes, - String sql, - LogicalType[] logicalTypes) { - final JdbcRowConverter rowConverter = dialect.getRowConverter(RowType.of(pkTypes)); - final Function keyExtractor = createRowKeyExtractor(logicalTypes, pkFields); - return JdbcBatchStatementExecutor.keyed( - sql, - keyExtractor, - (st, record) -> rowConverter.toExternal(keyExtractor.apply(record), st)); - } - private static JdbcBatchStatementExecutor createBufferReduceExecutor( JdbcDmlOptions opt, RuntimeContext ctx, @@ -182,7 +168,12 @@ public class JdbcDynamicOutputFormatBuilder implements Serializable { checkArgument(dmlOptions.getKeyFields().isPresent()); String[] pkNames = Arrays.stream(pkFields).mapToObj(k -> dmlOptions.getFieldNames()[k]).toArray(String[]::new); String deleteSql = dmlOptions.getDialect().getDeleteStatement(dmlOptions.getTableName(), pkNames); - return createKeyedRowExecutor(dmlOptions.getDialect(), pkFields, pkTypes, deleteSql, fieldTypes); + JdbcRowConverter rowConverter = dmlOptions.getDialect().getRowConverter(RowType.of(pkTypes)); + Function keyExtractor = createRowKeyExtractor(fieldTypes, pkFields); + return JdbcBatchStatementExecutor.keyed( + deleteSql, + keyExtractor, + (st, key) -> rowConverter.toExternal(key, st)); } private static Function createRowKeyExtractor(LogicalType[] logicalTypes, int[] pkFields) { diff --git a/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java b/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java index 906d5a0..3553432 100644 --- a/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java +++ b/flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java @@ -20,9 +20,13 @@ package org.apache.flink.connector.jdbc.table; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.DataTypes; +import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.TableEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.table.planner.factories.TestValuesTableFactory; import org.apache.flink.table.planner.runtime.utils.TableEnvUtil; +import org.apache.flink.table.planner.runtime.utils.TestData; import org.apache.flink.test.util.AbstractTestBase; import org.apache.flink.types.Row; @@ -42,11 +46,13 @@ import java.sql.DriverManager; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; +import java.sql.Statement; import java.util.Collections; import java.util.Iterator; import java.util.List; imp
[flink] branch master updated (401f56f -> f6f9cb7)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 401f56f [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' add f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' No new revisions were added by this update. Summary of changes: .../streaming/connectors/cassandra/CassandraSink.java| 4 ++-- .../connectors/kafka/shuffle/FlinkKafkaShuffle.java | 4 ++-- .../flink/streaming/api/datastream/DataStreamSink.java | 8 .../flink/streaming/api/graph/StreamGraphGenerator.java | 10 +- .../api/operators/collect/CollectStreamSink.java | 8 ...Transformation.java => LegacySinkTransformation.java} | 16 .../planner/plan/nodes/common/CommonPhysicalSink.scala | 4 ++-- .../planner/runtime/batch/sql/join/JoinITCase.scala | 4 ++-- 8 files changed, 29 insertions(+), 29 deletions(-) rename flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/{SinkTransformation.java => LegacySinkTransformation.java} (87%)
[flink] branch master updated (401f56f -> f6f9cb7)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 401f56f [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' add f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' No new revisions were added by this update. Summary of changes: .../streaming/connectors/cassandra/CassandraSink.java| 4 ++-- .../connectors/kafka/shuffle/FlinkKafkaShuffle.java | 4 ++-- .../flink/streaming/api/datastream/DataStreamSink.java | 8 .../flink/streaming/api/graph/StreamGraphGenerator.java | 10 +- .../api/operators/collect/CollectStreamSink.java | 8 ...Transformation.java => LegacySinkTransformation.java} | 16 .../planner/plan/nodes/common/CommonPhysicalSink.scala | 4 ++-- .../planner/runtime/batch/sql/join/JoinITCase.scala | 4 ++-- 8 files changed, 29 insertions(+), 29 deletions(-) rename flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/{SinkTransformation.java => LegacySinkTransformation.java} (87%)
[flink] branch master updated (401f56f -> f6f9cb7)
This is an automated email from the ASF dual-hosted git repository. kkloudas pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 401f56f [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' add f6f9cb7 [FLINK-19511] Rename the 'SinkTransformation' to 'LegacySinkTransformation' No new revisions were added by this update. Summary of changes: .../streaming/connectors/cassandra/CassandraSink.java| 4 ++-- .../connectors/kafka/shuffle/FlinkKafkaShuffle.java | 4 ++-- .../flink/streaming/api/datastream/DataStreamSink.java | 8 .../flink/streaming/api/graph/StreamGraphGenerator.java | 10 +- .../api/operators/collect/CollectStreamSink.java | 8 ...Transformation.java => LegacySinkTransformation.java} | 16 .../planner/plan/nodes/common/CommonPhysicalSink.scala | 4 ++-- .../planner/runtime/batch/sql/join/JoinITCase.scala | 4 ++-- 8 files changed, 29 insertions(+), 29 deletions(-) rename flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/{SinkTransformation.java => LegacySinkTransformation.java} (87%)
[flink] 02/03: [tmpfix] Set CI log level to TRACE for 'org.apache.flink.connector.base.source.reader' to debug FLINK-19448
This is an automated email from the ASF dual-hosted git repository. sewen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit d0e41a04b8bec19aedf5ebe63950bc24958dcb9d Author: Stephan Ewen AuthorDate: Mon Oct 12 17:44:07 2020 +0200 [tmpfix] Set CI log level to TRACE for 'org.apache.flink.connector.base.source.reader' to debug FLINK-19448 --- tools/ci/log4j.properties | 5 + 1 file changed, 5 insertions(+) diff --git a/tools/ci/log4j.properties b/tools/ci/log4j.properties index 4550a98..bc5dd0d 100644 --- a/tools/ci/log4j.properties +++ b/tools/ci/log4j.properties @@ -73,3 +73,8 @@ logger.consumer.level = OFF logger.queryablestate.name = org.apache.flink.queryablestate logger.queryablestate.level = TRACE logger.queryablestate.appenderRef.out.ref = ConsoleAppender + +# Enable TRACE logging for the sources to debug FLINK-19448 +logger.sources.name = org.apache.flink.connector.base.source.reader +logger.sources.level = TRACE +logger.sources.appenderRef.out.ref = ConsoleAppender
[flink] 03/03: [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()'
This is an automated email from the ASF dual-hosted git repository. sewen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit 401f56fe9d6b0271260edf9787cdcbfe4d03874d Author: Stephan Ewen AuthorDate: Mon Oct 12 13:43:57 2020 +0200 [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' The changed logic also fixes flaky thread shutdown logic as a side effect, because it no longer relies on thread interrupting. This closes #13593 --- .../source/reader/fetcher/SplitFetcherTest.java| 54 +++--- 1 file changed, 26 insertions(+), 28 deletions(-) diff --git a/flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTest.java b/flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTest.java index 5027e3f..5082ebf 100644 --- a/flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTest.java +++ b/flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/source/reader/fetcher/SplitFetcherTest.java @@ -120,16 +120,16 @@ public class SplitFetcherTest { final SplitFetcher fetcher = createFetcherWithSplit( "test-split", queue, new TestingSplitReader<>(finishedSplitFetch("test-split"))); - final QueueDrainerThread queueDrainer = new QueueDrainerThread(queue); + final QueueDrainerThread queueDrainer = new QueueDrainerThread(queue, fetcher, 1); queueDrainer.start(); - try { - fetcher.runOnce(); + fetcher.runOnce(); - assertTrue(queue.getAvailabilityFuture().isDone()); - } finally { - queueDrainer.shutdown(); - } + queueDrainer.sync(); + + // either we got the notification that the fetcher went idle after the queue was drained (thread finished) + // or the fetcher was already idle when the thread drained the queue (then we need no additional notification) + assertTrue(queue.getAvailabilityFuture().isDone() || queueDrainer.wasIdleWhenFinished()); } @Test @@ -139,18 +139,15 @@ public class SplitFetcherTest { final SplitFetcher fetcher = createFetcherWithSplit( "test-split", queue, new TestingSplitReader<>(finishedSplitFetch("test-split"))); - final QueueDrainerThread queueDrainer = new QueueDrainerThread(queue); + final QueueDrainerThread queueDrainer = new QueueDrainerThread(queue, fetcher, 1); queueDrainer.start(); final CompletableFuture future = queue.getAvailabilityFuture(); - try { - fetcher.runOnce(); + fetcher.runOnce(); + assertTrue(future.isDone()); - assertTrue(future.isDone()); - } finally { - queueDrainer.shutdown(); - } + queueDrainer.sync(); } @Test @@ -274,31 +271,32 @@ public class SplitFetcherTest { private static final class QueueDrainerThread extends CheckedThread { private final FutureCompletingBlockingQueue queue; - private volatile boolean running = true; + private final SplitFetcher fetcher; + private final int numFetchesToTake; - QueueDrainerThread(FutureCompletingBlockingQueue queue) { + private volatile boolean wasIdleWhenFinished; + + QueueDrainerThread(FutureCompletingBlockingQueue queue, SplitFetcher fetcher, int numFetchesToTake) { super("Queue Drainer"); setPriority(Thread.MAX_PRIORITY); this.queue = queue; + this.fetcher = fetcher; + this.numFetchesToTake = numFetchesToTake; } @Override public void go() throws Exception { - while (running) { - try { - queue.take(); - } - catch (InterruptedException ignored) { - Thread.currentThread().interrupt(); - // fall through the loop - } + int remaining = numFetchesToTake; + while (remaining > 0) { + remaining--; + queue.take(); } + + was
[flink] 01/03: [FLINK-19448][connector base] Explicitly check for un-expected condition that would leave an inconsistent state
This is an automated email from the ASF dual-hosted git repository. sewen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git commit 68c5c2ff779d82a1ff81ffaf60d8a1b283797db1 Author: Stephan Ewen AuthorDate: Mon Oct 12 17:38:30 2020 +0200 [FLINK-19448][connector base] Explicitly check for un-expected condition that would leave an inconsistent state This condition should never happen, but if it ever happened, it would leave the source in an idle state waiting for more input data, rather than shutting down. --- .../flink/connector/base/source/reader/SourceReaderBase.java | 7 +-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java b/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java index 1182cae..e1da654 100644 --- a/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java +++ b/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/reader/SourceReaderBase.java @@ -262,10 +262,13 @@ public abstract class SourceReaderBase
[flink] branch master updated (232b61f -> 401f56f)
This is an automated email from the ASF dual-hosted git repository. sewen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 232b61f [FLINK-17857][e2e] Fix Kubernetes e2e tests on Mac OS new 68c5c2f [FLINK-19448][connector base] Explicitly check for un-expected condition that would leave an inconsistent state new d0e41a0 [tmpfix] Set CI log level to TRACE for 'org.apache.flink.connector.base.source.reader' to debug FLINK-19448 new 401f56f [FLINK-19427][FLINK-19489][tests] Fix test conditions for 'SplitFetcherTest.testNotifiesWhenGoingIdleConcurrent()' The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../base/source/reader/SourceReaderBase.java | 7 ++- .../source/reader/fetcher/SplitFetcherTest.java| 54 +++--- tools/ci/log4j.properties | 5 ++ 3 files changed, 36 insertions(+), 30 deletions(-)
[flink] branch master updated (84a6025 -> 232b61f)
This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 84a6025 [FLINK-19522][jdbc] Add "scan.auto-commit" option for JDBC SQL connector add 232b61f [FLINK-17857][e2e] Fix Kubernetes e2e tests on Mac OS No new revisions were added by this update. Summary of changes: .../test-scripts/common_docker.sh | 5 - .../test-scripts/common_kubernetes.sh | 24 ++ .../test-scripts/test_kubernetes_application.sh| 2 +- .../test-scripts/test_kubernetes_embedded_job.sh | 2 +- .../test-scripts/test_kubernetes_session.sh| 2 +- 5 files changed, 31 insertions(+), 4 deletions(-)
[flink] branch master updated (84a6025 -> 232b61f)
This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 84a6025 [FLINK-19522][jdbc] Add "scan.auto-commit" option for JDBC SQL connector add 232b61f [FLINK-17857][e2e] Fix Kubernetes e2e tests on Mac OS No new revisions were added by this update. Summary of changes: .../test-scripts/common_docker.sh | 5 - .../test-scripts/common_kubernetes.sh | 24 ++ .../test-scripts/test_kubernetes_application.sh| 2 +- .../test-scripts/test_kubernetes_embedded_job.sh | 2 +- .../test-scripts/test_kubernetes_session.sh| 2 +- 5 files changed, 31 insertions(+), 4 deletions(-)
[flink] branch master updated (84a6025 -> 232b61f)
This is an automated email from the ASF dual-hosted git repository. rmetzger pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 84a6025 [FLINK-19522][jdbc] Add "scan.auto-commit" option for JDBC SQL connector add 232b61f [FLINK-17857][e2e] Fix Kubernetes e2e tests on Mac OS No new revisions were added by this update. Summary of changes: .../test-scripts/common_docker.sh | 5 - .../test-scripts/common_kubernetes.sh | 24 ++ .../test-scripts/test_kubernetes_application.sh| 2 +- .../test-scripts/test_kubernetes_embedded_job.sh | 2 +- .../test-scripts/test_kubernetes_session.sh| 2 +- 5 files changed, 31 insertions(+), 4 deletions(-)
