[flink] branch release-1.12 updated (800c6a6 -> 387eb5c)
This is an automated email from the ASF dual-hosted git repository. hxb pushed a change to branch release-1.12 in repository https://gitbox.apache.org/repos/asf/flink.git. from 800c6a6 [hotfix] Use AllocationID instead of JobID when slot cannot be marked as active. add 387eb5c [FLINK-22172][python] Fix the bug of shared resource among Python Operators of the same slot is not released No new revisions were added by this update. Summary of changes: .../api/runners/python/beam/BeamPythonFunctionRunner.java | 5 + .../api/runners/python/beam/PythonSharedResources.java | 14 -- 2 files changed, 1 insertion(+), 18 deletions(-)
[flink] branch master updated: [FLINK-22172][python] Fix the bug of shared resource among Python Operators of the same slot is not released
This is an automated email from the ASF dual-hosted git repository.
hxb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 023568b [FLINK-22172][python] Fix the bug of shared resource among
Python Operators of the same slot is not released
023568b is described below
commit 023568b83d971b439468d680bddc66d584c6f7c4
Author: huangxingbo
AuthorDate: Fri Apr 9 14:33:59 2021 +0800
[FLINK-22172][python] Fix the bug of shared resource among Python Operators
of the same slot is not released
This closes #15537.
---
.../api/runners/python/beam/BeamPythonFunctionRunner.java | 5 +
.../api/runners/python/beam/PythonSharedResources.java | 14 --
2 files changed, 1 insertion(+), 18 deletions(-)
diff --git
a/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/BeamPythonFunctionRunner.java
b/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/BeamPythonFunctionRunner.java
index 4e2e1d2..1e95b67 100644
---
a/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/BeamPythonFunctionRunner.java
+++
b/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/BeamPythonFunctionRunner.java
@@ -294,10 +294,7 @@ public abstract class BeamPythonFunctionRunner implements
PythonFunctionRunner {
try {
if (sharedResources != null) {
-if (sharedResources.getResourceHandle().release()) {
-// release sharedResources iff there are no more Python
operators sharing it
-sharedResources.close();
-}
+sharedResources.close();
} else {
// if sharedResources is not null, the close of
environmentManager will be managed
// in sharedResources,
diff --git
a/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/PythonSharedResources.java
b/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/PythonSharedResources.java
index 3f5ec9e..c81e723 100644
---
a/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/PythonSharedResources.java
+++
b/flink-python/src/main/java/org/apache/flink/streaming/api/runners/python/beam/PythonSharedResources.java
@@ -44,14 +44,10 @@ public final class PythonSharedResources implements
AutoCloseable {
/** Keep track of the PythonEnvironmentManagers of the Python operators in
one slot. */
private final List environmentManagers;
-/** Keep track of the number of Python operators sharing this Python
resource. */
-private int refCnt;
-
PythonSharedResources(JobBundleFactory jobBundleFactory, Environment
environment) {
this.jobBundleFactory = jobBundleFactory;
this.environment = environment;
this.environmentManagers = new ArrayList<>();
-this.refCnt = 0;
}
JobBundleFactory getJobBundleFactory() {
@@ -64,16 +60,6 @@ public final class PythonSharedResources implements
AutoCloseable {
void addPythonEnvironmentManager(PythonEnvironmentManager
environmentManager) {
environmentManagers.add(environmentManager);
-refCnt++;
-}
-
-/**
- * Release a Python operator which shares this Python resource. Returns
true if there are no
- * more Python operators sharing this Python resource.
- */
-boolean release() {
-refCnt--;
-return refCnt == 0;
}
@Override
[flink] branch release-1.12 updated (f541f9d -> 800c6a6)
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a change to branch release-1.12 in repository https://gitbox.apache.org/repos/asf/flink.git. from f541f9d [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup add 800c6a6 [hotfix] Use AllocationID instead of JobID when slot cannot be marked as active. No new revisions were added by this update. Summary of changes: .../apache/flink/runtime/taskexecutor/TaskExecutor.java | 17 - 1 file changed, 8 insertions(+), 9 deletions(-)
[flink] branch master updated: [hotfix] Use AllocationID instead of JobID when slot cannot be marked as active.
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new bcfaf0b [hotfix] Use AllocationID instead of JobID when slot cannot
be marked as active.
bcfaf0b is described below
commit bcfaf0b475c18ac03b689ed8d9fd238a5b7c847e
Author: Till Rohrmann
AuthorDate: Fri Apr 9 18:31:11 2021 +0200
[hotfix] Use AllocationID instead of JobID when slot cannot be marked as
active.
---
.../apache/flink/runtime/taskexecutor/TaskExecutor.java | 17 -
1 file changed, 8 insertions(+), 9 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
index afae229..2ddb346 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
@@ -1512,22 +1512,21 @@ public class TaskExecutor extends RpcEndpoint
implements TaskExecutorGateway {
if (isJobManagerConnectionValid(jobId, jobMasterId)) {
// mark accepted slots active
for (SlotOffer acceptedSlot : acceptedSlots) {
+final AllocationID allocationId =
acceptedSlot.getAllocationId();
try {
-if
(!taskSlotTable.markSlotActive(acceptedSlot.getAllocationId())) {
+if (!taskSlotTable.markSlotActive(allocationId)) {
// the slot is either free or releasing at the
moment
-final String message = "Could not mark slot "
+ jobId + " active.";
+final String message =
+"Could not mark slot " + allocationId
+ " active.";
log.debug(message);
jobMasterGateway.failSlot(
-getResourceID(),
-acceptedSlot.getAllocationId(),
-new FlinkException(message));
+getResourceID(), allocationId, new
FlinkException(message));
}
} catch (SlotNotFoundException e) {
-final String message = "Could not mark slot " +
jobId + " active.";
+final String message =
+"Could not mark slot " + allocationId + "
active.";
jobMasterGateway.failSlot(
-getResourceID(),
-acceptedSlot.getAllocationId(),
-new FlinkException(message));
+getResourceID(), allocationId, new
FlinkException(message));
}
offeredSlots.remove(acceptedSlot);
[flink] branch master updated (e294300 -> 8ef6bd2)
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from e294300 [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup add 8ef6bd2 [hotfix][docs] Add missing parenthesis No new revisions were added by this update. Summary of changes: docs/content.zh/docs/dev/datastream/operators/windows.md | 2 +- docs/content/docs/dev/datastream/operators/windows.md| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[flink-web] branch asf-site updated: [FLINK-22116] Setup .asf.yaml for publishing
This is an automated email from the ASF dual-hosted git repository. chesnay pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git The following commit(s) were added to refs/heads/asf-site by this push: new f810b35 [FLINK-22116] Setup .asf.yaml for publishing f810b35 is described below commit f810b35f138691e4a32356469399035b173497bd Author: Chesnay Schepler AuthorDate: Thu Apr 8 12:27:44 2021 +0200 [FLINK-22116] Setup .asf.yaml for publishing --- .asf.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.asf.yaml b/.asf.yaml new file mode 100644 index 000..08e79d7 --- /dev/null +++ b/.asf.yaml @@ -0,0 +1,2 @@ +publish: + whoami: asf-site \ No newline at end of file
[flink] branch release-1.11 updated (f90493f -> 35babaf)
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a change to branch release-1.11 in repository https://gitbox.apache.org/repos/asf/flink.git. from f90493f [FLINK-22142][ci] Don't log Kafka connector output to the AZP console add b653222 [hotfix][test] Add error message for CommonTestUtils#waitUntilCondition add e3066e4 [FLINK-21148][test] Removes runTest in runDetachedModeTest add 7c7f7fe [FLINK-21148][test] Makes the test fail early when running into a timeout add 9250c4a [FLINK-21148][test] Increases timeout add 6f1e1b5 [FLINK-21148][test] Refactors test code to use CommonTestUtils for loops add 278839b [hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles add 35babaf [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup No new revisions were added by this update. Summary of changes: .../runtime/entrypoint/ClusterEntrypoint.java | 43 ++- .../DispatcherResourceManagerComponent.java| 28 +- .../runtime/entrypoint/ClusterEntrypointTest.java | 424 + .../TestingHighAvailabilityServices.java | 18 +- .../TestingHighAvailabilityServicesBuilder.java| 21 + .../flink/runtime/testutils/CommonTestUtils.java | 18 +- .../flink/runtime/testutils/TestJvmProcess.java| 19 + .../testutils/TestingClusterEntrypointProcess.java | 138 +++ .../flink/runtime/util/BlockingShutdownTest.java | 11 +- .../recovery/ProcessFailureCancelingITCase.java| 2 +- .../apache/flink/yarn/YARNSessionFIFOITCase.java | 308 +++ .../flink/yarn/YARNSessionFIFOSecuredITCase.java | 16 +- .../java/org/apache/flink/yarn/YarnTestBase.java | 18 +- 13 files changed, 868 insertions(+), 196 deletions(-) create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/ClusterEntrypointTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/testutils/TestingClusterEntrypointProcess.java
[flink] branch release-1.12 updated (688378e -> f541f9d)
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a change to branch release-1.12 in repository https://gitbox.apache.org/repos/asf/flink.git. from 688378e [FLINK-21833][table-runtime-blink] Fix state of TemporalRowTimeJoinOperator increase unlimitedly even state ttl is enabled add a87c587 [FLINK-21148][test] Removes runTest in runDetachedModeTest add 1ee4cb4 [FLINK-21148][test] Makes the test fail early when running into a timeout add 3b7ef1c [FLINK-21148][test] Increases timeout add eebe63d [FLINK-21148][test] Refactors test code to use CommonTestUtils for loops add d12db81 [hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles add f541f9d [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup No new revisions were added by this update. Summary of changes: .../runtime/entrypoint/ClusterEntrypoint.java | 43 ++- .../DispatcherResourceManagerComponent.java| 28 +- .../runtime/entrypoint/ClusterEntrypointTest.java | 422 + .../TestingHighAvailabilityServices.java | 18 +- .../TestingHighAvailabilityServicesBuilder.java| 21 + .../flink/runtime/testutils/TestJvmProcess.java| 19 + .../testutils/TestingClusterEntrypointProcess.java | 136 +++ .../flink/runtime/util/BlockingShutdownTest.java | 11 +- .../recovery/ProcessFailureCancelingITCase.java| 2 +- .../apache/flink/yarn/YARNSessionFIFOITCase.java | 308 +++ .../flink/yarn/YARNSessionFIFOSecuredITCase.java | 16 +- .../java/org/apache/flink/yarn/YarnTestBase.java | 18 +- 12 files changed, 847 insertions(+), 195 deletions(-) create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/entrypoint/ClusterEntrypointTest.java create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/testutils/TestingClusterEntrypointProcess.java
[flink] 01/02: [hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit 3b6af3d348ee854e51a55e58371d0592fbee1cb8
Author: wangyang0918
AuthorDate: Sat Apr 3 01:03:42 2021 +0800
[hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles
---
.../org/apache/flink/yarn/YARNSessionFIFOITCase.java | 5 -
.../flink/yarn/YARNSessionFIFOSecuredITCase.java | 16 +---
.../test/java/org/apache/flink/yarn/YarnTestBase.java | 18 --
3 files changed, 25 insertions(+), 14 deletions(-)
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java
index e82396c..8ab421f 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java
@@ -92,7 +92,7 @@ public class YARNSessionFIFOITCase extends YarnTestBase {
}
/** Test regular operation, including command line parameter parsing. */
-void runDetachedModeTest(Map securityProperties) throws
Exception {
+ApplicationId runDetachedModeTest(Map securityProperties)
throws Exception {
LOG.info("Starting testDetachedMode()");
File exampleJarLocation = getTestJarPath("StreamingWordCount.jar");
@@ -169,10 +169,12 @@ public class YARNSessionFIFOITCase extends YarnTestBase {
+ ")");
LOG.info("Waiting until the job reaches FINISHED state");
+final ApplicationId applicationId =
getOnlyApplicationReport().getApplicationId();
CommonTestUtils.waitUntilCondition(
() ->
verifyStringsInNamedLogFiles(
new String[] {"switched from state RUNNING to
FINISHED"},
+applicationId,
"jobmanager.log"),
Deadline.fromNow(timeout),
testConditionIntervalInMillis,
@@ -242,6 +244,7 @@ public class YARNSessionFIFOITCase extends YarnTestBase {
}
LOG.info("Finished testDetachedMode()");
+return applicationId;
}
/**
diff --git
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java
index b3e188a..eae6d42 100644
---
a/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java
+++
b/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java
@@ -30,6 +30,7 @@ import org.apache.flink.yarn.util.TestHadoopModuleFactory;
import org.apache.flink.shaded.guava18.com.google.common.collect.Lists;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.security.AMRMTokenIdentifier;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceScheduler;
@@ -150,8 +151,8 @@ public class YARNSessionFIFOSecuredITCase extends
YARNSessionFIFOITCase {
SecurityOptions.KERBEROS_LOGIN_PRINCIPAL.key(),
SecureTestEnvironment.getHadoopServicePrincipal());
}
-runDetachedModeTest(securityProperties);
-verifyResultContainsKerberosKeytab();
+final ApplicationId applicationId =
runDetachedModeTest(securityProperties);
+verifyResultContainsKerberosKeytab(applicationId);
});
}
@@ -171,17 +172,18 @@ public class YARNSessionFIFOSecuredITCase extends
YARNSessionFIFOITCase {
SecurityOptions.KERBEROS_LOGIN_PRINCIPAL.key(),
SecureTestEnvironment.getHadoopServicePrincipal());
}
-runDetachedModeTest(securityProperties);
-verifyResultContainsKerberosKeytab();
+final ApplicationId applicationId =
runDetachedModeTest(securityProperties);
+verifyResultContainsKerberosKeytab(applicationId);
});
}
-private static void verifyResultContainsKerberosKeytab() throws Exception {
+private static void verifyResultContainsKerberosKeytab(ApplicationId
applicationId)
+throws Exception {
final String[] mustHave = {"Login successful for user", "using keytab
file"};
final boolean jobManagerRunsWithKerberos =
-verifyStringsInNamedLogFiles(mustHave, "jobmanager.log");
+verifyStringsInNamedLogFiles(mustHave, applicationId,
"jobmanager.log");
final boolean taskManagerRunsWithKerberos =
-
[flink] 02/02: [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup
This is an automated email from the ASF dual-hosted git repository.
trohrmann pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
commit e2943006aec1febcfd1c33eef1b1f44367019a98
Author: wangyang0918
AuthorDate: Thu Apr 1 17:24:50 2021 +0800
[FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as
shutdown hook for the cleanup
This closes #15396.
---
.../runtime/entrypoint/ClusterEntrypoint.java | 43 ++-
.../DispatcherResourceManagerComponent.java| 28 +-
.../runtime/entrypoint/ClusterEntrypointTest.java | 387 -
.../TestingHighAvailabilityServices.java | 18 +-
.../TestingHighAvailabilityServicesBuilder.java| 21 ++
.../flink/runtime/testutils/TestJvmProcess.java| 19 +
.../testutils/TestingClusterEntrypointProcess.java | 136
.../flink/runtime/util/BlockingShutdownTest.java | 11 +-
.../recovery/ProcessFailureCancelingITCase.java| 2 +-
9 files changed, 625 insertions(+), 40 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
index aab1c7a..69b090d 100755
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/ClusterEntrypoint.java
@@ -165,7 +165,7 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
shutDownHook =
ShutdownHookUtil.addShutdownHook(
-this::cleanupDirectories, getClass().getSimpleName(),
LOG);
+() -> this.closeAsync().join(),
getClass().getSimpleName(), LOG);
}
public CompletableFuture getTerminationFuture() {
@@ -198,6 +198,7 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
// clean up any partial state
shutDownAsync(
ApplicationStatus.FAILED,
+ShutdownBehaviour.STOP_APPLICATION,
ExceptionUtils.stringifyException(strippedThrowable),
false)
.get(
@@ -266,13 +267,18 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
if (throwable != null) {
shutDownAsync(
ApplicationStatus.UNKNOWN,
+ShutdownBehaviour.STOP_APPLICATION,
ExceptionUtils.stringifyException(throwable),
false);
} else {
// This is the general shutdown path. If a
separate more
// specific shutdown was
// already triggered, this will do nothing
-shutDownAsync(applicationStatus, null,
true);
+shutDownAsync(
+applicationStatus,
+ShutdownBehaviour.STOP_APPLICATION,
+null,
+true);
}
});
}
@@ -363,10 +369,13 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
@Override
public CompletableFuture closeAsync() {
+ShutdownHookUtil.removeShutdownHook(shutDownHook,
getClass().getSimpleName(), LOG);
+
return shutDownAsync(
ApplicationStatus.UNKNOWN,
+ShutdownBehaviour.STOP_PROCESS,
"Cluster entrypoint has been closed externally.",
-true)
+false)
.thenAccept(ignored -> {});
}
@@ -465,6 +474,7 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
private CompletableFuture shutDownAsync(
ApplicationStatus applicationStatus,
+ShutdownBehaviour shutdownBehaviour,
@Nullable String diagnostics,
boolean cleanupHaData) {
if (isShutDown.compareAndSet(false, true)) {
@@ -475,7 +485,7 @@ public abstract class ClusterEntrypoint implements
AutoCloseableAsync, FatalErro
diagnostics);
final CompletableFuture shutDownApplicationFuture =
-closeClusterComponent(applicationStatus, diagnostics);
+closeClusterComponent(applicationStatus,
shutdownBehaviour, diagnostics);
final CompletableFuture
[flink] branch master updated (aff7bb3 -> e294300)
This is an automated email from the ASF dual-hosted git repository. trohrmann pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from aff7bb3 [hotfix][table-runtime] Fix the exception info of raw format (#15418) new 3b6af3d [hotfix][yarn-tests] Check the applicationId in verifyStringsInNamedLogFiles new e294300 [FLINK-21008][coordination] Register ClusterEntrypoint#closeAsync as shutdown hook for the cleanup The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runtime/entrypoint/ClusterEntrypoint.java | 43 ++- .../DispatcherResourceManagerComponent.java| 28 +- .../runtime/entrypoint/ClusterEntrypointTest.java | 387 - .../TestingHighAvailabilityServices.java | 18 +- .../TestingHighAvailabilityServicesBuilder.java| 21 ++ .../flink/runtime/testutils/TestJvmProcess.java| 19 + .../testutils/TestingClusterEntrypointProcess.java | 136 .../flink/runtime/util/BlockingShutdownTest.java | 11 +- .../recovery/ProcessFailureCancelingITCase.java| 2 +- .../apache/flink/yarn/YARNSessionFIFOITCase.java | 5 +- .../flink/yarn/YARNSessionFIFOSecuredITCase.java | 16 +- .../java/org/apache/flink/yarn/YarnTestBase.java | 18 +- 12 files changed, 650 insertions(+), 54 deletions(-) create mode 100644 flink-runtime/src/test/java/org/apache/flink/runtime/testutils/TestingClusterEntrypointProcess.java
[flink] branch master updated (276e847 -> aff7bb3)
This is an automated email from the ASF dual-hosted git repository. kurt pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 276e847 [FLINK-20387][table] Support TIMESTAMP_LTZ as rowtime attribute add aff7bb3 [hotfix][table-runtime] Fix the exception info of raw format (#15418) No new revisions were added by this update. Summary of changes: .../apache/flink/table/formats/raw/RawFormatSerializationSchema.java| 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[flink] branch master updated (6faaa37 -> 276e847)
This is an automated email from the ASF dual-hosted git repository. jark pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 6faaa37 [FLINK-21519] Temporarily disable SQLClientHBaseITCase add 276e847 [FLINK-20387][table] Support TIMESTAMP_LTZ as rowtime attribute No new revisions were added by this update. Summary of changes: .../PythonStreamGroupWindowAggregateOperator.java | 15 +- ...PythonGroupWindowAggregateFunctionOperator.java | 35 +- ...ghPythonStreamGroupWindowAggregateOperator.java | 9 +- .../flink/table/catalog/DefaultSchemaResolver.java | 51 +- .../flink/table/catalog/SchemaResolutionTest.java | 113 .../java/org/apache/flink/table/api/Schema.java| 4 +- .../org/apache/flink/table/api/TableSchema.java| 11 +- .../types/logical/utils/LogicalTypeChecks.java | 22 +- .../apache/flink/table/api/TableSchemaTest.java| 28 +- .../table/planner/catalog/CatalogSchemaTable.java | 4 +- .../expressions/PlannerRowtimeAttribute.java | 7 +- .../functions/sql/FlinkSqlOperatorTable.java | 16 +- .../exec/serde/RelDataTypeJsonDeserializer.java| 5 +- .../exec/serde/RelDataTypeJsonSerializer.java | 2 + .../nodes/exec/stream/StreamExecLegacySink.java| 14 +- .../stream/StreamExecLocalWindowAggregate.java | 3 +- .../plan/nodes/exec/stream/StreamExecMatch.java| 4 +- ...WatermarkIntoTableSourceScanAcrossCalcRule.java | 7 +- .../table/planner/calcite/FlinkTypeFactory.scala | 21 +- .../calcite/RelTimeIndicatorConverter.scala| 117 +++- .../table/planner/codegen/ExprCodeGenerator.scala | 2 +- .../table/planner/codegen/GenerateUtils.scala | 9 +- .../table/planner/codegen/MatchCodeGenerator.scala | 5 +- .../codegen/WatermarkGeneratorCodeGenerator.scala | 5 +- .../plan/nodes/calcite/WatermarkAssigner.scala | 6 +- ...gicalCorrelateToJoinFromTemporalTableRule.scala | 8 + .../StreamPhysicalGroupWindowAggregateRule.scala | 1 - .../stream/StreamPhysicalIntervalJoinRule.scala| 13 + .../plan/schema/LegacyCatalogSourceTable.scala | 2 +- .../plan/schema/TimeIndicatorRelDataType.scala | 8 +- .../table/planner/plan/utils/WindowUtil.scala | 4 +- .../nodes/exec/serde/RelDataTypeSerdeTest.java | 3 +- .../jsonplan/GroupWindowAggregateJsonITCase.java | 2 +- .../stream/jsonplan/TemporalSortJsonITCase.java| 2 +- .../stream/jsonplan/WindowAggregateJsonITCase.java | 2 +- .../testDeduplication.out | 1 + .../testProcTimeHopWindow.out | 40 +- .../testProcTimeSessionWindow.out | 40 +- .../testProcTimeTumbleWindow.out | 46 +- .../testProcessingTimeInnerJoinWithOnClause.out| 78 +-- .../testJoinTemporalTable.out | 3 + ...testJoinTemporalTableWithProjectionPushDown.out | 3 + .../testMatch.out | 27 +- .../testProcTimeBoundedNonPartitionedRangeOver.out | 46 +- .../testProcTimeBoundedPartitionedRangeOver.out| 46 +- ...undedPartitionedRowsOverWithBuiltinProctime.out | 39 +- .../testProcTimeUnboundedPartitionedRangeOver.out | 46 +- ...stProctimeBoundedDistinctPartitionedRowOver.out | 46 +- ...edDistinctWithNonDistinctPartitionedRowOver.out | 46 +- .../testSortProcessingTime.out | 39 +- .../testEventTimeCumulateWindow.out| 52 +- .../testEventTimeHopWindow.out | 52 +- .../testEventTimeTumbleWindow.out | 52 +- .../testProcTimeCumulateWindow.out | 46 +- .../testProcTimeHopWindow.out | 46 +- .../testProcTimeTumbleWindow.out | 46 +- .../ExpandWindowTableFunctionTransposeRuleTest.xml | 6 +- .../planner/plan/stream/sql/MatchRecognizeTest.xml | 66 ++ .../plan/stream/sql/MiniBatchIntervalInferTest.xml | 2 +- .../stream/sql/RelTimeIndicatorConverterTest.xml | 2 +- .../plan/stream/sql/SourceWatermarkTest.xml| 97 +-- .../table/planner/plan/stream/sql/UnionTest.xml| 35 ++ .../planner/plan/stream/sql/WindowRankTest.xml | 26 +- .../plan/stream/sql/WindowTableFunctionTest.xml| 8 +- .../plan/stream/sql/agg/WindowAggregateTest.xml| 162 ++--- .../plan/stream/sql/join/IntervalJoinTest.xml | 140 +++-- .../stream/sql/join/TemporalFunctionJoinTest.xml | 8 +- .../plan/stream/sql/join/TemporalJoinTest.xml | 33 + .../plan/stream/sql/join/WindowJoinTest.xml| 64 +- .../stream/table/TemporalTableFunctionJoinTest.xml | 10 +- .../plan/schema/TimeIndicatorRelDataTypeTest.scala | 9 +- .../plan/stream/sql/MatchRecognizeTest.scala | 71 +++ .../plan/stream/sql/SourceWatermarkTest.scala | 132 ++-- .../table/planner/plan/stream/sql/UnionTest.scala | 78 ++- .../plan/stream/sql/join/IntervalJoinTest.scala| 56 ++
[flink] branch master updated (440ec74 -> 6faaa37)
This is an automated email from the ASF dual-hosted git repository. dwysakowicz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 440ec74 [FLINK_21972][table-planner-blink] check whether TemporalTableSourceSpec can be serialized or not add 6faaa37 [FLINK-21519] Temporarily disable SQLClientHBaseITCase No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/tests/util/hbase/SQLClientHBaseITCase.java| 2 ++ 1 file changed, 2 insertions(+)
[flink] branch master updated: [FLINK_21972][table-planner-blink] check whether TemporalTableSourceSpec can be serialized or not
This is an automated email from the ASF dual-hosted git repository.
godfrey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 440ec74 [FLINK_21972][table-planner-blink] check whether
TemporalTableSourceSpec can be serialized or not
440ec74 is described below
commit 440ec74c2ac7880e1e9ef91a08219ca1e6a28869
Author: zjuwangg
AuthorDate: Thu Mar 25 18:19:28 2021 +0800
[FLINK_21972][table-planner-blink] check whether TemporalTableSourceSpec
can be serialized or not
This closes #15370
---
.../exec/serde/ExecNodeGraphJsonPlanGenerator.java | 10 +
.../nodes/exec/stream/LookupJoinJsonPlanTest.java | 45 ++
2 files changed, 55 insertions(+)
diff --git
a/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/ExecNodeGraphJsonPlanGenerator.java
b/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/ExecNodeGraphJsonPlanGenerator.java
index 0b8e9c4..f18dcb4 100644
---
a/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/ExecNodeGraphJsonPlanGenerator.java
+++
b/flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/serde/ExecNodeGraphJsonPlanGenerator.java
@@ -144,6 +144,16 @@ public class ExecNodeGraphJsonPlanGenerator {
"%s does not implement
@JsonCreator annotation on constructor.",
node.getClass().getCanonicalName()));
}
+if (node instanceof StreamExecLookupJoin) {
+StreamExecLookupJoin streamExecLookupJoin =
(StreamExecLookupJoin) node;
+if (null
+== streamExecLookupJoin
+.getTemporalTableSourceSpec()
+.getTableSourceSpec()) {
+throw new TableException(
+"TemporalTableSourceSpec can not be
serialized.");
+}
+}
super.visitInputs(node);
}
};
diff --git
a/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinJsonPlanTest.java
b/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinJsonPlanTest.java
index 2801712..4a0227c 100644
---
a/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinJsonPlanTest.java
+++
b/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinJsonPlanTest.java
@@ -18,14 +18,22 @@
package org.apache.flink.table.planner.plan.nodes.exec.stream;
+import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.table.api.TableConfig;
import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.TableSchema;
+import
org.apache.flink.table.planner.runtime.utils.InMemoryLookupableTableSource;
import org.apache.flink.table.planner.utils.StreamTableTestUtil;
import org.apache.flink.table.planner.utils.TableTestBase;
+import org.apache.flink.types.Row;
import org.junit.Before;
import org.junit.Test;
+import java.util.ArrayList;
+
+import scala.collection.JavaConverters;
+
/** Test json serialization/deserialization for LookupJoin. */
public class LookupJoinJsonPlanTest extends TableTestBase {
@@ -102,4 +110,41 @@ public class LookupJoinJsonPlanTest extends TableTestBase {
+ "JOIN LookupTable FOR SYSTEM_TIME AS OF T.proctime
AS D \n"
+ "ON T.a = D.id\n");
}
+
+@Test
+public void testLegacyTableSourceException() {
+expectedException().expectMessage("TemporalTableSourceSpec can not be
serialized.");
+TableSchema tableSchema =
+TableSchema.builder()
+.field("id", Types.INT)
+.field("name", Types.STRING)
+.field("age", Types.INT)
+.build();
+InMemoryLookupableTableSource.createTemporaryTable(
+tEnv,
+false,
+JavaConverters.asScalaIteratorConverter(new
ArrayList().iterator())
+.asScala()
+.toList(),
+tableSchema,
+"LookupTable",
+true);
+String sinkTableDdl =
+"CREATE TABLE MySink (\n"
++ " a int,\n"
++ " b varchar,"
+
[flink-statefun-playground] branch release-3.0 created (now ebbf53e)
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a change to branch release-3.0 in repository https://gitbox.apache.org/repos/asf/flink-statefun-playground.git. at ebbf53e [release-3.0] Updating all examples for 3.0 release No new revisions were added by this update.
[flink-statefun-playground] branch dev updated: [release-3.0] Updating all examples for 3.0 release
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a commit to branch dev in repository https://gitbox.apache.org/repos/asf/flink-statefun-playground.git The following commit(s) were added to refs/heads/dev by this push: new ebbf53e [release-3.0] Updating all examples for 3.0 release ebbf53e is described below commit ebbf53e9bdc0809dd4533c54096ea7dc2db8fb80 Author: Tzu-Li (Gordon) Tai AuthorDate: Fri Apr 9 15:43:22 2021 +0800 [release-3.0] Updating all examples for 3.0 release --- deployments/aws-lambda/pom.xml| 4 ++-- java/greeter/Dockerfile | 9 - java/greeter/docker-compose.yml | 4 ++-- java/greeter/pom.xml | 4 ++-- java/greeter/statefun-sdk-java-3.0-SNAPSHOT.jar | Bin 1856788 -> 0 bytes java/shopping-cart/Dockerfile | 11 +-- java/shopping-cart/docker-compose.yml | 4 ++-- java/shopping-cart/pom.xml| 4 ++-- java/shopping-cart/statefun-sdk-java-3.0-SNAPSHOT.jar | Bin 1856788 -> 0 bytes java/showcase/pom.xml | 4 ++-- .../serving/StatefulFunctionsRuntimeProcesses.java| 4 ++-- python/greeter/Dockerfile | 4 python/greeter/docker-compose.yml | 4 ++-- python/greeter/lib-dev/README.md | 4 ...pache_flink_statefun-3.0_SNAPSHOT-py3-none-any.whl | Bin 32861 -> 0 bytes python/greeter/requirements.txt | 2 +- python/showcase/docker-compose.yml| 4 ++-- python/showcase/lib-dev/README.md | 4 ...pache_flink_statefun-3.0_SNAPSHOT-py3-none-any.whl | Bin 32861 -> 0 bytes python/showcase/setup.py | 4 ++-- 20 files changed, 22 insertions(+), 52 deletions(-) diff --git a/deployments/aws-lambda/pom.xml b/deployments/aws-lambda/pom.xml index 82228b7..63d9f6a 100644 --- a/deployments/aws-lambda/pom.xml +++ b/deployments/aws-lambda/pom.xml @@ -22,11 +22,11 @@ under the License. org.apache.flink aws-lambda-example -3.0-SNAPSHOT +3.0.0 jar -3.0-SNAPSHOT +3.0.0 8 8 diff --git a/java/greeter/Dockerfile b/java/greeter/Dockerfile index f87ea11..4049186 100644 --- a/java/greeter/Dockerfile +++ b/java/greeter/Dockerfile @@ -17,15 +17,6 @@ FROM maven:3.6.3-jdk-11 AS builder COPY src /usr/src/app/src COPY pom.xml /usr/src/app -# TODO remove these commented lines and the jar; this is needed now only because we don't have the latest Java SDK published to Maven central yet -COPY statefun-sdk-java-3.0-SNAPSHOT.jar /usr/src/app -RUN mvn install:install-file \ --Dfile=/usr/src/app/statefun-sdk-java-3.0-SNAPSHOT.jar \ --DgroupId=org.apache.flink \ --DartifactId=statefun-sdk-java \ --Dversion=3.0-SNAPSHOT \ --Dpackaging=jar \ --DgeneratePom=true RUN mvn -f /usr/src/app/pom.xml clean package # ... and run the web server! diff --git a/java/greeter/docker-compose.yml b/java/greeter/docker-compose.yml index 22b2dd3..77555c5 100644 --- a/java/greeter/docker-compose.yml +++ b/java/greeter/docker-compose.yml @@ -35,7 +35,7 @@ services: ### statefun-manager: -image: flink-statefun:3.0-SNAPSHOT +image: apache/flink-statefun:3.0.0-java11 expose: - "6123" ports: @@ -47,7 +47,7 @@ services: - ./module.yaml:/opt/statefun/modules/greeter/module.yaml statefun-worker: -image: flink-statefun:3.0-SNAPSHOT +image: apache/flink-statefun:3.0.0-java11 expose: - "6121" - "6122" diff --git a/java/greeter/pom.xml b/java/greeter/pom.xml index 89f7616..6ad0e01 100644 --- a/java/greeter/pom.xml +++ b/java/greeter/pom.xml @@ -22,11 +22,11 @@ under the License. org.apache.flink greeter-functions-app -3.0-SNAPSHOT +3.0.0 jar -3.0-SNAPSHOT +3.0.0 8 8 diff --git a/java/greeter/statefun-sdk-java-3.0-SNAPSHOT.jar b/java/greeter/statefun-sdk-java-3.0-SNAPSHOT.jar deleted file mode 100644 index 47f65ac..000 Binary files a/java/greeter/statefun-sdk-java-3.0-SNAPSHOT.jar and /dev/null differ diff --git a/java/shopping-cart/Dockerfile b/java/shopping-cart/Dockerfile index 61b8684..2e3864f 100644 --- a/java/shopping-cart/Dockerfile +++ b/java/shopping-cart/Dockerfile @@ -15,15 +15,6 @@ # Build the functions code ... FROM maven:3.6.3-jdk-11 AS builder -# TODO remove these commented lines and the jar; this is needed now only because we don't have the latest Java SDK published to Maven central yet -COPY statefun-sdk-java-3.0-SNAPSHOT.jar /usr/src/app/ -RUN mvn install:install-file \ - -Dfile=/usr/src/app/statefun-sdk-java-3.0-SNAPSHOT.jar \ - -DgroupId=org.apache.flink \ -
[flink] branch master updated (7eedc54 -> 7e9de61)
This is an automated email from the ASF dual-hosted git repository. roman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from 7eedc54 [hotfix] Fix the release version for JdbcXaSinkFunction add 7e9de61 [hotfix] Remove unnessary rollbackPreparedFromCheckpoint from JdbcXaSinkFunction.snapshot No new revisions were added by this update. Summary of changes: .../connector/jdbc/xa/JdbcXaSinkFunction.java | 22 -- 1 file changed, 22 deletions(-)
[flink] branch master updated (b17e7b5 -> 7eedc54)
This is an automated email from the ASF dual-hosted git repository. roman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git. from b17e7b5 [FLINK-22148][table] Planner rules should use RexCall#equsls to check whether two rexCalls are equivalent add 7eedc54 [hotfix] Fix the release version for JdbcXaSinkFunction No new revisions were added by this update. Summary of changes: .../java/org/apache/flink/connector/jdbc/xa/JdbcXaSinkFunction.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
