[flink] branch dependabot/npm_and_yarn/flink-runtime-web/web-dashboard/socket.io-parser-and-socket.io-4.2.3 deleted (was 4f66c765cda)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/flink-runtime-web/web-dashboard/socket.io-parser-and-socket.io-4.2.3 in repository https://gitbox.apache.org/repos/asf/flink.git was 4f66c765cda Bump socket.io-parser and socket.io in /flink-runtime-web/web-dashboard The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[flink] branch dependabot/npm_and_yarn/flink-runtime-web/web-dashboard/socket.io-parser-and-socket.io-4.2.4 created (now d6b1bb85ba0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/flink-runtime-web/web-dashboard/socket.io-parser-and-socket.io-4.2.4 in repository https://gitbox.apache.org/repos/asf/flink.git at d6b1bb85ba0 Bump socket.io-parser and socket.io in /flink-runtime-web/web-dashboard No new revisions were added by this update.
[flink] branch master updated (602261648db -> 149a5e34c1e)
This is an automated email from the ASF dual-hosted git repository. tzulitai pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 602261648db [FLINK-32347][checkpoint] Exceptions from the CompletedCheckpointStore are not registered by the CheckpointFailureManager. (#22793) add 149a5e34c1e [FLINNK-30859] Remove all Kafka connector code from main repo No new revisions were added by this update. Summary of changes: .../content.zh/docs/connectors/datastream/kafka.md | 659 - .../docs/connectors/table/upsert-kafka.md | 267 -- docs/content/docs/connectors/datastream/kafka.md | 752 -- docs/content/docs/connectors/table/upsert-kafka.md | 300 --- docs/setup_docs.sh |1 + .../5b9eed8a-5fb6-4373-98ac-3be2a71941b8 |5 - .../7602816f-5c01-4b7a-9e3e-235dfedec245 |2 +- .../e5126cae-f3fe-48aa-b6fb-60ae6cc3fcd5 |3 +- .../f7a4e6fa-e7de-48c9-a61e-c13e83f0c72e | 309 +-- .../flink-architecture-tests-production/pom.xml|5 - flink-architecture-tests/pom.xml |7 - .../8248e31e-b706-4471-bb65-4c927c65fccf |0 .../97dda445-f6bc-43e2-8106-5876ca0cd052 | 78 - .../archunit-violations/stored.rules |4 - flink-connectors/flink-connector-kafka/pom.xml | 338 --- .../apache/flink/connector/kafka/MetricUtil.java | 79 - .../kafka/sink/DefaultKafkaSinkContext.java| 79 - .../kafka/sink/FlinkKafkaInternalProducer.java | 375 --- .../connector/kafka/sink/KafkaCommittable.java | 104 - .../kafka/sink/KafkaCommittableSerializer.java | 57 - .../flink/connector/kafka/sink/KafkaCommitter.java | 146 -- .../kafka/sink/KafkaRecordSerializationSchema.java | 102 - .../KafkaRecordSerializationSchemaBuilder.java | 336 --- .../kafka/sink/KafkaSerializerWrapper.java | 91 - .../flink/connector/kafka/sink/KafkaSink.java | 138 - .../connector/kafka/sink/KafkaSinkBuilder.java | 215 -- .../flink/connector/kafka/sink/KafkaWriter.java| 473 .../connector/kafka/sink/KafkaWriterState.java | 60 - .../kafka/sink/KafkaWriterStateSerializer.java | 54 - .../flink/connector/kafka/sink/Recyclable.java | 49 - .../flink/connector/kafka/sink/TopicSelector.java | 31 - .../connector/kafka/sink/TransactionAborter.java | 131 - .../kafka/sink/TransactionalIdFactory.java | 41 - .../kafka/sink/TransactionsToAbortChecker.java | 95 - .../flink/connector/kafka/source/KafkaSource.java | 241 -- .../connector/kafka/source/KafkaSourceBuilder.java | 534 .../connector/kafka/source/KafkaSourceOptions.java | 65 - .../source/enumerator/KafkaSourceEnumState.java| 39 - .../enumerator/KafkaSourceEnumStateSerializer.java | 124 - .../source/enumerator/KafkaSourceEnumerator.java | 634 - .../initializer/NoStoppingOffsetsInitializer.java | 52 - .../enumerator/initializer/OffsetsInitializer.java | 186 -- .../initializer/OffsetsInitializerValidator.java | 39 - .../ReaderHandledOffsetsInitializer.java | 83 - .../initializer/SpecifiedOffsetsInitializer.java | 108 - .../initializer/TimestampOffsetsInitializer.java | 76 - .../enumerator/subscriber/KafkaSubscriber.java | 67 - .../subscriber/KafkaSubscriberUtils.java | 50 - .../subscriber/PartitionSetSubscriber.java | 76 - .../enumerator/subscriber/TopicListSubscriber.java | 63 - .../subscriber/TopicPatternSubscriber.java | 65 - .../source/metrics/KafkaSourceReaderMetrics.java | 330 --- .../source/reader/KafkaPartitionSplitReader.java | 532 .../kafka/source/reader/KafkaRecordEmitter.java| 81 - .../kafka/source/reader/KafkaSourceReader.java | 199 -- .../KafkaDeserializationSchemaWrapper.java | 65 - .../KafkaRecordDeserializationSchema.java | 129 - ...KafkaValueOnlyDeserializationSchemaWrapper.java | 58 - .../KafkaValueOnlyDeserializerWrapper.java | 106 - .../reader/fetcher/KafkaSourceFetcherManager.java | 112 - .../kafka/source/split/KafkaPartitionSplit.java| 152 -- .../split/KafkaPartitionSplitSerializer.java | 72 - .../source/split/KafkaPartitionSplitState.java | 56 - .../connectors/kafka/FlinkKafkaConsumer.java | 342 --- .../connectors/kafka/FlinkKafkaConsumerBase.java | 1227 - .../connectors/kafka/FlinkKafkaErrorCode.java | 27 - .../connectors/kafka/FlinkKafkaException.java | 44 - .../connectors/kafka/FlinkKafkaProducer.java | 1950 -- .../connectors/kafka/FlinkKafkaProducer011.java| 72 - .../connectors/kafka/FlinkKafkaProducerBase.java | 442 .../connectors/kafka/KafkaContextAware.java| 55 - .../kafka/KafkaDeserializationSchema.java | 84 - .../connectors/kafka/KafkaSerializationSchema.java | 61 -
[flink] branch master updated: [FLINK-32347][checkpoint] Exceptions from the CompletedCheckpointStore are not registered by the CheckpointFailureManager. (#22793)
This is an automated email from the ASF dual-hosted git repository.
srichter pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 602261648db [FLINK-32347][checkpoint] Exceptions from the
CompletedCheckpointStore are not registered by the CheckpointFailureManager.
(#22793)
602261648db is described below
commit 602261648dbc387c51ae113139486b2d1f0935e2
Author: Stefan Richter
AuthorDate: Mon Jun 26 19:22:10 2023 +0200
[FLINK-32347][checkpoint] Exceptions from the CompletedCheckpointStore are
not registered by the CheckpointFailureManager. (#22793)
Currently if an error occurs while saving a completed checkpoint in the
CompletedCheckpointStore, CheckpointCoordinator doesn't call
CheckpointFailureManager to handle the error. Such behavior leads to the fact,
that errors from CompletedCheckpointStore don't increase the failed checkpoints
count and 'execution.checkpointing.tolerable-failed-checkpoints' option does
not limit the number of errors of this kind in any way.
---
.../runtime/checkpoint/CheckpointCoordinator.java | 33
.../checkpoint/CheckpointCoordinatorTest.java | 91 ++
2 files changed, 110 insertions(+), 14 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
index 585a35ffb1c..41902b4af39 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
@@ -1330,6 +1330,7 @@ public class CheckpointCoordinator {
}
private void reportCompletedCheckpoint(CompletedCheckpoint
completedCheckpoint) {
+
failureManager.handleCheckpointSuccess(completedCheckpoint.getCheckpointID());
CompletedCheckpointStats completedCheckpointStats =
completedCheckpoint.getStatistic();
if (completedCheckpointStats != null) {
LOG.trace(
@@ -1403,7 +1404,6 @@ public class CheckpointCoordinator {
pendingCheckpoint.finalizeCheckpoint(
checkpointsCleaner, this::scheduleTriggerRequest,
executor);
-
failureManager.handleCheckpointSuccess(pendingCheckpoint.getCheckpointID());
return completedCheckpoint;
} catch (Exception e1) {
// abort the current pending checkpoint if we fails to finalize
the pending
@@ -1467,23 +1467,28 @@ public class CheckpointCoordinator {
checkpointsCleaner.cleanCheckpointOnFailedStoring(completedCheckpoint,
executor);
}
-reportFailedCheckpoint(checkpointId, exception);
+final CheckpointException checkpointException =
+new CheckpointException(
+"Could not complete the pending checkpoint " +
checkpointId + '.',
+
CheckpointFailureReason.FINALIZE_CHECKPOINT_FAILURE,
+exception);
+reportFailedCheckpoint(pendingCheckpoint, checkpointException);
sendAbortedMessages(tasksToAbort, checkpointId,
completedCheckpoint.getTimestamp());
-throw new CheckpointException(
-"Could not complete the pending checkpoint " +
checkpointId + '.',
-CheckpointFailureReason.FINALIZE_CHECKPOINT_FAILURE,
-exception);
+throw checkpointException;
}
}
-private void reportFailedCheckpoint(long checkpointId, Exception
exception) {
-PendingCheckpointStats pendingCheckpointStats =
-statsTracker.getPendingCheckpointStats(checkpointId);
-if (pendingCheckpointStats != null) {
-statsTracker.reportFailedCheckpoint(
-pendingCheckpointStats.toFailedCheckpoint(
-System.currentTimeMillis(), exception));
-}
+private void reportFailedCheckpoint(
+PendingCheckpoint pendingCheckpoint, CheckpointException
exception) {
+
+failureManager.handleCheckpointException(
+pendingCheckpoint,
+pendingCheckpoint.getProps(),
+exception,
+null,
+job,
+getStatsCallback(pendingCheckpoint),
+statsTracker);
}
void scheduleTriggerRequest() {
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
index 660671dfa8f..93d565644cb 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinatorTest.java
+++
b/flink-runtime/src/test/java/org/apa
[flink-kubernetes-operator] branch main updated: [FLINK-32412] Reduce JobID collision chance
This is an automated email from the ASF dual-hosted git repository.
gyfora pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new abf9d040 [FLINK-32412] Reduce JobID collision chance
abf9d040 is described below
commit abf9d040ae58caf8313ca7b71049d6709fa26ea3
Author: fabio.wanner
AuthorDate: Wed Jun 21 16:11:19 2023 +0200
[FLINK-32412] Reduce JobID collision chance
Instead of using Java's hashCode, which is an integer value (32bit), a long
representation of the uid is used. This decreases the chance for an ID
collision drastically:
For a 50% collision chance with random integers, 77000 numbers need to be
generated. For a long value (64 bit) a 50% change of a collision needs
5.1×10^9
random longs. For details look up: "the birthday problem".
A test is added to increase awareness of the problem when changing this
part of
the code.
---
.../operator/observer/sessionjob/FlinkSessionJobObserver.java | 6 +-
.../org/apache/flink/kubernetes/operator/utils/FlinkUtils.java | 4 +++-
.../org/apache/flink/kubernetes/operator/utils/FlinkUtilsTest.java | 7 +++
3 files changed, 15 insertions(+), 2 deletions(-)
diff --git
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/sessionjob/FlinkSessionJobObserver.java
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/sessionjob/FlinkSessionJobObserver.java
index c143bff5..42762b13 100644
---
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/sessionjob/FlinkSessionJobObserver.java
+++
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/sessionjob/FlinkSessionJobObserver.java
@@ -43,6 +43,8 @@ import java.util.List;
import java.util.Optional;
import java.util.stream.Collectors;
+import static
org.apache.flink.kubernetes.operator.utils.FlinkUtils.generateSessionJobFixedJobID;
+
/** The observer of {@link FlinkSessionJob}. */
public class FlinkSessionJobObserver extends
AbstractFlinkResourceObserver {
@@ -83,7 +85,9 @@ public class FlinkSessionJobObserver extends
AbstractFlinkResourceObserver();
for (JobStatusMessage jobStatusMessage : jobStatusMessages) {
var jobId = jobStatusMessage.getJobId();
-if (jobId.getLowerPart() == uid.hashCode()
+if (jobId.getLowerPart()
+== generateSessionJobFixedJobID(uid,
jobId.getUpperPart() + 1L)
+.getLowerPart()
&&
!jobStatusMessage.getJobState().isGloballyTerminalState()) {
matchedJobs.add(jobId);
}
diff --git
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java
index 68b23e34..3700b1f7 100644
---
a/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java
+++
b/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/FlinkUtils.java
@@ -55,6 +55,7 @@ import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Optional;
+import java.util.UUID;
import static
org.apache.flink.kubernetes.utils.Constants.LABEL_CONFIGMAP_TYPE_HIGH_AVAILABILITY;
@@ -363,7 +364,8 @@ public class FlinkUtils {
*/
public static JobID generateSessionJobFixedJobID(String uid, Long
generation) {
return new JobID(
-Preconditions.checkNotNull(uid).hashCode(),
Preconditions.checkNotNull(generation));
+
UUID.fromString(Preconditions.checkNotNull(uid)).getMostSignificantBits(),
+Preconditions.checkNotNull(generation));
}
/**
diff --git
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/utils/FlinkUtilsTest.java
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/utils/FlinkUtilsTest.java
index 770a7042..8fe98f41 100644
---
a/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/utils/FlinkUtilsTest.java
+++
b/flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/utils/FlinkUtilsTest.java
@@ -313,6 +313,13 @@ public class FlinkUtilsTest {
assertEquals(List.of(v1merged, volume2, volume3),
mergedPod.getSpec().getVolumes());
}
+@Test
+public void testJobIDGeneration() {
+JobID jobID =
+
FlinkUtils.generateSessionJobFixedJobID("----",
2L);
+assertEquals("0002", jobID.toString());
+}
+
private void createHAConfigMapWithData(
String configMapName, String namespace, String clus
[flink-ml] annotated tag release-2.3.0-rc1 updated (09196f62 -> 9adf4fc5)
This is an automated email from the ASF dual-hosted git repository. lindong pushed a change to annotated tag release-2.3.0-rc1 in repository https://gitbox.apache.org/repos/asf/flink-ml.git *** WARNING: tag release-2.3.0-rc1 was modified! *** from 09196f62 (tag) to 9adf4fc5 (tag) tagging 44b406877a98eaa8f991461bd84696cbee717a31 (commit) by Dong Lin on Mon Jun 26 19:59:10 2023 +0800 - Log - release-2.3.0-rc1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEr6zbCebw/yjJPWS8vu1PbLn3fQ4FAmSZfY4ACgkQvu1PbLn3 fQ6V0A/+OPSm6jupjzwfVXFkteuY/BfXXGmbpgDk2QXpyyNFXlmbAsanXbS/jaed ygErVnZd+lJjUlK5gQ31tDOvbL/PKH0f2OyLkKfkKutQ8qdJ5392mgKLmy3N8kJe TuDNloYvzIu3OpE5VfJgvcDGXc12jpX0BphnZoY1UM5oN6urwFngBOEMJCjsowyL rNazVhOn3tRLlMk43Oscw3Cvpen9+jB9Ifhe63gHvgNkachxDEJQ7OVhzRA5ZrQX 65xfWID6oMUitI+sO8FyUX1JX3Cp9tryzFvr1XrGHsoVYJ9ZHTZF7Efdy29iwDD/ f/7YAUvWqx2bwyaJHdE4E9ooj2lge3rronKuRNSHg8vsWR10IyF81r557vxhKgxn 7JH7349GpMZ/ZvyCqWs/gLy/sSk3QUicxl9URqM0/09gwvV+AMGFni+y4ddXQ9e3 KLtLM7AnuudD/u927CZhc7pZZBr7b51I9iVf9gvXFVRS39BuZenOwGuntsv+nlhi qp4u6L0SOL4juTbDXYDoNBs1XKVDSClOHOTwlsK+kiznF362uDQ47OuizW18Y+LM F+XvNzRTqz7YXtStXHpsC+ECGzu18QMUud9U54vVA4fWPuvIgrpErP62FI/C+M6P L2YaeDgtuXuLkoBOVLwH6e7Ez4tkAjCa0jHu/xX4GLpYgCjPKJ4= =/yr6 -END PGP SIGNATURE- --- from e62cdc56 [hotfix] Update config.toml for Flink ML 2.3.0 release add 2464b0a4 [release] Update version to 2.3.0 add 84001e38 [hotfix] Fix baseURL in docs/config.toml add 44b40687 [hotfix] Update Flink version used in build-your-own-project.md No new revisions were added by this update. Summary of changes: docs/config.toml | 2 +- docs/content/docs/try-flink-ml/java/build-your-own-project.md | 4 ++-- flink-ml-benchmark/pom.xml| 2 +- flink-ml-core/pom.xml | 2 +- flink-ml-dist/pom.xml | 2 +- flink-ml-examples/pom.xml | 2 +- flink-ml-iteration/flink-ml-iteration-1.15/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-1.16/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-1.17/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-common/pom.xml | 2 +- flink-ml-iteration/pom.xml| 2 +- flink-ml-lib/pom.xml | 2 +- flink-ml-python/pom.xml | 2 +- flink-ml-python/pyflink/ml/version.py | 2 +- flink-ml-servable-core/pom.xml| 2 +- flink-ml-servable-lib/pom.xml | 2 +- flink-ml-tests/pom.xml| 2 +- flink-ml-uber/pom.xml | 2 +- pom.xml | 2 +- 19 files changed, 20 insertions(+), 20 deletions(-)
svn commit: r62651 - in /dev/flink/flink-ml-2.3.0-rc1: ./ apache-flink-ml-2.3.0.tar.gz apache-flink-ml-2.3.0.tar.gz.asc apache-flink-ml-2.3.0.tar.gz.sha512 flink-ml-2.3.0-src.tgz flink-ml-2.3.0-src.tg
Author: lindong Date: Mon Jun 26 11:24:16 2023 New Revision: 62651 Log: Apache Flink ML, version 2.3.0, release candidate 1 Added: dev/flink/flink-ml-2.3.0-rc1/ dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz (with props) dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.asc dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.sha512 dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz (with props) dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.asc dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.sha512 Added: dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz == Binary file - no diff available. Propchange: dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz -- svn:mime-type = application/octet-stream Added: dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.asc == --- dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.asc (added) +++ dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.asc Mon Jun 26 11:24:16 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEr6zbCebw/yjJPWS8vu1PbLn3fQ4FAmSZIh4ACgkQvu1PbLn3 +fQ5A/hAAozZIDFiE6NUE8FBHwtqDsmJtwILpn/cWYUjIPowmM91mdxSwYcCryGC5 +ijnryNFBwMi3KtnDQlwny5U2OYsURCnIhClwDJsVQsr6KN16bBOvwiAMH1Q3Tye9 +iz+AfE3TtA0SI9d2sT2lNNeXLgeN0xl+H+mhEvuT/Ij5MX9vnBltHYi0iMYluZTL +PIgR2rtHrKSKUwiNk1hG/9I3cgdUORoXX8kHMQGeBIx7FGZKIBdLRg2fUMI4lyds +vrl/klDfblInChd8Bd3IasmdIs07To1FkvimBuopUaaBYLbhlEy2A93Y46rl/t6c +YG64YVFVBzhbWkkF3nlzDin+04FNA8WHRU3TivRmXvvhsveExjISzyiEhPRA88v6 +1obMapULlR0G7S/vQjbu8/Nu6/xqHZLo3gqnzt6jHjAkNV4EENdvR/n+8AkWppc1 +Vr0EZt4hoStVwVtB5ongWRXMNE4vaBM6p0GLFcjqVsBgAVNVpAQMMiBp8XxJSqM0 +RlEfQI/M6f5+L3ENvjBJ0lJIazXhPXXoAEA6FbOFlcZ7MfcoifDXAfTiYzrRsj5r +3y/AqSNniLCRQbWlT+nt4jZHP9JEkwdSy/Vn9vxL9ulky144c243pfnNaJNC4Kwk +kvnjMmxEZsVebWx1tFWFInMNvh97b0e9NDUiPDfha1jlKbyVTmM= +=Zo46 +-END PGP SIGNATURE- Added: dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.sha512 == --- dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.sha512 (added) +++ dev/flink/flink-ml-2.3.0-rc1/apache-flink-ml-2.3.0.tar.gz.sha512 Mon Jun 26 11:24:16 2023 @@ -0,0 +1 @@ +214d557c12b716dc6adea58243cd2d67d39b3d4c156b1c46709be86625b0f5ab1850aeb5746864f910b0d0f3a725464191586005f6e77c32ff5c4911d8f6d3bf apache-flink-ml-2.3.0.tar.gz Added: dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz == Binary file - no diff available. Propchange: dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz -- svn:mime-type = application/octet-stream Added: dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.asc == --- dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.asc (added) +++ dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.asc Mon Jun 26 11:24:16 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEr6zbCebw/yjJPWS8vu1PbLn3fQ4FAmSZD34ACgkQvu1PbLn3 +fQ4NDBAAvdZ0ZjYZk46jAXP3cEMK27ZsX3m0qfPtqFgRUuaRsJpPtUz3e6n+Iw8P +qSL0+1FTfa2TJB2C3u9w6XvZIy6QLn24lRA3NEI3lcuYHiQ0w6vCja6RZHOnoYvS +jUzpMMNMPbsLcALRgRpmrs78YCJdt8Q7C73HIZlxOIzsUnt29apKSlJOzKigkBTD +NJhqlevFhoUdCq4+Hm/cF+MmE053wlMV0J8I2BjdpA1k/1h5HDqbPJGCx8cXbPu0 +iG+GZe0GGFYnJNyeNc1nZNm200mZka/t2CeumJ2QAfMYPhwbpziUtEZ+yrA8e80K +Gtd2P307ee9rwXMP3itJ6ikn4Vn5jzOLsGM8K0I/bGtDyLvuEUAGSuHxt68KETm4 +SzduraAZtc5vDA2Qvni7uJqmSoQPXoZCsLr1XehHuRKmWUXsapnHCcZUgyk01YcG +tcDqu863i5N5o1K0yWYo9dhQ5ru22jQQR0uzfB+11RhwoBnkjvlAO+xlLmaojoXO +bn7QOt/HumyVSHtE3Jf6W+v32ZaEm9OEoR+UyAX2O2J6jhiJQY9JcGgAWUy51oWk +oVdBfyNjhUDMvHfusjBwyYaF9nJrFD1GAY0km3A9EvQCGigZNedFLApRXBI8757g +ZPJD0suE6ZzwZiIipXMjqTJ0cvxtIJOomEm2G7nPqXWeiLdBTPA= +=Wdr2 +-END PGP SIGNATURE- Added: dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.sha512 == --- dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.sha512 (added) +++ dev/flink/flink-ml-2.3.0-rc1/flink-ml-2.3.0-src.tgz.sha512 Mon Jun 26 11:24:16 2023 @@ -0,0 +1 @@ +7176f417572194578a4ebd848b4d52d589b778a954b2837d53a980bcbddfa2f41f90e882016da60a83a982d9732bdc1c93bb34fa3f3d1b42c637d47e3d7f9382 flink-ml-2.3.0-src.tgz
[flink] branch master updated: [FLINK-32265][runtime] Use system classloader in JM for jobs without extra jars
This is an automated email from the ASF dual-hosted git repository.
chesnay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new b3d0e51bdf9 [FLINK-32265][runtime] Use system classloader in JM for
jobs without extra jars
b3d0e51bdf9 is described below
commit b3d0e51bdf9374848b03843115c97fa092e2e6b2
Author: Shammon FY
AuthorDate: Mon Jun 26 19:10:04 2023 +0800
[FLINK-32265][runtime] Use system classloader in JM for jobs without extra
jars
---
.../librarycache/BlobLibraryCacheManager.java | 43 +--
.../jobmaster/JobManagerSharedServices.java| 3 +-
.../runtime/taskexecutor/TaskManagerServices.java | 3 +-
.../librarycache/BlobLibraryCacheManagerTest.java | 50 --
.../BlobLibraryCacheRecoveryITCase.java| 15 ++-
5 files changed, 87 insertions(+), 27 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java
index e2723dd2e1e..9af717008b6 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/execution/librarycache/BlobLibraryCacheManager.java
@@ -37,6 +37,7 @@ import javax.annotation.Nullable;
import javax.annotation.concurrent.GuardedBy;
import javax.annotation.concurrent.ThreadSafe;
+import java.io.Closeable;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
@@ -75,12 +76,18 @@ public class BlobLibraryCacheManager implements
LibraryCacheManager {
private final ClassLoaderFactory classLoaderFactory;
+/** If true, it will use system class loader when the jars and classpaths
of job are empty. */
+private final boolean wrapsSystemClassLoader;
+
//
public BlobLibraryCacheManager(
-PermanentBlobService blobService, ClassLoaderFactory
classLoaderFactory) {
+PermanentBlobService blobService,
+ClassLoaderFactory classLoaderFactory,
+boolean wrapsSystemClassLoader) {
this.blobService = checkNotNull(blobService);
this.classLoaderFactory = checkNotNull(classLoaderFactory);
+this.wrapsSystemClassLoader = wrapsSystemClassLoader;
}
@Override
@@ -226,11 +233,17 @@ public class BlobLibraryCacheManager implements
LibraryCacheManager {
verifyIsNotReleased();
if (resolvedClassLoader == null) {
+boolean systemClassLoader =
+wrapsSystemClassLoader && libraries.isEmpty() &&
classPaths.isEmpty();
resolvedClassLoader =
new ResolvedClassLoader(
-createUserCodeClassLoader(jobId,
libraries, classPaths),
+systemClassLoader
+?
ClassLoader.getSystemClassLoader()
+: createUserCodeClassLoader(
+jobId, libraries,
classPaths),
libraries,
-classPaths);
+classPaths,
+systemClassLoader);
} else {
resolvedClassLoader.verifyClassLoader(libraries,
classPaths);
}
@@ -357,7 +370,7 @@ public class BlobLibraryCacheManager implements
LibraryCacheManager {
}
private static final class ResolvedClassLoader implements
UserCodeClassLoader {
-private final URLClassLoader classLoader;
+private final ClassLoader classLoader;
/**
* Set of BLOB keys used for a previous job/task registration.
@@ -375,12 +388,15 @@ public class BlobLibraryCacheManager implements
LibraryCacheManager {
*/
private final Set classPaths;
+private final boolean wrapsSystemClassLoader;
+
private final Map releaseHooks;
private ResolvedClassLoader(
-URLClassLoader classLoader,
+ClassLoader classLoader,
Collection requiredLibraries,
-Collection requiredClassPaths) {
+Collection requiredClassPaths,
+boolean wrapsSystemClassLoader) {
this.classLoader = classLoader;
// NOTE: do not store the class paths, i.e. URLs, into a set for
performance reasons
@@ -391,6 +407,7 @@ public class BlobLibraryCacheManager implements
LibraryCacheManager {
classPaths.add(url.toString())
[flink-ml] branch release-2.3 updated: [hotfix] Update Flink version used in build-your-own-project.md
This is an automated email from the ASF dual-hosted git repository. lindong pushed a commit to branch release-2.3 in repository https://gitbox.apache.org/repos/asf/flink-ml.git The following commit(s) were added to refs/heads/release-2.3 by this push: new 44b40687 [hotfix] Update Flink version used in build-your-own-project.md 44b40687 is described below commit 44b406877a98eaa8f991461bd84696cbee717a31 Author: Dong Lin AuthorDate: Mon Jun 26 18:06:58 2023 +0800 [hotfix] Update Flink version used in build-your-own-project.md --- docs/content/docs/try-flink-ml/java/build-your-own-project.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/try-flink-ml/java/build-your-own-project.md b/docs/content/docs/try-flink-ml/java/build-your-own-project.md index a94e992b..ceefdf1b 100644 --- a/docs/content/docs/try-flink-ml/java/build-your-own-project.md +++ b/docs/content/docs/try-flink-ml/java/build-your-own-project.md @@ -80,7 +80,7 @@ the basic skeleton of a project, with some necessary Flink dependencies. $ mvn archetype:generate \ -DarchetypeGroupId=org.apache.flink \ -DarchetypeArtifactId=flink-quickstart-java \ --DarchetypeVersion=1.16.1 \ +-DarchetypeVersion=1.17.1 \ -DgroupId=kmeans-example \ -DartifactId=kmeans-example \ -Dversion=0.1 \
[flink-ml] 01/02: [release] Update version to 2.3.0
This is an automated email from the ASF dual-hosted git repository.
lindong pushed a commit to branch release-2.3
in repository https://gitbox.apache.org/repos/asf/flink-ml.git
commit 2464b0a4695d370c5176a36d0c6348bb1214b597
Author: Dong Lin
AuthorDate: Mon Jun 26 12:07:53 2023 +0800
[release] Update version to 2.3.0
---
docs/content/docs/try-flink-ml/java/build-your-own-project.md | 2 +-
flink-ml-benchmark/pom.xml| 2 +-
flink-ml-core/pom.xml | 2 +-
flink-ml-dist/pom.xml | 2 +-
flink-ml-examples/pom.xml | 2 +-
flink-ml-iteration/flink-ml-iteration-1.15/pom.xml| 2 +-
flink-ml-iteration/flink-ml-iteration-1.16/pom.xml| 2 +-
flink-ml-iteration/flink-ml-iteration-1.17/pom.xml| 2 +-
flink-ml-iteration/flink-ml-iteration-common/pom.xml | 2 +-
flink-ml-iteration/pom.xml| 2 +-
flink-ml-lib/pom.xml | 2 +-
flink-ml-python/pom.xml | 2 +-
flink-ml-python/pyflink/ml/version.py | 2 +-
flink-ml-servable-core/pom.xml| 2 +-
flink-ml-servable-lib/pom.xml | 2 +-
flink-ml-tests/pom.xml| 2 +-
flink-ml-uber/pom.xml | 2 +-
pom.xml | 2 +-
18 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/docs/content/docs/try-flink-ml/java/build-your-own-project.md
b/docs/content/docs/try-flink-ml/java/build-your-own-project.md
index 98efeef5..a94e992b 100644
--- a/docs/content/docs/try-flink-ml/java/build-your-own-project.md
+++ b/docs/content/docs/try-flink-ml/java/build-your-own-project.md
@@ -110,7 +110,7 @@ Change the dependencies provided in `pom.xml` as follows:
org.apache.flink
flink-ml-uber-1.17
- 2.3-SNAPSHOT
+ 2.3.0
provided
diff --git a/flink-ml-benchmark/pom.xml b/flink-ml-benchmark/pom.xml
index 348e3301..3988e1b6 100644
--- a/flink-ml-benchmark/pom.xml
+++ b/flink-ml-benchmark/pom.xml
@@ -23,7 +23,7 @@ under the License.
flink-ml-parent
org.apache.flink
-2.3-SNAPSHOT
+2.3.0
flink-ml-benchmark-${flink.main.version}
diff --git a/flink-ml-core/pom.xml b/flink-ml-core/pom.xml
index 35895885..b761c2dc 100644
--- a/flink-ml-core/pom.xml
+++ b/flink-ml-core/pom.xml
@@ -25,7 +25,7 @@ under the License.
org.apache.flink
flink-ml-parent
-2.3-SNAPSHOT
+2.3.0
flink-ml-core-${flink.main.version}
diff --git a/flink-ml-dist/pom.xml b/flink-ml-dist/pom.xml
index 33bafcee..75de889a 100644
--- a/flink-ml-dist/pom.xml
+++ b/flink-ml-dist/pom.xml
@@ -23,7 +23,7 @@ under the License.
org.apache.flink
flink-ml-parent
-2.3-SNAPSHOT
+2.3.0
flink-ml-dist-${flink.main.version}
diff --git a/flink-ml-examples/pom.xml b/flink-ml-examples/pom.xml
index 3d2bac36..8d291746 100644
--- a/flink-ml-examples/pom.xml
+++ b/flink-ml-examples/pom.xml
@@ -23,7 +23,7 @@ under the License.
flink-ml-parent
org.apache.flink
-2.3-SNAPSHOT
+2.3.0
4.0.0
diff --git a/flink-ml-iteration/flink-ml-iteration-1.15/pom.xml
b/flink-ml-iteration/flink-ml-iteration-1.15/pom.xml
index 37e67b5a..90392306 100644
--- a/flink-ml-iteration/flink-ml-iteration-1.15/pom.xml
+++ b/flink-ml-iteration/flink-ml-iteration-1.15/pom.xml
@@ -21,7 +21,7 @@ under the License.
org.apache.flink
flink-ml-iteration
-2.3-SNAPSHOT
+2.3.0
flink-ml-iteration-1.15
diff --git a/flink-ml-iteration/flink-ml-iteration-1.16/pom.xml
b/flink-ml-iteration/flink-ml-iteration-1.16/pom.xml
index 0dcdc879..7673ea1d 100644
--- a/flink-ml-iteration/flink-ml-iteration-1.16/pom.xml
+++ b/flink-ml-iteration/flink-ml-iteration-1.16/pom.xml
@@ -21,7 +21,7 @@ under the License.
org.apache.flink
flink-ml-iteration
-2.3-SNAPSHOT
+2.3.0
flink-ml-iteration-1.16
diff --git a/flink-ml-iteration/flink-ml-iteration-1.17/pom.xml
b/flink-ml-iteration/flink-ml-iteration-1.17/pom.xml
index 17fd6e83..3367538b 100644
--- a/flink-ml-iteration/flink-ml-iteration-1.17/pom.xml
+++ b/flink-ml-iteration/flink-ml-iteration-1.17/pom.xml
@@ -21,7 +21,7 @@ under the License.
org.apache.flink
flink-ml-iteration
-2.3-SNAPSHOT
+2.3.0
flink-ml-iteration-1.17
diff --git a/flink-ml-iteration/flink-ml-iteration-common/pom.xml
b/flink-ml-iteration/flink-ml-iteration-common/pom.xml
index 1ff29893..f8d608cc 100644
--- a/flink-ml-iteration/flink-ml-iteration-common/pom.xml
+++ b/flink-ml-iteration/flink-ml-iteration-common/pom.xml
@@ -21,7
[flink-ml] branch release-2.3 updated (e62cdc56 -> 84001e38)
This is an automated email from the ASF dual-hosted git repository. lindong pushed a change to branch release-2.3 in repository https://gitbox.apache.org/repos/asf/flink-ml.git from e62cdc56 [hotfix] Update config.toml for Flink ML 2.3.0 release new 2464b0a4 [release] Update version to 2.3.0 new 84001e38 [hotfix] Fix baseURL in docs/config.toml The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/config.toml | 2 +- docs/content/docs/try-flink-ml/java/build-your-own-project.md | 2 +- flink-ml-benchmark/pom.xml| 2 +- flink-ml-core/pom.xml | 2 +- flink-ml-dist/pom.xml | 2 +- flink-ml-examples/pom.xml | 2 +- flink-ml-iteration/flink-ml-iteration-1.15/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-1.16/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-1.17/pom.xml| 2 +- flink-ml-iteration/flink-ml-iteration-common/pom.xml | 2 +- flink-ml-iteration/pom.xml| 2 +- flink-ml-lib/pom.xml | 2 +- flink-ml-python/pom.xml | 2 +- flink-ml-python/pyflink/ml/version.py | 2 +- flink-ml-servable-core/pom.xml| 2 +- flink-ml-servable-lib/pom.xml | 2 +- flink-ml-tests/pom.xml| 2 +- flink-ml-uber/pom.xml | 2 +- pom.xml | 2 +- 19 files changed, 19 insertions(+), 19 deletions(-)
[flink-ml] 02/02: [hotfix] Fix baseURL in docs/config.toml
This is an automated email from the ASF dual-hosted git repository. lindong pushed a commit to branch release-2.3 in repository https://gitbox.apache.org/repos/asf/flink-ml.git commit 84001e382c591888deac3e574b802ca29abae275 Author: Dong Lin AuthorDate: Mon Jun 26 17:51:19 2023 +0800 [hotfix] Fix baseURL in docs/config.toml --- docs/config.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/config.toml b/docs/config.toml index 4810200c..4c849c1f 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -baseURL = '//nightlies.apache.org/flink/flink-ml-docs-release-$SHORT_RELEASE_VERSION' +baseURL = '//nightlies.apache.org/flink/flink-ml-docs-release-2.3' languageCode = 'en-us' title = 'Apache Flink Machine Learning Library' enableGitInfo = false
[flink] branch master updated: [FLINK-32413][runtime] Adds fallback error handler to DefaultLeaderElectionService
This is an automated email from the ASF dual-hosted git repository.
mapohl pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/master by this push:
new 14480c8ce4f [FLINK-32413][runtime] Adds fallback error handler to
DefaultLeaderElectionService
14480c8ce4f is described below
commit 14480c8ce4f6cd8f0f918f33d5662b7c62be8933
Author: Matthias Pohl
AuthorDate: Thu Jun 22 14:53:14 2023 +0200
[FLINK-32413][runtime] Adds fallback error handler to
DefaultLeaderElectionService
This enables test scenarios to catch unexpected errors that might have been
otherwise missed.
Signed-off-by: Matthias Pohl
---
.../DefaultLeaderElectionService.java | 19 +++-
.../JobMasterServiceLeadershipRunnerTest.java | 3 +-
.../DefaultLeaderElectionServiceTest.java | 50 +-
...KeeperLeaderElectionConnectionHandlingTest.java | 4 +-
4 files changed, 63 insertions(+), 13 deletions(-)
diff --git
a/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionService.java
b/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionService.java
index 940ac664df3..88bf9caa8e3 100644
---
a/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionService.java
+++
b/flink-runtime/src/main/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionService.java
@@ -115,9 +115,23 @@ public class DefaultLeaderElectionService extends
AbstractLeaderElectionService
*/
private final ExecutorService leadershipOperationExecutor;
+private final FatalErrorHandler fallbackErrorHandler;
+
public DefaultLeaderElectionService(LeaderElectionDriverFactory
leaderElectionDriverFactory) {
this(
leaderElectionDriverFactory,
+t ->
+LOG.debug(
+"Ignoring error notification since there's no
contender registered."));
+}
+
+@VisibleForTesting
+public DefaultLeaderElectionService(
+LeaderElectionDriverFactory leaderElectionDriverFactory,
+FatalErrorHandler fallbackErrorHandler) {
+this(
+leaderElectionDriverFactory,
+fallbackErrorHandler,
Executors.newSingleThreadExecutor(
new ExecutorThreadFactory(
"DefaultLeaderElectionService-leadershipOperationExecutor")));
@@ -126,9 +140,12 @@ public class DefaultLeaderElectionService extends
AbstractLeaderElectionService
@VisibleForTesting
DefaultLeaderElectionService(
LeaderElectionDriverFactory leaderElectionDriverFactory,
+FatalErrorHandler fallbackErrorHandler,
ExecutorService leadershipOperationExecutor) {
this.leaderElectionDriverFactory =
checkNotNull(leaderElectionDriverFactory);
+this.fallbackErrorHandler = checkNotNull(fallbackErrorHandler);
+
this.leaderContender = null;
this.issuedLeaderSessionID = null;
@@ -486,7 +503,7 @@ public class DefaultLeaderElectionService extends
AbstractLeaderElectionService
private void forwardErrorToLeaderContender(Throwable t) {
synchronized (lock) {
if (leaderContender == null) {
-LOG.debug("Ignoring error notification since there's no
contender registered.");
+fallbackErrorHandler.onFatalError(t);
return;
}
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java
index a4df85e5f3c..4d06cc9af17 100644
---
a/flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java
+++
b/flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java
@@ -683,7 +683,8 @@ class JobMasterServiceLeadershipRunnerTest {
// we need to use DefaultLeaderElectionService here because
JobMasterServiceLeadershipRunner
// in connection with the DefaultLeaderElectionService generates the
nested locking
final DefaultLeaderElectionService defaultLeaderElectionService =
-new
DefaultLeaderElectionService(testingLeaderElectionDriverFactory);
+new DefaultLeaderElectionService(
+testingLeaderElectionDriverFactory, fatalErrorHandler);
defaultLeaderElectionService.startLeaderElectionBackend();
// latch to detect when we reached the first synchronized section
having a lock on the
diff --git
a/flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/DefaultLeaderElectionServiceTest.java
b/flink-runtime/src/test/java/org/apache/flin
[flink] branch master updated (43f5d038fb2 -> 9b63099964b)
This is an automated email from the ASF dual-hosted git repository. renqs pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 43f5d038fb2 [FLINK-31413][hive] Change scope of flink-table-planner dependency from provided to test in Hive connector This closes #22620 add 9b63099964b [FLINK-32392][ci] Invalidate Maven repo cache on Azure every year to reduce its disk usage No new revisions were added by this update. Summary of changes: azure-pipelines.yml | 1 + tools/azure-pipelines/build-apache-repo.yml | 1 + tools/azure-pipelines/build-nightly-dist.yml | 8 tools/azure-pipelines/e2e-template.yml | 4 ++-- tools/azure-pipelines/jobs-template.yml | 11 +++ 5 files changed, 15 insertions(+), 10 deletions(-)
