[flink-connector-mongodb] branch main updated: [FLINK-32757][connectors/mongodb][build] Update the copyright with year 2023 in NOTICE files
This is an automated email from the ASF dual-hosted git repository. hong pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git The following commit(s) were added to refs/heads/main by this push: new fc656c4 [FLINK-32757][connectors/mongodb][build] Update the copyright with year 2023 in NOTICE files fc656c4 is described below commit fc656c420e9b20676bf5e67c0c1c059a5ad44216 Author: Jiabao Sun AuthorDate: Sat Aug 5 00:51:59 2023 +0800 [FLINK-32757][connectors/mongodb][build] Update the copyright with year 2023 in NOTICE files --- NOTICE | 2 +- flink-sql-connector-mongodb/src/main/resources/META-INF/NOTICE | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/NOTICE b/NOTICE index 5949650..d95f518 100644 --- a/NOTICE +++ b/NOTICE @@ -1,5 +1,5 @@ Apache Flink MongoDB Connector -Copyright 2014-2022 The Apache Software Foundation +Copyright 2014-2023 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/). diff --git a/flink-sql-connector-mongodb/src/main/resources/META-INF/NOTICE b/flink-sql-connector-mongodb/src/main/resources/META-INF/NOTICE index 9514eed..86f1af0 100644 --- a/flink-sql-connector-mongodb/src/main/resources/META-INF/NOTICE +++ b/flink-sql-connector-mongodb/src/main/resources/META-INF/NOTICE @@ -1,5 +1,5 @@ flink-sql-connector-mongodb -Copyright 2014-2022 The Apache Software Foundation +Copyright 2014-2023 The Apache Software Foundation This product includes software developed at The Apache Software Foundation (http://www.apache.org/).
[flink-web] branch asf-site updated: Rebuild website
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/flink-web.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 335155f19 Rebuild website
335155f19 is described below
commit 335155f1965d769ae63e45af2cf30461af5e0e73
Author: Danny Cranmer
AuthorDate: Fri Aug 4 16:27:35 2023 +0100
Rebuild website
---
.../index.html | 359 +++--
content/en/sitemap.xml | 7 +-
content/index.html | 13 +-
content/index.xml | 12 +-
.../news/2023/08/04/externalized-connectors.html | 10 +
content/posts/index.html | 57 ++--
content/posts/index.xml| 12 +-
content/posts/page/10/index.html | 61 ++--
content/posts/page/11/index.html | 57 ++--
content/posts/page/12/index.html | 62 ++--
content/posts/page/13/index.html | 66 ++--
content/posts/page/14/index.html | 60 ++--
content/posts/page/15/index.html | 55 ++--
content/posts/page/16/index.html | 54 ++--
content/posts/page/17/index.html | 66 ++--
content/posts/page/18/index.html | 64 ++--
content/posts/page/19/index.html | 50 +--
content/posts/page/2/index.html| 57 ++--
content/posts/page/20/index.html | 49 +--
content/posts/page/21/index.html | 53 +--
content/posts/page/22/index.html | 59 ++--
content/posts/page/{22 => 23}/index.html | 193 +--
content/posts/page/3/index.html| 59 ++--
content/posts/page/4/index.html| 64 ++--
content/posts/page/5/index.html| 71 ++--
content/posts/page/6/index.html| 67 ++--
content/posts/page/7/index.html| 58 ++--
content/posts/page/8/index.html| 57 ++--
content/posts/page/9/index.html| 61 ++--
content/sitemap.xml| 2 +-
content/zh/index.html | 15 +-
31 files changed, 988 insertions(+), 942 deletions(-)
diff --git a/content/index.html
b/content/2023/08/04/announcing-three-new-apache-flink-connectors-the-new-connector-versioning-strategy-and-externalization/index.html
similarity index 63%
copy from content/index.html
copy to
content/2023/08/04/announcing-three-new-apache-flink-connectors-the-new-connector-versioning-strategy-and-externalization/index.html
index 320b3303c..438253627 100644
--- a/content/index.html
+++
b/content/2023/08/04/announcing-three-new-apache-flink-connectors-the-new-connector-versioning-strategy-and-externalization/index.html
@@ -6,19 +6,20 @@
-
-
-
-https://flink.apache.org/; />
-Apache Flink® — Stateful Computations over Data Streams | Apache
Flink
+
+
+
+
+https://flink.apache.org/2023/08/04/announcing-three-new-apache-flink-connectors-the-new-connector-versioning-strategy-and-externalization/;
/>
+
+
+Announcing three new Apache Flink connectors, the new connector
versioning strategy and externalization | Apache Flink
-https://flink.apache.org/zh/;
title="Apache Flink Documentation">
-
-https://flink.apache.org/index.xml; title="Apache Flink" />
-
- Apache Flink® — Stateful Computations over Data Streams
- #
-
-
-https://flink.apache.org/img/flink-home-graphic.png;
- alt=""
- width="800px"
- style="display:block;margin-left:auto;margin-right:auto"
->
-
-
-
-
-
-
- All streaming use
cases
-
-
-
-Event-driven Applications
-Stream Batch Analytics
-Data Pipelines ETL
-
-Learn more
-
-
-
-
-
-
-
-
-
- Guaranteed
correctness
-
-
-
-Exactly-once state consistency
-Event-time processing
-Sophisticated late data handling
-
-Learn
more
-
-
+
+
+Announcing
three new Apache Flink connectors, the new connector versioning strategy and
externalization
+
+
+ August 4, 2023 -
-
-
-
-
-
-
-
Layered APIs
-
-
-
-SQL on Stream Batch Data
-DataStream API DataSet API
-ProcessFunction (Time State)
-
-Learn more
-
-
-
-
-
-
-
-
-
-
-
- Operational
Focus
-
-
-
-Flexible deployment
-High-availability setup
-Savepoints
-
-Learn more
-
-
+ Elphas Toringepi
-
+ https://twitter.com/elphastori;>(@elphastori)
-
-
-
-
-
[flink-web] branch asf-site updated: [FLINK-32736] Externalized connectors
This is an automated email from the ASF dual-hosted git repository. dannycranmer pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/flink-web.git The following commit(s) were added to refs/heads/asf-site by this push: new aca5d7a38 [FLINK-32736] Externalized connectors aca5d7a38 is described below commit aca5d7a38f0d30590679e039d7b34f73592ac40f Author: Elphas Toringepi AuthorDate: Thu Aug 3 18:44:52 2023 +0100 [FLINK-32736] Externalized connectors --- .../posts/2023-08-04-externalized-connectors.md| 77 ++ 1 file changed, 77 insertions(+) diff --git a/docs/content/posts/2023-08-04-externalized-connectors.md b/docs/content/posts/2023-08-04-externalized-connectors.md new file mode 100644 index 0..4386fd48e --- /dev/null +++ b/docs/content/posts/2023-08-04-externalized-connectors.md @@ -0,0 +1,77 @@ +--- +title: "Announcing three new Apache Flink connectors, the new connector versioning strategy and externalization" +date: "2023-08-04T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/04/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now supports three new connectors: [Amazon DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[Amazon DynamoDB](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - This connector includes a sink that provides at-least-once delivery guarantees. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - This connector includes a source and sink that provide at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector includes a sink that provides at-least-once guarantees. + +|Connector|Date Released|Supported Flink Versions| +|---|---|---| +|Amazon DynamoDB sink|2022-12-02|1.15+| +|MongoDB connector|2023-03-31|1.16+| +|OpenSearch sink|2022-12-21|1.16+| + +### List of Contributors + +The Apache Flink community would like to express gratitude to all the new connector contributors: + +Andriy Redko, Chesnay Schepler, Danny Cranmer, darenwkt, Hong Liang Teoh, Jiabao Sun, Leonid Ilyevsky, Martijn Visser, nir.tsruya, Sergey Nuyanzin, Weijie Guo, Yuri Gusev, Yuxin Tan + +## Externalized connectors + +The community has externalized connectors from [Flink’s main repository](https://github.com/apache/flink). This was driven to realise the following benefits: +- **Faster releases of connectors:** New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws. +- **Adding newer connector features to older Flink versions:** By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions. +- **More activity and contributions around connectors:** By easing the contribution and development process around connectors, we will see faster development and also more connectors. +- **Documentation:** Standardized documentation and user experience for the connectors, regardless of where they are maintained. +- **A faster Flink CI:** By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience fewer build instabilities (which mostly come from connectors). That should speed up Flink development. + +The following connectors have been moved to individual repositories: + +- [Kafka / Upsert-Kafka](https://github.com/apache/flink-connector-kafka) +- [Cassandra](https://github.com/apache/flink-connector-cassandra) +- [Elasticsearch](https://github.com/apache/flink-connector-elasticsearch/) +- [**MongoDB**](https://github.com/apache/flink-connector-mongodb) +- **[OpenSearch](https://github.com/apache/flink-connector-opensearch)** +- [RabbitMQ](https://github.com/apache/flink-connector-rabbitmq) +- [Google Cloud PubSub](https://github.com/apache/flink-connector-gcp-pubsub) +- [Pulsar](https://github.com/apache/flink-connector-pulsar/) +- [JDBC](https://github.com/apache/flink-connector-jdbc) +- [HBase](https://github.com/apache/flink-connector-hbase) +- [Hive](https://github.com/apache/flink-connector-hive) +- [AWS connectors](https://github.com/apache/flink-connector-aws): + - Firehose + - Kinesis + - **DynamoDB** + +###
svn commit: r63354 - in /dev/flink/flink-connector-mongodb-1.0.2-rc1: ./ flink-connector-mongodb-1.0.2-src.tgz flink-connector-mongodb-1.0.2-src.tgz.asc flink-connector-mongodb-1.0.2-src.tgz.sha512
Author: dannycranmer Date: Fri Aug 4 12:29:12 2023 New Revision: 63354 Log: Add flink-connector-mongodb-1.0.2-rc1 Added: dev/flink/flink-connector-mongodb-1.0.2-rc1/ dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz (with props) dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.asc dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.sha512 Added: dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz == Binary file - no diff available. Propchange: dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz -- svn:mime-type = application/octet-stream Added: dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.asc == --- dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.asc (added) +++ dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.asc Fri Aug 4 12:29:12 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEED3nyr7I1G8KWeFRFkfnB7BJf2NsFAmTM7w8ACgkQkfnB7BJf +2NvK+BAAlyrNF3mCis/pHLefVXSVkgH7tIkidgftCEXNOeeSA6dixLB0pnBSTHQ0 +e2I5/B4kJnZ46zU9Kb9HvrtgVmFEKLmbKUCrQCxOKpAoYJJOtDTKku3FmwXPS2w+ +mfl8BqPOaC1qkWKkIL49qW52BcVI7APUAY/kE8ps/KEugosM/Dmo7NrB8GfX0Szk +oacbAUsrXemmqdTroy5w84ZwpmTS/t82ac/nNe3IcC09Zc0bRZpKxY4E6yCWu0x0 ++mCUj/6Y6D1ciq5RXJghdfeF5Ni08wVZKGLZUPjcw1Cz2BKUw4rE6Vd44uWwCyGC +D6NewsNuuJ/v8K1xHjK3PB1pxuylGdsA4TNRwH0Gr8XUt2/3H+qkmDfh9uMKs3e4 +JvHihVia9FdTjuJmdoY3NZZR2sFYAI2RM6DX8de1ZphYVPHWl7ZyT3rkPxCvhJ9a +hnQAaeyJ6d1d3TrNl8SBa3QBKU6S9FDKUelWgyq3LjRQHOc2ejR0geUX5Q4CNGbB +N4CU3EG4oA/q7B5GRhe07JmBlO5kzNWxyiFxjz8LD+mb/CN4sYtD3XYU1XIanHmh +cTM68LDDZPv+qN5TAKCGTpOJwoodbnV6pR0C3mw6limSwOJDru+FKF1mnpHvuoR1 +zJwfygrXwhP2BB9qAe5oknAKpruvRK2MdRb97pvhDQBfPj54qyI= +=h2Rd +-END PGP SIGNATURE- Added: dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.sha512 == --- dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.sha512 (added) +++ dev/flink/flink-connector-mongodb-1.0.2-rc1/flink-connector-mongodb-1.0.2-src.tgz.sha512 Fri Aug 4 12:29:12 2023 @@ -0,0 +1 @@ +0644d424499486dc28774cd48e1279dfa7aacf3c705984188a3f223d0d9a835dfb2cd2cef077aed181062997607db7579b9d2c452fb2e8040d55315dd5f6c66d flink-connector-mongodb-1.0.2-src.tgz
[flink-connector-mongodb] annotated tag v1.0.2-rc1 updated (81eb188 -> f53a374)
This is an automated email from the ASF dual-hosted git repository. dannycranmer pushed a change to annotated tag v1.0.2-rc1 in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git *** WARNING: tag v1.0.2-rc1 was modified! *** from 81eb188 (commit) to f53a374 (tag) tagging 81eb188daf8ed7c699245318e5cf4076995cf494 (commit) replaces v1.0.0 by Danny Cranmer on Fri Aug 4 13:27:20 2023 +0100 - Log - v1.0.2-rc1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEED3nyr7I1G8KWeFRFkfnB7BJf2NsFAmTM7qkACgkQkfnB7BJf 2NsJuhAAqFhSLDbFaD/cW74EZ8zVlmdBHYsCcHaNWMAVIPtJqN+NCq2zB6H4cwtl a3vL6Sttf09L8qZXFyWdcnCKys6dmKpKpQx1YgvlWtYXfboqgitV6QvWK4d/wzxd A6cg4IV2kT0J1TGmUEOV2+is1x6MmkCCK3qmtXmlURQ1GRZmCs11VeJIEzXYlGVy pbC3XxyXSo6T5MBbSMGn81KHShKXWTYEtYBu2dPgb1OXD75HKD6qRdXwTQCi988p OMvbayb9KraOcQ0Z5THYmKf4WTLwQplWd5aBhHuWxp0RUNy59UaErljLOgFECUXE jl/H6ipL7tD58RzAtwHBgFMXeeCLgfZSbdjPIaXGAQeFwzRRmb8BRLTh3pA7zH9W bHbCwVNy1CW5v4/U0zCSuiCso8Vjp/x78z9ZnID6O/IoSO477EfeLrJzbC1V4lNY vOL0wkvtNeUd5wWg8GKaTnUwnKJiz8vGRi27FGtS2K+BHjhmL+0txZIM8FsXgfED eVcBPsh5bch0jHS0gt1wV16xGEX9ASayO/NgbZP41xWHjYVWiveteiaQ/xrKUCu3 5aC1Pzq29RojlnVX0YmgzLcisQ5Jre7HEsggXd2CdbP5+nognlcOk5KTR3INfE0F 8CP0Dye4pHOEOrqi9305dPILDLvyzDscoQRCWNeTf5X2tMQb6ng= =UabL -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[flink-connector-mongodb] 03/05: [hotfix] Test PRs against 1.17-SNAPSHOT and 1.18-SNAPSHOT
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch v1.0
in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git
commit a351ddf63e2e04aae403d164bc2faab5c5fff767
Author: Martijn Visser
AuthorDate: Thu Jun 15 10:54:56 2023 +0200
[hotfix] Test PRs against 1.17-SNAPSHOT and 1.18-SNAPSHOT
---
.github/workflows/push_pr.yml | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/push_pr.yml b/.github/workflows/push_pr.yml
index 335f108..8cc815d 100644
--- a/.github/workflows/push_pr.yml
+++ b/.github/workflows/push_pr.yml
@@ -23,6 +23,9 @@ concurrency:
cancel-in-progress: true
jobs:
compile_and_test:
+strategy:
+ matrix:
+flink: [1.17-SNAPSHOT, 1.18-SNAPSHOT]
uses: apache/flink-connector-shared-utils/.github/workflows/ci.yml@ci_utils
with:
- flink_version: 1.17.0
+ flink_version: ${{ matrix.flink }}
[flink-connector-mongodb] 01/05: [FLINK-32187] Remove dependency on flink-shaded
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch v1.0
in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git
commit 23b37894e2bdd82f4956533e331f3aab22576694
Author: Sergey Nuyanzin
AuthorDate: Thu May 25 12:06:10 2023 +0200
[FLINK-32187] Remove dependency on flink-shaded
---
.editorconfig | 2 +-
pom.xml| 12
tools/maven/checkstyle.xml | 2 +-
3 files changed, 2 insertions(+), 14 deletions(-)
diff --git a/.editorconfig b/.editorconfig
index b66771d..ab5d802 100644
--- a/.editorconfig
+++ b/.editorconfig
@@ -113,7 +113,7 @@ ij_java_doc_preserve_line_breaks = false
# ij_java_generate_final_locals = false
# ij_java_generate_final_parameters = false
# ij_java_if_brace_force = never
-ij_java_imports_layout =
org.apache.flink.**,|,org.apache.flink.shaded.**,|,*,|,javax.**,|,java.**,|,scala.**,|,$*
+ij_java_imports_layout =
org.apache.flink.**,|,|,*,|,javax.**,|,java.**,|,scala.**,|,$*
# ij_java_indent_case_from_switch = true
# ij_java_insert_inner_class_imports = false
# ij_java_insert_override_annotation = true
diff --git a/pom.xml b/pom.xml
index 65ff786..8389f9b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -54,7 +54,6 @@ under the License.
4.7.2
1.17.0
- 16.1
5.8.1
3.21.0
@@ -77,11 +76,6 @@ under the License.
-
- org.apache.flink
- flink-shaded-force-shading
-
-
@@ -194,12 +188,6 @@ under the License.
test
-
- org.apache.flink
-
flink-shaded-force-shading
- ${flink.shaded.version}
-
-
diff --git a/tools/maven/checkstyle.xml b/tools/maven/checkstyle.xml
index 3fc4711..eb62611 100644
--- a/tools/maven/checkstyle.xml
+++ b/tools/maven/checkstyle.xml
@@ -195,7 +195,7 @@ This file is based on the checkstyle file of Apache Beam.
+
value="org.apache.flink,*,javax,java,scala"/>
[flink-connector-mongodb] 04/05: [FLINK-32446][connectors/mongodb] MongoWriter should regularly check whether the latest flush time is arriving
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch v1.0
in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git
commit 1de499900fdb4d934d95c52808df6516d627ed58
Author: Jiabao Sun
AuthorDate: Wed Jun 28 12:02:44 2023 +0800
[FLINK-32446][connectors/mongodb] MongoWriter should regularly check
whether the latest flush time is arriving
This closes #12.
---
.../connector/mongodb/sink/writer/MongoWriter.java | 83 +++---
.../mongodb/sink/writer/MongoWriterITCase.java | 7 +-
.../connector/mongodb/testutils/MongoTestUtil.java | 13
3 files changed, 92 insertions(+), 11 deletions(-)
diff --git
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java
index df76891..5319959 100644
---
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java
+++
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/MongoWriter.java
@@ -33,6 +33,7 @@ import org.apache.flink.metrics.Counter;
import org.apache.flink.metrics.groups.SinkWriterMetricGroup;
import org.apache.flink.util.Collector;
import org.apache.flink.util.FlinkRuntimeException;
+import org.apache.flink.util.concurrent.ExecutorThreadFactory;
import com.mongodb.MongoException;
import com.mongodb.client.MongoClient;
@@ -45,6 +46,10 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
import static org.apache.flink.util.Preconditions.checkNotNull;
@@ -68,11 +73,18 @@ public class MongoWriter implements SinkWriter {
private final Collector> collector;
private final Counter numRecordsOut;
private final MongoClient mongoClient;
+private final long batchIntervalMs;
+private final int batchSize;
private boolean checkpointInProgress = false;
private volatile long lastSendTime = 0L;
private volatile long ackTime = Long.MAX_VALUE;
+private transient volatile boolean closed = false;
+private transient ScheduledExecutorService scheduler;
+private transient ScheduledFuture scheduledFuture;
+private transient volatile Exception flushException;
+
public MongoWriter(
MongoConnectionOptions connectionOptions,
MongoWriteOptions writeOptions,
@@ -83,6 +95,8 @@ public class MongoWriter implements SinkWriter {
this.writeOptions = checkNotNull(writeOptions);
this.serializationSchema = checkNotNull(serializationSchema);
this.flushOnCheckpoint = flushOnCheckpoint;
+this.batchIntervalMs = writeOptions.getBatchIntervalMs();
+this.batchSize = writeOptions.getBatchSize();
checkNotNull(initContext);
this.mailboxExecutor = checkNotNull(initContext.getMailboxExecutor());
@@ -105,10 +119,37 @@ public class MongoWriter implements SinkWriter {
// Initialize the mongo client.
this.mongoClient = MongoClients.create(connectionOptions.getUri());
+
+boolean flushOnlyOnCheckpoint = batchIntervalMs == -1 && batchSize ==
-1;
+
+if (!flushOnlyOnCheckpoint && batchIntervalMs > 0) {
+this.scheduler =
+Executors.newScheduledThreadPool(1, new
ExecutorThreadFactory("mongo-writer"));
+
+this.scheduledFuture =
+this.scheduler.scheduleWithFixedDelay(
+() -> {
+synchronized (MongoWriter.this) {
+if (!closed &&
isOverMaxBatchIntervalLimit()) {
+try {
+doBulkWrite();
+} catch (Exception e) {
+flushException = e;
+}
+}
+}
+},
+batchIntervalMs,
+batchIntervalMs,
+TimeUnit.MILLISECONDS);
+}
}
@Override
-public void write(IN element, Context context) throws IOException,
InterruptedException {
+public synchronized void write(IN element, Context context)
+throws IOException, InterruptedException {
+checkFlushException();
+
// do not allow new bulk writes until all actions are flushed
while (checkpointInProgress) {
mailboxExecutor.yield();
@@ -122,7 +163,9 @@ public class MongoWriter implements SinkWriter {
}
[flink-connector-mongodb] 02/05: [FLINK-31923] Run nightly builds against multiple branches and Flink versions
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch v1.0
in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git
commit 99755b44f9964cecd066641d2147aa134ed41b00
Author: Martijn Visser
AuthorDate: Thu Jun 15 10:54:10 2023 +0200
[FLINK-31923] Run nightly builds against multiple branches and Flink
versions
---
.github/workflows/weekly.yml | 24 ++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/weekly.yml b/.github/workflows/weekly.yml
index 577ff1f..bba17f1 100644
--- a/.github/workflows/weekly.yml
+++ b/.github/workflows/weekly.yml
@@ -26,7 +26,27 @@ jobs:
if: github.repository_owner == 'apache'
strategy:
matrix:
-flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT]
+flink_branches: [{
+ flink: 1.16-SNAPSHOT,
+ branch: main
+}, {
+ flink: 1.17-SNAPSHOT,
+ branch: main
+}, {
+ flink: 1.18-SNAPSHOT,
+ branch: main
+}, {
+ flink: 1.16.2,
+ branch: v1.0
+}, {
+ flink: 1.17.1,
+ branch: v1.0
+}, {
+ flink: 1.18-SNAPSHOT,
+ branch: v1.0
+}]
uses: apache/flink-connector-shared-utils/.github/workflows/ci.yml@ci_utils
with:
- flink_version: ${{ matrix.flink }}
+ flink_version: ${{ matrix.flink_branches.flink }}
+ connector_branch: ${{ matrix.flink_branches.branch }}
+ run_dependency_convergence: false
[flink-connector-mongodb] branch v1.0 updated (bea5957 -> 723b34a)
This is an automated email from the ASF dual-hosted git repository. dannycranmer pushed a change to branch v1.0 in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git from bea5957 [FLINK-32024][docs] Short code related to externalized connector retrieve version from its own data yaml new 23b3789 [FLINK-32187] Remove dependency on flink-shaded new 99755b4 [FLINK-31923] Run nightly builds against multiple branches and Flink versions new a351ddf [hotfix] Test PRs against 1.17-SNAPSHOT and 1.18-SNAPSHOT new 1de4999 [FLINK-32446][connectors/mongodb] MongoWriter should regularly check whether the latest flush time is arriving new 723b34a [FLINK-32348][connectors/mongodb][tests] Fix MongoDB tests are flaky and time out The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .editorconfig | 2 +- .github/workflows/push_pr.yml | 5 +- .github/workflows/weekly.yml | 24 ++- .../connector/mongodb/sink/writer/MongoWriter.java | 83 +++--- .../source/enumerator/MongoSourceEnumerator.java | 13 ++-- .../reader/split/MongoScanSourceSplitReader.java | 1 - .../mongodb/sink/writer/MongoWriterITCase.java | 7 +- .../connector/mongodb/testutils/MongoTestUtil.java | 13 pom.xml| 12 tools/maven/checkstyle.xml | 2 +- 10 files changed, 129 insertions(+), 33 deletions(-)
[flink-connector-mongodb] 05/05: [FLINK-32348][connectors/mongodb][tests] Fix MongoDB tests are flaky and time out
This is an automated email from the ASF dual-hosted git repository.
dannycranmer pushed a commit to branch v1.0
in repository https://gitbox.apache.org/repos/asf/flink-connector-mongodb.git
commit 723b34a56012bf857a5e7da540adfe12a44bfe4c
Author: Jiabao Sun
AuthorDate: Wed Jul 12 17:12:06 2023 +0800
[FLINK-32348][connectors/mongodb][tests] Fix MongoDB tests are flaky and
time out
This closes #13.
---
.../mongodb/source/enumerator/MongoSourceEnumerator.java| 13 +
.../source/reader/split/MongoScanSourceSplitReader.java | 1 -
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/enumerator/MongoSourceEnumerator.java
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/enumerator/MongoSourceEnumerator.java
index d13a843..951c527 100644
---
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/enumerator/MongoSourceEnumerator.java
+++
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/enumerator/MongoSourceEnumerator.java
@@ -97,6 +97,15 @@ public class MongoSourceEnumerator
continue;
}
+// close idle readers
+if (splitAssigner.noMoreSplits() && boundedness ==
Boundedness.BOUNDED) {
+context.signalNoMoreSplits(nextAwaiting);
+awaitingReader.remove();
+LOG.info(
+"All scan splits have been assigned, closing idle
reader {}", nextAwaiting);
+continue;
+}
+
Optional split = splitAssigner.getNext();
if (split.isPresent()) {
final MongoSourceSplit mongoSplit = split.get();
@@ -104,10 +113,6 @@ public class MongoSourceEnumerator
awaitingReader.remove();
LOG.info("Assign split {} to subtask {}", mongoSplit,
nextAwaiting);
break;
-} else if (splitAssigner.noMoreSplits() && boundedness ==
Boundedness.BOUNDED) {
-LOG.info("All splits have been assigned");
-
context.registeredReaders().keySet().forEach(context::signalNoMoreSplits);
-break;
} else {
// there is no available splits by now, skip assigning
break;
diff --git
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/split/MongoScanSourceSplitReader.java
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/split/MongoScanSourceSplitReader.java
index 4702f94..134fe73 100644
---
a/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/split/MongoScanSourceSplitReader.java
+++
b/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/split/MongoScanSourceSplitReader.java
@@ -119,7 +119,6 @@ public class MongoScanSourceSplitReader implements
MongoSourceSplitReader
[flink] branch master updated (34729c88914 -> aa98c18d2ba)
This is an automated email from the ASF dual-hosted git repository. xtsong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 34729c88914 [FLINK-32656][table] Deprecate ManagedTable related APIs add aa98c18d2ba [FLINK-32558][flink-java] Deprecate all DataSet API No new revisions were added by this update. Summary of changes: .../generated/pipeline_configuration.html | 6 - .../apache/flink/api/common/ExecutionConfig.java | 38 .../org/apache/flink/api/common/ExecutionMode.java | 8 + .../flink/configuration/PipelineOptions.java | 13 ++ flink-dist/src/main/assemblies/bin.xml | 6 +- flink-examples/flink-examples-batch/pom.xml| 42 ++-- .../flink/examples/java/clustering/KMeans.java | 13 ++ .../apache/flink/examples/java/distcp/DistCp.java | 8 + .../examples/java/graph/ConnectedComponents.java | 13 ++ .../flink/examples/java/graph/EnumTriangles.java | 13 ++ .../apache/flink/examples/java/graph/PageRank.java | 11 + .../java/graph/TransitiveClosureNaive.java | 13 ++ .../java/misc/CollectionExecutionExample.java | 14 ++ .../flink/examples/java/misc/PiEstimation.java | 13 ++ .../relational/EmptyFieldsCountAccumulator.java| 12 ++ .../examples/java/relational/TPCHQuery10.java | 13 ++ .../flink/examples/java/relational/TPCHQuery3.java | 13 ++ .../examples/java/relational/WebLogAnalysis.java | 13 ++ .../examples/java/util/DataSetDeprecationInfo.java | 20 +- .../flink/examples/java/wordcount/WordCount.java | 13 ++ .../examples/java/wordcount/WordCountPojo.java | 13 ++ flink-examples/flink-examples-streaming/pom.xml| 1 + .../streaming/examples/async/AsyncIOExample.java | 2 +- .../streaming/examples/gpu/MatrixVectorMul.java| 2 +- .../examples/iteration/IterateExample.java | 2 +- .../flink/streaming/examples/join/WindowJoin.java | 5 +- .../examples/sideoutput/SideOutputExample.java | 2 +- .../examples/socket/SocketWindowWordCount.java | 2 +- .../statemachine/KafkaEventsGeneratorJob.java | 2 +- .../examples/statemachine/StateMachineExample.java | 2 +- .../streaming/examples/utils/ParameterTool.java| 227 + .../examples/windowing/SessionWindowing.java | 2 +- .../streaming/examples/wordcount/util/CLI.java | 10 +- .../java/connectors/ChangelogSocketExample.java| 43 +++- .../flink/api/java/CollectionEnvironment.java | 7 + .../java/org/apache/flink/api/java/DataSet.java| 6 + .../flink/api/java/ExecutionEnvironment.java | 6 + .../api/java/ExecutionEnvironmentFactory.java | 11 +- .../apache/flink/api/java/LocalEnvironment.java| 7 + .../apache/flink/api/java/RemoteEnvironment.java | 7 + .../flink/api/java/aggregation/Aggregations.java | 11 +- .../UnsupportedAggregationTypeException.java | 11 +- .../flink/api/java/functions/FlatMapIterator.java | 6 + .../api/java/functions/FunctionAnnotation.java | 7 + .../api/java/functions/GroupReduceIterator.java| 6 + .../flink/api/java/io/CollectionInputFormat.java | 11 +- .../apache/flink/api/java/io/CsvOutputFormat.java | 7 + .../org/apache/flink/api/java/io/CsvReader.java| 34 ++- .../flink/api/java/io/DiscardingOutputFormat.java | 6 + .../flink/api/java/io/IteratorInputFormat.java | 11 +- .../api/java/io/LocalCollectionOutputFormat.java | 11 +- .../api/java/io/ParallelIteratorInputFormat.java | 11 +- .../flink/api/java/io/PrimitiveInputFormat.java| 7 + .../flink/api/java/io/PrintingOutputFormat.java| 6 + .../flink/api/java/io/RowCsvInputFormat.java | 11 +- .../flink/api/java/io/SplitDataProperties.java | 6 + .../apache/flink/api/java/io/TextInputFormat.java | 11 +- .../apache/flink/api/java/io/TextOutputFormat.java | 6 + .../flink/api/java/io/TextValueInputFormat.java| 11 +- .../api/java/io/TypeSerializerInputFormat.java | 6 + .../api/java/io/TypeSerializerOutputFormat.java| 6 + .../api/java/operators/AggregateOperator.java | 6 + .../flink/api/java/operators/CoGroupOperator.java | 6 + .../flink/api/java/operators/CrossOperator.java| 6 + .../api/java/operators/CustomUnaryOperation.java | 6 + .../apache/flink/api/java/operators/DataSink.java | 6 + .../flink/api/java/operators/DataSource.java | 6 + .../flink/api/java/operators/DeltaIteration.java | 6 + .../java/operators/DeltaIterationResultSet.java| 6 + .../flink/api/java/operators/DistinctOperator.java | 6 + .../flink/api/java/operators/FilterOperator.java | 6 + .../flink/api/java/operators/FlatMapOperator.java | 6 + .../api/java/operators/GroupCombineOperator.java | 6 + .../api/java/operators/GroupReduceOperator.java| 6 + .../apache/flink/api/java/operators/Grouping.java | 6 +
[flink] branch master updated (50622df1f01 -> 34729c88914)
This is an automated email from the ASF dual-hosted git repository. czweng pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/flink.git from 50622df1f01 [FLINK-29527][format] Make unknownFieldsIndices work for single ParquetReader (#21149) add 34729c88914 [FLINK-32656][table] Deprecate ManagedTable related APIs No new revisions were added by this update. Summary of changes: .../org/apache/flink/table/catalog/hive/HiveCatalogLock.java | 8 +++- .../main/java/org/apache/flink/table/api/TableDescriptor.java | 7 ++- .../org/apache/flink/table/catalog/ManagedTableListener.java | 7 ++- .../src/main/java/org/apache/flink/table/catalog/Catalog.java | 7 ++- .../main/java/org/apache/flink/table/catalog/CatalogLock.java | 10 +- .../org/apache/flink/table/connector/RequireCatalogLock.java | 7 ++- .../org/apache/flink/table/factories/ManagedTableFactory.java | 3 +++ 7 files changed, 43 insertions(+), 6 deletions(-)
