This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 575dded6 [doc] Add doc about how to contribute a plugin (#1690)
575dded6 is described below
commit 575dded68c3bbe8bd6fd0680b4bed7eaaaa146da
Author: Wenjun Ruan <[email protected]>
AuthorDate: Wed Apr 13 16:03:12 2022 +0800
[doc] Add doc about how to contribute a plugin (#1690)
* [doc] Add how to contribute a plugin doc
* fix dead link
---
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
docs/en/contribution/contribute-plugin.md | 125 ++++++++++++++++++++++++++++++
2 files changed, 126 insertions(+), 1 deletion(-)
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 52a63326..3752eb8b 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -24,5 +24,5 @@ Feel free to ping committers for the review!
* [ ] Code changed are covered with tests, or it does not need tests for
reason:
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
- [New License
Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/development/new-license.md)
+ [New License
Guide](https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
diff --git a/docs/en/contribution/contribute-plugin.md
b/docs/en/contribution/contribute-plugin.md
new file mode 100644
index 00000000..95a95906
--- /dev/null
+++ b/docs/en/contribution/contribute-plugin.md
@@ -0,0 +1,125 @@
+# Contribute Spark Plugins
+
+There are two parent modules for Spark plugins:
+
+1.
[seatunnel-connectors-spark](https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-connectors/seatunnel-connectors-spark)
+2.
[seatunnel-transforms-spark](https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-transforms/seatunnel-transforms-spark)
+
+Once you want to contribute a new plugin, you need to:
+
+## Create plugin module
+Create your plugin module under the corresponding parent plugin module.
+For example, if you want to add a new Spark connector plugin, you need to
create a new module under the `seatunnel-connectors-spark` module.
+
+```java
+<project xmlns="http://maven.apache.org/POM/4.0.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <parent>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-connectors-spark</artifactId>
+ <version>${revision}</version>
+ </parent>
+ <modelVersion>4.0.0</modelVersion>
+
+ <artifactId>seatunnel-connector-spark-hello</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-api-spark</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+</project>
+```
+## Add plugin implementation
+
+You need to implement the `Connector` service provider interface. e.g.
`BaseSource`/`BaseSink`.
+
+Conveniently, there are some abstract class can help you easy to create your
plugin. If you want to create a source connector,
+you can implement with `SparkBatchSource`/`SparkStreamingSource`. If you want
to create a sink connector, you can implement with
`SparkBatchSink`/`SparkStreamingSink`.
+
+The methods defined in `SparkBatchSource` are some lifecycle methods. will be
executed by SeaTunnel engine.
+The execution order of the lifecycle methods is: `checkConfig` -> `prepare` ->
`getData` -> `close`.
+
+```java
+import java.util.Date;
+
+public class Hello extends SparkBatchSource {
+ @Override
+ public Dataset<Row> getData(SparkEnvironment env) {
+ // do your logic here to generate data
+ Dataset<Row> dataset = null;
+ return dataset;
+ }
+
+ @Override
+ public CheckResult checkConfig() {
+ return super.checkConfig();
+ }
+
+ @Override
+ public void prepare(SparkEnvironment env) {
+ super.prepare(env);
+ }
+
+ @Override
+ public void close() throws Exception {
+ super.close();
+ }
+
+ @Override
+ public String getPluginName() {
+ return "hello";
+ }
+}
+```
+The `getPluginName` method is used to identify the plugin name.
+
+After you finish your implementation, you need to add a service provider to
the `META-INF/services` file.
+The file name should be `org.apache.seatunnel.spark.BaseSparkSource` or
`org.apache.seatunnel.spark.BaseSparkSink`, dependents on the plugin type.
+The content of the file should be the fully qualified class name of your
implementation.
+
+## Add plugin to the distribution
+
+You need to add your plugin to the `seatunnel-core-spark` module, then the
plugin will in distribution.
+```java
+<dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>seatunnel-connector-spark-hello</artifactId>
+ <version>${project.version}</version>
+</dependency>
+```
+
+# Contribute Flink Plugins
+
+The steps to contribute a Flink plugin is similar to the steps to contribute a
Spark plugin.
+Different from Spark, you need to add your plugin in Flink plugin modules.
+
+# Add e2e tests for your plugin
+
+Once you add a new plugin, it is recommended to add e2e tests for it. We have
a `seatunnel-e2e` module to help you to do this.
+
+For example, if you want to add an e2e test for your flink connector, you can
create a new test in `seatunnel-flink-e2e` module.
+And extend the FlinkContainer class in the test.
+
+```java
+public class HellpSourceIT extends FlinkContainer {
+
+ @Test
+ public void testHellpSource() throws IOException, InterruptedException {
+ Container.ExecResult execResult =
executeSeaTunnelFlinkJob("/hello/hellosource.conf");
+ Assert.assertEquals(0, execResult.getExitCode());
+ // do some other assertion here
+ }
+
+```
+Once your class implements the `FlinkContainer` interface, it will auto create
a Flink cluster in Docker, and you just need to
+execute the `executeSeaTunnelFlinkJob` method with your SeaTunnel
configuration file, it will submit the SeaTunnel job.
+
+In most times, you need to start a third-part datasource in your test, for
example, if you add a clickhouse connectors, you may need to
+start a Clickhouse database in your test. You can use `GenericContainer` to
start a container.
+
+It should be noted that your e2e test class should be named ending with `IT`.
By default, we will not execute the test if the class name ending with `IT`.
+You can add `-DskipIT=false` to execute the e2e test, it will rely on a Docker
environment.
