This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 04b49e6d7 [Doc] seatunnel run with flink operator error #3998 (#4024)
04b49e6d7 is described below
commit 04b49e6d73f81bcb8cbdf911696fcacf67b87209
Author: 好好学习的郝 <[email protected]>
AuthorDate: Wed Feb 1 12:14:37 2023 +0800
[Doc] seatunnel run with flink operator error #3998 (#4024)
---
docs/en/start-v2/kubernetes/kubernetes.mdx | 130 ++++++++++++++++-------------
release-note.md | 2 +
2 files changed, 75 insertions(+), 57 deletions(-)
diff --git a/docs/en/start-v2/kubernetes/kubernetes.mdx
b/docs/en/start-v2/kubernetes/kubernetes.mdx
index 5ec7ceedb..9157a423e 100644
--- a/docs/en/start-v2/kubernetes/kubernetes.mdx
+++ b/docs/en/start-v2/kubernetes/kubernetes.mdx
@@ -42,7 +42,7 @@ To run the image with SeaTunnel, first create a `Dockerfile`:
```Dockerfile
FROM flink:1.13
-ENV SEATUNNEL_VERSION="2.3.0-beta"
+ENV SEATUNNEL_VERSION="2.3.0"
ENV SEATUNNEL_HOME = "/opt/seatunnel"
RUN mkdir -p $SEATUNNEL_HOME
@@ -52,18 +52,17 @@ RUN tar -xzvf
apache-seatunnel-incubating-${SEATUNNEL_VERSION}-bin.tar.gz
RUN cp -r apache-seatunnel-incubating-${SEATUNNEL_VERSION}/* $SEATUNNEL_HOME/
RUN rm -rf apache-seatunnel-incubating-${SEATUNNEL_VERSION}*
-RUN rm -rf $SEATUNNEL_HOME/connectors/seatunnel
```
Then run the following commands to build the image:
```bash
-docker build -t seatunnel:2.3.0-beta-flink-1.13 -f Dockerfile .
+docker build -t seatunnel:2.3.0-flink-1.13 -f Dockerfile .
```
-Image `seatunnel:2.3.0-beta-flink-1.13` need to be present in the host
(minikube) so that the deployment can take place.
+Image `seatunnel:2.3.0-flink-1.13` need to be present in the host (minikube)
so that the deployment can take place.
Load image to minikube via:
```bash
-minikube image load seatunnel:2.3.0-beta-flink-1.13
+minikube image load seatunnel:2.3.0-flink-1.13
```
</TabItem>
@@ -79,20 +78,23 @@ minikube image load seatunnel:2.3.0-beta-flink-1.13
]}>
<TabItem value="flink">
-The steps below provide a quick walk-through on setting up the Flink
Kubernetes Operator.
+The steps below provide a quick walk-through on setting up the Flink
Kubernetes Operator.
+You can refer to [Flink Kubernetes Operator - Quick
Start](https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/)
for more details.
+
+> Notice: All the Kubernetes resources bellow are created in default namespace.
Install the certificate manager on your Kubernetes cluster to enable adding
the webhook component (only needed once per Kubernetes cluster):
```bash
-kubectl create -f
https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml
+kubectl create -f
https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
```
Now you can deploy the latest stable Flink Kubernetes Operator version using
the included Helm chart:
```bash
+helm repo add flink-operator-repo
https://downloads.apache.org/flink/flink-kubernetes-operator-1.3.1/
-helm repo add flink-operator-repo
https://downloads.apache.org/flink/flink-kubernetes-operator-0.1.0/
-
-helm install flink-kubernetes-operator
flink-operator-repo/flink-kubernetes-operator
+helm install flink-kubernetes-operator
flink-operator-repo/flink-kubernetes-operator \
+--set image.repository=apache/flink-kubernetes-operator
```
You may verify your installation via `kubectl`:
@@ -119,94 +121,93 @@ flink-kubernetes-operator-5f466b8549-mgchb
1/1 Running 3 (23h
]}>
<TabItem value="flink">
-In this guide we are going to use
[flink.streaming.conf](https://github.com/apache/incubator-seatunnel/blob/dev/config/flink.streaming.conf.template):
+In this guide we are going to use
[seatunnel.streaming.conf](https://github.com/apache/incubator-seatunnel/blob/2.3.0-release/config/v2.streaming.conf.template):
- ```conf
+```conf
env {
execution.parallelism = 1
+ job.mode = "STREAMING"
+ checkpoint.interval = 2000
}
source {
- FakeSourceStream {
+ FakeSource {
result_table_name = "fake"
- field_name = "name,age"
+ row.num = 160000
+ schema = {
+ fields {
+ name = "string"
+ age = "int"
+ }
+ }
}
}
transform {
- sql {
- sql = "select name,age from fake"
- }
+
}
sink {
- ConsoleSink {}
+ Console {}
}
- ```
-
-This configuration need to be present when we are going to deploy the
application (SeaTunnel) to Flink cluster (on Kubernetes), we also need to
configure a Pod to Use a PersistentVolume for Storage.
-- Create `/mnt/data` on your Node. Open a shell to the single Node in your
cluster. How you open a shell depends on how you set up your cluster. For
example, in our case weare using Minikube, you can open a shell to your Node by
entering `minikube ssh`.
-In your shell on that Node, create a /mnt/data directory:
-```bash
-minikube ssh
-
-# This assumes that your Node uses "sudo" to run commands
-# as the superuser
-sudo mkdir /mnt/data
```
-- Copy application (SeaTunnel) configuration files to your Node.
+
+Generate a configmap named seatunnel-config in Kubernetes for the
seatunnel.streaming.conf so that we can mount the config content in pod.
```bash
-minikube cp flink.streaming.conf /mnt/data/flink.streaming.conf
+kubectl create cm seatunnel-config \
+--from-file=seatunnel.streaming.conf=seatunnel.streaming.conf
```
Once the Flink Kubernetes Operator is running as seen in the previous steps
you are ready to submit a Flink (SeaTunnel) job:
- Create `seatunnel-flink.yaml` FlinkDeployment manifest:
```yaml
-apiVersion: flink.apache.org/v1alpha1
+apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
- namespace: default
name: seatunnel-flink-streaming-example
spec:
- image: seatunnel:2.3.0-beta-flink-1.13
- flinkVersion: v1_14
+ image: seatunnel:2.3.0-flink-1.13
+ flinkVersion: v1_13
flinkConfiguration:
taskmanager.numberOfTaskSlots: "2"
serviceAccount: flink
jobManager:
replicas: 1
resource:
- memory: "2048m"
+ memory: "1024m"
cpu: 1
taskManager:
resource:
- memory: "2048m"
- cpu: 2
+ memory: "1024m"
+ cpu: 1
podTemplate:
spec:
containers:
- name: flink-main-container
volumeMounts:
- - mountPath: /data
- name: config-volume
+ - name: seatunnel-config
+ mountPath: /data/seatunnel.streaming.conf
+ subPath: seatunnel.streaming.conf
volumes:
- - name: config-volume
- hostPath:
- path: "/mnt/data"
- type: Directory
-
+ - name: seatunnel-config
+ configMap:
+ name: seatunnel-config
+ items:
+ - key: seatunnel.streaming.conf
+ path: seatunnel.streaming.conf
job:
- jarURI: local:///opt/seatunnel/lib/seatunnel-flink-starter.jar
- entryClass: org.apache.seatunnel.core.starter.flink.SeaTunnelFlink
- args: ["--config", "/data/flink.streaming.conf"]
+ jarURI: local:///opt/seatunnel/starter/seatunnel-flink-starter.jar
+ entryClass: org.apache.seatunnel.core.starter.flink.SeatunnelFlink
+ args: ["--config", "/data/seatunnel.streaming.conf"]
parallelism: 2
upgradeMode: stateless
-
```
+
- Run the example application:
```bash
kubectl apply -f seatunnel-flink.yaml
```
+
</TabItem>
</Tabs>
@@ -225,6 +226,17 @@ You may follow the logs of your job, after a successful
startup (which can take
```bash
kubectl logs -f deploy/seatunnel-flink-streaming-example
```
+looks like the below:
+
+```shell
+...
+2023-01-31 12:13:54,349 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
SeaTunnel FakeSource -> Sink Writer: Console (1/1)
(1665d2d011b2f6cf6525c0e5e75ec251) switched from SCHEDULED to DEPLOYING.
+2023-01-31 12:13:56,684 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
Source: SeaTunnel FakeSource -> Sink Writer: Console (1/1) (attempt #0) with
attempt id 1665d2d011b2f6cf6525c0e5e75ec251 to
seatunnel-flink-streaming-example-taskmanager-1-1 @ 100.103.244.106
(dataPort=39137) with allocation id fbe162650c4126649afcdaff00e46875
+2023-01-31 12:13:57,794 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
SeaTunnel FakeSource -> Sink Writer: Console (1/1)
(1665d2d011b2f6cf6525c0e5e75ec251) switched from DEPLOYING to INITIALIZING.
+2023-01-31 12:13:58,203 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
SeaTunnel FakeSource -> Sink Writer: Console (1/1)
(1665d2d011b2f6cf6525c0e5e75ec251) switched from INITIALIZING to RUNNING.
+```
+
+If OOM error accur in the log, you can decrease the `row.num` value in
seatunnel.streaming.conf
To expose the Flink Dashboard you may add a port-forward rule:
```bash
@@ -241,16 +253,20 @@ kubectl logs \
--tail=-1 \
-f
```
-looks like the below (your content may be different since we use
`FakeSourceStream` to automatically generate random stream data):
+looks like the below (your content may be different since we use `FakeSource`
to automatically generate random stream data):
```shell
-+I[Kid Xiong, 1650316786086]
-+I[Ricky Huo, 1650316787089]
-+I[Ricky Huo, 1650316788089]
-+I[Ricky Huo, 1650316789090]
-+I[Kid Xiong, 1650316790090]
-+I[Kid Xiong, 1650316791091]
-+I[Kid Xiong, 1650316792092]
+...
+subtaskIndex=0: row=159991 : VVgpp, 978840000
+subtaskIndex=0: row=159992 : JxrOC, 1493825495
+subtaskIndex=0: row=159993 : YmCZR, 654146216
+subtaskIndex=0: row=159994 : LdmUn, 643140261
+subtaskIndex=0: row=159995 : tURkE, 837012821
+subtaskIndex=0: row=159996 : uPDfd, 2021489045
+subtaskIndex=0: row=159997 : mjrdG, 2074957853
+subtaskIndex=0: row=159998 : xbeUi, 864518418
+subtaskIndex=0: row=159999 : sSWLb, 1924451911
+subtaskIndex=0: row=160000 : AuPlM, 1255017876
```
To stop your job and delete your FlinkDeployment you can simply:
diff --git a/release-note.md b/release-note.md
index 31a985e71..616a0d0d6 100644
--- a/release-note.md
+++ b/release-note.md
@@ -52,6 +52,8 @@
- [Core]Fix Local Mode can't deserialize split (#3817)
- [Metrics] Fix Metrics will lose when Job be canceled. #3797
+### Documents
+- [Doc] seatunnel run with flink operator error #3998
## Test
### E2E
