[GitHub] [incubator-seatunnel] hailin0 opened a new issue, #3178: [WIP][Umbrella] SeaTunnel Transform API

[GitBox]

hailin0 opened a new issue, #3178:
URL: https://github.com/apache/incubator-seatunnel/issues/3178

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22)
 and found no similar issues.
   
   
   ### Describe the proposal
   
   # Backgroud #2678
   
   Currently, the transform code is bound to the single engine and cannot be 
shared to other engine using.
   
   I propose that we create `transform-v2` module to unify transform implement, 
like source and sink, it is decoupled from the engine and can run on different 
engines.
   
   Furthermore,  we can use the `translation` module to integrate transform to 
seatunnel, flink, spark engine execute.
   
   In order to ensure seatunnel's positioning as a data integration platform 
and not introduce work beyond the plan, the `transform-v2` will only support 
UDF level data conversion, and And unsupported sql transform(because st-engine 
unsupported sql parse & analysis).
   
   # Objective
   - Supports running on different engines
   - Supports update fields datatype & value & orders
   - Supports delete\add fields
   
   # Overall Design
   
   The Transform base process contains:
   - Transform implement
   
   <img width="564" alt="image" 
src="https://user-images.githubusercontent.com/14371345/197688154-2f38b996-3a42-4443-af13-bde9b74978bd.png";>
   <img width="599" alt="image" 
src="https://user-images.githubusercontent.com/14371345/197688226-c7e80f59-369b-4e4c-980d-8b77cae3bd56.png";>
   
   - Transform translation layer
     - Adapt to flink engine
     - Adapt to spark engine
     - Adapt to seatunnel engine
   
   <img width="721" alt="image" 
src="https://user-images.githubusercontent.com/14371345/197688271-8146086f-2b03-48ff-84b4-b4350998b0c6.png";>
   
   
   ### Task list
   
   Translation layer
   
   - [ ] #3145
   - [ ] Support transform-v2 for flink
   - [ ] Support transform-v2 for spark
   
   Transform
   - [ ] Substring transform
   - [ ] Convert date & time & timestamp transform
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org