Issue Name: Expand the data source center into a connection centerIssue
Background:1、The DolphinScheduler has a component of the Datasource Center
that is used to manage external connections for SQL tasks, such as MySQL,
hive, spark, etc.2、But not only SQL tasks, but also some other
DolphinScheduler task plugins require external connections, such as AWS EMR
tasks, Zeppelin tasks, K8S tasks, etc. We can enrich the scenarios that
require the Datasource Center to manage connections, especially those
external systems with credentials, and upgrade them to the Connection
Center.Related Issues:

[Feature] Add connection center feature for DS #10283
<https://github.com/apache/dolphinscheduler/issues/10283>

### Issue Target:
1、Change the name of the Datasource Center to Connection Center.2、Reconstruct
some AWS EMR, Zeppelin, K8S, and Sagemaker task plugins to facilitate user
management of external connections in the connection center.3、Cluster
Management and K8S Namespace Management in the security center are removed
because managing K8S clusters is not the job of Big data orchestration
tools. Users can configure K8S connections for K8S task plugins in the
connection center.Design scheme (taking Zeppelin as an example):1、Support
for Zepplein connections in the Connection Center

   -

   Add the zeppelin module to the datasource-plugin module, and add key
   classes such as ZeppelinConnectionParam, ZeppelinDataSourceParamDTO,
   ZeppelinDataSourceProcessor, ZeppelinDataSourceChannel,
   ZeppelinDataSourceChannelFactory, and ZeppelinDataSourceClient。

   *Several key base class designs that need to be discussed*:
   ZeppelinConnectionParam:

    @Data
    @JsonInclude(JsonInclude.Include.NON_NULL)
    public class ZeppelinConnectionParam implements ConnectionParam {

        protected String user;

        protected String password;

        protected String host;

        protected int port = 8080;
    }

   ZeppelinDataSourceProcessor:

   This class implements the DataSourceProcessorinterface and rewrites all
   methods inside, including the default testConnection method, which is
   used to detect whether Zepplein can connect.

    @Override
        public Result<Object> checkConnection(DbType type,
ConnectionParam connectionParam) {
            Result<Object> result = new Result<>();
            // something
            if (type == DbType.ZEPPELIN) {
                DataSourceProcessor zeppelinDataSourceProcessor =
DataSourceUtils.getDatasourceProcessor(type);
                if
(zeppelinDataSourceProcessor.testConnection(connectionParam)) {
                    putMsg(result, Status.SUCCESS);
                } else {
                    putMsg(result, Status.CONNECT_DATASOURCE_FAILURE);
                }
                return result;
            }
            // something
        }

   -

   Front end design of zeppelin connection:

   [image: image-20230613113210281]

2、Support the selection of online Zeppelin connections in the Zeppelintask

   -

   Zeppelin task front-end interface changes (removed input for username
   and password)

   [image: image-20230613113422370]
   -

   Add the following parameters to the model of tasks/use zeppelin.ts

    type: 'ZEPPELIN',
    displayRows: 10,
    restEndpoint: '',
    username: '',
    password: ''

   The final model is:

    const model = reactive({
        name: '',
        taskType: 'ZEPPELIN',
        flag: 'YES',
        description: '',
        timeoutFlag: false,
        localParams: [],
        environmentCode: null,
        failRetryInterval: 1,
        failRetryTimes: 0,
        workerGroup: 'default',
        delayTime: 0,
        timeout: 30,
        timeoutNotifyStrategy: ['WARN'],
        type: 'ZEPPELIN',
        displayRows: 10,
        restEndpoint: '',
        username: '',
        password: ''
      } as INodeData)

   -

   Before the use-zeppelin function return, fill in the username, password,
   and restEndpoint parameters with the parameters of the connection
   selected in the drop-down box.

   Obtain the selected connection ID, query its detailed information in the
   database based on the ID, extract the username, password, restEndpoint
   in the information, and password should be encrypted and may need to be
   decrypted.

    return {
        json: [
          Fields.useName(from),
          ...Fields.useTaskDefinition({ projectCode, from, readonly,
data, model }),
          Fields.useRunFlag(),
          Fields.useCache(),
          Fields.useDescription(),
          Fields.useTaskPriority(),
          Fields.useWorkerGroup(),
          Fields.useEnvironmentName(model, !data?.id),
          ...Fields.useTaskGroup(model, projectCode),
          ...Fields.useFailed(),
          Fields.useDelayTime(model),
          ...Fields.useTimeoutAlarm(model),
          ...Fields.useZeppelin(model),
          Fields.usePreTasks()
        ] as IJsonItem[],
        model
      }

Reply via email to