Thanks for your advise and suggestions. It may has some exceptions if I implement SSH Task as you said. Now, let me explain my ideas.
e.g My workflow DAG has 100 or more task, it's all ssh task. I have two questions. 1??We must limit SSH connection and reuse the SSH connections, otherwise the remote server will be refused. 2??If we config all of these task, we have a lot of repeat work to do. By the way, the way I'm currently verifying is I use local params to config the remote server's host,user,password ------------------ ???????? ------------------ ??????: "lidong dai"<[email protected]>; ????????: 2020??5??20??(??????) ????10:56 ??????: "dev"<[email protected]>; ????: Re: [Feature] Support SSH Task hi, I see you will Jsch, does his license compitable with apache v2 license, I see the following words " JSch 0.0.* was released under the GNU LGPL license. Later, we have switched over to a BSD-style license. " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, LGPL license not compitable with apache license. but BSD is ok , so the version of Jsch is key point second, Connection Pool is heavy resource, Please use carefully and I'm a little confused, why there must be add a new Table, other task types also no seperate table, could you consider task_type and task_json in t_ds_task_instance ? Best Regards --------------- DolphinScheduler(Incubator) PPMC Lidong Dai ?????? [email protected] --------------- ?????? <[email protected]> ??2020??5??20?????? ????9:57?????? > I want to describe my idea. > > > First, we must design a host table that's named t_ds_ssh_host, e.g > > > - id > - name > - ip / host > - user > - password > - max_connection > - create_time > - update_time > - user_id > > > Second, Shell can execute in worker server or remote host. We can choose a > host in host's list. > > > Third, when workflow running, we must maintain the remote host's > connection pool. > > > Finally, when workflow finish, we must release the connection pool. > > > Why do we have to maintain a connection pool. Because it is very easy to > get exceptions when it's has many SSH task. > > > > > ------------------&nbsp;????????&nbsp;------------------ > ??????:&nbsp;"lidong dai"<[email protected]&gt;; > ????????:&nbsp;2020??5??20??(??????) ????7:12 > ??????:&nbsp;"dev"<[email protected]&gt;; > > ????:&nbsp;Re: [Feature] Support SSH Task > > > > glad to hear that you will implement this feature > > > > Best Regards > --------------- > DolphinScheduler(Incubator) PPMC > Lidong Dai ?????? > [email protected] > --------------- > > > ?????? <[email protected]&gt; ??2020??5??20?????? ????3:47?????? > > &gt; My code is not perfect yet. I will write a detailed design document. > Then > &gt; I will realize this feature about our discussion result. > &gt; > &gt; > &gt; ------------------&amp;nbsp;????????&amp;nbsp;------------------ > &gt; ??????:&amp;nbsp;"wenhemin"<[email protected]&amp;gt;; > &gt; ????????:&amp;nbsp;2020??5??18??(??????) ????7:50 > &gt; ??????:&amp;nbsp;"??????"<[email protected]&amp;gt;;"dev"< > [email protected] > &gt; &amp;gt;; > &gt; > &gt; ????:&amp;nbsp;Re: [Feature] Support SSH Task and Support dummy task > like airflow > &gt; > &gt; > &gt; > &gt; Thanks for writing detailed documentation. I think this is also a > missing > &gt; feature of DS. > &gt; About the extension point: > &gt; 1.Can ssh tasks be merged into shell tasks. Essentially, they all > execute > &gt; shell commands. > &gt; 2.About dummy task, DS has the function of disable nodes, I do n??t > know if > &gt; this requirement is met. > &gt; > &gt; The script from AirFlow to Dolphin is great. > &gt; > &gt; &amp;gt; ?? 2020??5??18????09:28???????? <[email protected]&amp;gt; ?????? > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; OK, 3Q! > &gt; &amp;gt; > &gt; &amp;gt; First, I will ensure that open source can use. > &gt; &amp;gt; > &gt; &amp;gt; Second, I think we must discuss deeply. I write a more > detailed > &gt; document. You can check the attachment. I also send the document to > &gt; DaiLidong. > &gt; &amp;gt; > &gt; &amp;gt; Third,&amp;nbsp; I'll give you the error of not using SSH > connection pool. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; ------------------ ???????? ------------------ > &gt; &amp;gt; ??????: "wenhemin"<[email protected]&amp;gt;; > &gt; &amp;gt; ????????: 2020??5??14??(??????) ????7:26 > &gt; &amp;gt; ??????: "??????"<[email protected]&amp;gt;; > &gt; &amp;gt; ????: Re: [Feature] Support SSH Task and Support dummy task > like airflow > &gt; &amp;gt; > &gt; &amp;gt; Great! > &gt; &amp;gt; I think, Can ssh tasks be merged into shell tasks,&amp;nbsp; > execute > &gt; script locally or remotely, Configure on the front end. > &gt; &amp;gt; About ssh connect pool, I did not find it necessary to use > the > &gt; connection pool. > &gt; &amp;gt; > &gt; &amp;gt; BTW, Look at the code to introduce additional jar packages, > You also > &gt; need to ensure that open source can use the license of this jar > package. > &gt; &amp;gt; > &gt; &amp;gt;&amp;gt; ?? 2020??5??14????16:20???????? <[email protected] <mailto: > [email protected]&amp;gt;&amp;gt; > &gt; ?????? > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; 1. The priority between these tasks is also depended > on the > &gt; dolphin DAG define. When the front task is not finished, it not > execute > &gt; next task. > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; 2. I extend ssh task. I also use local params to > config ssh host, > &gt; user and password. > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; E.g: > &gt; &amp;gt;&amp;gt; public static AbstractTask > newTask(TaskExecutionContext > &gt; taskExecutionContext, Logger logger) > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; throws > IllegalArgumentException { > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; Boolean enable = > &gt; > JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable"); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; if (enable != null > &amp;amp;&amp;amp; enable == false ) { > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; return new > &gt; DummyTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; } > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; switch > &gt; > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) { > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SHELL: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; ShellTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case > PROCEDURE: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; ProcedureTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SQL: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; SqlTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case MR: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; MapReduceTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SPARK: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; SparkTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case FLINK: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; FlinkTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case PYTHON: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; PythonTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case HTTP: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; HttpTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case DATAX: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; DataxTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SQOOP: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; SqoopTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; case SSH: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > return new > &gt; SSHTask(taskExecutionContext, logger); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; default: > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > logger.error("unsupport task > &gt; type: {}", taskExecutionContext.getTaskType()); > &gt; > &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > throw new > &gt; IllegalArgumentException("not support task type"); > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; } > &gt; &amp;gt;&amp;gt; } > &gt; &amp;gt;&amp;gt; 3. I am not sure that it supports window or not. > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; ------------------ ???????? ------------------ > &gt; &amp;gt;&amp;gt; ??????: "wenhemin"<[email protected] <mailto: > [email protected]&amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt; ????????: 2020??5??14??(??????) ????3:46 > &gt; &amp;gt;&amp;gt; ??????: "??????"<[email protected] <mailto:[email protected] > &amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt; ????: Re: [Feature] Support SSH Task and Support dummy > task like > &gt; airflow > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; Sorry, My previous description is not very clear. > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt; I want to ask some questions: > &gt; &amp;gt;&amp;gt; 1.How to control the priority between ssh tasks? > There may be > &gt; some ssh tasks that have been waiting for execution. > &gt; &amp;gt;&amp;gt; 2.I understand what you want to solve is the problem > of executing > &gt; remote ssh scripts in batches. > &gt; &amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp; So, not sure how to use this > function. > &gt; &amp;gt;&amp;gt; 3.I don't know if this supports windows system. > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; ?? 2020??5??13????20:56???????? <[email protected] > <mailto: > &gt; [email protected]&amp;gt;&amp;gt; ?????? > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; I use spin lock. Here is my code. Of course > , it's not > &gt; perfect. I just do a test. To my surprise, it is the result of the > &gt; execution is the same as the AirFlow > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > ???????????????????????????????????????????????????????????????????????????????????????????????????????? AirFlow > &gt; ???????????? > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; ------------------ ???????? ------------------ > &gt; &amp;gt;&amp;gt;&amp;gt; ??????: "whm_777"<[email protected] <mailto: > [email protected] > &gt; &amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt; ????????: 2020??5??13??(??????) ????7:21 > &gt; &amp;gt;&amp;gt;&amp;gt; ??????: "??????"<[email protected] <mailto: > [email protected]&amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt; ????: Re: [Feature] Support SSH Task and > Support dummy task > &gt; like airflow > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; You can modify the maximum number of linux > ssh connections. > &gt; &amp;gt;&amp;gt;&amp;gt; If use ssh connection pool, How to control > the priority of > &gt; ssh? > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ?? 2020??5??13????18:01???????? < > [email protected] <mailto: > &gt; [email protected]&amp;gt;&amp;gt; ?????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; First 3Q?? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; I&amp;nbsp; use more than 100 task > node. But SSH connections > &gt; are limited. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; > ??????????100??????????????????????SSH????????????????????????????????????????????????SSH??????????????????????????????DAG??????AirFlow???????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; <[email protected] > &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ------------------ ???????? > ------------------ > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ??????: "whm_777"<[email protected] > <mailto:[email protected] > &gt; &amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ????????: 2020??5??13??(??????) ????5:50 > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ??????: "??????"<[email protected] <mailto: > [email protected] > &gt; &amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; ????: Re: [Feature] Support SSH Task > and Support dummy task > &gt; like airflow > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; E.g. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; rtn_code=`ssh -o > ServerAliveInterval=60 -p xxxx > &gt; [email protected] <mailto:[email protected]&amp;gt; ??shell > &gt; command&amp;nbsp; &amp;gt;/dev/null 2&amp;gt;&amp;amp;1; echo $?'` > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; if [ "$rtn_code" -eq 0 ]; then > &gt; > &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > echo "????" > &gt; > &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > exit 0 > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; else > &gt; > &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > echo "????" > &gt; > &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp; > exit 1 > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; fi > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Batch shell command is not supported. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; Multiple servers can be split into > multiple task nodes. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ?? 2020??5??13????17:40???????? < > [email protected] <mailto: > &gt; [email protected]&amp;gt;&amp;gt; ?????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; Could you give me a > example??3Q. ???????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; By the way, I have more than > 100 tasks in one DAG. > &gt; These tasks connect two other server to execute. So SSH tasks must > have > &gt; pool to manager. Now I use JSch and realize a simple pool. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ??????????????????????????????????100???? SSH > ?????????????????????????????????????????????????? > &gt; SSH ?????????????????????????????????????????????????? JSch???????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ------------------ ???????? > ------------------ > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ??????: "wenhemin"< > [email protected] <mailto: > &gt; [email protected]&amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ????????: 2020??5??13??(??????) ????5:24 > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ??????: "dev"< > [email protected] <mailto: > &gt; [email protected]&amp;gt;&amp;gt;; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; ????: Re: [Feature] Support > SSH Task and Support dummy > &gt; task like airflow > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; The shell node is supports > remote calling, and get > &gt; the remote command result code. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ?? > 2020??5??13????15:16???????? <[email protected] > &gt; <mailto:[email protected]&amp;gt;&amp;gt; ?????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dear ALL?? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Support Linux SSH > Task ???? Linux SSH ???? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > ???????????????????????????????????????????????????????? Shell ??????Shell > &gt; ???????????????????????????????????? Worker ???????????????????????????????????????????????????? Shell > &gt; ?????????????????????????????????????????????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; For example, in my > project, the workflow's tasks > &gt; want to execute shell scripts where are in different server's > different > &gt; directory. When worker execute these shell scripts, it must use the > same > &gt; user to login these server. Also, the worker can get the executing > state of > &gt; these server. We can config these server 's host,user and password. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; SSH Task is very > useful for most user SSH > &gt; ???????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ?????????????????????? Shell > &gt; ???????????????????????????????????????????????????????????????? Worker?????????? Worker > &gt; ?????????????????????????????????????????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; In > dolphinscheduler, the most executing tasks > &gt; are in different servers who are not workers. These servers also have > their > &gt; different fixed services. We just have to pass different parameters to > &gt; schedule these shell scripts to execute. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python has a module > to execute ssh script Python > &gt; ??????????????????????????SSH Shell ???? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python > ??????????????????????SSH Shell??????????????????????paramiko?? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Python has a module > that can execute SSH Shell > &gt; script. It's paramiko. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Others ???????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > ???????????????????????????????????????????????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; I found this > described in previous feature, but > &gt; it was relatively simple. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Feature URL > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ???????????? Shell Task > &gt; ?????????????????????????????????????????????????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; In addition, it is > very inconvenient for me to > &gt; perform remote tasks through Shell Task. Here is my script. I don't > know if > &gt; there's a better way. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; sshpass -p > 'password' ssh user@host echo 'ssh > &gt; success' echo 'Hello World' -&amp;amp;gt; > /home/dolphinscheduler/test/hello.txt > &gt; echo 'end' > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Support dummy task > like airflow ?????? Airflow > &gt; ???????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ???????????????????????????????? > DAG ??????DAG > &gt; ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > Dummy > &gt; > Task????????????????????????????????????????????????????????????????????????AirFlow??????????????DummyOperator???????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; For example, in my > project, it has a productized > &gt; DAG file. The file contains different modules, some of which are > &gt; interdependent and some of which are not. When customers purchase > different > &gt; modules, we need to set some tasks as dummy tasks, which some modules > are > &gt; not purchased and the purchased module is not dependent. Because of > this > &gt; setting, these dummy tasks are actually not executed. The benefits of > this > &gt; setup are product unity and diagram integrity. In airflow, these task > &gt; execute by dummy operator. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; ** Realize ????????** > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dummy Task > ???????????????????????????????????????????????????????????????? > &gt; dummy ?????????????????????????? Dummy Task?? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; Dummy Task is easy > to realize, but it need to > &gt; use with other different tasks. When the task's executed type is set > to > &gt; dummy type, the task are executed as a dummy task and the real task > is not > &gt; executed. > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; ????????????????????????????????????Fork?????????????????????????????????????????????????????????? > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; &amp;gt; By the way??I > already realize these two&amp;amp;nbsp; > &gt; features in my fork branch.&amp;amp;nbsp;Whether the follow-up > release can be > &gt; supported > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > &gt; &amp;gt;&amp;gt;&amp;gt; > <SSHClient.java&amp;gt;<SSHPool.java&amp;gt;<SSHTask.java&amp;gt; > &gt; &amp;gt;&amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; <??????????????Dolphin????????????.pdf&amp;gt;
