There is only two ways that I can think to do about this.
1. Simple SSH connect pool 2. Control the number of task's concurrent ------------------ ???????? ------------------ ??????: "Hemin Wen"<[email protected]>; ????????: 2020??5??22??(??????) ????7:29 ??????: "dev"<[email protected]>; ????: Re: [Feature] Support SSH Task What other friends think about this discuss -------------------- DolphinScheduler(Incubator) Commtter Hemin Wen ?????? [email protected] -------------------- ?????? <[email protected]> ??2020??5??22?????? ????10:20?????? > Your are right, it can be solved the problem by modifying the > number of server ssh connections.&nbsp; > > The real reason is that I want to control the&nbsp; number of SSH > connections and reuse the SSH connection, Because the remote server is > managed by other team. > > Otherwise, I think that we can also control the number of parallel tasks > in a workflow. > > > It is a good idea that we can use the encapsulated ssh connection to get > clear > exception information&nbsp; > > > > > > > > ------------------&nbsp;????????&nbsp;------------------ > ??????:&nbsp;"Hemin Wen"<[email protected]&gt;; > ????????:&nbsp;2020??5??21??(??????) ????11:07 > ??????:&nbsp;"dev"<[email protected]&gt;; > > ????:&nbsp;Re: [Feature] Support SSH Task > > > > Thanks. > > About ssh connections pool, > 1. If the connection is not enough, Can it be solved by modifying the > number of server ssh connections. > 2. I think we have to look at whether a server's parallel ssh tasks will > reach the bottleneck of server connections. > > I think it is better to use the encapsulated ssh connection to get clear > exception information. > > -------------------- > DolphinScheduler(Incubator) Commtter > Hemin Wen&nbsp; ?????? > [email protected] > -------------------- > > > ?????? <[email protected]&gt; ??2020??5??20?????? ????11:29?????? > > &gt; Thanks for your advise and suggestions. It may has some exceptions if > I > &gt; implement SSH Task as you said. Now, let me explain my ideas. > &gt; > &gt; > &gt; e.g > &gt; > &gt; > &gt; My workflow DAG has 100 or more task, it's all ssh task. I > have&amp;nbsp; two > &gt; questions. > &gt; > &gt; > &gt; 1??We must limit SSH connection and reuse the SSH connections, > otherwise > &gt; the remote server will be refused. > &gt; > &gt; > &gt; 2??If we config all of these task, we have a lot of repeat work to do. > &gt; > &gt; > &gt; By the way, the way I'm currently verifying is I use local params to > &gt; config the remote server's host,user,password > &gt; > &gt; > &gt; ------------------&amp;nbsp;????????&amp;nbsp;------------------ > &gt; ??????:&amp;nbsp;"lidong dai"<[email protected]&amp;gt;; > &gt; ????????:&amp;nbsp;2020??5??20??(??????) ????10:56 > &gt; ??????:&amp;nbsp;"dev"<[email protected]&amp;gt;; > &gt; > &gt; ????:&amp;nbsp;Re: [Feature] Support SSH Task > &gt; > &gt; > &gt; > &gt; hi, I see you will Jsch, does his license compitable with apache v2 > &gt; license, I see the following words " > &gt; JSch 0.0.* was released under the GNU LGPL license. Later, we have > switched > &gt; over to a BSD-style license. > &gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt . you know, > LGPL > &gt; license not compitable with apache license. but BSD is ok , so the > version > &gt; of Jsch is key point > &gt; > &gt; second, Connection Pool is heavy resource, Please use carefully > &gt; > &gt; and I'm a little confused, why there must be add a new > Table,&amp;nbsp; other > &gt; task > &gt; types also no seperate table,&amp;nbsp; could you consider task_type > and > &gt; task_json > &gt; in t_ds_task_instance ? > &gt; > &gt; > &gt; > &gt; Best Regards > &gt; --------------- > &gt; DolphinScheduler(Incubator) PPMC > &gt; Lidong Dai ?????? > &gt; [email protected] > &gt; --------------- > &gt; > &gt; > &gt; ?????? <[email protected]&amp;gt; ??2020??5??20?????? ????9:57?????? > &gt; > &gt; &amp;gt; I want to describe my idea. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; First, we must design a host table that's named > t_ds_ssh_host, e.g > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; - id > &gt; &amp;gt; - name > &gt; &amp;gt; - ip / host > &gt; &amp;gt; - user > &gt; &amp;gt; - password > &gt; &amp;gt; - max_connection > &gt; &amp;gt; - create_time > &gt; &amp;gt; - update_time > &gt; &amp;gt; - user_id > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; Second, Shell can execute in worker server or remote host. > We can > &gt; choose a > &gt; &amp;gt; host in host's list. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; Third, when workflow running, we must maintain the remote > host's > &gt; &amp;gt; connection pool. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; Finally, when workflow finish, we must release the > connection pool. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; Why do we have to maintain a connection pool. Because it is > very easy > &gt; to > &gt; &amp;gt; get exceptions when it's has many SSH task. > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > ------------------&amp;amp;nbsp;????????&amp;amp;nbsp;------------------ > &gt; &amp;gt; ??????:&amp;amp;nbsp;"lidong dai"<[email protected] > &amp;amp;gt;; > &gt; &amp;gt; ????????:&amp;amp;nbsp;2020??5??20??(??????) ????7:12 > &gt; &amp;gt; ??????:&amp;amp;nbsp;"dev"<[email protected] > &amp;amp;gt;; > &gt; &amp;gt; > &gt; &amp;gt; ????:&amp;amp;nbsp;Re: [Feature] Support SSH Task > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; glad to hear that you will implement this feature > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; Best Regards > &gt; &amp;gt; --------------- > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC > &gt; &amp;gt; Lidong Dai ?????? > &gt; &amp;gt; [email protected] > &gt; &amp;gt; --------------- > &gt; &amp;gt; > &gt; &amp;gt; > &gt; &amp;gt; ?????? <[email protected]&amp;amp;gt; ??2020??5??20?????? ????3:47?????? > &gt; &amp;gt; > &gt; &amp;gt; &amp;amp;gt; My code is not perfect yet. I will write a > detailed design > &gt; document. > &gt; &amp;gt; Then > &gt; &amp;gt; &amp;amp;gt; I will realize this feature about our > discussion result. > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &gt; > ------------------&amp;amp;amp;nbsp;????????&amp;amp;amp;nbsp;------------------ > &gt; &amp;gt; &amp;amp;gt; ??????:&amp;amp;amp;nbsp;"wenhemin"< > [email protected]&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; ????????:&amp;amp;amp;nbsp;2020??5??18??(??????) ????7:50 > &gt; &amp;gt; &amp;amp;gt; ??????:&amp;amp;amp;nbsp;"??????"<[email protected] > &amp;amp;amp;gt;;"dev"< > &gt; &amp;gt; [email protected] > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; ????:&amp;amp;amp;nbsp;Re: [Feature] Support SSH > Task and Support > &gt; dummy task > &gt; &amp;gt; like airflow > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; Thanks for writing detailed documentation. I > think this is > &gt; also a > &gt; &amp;gt; missing > &gt; &amp;gt; &amp;amp;gt; feature of DS. > &gt; &amp;gt; &amp;amp;gt; About the extension point: > &gt; &amp;gt; &amp;amp;gt; 1.Can ssh tasks be merged into shell tasks. > Essentially, > &gt; they all > &gt; &amp;gt; execute > &gt; &amp;gt; &amp;amp;gt; shell commands. > &gt; &amp;gt; &amp;amp;gt; 2.About dummy task, DS has the function of > disable nodes, I > &gt; do n??t > &gt; &amp;gt; know if > &gt; &amp;gt; &amp;amp;gt; this requirement is met. > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; The script from AirFlow to Dolphin is great. > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ?? 2020??5??18????09:28???????? < > [email protected]&amp;amp;amp;gt; > &gt; ?????? > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; OK, 3Q! > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, I will ensure that open > source can use. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second, I think we must > discuss deeply. I write > &gt; a more > &gt; &amp;gt; detailed > &gt; &amp;gt; &amp;amp;gt; document. You can check the attachment. I also > send the > &gt; document to > &gt; &amp;gt; &amp;amp;gt; DaiLidong. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Third,&amp;amp;amp;nbsp; I'll > give you the error of > &gt; not using SSH > &gt; &amp;gt; connection pool. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ------------------ ???????? > ------------------ > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ??????: "wenhemin"< > [email protected]&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ????????: 2020??5??14??(??????) ????7:26 > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ??????: "??????"<[email protected] > &amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ????: Re: [Feature] Support SSH > Task and Support > &gt; dummy task > &gt; &amp;gt; like airflow > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Great! > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I think, Can ssh tasks be > merged into shell > &gt; tasks,&amp;amp;amp;nbsp; > &gt; &amp;gt; execute > &gt; &amp;gt; &amp;amp;gt; script locally or remotely, Configure on the > front end. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About ssh connect pool, I did > not find it > &gt; necessary to use > &gt; &amp;gt; the > &gt; &amp;gt; &amp;amp;gt; connection pool. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; BTW, Look at the code to > introduce additional > &gt; jar packages, > &gt; &amp;gt; You also > &gt; &amp;gt; &amp;amp;gt; need to ensure that open source can use the > license of this > &gt; jar > &gt; &amp;gt; package. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; ?? > 2020??5??14????16:20???????? < > &gt; [email protected] <mailto: > &gt; &amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; ?????? > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 1. The > priority between these tasks > &gt; is also depended > &gt; &amp;gt; on the > &gt; &amp;gt; &amp;amp;gt; dolphin DAG define. When the front task is not > finished, it > &gt; not > &gt; &amp;gt; execute > &gt; &amp;gt; &amp;amp;gt; next task. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 2. I extend > ssh task. I also use > &gt; local params to > &gt; &amp;gt; config ssh host, > &gt; &amp;gt; &amp;amp;gt; user and password. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; E.g: > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; public static > AbstractTask > &gt; &amp;gt; newTask(TaskExecutionContext > &gt; &amp;gt; &amp;amp;gt; taskExecutionContext, Logger logger) > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; throws > &gt; &amp;gt; IllegalArgumentException { > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; Boolean > &gt; enable = > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable"); > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; if > &gt; (enable != null > &gt; &amp;gt; &amp;amp;amp;amp;&amp;amp;amp;amp; enable == false ) { > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; return new > &gt; &amp;gt; &amp;amp;gt; DummyTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; } > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; switch > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) { > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case SHELL: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; ShellTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case > &gt; &amp;gt; PROCEDURE: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; ProcedureTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case SQL: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; SqlTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case MR: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; MapReduceTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case SPARK: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; SparkTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case FLINK: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; FlinkTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case PYTHON: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; PythonTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case HTTP: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; HttpTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case DATAX: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; DataxTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case SQOOP: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; SqoopTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; case SSH: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; return new > &gt; &amp;gt; &amp;amp;gt; SSHTask(taskExecutionContext, logger); > &gt; &amp;gt; &amp;amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; default: > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; logger.error("unsupport task > &gt; &amp;gt; &amp;amp;gt; type: {}", taskExecutionContext.getTaskType()); > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; throw new > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException("not support task > type"); > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; } > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; } > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 3. I am not > sure that it supports > &gt; window or not. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > ------------------ ???????? > &gt; ------------------ > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; ??????: > "wenhemin"<[email protected] > &gt; <mailto: > &gt; &amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; ????????: > 2020??5??14??(??????) ????3:46 > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; ??????: "??????"< > [email protected] > &gt; <mailto:[email protected] > &gt; &amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; ????: Re: > [Feature] Support SSH Task > &gt; and Support dummy > &gt; &amp;gt; task like > &gt; &amp;gt; &amp;amp;gt; airflow > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; Sorry, My > previous description is > &gt; not very clear. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; I want to ask > some questions: > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 1.How to > control the priority > &gt; between ssh tasks? > &gt; &amp;gt; There may be > &gt; &amp;gt; &amp;amp;gt; some ssh tasks that have been waiting for > execution. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 2.I understand > what you want to > &gt; solve is the problem > &gt; &amp;gt; of executing > &gt; &amp;gt; &amp;amp;gt; remote ssh scripts in batches. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; So, not > &gt; sure how to use this > &gt; &amp;gt; function. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; 3.I don't know > if this supports > &gt; windows system. > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ?? 2020??5??13????20:56???????? < > &gt; [email protected] > &gt; &amp;gt; <mailto: > &gt; &amp;gt; &amp;amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt; > ?????? > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; I use spin lock. Here > &gt; is my code. Of course > &gt; &amp;gt; , it's not > &gt; &amp;gt; &amp;amp;gt; perfect. I just do a test. To my surprise, it > is the result > &gt; of the > &gt; &amp;gt; &amp;amp;gt; execution is the same as the AirFlow > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; ???????????????????????????????????????????????????????????????????????????????????????????????????????? AirFlow > &gt; &amp;gt; &amp;amp;gt; ???????????? > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ------------------ ???????? > &gt; ------------------ > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ??????: "whm_777"< > &gt; [email protected] <mailto: > &gt; &amp;gt; [email protected] > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ????????: 2020??5??13??(??????) > &gt; ????7:21 > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ??????: "??????"< > &gt; [email protected] <mailto: > &gt; &amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ????: Re: [Feature] > &gt; Support SSH Task and > &gt; &amp;gt; Support dummy task > &gt; &amp;gt; &amp;amp;gt; like airflow > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; You can modify the > &gt; maximum number of linux > &gt; &amp;gt; ssh connections. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; If use ssh connection > &gt; pool, How to control > &gt; &amp;gt; the priority of > &gt; &amp;gt; &amp;amp;gt; ssh? > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ?? > &gt; 2020??5??13????18:01???????? < > &gt; &amp;gt; [email protected] <mailto: > &gt; &amp;gt; &amp;amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt; > ?????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; First 3Q?? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; I&amp;amp;amp;nbsp; use more than 100 task > &gt; &amp;gt; node. But SSH connections > &gt; &amp;gt; &amp;amp;gt; are limited. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > ??????????100??????????????????????SSH????????????????????????????????????????????????SSH??????????????????????????????DAG??????AirFlow???????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; <[email protected] > &gt; &amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ------------------ ???????? > &gt; &amp;gt; ------------------ > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ??????: > &gt; "whm_777"<[email protected] > &gt; &amp;gt; <mailto:[email protected] > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ????????: > &gt; 2020??5??13??(??????) ????5:50 > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ??????: "??????"< > &gt; [email protected] <mailto: > &gt; &amp;gt; [email protected] > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; ????: Re: > &gt; [Feature] Support SSH Task > &gt; &amp;gt; and Support dummy task > &gt; &amp;gt; &amp;amp;gt; like airflow > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; E.g. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; rtn_code=`ssh -o > &gt; &amp;gt; ServerAliveInterval=60 -p xxxx > &gt; &amp;gt; &amp;amp;gt; [email protected] <mailto: > [email protected]&amp;amp;amp;gt; > &gt; ??shell > &gt; &amp;gt; &amp;amp;gt; command&amp;amp;amp;nbsp; > &amp;amp;amp;gt;/dev/null > &gt; 2&amp;amp;amp;gt;&amp;amp;amp;amp;1; echo $?'` > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; if [ > &gt; "$rtn_code" -eq 0 ]; then > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; echo "????" > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; exit 0 > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; else > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; echo "????" > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > &gt; &amp;gt; exit 1 > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; fi > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; Batch shell > &gt; command is not supported. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; Multiple > &gt; servers can be split into > &gt; &amp;gt; multiple task nodes. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ?? 2020??5??13????17:40???????? < > &gt; &amp;gt; [email protected] <mailto: > &gt; &amp;gt; &amp;amp;gt; [email protected]&amp;amp;amp;gt;&amp;amp;amp;gt; > ?????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; Could you give me a > &gt; &amp;gt; example??3Q. ???????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; By the way, I have more than > &gt; &amp;gt; 100 tasks in one DAG. > &gt; &amp;gt; &amp;amp;gt; These tasks connect two other server to > execute. So SSH > &gt; tasks must > &gt; &amp;gt; have > &gt; &amp;gt; &amp;amp;gt; pool to manager. Now I use JSch and realize a > simple pool. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ??????????????????????????????????100???? SSH > &gt; &amp;gt; ?????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; SSH ?????????????????????????????????????????????????? JSch???????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ------------------ ???????? > &gt; &amp;gt; ------------------ > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ??????: "wenhemin"< > &gt; &amp;gt; [email protected] <mailto: > &gt; &amp;gt; &amp;amp;gt; [email protected] > &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ????????: 2020??5??13??(??????) ????5:24 > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ??????: "dev"< > &gt; &amp;gt; [email protected] <mailto: > &gt; &amp;gt; &amp;amp;gt; [email protected] > &amp;amp;amp;gt;&amp;amp;amp;gt;; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; ????: Re: [Feature] Support > &gt; &amp;gt; SSH Task and Support dummy > &gt; &amp;gt; &amp;amp;gt; task like airflow > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; The shell node is supports > &gt; &amp;gt; remote calling, and get > &gt; &amp;gt; &amp;amp;gt; the remote command result code. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; ?? > &gt; &amp;gt; 2020??5??13????15:16???????? <[email protected] > &gt; &amp;gt; &amp;amp;gt; <mailto:[email protected]&amp;amp;amp;gt;&amp;amp;amp;gt; > ?????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Dear ALL?? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Support Linux SSH > &gt; &amp;gt; Task ???? Linux SSH ???? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; ???????????????????????????????????????????????????????? Shell ??????Shell > &gt; &amp;gt; &amp;amp;gt; ???????????????????????????????????? Worker > ???????????????????????????????????????????????????? Shell > &gt; &amp;gt; &amp;amp;gt; ?????????????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; For example, in my > &gt; &amp;gt; project, the workflow's tasks > &gt; &amp;gt; &amp;amp;gt; want to execute shell scripts where are in > different server's > &gt; &amp;gt; different > &gt; &amp;gt; &amp;amp;gt; directory. When worker execute these shell > scripts, it must > &gt; use the > &gt; &amp;gt; same > &gt; &amp;gt; &amp;amp;gt; user to login these server. Also, the worker > can get the > &gt; executing > &gt; &amp;gt; state of > &gt; &amp;gt; &amp;amp;gt; these server. We can config these server 's > host,user and > &gt; password. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; SSH Task is very > &gt; &amp;gt; useful for most user SSH > &gt; &amp;gt; &amp;amp;gt; ???????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; ?????????????????????? Shell > &gt; &amp;gt; &amp;amp;gt; ???????????????????????????????????????????????????????????????? Worker?????????? > Worker > &gt; &amp;gt; &amp;amp;gt; ?????????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; In > &gt; &amp;gt; dolphinscheduler, the most executing tasks > &gt; &amp;gt; &amp;amp;gt; are in different servers who are not workers. > These servers > &gt; also have > &gt; &amp;gt; their > &gt; &amp;gt; &amp;amp;gt; different fixed services. We just have to pass > different > &gt; parameters to > &gt; &amp;gt; &amp;amp;gt; schedule these shell scripts to execute. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Python has a module > &gt; &amp;gt; to execute ssh script Python > &gt; &amp;gt; &amp;amp;gt; ??????????????????????????SSH Shell ???? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Python > &gt; &amp;gt; ??????????????????????SSH Shell??????????????????????paramiko?? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Python has a module > &gt; &amp;gt; that can execute SSH Shell > &gt; &amp;gt; &amp;amp;gt; script. It's paramiko. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Others ???????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; ???????????????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; I found this > &gt; &amp;gt; described in previous feature, but > &gt; &amp;gt; &amp;amp;gt; it was relatively simple. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Feature URL > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; ???????????? Shell Task > &gt; &amp;gt; &amp;amp;gt; ?????????????????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; In addition, it is > &gt; &amp;gt; very inconvenient for me to > &gt; &amp;gt; &amp;amp;gt; perform remote tasks through Shell Task. Here > is my script. > &gt; I don't > &gt; &amp;gt; know if > &gt; &amp;gt; &amp;amp;gt; there's a better way. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; sshpass -p > &gt; &amp;gt; 'password' ssh user@host echo 'ssh > &gt; &amp;gt; &amp;amp;gt; success' echo 'Hello World' > -&amp;amp;amp;amp;gt; > &gt; &amp;gt; /home/dolphinscheduler/test/hello.txt > &gt; &amp;gt; &amp;amp;gt; echo 'end' > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Support dummy task > &gt; &amp;gt; like airflow ?????? Airflow > &gt; &amp;gt; &amp;amp;gt; ???????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; ???????????????????????????????? > &gt; &amp;gt; DAG ??????DAG > &gt; &amp;gt; &amp;amp;gt; > &gt; ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > &gt; &amp;gt; Dummy > &gt; &amp;gt; &amp;amp;gt; > &gt; &amp;gt; > &gt; > Task????????????????????????????????????????????????????????????????????????AirFlow??????????????DummyOperator???????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; For example, in my > &gt; &amp;gt; project, it has a productized > &gt; &amp;gt; &amp;amp;gt; DAG file. The file contains different modules, > some of which > &gt; are > &gt; &amp;gt; &amp;amp;gt; interdependent and some of which are not. When > customers > &gt; purchase > &gt; &amp;gt; different > &gt; &amp;gt; &amp;amp;gt; modules, we need to set some tasks as dummy > tasks, which > &gt; some modules > &gt; &amp;gt; are > &gt; &amp;gt; &amp;amp;gt; not purchased and the purchased module is not > dependent. > &gt; Because of > &gt; &amp;gt; this > &gt; &amp;gt; &amp;amp;gt; setting, these dummy tasks are actually not > executed. The > &gt; benefits of > &gt; &amp;gt; this > &gt; &amp;gt; &amp;amp;gt; setup are product unity and diagram integrity. > In airflow, > &gt; these task > &gt; &amp;gt; &amp;amp;gt; execute by dummy operator. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; ** Realize ????????** > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Dummy Task > &gt; &amp;gt; ???????????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; dummy ?????????????????????????? Dummy Task?? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; Dummy Task is easy > &gt; &amp;gt; to realize, but it need to > &gt; &amp;gt; &amp;amp;gt; use with other different tasks. When the task's > executed > &gt; type is set > &gt; &amp;gt; to > &gt; &amp;gt; &amp;amp;gt; dummy type, the task are executed as a dummy > task and the > &gt; real task > &gt; &amp;gt; is not > &gt; &amp;gt; &amp;amp;gt; executed. > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > ????????????????????????????????????Fork?????????????????????????????????????????????????????????? > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;amp;amp;gt; By the way??I > &gt; &amp;gt; already realize these two&amp;amp;amp;amp;nbsp; > &gt; &amp;gt; &amp;amp;gt; features in my fork > branch.&amp;amp;amp;amp;nbsp;Whether the > &gt; follow-up > &gt; &amp;gt; release can be > &gt; &amp;gt; &amp;amp;gt; supported > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; > &gt; > <SSHClient.java&amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > <??????????????Dolphin????????????.pdf&amp;amp;amp;gt;
