Good idea! This can control the number of concurrency for each task type.
-------------------- DolphinScheduler(Incubator) Commtter Hemin Wen 温合民 [email protected] -------------------- lidong dai <[email protected]> 于2020年5月26日周二 下午6:39写道： > good idea, I think "Control the number of task's concurrent" may be have > some other methods, for now, we don't have this feature, but when I > communicate with the team of China Unicom， they provide a new solve > solution： > we can add a task concurrent page, you can set a variable value like > 'ETL_JOB' =10 and so on, then give the task type add a option for choosing > ETL_JOB or other variable, and there may be many task also choose ETL_JOB > variable. when these tasks begin to running at the same time, the Master > Server will check whether a task begin to run or wait(if the running tasks > num = 10) > > > Best egards > --------------- > DolphinScheduler(Incubator) PPMC > Lidong Dai 代立冬 > [email protected] > --------------- > > > 裴龙武 <[email protected]> 于2020年5月26日周二 下午2:34写道： > > > There is only two ways that I can think to do about this. > > > > > > 1. Simple SSH connect pool > > 2. Control the number of task's concurrent > > > > > > ------------------ 原始邮件 ------------------ > > 发件人: "Hemin Wen"<[email protected]>; > > 发送时间: 2020年5月22日(星期五) 晚上7:29 > > 收件人: "dev"<[email protected]>; > > > > 主题: Re: [Feature] Support SSH Task > > > > > > > > What other friends think about this discuss > > > > > > -------------------- > > DolphinScheduler(Incubator) Commtter > > Hemin Wen 温合民 > > [email protected] > > -------------------- > > > > > > 裴龙武 <[email protected]> 于2020年5月22日周五 上午10:20写道： > > > > > Your are right, it can be solved the problem by modifying the > > > number of server ssh connections.&nbsp; > > > > > > The real reason is that I want to control the&nbsp; number of > SSH > > > connections and reuse the SSH connection, Because the remote server > is > > > managed by other team. > > > > > > Otherwise, I think that we can also control the number of parallel > > tasks > > > in a workflow. > > > > > > > > > It is a good idea that we can use the encapsulated ssh connection to > > get > > > clear > > > exception information&nbsp; > > > > > > > > > > > > > > > > > > > > > > > > ------------------&nbsp;原始邮件&nbsp;------------------ > > > 发件人:&nbsp;"Hemin Wen"<[email protected]&gt;; > > > 发送时间:&nbsp;2020年5月21日(星期四) 中午11:07 > > > 收件人:&nbsp;"dev"<[email protected]&gt;; > > > > > > 主题:&nbsp;Re: [Feature] Support SSH Task > > > > > > > > > > > > Thanks. > > > > > > About ssh connections pool, > > > 1. If the connection is not enough, Can it be solved by modifying > the > > > number of server ssh connections. > > > 2. I think we have to look at whether a server's parallel ssh tasks > > will > > > reach the bottleneck of server connections. > > > > > > I think it is better to use the encapsulated ssh connection to get > > clear > > > exception information. > > > > > > -------------------- > > > DolphinScheduler(Incubator) Commtter > > > Hemin Wen&nbsp; 温合民 > > > [email protected] > > > -------------------- > > > > > > > > > 裴龙武 <[email protected]&gt; 于2020年5月20日周三 下午11:29写道： > > > > > > &gt; Thanks for your advise and suggestions. It may has some > > exceptions if > > > I > > > &gt; implement SSH Task as you said. Now, let me explain my > ideas. > > > &gt; > > > &gt; > > > &gt; e.g > > > &gt; > > > &gt; > > > &gt; My workflow DAG has 100 or more task, it's all ssh task. I > > > have&amp;nbsp; two > > > &gt; questions. > > > &gt; > > > &gt; > > > &gt; 1、We must limit SSH connection and reuse the SSH > connections, > > > otherwise > > > &gt; the remote server will be refused. > > > &gt; > > > &gt; > > > &gt; 2、If we config all of these task, we have a lot of repeat > > work to do. > > > &gt; > > > &gt; > > > &gt; By the way, the way I'm currently verifying is I use local > > params to > > > &gt; config the remote server's host,user,password > > > &gt; > > > &gt; > > > &gt; > > ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------ > > > &gt; 发件人:&amp;nbsp;"lidong dai"<[email protected] > > &amp;gt;; > > > &gt; 发送时间:&amp;nbsp;2020年5月20日(星期三) 晚上10:56 > > > &gt; 收件人:&amp;nbsp;"dev"<[email protected] > > &amp;gt;; > > > &gt; > > > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task > > > &gt; > > > &gt; > > > &gt; > > > &gt; hi, I see you will Jsch, does his license compitable with > > apache v2 > > > &gt; license, I see the following words " > > > &gt; JSch 0.0.* was released under the GNU LGPL license. Later, > > we have > > > switched > > > &gt; over to a BSD-style license. > > > &gt; " on https://github.com/is/jsch/blob/master/LICENSE.txt . > > you know, > > > LGPL > > > &gt; license not compitable with apache license. but BSD is ok , > > so the > > > version > > > &gt; of Jsch is key point > > > &gt; > > > &gt; second, Connection Pool is heavy resource, Please use > > carefully > > > &gt; > > > &gt; and I'm a little confused, why there must be add a new > > > Table,&amp;nbsp; other > > > &gt; task > > > &gt; types also no seperate table,&amp;nbsp; could you > > consider task_type > > > and > > > &gt; task_json > > > &gt; in t_ds_task_instance ? > > > &gt; > > > &gt; > > > &gt; > > > &gt; Best Regards > > > &gt; --------------- > > > &gt; DolphinScheduler(Incubator) PPMC > > > &gt; Lidong Dai 代立冬 > > > &gt; [email protected] > > > &gt; --------------- > > > &gt; > > > &gt; > > > &gt; 裴龙武 <[email protected]&amp;gt; 于2020年5月20日周三 下午9:57写道： > > > &gt; > > > &gt; &amp;gt; I want to describe my idea. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; First, we must design a host table that's > named > > > t_ds_ssh_host, e.g > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; - id > > > &gt; &amp;gt; - name > > > &gt; &amp;gt; - ip / host > > > &gt; &amp;gt; - user > > > &gt; &amp;gt; - password > > > &gt; &amp;gt; - max_connection > > > &gt; &amp;gt; - create_time > > > &gt; &amp;gt; - update_time > > > &gt; &amp;gt; - user_id > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Second, Shell can execute in worker server or > > remote host. > > > We can > > > &gt; choose a > > > &gt; &amp;gt; host in host's list. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Third, when workflow running, we must maintain > > the remote > > > host's > > > &gt; &amp;gt; connection pool. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Finally, when workflow finish, we must release > > the > > > connection pool. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Why do we have to maintain a connection pool. > > Because it is > > > very easy > > > &gt; to > > > &gt; &amp;gt; get exceptions when it's has many SSH task. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > > > > ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------ > > > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"< > > [email protected] > > > &amp;amp;gt;; > > > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12 > > > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"< > > [email protected] > > > &amp;amp;gt;; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH > > Task > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; glad to hear that you will implement this > > feature > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Best Regards > > > &gt; &amp;gt; --------------- > > > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC > > > &gt; &amp;gt; Lidong Dai 代立冬 > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; --------------- > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 裴龙武 <[email protected]&amp;amp;gt; > > 于2020年5月20日周三 下午3:47写道： > > > &gt; &amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; My code is not perfect yet. I > > will write a > > > detailed design > > > &gt; document. > > > &gt; &amp;gt; Then > > > &gt; &amp;gt; &amp;amp;gt; I will realize this feature > > about our > > > discussion result. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > 发件人:&amp;amp;amp;nbsp;"wenhemin"< > > > [email protected]&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > 发送时间:&amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50 > > > &gt; &amp;gt; &amp;amp;gt; > > 收件人:&amp;amp;amp;nbsp;"裴龙武"<[email protected] > > > &amp;amp;amp;gt;;"dev"< > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re: > > [Feature] Support SSH > > > Task and Support > > > &gt; dummy task > > > &gt; &amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Thanks for writing detailed > > documentation. I > > > think this is > > > &gt; also a > > > &gt; &amp;gt; missing > > > &gt; &amp;gt; &amp;amp;gt; feature of DS. > > > &gt; &amp;gt; &amp;amp;gt; About the extension point: > > > &gt; &amp;gt; &amp;amp;gt; 1.Can ssh tasks be merged > into > > shell tasks. > > > Essentially, > > > &gt; they all > > > &gt; &amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; shell commands. > > > &gt; &amp;gt; &amp;amp;gt; 2.About dummy task, DS has > the > > function of > > > disable nodes, I > > > &gt; do n’t > > > &gt; &amp;gt; know if > > > &gt; &amp;gt; &amp;amp;gt; this requirement is met. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; The script from AirFlow to > > Dolphin is great. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 在 > > 2020年5月18日，09:28，裴龙武 < > > > [email protected]&amp;amp;amp;gt; > > > &gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; OK, 3Q! > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; First, I > > will ensure that open > > > source can use. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Second, > I > > think we must > > > discuss deeply. I write > > > &gt; a more > > > &gt; &amp;gt; detailed > > > &gt; &amp;gt; &amp;amp;gt; document. You can check the > > attachment. I also > > > send the > > > &gt; document to > > > &gt; &amp;gt; &amp;amp;gt; DaiLidong. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > Third,&amp;amp;amp;nbsp; I'll > > > give you the error of > > > &gt; not using SSH > > > &gt; &amp;gt; connection pool. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > ------------------ 原始邮件 > > > ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发件人: > > "wenhemin"< > > > [email protected]&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 发送时间: > > 2020年5月14日(星期四) 晚上7:26 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 收件人: > > "裴龙武"<[email protected] > > > &amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 主题: Re: > > [Feature] Support SSH > > > Task and Support > > > &gt; dummy task > > > &gt; &amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Great! > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I think, > > Can ssh tasks be > > > merged into shell > > > &gt; tasks,&amp;amp;amp;nbsp; > > > &gt; &amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; script locally or remotely, > > Configure on the > > > front end. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About > ssh > > connect pool, I did > > > not find it > > > &gt; necessary to use > > > &gt; &amp;gt; the > > > &gt; &amp;gt; &amp;amp;gt; connection pool. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; BTW, > Look > > at the code to > > > introduce additional > > > &gt; jar packages, > > > &gt; &amp;gt; You also > > > &gt; &amp;gt; &amp;amp;gt; need to ensure that open > > source can use the > > > license of this > > > &gt; jar > > > &gt; &amp;gt; package. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 在 > > > 2020年5月14日，16:20，裴龙武 < > > > &gt; [email protected] <mailto: > > > &gt; &amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 1. The > > > priority between these tasks > > > &gt; is also depended > > > &gt; &amp;gt; on the > > > &gt; &amp;gt; &amp;amp;gt; dolphin DAG define. When the > > front task is not > > > finished, it > > > &gt; not > > > &gt; &amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; next task. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 2. I extend > > > ssh task. I also use > > > &gt; local params to > > > &gt; &amp;gt; config ssh host, > > > &gt; &amp;gt; &amp;amp;gt; user and password. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; E.g: > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; public static > > > AbstractTask > > > &gt; &amp;gt; newTask(TaskExecutionContext > > > &gt; &amp;gt; &amp;amp;gt; taskExecutionContext, Logger > > logger) > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; throws > > > &gt; &amp;gt; IllegalArgumentException { > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > Boolean > > > &gt; enable = > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable"); > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > if > > > &gt; (enable != null > > > &gt; &amp;gt; &amp;amp;amp;amp;&amp;amp;amp;amp; > > enable == false ) { > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > DummyTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > } > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > switch > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) { > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case SHELL: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > ShellTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case > > > &gt; &amp;gt; PROCEDURE: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > ProcedureTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case SQL: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; SqlTask(taskExecutionContext, > > logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case MR: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > MapReduceTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case SPARK: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > SparkTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case FLINK: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > FlinkTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case PYTHON: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > PythonTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case HTTP: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > HttpTask(taskExecutionContext, > > logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case DATAX: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > DataxTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case SQOOP: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; > > SqoopTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; case SSH: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; SSHTask(taskExecutionContext, > > logger); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; default: > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; logger.error("unsupport task > > > &gt; &amp;gt; &amp;amp;gt; type: {}", > > taskExecutionContext.getTaskType()); > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; throw new > > > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException("not > > support task > > > type"); > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > } > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; } > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 3. I am not > > > sure that it supports > > > &gt; window or not. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > ------------------ 原始邮件 > > > &gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 发件人: > > > "wenhemin"<[email protected] > > > &gt; <mailto: > > > &gt; &amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间: > > > 2020年5月14日(星期四) 下午3:46 > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: "裴龙武"< > > > [email protected] > > > &gt; <mailto:[email protected] > > > &gt; &amp;gt; &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re: > > > [Feature] Support SSH Task > > > &gt; and Support dummy > > > &gt; &amp;gt; task like > > > &gt; &amp;gt; &amp;amp;gt; airflow > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; Sorry, My > > > previous description is > > > &gt; not very clear. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; I want to ask > > > some questions: > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 1.How to > > > control the priority > > > &gt; between ssh tasks? > > > &gt; &amp;gt; There may be > > > &gt; &amp;gt; &amp;amp;gt; some ssh tasks that have been > > waiting for > > > execution. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 2.I understand > > > what you want to > > > &gt; solve is the problem > > > &gt; &amp;gt; of executing > > > &gt; &amp;gt; &amp;amp;gt; remote ssh scripts in > batches. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > So, not > > > &gt; sure how to use this > > > &gt; &amp;gt; function. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; 3.I don't know > > > if this supports > > > &gt; windows system. > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 在 > > 2020年5月13日，20:56，裴龙武 < > > > &gt; [email protected] > > > &gt; &amp;gt; <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > 写道： > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; I use > > spin lock. Here > > > &gt; is my code. Of course > > > &gt; &amp;gt; , it's not > > > &gt; &amp;gt; &amp;amp;gt; perfect. I just do a test. To > > my surprise, it > > > is the result > > > &gt; of the > > > &gt; &amp;gt; &amp;amp;gt; execution is the same as the > > AirFlow > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; > > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow > > > &gt; &amp;gt; &amp;amp;gt; 相同的结果。 > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > ------------------ 原始邮件 > > > &gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发件人: > > "whm_777"< > > > &gt; [email protected] <mailto: > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 发送时间: > > 2020年5月13日(星期三) > > > &gt; 晚上7:21 > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 收件人: > > "裴龙武"< > > > &gt; [email protected] <mailto: > > > &gt; &amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; 主题: Re: > > [Feature] > > > &gt; Support SSH Task and > > > &gt; &amp;gt; Support dummy task > > > &gt; &amp;gt; &amp;amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; You can > > modify the > > > &gt; maximum number of linux > > > &gt; &amp;gt; ssh connections. > > > &gt; &amp;gt; &amp;amp;gt; > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; If use > > ssh connection > > > &gt; pool, How to control > > > &gt; &amp;gt; the priority of > > > &gt; &amp;gt; &amp;amp;gt; ssh? > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > 在 > > > &gt; 2020年5月13日，18:01，裴龙武 < > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > 写道： > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > First 3Q， > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; I&amp;amp;amp;nbsp; use more than 100 task > > > &gt; &amp;gt; node. But SSH connections > > > &gt; &amp;gt; &amp;amp;gt; are limited. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; <[email protected] > > > &gt; &amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; ------------------ 原始邮件 > > > &gt; &amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > 发件人: > > > &gt; "whm_777"<[email protected] > > > &gt; &amp;gt; <mailto:[email protected] > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > 发送时间: > > > &gt; 2020年5月13日(星期三) 下午5:50 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > 收件人: "裴龙武"< > > > &gt; [email protected] <mailto: > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > 主题: Re: > > > &gt; [Feature] Support SSH Task > > > &gt; &amp;gt; and Support dummy task > > > &gt; &amp;gt; &amp;amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > E.g. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; rtn_code=`ssh -o > > > &gt; &amp;gt; ServerAliveInterval=60 -p xxxx > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > <mailto: > > > [email protected]&amp;amp;amp;gt; > > > &gt; ‘shell > > > &gt; &amp;gt; &amp;amp;gt; command&amp;amp;amp;nbsp; > > > &amp;amp;amp;gt;/dev/null > > > &gt; 2&amp;amp;amp;gt;&amp;amp;amp;amp;1; echo $?'` > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > if [ > > > &gt; "$rtn_code" -eq 0 ]; then > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; echo "成功" > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; exit 0 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > else > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; echo "失败" > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp;&amp;amp;amp;nbsp; > > > &gt; &amp;gt; exit 1 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > fi > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > Batch shell > > > &gt; command is not supported. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > Multiple > > > &gt; servers can be split into > > > &gt; &amp;gt; multiple task nodes. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 在 2020年5月13日，17:40，裴龙武 < > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > 写道： > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; Could you give me a > > > &gt; &amp;gt; example，3Q. 能否给我一个例子，谢谢！ > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; By the way, I have more than > > > &gt; &amp;gt; 100 tasks in one DAG. > > > &gt; &amp;gt; &amp;amp;gt; These tasks connect two other > > server to > > > execute. So SSH > > > &gt; tasks must > > > &gt; &amp;gt; have > > > &gt; &amp;gt; &amp;amp;gt; pool to manager. Now I use > > JSch and realize a > > > simple pool. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 顺带说一下，在我的实际场景中，我有100多个 SSH > > > &gt; &amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以 > > > &gt; &amp;gt; &amp;amp;gt; SSH 任务进行连接时，必须使用连接池进行管理。当前我使用 > > JSch，并实现了一个简单的连接池。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; ------------------ 原始邮件 > > > &gt; &amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 发件人: "wenhemin"< > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 发送时间: 2020年5月13日(星期三) 下午5:24 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 收件人: "dev"< > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; > > [email protected] > > > &amp;amp;amp;gt;&amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; 主题: Re: [Feature] Support > > > &gt; &amp;gt; SSH Task and Support dummy > > > &gt; &amp;gt; &amp;amp;gt; task like airflow > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; The shell node is supports > > > &gt; &amp;gt; remote calling, and get > > > &gt; &amp;gt; &amp;amp;gt; the remote command result > code. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; 在 > > > &gt; &amp;gt; 2020年5月13日，15:16，裴龙武 <[email protected] > > > &gt; &amp;gt; &amp;amp;gt; <mailto:[email protected] > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > 写道： > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Dear ALL： > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Support Linux SSH > > > &gt; &amp;gt; Task 支持 Linux SSH 任务 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 Shell 脚本，Shell > > > &gt; &amp;gt; &amp;amp;gt; 脚本是保存在业务服务器的固定目录。当 Worker > > > 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell > > > &gt; &amp;gt; &amp;amp;gt; > > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; For example, in my > > > &gt; &amp;gt; project, the workflow's tasks > > > &gt; &amp;gt; &amp;amp;gt; want to execute shell scripts > > where are in > > > different server's > > > &gt; &amp;gt; different > > > &gt; &amp;gt; &amp;amp;gt; directory. When worker > execute > > these shell > > > scripts, it must > > > &gt; use the > > > &gt; &amp;gt; same > > > &gt; &amp;gt; &amp;amp;gt; user to login these server. > > Also, the worker > > > can get the > > > &gt; executing > > > &gt; &amp;gt; state of > > > &gt; &amp;gt; &amp;amp;gt; these server. We can config > > these server 's > > > host,user and > > > &gt; password. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; SSH Task is very > > > &gt; &amp;gt; useful for most user SSH > > > &gt; &amp;gt; &amp;amp;gt; 任务对大多数用户是非常有用的 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; 分布式调度任务所执行的 Shell > > > &gt; &amp;gt; &amp;amp;gt; > > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 > > > Worker > > > &gt; &amp;gt; &amp;amp;gt; > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; In > > > &gt; &amp;gt; dolphinscheduler, the most executing tasks > > > &gt; &amp;gt; &amp;amp;gt; are in different servers who > > are not workers. > > > These servers > > > &gt; also have > > > &gt; &amp;gt; their > > > &gt; &amp;gt; &amp;amp;gt; different fixed services. We > > just have to pass > > > different > > > &gt; parameters to > > > &gt; &amp;gt; &amp;amp;gt; schedule these shell scripts > > to execute. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Python has a module > > > &gt; &amp;gt; to execute ssh script Python > > > &gt; &amp;gt; &amp;amp;gt; 有固定的工具包，可执行这些SSH Shell 脚本 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Python > > > &gt; &amp;gt; 有一个可执行远程服务器SSH Shell脚本的模块，其名字为：paramiko。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Python has a module > > > &gt; &amp;gt; that can execute SSH Shell > > > &gt; &amp;gt; &amp;amp;gt; script. It's paramiko. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Others 其他内容 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; I found this > > > &gt; &amp;gt; described in previous feature, but > > > &gt; &amp;gt; &amp;amp;gt; it was relatively simple. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Feature URL > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; 另外，我通过 Shell Task > > > &gt; &amp;gt; &amp;amp;gt; > > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; In addition, it is > > > &gt; &amp;gt; very inconvenient for me to > > > &gt; &amp;gt; &amp;amp;gt; perform remote tasks through > > Shell Task. Here > > > is my script. > > > &gt; I don't > > > &gt; &amp;gt; know if > > > &gt; &amp;gt; &amp;amp;gt; there's a better way. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; sshpass -p > > > &gt; &amp;gt; 'password' ssh user@host echo 'ssh > > > &gt; &amp;gt; &amp;amp;gt; success' echo 'Hello World' > > > -&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; /home/dolphinscheduler/test/hello.txt > > > &gt; &amp;gt; &amp;amp;gt; echo 'end' > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Support dummy task > > > &gt; &amp;gt; like airflow 支持像 Airflow > > > &gt; &amp;gt; &amp;amp;gt; 中的虚拟任务 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; 场景描述：项目中，有已经产品化的 > > > &gt; &amp;gt; DAG 文件，DAG > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为 > > > &gt; &amp;gt; Dummy > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; For example, in my > > > &gt; &amp;gt; project, it has a productized > > > &gt; &amp;gt; &amp;amp;gt; DAG file. The file contains > > different modules, > > > some of which > > > &gt; are > > > &gt; &amp;gt; &amp;amp;gt; interdependent and some of > > which are not. When > > > customers > > > &gt; purchase > > > &gt; &amp;gt; different > > > &gt; &amp;gt; &amp;amp;gt; modules, we need to set some > > tasks as dummy > > > tasks, which > > > &gt; some modules > > > &gt; &amp;gt; are > > > &gt; &amp;gt; &amp;amp;gt; not purchased and the > > purchased module is not > > > dependent. > > > &gt; Because of > > > &gt; &amp;gt; this > > > &gt; &amp;gt; &amp;amp;gt; setting, these dummy tasks > are > > actually not > > > executed. The > > > &gt; benefits of > > > &gt; &amp;gt; this > > > &gt; &amp;gt; &amp;amp;gt; setup are product unity and > > diagram integrity. > > > In airflow, > > > &gt; these task > > > &gt; &amp;gt; &amp;amp;gt; execute by dummy operator. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; ** Realize 实现方式** > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Dummy Task > > > &gt; &amp;gt; 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为 > > > &gt; &amp;gt; &amp;amp;gt; dummy 时，实际的任务不执行，执行 Dummy > Task。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; Dummy Task is easy > > > &gt; &amp;gt; to realize, but it need to > > > &gt; &amp;gt; &amp;amp;gt; use with other different > > tasks. When the task's > > > executed > > > &gt; type is set > > > &gt; &amp;gt; to > > > &gt; &amp;gt; &amp;amp;gt; dummy type, the task are > > executed as a dummy > > > task and the > > > &gt; real task > > > &gt; &amp;gt; is not > > > &gt; &amp;gt; &amp;amp;gt; executed. > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。 > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;amp;amp;gt; By the way，I > > > &gt; &amp;gt; already realize these > > two&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; features in my fork > > > branch.&amp;amp;amp;amp;nbsp;Whether the > > > &gt; follow-up > > > &gt; &amp;gt; release can be > > > &gt; &amp;gt; &amp;amp;gt; supported > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > > > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > <SSHClient.java&amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;gt;&amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;gt; >
