I think your idea is good, haha
Best Regards --------------- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 [email protected] --------------- Hemin Wen <[email protected]> 于2020年5月27日周三 上午11:53写道： > Is it possible to extend the data source and increase the ssh type. > I look at the data structure of DB is ok. > > > -------------------- > DolphinScheduler(Incubator) Commtter > Hemin Wen 温合民 > [email protected] > -------------------- > > > 裴龙武 <[email protected]> 于2020年5月27日周三 上午10:55写道： > > > Good idea! It's can solve the exceptions where the numbers of SSH > > connections exceeds. > > > > > > We also have another problem of how to config remote server easily when > we > > have hundreds of tasks. > > > > > > Now I use the local param to config remote server's IP,username,password. > > But every task needs to config. > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > > 发件人: "lidong dai"<[email protected]>; > > 发送时间: 2020年5月26日(星期二) 晚上6:38 > > 收件人: "dev"<[email protected]>; > > > > 主题: Re: [Feature] Support SSH Task > > > > > > > > good idea, I think "Control the number of task's concurrent" may be have > > some other methods, for now, we don't have this feature, but when I > > communicate with the team of China Unicom， they provide a new solve > > solution： > > we can add a task concurrent page, you can set a variable value > like > > 'ETL_JOB' =10 and so on, then give the task type add a option for > choosing > > ETL_JOB or other variable, and there may be many task also choose ETL_JOB > > variable. when these tasks begin to running at the same time, the > > Master > > Server will check whether a task begin to run or wait(if the running > tasks > > num = 10) > > > > > > Best egards > > --------------- > > DolphinScheduler(Incubator) PPMC > > Lidong Dai 代立冬 > > [email protected] > > --------------- > > > > > > 裴龙武 <[email protected]> 于2020年5月26日周二 下午2:34写道： > > > > > There is only two ways that I can think to do about this. > > > > > > > > > 1. Simple SSH connect pool > > > 2. Control the number of task's concurrent > > > > > > > > > ------------------&nbsp;原始邮件&nbsp;------------------ > > > 发件人:&nbsp;"Hemin Wen"<[email protected]&gt;; > > > 发送时间:&nbsp;2020年5月22日(星期五) 晚上7:29 > > > 收件人:&nbsp;"dev"<[email protected]&gt;; > > > > > > 主题:&nbsp;Re: [Feature] Support SSH Task > > > > > > > > > > > > What other friends think about this discuss > > > > > > > > > -------------------- > > > DolphinScheduler(Incubator) Commtter > > > Hemin Wen&nbsp; 温合民 > > > [email protected] > > > -------------------- > > > > > > > > > 裴龙武 <[email protected]&gt; 于2020年5月22日周五 上午10:20写道： > > > > > > &gt; Your are right, it can be solved the problem by modifying > the > > > &gt; number of server ssh connections.&amp;nbsp; > > > &gt; > > > &gt; The real reason is that I want to control the&amp;nbsp; > > number of SSH > > > &gt; connections and reuse the SSH connection, Because the > remote > > server is > > > &gt; managed by other team. > > > &gt; > > > &gt; Otherwise, I think that we can also control the number of > > parallel > > > tasks > > > &gt; in a workflow. > > > &gt; > > > &gt; > > > &gt; It is a good idea that we can use the encapsulated ssh > > connection to > > > get > > > &gt; clear > > > &gt; exception information&amp;nbsp; > > > &gt; > > > &gt; > > > &gt; > > > &gt; > > > &gt; > > > &gt; > > > &gt; > > > &gt; > > ------------------&amp;nbsp;原始邮件&amp;nbsp;------------------ > > > &gt; 发件人:&amp;nbsp;"Hemin Wen"<[email protected] > > &amp;gt;; > > > &gt; 发送时间:&amp;nbsp;2020年5月21日(星期四) 中午11:07 > > > &gt; 收件人:&amp;nbsp;"dev"<[email protected] > > &amp;gt;; > > > &gt; > > > &gt; 主题:&amp;nbsp;Re: [Feature] Support SSH Task > > > &gt; > > > &gt; > > > &gt; > > > &gt; Thanks. > > > &gt; > > > &gt; About ssh connections pool, > > > &gt; 1. If the connection is not enough, Can it be solved by > > modifying the > > > &gt; number of server ssh connections. > > > &gt; 2. I think we have to look at whether a server's parallel > > ssh tasks > > > will > > > &gt; reach the bottleneck of server connections. > > > &gt; > > > &gt; I think it is better to use the encapsulated ssh connection > > to get > > > clear > > > &gt; exception information. > > > &gt; > > > &gt; -------------------- > > > &gt; DolphinScheduler(Incubator) Commtter > > > &gt; Hemin Wen&amp;nbsp; 温合民 > > > &gt; [email protected] > > > &gt; -------------------- > > > &gt; > > > &gt; > > > &gt; 裴龙武 <[email protected]&amp;gt; 于2020年5月20日周三 下午11:29写道： > > > &gt; > > > &gt; &amp;gt; Thanks for your advise and suggestions. It may > > has some > > > exceptions if > > > &gt; I > > > &gt; &amp;gt; implement SSH Task as you said. Now, let me > > explain my ideas. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; e.g > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; My workflow DAG has 100 or more task, it's all > > ssh task. I > > > &gt; have&amp;amp;nbsp; two > > > &gt; &amp;gt; questions. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 1、We must limit SSH connection and reuse the > > SSH connections, > > > &gt; otherwise > > > &gt; &amp;gt; the remote server will be refused. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 2、If we config all of these task, we have a > lot > > of repeat > > > work to do. > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; By the way, the way I'm currently verifying is > > I use local > > > params to > > > &gt; &amp;gt; config the remote server's host,user,password > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > > > > ------------------&amp;amp;nbsp;原始邮件&amp;amp;nbsp;------------------ > > > &gt; &amp;gt; 发件人:&amp;amp;nbsp;"lidong dai"< > > [email protected] > > > &amp;amp;gt;; > > > &gt; &amp;gt; 发送时间:&amp;amp;nbsp;2020年5月20日(星期三) 晚上10:56 > > > &gt; &amp;gt; 收件人:&amp;amp;nbsp;"dev"< > > [email protected] > > > &amp;amp;gt;; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 主题:&amp;amp;nbsp;Re: [Feature] Support SSH > > Task > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; hi, I see you will Jsch, does his license > > compitable with > > > apache v2 > > > &gt; &amp;gt; license, I see the following words " > > > &gt; &amp;gt; JSch 0.0.* was released under the GNU LGPL > > license. Later, > > > we have > > > &gt; switched > > > &gt; &amp;gt; over to a BSD-style license. > > > &gt; &amp;gt; " on > > https://github.com/is/jsch/blob/master/LICENSE.txt . > > > you know, > > > &gt; LGPL > > > &gt; &amp;gt; license not compitable with apache license. > but > > BSD is ok , > > > so the > > > &gt; version > > > &gt; &amp;gt; of Jsch is key point > > > &gt; &amp;gt; > > > &gt; &amp;gt; second, Connection Pool is heavy resource, > > Please use > > > carefully > > > &gt; &amp;gt; > > > &gt; &amp;gt; and I'm a little confused, why there must be > > add a new > > > &gt; Table,&amp;amp;nbsp; other > > > &gt; &amp;gt; task > > > &gt; &amp;gt; types also no seperate > table,&amp;amp;nbsp; > > could you > > > consider task_type > > > &gt; and > > > &gt; &amp;gt; task_json > > > &gt; &amp;gt; in t_ds_task_instance ? > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; Best Regards > > > &gt; &amp;gt; --------------- > > > &gt; &amp;gt; DolphinScheduler(Incubator) PPMC > > > &gt; &amp;gt; Lidong Dai 代立冬 > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; --------------- > > > &gt; &amp;gt; > > > &gt; &amp;gt; > > > &gt; &amp;gt; 裴龙武 <[email protected]&amp;amp;gt; > > 于2020年5月20日周三 下午9:57写道： > > > &gt; &amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; I want to describe my idea. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; First, we must design a host > > table that's named > > > &gt; t_ds_ssh_host, e.g > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; - id > > > &gt; &amp;gt; &amp;amp;gt; - name > > > &gt; &amp;gt; &amp;amp;gt; - ip / host > > > &gt; &amp;gt; &amp;amp;gt; - user > > > &gt; &amp;gt; &amp;amp;gt; - password > > > &gt; &amp;gt; &amp;amp;gt; - max_connection > > > &gt; &amp;gt; &amp;amp;gt; - create_time > > > &gt; &amp;gt; &amp;amp;gt; - update_time > > > &gt; &amp;gt; &amp;amp;gt; - user_id > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Second, Shell can execute in > > worker server or > > > remote host. > > > &gt; We can > > > &gt; &amp;gt; choose a > > > &gt; &amp;gt; &amp;amp;gt; host in host's list. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Third, when workflow running, > > we must maintain > > > the remote > > > &gt; host's > > > &gt; &amp;gt; &amp;amp;gt; connection pool. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Finally, when workflow > finish, > > we must release > > > the > > > &gt; connection pool. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Why do we have to maintain a > > connection pool. > > > Because it is > > > &gt; very easy > > > &gt; &amp;gt; to > > > &gt; &amp;gt; &amp;amp;gt; get exceptions when it's has > > many SSH task. > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; > > > > > > ------------------&amp;amp;amp;nbsp;原始邮件&amp;amp;amp;nbsp;------------------ > > > &gt; &amp;gt; &amp;amp;gt; > > 发件人:&amp;amp;amp;nbsp;"lidong dai"< > > > [email protected] > > > &gt; &amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > 发送时间:&amp;amp;amp;nbsp;2020年5月20日(星期三) 晚上7:12 > > > &gt; &amp;gt; &amp;amp;gt; > > 收件人:&amp;amp;amp;nbsp;"dev"< > > > [email protected] > > > &gt; &amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; 主题:&amp;amp;amp;nbsp;Re: > > [Feature] Support SSH > > > Task > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; glad to hear that you will > > implement this > > > feature > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; Best Regards > > > &gt; &amp;gt; &amp;amp;gt; --------------- > > > &gt; &amp;gt; &amp;amp;gt; DolphinScheduler(Incubator) > > PPMC > > > &gt; &amp;gt; &amp;amp;gt; Lidong Dai 代立冬 > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; --------------- > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; 裴龙武 <[email protected] > > &amp;amp;amp;gt; > > > 于2020年5月20日周三 下午3:47写道： > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; My code > > is not perfect yet. I > > > will write a > > > &gt; detailed design > > > &gt; &amp;gt; document. > > > &gt; &amp;gt; &amp;amp;gt; Then > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; I will > > realize this feature > > > about our > > > &gt; discussion result. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > ------------------&amp;amp;amp;amp;nbsp;原始邮件&amp;amp;amp;amp;nbsp;------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 发件人:&amp;amp;amp;amp;nbsp;"wenhemin"< > > > &gt; [email protected]&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 发送时间:&amp;amp;amp;amp;nbsp;2020年5月18日(星期一) 晚上7:50 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 收件人:&amp;amp;amp;amp;nbsp;"裴龙武"<[email protected] > > > &gt; &amp;amp;amp;amp;gt;;"dev"< > > > &gt; &amp;gt; &amp;amp;gt; > > [email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > 主题:&amp;amp;amp;amp;nbsp;Re: > > > [Feature] Support SSH > > > &gt; Task and Support > > > &gt; &amp;gt; dummy task > > > &gt; &amp;gt; &amp;amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; Thanks > > for writing detailed > > > documentation. I > > > &gt; think this is > > > &gt; &amp;gt; also a > > > &gt; &amp;gt; &amp;amp;gt; missing > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; feature > > of DS. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; About > the > > extension point: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 1.Can > ssh > > tasks be merged into > > > shell tasks. > > > &gt; Essentially, > > > &gt; &amp;gt; they all > > > &gt; &amp;gt; &amp;amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; shell > > commands. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 2.About > > dummy task, DS has the > > > function of > > > &gt; disable nodes, I > > > &gt; &amp;gt; do n’t > > > &gt; &amp;gt; &amp;amp;gt; know if > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; this > > requirement is met. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; The > > script from AirFlow to > > > Dolphin is great. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; 在 > > > 2020年5月18日，09:28，裴龙武 < > > > &gt; [email protected]&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; OK, 3Q! > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; First, I > > > will ensure that open > > > &gt; source can use. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; Second, I > > > think we must > > > &gt; discuss deeply. I write > > > &gt; &amp;gt; a more > > > &gt; &amp;gt; &amp;amp;gt; detailed > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > document. > > You can check the > > > attachment. I also > > > &gt; send the > > > &gt; &amp;gt; document to > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > DaiLidong. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > Third,&amp;amp;amp;amp;nbsp; I'll > > > &gt; give you the error of > > > &gt; &amp;gt; not using SSH > > > &gt; &amp;gt; &amp;amp;gt; connection pool. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > ------------------ 原始邮件 > > > &gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; 发件人: > > > "wenhemin"< > > > &gt; [email protected]&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; 发送时间: > > > 2020年5月14日(星期四) 晚上7:26 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; 收件人: > > > "裴龙武"<[email protected] > > > &gt; &amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; 主题: Re: > > > [Feature] Support SSH > > > &gt; Task and Support > > > &gt; &amp;gt; dummy task > > > &gt; &amp;gt; &amp;amp;gt; like airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; Great! > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; I think, > > > Can ssh tasks be > > > &gt; merged into shell > > > &gt; &amp;gt; tasks,&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script > > locally or remotely, > > > Configure on the > > > &gt; front end. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; About ssh > > > connect pool, I did > > > &gt; not find it > > > &gt; &amp;gt; necessary to use > > > &gt; &amp;gt; &amp;amp;gt; the > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > connection pool. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; BTW, Look > > > at the code to > > > &gt; introduce additional > > > &gt; &amp;gt; jar packages, > > > &gt; &amp;gt; &amp;amp;gt; You also > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; need to > > ensure that open > > > source can use the > > > &gt; license of this > > > &gt; &amp;gt; jar > > > &gt; &amp;gt; &amp;amp;gt; package. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 在 > > > &gt; 2020年5月14日，16:20，裴龙武 < > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1. The > > > &gt; priority between these tasks > > > &gt; &amp;gt; is also depended > > > &gt; &amp;gt; &amp;amp;gt; on the > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dolphin > > DAG define. When the > > > front task is not > > > &gt; finished, it > > > &gt; &amp;gt; not > > > &gt; &amp;gt; &amp;amp;gt; execute > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; next > task. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2. I extend > > > &gt; ssh task. I also use > > > &gt; &amp;gt; local params to > > > &gt; &amp;gt; &amp;amp;gt; config ssh host, > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user and > > password. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; E.g: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; public static > > > &gt; AbstractTask > > > &gt; &amp;gt; &amp;amp;gt; newTask(TaskExecutionContext > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > taskExecutionContext, Logger > > > logger) > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; throws > > > &gt; &amp;gt; &amp;amp;gt; IllegalArgumentException { > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > Boolean > > > &gt; &amp;gt; enable = > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > JSONUtils.parseObject(taskExecutionContext.getTaskParams()).getBoolean("enable"); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > if > > > &gt; &amp;gt; (enable != null > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; > > > enable == false ) { > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > DummyTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > } > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > switch > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > (EnumUtils.getEnum(TaskType.class,taskExecutionContext.getTaskType())) { > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case SHELL: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > ShellTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case > > > &gt; &amp;gt; &amp;amp;gt; PROCEDURE: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > ProcedureTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case SQL: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > SqlTask(taskExecutionContext, > > > logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case MR: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > MapReduceTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case SPARK: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > SparkTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case FLINK: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > FlinkTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case PYTHON: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > PythonTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case HTTP: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > HttpTask(taskExecutionContext, > > > logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case DATAX: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > DataxTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case SQOOP: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > SqoopTask(taskExecutionContext, logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; case SSH: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; return new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > SSHTask(taskExecutionContext, > > > logger); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; default: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; logger.error("unsupport task > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; type: > {}", > > > taskExecutionContext.getTaskType()); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; throw new > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > IllegalArgumentException("not > > > support task > > > &gt; type"); > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > } > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; } > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3. I am not > > > &gt; sure that it supports > > > &gt; &amp;gt; window or not. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; ------------------ 原始邮件 > > > &gt; &amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发件人: > > > &gt; "wenhemin"<[email protected] > > > &gt; &amp;gt; <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 发送时间: > > > &gt; 2020年5月14日(星期四) 下午3:46 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 收件人: "裴龙武"< > > > &gt; [email protected] > > > &gt; &amp;gt; <mailto:[email protected] > > > &gt; &amp;gt; &amp;amp;gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 主题: Re: > > > &gt; [Feature] Support SSH Task > > > &gt; &amp;gt; and Support dummy > > > &gt; &amp;gt; &amp;amp;gt; task like > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; Sorry, My > > > &gt; previous description is > > > &gt; &amp;gt; not very clear. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; I want to ask > > > &gt; some questions: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 1.How to > > > &gt; control the priority > > > &gt; &amp;gt; between ssh tasks? > > > &gt; &amp;gt; &amp;amp;gt; There may be > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; some ssh > > tasks that have been > > > waiting for > > > &gt; execution. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 2.I understand > > > &gt; what you want to > > > &gt; &amp;gt; solve is the problem > > > &gt; &amp;gt; &amp;amp;gt; of executing > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; remote > > ssh scripts in batches. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > So, not > > > &gt; &amp;gt; sure how to use this > > > &gt; &amp;gt; &amp;amp;gt; function. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; 3.I don't know > > > &gt; if this supports > > > &gt; &amp;gt; windows system. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > 在 > > > 2020年5月13日，20:56，裴龙武 < > > > &gt; &amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; <mailto: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > I > > use > > > spin lock. Here > > > &gt; &amp;gt; is my code. Of course > > > &gt; &amp;gt; &amp;amp;gt; , it's not > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perfect. > > I just do a test. To > > > my surprise, it > > > &gt; is the result > > > &gt; &amp;gt; of the > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > execution > > is the same as the > > > AirFlow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > 我通过模拟自选锁方式实现，附件中是我的代码，当然，这并不完善。我拿这个做了测试。令我惊喜的是，我得到了和 AirFlow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 相同的结果。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > ------------------ 原始邮件 > > > &gt; &amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > 发件人: > > > "whm_777"< > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > 发送时间: > > > 2020年5月13日(星期三) > > > &gt; &amp;gt; 晚上7:21 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > 收件人: > > > "裴龙武"< > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > 主题: Re: > > > [Feature] > > > &gt; &amp;gt; Support SSH Task and > > > &gt; &amp;gt; &amp;amp;gt; Support dummy task > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like > > airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > You can > > > modify the > > > &gt; &amp;gt; maximum number of linux > > > &gt; &amp;gt; &amp;amp;gt; ssh connections. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > If > > use > > > ssh connection > > > &gt; &amp;gt; pool, How to control > > > &gt; &amp;gt; &amp;amp;gt; the priority of > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; ssh? > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > 在 > > > &gt; &amp;gt; 2020年5月13日，18:01，裴龙武 < > > > &gt; &amp;gt; &amp;amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > First 3Q， > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; I&amp;amp;amp;amp;nbsp; use more than 100 > > task > > > &gt; &amp;gt; &amp;amp;gt; node. But SSH connections > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are > > limited. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > 我是使用了100多个任务节点，但服务器SSH连接是有限制的，超过后，就会报错了。下面是我扩展SSH任务节点后的一张截图，另外这个DAG是我从AirFlow转换过来的。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; <[email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; ------------------ 原始邮件 > > > &gt; &amp;gt; &amp;amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > 发件人: > > > &gt; &amp;gt; "whm_777"<[email protected] > > > &gt; &amp;gt; &amp;amp;gt; <mailto:[email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > 发送时间: > > > &gt; &amp;gt; 2020年5月13日(星期三) 下午5:50 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > 收件人: "裴龙武"< > > > &gt; &amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; [email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > 主题: Re: > > > &gt; &amp;gt; [Feature] Support SSH Task > > > &gt; &amp;gt; &amp;amp;gt; and Support dummy task > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; like > > airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > E.g. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; rtn_code=`ssh -o > > > &gt; &amp;gt; &amp;amp;gt; ServerAliveInterval=60 -p > xxxx > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > [email protected] <mailto: > > > &gt; [email protected]&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; ‘shell > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > command&amp;amp;amp;amp;nbsp; > > > &gt; &amp;amp;amp;amp;gt;/dev/null > > > &gt; &amp;gt; > > 2&amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;1; echo $?'` > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > if [ > > > &gt; &amp;gt; "$rtn_code" -eq 0 ]; then > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; echo "成功" > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; exit 0 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > else > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; echo "失败" > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp;&amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; exit 1 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > fi > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > Batch shell > > > &gt; &amp;gt; command is not supported. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > Multiple > > > &gt; &amp;gt; servers can be split into > > > &gt; &amp;gt; &amp;amp;gt; multiple task nodes. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 在 2020年5月13日，17:40，裴龙武 < > > > &gt; &amp;gt; &amp;amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; Could you give me a > > > &gt; &amp;gt; &amp;amp;gt; example，3Q. 能否给我一个例子，谢谢！ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; By the way, I have more than > > > &gt; &amp;gt; &amp;amp;gt; 100 tasks in one DAG. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; These > > tasks connect two other > > > server to > > > &gt; execute. So SSH > > > &gt; &amp;gt; tasks must > > > &gt; &amp;gt; &amp;amp;gt; have > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; pool to > > manager. Now I use > > > JSch and realize a > > > &gt; simple pool. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 顺带说一下，在我的实际场景中，我有100多个 SSH > > > &gt; &amp;gt; &amp;amp;gt; 任务，这些任务连接两台任务服务器进行任务执行。所以 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; SSH > > 任务进行连接时，必须使用连接池进行管理。当前我使用 > > > JSch，并实现了一个简单的连接池。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; ------------------ 原始邮件 > > > &gt; &amp;gt; &amp;amp;gt; ------------------ > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 发件人: "wenhemin"< > > > &gt; &amp;gt; &amp;amp;gt; [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > [email protected] > > > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 发送时间: 2020年5月13日(星期三) 下午5:24 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 收件人: "dev"< > > > &gt; &amp;gt; &amp;amp;gt; > > [email protected] <mailto: > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > [email protected] > > > &gt; &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; 主题: Re: [Feature] Support > > > &gt; &amp;gt; &amp;amp;gt; SSH Task and Support dummy > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; task > like > > airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; The shell node is supports > > > &gt; &amp;gt; &amp;amp;gt; remote calling, and get > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; the > > remote command result code. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 在 > > > &gt; &amp;gt; &amp;amp;gt; 2020年5月13日，15:16，裴龙武 < > > [email protected] > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; <mailto: > > [email protected] > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; 写道： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dear ALL： > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support Linux SSH > > > &gt; &amp;gt; &amp;amp;gt; Task 支持 Linux SSH 任务 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; 场景描述：当前项目中，工作流的任务的目标是执行不同服务器 > > Shell 脚本，Shell > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > 脚本是保存在业务服务器的固定目录。当 Worker > > > &gt; 调度执行时，需要通过固定用户登录这些服务器，然后执行 Shell > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 脚本并获取这些任务执行的状态，其中服务器地址、用户名、密码可配置。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my > > > &gt; &amp;gt; &amp;amp;gt; project, the workflow's tasks > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; want to > > execute shell scripts > > > where are in > > > &gt; different server's > > > &gt; &amp;gt; &amp;amp;gt; different > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > directory. When worker execute > > > these shell > > > &gt; scripts, it must > > > &gt; &amp;gt; use the > > > &gt; &amp;gt; &amp;amp;gt; same > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; user to > > login these server. > > > Also, the worker > > > &gt; can get the > > > &gt; &amp;gt; executing > > > &gt; &amp;gt; &amp;amp;gt; state of > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; these > > server. We can config > > > these server 's > > > &gt; host,user and > > > &gt; &amp;gt; password. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; SSH Task is very > > > &gt; &amp;gt; &amp;amp;gt; useful for most user SSH > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > 任务对大多数用户是非常有用的 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 分布式调度任务所执行的 Shell > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 脚本是处于不同的业务服务器，都有其固定的业务，这些业务服务器不是 Worker，只是需要 > > > &gt; Worker > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > 调度执行，我们只需要传递不同的参数，让服务器执行任务脚本即可。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; In > > > &gt; &amp;gt; &amp;amp;gt; dolphinscheduler, the most > > executing tasks > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; are in > > different servers who > > > are not workers. > > > &gt; These servers > > > &gt; &amp;gt; also have > > > &gt; &amp;gt; &amp;amp;gt; their > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > different > > fixed services. We > > > just have to pass > > > &gt; different > > > &gt; &amp;gt; parameters to > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; schedule > > these shell scripts > > > to execute. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module > > > &gt; &amp;gt; &amp;amp;gt; to execute ssh script Python > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > 有固定的工具包，可执行这些SSH Shell 脚本 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python > > > &gt; &amp;gt; &amp;amp;gt; 有一个可执行远程服务器SSH > > Shell脚本的模块，其名字为：paramiko。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Python has a module > > > &gt; &amp;gt; &amp;amp;gt; that can execute SSH Shell > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; script. > > It's paramiko. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Others 其他内容 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > 我发现之前的改进功能中也有关于这个的描述，不过相对简单。功能更新地址 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; I found this > > > &gt; &amp;gt; &amp;amp;gt; described in previous > feature, > > but > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; it was > > relatively simple. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Feature URL > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 另外，我通过 Shell Task > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > 方式去执行远程任务会非常不便，下面是我的脚本，不知道是否有更好的方式。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; In addition, it is > > > &gt; &amp;gt; &amp;amp;gt; very inconvenient for me to > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; perform > > remote tasks through > > > Shell Task. Here > > > &gt; is my script. > > > &gt; &amp;gt; I don't > > > &gt; &amp;gt; &amp;amp;gt; know if > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; there's > a > > better way. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; sshpass -p > > > &gt; &amp;gt; &amp;amp;gt; 'password' ssh user@host > echo > > 'ssh > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; success' > > echo 'Hello World' > > > &gt; -&amp;amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > /home/dolphinscheduler/test/hello.txt > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; echo > 'end' > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Support dummy task > > > &gt; &amp;gt; &amp;amp;gt; like airflow 支持像 Airflow > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; 中的虚拟任务 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; 场景描述：项目中，有已经产品化的 > > > &gt; &amp;gt; &amp;amp;gt; DAG 文件，DAG > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; > > > 文件中包括不同的模块，这些模块之间的有些点是相互依赖的，有些不是，在用户购买不同模块时，需要把未购买模块且其他已购模块未依赖的点设置为 > > > &gt; &amp;gt; &amp;amp;gt; Dummy > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > Task，这样实际这些任务就不会执行，这样设置的好处是产品统一性和图的完整性，在AirFlow中，这些是通过DummyOperator完成的。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; For example, in my > > > &gt; &amp;gt; &amp;amp;gt; project, it has a productized > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; DAG > file. > > The file contains > > > different modules, > > > &gt; some of which > > > &gt; &amp;gt; are > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > interdependent and some of > > > which are not. When > > > &gt; customers > > > &gt; &amp;gt; purchase > > > &gt; &amp;gt; &amp;amp;gt; different > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; modules, > > we need to set some > > > tasks as dummy > > > &gt; tasks, which > > > &gt; &amp;gt; some modules > > > &gt; &amp;gt; &amp;amp;gt; are > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; not > > purchased and the > > > purchased module is not > > > &gt; dependent. > > > &gt; &amp;gt; Because of > > > &gt; &amp;gt; &amp;amp;gt; this > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setting, > > these dummy tasks are > > > actually not > > > &gt; executed. The > > > &gt; &amp;gt; benefits of > > > &gt; &amp;gt; &amp;amp;gt; this > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; setup > are > > product unity and > > > diagram integrity. > > > &gt; In airflow, > > > &gt; &amp;gt; these task > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; execute > > by dummy operator. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; ** Realize 实现方式** > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task > > > &gt; &amp;gt; &amp;amp;gt; > > 本身实现很简单，只是需要与其他任务配合使用，但任务执行方式设置为 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy > > 时，实际的任务不执行，执行 Dummy Task。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; Dummy Task is easy > > > &gt; &amp;gt; &amp;amp;gt; to realize, but it need to > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; use with > > other different > > > tasks. When the task's > > > &gt; executed > > > &gt; &amp;gt; type is set > > > &gt; &amp;gt; &amp;amp;gt; to > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; dummy > > type, the task are > > > executed as a dummy > > > &gt; task and the > > > &gt; &amp;gt; real task > > > &gt; &amp;gt; &amp;amp;gt; is not > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > executed. > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; 顺带说一下，因为项目着急测试使用，我Fork了开发版本，实现两种任务类型。在后续的版本中是否能够支持。 > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;amp;amp;gt; By the way，I > > > &gt; &amp;gt; &amp;amp;gt; already realize these > > > two&amp;amp;amp;amp;amp;nbsp; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; features > > in my fork > > > &gt; branch.&amp;amp;amp;amp;amp;nbsp;Whether the > > > &gt; &amp;gt; follow-up > > > &gt; &amp;gt; &amp;amp;gt; release can be > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > supported > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &gt; > > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; > > > &gt; &amp;gt; > > > &gt; > > > > > > <SSHClient.java&amp;amp;amp;amp;gt;<SSHPool.java&amp;amp;amp;amp;gt;<SSHTask.java&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > > &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; &amp;gt; &amp;amp;gt; &amp;amp;amp;gt; > > &amp;amp;amp;amp;gt; > > > &gt; <项目场景中关于Dolphin的一些扩展点.pdf&amp;amp;amp;amp;gt; >
