Just curious: how do you use python to kick off a Java pipeline in airflow?
And, could you go to a direction that you have an airflow operator(maybe a bash operator to execute a Jar?) to run such Java code: result = p.run() result.wait_until_finish(duration=TIMEOUT_SEC) if not PipelineState.is_terminal(result.state): result.cancel() ? On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel <[email protected]> wrote: > sorry for not being explicit. My pipeline is in java and i am > executing it using python in airflow. > I would like from airflow to cancel the pipeline if running for more > than x minutes. > Currently i am doing this using the cli, but it is not optimal > > chaim > > On Mon, Jul 8, 2019 at 7:24 PM Mark Liu <[email protected]> wrote: > > > > Hi Chaim, > > > > You can checkout PipelineResult class and do something like: > > > > result = p.run() > > result.wait_until_finish(duration=TIMEOUT_SEC) > > if not PipelineState.is_terminal(result.state): > > result.cancel() > > > > The implementation of PipelineResult depends on what runner you choose. > And you may find more useful functions in its subclass. > > > > Mark > > > > > > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <[email protected]> wrote: > >> > >> Hi, > >> I have a pipeline that usually takes 15-30 minutes. Sometimes things > >> get stuck (from 3rd party side). I would like to know if there is a > >> way to cancel the job if it is running for more than x minutes? I know > >> there is a cli command but i would like it either on the pipeline > >> config or in the python sdk. > >> Any ideas? > >> > >> Chaim Turkel > >> > >> -- > >> > >> > >> Loans are funded by > >> FinWise Bank, a Utah-chartered bank located in Sandy, > >> Utah, member FDIC, Equal > >> Opportunity Lender. Merchant Cash Advances are > >> made by Behalf. For more > >> information on ECOA, click here > >> <https://www.behalf.com/legal/ecoa/>. For important information about > >> opening a new > >> account, review Patriot Act procedures here > >> <https://www.behalf.com/legal/patriot/>. > >> Visit Legal > >> <https://www.behalf.com/legal/> to > >> review our comprehensive program terms, > >> conditions, and disclosures. > > -- > > > Loans are funded by > FinWise Bank, a Utah-chartered bank located in Sandy, > Utah, member FDIC, Equal > Opportunity Lender. Merchant Cash Advances are > made by Behalf. For more > information on ECOA, click here > <https://www.behalf.com/legal/ecoa/>. For important information about > opening a new > account, review Patriot Act procedures here > <https://www.behalf.com/legal/patriot/>. > Visit Legal > <https://www.behalf.com/legal/> to > review our comprehensive program terms, > conditions, and disclosures. >
