Re: pipeline timeout

[Hannah Jiang]

Hi Chaim

If you want to cancel the job from airflow, here is how you can set
timeout. (Using *execution_timeout*)

aggregate_db_message_job = BashOperator(
task_id='aggregate_db_message_job',
execution_timeout=timedelta(hours=3),
pool='ep_data_pipeline_db_msg_agg',
bash_command=aggregate_db_message_job_cmd,
dag=dag)

Hope this helps.
Hannah

On Tue, Jul 9, 2019 at 11:15 AM Rui Wang  wrote:

> Just curious: how do you use python to kick off a Java pipeline in
> airflow?
>
> And, could you go to a direction that you have an airflow operator(maybe
> a  bash operator to execute a Jar?) to run such Java code:
>
> result = p.run()
> result.wait_until_finish(duration=TIMEOUT_SEC)
> if not PipelineState.is_terminal(result.state):
>   result.cancel()
>
> ?
>
> On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel  wrote:
>
>> sorry for not being explicit. My pipeline is in java and i am
>> executing it using python in airflow.
>> I would like from airflow to cancel the pipeline if running for more
>> than x minutes.
>> Currently i am doing this using the cli, but it is not optimal
>>
>> chaim
>>
>> On Mon, Jul 8, 2019 at 7:24 PM Mark Liu  wrote:
>> >
>> > Hi Chaim,
>> >
>> > You can checkout PipelineResult class and do something like:
>> >
>> > result = p.run()
>> > result.wait_until_finish(duration=TIMEOUT_SEC)
>> > if not PipelineState.is_terminal(result.state):
>> >   result.cancel()
>> >
>> > The implementation of PipelineResult depends on what runner you choose.
>> And you may find more useful functions in its subclass.
>> >
>> > Mark
>> >
>> >
>> > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel  wrote:
>> >>
>> >> Hi,
>> >>   I have a pipeline that usually takes 15-30 minutes. Sometimes things
>> >> get stuck (from 3rd party side). I would like to know if there is a
>> >> way to cancel the job if it is running for more than x minutes? I know
>> >> there is a cli command but i would like it either on the pipeline
>> >> config or in the python sdk.
>> >> Any ideas?
>> >>
>> >> Chaim Turkel
>> >>
>> >> --
>> >>
>> >>
>> >> Loans are funded by
>> >> FinWise Bank, a Utah-chartered bank located in Sandy,
>> >> Utah, member FDIC, Equal
>> >> Opportunity Lender. Merchant Cash Advances are
>> >> made by Behalf. For more
>> >> information on ECOA, click here
>> >> . For important information about
>> >> opening a new
>> >> account, review Patriot Act procedures here
>> >> .
>> >> Visit Legal
>> >>  to
>> >> review our comprehensive program terms,
>> >> conditions, and disclosures.
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> . For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> .
>> Visit Legal
>>  to
>> review our comprehensive program terms,
>> conditions, and disclosures.
>>
>

Re: pipeline timeout

[Rui Wang]

Just curious: how do you use python to kick off a Java pipeline in airflow?

And, could you go to a direction that you have an airflow operator(maybe a
bash operator to execute a Jar?) to run such Java code:

result = p.run()
result.wait_until_finish(duration=TIMEOUT_SEC)
if not PipelineState.is_terminal(result.state):
  result.cancel()

?

On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel  wrote:

> sorry for not being explicit. My pipeline is in java and i am
> executing it using python in airflow.
> I would like from airflow to cancel the pipeline if running for more
> than x minutes.
> Currently i am doing this using the cli, but it is not optimal
>
> chaim
>
> On Mon, Jul 8, 2019 at 7:24 PM Mark Liu  wrote:
> >
> > Hi Chaim,
> >
> > You can checkout PipelineResult class and do something like:
> >
> > result = p.run()
> > result.wait_until_finish(duration=TIMEOUT_SEC)
> > if not PipelineState.is_terminal(result.state):
> >   result.cancel()
> >
> > The implementation of PipelineResult depends on what runner you choose.
> And you may find more useful functions in its subclass.
> >
> > Mark
> >
> >
> > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel  wrote:
> >>
> >> Hi,
> >>   I have a pipeline that usually takes 15-30 minutes. Sometimes things
> >> get stuck (from 3rd party side). I would like to know if there is a
> >> way to cancel the job if it is running for more than x minutes? I know
> >> there is a cli command but i would like it either on the pipeline
> >> config or in the python sdk.
> >> Any ideas?
> >>
> >> Chaim Turkel
> >>
> >> --
> >>
> >>
> >> Loans are funded by
> >> FinWise Bank, a Utah-chartered bank located in Sandy,
> >> Utah, member FDIC, Equal
> >> Opportunity Lender. Merchant Cash Advances are
> >> made by Behalf. For more
> >> information on ECOA, click here
> >> . For important information about
> >> opening a new
> >> account, review Patriot Act procedures here
> >> .
> >> Visit Legal
> >>  to
> >> review our comprehensive program terms,
> >> conditions, and disclosures.
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> . For important information about
> opening a new
> account, review Patriot Act procedures here
> .
> Visit Legal
>  to
> review our comprehensive program terms,
> conditions, and disclosures.
>

Re: pipeline timeout

[Chaim Turkel]

sorry for not being explicit. My pipeline is in java and i am
executing it using python in airflow.
I would like from airflow to cancel the pipeline if running for more
than x minutes.
Currently i am doing this using the cli, but it is not optimal

chaim

On Mon, Jul 8, 2019 at 7:24 PM Mark Liu  wrote:
>
> Hi Chaim,
>
> You can checkout PipelineResult class and do something like:
>
> result = p.run()
> result.wait_until_finish(duration=TIMEOUT_SEC)
> if not PipelineState.is_terminal(result.state):
>   result.cancel()
>
> The implementation of PipelineResult depends on what runner you choose. And 
> you may find more useful functions in its subclass.
>
> Mark
>
>
> On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel  wrote:
>>
>> Hi,
>>   I have a pipeline that usually takes 15-30 minutes. Sometimes things
>> get stuck (from 3rd party side). I would like to know if there is a
>> way to cancel the job if it is running for more than x minutes? I know
>> there is a cli command but i would like it either on the pipeline
>> config or in the python sdk.
>> Any ideas?
>>
>> Chaim Turkel
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> . For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> .
>> Visit Legal
>>  to
>> review our comprehensive program terms,
>> conditions, and disclosures.

-- 


Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more
information on ECOA, click here 
. For important information about 
opening a new
account, review Patriot Act procedures here 
.
Visit Legal 
 to
review our comprehensive program terms, 
conditions, and disclosures. 

Re: pipeline timeout

[Mark Liu]

Hi Chaim,

You can checkout PipelineResult

class and do something like:

result = p.run()
result.wait_until_finish(duration=TIMEOUT_SEC)
if not PipelineState.is_terminal(result.state):
  result.cancel()

The implementation of PipelineResult

depends on what runner you choose. And you may find more useful functions
in its subclass.

Mark


On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel  wrote:

> Hi,
>   I have a pipeline that usually takes 15-30 minutes. Sometimes things
> get stuck (from 3rd party side). I would like to know if there is a
> way to cancel the job if it is running for more than x minutes? I know
> there is a cli command but i would like it either on the pipeline
> config or in the python sdk.
> Any ideas?
>
> Chaim Turkel
>
> --
>
>
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy,
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are
> made by Behalf. For more
> information on ECOA, click here
> . For important information about
> opening a new
> account, review Patriot Act procedures here
> .
> Visit Legal
>  to
> review our comprehensive program terms,
> conditions, and disclosures.
>

pipeline timeout

[Chaim Turkel]

Hi,
  I have a pipeline that usually takes 15-30 minutes. Sometimes things
get stuck (from 3rd party side). I would like to know if there is a
way to cancel the job if it is running for more than x minutes? I know
there is a cli command but i would like it either on the pipeline
config or in the python sdk.
Any ideas?

Chaim Turkel

-- 


Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more
information on ECOA, click here 
. For important information about 
opening a new
account, review Patriot Act procedures here 
.
Visit Legal 
 to
review our comprehensive program terms, 
conditions, and disclosures.