Hi Chaim
If you want to cancel the job from airflow, here is how you can set
timeout. (Using *execution_timeout*)
aggregate_db_message_job = BashOperator(
task_id='aggregate_db_message_job',
execution_timeout=timedelta(hours=3),
pool='ep_data_pipeline_db_msg_agg',
bash_command=aggregate_db_message_job_cmd,
dag=dag)
Hope this helps.
Hannah
On Tue, Jul 9, 2019 at 11:15 AM Rui Wang <ruw...@google.com> wrote:
> Just curious: how do you use python to kick off a Java pipeline in
> airflow?
>
> And, could you go to a direction that you have an airflow operator(maybe
> a bash operator to execute a Jar?) to run such Java code:
>
> result = p.run()
> result.wait_until_finish(duration=TIMEOUT_SEC)
> if not PipelineState.is_terminal(result.state):
> result.cancel()
>
> ?
>
> On Tue, Jul 9, 2019 at 1:14 AM Chaim Turkel <ch...@behalf.com> wrote:
>
>> sorry for not being explicit. My pipeline is in java and i am
>> executing it using python in airflow.
>> I would like from airflow to cancel the pipeline if running for more
>> than x minutes.
>> Currently i am doing this using the cli, but it is not optimal
>>
>> chaim
>>
>> On Mon, Jul 8, 2019 at 7:24 PM Mark Liu <mark...@google.com> wrote:
>> >
>> > Hi Chaim,
>> >
>> > You can checkout PipelineResult class and do something like:
>> >
>> > result = p.run()
>> > result.wait_until_finish(duration=TIMEOUT_SEC)
>> > if not PipelineState.is_terminal(result.state):
>> > result.cancel()
>> >
>> > The implementation of PipelineResult depends on what runner you choose.
>> And you may find more useful functions in its subclass.
>> >
>> > Mark
>> >
>> >
>> > On Sun, Jul 7, 2019 at 12:59 AM Chaim Turkel <ch...@behalf.com> wrote:
>> >>
>> >> Hi,
>> >> I have a pipeline that usually takes 15-30 minutes. Sometimes things
>> >> get stuck (from 3rd party side). I would like to know if there is a
>> >> way to cancel the job if it is running for more than x minutes? I know
>> >> there is a cli command but i would like it either on the pipeline
>> >> config or in the python sdk.
>> >> Any ideas?
>> >>
>> >> Chaim Turkel
>> >>
>> >> --
>> >>
>> >>
>> >> Loans are funded by
>> >> FinWise Bank, a Utah-chartered bank located in Sandy,
>> >> Utah, member FDIC, Equal
>> >> Opportunity Lender. Merchant Cash Advances are
>> >> made by Behalf. For more
>> >> information on ECOA, click here
>> >> <https://www.behalf.com/legal/ecoa/>. For important information about
>> >> opening a new
>> >> account, review Patriot Act procedures here
>> >> <https://www.behalf.com/legal/patriot/>.
>> >> Visit Legal
>> >> <https://www.behalf.com/legal/> to
>> >> review our comprehensive program terms,
>> >> conditions, and disclosures.
>>
>> --
>>
>>
>> Loans are funded by
>> FinWise Bank, a Utah-chartered bank located in Sandy,
>> Utah, member FDIC, Equal
>> Opportunity Lender. Merchant Cash Advances are
>> made by Behalf. For more
>> information on ECOA, click here
>> <https://www.behalf.com/legal/ecoa/>. For important information about
>> opening a new
>> account, review Patriot Act procedures here
>> <https://www.behalf.com/legal/patriot/>.
>> Visit Legal
>> <https://www.behalf.com/legal/> to
>> review our comprehensive program terms,
>> conditions, and disclosures.
>>
>
