Re: pipeline timeout

Hi Chaim If you want to cancel the job from airflow, here is how you can set timeout. (Using *execution_timeout*) aggregate_db_message_job = BashOperator( task_id='aggregate_db_message_job', execution_timeout=timedelta(hours=3), pool='ep_data_pipeline_db_msg_agg', bash_command=agg

Re: pipeline timeout

Just curious: how do you use python to kick off a Java pipeline in airflow? And, could you go to a direction that you have an airflow operator(maybe a bash operator to execute a Jar?) to run such Java code: result = p.run() result.wait_until_finish(duration=TIMEOUT_SEC) if not PipelineState.is_te

Re: pipeline timeout

sorry for not being explicit. My pipeline is in java and i am executing it using python in airflow. I would like from airflow to cancel the pipeline if running for more than x minutes. Currently i am doing this using the cli, but it is not optimal chaim On Mon, Jul 8, 2019 at 7:24 PM Mark Liu wr

Re: pipeline timeout

Hi Chaim, You can checkout PipelineResult class and do something like: result = p.run() result.wait_until_finish(duration=TIMEOUT_SEC) if not PipelineState.is_terminal(result.state): result.cancel() The

pipeline timeout

Hi, I have a pipeline that usually takes 15-30 minutes. Sometimes things get stuck (from 3rd party side). I would like to know if there is a way to cancel the job if it is running for more than x minutes? I know there is a cli command but i would like it either on the pipeline config or in the py