The use case I'm referring to is that you can't use xcom to let a task read information from it's past attempts, because when a task starts it's xcom is automatically deleted.
My specific use case is that we have a custom LivyOperator that calls Livy to start batch Spark Jobs. When you start a batch job Livy returns a job_id Sometimes our operator can fail for one reason or another (for instance if Livy is unreachable for a while) When the task retries, it calls Livy again, which start the same spark job, but the problem is that the spark job from the first attempt can still be running, and then we have a batch job that runs twice simultaneously and creates duplicates in the output. What we tried to do is getting the job_id from the first try, to check if the job is still running, and wait for it to complete if it is. We tried using xcom to let the task send a message to itself (to it's next try) but xcom is meant for "inter-task communication" only so this doesn't work and is not intended to work. On Mon, 1 Jun 2020 at 13:15, Ash Berlin-Taylor <[email protected]> wrote: > Hi Furcy, > > Can you give a concrete example of what you mean by intra-task xcom? > Depending your use case this may already be possible. > > On Jun 1 2020, at 11:45 am, Furcy Pin <[email protected]> wrote: > > Hello, > > I would like to open a feature request for Airflow to support "intra-task > xcom". > > It seems that there are several distinct use cases for it already > and only ugly workarounds and I wanted to list them in a JIRA ticket. > > I wanted to summarize links to the use cases and past attempts, > and the recommended approach (which apparently would be to create > a distinct feature from xcom to support this, it could be calle intra-com > or self-com ?) > > Do you know if such ticket already exists? I couldn't find one. > Also I can't create any ticket due to some obscure bug (see my other > email). > > Thanks, > > Furcy > >
