tirkarthi commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-1843374660
Related : https://github.com/apache/airflow/issues/19929 . It seems this was
reported earlier but got closed as stale issue.
--
This is an automated message from the Apache Git
potiuk commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-1850908399
Yes. Probably it should be changed. The problem is that this would have to
bring the task from deferred state first to run it's on_kill method, but that
would require someone to imp
nirben82 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-1853973052
Thanks @potiuk.
1. Do you know if someone is currently working on this?
2. Could you please provide some more specifics on what needs to be
implemented, so we can understand
potiuk commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-1854039714
1, Not that I am aware of. But you are asking random maintainer who might
not know everything. In OSS everything is in the open so if this is not
mentioned here, there is no other p
pustoshilov-d commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2049354769
We're working with Slurm deferrable operator and waiting this feature so
much 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, plea
potiuk commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2049363035
> We're working with Slurm deferrable operator and waiting this feature so
much 🙏
Just a kind reminder: if you and your company waits for a feature badly, the
most certain wa
Lee-W commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2052002194
Assign it ot @sunank200 as he already created a PR to avoid others work on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on t
tirkarthi commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2052040917
I guess this issue needs a more general solution across core triggered and
the linked PR is more specific to a particular operator. One of the solutions
we are trying to implemen
tirkarthi commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2053554297
I just found out that there is already a cleanup method as part of the
interface. The problem is that cleanup is called during triggerer restarts too
as part of deployment due to
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2228847790
@sunank200 which approach are we planning to take on this issue?
1. Implement the workaround I described in
https://github.com/apache/airflow/issues/36090#issuecomment
dh-racheldean commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2057705616
If i was feeling super-hacky in my own fork, I'd be tweaking
`models.Trigger.clean_unused()`, and adapting `submit_failure` to do the
hilarious re-schedule. In theory, just t
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2094972855
## WARNING
The proposed solution of capturing `asyncio.CancelledError` in a try/except
is NOT safe!
The following PRs have implemented this:
- https://github
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2096716595
For those watching, I have done a little more testing with the workaround
proposed in
https://github.com/apache/airflow/issues/36090#issuecomment-2094972855, and at
least i
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2096725326
Same here. I have created the PR for BigQuery/InsertJobOperator based on
suggestions. Please review: https://github.com/apache/airflow/pull/39442
--
This is an automated messag
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2096817280
PR for DataprocCreateClusterOperator:
https://github.com/apache/airflow/pull/39446
--
This is an automated message from the Apache Git Service.
To respond to the message, pleas
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2096845024
PR for DataprocSubmitJobOperator:
https://github.com/apache/airflow/pull/39447
--
This is an automated message from the Apache Git Service.
To respond to the message, please lo
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2097846420
@thesuperzapper the problem with suggested approach is that when you're
working with triggers and the Triggerer component, you typically don't interact
directly with TaskInstance
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2097923781
@thesuperzapper also `BaseTrigger` already has task_instance which can be
directly be used
--
This is an automated message from the Apache Git Service.
To respond to the messag
rzilkha commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2098220576
Hi Everyone,
I noticed that you are making workaround for some of the providers, can
someone say if **EmrContainerOperator** is also going to fixed anytime soon? is
there a
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2098360793
Please re-review: https://github.com/apache/airflow/pull/39442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2098851751
Please re-review: https://github.com/apache/airflow/pull/39446
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub an
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2098852350
Please re-review: https://github.com/apache/airflow/pull/39447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHu
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101913601
@sunank200 I see you have removed the check for a RUNNING state, and now
only check if it's not in a DEFERRED state, for example:
https://github.com/apache/airflow/bl
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101920452
@sunank200 also note that using the `self.task_instance` of the BaseTrigger
will require __at least Airflow 2.6.0__, it was added by this PR
https://github.com/apache/airflo
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2101966602
@sunank200 also we will need to update all the other providers that set an
`on_kill` and can be deferred (unless you are already doing this).
This search might help us
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2164294237
@sunank200 @akaul my original solution proposed in
https://github.com/apache/airflow/issues/36090#issuecomment-2094972855 had a
critical part missing, which means that the s
sunank200 commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2167393343
> @sunank200 @akaul my original solution proposed in
https://github.com/apache/airflow/issues/36090#issuecomment-2094972855 had a
critical part missing, which means that the solu
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2167397321
> @thesuperzapper I tried clearing the task for
`DataprocCreateClusterOperator` and it deleted the provisioning cluster for me.
@sunank200 it's a race condition, it
potiuk commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2168145008
Probably worth implementing it on the "airflow" level as you mentioned. Ate
you willing to submit a PR on that @thesuperzapper ? Did I understand well?
--
This is an automated mes
thesuperzapper commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2168436868
> Probably worth implementing it on the "airflow" level as you mentioned.
Ate you willing to submit a PR on that @thesuperzapper ? Did I understand well?
@potiuk I can
potiuk commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2168461614
As usual with OSS - we need someone to take a lead on that, so yeah - I see
you would love to discuss the solution but it needs someone to take the lead
and create a PR for that
eladkal commented on issue #36090:
URL: https://github.com/apache/airflow/issues/36090#issuecomment-2168492501
@sunank200 reported that he is working on a generic solution.
@sunank200 is that still the case?
--
This is an automated message from the Apache Git Service.
To respond to the
32 matches
Mail list logo
