villebro commented on PR #36368:
URL: https://github.com/apache/superset/pull/36368#issuecomment-3854554524
> Sync join-and-wait - the whole pitch of GTF is getting long-running work
off web workers, but when a sync caller hits a deduplicated task, join-and-wait
blocks the web worker polling until the async task completes doing zero useful
work. What's the scenario where that's the right call vs just forcing callers
to .schedule() and returning a task UUID?
There are a few motivations:
- Celery isn't a hard requirement right now. To make sure we can provide DRY
execution paths for sync and async workflows, we need to be able to have
support for executing tasks synchronously, while still being able to leverage
the deduplication and abort functionality that the framework offers. A prime
candidate for this type of consolidation is collapsing the current sync vs
async chart query flows into a single GTF based flow.
- Even with async capabilities, you may have workflows where you prefer sync
execution. I noticed we're currently generating thumbnails synchronously with
the `superset compute-thumbnails` CLI command. Providing async only would
require extra boilerplate for awaiting task completion. There could be other
similar cases even in async flows, when a task has a dependency on another
task, and wants to block until the dependent task finishes. This is going into
DAG territory, and not something that we'd widely encourage, but I can see the
cases where this could be handy.
> Also noticed the @task decorator checks
is_feature_enabled("GLOBAL_TASK_FRAMEWORK") at import time, not at
call/schedule time. So if any module with @task-decorated functions gets
imported during app startup or Celery autodiscovery with the flag off, the app
just crashes. That kind of kills the coexistence rollout strategy - deferring
the check to **call**/.schedule() would fix it.
Good catch, I will inline these checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
