Re: [I] Correct example for TaskFlow api [airflow]
potiuk closed issue #53637: Correct example for TaskFlow api URL: https://github.com/apache/airflow/issues/53637 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
obarisk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3998224804 Need to check the latest version before move further. at lease before 3.1.3, I don't think this one is a good first issue. (if we'd love to really fix it instead of a minor fix of those examples.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
potiuk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3986445226 @obarisk We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes). This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready. We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3568697581 Thanks for the clarification @obarisk! Apologies for stirring the soup as was trying to understand the validity of this PR prior to migration. I was trying to replicate the error and having issues earlier. Your sharing the details regarding the workaround tried out helps me understand better now. Am able to replicate after adding few additional lines per documentation. As reviewers for PR-54693, do you have any advice TP @uranusjr, Ash @ashb, Amogh @amoghrajesh, so we can decide the approach and place to fix this issue? TIA! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
obarisk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3566276957 I don't get your question. In this issue, the error is should be replicated by copy the example from the document it was 3.0.3 https://airflow.apache.org/docs/apache-airflow/3.0.3/core-concepts/taskflow.html#context , the document is still the same with latest(3.1.3) https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/taskflow.html#context The PR https://github.com/apache/airflow/pull/54693 is kind of solution. So there won't be errors. But..., I closed the pr because the PR use non exposed methods. Please read the discussion in https://github.com/apache/airflow/pull/54693 🙏🏼 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3559588504 https://github.com/user-attachments/assets/96ea7016-b158-49bb-95de-6d2f96be3d18"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637:
URL: https://github.com/apache/airflow/issues/53637#issuecomment-3556235050
I was trying to replicate the error using DAG code as below, however not
seeing any errors.
Can you please suggest any changes needed to replicate the error @obarisk or
someone else?
TIA!
#
"""Example DAG demonstrating the issue of PR53637."""
```
from datetime import datetime
from airflow.models.taskinstance import TaskInstance
#from airflow.models.dagrun import DagRun #commenting to try and replicate
error scenario
from airflow.sdk.api.datamodels._generated import DagRun
from airflow.sdk import dag, task
@dag(
dag_id='pr_53637_context_types',
schedule=None,
start_date=datetime(2025, 11, 16),
catchup=False,
tags=['issue_pr53637'],
)
def document_context():
@task
def print_ti_info(task_instance: TaskInstance, dag_run: DagRun):
print(f"Run ID: {task_instance.run_id}")
print(dag_run.__class__)
## airflow >= 3.0 airflow.sdk.api.datamodels._generated.DagRun
## airflow < 3.0 airflow.models.dagrun.DagRun
print_ti_info()
document_context()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Correct example for TaskFlow api [airflow]
potiuk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3547265189 > I'm sorry that I cann't answer any of the design nor the existence question. I don't thing @Srabasti is asking you particularly to solve it, somene could help but mostly it's up to someone to see the current state of things (we are currently in the stage of moving of stuff for APIs between airflow-core and task-sdk) and find out what was really the reasoning, why things were there and how to fix it. It's not **obvious** but it's also **changing** currently - and depends on which parts have been moved and which weren't. > But, without fixing it, the document will always provide the incorrect information. Well. That's a bit of truism. But It is important to discuss and agree **how** to fix it best. I think personally (and that was my comment to @cmarteepants above who I think originally moved the examples to the .rst files) that tn order to fix it (and keep it working) - it needs to be not only fixed but turned back into our "include" pattern and it should really be part of the https://cwiki.apache.org/confluence/display/AIRFLOW/Examples+Refurbish initiatives where we plan to - eventually - review and make our examples far more consistent and usable. The idea with include patterns of ours is that all the example dags we include are also PARSED during our tests and prek-hooks - which means that they **at least** do not cause import errrors and warnings with the current "main` state of things - which means that it's far more difficult to get such an out-dated example that is lying around until someone notices, because our CI system makes sure that whenever someone modifies and refactors code, the examples are still **working** with it - so they basically force anyone who is refactoring the code to also modify the examples. I am not sure **how exactly** to fix this particular example, but I think after this fix is complete our goal is not only to fix it but also leave it in the state that it will be more difficult to break it in the future accidentally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
obarisk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3544781008 I'm sorry that I cann't answer any of the **design** nor the **existence** question. I'm not capable to answer those. Sorry. I beleive the api or core concept should be somewhere in the documents or AIPs. However, this ticket is simply a documentation fix, the issue is way simple that > One/Two of the example(s) in the document is incorrent and cann't be executed on airflow >= 3.0. It cann't be fixed with a pythonic/airflow-style way because the correct fix depends on a non-exposed type. But, without fix it, the document will always shares incorrect information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3543658728 Thanks for your reply and sharing the details @obarisk! Looks like the root cause is to fix the use of "airflow.sdk.api.datamodels._generated.DagRun" to use the internal/private API (since the underscore in _generated). FYI - looks like "airflow.sdk.api.datamodels._generated.DagRun" does not exist as below screenshot. https://github.com/user-attachments/assets/d6138df9-c9ae-4629-a0ae-756e5fa3c1c9"; /> Instead found it in below link. Observed it does not have both "queued_at" and "duration" https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/api/datamodels/_generated.py My question is what is the utility of "queued_at" and "duration" field at this point of time in Airflow 3.X? Accordingly we need to decide whether to include it in https://airflow.apache.org/docs/task-sdk/stable/api.html, correct? As an example, if duration is a calculated field (difference between start and end times), we might not need to add it here, since users can calculate at their end. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
obarisk commented on issue #53637:
URL: https://github.com/apache/airflow/issues/53637#issuecomment-3538842272
@Srabasti
to replicate the issue, all you need to do is
1. have an airflow 3.0+ environment
2. copy the example dag to your environment
to debug the issue,
modify a bit of the example,
```python
from airflow.models.taskinstance import TaskInstance
from airflow.models.dagrun import DagRun
@dag(schedule=None)
def document_context():
@task
def print_ti_info(task_instance: TaskInstance, dag_run: DagRun):
print(f"Run ID: {task_instance.run_id}")
print(dag_runt.__class__)
## airflow >= 3.0 airflow.sdk.api.datamodels._generated.DagRun
## airflow < 3.0 airflow.models.dagrun.DagRun
print_ti_info()
document_context()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Correct example for TaskFlow api [airflow]
obarisk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3538835732 @Srabasti please take a look of the closed PR. This issue is not only because of `there's no queued_at` the issue is during runtime, the type of `dag_run` in the execution context is `airflow.sdk.api.datamodels._generated.DagRun` instead of `airflow.models.dagrun.DagRun`. `queued_at` is an attribute of `airflow.models.dagrun.DagRun` but it's no longer available (airflow >= 3). at the same time, it's not **correct** that we use `airflow.sdk.api.datamodels._generated.DagRun` for type hinting (it's not a public api). the propery way is waiting for this suggestion https://github.com/apache/airflow/pull/54693#discussion_r2314763686 > We should probably expose some types (maybe in airflow.sdk.types) for users to import for type annotations like here. It is recommended too; since we expose Context as a public type, things in Context (and things in those things) should also be exposed somehow. If someday we can have the type, we can have the correct example. Note, the example can't be successfully executed on airflow >= 3 So. this is an issue. But we can't solve this with only fixing the example in the document -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3536886642 Hello @obarisk , Can you please confirm if you are able to replicate the issue? Kindly share the steps to do so. If not, shall we close this @potiuk @vikramkoka, since open since 7/2025 and it is already 11/2025? I can pick this up once the requirements are clear on the changes to fix the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
Srabasti commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3525595141 Is this issue already resolved from last commit? Can someone kindly share how to replicate the above "Attribute Errors" as not clear from the description? From the documentation link below, still seeing "queued_at" is mentioned. https://airflow.apache.org/docs/apache-airflow/3.0.3/core-concepts/taskflow.html#context https://github.com/user-attachments/assets/84429af8-2a96-4a10-8d51-58fcd2dde65b"; /> I see context variables in link below Airflow engine passes "run_id" and "dag_run" by default for all the templates. https://airflow.apache.org/docs/apache-airflow/3.0.3/templates-ref.html#templates-variables Looks to me since "queued_at" is not present, all the places it is mentioned in the documentation needs to be removed. Please advise if any additional changes are needed. Since not aware of the history behind this, wanted to confirm that adding this field now is not in scope for Airflow engine anymore by the community. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Correct example for TaskFlow api [airflow]
potiuk commented on issue #53637: URL: https://github.com/apache/airflow/issues/53637#issuecomment-3103636082 FYI: @cmarteepants -> this is one of the reasons I always prefer "example-include" case and actual import and validation (and often running) the actual example files that parts are included in the docs :D -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
