Re: [PR] Add start execution from triggerer support to dynamic task mapping [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #39912: URL: https://github.com/apache/airflow/pull/39912#discussion_r1682284981 ## airflow/models/taskinstance.py: ## @@ -1595,13 +1595,20 @@ def _defer_task( next_kwargs = exception.kwargs timeout = exception.timeout elif

Re: [PR] Add start execution from triggerer support to dynamic task mapping [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on PR #39912: URL: https://github.com/apache/airflow/pull/39912#issuecomment-2235789834 Probably shouldn’t be an approval (the exception class likely needs discussion, and I would prefer the function in AbstractOperator to be a part of this PR), but close enough. -- Th

Re: [I] airflow issue [airflow]

2024-07-18 Thread via GitHub
nathadfield closed issue #40849: airflow issue URL: https://github.com/apache/airflow/issues/40849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commi

Re: [I] airflow issue [airflow]

2024-07-18 Thread via GitHub
nathadfield commented on issue #40849: URL: https://github.com/apache/airflow/issues/40849#issuecomment-2235796610 Closing this as there is no information here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL a

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2235821170 @potiuk Is there a way to ignore the required license header in sql files within tests? Searched through the project but didn't find any example, saw other sql file and seemed that one w

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2235827478 Never mind the hive.sql also has the header, problem is I use the content of the sql for my assertion in the test, as I could have it as a python string as it would be sensitive to format

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1677325616 ## airflow/dag_processing/processor.py: ## @@ -762,8 +762,29 @@ def _execute_dag_callbacks(self, dagbag: DagBag, request: DagCallbackRequest, se if callba

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682363545 ## airflow/dag_processing/processor.py: ## @@ -762,8 +762,29 @@ def _execute_dag_callbacks(self, dagbag: DagBag, request: DagCallbackRequest, se if callbac

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2235841226 I've added the license header in the sql file and written a function which filters out the license header so the assertion still works -- This is an automated message from the Apache Gi

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682365659 ## airflow/dag_processing/processor.py: ## @@ -796,8 +818,16 @@ def _execute_task_callbacks(self, dagbag: DagBag | None, request: TaskCallbackRe if task:

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682367304 ## airflow/exceptions.py: ## @@ -372,9 +372,15 @@ class TaskDeferred(BaseException): Signal an operator moving to deferred state. Special exception raise

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682369622 ## airflow/models/taskinstance.py: ## @@ -715,6 +715,12 @@ def _execute_task(task_instance: TaskInstance | TaskInstancePydantic, context: C if not task

Re: [I] Release Airflow 3.0 [airflow]

2024-07-18 Thread via GitHub
gopidesupavan commented on issue #39593: URL: https://github.com/apache/airflow/issues/39593#issuecomment-2235849427 Hi @kaxil , Would like to take part of this Airflow 3 Journey , happy to contribute here. I can take look into this `Improvements to Sensors`. -- This is an automated mess

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682367304 ## airflow/exceptions.py: ## @@ -372,9 +372,15 @@ class TaskDeferred(BaseException): Signal an operator moving to deferred state. Special exception raise

Re: [PR] Add pre-commit to check attrs of TI and TIHistory [airflow]

2024-07-18 Thread via GitHub
ephraimbuddy commented on code in PR #40846: URL: https://github.com/apache/airflow/pull/40846#discussion_r1682396100 ## airflow/models/taskinstancehistory.py: ## @@ -44,8 +64,76 @@ class TaskInstanceHistory(Base): dag_id = Column(StringID(), nullable=False) run_id = C

Re: [PR] Make BashOperator compatible with Internal API AIP-44 [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40847: URL: https://github.com/apache/airflow/pull/40847#discussion_r1682408210 ## airflow/models/renderedtifields.py: ## @@ -153,6 +154,23 @@ def _redact(self): for field, rendered in self.rendered_fields.items(): self.ren

Re: [PR] Extend get datasets endpoint to include dataset aliases [airflow]

2024-07-18 Thread via GitHub
uranusjr commented on code in PR #40830: URL: https://github.com/apache/airflow/pull/40830#discussion_r1682426433 ## airflow/api_connexion/schemas/dataset_schema.py: ## @@ -59,6 +60,18 @@ class Meta: updated_at = auto_field() +class DatasetAliasSchema(SQLAlchemySchema):

Re: [PR] Resolve google cloud sql deprecation in tests [airflow]

2024-07-18 Thread via GitHub
eladkal merged PR #40834: URL: https://github.com/apache/airflow/pull/40834 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.

(airflow) branch main updated: Resolve CloudSQLDatabaseHook deprecation warning (#40834)

2024-07-18 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 67befb71bf Resolve CloudSQLDatabaseHook deprecatio

Re: [I] Making `mssql` provider to support only `mssql+pymssql` dialect [airflow]

2024-07-18 Thread via GitHub
dabla commented on issue #39586: URL: https://github.com/apache/airflow/issues/39586#issuecomment-2235950013 > @Taragolis , requesting your comments on this. > > I was working on updating `mssql` provider to use `sqlalchemy_url` property in `get_uri` and got the idea for this issue.

Re: [PR] Make BashOperator compatible with Internal API AIP-44 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40847: URL: https://github.com/apache/airflow/pull/40847#discussion_r1682454635 ## airflow/models/renderedtifields.py: ## @@ -153,6 +154,23 @@ def _redact(self): for field, rendered in self.rendered_fields.items(): self.rende

Re: [PR] Make BashOperator compatible with Internal API AIP-44 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40847: URL: https://github.com/apache/airflow/pull/40847#discussion_r1682454635 ## airflow/models/renderedtifields.py: ## @@ -153,6 +154,23 @@ def _redact(self): for field, rendered in self.rendered_fields.items(): self.rende

Re: [I] Making `mssql` provider to support only `mssql+pymssql` dialect [airflow]

2024-07-18 Thread via GitHub
dabla commented on issue #39586: URL: https://github.com/apache/airflow/issues/39586#issuecomment-2235959531 An overview of all enhancements and fixes I did regarding hooks: [40665](https://github.com/apache/airflow/pull/40665) [40669](https://github.com/apache/airflow/pull/40669)

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
pankajastro commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682428042 ## airflow/dag_processing/manager.py: ## @@ -580,6 +580,7 @@ def _run_parsing_loop(self): pass elif isinstance(agent_signal,

Re: [I] map_index_template not compute when task marked success manually [airflow]

2024-07-18 Thread via GitHub
raphaelauv commented on issue #39092: URL: https://github.com/apache/airflow/issues/39092#issuecomment-2236009489 @karenbraganz are you still working on this issue ? thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682529754 ## airflow/sensors/time_sensor.py: ## @@ -40,11 +40,11 @@ class TimeSensor(BaseSensorOperator): """ -def __init__(self, *, target_time: datetime.time,

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682530707 ## airflow/sensors/time_sensor.py: ## @@ -40,11 +41,11 @@ class TimeSensor(BaseSensorOperator): """ -def __init__(self, *, target_time: datetime.time,

[PR] openlineage: Add AirflowRunFacet for dag runEvents [airflow]

2024-07-18 Thread via GitHub
dolfinus opened a new pull request, #40854: URL: https://github.com/apache/airflow/pull/40854 closes: #40798 * Make `AirflowRunFacet` attributes `task`, `taskInstance` and `taskUuid` optional. * Add the same facet to DAG run events with eventType=STARTED. ---

Re: [I] Update web ui to allow for using strict content security policies [airflow]

2024-07-18 Thread via GitHub
vandanthaker-ey commented on issue #20934: URL: https://github.com/apache/airflow/issues/20934#issuecomment-2236207664 @potiuk there are still some references to inline scripts and styles and style attributes in latest version of airflow. due to which it's difficult to set strict CSP. which

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682545882 ## airflow/dag_processing/manager.py: ## @@ -580,6 +580,7 @@ def _run_parsing_loop(self): pass elif isinstance(agent_signal, C

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682673931 ## airflow/dag_processing/processor.py: ## @@ -763,8 +763,29 @@ def _execute_dag_callbacks(self, dagbag: DagBag, request: DagCallbackRequest, se if callba

[PR] Add changelog for airflow python client 2.9.3 [airflow]

2024-07-18 Thread via GitHub
utkarsharma2 opened a new pull request, #40857: URL: https://github.com/apache/airflow/pull/40857 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] Update web ui to allow for using strict content security policies [airflow]

2024-07-18 Thread via GitHub
potiuk commented on issue #20934: URL: https://github.com/apache/airflow/issues/20934#issuecomment-2236270394 Wher did you make an assumption the commit is merged from? This AIP is still in progress If you have not noticed. We are discussing it as part of Airlfow 2 workstream to comp

Re: [PR] openlineage: migrate OpenLineage provider to V2 facets. [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #39530: URL: https://github.com/apache/airflow/pull/39530#issuecomment-2236275910 almost :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubs

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682692872 ## airflow/dag_processing/processor.py: ## @@ -796,8 +818,16 @@ def _execute_task_callbacks(self, dagbag: DagBag | None, request: TaskCallbackRe if task:

Re: [PR] Add changelog for airflow python client 2.9.3 [airflow]

2024-07-18 Thread via GitHub
utkarsharma2 merged PR #40857: URL: https://github.com/apache/airflow/pull/40857 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@air

(airflow) branch main updated (67befb71bf -> 695542678b)

2024-07-18 Thread utkarsharma
This is an automated email from the ASF dual-hosted git repository. utkarsharma pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 67befb71bf Resolve CloudSQLDatabaseHook deprecation warning (#40834) add 695542678b Add changelog for airflow

(airflow) branch v2-9-test updated: Add changelog for airflow python client 2.9.3 (#40857)

2024-07-18 Thread utkarsharma
This is an automated email from the ASF dual-hosted git repository. utkarsharma pushed a commit to branch v2-9-test in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/v2-9-test by this push: new fa7cfbdbf5 Add changelog for airflow

Re: [PR] Generalize caching of connection in DbApiHook to improve performance [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40751: URL: https://github.com/apache/airflow/pull/40751#issuecomment-2236306401 I think the cause of the problem is that some tests effectively started to expect the Database - likely they simply now require and use session when they did not before. The advice to se

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow Part 2 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40802: URL: https://github.com/apache/airflow/pull/40802#issuecomment-2236308607 Marked it for 2.10.0 but some tests are still failing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo

Re: [PR] feat: support rendering ObjectStoragePath value [airflow]

2024-07-18 Thread via GitHub
potiuk merged PR #40638: URL: https://github.com/apache/airflow/pull/40638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [PR] feat: support rendering ObjectStoragePath value [airflow]

2024-07-18 Thread via GitHub
boring-cyborg[bot] commented on PR #40638: URL: https://github.com/apache/airflow/pull/40638#issuecomment-2236311297 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.

(airflow) branch main updated: feat: support rendering ObjectStoragePath value (#40638)

2024-07-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f88fdbabb1 feat: support rendering ObjectStoragePat

[PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
utkarsharma2 opened a new pull request, #40858: URL: https://github.com/apache/airflow/pull/40858 Time for python client 2.9.3rc1! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific com

Re: [I] ObjectStoragePath should be template-able [airflow]

2024-07-18 Thread via GitHub
potiuk closed issue #40361: ObjectStoragePath should be template-able URL: https://github.com/apache/airflow/issues/40361 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To uns

Re: [PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40858: URL: https://github.com/apache/airflow/pull/40858#issuecomment-2236316603 Ah. Would you also regenerate the docs for Airflow ? I see they have (TM) in 2.9.3 docs ? -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40858: URL: https://github.com/apache/airflow/pull/40858#issuecomment-2236320759 Actually I see there are still some places with TM. Let me add another PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[PR] Switch still some remaining places in Airflow docs with TM to (R) [airflow]

2024-07-18 Thread via GitHub
potiuk opened a new pull request, #40860: URL: https://github.com/apache/airflow/pull/40860 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

Re: [PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40858: URL: https://github.com/apache/airflow/pull/40858#issuecomment-2236326466 https://github.com/apache/airflow/pull/40860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40858: URL: https://github.com/apache/airflow/pull/40858#issuecomment-2236328935 It's the same process as in preparaing regular doc for release but this one shoud be used BTW. `breeze release-management publish-docs --override-versioned` -- This is an autom

Re: [PR] Sync v2-9-stable with v2-9-test to release python client v2.9.3 [airflow]

2024-07-18 Thread via GitHub
utkarsharma2 commented on PR #40858: URL: https://github.com/apache/airflow/pull/40858#issuecomment-2236329024 > Ah. Would you also regenerate the docs for Airflow ? I see they have (TM) in 2.9.3 docs ? Sure @potiuk, I'll regenerate them, once the PR is merged. -- This is an autom

[I] KubernetesExecutor does not save logs [airflow]

2024-07-18 Thread via GitHub
mrybas opened a new issue, #40861: URL: https://github.com/apache/airflow/issues/40861 ### Official Helm Chart version 1.14.0 (latest released) ### Apache Airflow version v2.9.3 ### Kubernetes Version v1.29.5 ### Helm Chart configuration ```

Re: [I] KubernetesExecutor does not save logs [airflow]

2024-07-18 Thread via GitHub
boring-cyborg[bot] commented on issue #40861: URL: https://github.com/apache/airflow/issues/40861#issuecomment-2236330356 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for ap

Re: [I] DatabricksRunNowOperator failing as named parameters Jinja templating not getting resolved [airflow]

2024-07-18 Thread via GitHub
Stormhand commented on issue #40788: URL: https://github.com/apache/airflow/issues/40788#issuecomment-2236337557 Hello, these changes also broke my code where im using jinja templating in the **notebook_params** for **DatabricksRunNowOperator** -- This is an automated message from the

Re: [PR] Switch still some remaining places in Airflow docs with TM to (R) [airflow]

2024-07-18 Thread via GitHub
potiuk merged PR #40860: URL: https://github.com/apache/airflow/pull/40860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Switch still some remaining places in Airflow docs with TM to (R) (#40860)

2024-07-18 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 54d1879c70 Switch still some remaining places in Ai

Re: [I] Add 'dag_id' and 'dag_run_id' to the default template variables [airflow]

2024-07-18 Thread via GitHub
pedro-cf closed issue #40859: Add 'dag_id' and 'dag_run_id' to the default template variables URL: https://github.com/apache/airflow/issues/40859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] DatabricksRunNowOperator failing as named parameters Jinja templating not getting resolved [airflow]

2024-07-18 Thread via GitHub
vatsrahul1001 commented on issue #40788: URL: https://github.com/apache/airflow/issues/40788#issuecomment-2236360577 @boraberke are you working on a fix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] Task with "wait_for_downstream" doesn't wait, because start_date is mishandled [airflow]

2024-07-18 Thread via GitHub
shai-ikko commented on issue #40841: URL: https://github.com/apache/airflow/issues/40841#issuecomment-2236365767 There may have been some misunderstanding here - I am willing, in principle, to submit a PR, but I have no clue about the fix. The bug submission form promised that guidance may

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2236366630 It's a pitty that I can't use the functools.lru_cache decorator? I could use the [cachetools cache decorator](https://cachetools.readthedocs.io/en/latest/), but then that would add an e

Re: [PR] AIP-62: add hook lineage collection support for S3Hook [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40819: URL: https://github.com/apache/airflow/pull/40819#discussion_r1682749953 ## airflow/lineage/hook.py: ## @@ -139,10 +139,10 @@ class NoOpCollector(HookLineageCollector): It is used when you want to disable lineage collection. """

Re: [PR] AIP-62: add hook lineage collection support for S3Hook [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40819: URL: https://github.com/apache/airflow/pull/40819#discussion_r1682748569 ## airflow/lineage/hook.py: ## @@ -139,10 +139,10 @@ class NoOpCollector(HookLineageCollector): It is used when you want to disable lineage collection. """

Re: [PR] AIP-62: add hook lineage collection support for S3Hook [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40819: URL: https://github.com/apache/airflow/pull/40819#discussion_r1682754790 ## generated/provider_dependencies.json: ## @@ -57,6 +57,7 @@ "cross-providers-deps": [ "apache.hive", "cncf.kubernetes", + "common.compat", R

(airflow) branch aip-62/object-storage updated (ca81f78e17 -> 6a1308908e)

2024-07-18 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch aip-62/object-storage in repository https://gitbox.apache.org/repos/asf/airflow.git omit ca81f78e17 openlineage: add support for hook lineage for Object Store add 6a1308908e openlin

Re: [PR] AIP-62: add hook lineage collection support for S3Hook [airflow]

2024-07-18 Thread via GitHub
potiuk commented on code in PR #40819: URL: https://github.com/apache/airflow/pull/40819#discussion_r1682754790 ## generated/provider_dependencies.json: ## @@ -57,6 +57,7 @@ "cross-providers-deps": [ "apache.hive", "cncf.kubernetes", + "common.compat", R

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2236391589 > It's a pitty that I can't use the functools.lru_cache decorator? I could use the [cachetools cache decorator](https://cachetools.readthedocs.io/en/latest/), but then that would add an

Re: [I] OpenLineage: Add AirflowRunFacet to DAG run events [airflow]

2024-07-18 Thread via GitHub
dolfinus commented on issue #40798: URL: https://github.com/apache/airflow/issues/40798#issuecomment-2236406223 Implementation: #40854 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Setting podAnnotations for workers also adds them to other deployments such as scheduler [airflow]

2024-07-18 Thread via GitHub
potiuk commented on issue #40804: URL: https://github.com/apache/airflow/issues/40804#issuecomment-2236409034 Anyone willing to provide a fix are welcome to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[PR] Add a check in BigQueryInsertJobOperator to verify the Job state before marking it as success [airflow]

2024-07-18 Thread via GitHub
kandharvishnu opened a new pull request, #40863: URL: https://github.com/apache/airflow/pull/40863

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow Part 2 [airflow]

2024-07-18 Thread via GitHub
howardyoo commented on PR #40802: URL: https://github.com/apache/airflow/pull/40802#issuecomment-2236427079 Thank you, Jarek.Will take a look at them ASAP.Sent from my iPhoneOn Jul 18, 2024, at 6:53 AM, Jarek Potiuk ***@***.***> wrote: Marked it for 2.10.0 but some tests are still failin

Re: [PR] openlineage: migrate OpenLineage provider to V2 facets. [airflow]

2024-07-18 Thread via GitHub
JDarDagran commented on PR #39530: URL: https://github.com/apache/airflow/pull/39530#issuecomment-2236427967 I think I need to drop tests checking what is imported with different availability of OL client. mocking imports breaks other tests when running in parallel. -- This is an automat

Re: [I] Remove StatsD and replace it with Open Telemetry as first-class citizen [airflow]

2024-07-18 Thread via GitHub
dirrao commented on issue #40800: URL: https://github.com/apache/airflow/issues/40800#issuecomment-2236430052 Hi @josix Let me know if you need any help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

(airflow) branch aip-62/object-storage updated (6a1308908e -> 2313f98984)

2024-07-18 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch aip-62/object-storage in repository https://gitbox.apache.org/repos/asf/airflow.git discard 6a1308908e openlineage: add support for hook lineage for Object Store add 2313f98984 openlin

[PR] Fix named parameters templating in Databricks operators [airflow]

2024-07-18 Thread via GitHub
boraberke opened a new pull request, #40864: URL: https://github.com/apache/airflow/pull/40864 This PR fixes the many named parameters that was templated and was broken with #40471. The following operators are affected: - DatabricksCreateJobsOperator - DatabricksSubmitRunOpe

(airflow) branch constraints-main updated: Updating constraints. Github run id:9991321534

2024-07-18 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 0f4e75e5fa Updating con

Re: [PR] openlineage: Add AirflowRunFacet for dag runEvents [airflow]

2024-07-18 Thread via GitHub
mobuchowski commented on PR #40854: URL: https://github.com/apache/airflow/pull/40854#issuecomment-2236446811 @dolfinus I think this should be a different facet or added to `AirflowJobFacet` that we attach to DagRun events. -- This is an automated message from the Apache Git Service. To r

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682813058 ## airflow/sensors/time_sensor.py: ## @@ -72,13 +73,8 @@ def __init__(self, *, target_time: datetime.time, **kwargs) -> None: self.target_datetime = tim

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682813959 ## airflow/sensors/time_delta.py: ## @@ -73,14 +73,10 @@ def execute(self, context: Context) -> bool | NoReturn: # If the target datetime is in the pas

Re: [PR] openlineage: Add AirflowRunFacet for dag runEvents [airflow]

2024-07-18 Thread via GitHub
dolfinus commented on PR #40854: URL: https://github.com/apache/airflow/pull/40854#issuecomment-2236466647 > Added to AirflowJobFacet that we attach to DagRun events. DagRun information is a part of RunFacet, and cannot be JobFacet. > I think this should be a different facet

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682818170 ## airflow/exceptions.py: ## @@ -372,9 +372,15 @@ class TaskDeferred(BaseException): Signal an operator moving to deferred state. Special exception rais

Re: [PR] openlineage: Add AirflowRunFacet for dag runEvents [airflow]

2024-07-18 Thread via GitHub
mobuchowski commented on PR #40854: URL: https://github.com/apache/airflow/pull/40854#issuecomment-2236502492 Why not attach it to some new facet as `AirflowDagRunFacet`? Basically, `AirflowRunFacet` is tightly coupled to what we have during task instance execution, and don't think making i

Re: [PR] Make task go to terminal state from triggerer without needing a worker [airflow]

2024-07-18 Thread via GitHub
sunank200 commented on code in PR #40084: URL: https://github.com/apache/airflow/pull/40084#discussion_r1682831071 ## airflow/dag_processing/processor.py: ## @@ -763,8 +763,29 @@ def _execute_dag_callbacks(self, dagbag: DagBag, request: DagCallbackRequest, se if callba

Re: [PR] openlineage: Add AirflowRunFacet for dag runEvents [airflow]

2024-07-18 Thread via GitHub
dolfinus commented on PR #40854: URL: https://github.com/apache/airflow/pull/40854#issuecomment-2236511831 > Why not attach it to some new facet as AirflowDagRunFacet? What should be the key of this facet? `"airflow"` or something else? For OL consumer, having runEvents with facets wh

Re: [PR] Added support for replace option when using insert_rows with MsSqlHook [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40836: URL: https://github.com/apache/airflow/pull/40836#issuecomment-2236516883 > > It's a pitty that I can't use the functools.lru_cache decorator? I could use the [cachetools cache decorator](https://cachetools.readthedocs.io/en/latest/), but then that would add a

Re: [I] Remove StatsD and replace it with Open Telemetry as first-class citizen [airflow]

2024-07-18 Thread via GitHub
raphaelauv commented on issue #40800: URL: https://github.com/apache/airflow/issues/40800#issuecomment-2236525455 also provide a simple grafana dashboard in the example stack ( cause the main airflow grafana dashboard open source is based on statsd ) would help a lot -- This is an autom

[PR] Bump packaging version to 23.0 in order to fix issue with older otel [airflow]

2024-07-18 Thread via GitHub
potiuk opened a new pull request, #40865: URL: https://github.com/apache/airflow/pull/40865 The packaging 22.0 installed together with older version of opentelemetry libraries breaks context retrieval. It works with newer versions of otel libraries, but it seems reasonable to limit packagin

Re: [PR] Add task duration plot across dagruns [airflow]

2024-07-18 Thread via GitHub
tirkarthi commented on PR #40755: URL: https://github.com/apache/airflow/pull/40755#issuecomment-2236537735 > > Clicking on the data point takes the user to specific task instance detail tab. > > ❤️ this. Do you predict the user clicking this would be more interested in the details t

Re: [PR] Configure HttpHook's auth_type from Connection [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-2236552399 > > Af few providers are failing due to the get_extra_json method fom above #39811, which means those providers would be dependant on a newer Airflow version. > > We should definit

Re: [PR] Add task duration plot across dagruns [airflow]

2024-07-18 Thread via GitHub
tirkarthi commented on PR #40755: URL: https://github.com/apache/airflow/pull/40755#issuecomment-2236582957 It looks easier to implement switching between line and bar chart. In the latest commit I have added a checkbox similar to "show landing time" in run duration without persistence to m

Re: [PR] Configure HttpHook's auth_type from Connection [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-2236599263 > > > Af few providers are failing due to the get_extra_json method fom above #39811, which means those providers would be dependant on a newer Airflow version. > > > > > > We

Re: [PR] Add task duration plot across dagruns [airflow]

2024-07-18 Thread via GitHub
tirkarthi commented on PR #40755: URL: https://github.com/apache/airflow/pull/40755#issuecomment-2236599904 > Love the idea of showing task duration even when nothing is selected. > I'm still partial to a bar chart over a line graph. The discrete task instance or dag run is more important

Re: [PR] Add task duration plot across dagruns [airflow]

2024-07-18 Thread via GitHub
tirkarthi commented on PR #40755: URL: https://github.com/apache/airflow/pull/40755#issuecomment-2236604813 @bbovenzi @ketozhang If someone can also help in fixing the typescript issue in the PR description moment import that would also help in making the PR green for further review. I am

Re: [PR] Configure HttpHook's auth_type from Connection [airflow]

2024-07-18 Thread via GitHub
potiuk commented on PR #35591: URL: https://github.com/apache/airflow/pull/35591#issuecomment-2236611750 > > > > Af few providers are failing due to the get_extra_json method fom above #39811, which means those providers would be dependant on a newer Airflow version. > > > > > >

[PR] Replace usages of task context logger with the log table [airflow]

2024-07-18 Thread via GitHub
dstandish opened a new pull request, #40867: URL: https://github.com/apache/airflow/pull/40867 Shipping messages to TI logs, which unfortunately in many cases means uploading to s3, is inefficient, and not really something the scheduler should be doing. Instead we can use an old but

(airflow) branch v2-9-test updated (fa7cfbdbf5 -> 1bf613e9c0)

2024-07-18 Thread utkarsharma
This is an automated email from the ASF dual-hosted git repository. utkarsharma pushed a change to branch v2-9-test in repository https://gitbox.apache.org/repos/asf/airflow.git from fa7cfbdbf5 Add changelog for airflow python client 2.9.3 (#40857) add d833b091f6 Switch still some remai

Re: [I] map_index_template not compute when task marked success manually [airflow]

2024-07-18 Thread via GitHub
karenbraganz commented on issue #39092: URL: https://github.com/apache/airflow/issues/39092#issuecomment-2236647566 I created a draft PR but was informed that rendering the Jinja template on the webserver is not viable. @uranusjr could you please clarify why this is the case? I accidentally

Re: [PR] Microsoft Power BI operator to refresh the dataset [airflow]

2024-07-18 Thread via GitHub
dabla commented on PR #40356: URL: https://github.com/apache/airflow/pull/40356#issuecomment-2236651981 > Hey @dabla, I added the Triggers class to extend the operator to works in deferable mode while extending async calls to MS graph operator. now this operator could support both async and

Re: [I] UI indication for parallelism, concurrency, or pool limit reached [airflow]

2024-07-18 Thread via GitHub
eladkal closed issue #18378: UI indication for parallelism, concurrency, or pool limit reached URL: https://github.com/apache/airflow/issues/18378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] UI indication for parallelism, concurrency, or pool limit reached [airflow]

2024-07-18 Thread via GitHub
eladkal commented on issue #18378: URL: https://github.com/apache/airflow/issues/18378#issuecomment-2236654875 Fixed in https://github.com/apache/airflow/pull/38449 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

(airflow) branch aip-62/object-storage updated (2313f98984 -> b4dd98d009)

2024-07-18 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch aip-62/object-storage in repository https://gitbox.apache.org/repos/asf/airflow.git discard 2313f98984 openlineage: add support for hook lineage for Object Store add b4dd98d009 openlin

  1   2   3   >