[I] BashOperator Windows compatibility [airflow]

2024-07-26 Thread via GitHub
SledgeHammer01 opened a new issue, #41063: URL: https://github.com/apache/airflow/issues/41063 ### Apache Airflow version 2.9.3 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? When attempting to run the simple tutorial

Re: [I] BashOperator Windows compatibility [airflow]

2024-07-26 Thread via GitHub
boring-cyborg[bot] commented on issue #41063: URL: https://github.com/apache/airflow/issues/41063#issuecomment-2253755140 Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for

[PR] fix: add argument include_xcom in method rsolve an optional value [airflow]

2024-07-26 Thread via GitHub
Lee-W opened a new pull request, #41062: URL: https://github.com/apache/airflow/pull/41062 The https://github.com/apache/airflow/pull/39912, we thought `resolve` is an internal method and added an additional required keyword argument. However, we found out some existing DAGs broke due to

Re: [I] Consolidate "Serialization" code [airflow]

2024-07-26 Thread via GitHub
gyli commented on issue #40974: URL: https://github.com/apache/airflow/issues/40974#issuecomment-2253725504 Thanks for the explanation! There are 3 points that I really agree, and can explore in this issue: 1. It makes more sense to move serde modules to providers, and I also think it

[PR] SSHHook: check if existing connection is still alive [airflow]

2024-07-26 Thread via GitHub
dolfinus opened a new pull request, #41061: URL: https://github.com/apache/airflow/pull/41061 #40377 started reusing in `get_conn()` the same paramiko SSHClient, which drastically improves performance of sequential execution of SSH commands. But connection could be already

Re: [PR] Support adding custom TI Deps to help DagRun make more flexible TI scheduling decisions [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] closed pull request #37778: Support adding custom TI Deps to help DagRun make more flexible TI scheduling decisions URL: https://github.com/apache/airflow/pull/37778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] DataprocCreateClusterOperator doesn't read softwareConfig properties as it should [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] closed issue #38127: DataprocCreateClusterOperator doesn't read softwareConfig properties as it should URL: https://github.com/apache/airflow/issues/38127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] Chart Postgres:15.4.1 and airflow default version description: 2.9.1 update [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] closed pull request #39879: Chart Postgres:15.4.1 and airflow default version description: 2.9.1 update URL: https://github.com/apache/airflow/pull/39879 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] DataprocCreateClusterOperator doesn't read softwareConfig properties as it should [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] commented on issue #38127: URL: https://github.com/apache/airflow/issues/38127#issuecomment-2253667347 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] Add config to avoid one DAG run when unpausing a DAG with catchup=False [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] commented on PR #38168: URL: https://github.com/apache/airflow/pull/38168#issuecomment-2253667323 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] doc: helm chart airflow version [airflow]

2024-07-26 Thread via GitHub
github-actions[bot] commented on PR #40087: URL: https://github.com/apache/airflow/pull/40087#issuecomment-2253667279 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for

Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on code in PR #40924: URL: https://github.com/apache/airflow/pull/40924#discussion_r1693641088 ## airflow/sensors/base.py: ## @@ -114,15 +118,33 @@ def _orig_start_date( ) +class SkipPolicy(str, enum.Enum): +"""Class with sensor's skip

Re: [PR] Refactor `BaseSensorOperator` introduce `skip_policy` parameter [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on code in PR #40924: URL: https://github.com/apache/airflow/pull/40924#discussion_r1693641088 ## airflow/sensors/base.py: ## @@ -114,15 +118,33 @@ def _orig_start_date( ) +class SkipPolicy(str, enum.Enum): +"""Class with sensor's skip

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253563168 Okay .Also all the providers are affected , could you confirm me that the design is okay before I do the big refacto , thanks -- This is an automated message from the Apache Git

Re: [I] Windows support for Airflow [airflow]

2024-07-26 Thread via GitHub
SledgeHammer01 commented on issue #10388: URL: https://github.com/apache/airflow/issues/10388#issuecomment-2253540208 in subprocess.py, this line in `SubprocessHook::run_command` also causes an issue. `cwd = stack.enter_context(TemporaryDirectory(prefix="airflowtmp"))`

(airflow) branch main updated: Update VertexAI system tests (#41052)

2024-07-26 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 047d139549 Update VertexAI system tests (#41052)

Re: [PR] Update VertexAI system tests [airflow]

2024-07-26 Thread via GitHub
eladkal merged PR #41052: URL: https://github.com/apache/airflow/pull/41052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Make EMR Container Trigger max attempts retries match the Operator (#41008)

2024-07-26 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new c519920661 Make EMR Container Trigger max

Re: [I] EmrContainerOperator in Async mode doesn't respect default "infinite" polling number [airflow]

2024-07-26 Thread via GitHub
vincbeck closed issue #40483: EmrContainerOperator in Async mode doesn't respect default "infinite" polling number URL: https://github.com/apache/airflow/issues/40483 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Make EMR Container Trigger max attempts retries match the Operator [airflow]

2024-07-26 Thread via GitHub
vincbeck merged PR #41008: URL: https://github.com/apache/airflow/pull/41008 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated: Fix `RdsStopDbOperator` operator in deferrable mode (#41059)

2024-07-26 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 83ca61a501 Fix `RdsStopDbOperator` operator in

Re: [PR] Fix `RdsStopDbOperator` operator in deferrable mode [airflow]

2024-07-26 Thread via GitHub
vincbeck merged PR #41059: URL: https://github.com/apache/airflow/pull/41059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Align timers and timing metrics (ms) across all metrics loggers [airflow]

2024-07-26 Thread via GitHub
ferruzzi commented on PR #39908: URL: https://github.com/apache/airflow/pull/39908#issuecomment-2253458521 I'm fine with millis. My main problem is standardizing on the statsd and breaking everyone else. In fact, after the last Airflow 3 dev call where they were discussing dropping

(airflow) branch constraints-main updated: Updating constraints. Github run id:10116617753

2024-07-26 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 49b99b5a2b Updating

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253402552 The code Ilooks good - just make it into common.compat provider and use it from there for every provider that is affected (and add dependency to "apache-airflow-provder-common-compat"

(airflow) branch main updated: Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new f6c7388cfa Create SQLAlchemy engine from

Re: [PR] Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method [airflow]

2024-07-26 Thread via GitHub
potiuk merged PR #40669: URL: https://github.com/apache/airflow/pull/40669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] fix: call listener manager on_task_instance_failed() after ti state is changed [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41053: URL: https://github.com/apache/airflow/pull/41053#issuecomment-2253347350 Makes sense. But would love other look :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Tasks are stuck in scheduled state when AIRFLOW__CORE__PARALLELISM env is set to 0 in Kubernetes Executor [airflow]

2024-07-26 Thread via GitHub
shubham22 commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2253340939 @vatsrahul1001 - do we actually document that 0 is unlimited parallelism? From product point of view, I don't think we need to actually support it. It is always a good idea to

Re: [PR] Change `task_id` from `send_email` to `send_email_notification` in `taskflow.rst` [airflow]

2024-07-26 Thread via GitHub
shahar1 merged PR #41060: URL: https://github.com/apache/airflow/pull/41060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (1ff3717f35 -> 1def3f151a)

2024-07-26 Thread shahar
This is an automated email from the ASF dual-hosted git repository. shahar pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 1ff3717f35 Update "what goes to next release" process (#41058) add 1def3f151a Change `task_id` from `send_email`

Re: [PR] Change `task_id` from `send_email` to `send_email_notification` in `taskflow.rst` [airflow]

2024-07-26 Thread via GitHub
boring-cyborg[bot] commented on PR #41060: URL: https://github.com/apache/airflow/pull/41060#issuecomment-2253339517 Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional

Re: [I] Unable to pass {username} with slashes to Airflow REST API [airflow]

2024-07-26 Thread via GitHub
jaklan commented on issue #39887: URL: https://github.com/apache/airflow/issues/39887#issuecomment-2253329133 @Taragolis any hints how to resolve that issue? Also encounter that problem on MWAA because of `assumed-role/` prefix -- This is an automated message from the Apache Git Service.

Re: [I] Tasks are stuck in scheduled state when AIRFLOW__CORE__PARALLELISM env is set to 0 in Kubernetes Executor [airflow]

2024-07-26 Thread via GitHub
o-nikolas commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2253322717 Just to add to that, the metrics code will also log this when parallelism is set to 0. So multiple things need to be fixed if we wish to support this: > INFO - Executor

Re: [I] Tasks are stuck in scheduled state when AIRFLOW__CORE__PARALLELISM env is set to 0 in Kubernetes Executor [airflow]

2024-07-26 Thread via GitHub
o-nikolas commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2253320678 @ephraimbuddy I was able to reproduce this with another executor (SequentialExecutor) and I get the same behaviour. I looked more closely at our docs and configuration

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253311221 We could create a new DeprecatedAirflowCoreException that we ignore in unit tests and that get show to users of "old" providers versions WDYT ? and if no , does my

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2253302264 > Perfect, thanks for the clarification. I got confused  It happens once every blue moon, so yeah that's easy to understand :) -- This is an automated message from the Apache

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253289635 > but that mean only operators from official providers of the apache-airflow projects will show the warnings , is not that a problem ? Well. It's logically impossible to influence

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
kacpermuda commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2253233559 Perfect, thanks for the clarification. I got confused  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Fix default behaviour for init function in AppriseNotifier [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on code in PR #41054: URL: https://github.com/apache/airflow/pull/41054#discussion_r1693417148 ## airflow/providers/apprise/notifications/apprise.py: ## @@ -52,9 +51,9 @@ def __init__( *, body: str, title: str | None = None, -

[PR] Changed tas_id from send_email to send_email_notification [airflow]

2024-07-26 Thread via GitHub
jay-law opened a new pull request, #41060: URL: https://github.com/apache/airflow/pull/41060 Renaming the `task_id` within the `EmailOperator` example from `send_email` to `send_email_notification`. `send_email` is an existing method within `airflow.utils.email`. Changing the duplicate

Re: [PR] Changed tas_id from send_email to send_email_notification [airflow]

2024-07-26 Thread via GitHub
boring-cyborg[bot] commented on PR #41060: URL: https://github.com/apache/airflow/pull/41060#issuecomment-2253214951 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253199584 but that mean only operators from official providers of the apache-airflow projects will show the warnings , is not that a problem ? -- This is an automated message from the

Re: [I] Add `dag_id` and `map_index` to the default template variables [airflow]

2024-07-26 Thread via GitHub
pedro-cf commented on issue #40958: URL: https://github.com/apache/airflow/issues/40958#issuecomment-2253154059 > There's currently an open PR (#41039) for solving #34158. When this PR is merged, will there still be a justification for the suggested feature? I believe so, these are

Re: [I] Add `dag_id` and `map_index` to the default template variables [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on issue #40958: URL: https://github.com/apache/airflow/issues/40958#issuecomment-2253144024 > > 1. Is it a feature suggestion, or a suggestion for improving the documentation? > > > > Please clarify the description - it would also be useful to provide some code

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253142027 You can't do it in BaseOperator. The compatibility code shoudl be implemented in those operators that gets incompatible (because eventually you will have - say - 2.7.1 BaseSensor and

Re: [PR] Build documentation in canary runs using ASF self-hosted AMD runners [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41057: URL: https://github.com/apache/airflow/pull/41057#issuecomment-2253115645 Something does not work :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
potiuk merged PR #41058: URL: https://github.com/apache/airflow/pull/41058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (29610ded9f -> 1ff3717f35)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 29610ded9f Increase frequency of canary runs to 4xDay (#41056) add 1ff3717f35 Update "what goes to next release"

Re: [I] Add `dag_id` and `map_index` as default variables to https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html [airflow]

2024-07-26 Thread via GitHub
pedro-cf commented on issue #40958: URL: https://github.com/apache/airflow/issues/40958#issuecomment-2253111744 > 1. Is it a feature suggestion, or a suggestion for improving the documentation? > > Please clarify the description - it would also be useful to provide some code

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253111327 thanks @potiuk for the guidance ,but I still don't know how to do that in the BaseSensorOperator. I'm quit lost :sweat_smile: -- This is an automated message from the Apache Git

Re: [I] Add `dag_id` and `map_index` as default variables to https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on issue #40958: URL: https://github.com/apache/airflow/issues/40958#issuecomment-2253103734 1. Is it a feature suggestion, or a suggestion for improving the documentation? Please clarify the description - it would also be useful to provide some code examples that will

Re: [PR] Make EMR Container Trigger max attempts retries match the Operator [airflow]

2024-07-26 Thread via GitHub
akomisarek commented on PR #41008: URL: https://github.com/apache/airflow/pull/41008#issuecomment-2253097655 Thanks, a lot it seems it would solve part of our problems! :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[PR] Fix `RdsStopDbOperator` operator in deferrable mode [airflow]

2024-07-26 Thread via GitHub
vincbeck opened a new pull request, #41059: URL: https://github.com/apache/airflow/pull/41059 The operator `RdsStopDbOperator` when used in deferrable mode does not work it uses a waiter that does not exist. I also fixed some inconsistencies in RDS operators

Re: [I] bigquery_to_postgres performance issue [airflow]

2024-07-26 Thread via GitHub
shahar1 closed issue #40948: bigquery_to_postgres performance issue URL: https://github.com/apache/airflow/issues/40948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] bigquery_to_postgres performance issue [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on issue #40948: URL: https://github.com/apache/airflow/issues/40948#issuecomment-2253090278 I have to agree with @raphaelauv - Airflow is not suitable for this scale of data transfers, and other technologies should be considered, such as Apache Beam or Apache Spark.

(airflow) branch main updated (f0ef69198e -> 29610ded9f)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f0ef69198e Check dataset_alias exists in inlets when use it to retrieve inlet_events (#41043) add 29610ded9f

Re: [PR] Increase frequency of canary runs to 4xDay [airflow]

2024-07-26 Thread via GitHub
potiuk merged PR #41056: URL: https://github.com/apache/airflow/pull/41056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on code in PR #41058: URL: https://github.com/apache/airflow/pull/41058#discussion_r1693308184 ## README.md: ## @@ -477,17 +477,34 @@ Generally we release `MINOR` versions of Airflow from a branch that is named aft `2.7.*` releases are released from

Re: [PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on code in PR #41058: URL: https://github.com/apache/airflow/pull/41058#discussion_r1693299174 ## README.md: ## @@ -477,17 +477,34 @@ Generally we release `MINOR` versions of Airflow from a branch that is named aft `2.7.*` releases are released from

Re: [PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
shahar1 commented on code in PR #41058: URL: https://github.com/apache/airflow/pull/41058#discussion_r1693299174 ## README.md: ## @@ -477,17 +477,34 @@ Generally we release `MINOR` versions of Airflow from a branch that is named aft `2.7.*` releases are released from

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2253043930 https://github.com/apache/airflow/pull/41058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
potiuk opened a new pull request, #41058: URL: https://github.com/apache/airflow/pull/41058 --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] Update "what goes to next release" process [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41058: URL: https://github.com/apache/airflow/pull/41058#issuecomment-2253044288 cc: @kacpermuda -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2253017856 > It won't get lost. Until 9th of August (RC1 for 2.10.0 planned this day) the v2-10-test branch will continue to be rebased on main. What is going to be merged till then is going to be

Re: [I] duplicate key value violates unique constraint serialized_dag_pkey for dynamic dag generation [airflow]

2024-07-26 Thread via GitHub
millin commented on issue #40082: URL: https://github.com/apache/airflow/issues/40082#issuecomment-2253016337 I did a research. This error is the result of multiple dag-processors simultaneously trying to insert a new `serialized dag` into the table. After some time, one of the

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2253014349 It won't get lost. Until 9th of August (RC1 for 2.10.0 planned this day) the v2-10-test branch will continue to be rebased on main. What is going to be merged till then is going to be

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2253005692 Yes. You cannot have a warning for 2.7-2.9 and do not have an option to remove the warning other than upgrading. Yes the idea is simple. Check airlfow version and do it

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2252990180 @potiuk should I make the warning conditional to the minimal airflow-core compatible version of the package of the super class ? it yes , do you have an elegant idea how ?

Re: [PR] Increase frequency of canary runs to 4xDay [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41056: URL: https://github.com/apache/airflow/pull/41056#issuecomment-2252989789 Related #40925 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] Build documentation in canary runs using ASF self-hosted AMD runners [airflow]

2024-07-26 Thread via GitHub
potiuk opened a new pull request, #41057: URL: https://github.com/apache/airflow/pull/41057 The runners have more disk space which is needed to checkout and build full documentation. Related: #40925 --- **^ Add meaningful description above**

Re: [PR] full refacto sensors error management [airflow]

2024-07-26 Thread via GitHub
raphaelauv commented on PR #41047: URL: https://github.com/apache/airflow/pull/41047#issuecomment-2252968691 @potiuk what do you recommend to show the warnings only when providers use at least airflow 2.10.0 ? thanks -- This is an automated message from the Apache Git Service. To respond

[PR] Increase frequency of canary runs to 4xDay [airflow]

2024-07-26 Thread via GitHub
potiuk opened a new pull request, #41056: URL: https://github.com/apache/airflow/pull/41056 We have not seen a significant increase in a number of FT runners used when we run Canary builds 2x day, switching it to 4x day will speed up constraints and cache refresh as well as will allow to

Re: [PR] AIP-64: Add task instance history list endpoint [airflow]

2024-07-26 Thread via GitHub
ephraimbuddy commented on code in PR #40988: URL: https://github.com/apache/airflow/pull/40988#discussion_r1693225577 ## airflow/www/static/js/types/api-generated.ts: ## @@ -4347,6 +4397,69 @@ export interface operations { 404: components["responses"]["NotFound"];

Re: [PR] Reassign downstream and upstream task ids after operator's unmapping [airflow]

2024-07-26 Thread via GitHub
kacpermuda commented on PR #40722: URL: https://github.com/apache/airflow/pull/40722#issuecomment-2252951551 @uranusjr @potiuk Any way we could merge this one, or at least assign Airflow2.10 milestone to it so it doesn't get lost somewhere?  -- This is an automated message from the

Re: [PR] AIP-64: Add task instance history list endpoint [airflow]

2024-07-26 Thread via GitHub
bbovenzi commented on code in PR #40988: URL: https://github.com/apache/airflow/pull/40988#discussion_r1693221007 ## airflow/www/static/js/types/api-generated.ts: ## @@ -4347,6 +4397,69 @@ export interface operations { 404: components["responses"]["NotFound"]; };

Re: [PR] Send context using in venv operator [airflow]

2024-07-26 Thread via GitHub
41 UTC] {process_utils.py:191} INFO - 'ds_nodash': '20240726', [2024-07-26, 14:57:41 UTC] {process_utils.py:191} INFO - 'expanded_ti_count': None, [2024-07-26, 14:57:41 UTC] {process_utils.py:191} INFO - 'logical_date': '2024-07-26T14:57:31.727299+00:00', [2024-07-26, 14:57:

(airflow) branch main updated (277e746fa4 -> f0ef69198e)

2024-07-26 Thread weilee
This is an automated email from the ASF dual-hosted git repository. weilee pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 277e746fa4 openlineage: update docs on openlineage methods (#41051) add f0ef69198e Check dataset_alias exists in

Re: [PR] Check dataset_alias in inlets when use it to retrieve inlet_evnets [airflow]

2024-07-26 Thread via GitHub
Lee-W merged PR #41043: URL: https://github.com/apache/airflow/pull/41043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] EmrCreateJobFlowOperator should not wait_for_completion if wait_for_completion=false deferrable=true [airflow]

2024-07-26 Thread via GitHub
Prab-27 commented on issue #40966: URL: https://github.com/apache/airflow/issues/40966#issuecomment-225288 I am using WSL2 set up can I work on this issue here OS is Mac ? .If yes , I would like to work on this issue as my first contribution. -- This is an automated message from

Re: [I] Tasks are stuck in scheduled state when AIRFLOW__CORE__PARALLELISM env is set to 0 in Kubernetes Executor [airflow]

2024-07-26 Thread via GitHub
ephraimbuddy commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2252884635 cc @o-nikolas . It seems related to your PR on multi-executors https://github.com/apache/airflow/pull/40017 and is a breaking change -- This is an automated message from

Re: [PR] Support multiple XCom output in the BaseOperator [airflow]

2024-07-26 Thread via GitHub
jannisko commented on PR #37297: URL: https://github.com/apache/airflow/pull/37297#issuecomment-2252886312 @joycechan-KSO yes this feature was available for the [TaskFlow](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html) style operators already, but missing for

[I] Tasks are stuck in scheduled state when AIRFLOW__CORE__PARALLELISM env is set to 0 in Kubernetes Executor [airflow]

2024-07-26 Thread via GitHub
vatsrahul1001 opened a new issue, #41055: URL: https://github.com/apache/airflow/issues/41055 ### Apache Airflow version 2.10.0b1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Tasks are stuck in the scheduled state

[PR] Fix default behaviour for init function in AppriseNotifier [airflow]

2024-07-26 Thread via GitHub
romsharon98 opened a new pull request, #41054: URL: https://github.com/apache/airflow/pull/41054 closes: https://github.com/apache/airflow/pull/38593/ --- **^ Add meaningful description above** Read the **[Pull Request

Re: [PR] openlineage: update docs on openlineage methods [airflow]

2024-07-26 Thread via GitHub
mobuchowski merged PR #41051: URL: https://github.com/apache/airflow/pull/41051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

(airflow) branch main updated (592eafa82a -> 277e746fa4)

2024-07-26 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 592eafa82a openlineage, docs: add missing execution_timeout conf value (#41042) add 277e746fa4 openlineage:

[PR] fix: call listener manager on_task_instance_failed() after ti state is changed [airflow]

2024-07-26 Thread via GitHub
kacpermuda opened a new pull request, #41053: URL: https://github.com/apache/airflow/pull/41053 Now we are calling `get_listener_manager().hook.on_task_instance_failed` with `ti` that still has the `RUNNING` state assigned. By using only `ti.state`, there is no way of knowing

Re: [PR] Bring back ARM image building using ASF self-hosted ARM runners [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41049: URL: https://github.com/apache/airflow/pull/41049#issuecomment-2252837815 Unfortunately - this one will have to wait until ASF runners are migrated to Ubuntu 22.04 - because Ubuntu 20.04 does not have installers for ARM Python. -- This is an automated

Re: [PR] [Databricks Provider] Revert PRs #40864 and #40471 [airflow]

2024-07-26 Thread via GitHub
boraberke commented on PR #41050: URL: https://github.com/apache/airflow/pull/41050#issuecomment-2252806242 > cc: @boraberke for your information. I guess it would be better if we rework on the earlier issue peacefully and clear the path for the upcoming release. Hope you're comfortable

[PR] Update VertexAI system tests [airflow]

2024-07-26 Thread via GitHub
MaksYermak opened a new pull request, #41052: URL: https://github.com/apache/airflow/pull/41052 In this PR I have updated the System Tests for VertexAI operators. --- **^ Add meaningful description above** Read the **[Pull Request

(airflow) branch bring-back-arm-image-building updated (950fd83c08 -> 7fc594c02b)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch bring-back-arm-image-building in repository https://gitbox.apache.org/repos/asf/airflow.git discard 950fd83c08 Bring back ARM image building using ASF self-hosted ARM runners add

Re: [PR] Send context using in venv operator [airflow]

2024-07-26 Thread via GitHub
potiuk commented on PR #41039: URL: https://github.com/apache/airflow/pull/41039#issuecomment-2252767334 Always Be Rebased - this has been solved in main. Generally when in doubt, always rebase your PR. Your PR is 10 commits behind already:

Re: [PR] [Databricks Provider] Revert PRs #40864 and #40471 [airflow]

2024-07-26 Thread via GitHub
pankajkoti commented on PR #41050: URL: https://github.com/apache/airflow/pull/41050#issuecomment-2252772660 cc: @boraberke for your information. I guess it would be better if rework on the earlier issue peacefully and clear the path of the upcoming release. Hope you're comfortable with

[PR] openlineage: update docs on openlineage methods [airflow]

2024-07-26 Thread via GitHub
kacpermuda opened a new pull request, #41051: URL: https://github.com/apache/airflow/pull/41051 Some clarifications to Ol docs after receiving user feedback. --- **^ Add meaningful description above** Read the **[Pull Request

[I] Make Databricks operators' `json` parameter compatible with XComs, Jinja expression values [airflow]

2024-07-26 Thread via GitHub
josh-fell opened a new issue, #35433: URL: https://github.com/apache/airflow/issues/35433 ### Body A few Databricks operators exist which have a `json` parameter that can be a JSON object which contains any number of API parameters. In the constructors of these operators, the `json`

Re: [PR] openlineage, docs: add missing execution_timeout conf value [airflow]

2024-07-26 Thread via GitHub
potiuk merged PR #41042: URL: https://github.com/apache/airflow/pull/41042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] [Databricks Provider] Revert PRs #40864 and #40471 [airflow]

2024-07-26 Thread via GitHub
pankajkoti opened a new pull request, #41050: URL: https://github.com/apache/airflow/pull/41050 This reverts PR #40864 and PR #40471. Previously, PR https://github.com/apache/airflow/pull/40471 was contributed to address issue https://github.com/apache/airflow/issues/35433.

(airflow) branch main updated: openlineage, docs: add missing execution_timeout conf value (#41042)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 592eafa82a openlineage, docs: add missing

(airflow) branch bring-back-arm-image-building updated (6e1b432b2a -> 409699cea4)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch bring-back-arm-image-building in repository https://gitbox.apache.org/repos/asf/airflow.git discard 6e1b432b2a Bring back ARM image building using ASF self-hosted ARM runners add

(airflow) branch bring-back-arm-image-building updated (d4363c3be6 -> 950fd83c08)

2024-07-26 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch bring-back-arm-image-building in repository https://gitbox.apache.org/repos/asf/airflow.git omit d4363c3be6 Bring back ARM image building using ASF self-hosted ARM runners add

  1   2   >