Re: [I] Status of testing of Apache Airflow 2.8.3rc1 [airflow]

2024-03-08 Thread via GitHub
nathadfield commented on issue #37982: URL: https://github.com/apache/airflow/issues/37982#issuecomment-1985300858 #37828 is all good from me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
kosteev commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985334309 In perform_heartbeat, job.heartrate is used. The issue is that for scheduler it is defined in SchedulerJobRunner (previously was SchedulerJob): https://github.com/apache/airf

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
kosteev commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985337125 About reproduction scenario: "If heartbeat sec is greater than threshold, then yes you'd expect to see the warning" - yes, but in the scenario you modify [scheduler]job_heartbeat

Re: [I] Add a compatibility with Opensearch search for log management [airflow]

2024-03-08 Thread via GitHub
erikvatt commented on issue #33619: URL: https://github.com/apache/airflow/issues/33619#issuecomment-1985350725 @cjames23 Have you started working on OpenSearch log integration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: [I] 404 Pod not found can be seen often [airflow]

2024-03-08 Thread via GitHub
paramjeet01 commented on issue #37090: URL: https://github.com/apache/airflow/issues/37090#issuecomment-1985362791 @dirrao , Can you please take a look into this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use t

Re: [PR] Add the dataset_expression as part of DagModel and DAGDetailSchema [airflow]

2024-03-08 Thread via GitHub
sunank200 commented on code in PR #37826: URL: https://github.com/apache/airflow/pull/37826#discussion_r1517481946 ## airflow/models/dag.py: ## @@ -3115,6 +3119,8 @@ def bulk_write_to_db( ) orm_dag.schedule_interval = dag.schedule_interval

[PR] Validate tasks total priority weight in DagBag [airflow]

2024-03-08 Thread via GitHub
Taragolis opened a new pull request, #37990: URL: https://github.com/apache/airflow/pull/37990 This one is continuation of https://github.com/apache/airflow/pull/34168. Validation happen independently of the Database Type and expects that the task's priority_weight_total i

Re: [PR] Validate tasks total priority weight in DagBag [airflow]

2024-03-08 Thread via GitHub
Taragolis closed pull request #37990: Validate tasks total priority weight in DagBag URL: https://github.com/apache/airflow/pull/37990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

svn commit: r67798 - /dev/airflow/providers/ /release/airflow/providers/

2024-03-08 Thread eladkal
Author: eladkal Date: Fri Mar 8 10:37:49 2024 New Revision: 67798 Log: Release Airflow Providers on 2024-03-08 Added: release/airflow/providers/apache_airflow_providers_amazon-8.19.0-py3-none-any.whl - copied unchanged from r67797, dev/airflow/providers/apache_airflow_providers_amazo

Re: [PR] Add the dataset_expression as part of DagModel and DAGDetailSchema [airflow]

2024-03-08 Thread via GitHub
uranusjr commented on code in PR #37826: URL: https://github.com/apache/airflow/pull/37826#discussion_r1517546336 ## airflow/models/dag.py: ## @@ -3032,6 +3034,16 @@ def bulk_sync_to_db( ) return cls.bulk_write_to_db(dags=dags, session=session) +def simpl

Re: [PR] Add the dataset_expression as part of DagModel and DAGDetailSchema [airflow]

2024-03-08 Thread via GitHub
phanikumv merged PR #37826: URL: https://github.com/apache/airflow/pull/37826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflo

(airflow) branch main updated (08efc32fd1 -> 832d2f561a)

2024-03-08 Thread phanikumv
This is an automated email from the ASF dual-hosted git repository. phanikumv pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 08efc32fd1 Fix celery command deprecation message with right version (#37926) add 832d2f561a Add the dataset_e

Re: [PR] Add documentation for packages - 2024-03-04 [airflow-site]

2024-03-08 Thread via GitHub
eladkal merged PR #972: URL: https://github.com/apache/airflow-site/pull/972 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

(airflow-site) branch add-documentation-2024-03-04 deleted (was ff359eb980)

2024-03-08 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch add-documentation-2024-03-04 in repository https://gitbox.apache.org/repos/asf/airflow-site.git was ff359eb980 Add documentation for packages - 2024-03-04 The revisions that were on this b

[PR] Update providers metadata 2024-03-08 [airflow]

2024-03-08 Thread via GitHub
eladkal opened a new pull request, #37991: URL: https://github.com/apache/airflow/pull/37991 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [I] Status of testing Providers that were prepared on March 04, 2024 [airflow]

2024-03-08 Thread via GitHub
eladkal commented on issue #37890: URL: https://github.com/apache/airflow/issues/37890#issuecomment-1985470350 Thank you everyone. Providers are released. I invite everyone to help improve providers for the next release, a list of open issues can be found [here](https://github.com/ap

Re: [I] Status of testing Providers that were prepared on March 04, 2024 [airflow]

2024-03-08 Thread via GitHub
eladkal closed issue #37890: Status of testing Providers that were prepared on March 04, 2024 URL: https://github.com/apache/airflow/issues/37890 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985472529 > so, now this heartrate: int = conf.getint("scheduler", "SCHEDULER_HEARTBEAT_SEC") from SchedulerJobRunner is not used anywhere Indeed. > "If heartbeat sec is greater

Re: [I] 404 Pod not found can be seen often [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37090: URL: https://github.com/apache/airflow/issues/37090#issuecomment-1985479855 Generally speaking the issue will be looked at someone when they pick interest in fixing it - I think there is no reason to expect that particular person will take a look at it eve

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
kosteev commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985504904 I will try to prepare a fix, but I do not want to commit to this, so that if anyone wants to fix it. I will try to work on it today, early next week. -- This is an automated m

[PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk opened a new pull request, #37992: URL: https://github.com/apache/airflow/pull/37992 Sinc #34026 scheduler heartrate has not been properly calculated. We missed the check for SchedulerJob type and setting heartrate value from `scheduler_health_check_threshold`. This PR fixes i

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985513790 I actualy took a quick look and it turned out that it was removed after adding SchedulerJobRunner. The SchedulerJob check has been removed by #34026 a bit later. I prepared a

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985518074 Actually not even there -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the speci

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985536704 @kosteev - are you sure it's affecting 2.6.3 Airflow ? from what I traced #34026 has been only part of v2-8 branch. Maybe there was another reason before in 2.6.3 - but quite certa

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985539571 The 2.6.3 version is: ``` def is_alive(self, grace_multiplier=2.1): """ Is this job currently alive. We define alive as in a state

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985546310 Ahh.. those parameters are confusing and I am mixing them ... it is indeed way older sorry (threshold mixed with secs) -- This is an automated message from the Apache Git Service.

Re: [I] `AirbyteTriggerSyncOperator` does not work on deferrable mode [airflow]

2024-03-08 Thread via GitHub
josix commented on issue #37449: URL: https://github.com/apache/airflow/issues/37449#issuecomment-1985553064 I could help investigate it if there is no one working on this. Please feel free to assign me, thanks. -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
dirrao commented on code in PR #37992: URL: https://github.com/apache/airflow/pull/37992#discussion_r1517618531 ## airflow/jobs/job.py: ## @@ -265,6 +265,8 @@ def _merge_from(self, job: Job | JobPydantic | None): def _heartrate(job_type: str) -> float: if job_type

(airflow) branch constraints-main updated: Updating constraints. Github run id:8202147560

2024-03-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new b2dd2dba48 Updating con

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk commented on code in PR #37992: URL: https://github.com/apache/airflow/pull/37992#discussion_r1517620372 ## airflow/jobs/job.py: ## @@ -265,6 +265,8 @@ def _merge_from(self, job: Job | JobPydantic | None): def _heartrate(job_type: str) -> float: if job_type

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk commented on code in PR #37992: URL: https://github.com/apache/airflow/pull/37992#discussion_r1517621593 ## airflow/jobs/job.py: ## @@ -265,6 +265,8 @@ def _merge_from(self, job: Job | JobPydantic | None): def _heartrate(job_type: str) -> float: if job_type

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985564821 Tracked it down (there were quite a few refactors) and indeed it was #30255 that introduced it -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565030 So it's there since 2.6.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specifi

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565404 (just wanted to make sure how far back it goes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk commented on PR #37992: URL: https://github.com/apache/airflow/pull/37992#issuecomment-1985565811 I actually tracked it down wrongly - it's there since 2.6.0 and https://github.com/apache/airflow/pull/30255 -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] Add color formatting for ANSI chars in logs from task executions [airflow]

2024-03-08 Thread via GitHub
jscheffl commented on PR #37985: URL: https://github.com/apache/airflow/pull/37985#issuecomment-1985583083 Oh, wow 12 approvals in <12h :-D I found one glitch with the re-work, please do not merge (yet), need to fix this in the afternoon so that the linkify is working again. -- This is

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
kosteev commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985591616 I prepared a fix like this https://github.com/apache/airflow/pull/37993 But I see you have already fix (and implemented differently). Should I close my PR? -- This is an a

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985600389 Yeah - I think the way where it is closer to heartrate property is a bit better and easier to reason about. -- This is an automated message from the Apache Git Service. To respond

Re: [PR] Fix bug with using [scheduler]job_heartbeat_sec instead of [scheduler]scheduler_heartbeat_sec for Airflow scheduler. [airflow]

2024-03-08 Thread via GitHub
kosteev closed pull request #37993: Fix bug with using [scheduler]job_heartbeat_sec instead of [scheduler]scheduler_heartbeat_sec for Airflow scheduler. URL: https://github.com/apache/airflow/pull/37993 -- This is an automated message from the Apache Git Service. To respond to the message, p

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37971: URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985603095 We have a bit of a mess there - because the JOB (scheduler/job_heartrabeat_sec) has been used and abused by different jobs, so likely better to keep it in JOB. -- This is an aut

Re: [PR] Add color formatting for ANSI chars in logs from task executions [airflow]

2024-03-08 Thread via GitHub
potiuk commented on code in PR #37985: URL: https://github.com/apache/airflow/pull/37985#discussion_r1517653733 ## airflow/www/jest.config.js: ## @@ -32,6 +32,7 @@ const config = { transformIgnorePatterns: [ `node_modules/(?!${[ // specify modules that needs to be

[PR] [OpenLineage] fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised [airflow]

2024-03-08 Thread via GitHub
kacpermuda opened a new pull request, #37994: URL: https://github.com/apache/airflow/pull/37994 When missing [openlineage] section in airflow.cfg or some env variables, an error was raised. We should avoid that and treat lack of configuration as empty configuration. T

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
dbrtly commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985618402 If cloud logging streamed the logs directly into the airflow gcs logs path, would that be a pathway to seeing the logs in airflow? -- This is an automated message from the Apache

Re: [PR] [OpenLineage] fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised [airflow]

2024-03-08 Thread via GitHub
mobuchowski merged PR #37994: URL: https://github.com/apache/airflow/pull/37994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airf

(airflow) branch main updated (832d2f561a -> ea5238a81b)

2024-03-08 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 832d2f561a Add the dataset_expression as part of DagModel and DAGDetailSchema (#37826) add ea5238a81b fix: A

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985648227 > If cloud logging streamed the logs directly into the airflow gcs logs path, would that be a pathway to seeing the logs in airflow? No. GCS is object storage. It's not a stre

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
dbrtly commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985657810 I’m glossing over details of specific jargon. Cloud run already streams logs to cloud logging. I can see a directory in the cloud composer bucket for logs. If cloud loggin

Re: [PR] Add `ADLSCreateObjectOperator` [airflow]

2024-03-08 Thread via GitHub
eladkal commented on PR #37821: URL: https://github.com/apache/airflow/pull/37821#issuecomment-1985669385 OK now i get it. the claim is that the changes were added in https://github.com/apache/airflow/pull/37103 by mistake :) This is why separating unrelated changes in PRs is so impor

Re: [I] Add a compatibility with Opensearch search for log management [airflow]

2024-03-08 Thread via GitHub
eladkal commented on issue #33619: URL: https://github.com/apache/airflow/issues/33619#issuecomment-1985675070 I think that https://github.com/apache/airflow/pull/34693 has most of the needed logic for AWS integration it just require adjustments after the OpenSearch provider was implemented

Re: [I] Scheduler fails to schedule DagRuns due to persistent DAG record lock [airflow]

2024-03-08 Thread via GitHub
renanxx1 commented on issue #36920: URL: https://github.com/apache/airflow/issues/36920#issuecomment-1985698998 I faced the same issue on Airflow 2.8.1. The scheduler was skipping dags execution with the message: "DAG scheduling was skipped, probably because the DAG record was locked"

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk merged PR #37992: URL: https://github.com/apache/airflow/pull/37992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

2024-03-08 Thread via GitHub
potiuk closed issue #37971: Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec URL: https://github.com/apache/airflow/issues/37971 -- This is an automated message from the Apache Git Service. To respond to the message, please log

(airflow) branch main updated (ea5238a81b -> 01e40abc75)

2024-03-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from ea5238a81b fix: Add fallbacks when retrieving Airflow configuration to avoid errors being raised (#37994) add 01e

Re: [PR] Fix a bug where scheduler heartrate parameter were not used [airflow]

2024-03-08 Thread via GitHub
potiuk commented on code in PR #37992: URL: https://github.com/apache/airflow/pull/37992#discussion_r1517721921 ## airflow/jobs/job.py: ## @@ -265,6 +265,8 @@ def _merge_from(self, job: Job | JobPydantic | None): def _heartrate(job_type: str) -> float: if job_type

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985727697 Well. if you find a way to create GCS objects (not stream to GCS - that is impossible) after the job is complete, and you follow naming convention of Airflow (which you have to be w

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985742361 So maybe just to explain how logging works when REMOTE logging uses GCS - because you might not be aware of it. GCS is an object storage and streaming to it is impossible as mention

Re: [I] CloudRunExecuteJobOperator - logs from inside the container are not shown in Airflow [airflow]

2024-03-08 Thread via GitHub
dbrtly commented on issue #36963: URL: https://github.com/apache/airflow/issues/36963#issuecomment-1985746090 Thanks Jarek -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [I] In the DAG Graph view, all DAG runs disappear after selecting a Run State [airflow]

2024-03-08 Thread via GitHub
rgelsi commented on issue #37966: URL: https://github.com/apache/airflow/issues/37966#issuecomment-1985758515 The following error is displayed in the Browser Console: ``` response: "{\"error\": \"Can not provide filters when dag_run_id filter is selected.\"}" responseText: "{\"error

(airflow) branch main updated: Update providers metadata 2024-03-08 (#37991)

2024-03-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new bcb4e6603a Update providers metadata 2024-03-08 (#3

Re: [PR] Update providers metadata 2024-03-08 [airflow]

2024-03-08 Thread via GitHub
potiuk merged PR #37991: URL: https://github.com/apache/airflow/pull/37991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[PR] Update to latest uv [airflow]

2024-03-08 Thread via GitHub
potiuk opened a new pull request, #37995: URL: https://github.com/apache/airflow/pull/37995 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pul

Re: [I] Status of testing of Apache Airflow 2.8.3rc1 [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on issue #37982: URL: https://github.com/apache/airflow/issues/37982#issuecomment-1985848574 Checked my change and it works as expected -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Update to latest uv [airflow]

2024-03-08 Thread via GitHub
potiuk commented on PR #37995: URL: https://github.com/apache/airflow/pull/37995#issuecomment-1985853546 Ha... First incompatibility :) encountered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

Re: [PR] Make the method `BaseAuthManager.is_authorized_custom_view` abstract [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on PR #37915: URL: https://github.com/apache/airflow/pull/37915#issuecomment-1985863531 Hmmm, I can understand the MENU change looks unrelated. I can possibly split the PR in 2 if that helps. I can also a unit test to cover the plugin path -- This is an automated messag

Re: [PR] Add `ADLSCreateObjectOperator` [airflow]

2024-03-08 Thread via GitHub
dabla commented on PR #37821: URL: https://github.com/apache/airflow/pull/37821#issuecomment-1985881678 > OK now i get it. the claim is that the changes were added in #37103 by mistake :) This is why separating unrelated changes in PRs is so important it. It would have been much easier to u

Re: [PR] Make the method `BaseAuthManager.is_authorized_custom_view` abstract [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on PR #37915: URL: https://github.com/apache/airflow/pull/37915#issuecomment-1985892661 Actually, let me explain why I combined the two. The idea initially was to call `is_authorized_custom_view`in this case: https://github.com/apache/airflow/blob/main/airflow/providers/a

Re: [PR] Add system test to test the AWS auth manager [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on code in PR #37947: URL: https://github.com/apache/airflow/pull/37947#discussion_r1517873411 ## airflow/providers/amazon/aws/auth_manager/views/auth.py: ## @@ -93,7 +93,7 @@ def login_callback(self): user_id=attributes["id"][0], gro

Re: [I] In the DAG Graph view, all DAG runs disappear after selecting a Run State [airflow]

2024-03-08 Thread via GitHub
bbovenzi commented on issue #37966: URL: https://github.com/apache/airflow/issues/37966#issuecomment-1985905459 That makes sense. If you have a dag run selected, we shouldn't let you filter out your selection. So either setting a filter clears your selection or filters are disabled when

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517880761 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517880761 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517880761 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517892095 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

Re: [PR] Update to latest uv [airflow]

2024-03-08 Thread via GitHub
potiuk commented on PR #37995: URL: https://github.com/apache/airflow/pull/37995#issuecomment-1985929554 Closing in favour of #37996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Update to latest uv [airflow]

2024-03-08 Thread via GitHub
potiuk closed pull request #37995: Update to latest uv URL: https://github.com/apache/airflow/pull/37995 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[PR] Disable `uv` upgrade checks temporarily [airflow]

2024-03-08 Thread via GitHub
potiuk opened a new pull request, #37996: URL: https://github.com/apache/airflow/pull/37996 Until https://github.com/astral-sh/uv/issues/2302 is solved we should disable UV upgrade check as 0.1.16 version of `uv` does not properly validate our pyproject.toml with Draft PEP-0639 extension (l

[PR] Add back methods in security manager [airflow]

2024-03-08 Thread via GitHub
vincbeck opened a new pull request, #37997: URL: https://github.com/apache/airflow/pull/37997 I wrongly removed 2 methods in #37889. Adding them back. It makes fail any auth manager but FAB. I'll add a test in a separate PR to catch it. The reason why I dont add the test in this PR is: -

Re: [PR] Add system test to test the AWS auth manager [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on code in PR #37947: URL: https://github.com/apache/airflow/pull/37947#discussion_r1517901096 ## airflow/www/security_manager.py: ## @@ -348,3 +348,9 @@ def _is_authorized_category_menu(self, category: str) -> Callable: self._get_auth_manager_i

Re: [PR] Add system test to test the AWS auth manager [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on code in PR #37947: URL: https://github.com/apache/airflow/pull/37947#discussion_r1517910703 ## airflow/www/security_manager.py: ## @@ -348,3 +348,9 @@ def _is_authorized_category_menu(self, category: str) -> Callable: self._get_auth_manager_i

Re: [I] Status of testing Providers that were prepared on March 04, 2024 [airflow]

2024-03-08 Thread via GitHub
dabla commented on issue #37890: URL: https://github.com/apache/airflow/issues/37890#issuecomment-1985950730 Maybe a bit late but I tested [#37103](https://github.com/apache/airflow/pull/37103) and I can confirm it works :) -- This is an automated message from the Apache Git Service. To

[PR] Implement wait_job_until_complete parameter for KubernetesJobOperator [airflow]

2024-03-08 Thread via GitHub
moiseenkov opened a new pull request, #37998: URL: https://github.com/apache/airflow/pull/37998 PR introduces the optional `wait_job_until_complete` parameter for the `KubernetesJobOperator` that blocks execution until the created job is complete. The default value is `False` thus behavio

Re: [I] Status of testing Providers that were prepared on March 04, 2024 [airflow]

2024-03-08 Thread via GitHub
potiuk commented on issue #37890: URL: https://github.com/apache/airflow/issues/37890#issuecomment-1985985361 Never too late for good news :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s

Re: [PR] Make the method `BaseAuthManager.is_authorized_custom_view` abstract [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on PR #37915: URL: https://github.com/apache/airflow/pull/37915#issuecomment-1985982769 I also added a unit test that loads the index page. The difference with the test being implemented in #37947, is this one is a unit test. No real API call is made against AWS. This tes

Re: [PR] Add back methods in security manager [airflow]

2024-03-08 Thread via GitHub
vincbeck commented on PR #37997: URL: https://github.com/apache/airflow/pull/37997#issuecomment-1985984976 Test implemented in #37915 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
JDarDagran commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517942728 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.Pyth

Re: [PR] Disable `uv` upgrade checks temporarily [airflow]

2024-03-08 Thread via GitHub
potiuk merged PR #37996: URL: https://github.com/apache/airflow/pull/37996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated: Disable `uv` upgrade checks temporarily (#37996)

2024-03-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 66f28f2fc9 Disable `uv` upgrade checks temporarily

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517976751 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
mobuchowski commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517981734 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.Pyt

Re: [PR] openlineage: add `opt-in` option [airflow]

2024-03-08 Thread via GitHub
tatiana commented on code in PR #37725: URL: https://github.com/apache/airflow/pull/37725#discussion_r1517976751 ## airflow/providers/openlineage/provider.yaml: ## @@ -77,6 +77,14 @@ config: example: "airflow.operators.bash.BashOperator;airflow.operators.python.PythonO

[I] ElasticSearch log fetching hook is not granular enough in its index filter [airflow]

2024-03-08 Thread via GitHub
nsAstro opened a new issue, #37999: URL: https://github.com/apache/airflow/issues/37999 ### Apache Airflow Provider(s) elasticsearch ### Versions of Apache Airflow Providers Latest version of the ElasticSearch provider ### Apache Airflow version 2.9 but all

(airflow) branch constraints-main updated: Updating constraints. Github run id:8206300248

2024-03-08 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 3a034462ec Updating con

Re: [PR] Properly serialize TaskInstancePydantic and DagRunPydantic [airflow]

2024-03-08 Thread via GitHub
dstandish commented on PR #37855: URL: https://github.com/apache/airflow/pull/37855#issuecomment-1986111612 @potiuk initially I was going to write a check that all pydantic models we define can round trip. However, as yet some pydantic models do not have types defined (as in DAT.THE_MODEL)

(airflow) branch main updated: Add back methods in security manager (#37997)

2024-03-08 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b3cd32f6d8 Add back methods in security manager (

Re: [PR] Add back methods in security manager [airflow]

2024-03-08 Thread via GitHub
vincbeck merged PR #37997: URL: https://github.com/apache/airflow/pull/37997 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

Re: [PR] Raising Exception if dag is not in active state while triggering dag [airflow]

2024-03-08 Thread via GitHub
eladkal commented on PR #37836: URL: https://github.com/apache/airflow/pull/37836#issuecomment-1986128883 If we do this it must be consistent between CLI and API. > I think there were related discussions in the past about how to treat a paused DAG when it is triggered I think t

Re: [PR] Implement `filter_permitted_dag_ids` in AWS auth manager [airflow]

2024-03-08 Thread via GitHub
ferruzzi commented on code in PR #37666: URL: https://github.com/apache/airflow/pull/37666#discussion_r1518053030 ## airflow/providers/amazon/aws/auth_manager/aws_auth_manager.py: ## @@ -443,6 +443,62 @@ def batch_is_authorized_variable( ] return self.avp_facad

[PR] Fix regexp used to bump version of provider when preparing docs [airflow]

2024-03-08 Thread via GitHub
potiuk opened a new pull request, #38000: URL: https://github.com/apache/airflow/pull/38000 The regexp used to bump version of provider also matched `excluded-python-versions` and did not allow to bump version of papermill. Updated regexp only matches actual `versions:` key.

Re: [I] ElasticSearch log fetching hook is not granular enough in its index filter [airflow]

2024-03-08 Thread via GitHub
eladkal commented on issue #37999: URL: https://github.com/apache/airflow/issues/37999#issuecomment-1986140122 cc @Owen-CH-Leung You contributed several elastic PRs in the past. Can you assist with triaging this issue? -- This is an automated message from the Apache Git Service. To respon

Re: [PR] Fix regexp used to bump version of provider when preparing docs [airflow]

2024-03-08 Thread via GitHub
potiuk merged PR #38000: URL: https://github.com/apache/airflow/pull/38000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

(airflow) branch main updated (b3cd32f6d8 -> c616cd307a)

2024-03-08 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from b3cd32f6d8 Add back methods in security manager (#37997) add c616cd307a Fix regexp used to bump version of provide

  1   2   >