[GitHub] [airflow] pankajkoti commented on pull request #32950: Extend task context logging support for remote logging using AWS S3

2023-07-31 Thread via GitHub
pankajkoti commented on PR #32950: URL: https://github.com/apache/airflow/pull/32950#issuecomment-1659582952 @uranusjr yes, since the base PR https://github.com/apache/airflow/pull/32646 is not merged to main yet, the keyword argument identifier is not available yet for the `set_context` me

[GitHub] [airflow] bbovenzi opened a new pull request, #32992: Only load task action modal if user can edit

2023-07-31 Thread via GitHub
bbovenzi opened a new pull request, #32992: URL: https://github.com/apache/airflow/pull/32992 We checked that a View only user couldn't clear or mark as task. But the modal was still mounting, and the dry run request was still triggering. Now the modal will only mount if the user has permis

[GitHub] [airflow] pegasas commented on issue #32966: Enable Airflow DAG processor horizontally scalable

2023-07-31 Thread via GitHub
pegasas commented on issue #32966: URL: https://github.com/apache/airflow/issues/32966#issuecomment-1659534298 Hi, @dirrao , @raphaelauv , @potiuk I have one performance concern, maybe it is naive: Our architecture only allows one scheduler to access meta database, Will hori

[GitHub] [airflow] bbovenzi commented on pull request #32964: fix: front dagview - explicit wording - next run

2023-07-31 Thread via GitHub
bbovenzi commented on PR #32964: URL: https://github.com/apache/airflow/pull/32964#issuecomment-1659485125 This is why we added a tooltip to give more information. Is there something else you'd like here? https://github.com/apache/airflow/assets/4600967/0282e29a-a725-4807-be84-feee92b

[GitHub] [airflow] Lee-W commented on a diff in pull request #32809: Deferrable mode for Sqs Sensor

2023-07-31 Thread via GitHub
Lee-W commented on code in PR #32809: URL: https://github.com/apache/airflow/pull/32809#discussion_r1280051620 ## airflow/providers/amazon/aws/sensors/sqs.py: ## @@ -131,19 +165,7 @@ def poll_sqs(self, sqs_conn: BaseAwsConnection) -> Collection: receive_message_kwa

[GitHub] [airflow] Lee-W commented on a diff in pull request #32809: Deferrable mode for Sqs Sensor

2023-07-31 Thread via GitHub
Lee-W commented on code in PR #32809: URL: https://github.com/apache/airflow/pull/32809#discussion_r1280049380 ## airflow/providers/amazon/aws/triggers/sqs.py: ## @@ -0,0 +1,171 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agree

[GitHub] [airflow] jsgh7276 commented on issue #32859: DAG triggering page does not reflect conf parameter in the URL query string

2023-07-31 Thread via GitHub
jsgh7276 commented on issue #32859: URL: https://github.com/apache/airflow/issues/32859#issuecomment-1659440926 I understood what you are considering now and I’ll happily wait for the fix. Thank you so much for your help. -- This is an automated message from the Apache Git Service. To res

[GitHub] [airflow] uranusjr commented on a diff in pull request #32883: [WIP]:Refactor Sqlalchemy queries to 2.0 style (Part 7)

2023-07-31 Thread via GitHub
uranusjr commented on code in PR #32883: URL: https://github.com/apache/airflow/pull/32883#discussion_r1280012837 ## airflow/operators/subdag.py: ## @@ -112,7 +113,7 @@ def _validate_pool(self, session): conflicts = [t for t in self.subdag.tasks if t.pool == self.po

[GitHub] [airflow] potiuk opened a new pull request, #32991: Attempt to get logs from failed flaky bakfill test

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32991: URL: https://github.com/apache/airflow/pull/32991 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelin

[GitHub] [airflow] hussein-awala opened a new pull request, #32990: [WIP] Timeout Airflow TI and Sensors when using defferable mode

2023-07-31 Thread via GitHub
hussein-awala opened a new pull request, #32990: URL: https://github.com/apache/airflow/pull/32990 closes: #32638 This PR handles the trigger timeout and raise `AirflowTaskTimeout` or `AirflowSensorTimeout` instead of failing the TI with `TaskDeferralError`. It depr

[GitHub] [airflow] uranusjr commented on a diff in pull request #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
uranusjr commented on code in PR #32983: URL: https://github.com/apache/airflow/pull/32983#discussion_r1279998041 ## docs/apache-airflow/core-concepts/executor/index.rst: ## @@ -77,4 +77,4 @@ There are two types of executor - those that run tasks *locally* (inside the ``s ..

[GitHub] [airflow] uranusjr commented on a diff in pull request #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
uranusjr commented on code in PR #32983: URL: https://github.com/apache/airflow/pull/32983#discussion_r1279998041 ## docs/apache-airflow/core-concepts/executor/index.rst: ## @@ -77,4 +77,4 @@ There are two types of executor - those that run tasks *locally* (inside the ``s ..

[GitHub] [airflow] uranusjr commented on a diff in pull request #32887: OpenAPI Spec fix nullable alongside $ref

2023-07-31 Thread via GitHub
uranusjr commented on code in PR #32887: URL: https://github.com/apache/airflow/pull/32887#discussion_r1279992140 ## airflow/api_connexion/openapi/v1.yaml: ## @@ -4659,7 +4642,9 @@ components: *Changed in version 2.4.0*: 'sensing' state has been removed. *Chang

[GitHub] [airflow] potiuk merged pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
potiuk merged PR #32984: URL: https://github.com/apache/airflow/pull/32984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Move redis subscribe to poke() method in Redis Sensor (#32984)

2023-07-31 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 17a3dd4135 Move redis subscribe to poke() method in

[GitHub] [airflow] pierrejeambrun commented on pull request #32965: Adding missing start date for trigger DAGs

2023-07-31 Thread via GitHub
pierrejeambrun commented on PR #32965: URL: https://github.com/apache/airflow/pull/32965#issuecomment-1659350553 Can you elaborate a little bit more on why this is needed ? Scheduler is already in charge of updating the dagrun start_date to `timezone.utcnow()` when starting a new run.

[GitHub] [airflow] potiuk opened a new pull request, #32989: Bring back mysql-connector-python as required depednency

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32989: URL: https://github.com/apache/airflow/pull/32989 The mysql-connector-python 8.1.0 was released 18th of June with Protobuf 4 support - thus resolving dependency conflicts we had. --- **^ Add meaningful descriptio

[GitHub] [airflow] pierrejeambrun commented on pull request #32887: OpenAPI Spec fix nullable alongside $ref

2023-07-31 Thread via GitHub
pierrejeambrun commented on PR #32887: URL: https://github.com/apache/airflow/pull/32887#issuecomment-1659308735 CI should be green now. The hard part is that both `connexion` package and the `openapi-generator` need to understand the spec. This limits us to not use certain syntax. -- Th

[GitHub] [airflow] ferruzzi commented on pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
ferruzzi commented on PR #32984: URL: https://github.com/apache/airflow/pull/32984#issuecomment-1659308093 Not that my approval means much at this point, but Approved? Again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub a

[GitHub] [airflow] syedahsn commented on a diff in pull request #32809: Deferrable mode for Sqs Sensor

2023-07-31 Thread via GitHub
syedahsn commented on code in PR #32809: URL: https://github.com/apache/airflow/pull/32809#discussion_r1279961526 ## airflow/providers/amazon/aws/triggers/sqs.py: ## @@ -0,0 +1,171 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license ag

[GitHub] [airflow] syedahsn commented on a diff in pull request #32809: Deferrable mode for Sqs Sensor

2023-07-31 Thread via GitHub
syedahsn commented on code in PR #32809: URL: https://github.com/apache/airflow/pull/32809#discussion_r1279960782 ## airflow/providers/amazon/aws/sensors/sqs.py: ## @@ -131,19 +165,7 @@ def poll_sqs(self, sqs_conn: BaseAwsConnection) -> Collection: receive_message_

[GitHub] [airflow] potiuk commented on a diff in pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
potiuk commented on code in PR #32984: URL: https://github.com/apache/airflow/pull/32984#discussion_r1279956787 ## airflow/providers/redis/sensors/redis_pub_sub.py: ## @@ -42,7 +42,7 @@ def __init__(self, *, channels: list[str] | str, redis_conn_id: str, **kwargs) - se

[GitHub] [airflow] hussein-awala commented on a diff in pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
hussein-awala commented on code in PR #32984: URL: https://github.com/apache/airflow/pull/32984#discussion_r1279953988 ## airflow/providers/redis/sensors/redis_pub_sub.py: ## @@ -42,7 +42,7 @@ def __init__(self, *, channels: list[str] | str, redis_conn_id: str, **kwargs) -

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #32984: URL: https://github.com/apache/airflow/pull/32984#discussion_r1279952713 ## airflow/providers/redis/sensors/redis_pub_sub.py: ## @@ -42,7 +42,7 @@ def __init__(self, *, channels: list[str] | str, redis_conn_id: str, **kwargs) -

[GitHub] [airflow] hussein-awala commented on a diff in pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
hussein-awala commented on code in PR #32984: URL: https://github.com/apache/airflow/pull/32984#discussion_r1279946200 ## airflow/providers/redis/sensors/redis_pub_sub.py: ## @@ -42,7 +42,7 @@ def __init__(self, *, channels: list[str] | str, redis_conn_id: str, **kwargs) -

[GitHub] [airflow] ferruzzi commented on pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
ferruzzi commented on PR #32984: URL: https://github.com/apache/airflow/pull/32984#issuecomment-1659287121 Interesting. It's a shame subscribe() didn't return a value you could have used instead of creating an new field, but oh well. re-approved; CI is green -- This is an automate

[GitHub] [airflow] ferruzzi commented on pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
ferruzzi commented on PR #32984: URL: https://github.com/apache/airflow/pull/32984#issuecomment-1659281664 Sounds good, I'll take another look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32980: Refactor account_url use in WasbHook

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #32980: URL: https://github.com/apache/airflow/pull/32980#discussion_r1279947537 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -589,6 +585,10 @@ async def get_async_conn(self) -> AsyncBlobServiceClient: ) ret

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32980: Refactor account_url use in WasbHook

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #32980: URL: https://github.com/apache/airflow/pull/32980#discussion_r1279946992 ## airflow/providers/microsoft/azure/hooks/wasb.py: ## @@ -184,15 +184,17 @@ def get_conn(self) -> BlobServiceClient: # connection_string auth takes pri

[GitHub] [airflow] mobuchowski opened a new pull request, #32988: listener: move success hook to after SQLAlchemy commit

2023-07-31 Thread via GitHub
mobuchowski opened a new pull request, #32988: URL: https://github.com/apache/airflow/pull/32988 Similarly to https://github.com/apache/airflow/pull/32716 - slightly change the place of where `on_task_instance_success` listener is called. This currently takes place between when SQLAl

[airflow] branch main updated: Fix example_local_to_wasb system test (#32982)

2023-07-31 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 05494e5d8e Fix example_local_to_wasb system test (#

[GitHub] [airflow] potiuk merged pull request #32982: Fix example_local_to_wasb system test

2023-07-31 Thread via GitHub
potiuk merged PR #32982: URL: https://github.com/apache/airflow/pull/32982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch listener-move-complete created (now 09098ec749)

2023-07-31 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a change to branch listener-move-complete in repository https://gitbox.apache.org/repos/asf/airflow.git at 09098ec749 listener: move success hook to after SQLAlchemy commit This branch includes the foll

[airflow] 01/01: listener: move success hook to after SQLAlchemy commit

2023-07-31 Thread mobuchowski
This is an automated email from the ASF dual-hosted git repository. mobuchowski pushed a commit to branch listener-move-complete in repository https://gitbox.apache.org/repos/asf/airflow.git commit 09098ec7496deca3f89a8f04aca3f0b342634d76 Author: Maciej Obuchowski AuthorDate: Mon Jul 31 20:20:43

[GitHub] [airflow] potiuk commented on pull request #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
potiuk commented on PR #32983: URL: https://github.com/apache/airflow/pull/32983#issuecomment-1659248070 > thanks @potiuk for the review , my first rephrase attempt was ambiguous , could you check my second attempt , thanks Better :) -- This is an automated message from the Apache

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32987: Move security manager override to its own module

2023-07-31 Thread via GitHub
vandonr-amz commented on code in PR #32987: URL: https://github.com/apache/airflow/pull/32987#discussion_r1279918078 ## airflow/auth/managers/fab/auth/__init__.py: ## @@ -15,3 +14,252 @@ # KIND, either express or implied. See the License for the # specific language governing

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #32987: Move security manager override to its own module

2023-07-31 Thread via GitHub
vandonr-amz commented on code in PR #32987: URL: https://github.com/apache/airflow/pull/32987#discussion_r1279918078 ## airflow/auth/managers/fab/auth/__init__.py: ## @@ -15,3 +14,252 @@ # KIND, either express or implied. See the License for the # specific language governing

[GitHub] [airflow] raphaelauv commented on pull request #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
raphaelauv commented on PR #32983: URL: https://github.com/apache/airflow/pull/32983#issuecomment-1659217542 thanks @potiuk for the review , ma first rephrase attempt was ambiguous , could you check my second attempt , thanks -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] elements-ix closed issue #32981: Mounting DAGs from a private repo with git-sync: git-sync-init "git-s...@github.com: Permission denied (publickey)." regardless of configuration

2023-07-31 Thread via GitHub
elements-ix closed issue #32981: Mounting DAGs from a private repo with git-sync: git-sync-init "git-s...@github.com: Permission denied (publickey)." regardless of configuration URL: https://github.com/apache/airflow/issues/32981 -- This is an automated message from the Apache Git Service. T

[GitHub] [airflow] vincbeck opened a new pull request, #32987: Move security manager override to its own module

2023-07-31 Thread via GitHub
vincbeck opened a new pull request, #32987: URL: https://github.com/apache/airflow/pull/32987 Move the security manager to its own module. In the near future, we'll need to add files specific to the security manager override and having a module for it makes it easier. There is no code chang

[GitHub] [airflow] vandonr-amz commented on issue #32638: Find a solution to handle the execution timeout in Triggers

2023-07-31 Thread via GitHub
vandonr-amz commented on issue #32638: URL: https://github.com/apache/airflow/issues/32638#issuecomment-1659212392 Ha yes indeed, I was mastaken, I was thinking about the `timeout` parameter on `defer` https://github.com/apache/airflow/blob/12b0b6b60721d7ebfc4e5440a7ca9bf10efb8d11/ai

[GitHub] [airflow] getaaron opened a new issue, #32986: 🧟 Slow Zombie query causes scheduler heartbeat failures without errors

2023-07-31 Thread via GitHub
getaaron opened a new issue, #32986: URL: https://github.com/apache/airflow/issues/32986 ### Apache Airflow version 2.6.3 ### What happened Due to an error with our postgres database (the stats on the tables were stale) this query: https://github.com/apache/airfl

[GitHub] [airflow] hussein-awala commented on a diff in pull request #32925: Check google provider version in GCSToS3Operator before provoide match_glob param

2023-07-31 Thread via GitHub
hussein-awala commented on code in PR #32925: URL: https://github.com/apache/airflow/pull/32925#discussion_r1279899324 ## airflow/providers/amazon/aws/transfers/gcs_to_s3.py: ## @@ -31,6 +34,9 @@ from airflow.utils.context import Context +google_provider_version = versi

[GitHub] [airflow] potiuk commented on pull request #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
potiuk commented on PR #32983: URL: https://github.com/apache/airflow/pull/32983#issuecomment-1659196139 Well. Actualy the docs were right before. All executors - no matter if "remote" or "local" have an internal in-scheduler executor that runs as sub-process. For example this is what commu

[GitHub] [airflow] hussein-awala commented on issue #32638: Find a solution to handle the execution timeout in Triggers

2023-07-31 Thread via GitHub
hussein-awala commented on issue #32638: URL: https://github.com/apache/airflow/issues/32638#issuecomment-1659192636 > But triggers already have a timeout mechanism. The current mechanism is not an implementation for timeout (time-based), but it's based on an extra parameter called `m

[GitHub] [airflow] dstandish opened a new pull request, #32985: Fail stop feature can work with setup / teardown

2023-07-31 Thread via GitHub
dstandish opened a new pull request, #32985: URL: https://github.com/apache/airflow/pull/32985 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[GitHub] [airflow] potiuk commented on pull request #32974: Make SQLExecute Query signature consistent with other SQL operators

2023-07-31 Thread via GitHub
potiuk commented on PR #32974: URL: https://github.com/apache/airflow/pull/32974#issuecomment-1659183131 Should be good to go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [airflow] potiuk commented on pull request #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
potiuk commented on PR #32984: URL: https://github.com/apache/airflow/pull/32984#issuecomment-1659180743 revealed after #32978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific commen

[GitHub] [airflow] potiuk opened a new pull request, #32984: Move redis subscribe to poke() method in Redis Sensor

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32984: URL: https://github.com/apache/airflow/pull/32984 In RedisPubSubSensor subscription has been done in constructor, which was pretty wrong - for example it means that when scheduler parses the sensor, it involves subscribing to the messages and commmu

[GitHub] [airflow] raphaelauv opened a new pull request, #32983: doc: local executors - rephrase scheduler process

2023-07-31 Thread via GitHub
raphaelauv opened a new pull request, #32983: URL: https://github.com/apache/airflow/pull/32983 this note is at the end of the page (after the listing of the Remote Executors) let's make it super explicit -- This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [airflow] potiuk opened a new pull request, #32982: Fix example_local_to_wasb system test

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32982: URL: https://github.com/apache/airflow/pull/32982 The test - when collected - throws Airflow Exception. This has been revealed by #32978 --- **^ Add meaningful description above** Read the **[Pull Request

[airflow] branch main updated: Rename `db upgrade` to `db migrate` and add `connections create-default-connections` (#32810)

2023-07-31 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 1e20ef215a Rename `db upgrade` to `db migrate` and

[GitHub] [airflow] potiuk merged pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
potiuk merged PR #32810: URL: https://github.com/apache/airflow/pull/32810 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Fix system-test pytest marker (#32978)

2023-07-31 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 11629f65f1 Fix system-test pytest marker (#32978)

[GitHub] [airflow] ferruzzi merged pull request #32978: Fix system-test pytest marker

2023-07-31 Thread via GitHub
ferruzzi merged PR #32978: URL: https://github.com/apache/airflow/pull/32978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[airflow] branch main updated: Make the decorators of `PythonOperator` sub-classes extend its decorator (#32845)

2023-07-31 Thread husseinawala
This is an automated email from the ASF dual-hosted git repository. husseinawala pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 8f12e7e4a9 Make the decorators of `PythonOper

[GitHub] [airflow] hussein-awala merged pull request #32845: Make the decorators of `PythonOperator` sub-classes extend its decorator

2023-07-31 Thread via GitHub
hussein-awala merged PR #32845: URL: https://github.com/apache/airflow/pull/32845 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@ai

[GitHub] [airflow] rohan472000 commented on issue #32870: BigQueryValueCheckOperator does not accept non-default project_id, which lead to jobs with impersonation_chain to fail.

2023-07-31 Thread via GitHub
rohan472000 commented on issue #32870: URL: https://github.com/apache/airflow/issues/32870#issuecomment-1659065242 `BigQueryValueCheckOperator` is primarily used for data validation and quality checks, while the `BigQueryInsertJobOperator` or other mentioned operator (in issue description)

[GitHub] [airflow] jens-scheffler-bosch commented on issue #32800: Issue with fetching log file via download button

2023-07-31 Thread via GitHub
jens-scheffler-bosch commented on issue #32800: URL: https://github.com/apache/airflow/issues/32800#issuecomment-1659057535 Okay, I can re-produce this problem. I created an artificial large log file with ~110MB. First I copied it many/multiple times from the same "real" log file and saw th

[GitHub] [airflow] pankajkoti commented on pull request #32980: Refactor account_url use in WasbHook

2023-07-31 Thread via GitHub
pankajkoti commented on PR #32980: URL: https://github.com/apache/airflow/pull/32980#issuecomment-1659046343 cc: @manmeetkaur -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [airflow] elements-ix opened a new issue, #32981: Mounting DAGs from a private repo with git-sync: git-sync-init "git-s...@github.com: Permission denied (publickey)." regardless of configurat

2023-07-31 Thread via GitHub
elements-ix opened a new issue, #32981: URL: https://github.com/apache/airflow/issues/32981 ### Official Helm Chart version 1.10.0 (latest released) ### Apache Airflow version 2.6.2 ### Kubernetes Version v1.24.10+k3s1 ### Helm Chart configuration

[GitHub] [airflow] Adaverse commented on a diff in pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
Adaverse commented on code in PR #32810: URL: https://github.com/apache/airflow/pull/32810#discussion_r1279807909 ## airflow/cli/cli_parser.py: ## @@ -137,9 +137,12 @@ def get_long_option(arg: Arg): def _add_command(subparsers: argparse._SubParsersAction, sub: CLICommand) -

[GitHub] [airflow] rohan472000 commented on issue #32870: BigQueryValueCheckOperator does not accept non-default project_id, which lead to jobs with impersonation_chain to fail.

2023-07-31 Thread via GitHub
rohan472000 commented on issue #32870: URL: https://github.com/apache/airflow/issues/32870#issuecomment-1659024102 @GaryLiuTelus can u show me screenshot of the import of `BigQueryValueCheckOperator` operator? -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
jedcunningham commented on code in PR #32810: URL: https://github.com/apache/airflow/pull/32810#discussion_r1279791403 ## airflow/cli/cli_parser.py: ## @@ -137,9 +137,12 @@ def get_long_option(arg: Arg): def _add_command(subparsers: argparse._SubParsersAction, sub: CLIComma

[GitHub] [airflow] ephraimbuddy opened a new pull request, #32980: Refactor account_url use in WasbHook

2023-07-31 Thread via GitHub
ephraimbuddy opened a new pull request, #32980: URL: https://github.com/apache/airflow/pull/32980 This PR moves the account url setting to one place. Tested this by making connection to azure using the different methods -- This is an automated message from the Apache Git Service.

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32951: Clean-up of our new security page

2023-07-31 Thread via GitHub
jedcunningham commented on code in PR #32951: URL: https://github.com/apache/airflow/pull/32951#discussion_r127984 ## docs/apache-airflow/security/security_model.rst: ## @@ -0,0 +1,129 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributo

[GitHub] [airflow] rohan472000 opened a new pull request, #32979: Change log level from ERROR to INFO

2023-07-31 Thread via GitHub
rohan472000 opened a new pull request, #32979: URL: https://github.com/apache/airflow/pull/32979 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-gui

[airflow] branch main updated: Fix connection parameters of `SnowflakeValueCheckOperator` (#32605)

2023-07-31 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 2ab78ec441 Fix connection parameters of `SnowflakeV

[GitHub] [airflow] potiuk closed issue #32551: SnowflakeValueCheckOperator - database, warehouse, schema parameters doesn't ovveride connection

2023-07-31 Thread via GitHub
potiuk closed issue #32551: SnowflakeValueCheckOperator - database, warehouse, schema parameters doesn't ovveride connection URL: https://github.com/apache/airflow/issues/32551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] potiuk merged pull request #32605: Fix connection parameters of `SnowflakeValueCheckOperator`

2023-07-31 Thread via GitHub
potiuk merged PR #32605: URL: https://github.com/apache/airflow/pull/32605 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Handle logout by auth manager (#32819)

2023-07-31 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 2b0d88e450 Handle logout by auth manager (#32819)

[GitHub] [airflow] ferruzzi merged pull request #32819: Handle logout by auth manager

2023-07-31 Thread via GitHub
ferruzzi merged PR #32819: URL: https://github.com/apache/airflow/pull/32819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] potiuk commented on pull request #32951: Clean-up of our new security page

2023-07-31 Thread via GitHub
potiuk commented on PR #32951: URL: https://github.com/apache/airflow/pull/32951#issuecomment-1658995430 > Overall looks good. Feel free to take or leave all of these suggestions. I'll take 'em -- This is an automated message from the Apache Git Service. To respond to the message, p

[GitHub] [airflow] potiuk commented on a diff in pull request #32951: Clean-up of our new security page

2023-07-31 Thread via GitHub
potiuk commented on code in PR #32951: URL: https://github.com/apache/airflow/pull/32951#discussion_r1279766526 ## docs/apache-airflow/security/security_model.rst: ## @@ -0,0 +1,129 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor licen

[airflow] branch main updated: Increase the number of attempts in AWS system test `example_rds_export` (#32976)

2023-07-31 Thread ferruzzi
This is an automated email from the ASF dual-hosted git repository. ferruzzi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 86193f5608 Increase the number of attempts in AWS

[GitHub] [airflow] ferruzzi merged pull request #32976: Increase the number of attempts in AWS system test `example_rds_export`

2023-07-31 Thread via GitHub
ferruzzi merged PR #32976: URL: https://github.com/apache/airflow/pull/32976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] ferruzzi commented on a diff in pull request #31816: openlineage, sagemaker: add OpenLineage support for SageMaker's Processing, Transform and Training operators

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #31816: URL: https://github.com/apache/airflow/pull/31816#discussion_r1279764127 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -222,6 +230,7 @@ def __init__( self.max_attempts = max_attempts or 60 self.max_ingest

[GitHub] [airflow] potiuk commented on pull request #32978: Fix system-test pytest marker

2023-07-31 Thread via GitHub
potiuk commented on PR #32978: URL: https://github.com/apache/airflow/pull/32978#issuecomment-1658989016 cc: @VladaZakharova @ferruzzi -> You might be interested in this one, the system tests were not collected automaticaly, but now they should be collected when you run the whole folder `py

[GitHub] [airflow] jens-scheffler-bosch commented on issue #32859: DAG triggering page does not reflect conf parameter in the URL query string

2023-07-31 Thread via GitHub
jens-scheffler-bosch commented on issue #32859: URL: https://github.com/apache/airflow/issues/32859#issuecomment-1658985036 Sorry @jsgh7276 that my response took longer. I did a explicit test and can confirm actually that this is a bug. If you pass the `conf`parameter in the URL this is alw

[GitHub] [airflow] potiuk opened a new pull request, #32978: Fix system-test pytest marker

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32978: URL: https://github.com/apache/airflow/pull/32978 We have an automated system-test pytest marker that applies the pytest.mark.system marker to all system tests. It has been implemented in a strange way as it was applying the marker to all provider t

[GitHub] [airflow] ferruzzi commented on a diff in pull request #32819: Handle logout by auth manager

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #32819: URL: https://github.com/apache/airflow/pull/32819#discussion_r1279757896 ## airflow/auth/managers/base_auth_manager.py: ## @@ -63,6 +63,11 @@ def get_url_login(self, **kwargs) -> str: """Return the login page url.""" ...

[GitHub] [airflow] blpham commented on a diff in pull request #32846: Adding Support for Google Cloud's Data Pipelines Run Operator

2023-07-31 Thread via GitHub
blpham commented on code in PR #32846: URL: https://github.com/apache/airflow/pull/32846#discussion_r1279753138 ## airflow/providers/google/cloud/example_dags/google-datapipeline.py: ## Review Comment: Yes you're right, the docstring is incorrect. Moved DAGs file into the

[GitHub] [airflow] pankajkoti commented on pull request #32977: Extend task context logging support for remote logging using Elasticsearch

2023-07-31 Thread via GitHub
pankajkoti commented on PR #32977: URL: https://github.com/apache/airflow/pull/32977#issuecomment-1658976535 Backward compatibility test by putting this change on top of main branch locally: https://github.com/apache/airflow/assets/10206082/ab38eebe-2229-4887-865b-b02c7cbebb4e";> *

[GitHub] [airflow] pankajkoti opened a new pull request, #32977: Extend task context logging support for remote logging using Elasticsearch

2023-07-31 Thread via GitHub
pankajkoti opened a new pull request, #32977: URL: https://github.com/apache/airflow/pull/32977 With the addition of task context logging feature in PR #32646, this PR extends the feature to Elasticsearch when is it set as remote logging store. Here, backward compatibility is ensured

[GitHub] [airflow] jedcunningham commented on a diff in pull request #32951: Clean-up of our new security page

2023-07-31 Thread via GitHub
jedcunningham commented on code in PR #32951: URL: https://github.com/apache/airflow/pull/32951#discussion_r1279727469 ## docs/apache-airflow/security/releasing_security_patches.rst: ## @@ -0,0 +1,49 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31816: openlineage, sagemaker: add OpenLineage support for SageMaker's Processing, Transform and Training operators

2023-07-31 Thread via GitHub
vandonr-amz commented on code in PR #31816: URL: https://github.com/apache/airflow/pull/31816#discussion_r1279745380 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -658,11 +708,66 @@ def execute_complete(self, context, event=None): else: self

[airflow] branch main updated: Add Amazon EventBridge PutRule hook and operator (#32869)

2023-07-31 Thread vincbeck
This is an automated email from the ASF dual-hosted git repository. vincbeck pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 196d336585 Add Amazon EventBridge PutRule hook an

[GitHub] [airflow] vincbeck merged pull request #32869: Add Amazon EventBridge PutRule hook and operator

2023-07-31 Thread via GitHub
vincbeck merged PR #32869: URL: https://github.com/apache/airflow/pull/32869 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow

[GitHub] [airflow] vincbeck commented on a diff in pull request #32819: Handle logout by auth manager

2023-07-31 Thread via GitHub
vincbeck commented on code in PR #32819: URL: https://github.com/apache/airflow/pull/32819#discussion_r1279733811 ## airflow/auth/managers/base_auth_manager.py: ## @@ -63,6 +63,11 @@ def get_url_login(self, **kwargs) -> str: """Return the login page url.""" ...

[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #32819: Handle logout by auth manager

2023-07-31 Thread via GitHub
pierrejeambrun commented on code in PR #32819: URL: https://github.com/apache/airflow/pull/32819#discussion_r1279730314 ## airflow/auth/managers/base_auth_manager.py: ## @@ -63,6 +63,11 @@ def get_url_login(self, **kwargs) -> str: """Return the login page url."""

[GitHub] [airflow] Adaverse commented on a diff in pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
Adaverse commented on code in PR #32810: URL: https://github.com/apache/airflow/pull/32810#discussion_r1279728858 ## airflow/cli/cli_parser.py: ## @@ -65,6 +65,9 @@ ALL_COMMANDS_DICT: dict[str, CLICommand] = {sp.name: sp for sp in airflow_commands} +# Hides the subcommands

[GitHub] [airflow] Adaverse commented on a diff in pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
Adaverse commented on code in PR #32810: URL: https://github.com/apache/airflow/pull/32810#discussion_r1279720576 ## tests/cli/commands/test_db_command.py: ## @@ -37,8 +37,11 @@ def setup_class(cls): @mock.patch("airflow.cli.commands.db_command.db.initdb") def test_c

[GitHub] [airflow] ferruzzi commented on a diff in pull request #31816: openlineage, sagemaker: add OpenLineage support for SageMaker's Processing, Transform and Training operators

2023-07-31 Thread via GitHub
ferruzzi commented on code in PR #31816: URL: https://github.com/apache/airflow/pull/31816#discussion_r127971 ## airflow/providers/amazon/aws/operators/sagemaker.py: ## @@ -647,9 +693,13 @@ def execute(self, context: Context) -> dict: method_name="execute_co

[GitHub] [airflow] potiuk merged pull request #32975: Output for quarantined tests is not hidden by default in CI

2023-07-31 Thread via GitHub
potiuk merged PR #32975: URL: https://github.com/apache/airflow/pull/32975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.a

[airflow] branch main updated: Output for quarantined tests is not hidden by default in CI (#32975)

2023-07-31 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 357d35c76d Output for quarantined tests is not hidd

[GitHub] [airflow] vincbeck commented on a diff in pull request #32819: Handle logout by auth manager

2023-07-31 Thread via GitHub
vincbeck commented on code in PR #32819: URL: https://github.com/apache/airflow/pull/32819#discussion_r1279711521 ## tests/www/views/test_session.py: ## @@ -95,10 +95,4 @@ def test_check_active_user(app, user_client): user.active = False resp = user_client.get("/home")

[GitHub] [airflow] vincbeck opened a new pull request, #32976: Increase the number of attempts in AWS system test `example_rds_export`

2023-07-31 Thread via GitHub
vincbeck opened a new pull request, #32976: URL: https://github.com/apache/airflow/pull/32976 This system test failed recently because the number of attempts exceeded. See exceptions below: ``` ERROR [airflow.task] Task failed with exception -- 1348 | Traceback (most recent c

[GitHub] [airflow] potiuk opened a new pull request, #32975: Output for quarantined tests is not hidden by default in CI

2023-07-31 Thread via GitHub
potiuk opened a new pull request, #32975: URL: https://github.com/apache/airflow/pull/32975 When we ran quarantined tests the output of the tests was hidden, because we are not running the tests in parallell and only parallel tests are summarising test results after they finish and hide the

[GitHub] [airflow] Adaverse commented on a diff in pull request #32810: Rename `db upgrade` to `db migrate` and add `connections create-default-connections`

2023-07-31 Thread via GitHub
Adaverse commented on code in PR #32810: URL: https://github.com/apache/airflow/pull/32810#discussion_r1279671156 ## tests/cli/commands/test_db_command.py: ## @@ -37,8 +37,11 @@ def setup_class(cls): @mock.patch("airflow.cli.commands.db_command.db.initdb") def test_c

[GitHub] [airflow] ferruzzi commented on pull request #32869: Add Amazon EventBridge PutRule hook and operator

2023-07-31 Thread via GitHub
ferruzzi commented on PR #32869: URL: https://github.com/apache/airflow/pull/32869#issuecomment-1658851294 The method you renamed didn't get changed in some tests. :P -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

  1   2   >