[airflow] branch main updated (dffb0d27f2 -> 400f837dfc)
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from dffb0d27f2 Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor` (#22737) add 400f837dfc Move database config move note to `main` section (#22929) No new revisions were added by this update. Summary of changes: UPDATING.md | 26 +- 1 file changed, 13 insertions(+), 13 deletions(-)
[GitHub] [airflow] eladkal merged pull request #22929: Move database config move note to `main` section
eladkal merged PR #22929: URL: https://github.com/apache/airflow/pull/22929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22929: Move database config move note to `main` section
github-actions[bot] commented on PR #22929: URL: https://github.com/apache/airflow/pull/22929#issuecomment-1096109089 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22930: Use full version string for deprecated config
github-actions[bot] commented on PR #22930: URL: https://github.com/apache/airflow/pull/22930#issuecomment-1096107359 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor` (#22737)
This is an automated email from the ASF dual-hosted git repository.
eladkal pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new dffb0d27f2 Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor
of `S3KeySensor` (#22737)
dffb0d27f2 is described below
commit dffb0d27f25b9a1f2497535cb87de3b889aae9d0
Author: Vincent <97131062+vincb...@users.noreply.github.com>
AuthorDate: Tue Apr 12 01:22:51 2022 -0400
Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor`
(#22737)
Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor`
Co-authored-by: Ash Berlin-Taylor
---
.../amazon/aws/example_dags/example_s3_bucket.py | 58 +-
airflow/providers/amazon/aws/hooks/s3.py | 59 +-
airflow/providers/amazon/aws/sensors/s3.py | 218 +
airflow/sensors/s3_prefix_sensor.py| 4 +-
.../operators/s3.rst | 57 +-
tests/providers/amazon/aws/hooks/test_s3.py| 20 ++
tests/providers/amazon/aws/sensors/test_s3_key.py | 160 +--
.../providers/amazon/aws/sensors/test_s3_prefix.py | 41 ++--
8 files changed, 399 insertions(+), 218 deletions(-)
diff --git a/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
b/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
index d04bbab048..3e1c41d862 100644
--- a/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
+++ b/airflow/providers/amazon/aws/example_dags/example_s3_bucket.py
@@ -17,6 +17,7 @@
import os
from datetime import datetime
+from typing import List
from airflow.models.baseoperator import chain
from airflow.models.dag import DAG
@@ -27,11 +28,15 @@ from airflow.providers.amazon.aws.operators.s3 import (
S3GetBucketTaggingOperator,
S3PutBucketTaggingOperator,
)
+from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
BUCKET_NAME = os.environ.get('BUCKET_NAME', 'test-airflow-12345')
+KEY = os.environ.get('KEY', 'key')
+KEY_2 = os.environ.get('KEY_2', 'key2')
TAG_KEY = os.environ.get('TAG_KEY', 'test-s3-bucket-tagging-key')
TAG_VALUE = os.environ.get('TAG_VALUE', 'test-s3-bucket-tagging-value')
+
with DAG(
dag_id='example_s3_bucket',
schedule_interval=None,
@@ -39,6 +44,22 @@ with DAG(
catchup=False,
tags=['example'],
) as dag:
+# [START howto_sensor_s3_key_function_definition]
+def check_fn(files: List) -> bool:
+"""
+Example of custom check: check if all files are bigger than 1kB
+
+:param files: List of S3 object attributes.
+Format: [{
+'Size': int
+}]
+:return: true if the criteria is met
+:rtype: bool
+"""
+return all(f.get('Size', 0) > 1024 for f in files)
+
+# [END howto_sensor_s3_key_function_definition]
+
# [START howto_operator_s3_create_bucket]
create_bucket = S3CreateBucketOperator(
task_id='s3_create_bucket',
@@ -69,10 +90,45 @@ with DAG(
)
# [END howto_operator_s3_delete_bucket_tagging]
+# [START howto_sensor_s3_key_single_key]
+# Check if a file exists
+s3_sensor_one_key = S3KeySensor(
+task_id="s3_sensor_one_key",
+bucket_name=BUCKET_NAME,
+bucket_key=KEY,
+)
+# [END howto_sensor_s3_key_single_key]
+
+# [START howto_sensor_s3_key_multiple_keys]
+# Check if both files exist
+s3_sensor_two_keys = S3KeySensor(
+task_id="s3_sensor_two_keys",
+bucket_name=BUCKET_NAME,
+bucket_key=[KEY, KEY_2],
+)
+# [END howto_sensor_s3_key_multiple_keys]
+
+# [START howto_sensor_s3_key_function]
+# Check if a file exists and match a certain pattern defined in check_fn
+s3_sensor_key_function = S3KeySensor(
+task_id="s3_sensor_key_function",
+bucket_name=BUCKET_NAME,
+bucket_key=KEY,
+check_fn=check_fn,
+)
+# [END howto_sensor_s3_key_function]
+
# [START howto_operator_s3_delete_bucket]
delete_bucket = S3DeleteBucketOperator(
task_id='s3_delete_bucket', bucket_name=BUCKET_NAME, force_delete=True
)
# [END howto_operator_s3_delete_bucket]
-chain(create_bucket, put_tagging, get_tagging, delete_tagging,
delete_bucket)
+chain(
+create_bucket,
+put_tagging,
+get_tagging,
+delete_tagging,
+[s3_sensor_one_key, s3_sensor_two_keys, s3_sensor_key_function],
+delete_bucket,
+)
diff --git a/airflow/providers/amazon/aws/hooks/s3.py
b/airflow/providers/amazon/aws/hooks/s3.py
index fc23255a78..38416b4e89 100644
--- a/airflow/providers/amazon/aws/hooks/s3.py
+++ b/airflow/providers/amazon/aws/hooks/s3.py
@@ -348,26 +348,71 @@ class S3Hook(AwsBaseHook):
return self._list_key_object_filter(keys, from_datetime, to_datetime)
+@provide_bucket_name
+
[GitHub] [airflow] eladkal merged pull request #22737: Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor`
eladkal merged PR #22737: URL: https://github.com/apache/airflow/pull/22737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22737: Deprecate `S3PrefixSensor` and `S3KeySizeSensor` in favor of `S3KeySensor`
github-actions[bot] commented on PR #22737: URL: https://github.com/apache/airflow/pull/22737#issuecomment-1096071088 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #22832: Deprecate `DummyOperator` in favor of `EmptyOperator`
eladkal commented on PR #22832: URL: https://github.com/apache/airflow/pull/22832#issuecomment-1096065814 hi @ashb @uranusjr WDYT? I'd like to get it in 2.3.0 if possible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] tirkarthi opened a new pull request, #22934: Use appropriate TaskInstance object related to TaskFail to get execution_date
tirkarthi opened a new pull request, #22934: URL: https://github.com/apache/airflow/pull/22934 `TaskFail` instance has no execution_date attribute. Fetch execution_date from the relevant `TaskInstance` object to form the dictionary key tuple. closes: #22933 related: #22933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] tanelk commented on issue #22779: Paused DAG manual running task has no external_ executor_ id
tanelk commented on issue #22779: URL: https://github.com/apache/airflow/issues/22779#issuecomment-1095987096 Did I understand correctly - in 2.2.t the external_executor_id is filled, but the issue still there? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22932: Added check before self.clear_xcom_data to ensure XCom is not cleared when resuming from deferral.
github-actions[bot] commented on PR #22932: URL: https://github.com/apache/airflow/pull/22932#issuecomment-1095974045 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #22933: Task duration page crashes when there is a failed task entry
tirkarthi commented on issue #22933:
URL: https://github.com/apache/airflow/issues/22933#issuecomment-1095972343
Sample test case, I will submit a PR shortly.
```python
def test_task_fail_duration(app, admin_client, dag_maker, session):
"""Task duration page with a TaskFail entry should render without
error."""
with dag_maker() as dag:
op1 = BashOperator(task_id='fail', bash_command='sleep 5 && exit 1')
op2 = BashOperator(task_id='success', bash_command='exit 0')
with pytest.raises(AirflowException):
op1.run()
op2.run()
op1_fails = (
session.query(TaskFail)
.filter(
TaskFail.task_id == 'fail',
TaskFail.dag_id == dag.dag_id,
)
.all()
)
op2_fails = (
session.query(TaskFail)
.filter(
TaskFail.task_id == 'success',
TaskFail.dag_id == dag.dag_id,
)
.all()
)
assert len(op1_fails) == 1
assert len(op2_fails) == 0
with unittest.mock.patch.object(app, 'dag_bag') as mocked_dag_bag:
mocked_dag_bag.get_dag.return_value = dag
resp = admin_client.get(f"dags/{dag.dag_id}/duration",
follow_redirects=True)
breakpoint()
assert resp.status_code == 200
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] tirkarthi opened a new issue, #22933: Task duration page crashes when there is a failed task entry
tirkarthi opened a new issue, #22933:
URL: https://github.com/apache/airflow/issues/22933
### Apache Airflow version
2.2.5 (latest released)
### What happened
Task duration page crashes when there is a failed task in the list of task
instances. This happens since the chart calculates duration and accesses
`execution_date` on `TaskFail` instances which don't exist. The query should be
modified such that relevant `TaskInstance` execution_date field should be used
based on matching run_id between `TaskInstance` and `TaskFail`.
### What you think should happen instead
_No response_
### How to reproduce
1. Create a dag that fails.
2. After an execution visit the task duration page.
```python
import datetime
from airflow import DAG
from airflow.operators.bash import BashOperator
with DAG(
"raise_exception_1",
description="DAG that cuases failure",
tags=["example-failure"],
catchup=False,
start_date=datetime.datetime(2022, 3, 28),
default_args={
"depends_on_past": False,
"email": ["airf...@example.com"],
"email_on_failure": False,
"email_on_retry": False,
"retries": 0,
},
) as dag:
t1 = BashOperator(task_id="sleep_5", bash_command="sleep 5 && invalid")
t1 = BashOperator(task_id="sleep_10", bash_command="sleep 10")
```
```python
Python version: 3.10.4
Airflow version: 2.3.0.dev0
Node: laptop
---
Traceback (most recent call last):
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/app.py",
line 2447, in wsgi_app
response = self.full_dispatch_request()
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/app.py",
line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/app.py",
line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/_compat.py",
line 39, in reraise
raise value
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/app.py",
line 1950, in full_dispatch_request
rv = self.dispatch_request()
File
"/home/karthikeyan/stuff/python/airflow/.env/lib/python3.10/site-packages/flask/app.py",
line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/karthikeyan/stuff/python/airflow/airflow/www/auth.py", line
40, in decorated
return func(*args, **kwargs)
File "/home/karthikeyan/stuff/python/airflow/airflow/www/decorators.py",
line 80, in wrapper
return f(*args, **kwargs)
File "/home/karthikeyan/stuff/python/airflow/airflow/utils/session.py",
line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/home/karthikeyan/stuff/python/airflow/airflow/www/views.py", line
2911, in duration
failed_task_instance.execution_date,
AttributeError: 'TaskFail' object has no attribute 'execution_date'
```
### Operating System
Ubuntu 20.04
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] nitra-1 commented on issue #21668: Airflow airflow:2.0.2 UI is not getting updated with dag status
nitra-1 commented on issue #21668: URL: https://github.com/apache/airflow/issues/21668#issuecomment-1095949554 Any update on this issue ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] wangzhijunwang commented on issue #22779: Paused DAG manual running task has no external_ executor_ id
wangzhijunwang commented on issue #22779: URL: https://github.com/apache/airflow/issues/22779#issuecomment-1095917067 @tanelk Sorry to bother you again! First, I use the Web UI to operate，And the example of my operation this time comes from https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html  By upgrading to version 2.2.4, I did find that running task external external_executor_id was filled, but version 2.1.4 external_executor_id is empty -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pingzh commented on a diff in pull request #22917: remove stale serialized dags
pingzh commented on code in PR #22917: URL: https://github.com/apache/airflow/pull/22917#discussion_r847844780 ## airflow/models/serialized_dag.py: ## @@ -220,6 +220,36 @@ def remove_dag(cls, dag_id: str, session: Session = None): """ session.execute(cls.__table__.delete().where(cls.dag_id == dag_id)) +@classmethod +@provide_session +def remove_unknown_dags_in_dag_file( +cls, active_dag_ids: List[str], dag_file_path: str, session=None +) -> List[str]: +""" +Deletes serialized_dags whose dag_ids are not in the active_dag_ids for a given dag_file_path + +:return: a list of inactive dag ids +:rtype: list[str] +""" +file_path_hash = DagCode.dag_fileloc_hash(dag_file_path) +# there is a db index on fileloc_hash +rows = session.query(cls.dag_id).filter(cls.fileloc_hash == file_path_hash).all() +dag_ids_from_db = [i[0] for i in rows] + +dag_ids_to_remove = [dag_id for dag_id in dag_ids_from_db if dag_id not in active_dag_ids] + +if dag_ids_to_remove: +# delete one by one to avoid locking the table Review Comment: good question. I am not sure about the other dbs, as we only use mysql. but i think deleting with small batch could have smaller load in db. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pingzh commented on a diff in pull request #22917: remove stale serialized dags
pingzh commented on code in PR #22917: URL: https://github.com/apache/airflow/pull/22917#discussion_r847844344 ## airflow/models/serialized_dag.py: ## @@ -220,6 +220,36 @@ def remove_dag(cls, dag_id: str, session: Session = None): """ session.execute(cls.__table__.delete().where(cls.dag_id == dag_id)) +@classmethod +@provide_session +def remove_unknown_dags_in_dag_file( +cls, active_dag_ids: List[str], dag_file_path: str, session=None +) -> List[str]: +""" +Deletes serialized_dags whose dag_ids are not in the active_dag_ids for a given dag_file_path + +:return: a list of inactive dag ids +:rtype: list[str] +""" +file_path_hash = DagCode.dag_fileloc_hash(dag_file_path) +# there is a db index on fileloc_hash +rows = session.query(cls.dag_id).filter(cls.fileloc_hash == file_path_hash).all() +dag_ids_from_db = [i[0] for i in rows] + +dag_ids_to_remove = [dag_id for dag_id in dag_ids_from_db if dag_id not in active_dag_ids] Review Comment: good call. but i don't think it is necessary to check here. if we think collision is an issue, we might want to change the way it is generated. let me know your thoughts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ReadytoRocc opened a new pull request, #22932: Added check before self.clear_xcom_data to ensure XCom is not cleared when resuming from deferral.
ReadytoRocc opened a new pull request, #22932: URL: https://github.com/apache/airflow/pull/22932 closes: #22931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ReadytoRocc opened a new issue, #22931: XCom is cleared when a task resumes from deferral.
ReadytoRocc opened a new issue, #22931:
URL: https://github.com/apache/airflow/issues/22931
### Apache Airflow version
2.2.5 (latest released)
### What happened
A task's XCom value is cleared when a task is rescheduled after being
deferred.
### What you think should happen instead
XCom should not be cleared in this case, as it is still the same task run.
### How to reproduce
```
from datetime import datetime, timedelta
from airflow import DAG
from airflow.models import BaseOperator
from airflow.triggers.temporal import TimeDeltaTrigger
class XComPushDeferOperator(BaseOperator):
def execute(self, context):
context["ti"].xcom_push("test", "test_value")
self.defer(
trigger=TimeDeltaTrigger(delta=timedelta(seconds=10)),
method_name="next",
)
def next(self, context, event=None):
pass
with DAG(
"xcom_clear", schedule_interval=None, start_date=datetime(2022, 4, 11),
) as dag:
XComPushDeferOperator(task_id="xcom_push")
```
### Operating System
macOS
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] jedcunningham opened a new pull request, #22930: Use full version string for deprecated config
jedcunningham opened a new pull request, #22930: URL: https://github.com/apache/airflow/pull/22930 Since this is shown to the users in docs, we should use the full version string. Before:  After:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #22928: Update ImportError items instead of deleting and recreating them
jedcunningham commented on code in PR #22928: URL: https://github.com/apache/airflow/pull/22928#discussion_r847827828 ## airflow/dag_processing/processor.py: ## @@ -528,23 +529,34 @@ def update_import_errors(session: Session, dagbag: DagBag) -> None: :param session: session for ORM operations :param dagbag: DagBag containing DAGs with import errors """ -# Clear the errors of the processed files -for dagbag_file in dagbag.file_last_changed: -session.query(errors.ImportError).filter( -errors.ImportError.filename.startswith(dagbag_file) -).delete(synchronize_session="fetch") +last_changed = copy.copy(dagbag.file_last_changed) Review Comment: I can't say I like this variable name. I had to walk this whole section to figure out what it was. Maybe instead we name this `all_parsed_files`, keep a separate set of files that had errors, and iterate `all_parsed_files - files_with_errors`? ## airflow/dag_processing/processor.py: ## @@ -528,23 +529,34 @@ def update_import_errors(session: Session, dagbag: DagBag) -> None: :param session: session for ORM operations :param dagbag: DagBag containing DAGs with import errors """ -# Clear the errors of the processed files -for dagbag_file in dagbag.file_last_changed: -session.query(errors.ImportError).filter( -errors.ImportError.filename.startswith(dagbag_file) -).delete(synchronize_session="fetch") +last_changed = copy.copy(dagbag.file_last_changed) +import_error_files = [x.filename for x in session.query(errors.ImportError.filename).all()] Review Comment: Maybe `existing_import_error_files`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19949: Only update label when adopting to prevent new pods
github-actions[bot] commented on PR #19949: URL: https://github.com/apache/airflow/pull/19949#issuecomment-1095719461 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21716: Respect user-configured timezone when generating DAG run ID
github-actions[bot] commented on PR #21716: URL: https://github.com/apache/airflow/pull/21716#issuecomment-1095719409 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham opened a new pull request, #22929: Move database config move note to `main` section
jedcunningham opened a new pull request, #22929: URL: https://github.com/apache/airflow/pull/22929 This was accidentally added to the `2.2.4` notes instead of `main`. (I also added a missing "the") -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Fix changelog spelling (#22926)
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new a9e790cf61 Fix changelog spelling (#22926) a9e790cf61 is described below commit a9e790cf613a90ed8fa22b7cb38afaff8cdcc5a5 Author: Jed Cunningham <66968678+jedcunning...@users.noreply.github.com> AuthorDate: Mon Apr 11 17:26:42 2022 -0600 Fix changelog spelling (#22926) --- CHANGELOG.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 289964bd04..5289344fad 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -9,7 +9,7 @@ Bug Fixes - Fix incorrect data provided to tries & landing times charts (#21928) - Fix assignment of unassigned triggers (#21770) - Fix triggerer ``--capacity`` parameter (#21753) -- Fix graph autorefresh on page load (#21736) +- Fix graph auto-refresh on page load (#21736) - Fix filesystem sensor for directories (#21729) - Fix stray ``order_by(TaskInstance.execution_date)`` (#21705) - Correctly handle multiple '=' in LocalFileSystem secrets. (#21694) @@ -24,7 +24,7 @@ Bug Fixes - Reduce DB load incurred by Stale DAG deactivation (#21399) - Fix race condition between triggerer and scheduler (#21316) - Fix trigger dag redirect from task instance log view (#21239) -- Log traceback in trigger excs (#21213) +- Log traceback in trigger exceptions (#21213) - A trigger might use a connection; make sure we mask passwords (#21207) - Update ``ExternalTaskSensorLink`` to handle templated ``external_dag_id`` (#21192) - Ensure ``clear_task_instances`` sets valid run state (#21116)
[GitHub] [airflow] jedcunningham merged pull request #22926: Fix changelog spelling
jedcunningham merged PR #22926: URL: https://github.com/apache/airflow/pull/22926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy opened a new pull request, #22928: Update ImportError items instead of deleting and recreating them
ephraimbuddy opened a new pull request, #22928: URL: https://github.com/apache/airflow/pull/22928 Each time a dag with import error is parsed, the ImportError record is deleted and a new record is created. For example, say I have two dags with import errors, initially, the import error id will be dag_1:import_error.id=1, dag2:import_error.id=2. In the next dag parsing, the import error will increase. dag_1:import_error.id=3, dag_2:import_error.id=4 and it continues like that. This makes it impossible for the get import error REST API endpoint to be consistent This PR fixes this issue by updating the existing record and creating a new one if no record exists -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22926: Fix changelog spelling
github-actions[bot] commented on PR #22926: URL: https://github.com/apache/airflow/pull/22926#issuecomment-1095682428 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on pull request #22927: Add MSSQL link to contributing quick start.
pierrejeambrun commented on PR #22927: URL: https://github.com/apache/airflow/pull/22927#issuecomment-1095671098 @potiuk Now that breeze2 is out, as requested here is the small change in the doc :) Great work btw on breeze, the interface is great :tada: (kind of make me want to work on this part now :p) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun opened a new pull request, #22927: Add MSSQL link to contributing quick start.
pierrejeambrun opened a new pull request, #22927: URL: https://github.com/apache/airflow/pull/22927 Small PR to update the `CONTRIBUTING_QUICK_START.rst` doc to include the link to mssql form the host. Also tried to homogenize the doc between `setup and develop` sections for PyCharm/VS Code/Gitpod online workspaces. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch constraints-main updated: Updating constraints. Build id:2151407881
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 4c56c47089 Updating constraints. Build id:2151407881 4c56c47089 is described below commit 4c56c47089328676e3128ecbccd8ecebb174b6fa Author: Automated GitHub Actions commit AuthorDate: Mon Apr 11 22:47:28 2022 + Updating constraints. Build id:2151407881 This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of 'refs/heads/main' in 'apache/airflow' with commit sha d8889da29ccfcbecd2c89b9e8e278c480767d678. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.10.txt | 39 --- constraints-3.7.txt | 39 --- constraints-3.8.txt | 39 --- constraints-3.9.txt | 39 --- constraints-no-providers-3.10.txt | 4 ++-- constraints-no-providers-3.7.txt | 4 ++-- constraints-no-providers-3.8.txt | 4 ++-- constraints-no-providers-3.9.txt | 4 ++-- constraints-source-providers-3.10.txt | 10 - constraints-source-providers-3.7.txt | 10 - constraints-source-providers-3.8.txt | 10 - constraints-source-providers-3.9.txt | 10 - 12 files changed, 100 insertions(+), 112 deletions(-) diff --git a/constraints-3.10.txt b/constraints-3.10.txt index 6527dd9db2..831416adee 100644 --- a/constraints-3.10.txt +++ b/constraints-3.10.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2022-04-11T11:26:08Z +# This constraints file was automatically generated on 2022-04-11T22:42:03Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -60,7 +60,7 @@ ansiwrap==0.8.4 anyio==3.5.0 apache-airflow-providers-airbyte==2.1.4 apache-airflow-providers-alibaba==1.1.1 -apache-airflow-providers-amazon==3.2.0 +apache-airflow-providers-amazon==3.3.0 apache-airflow-providers-apache-beam==3.3.0 apache-airflow-providers-apache-cassandra==2.1.3 apache-airflow-providers-apache-drill==1.0.4 @@ -68,31 +68,31 @@ apache-airflow-providers-apache-druid==2.3.3 apache-airflow-providers-apache-hdfs==2.2.3 apache-airflow-providers-apache-hive==2.3.2 apache-airflow-providers-apache-kylin==2.0.4 -apache-airflow-providers-apache-livy==2.2.2 +apache-airflow-providers-apache-livy==2.2.3 apache-airflow-providers-apache-pig==2.0.4 apache-airflow-providers-apache-pinot==2.0.4 apache-airflow-providers-apache-spark==2.1.3 apache-airflow-providers-apache-sqoop==2.1.3 apache-airflow-providers-asana==1.1.3 -apache-airflow-providers-celery==2.1.3 +apache-airflow-providers-celery==2.1.4 apache-airflow-providers-cloudant==2.0.4 apache-airflow-providers-cncf-kubernetes==3.1.2 apache-airflow-providers-databricks==2.5.0 apache-airflow-providers-datadog==2.0.4 apache-airflow-providers-dbt-cloud==1.0.2 apache-airflow-providers-dingding==2.0.4 -apache-airflow-providers-discord==2.0.4 -apache-airflow-providers-docker==2.5.2 -apache-airflow-providers-elasticsearch==2.2.0 +apache-airflow-providers-discord==2.1.4 +apache-airflow-providers-docker==2.6.0 +apache-airflow-providers-elasticsearch==3.0.3 apache-airflow-providers-exasol==2.1.3 apache-airflow-providers-facebook==2.2.3 -apache-airflow-providers-google==6.7.0 +apache-airflow-providers-google==6.8.0 apache-airflow-providers-grpc==2.0.4 -apache-airflow-providers-hashicorp==2.1.4 +apache-airflow-providers-hashicorp==2.2.0 apache-airflow-providers-jdbc==2.1.3 -apache-airflow-providers-jenkins==2.0.7 +apache-airflow-providers-jenkins==2.1.0 apache-airflow-providers-jira==2.0.4 -apache-airflow-providers-microsoft-azure==3.7.2 +apache-airflow-providers-microsoft-azure==3.8.0 apache-airflow-providers-microsoft-mssql==2.1.3 apache-airflow-providers-microsoft-winrm==2.0.5 apache-airflow-providers-mongo==2.3.3 @@ -106,20 +106,20 @@ apache-airflow-providers-pagerduty==2.1.3 apache-airflow-providers-papermill==2.2.3 apache-airflow-providers-plexus==2.0.4 apache-airflow-providers-postgres==4.1.0 -apache-airflow-providers-presto==2.1.2 +apache-airflow-providers-presto==2.2.0 apache-airflow-providers-qubole==2.1.3 apache-airflow-providers-redis==2.0.4 apache-airflow-providers-salesforce==3.4.3 apache-airflow-providers-samba==3.0.4 apache-airflow-providers-segment==2.0.4 -apache-airflow-providers-sftp==2.5.2
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22925: Update Dockerfile - set proper airflow version
boring-cyborg[bot] commented on PR #22925: URL: https://github.com/apache/airflow/pull/22925#issuecomment-1095644655 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] piotrekklis opened a new pull request, #22925: Update Dockerfile - set proper airflow version
piotrekklis opened a new pull request, #22925: URL: https://github.com/apache/airflow/pull/22925 Currently set airflow version won't let to build the image --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22699: add allow_manual DAG argument to prevent scheduled DAGs from being triggered manually
uranusjr commented on PR #22699: URL: https://github.com/apache/airflow/pull/22699#issuecomment-1095613028 You need to find all `create_dagrun` calls that creates manual runs, and decide how this needs to be catched and handled. I believe there are at least two, one in CLI and one in web UI, and you need to emit the correct response in those situations as appropriate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (b8da7a3c8f -> d8889da29c)
This is an automated email from the ASF dual-hosted git repository. kamilbregula pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from b8da7a3c8f Remove installation instructions from Breeze's cheatsheet (#22923) add d8889da29c Move the database configuration to a new section (#22284) No new revisions were added by this update. Summary of changes: CHANGELOG.txt | 2 +- UPDATING.md| 17 ++ airflow/cli/commands/info_command.py | 2 +- airflow/cli/commands/standalone_command.py | 2 +- airflow/config_templates/config.yml| 250 +++-- airflow/config_templates/default_airflow.cfg | 141 ++-- airflow/config_templates/default_test.cfg | 6 +- airflow/configuration.py | 18 +- airflow/dag_processing/manager.py | 2 +- airflow/jobs/scheduler_job.py | 2 +- .../dags_in_image_template.yaml| 2 +- .../dags_in_volume_template.yaml | 2 +- .../git_sync_template.yaml | 2 +- .../basic_template.yaml| 2 +- airflow/models/base.py | 6 +- airflow/settings.py| 22 +- airflow/utils/db.py| 2 +- airflow/utils/retries.py | 2 +- airflow/utils/sqlalchemy.py| 4 +- airflow/www/app.py | 4 +- chart/templates/_helpers.yaml | 8 + chart/values.schema.json | 7 +- chart/values.yaml | 4 +- docs/apache-airflow/cli-and-env-variables-ref.rst | 2 +- docs/apache-airflow/howto/set-config.rst | 22 +- docs/apache-airflow/howto/set-up-database.rst | 10 +- docs/apache-airflow/production-deployment.rst | 2 +- docs/apache-airflow/start/docker-compose.yaml | 2 + docs/docker-stack/README.md| 2 +- docs/docker-stack/index.rst| 2 +- docs/helm-chart/airflow-configuration.rst | 2 +- docs/helm-chart/production-guide.rst | 2 +- .../ci/docker-compose/backend-mssql-bullseye.yml | 2 +- scripts/ci/docker-compose/backend-mssql-buster.yml | 2 +- scripts/ci/docker-compose/backend-mysql.yml| 2 +- scripts/ci/docker-compose/backend-postgres.yml | 2 +- scripts/ci/docker-compose/backend-sqlite.yml | 2 +- tests/charts/test_airflow_common.py| 2 + tests/cli/commands/test_info_command.py| 8 +- tests/core/test_config_templates.py| 2 + tests/core/test_configuration.py | 26 +-- tests/core/test_sqlalchemy_config.py | 10 +- tests/dag_processing/test_manager.py | 12 +- tests/models/test_base.py | 6 +- tests/test_utils/perf/sql_queries.py | 2 +- 45 files changed, 346 insertions(+), 287 deletions(-)
[GitHub] [airflow] mik-laj merged pull request #22284: Move the database configuration to a new section
mik-laj merged PR #22284: URL: https://github.com/apache/airflow/pull/22284 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22284: Move the database configuration to a new section
boring-cyborg[bot] commented on PR #22284: URL: https://github.com/apache/airflow/pull/22284#issuecomment-1095612612 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #15930: Move the database configuration to a new section
mik-laj closed issue #15930: Move the database configuration to a new section URL: https://github.com/apache/airflow/issues/15930 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22699: add allow_manual DAG argument to prevent scheduled DAGs from being triggered manually
uranusjr commented on code in PR #22699: URL: https://github.com/apache/airflow/pull/22699#discussion_r847768433 ## airflow/exceptions.py: ## @@ -298,3 +298,7 @@ def __repr__(self) -> str: class TaskDeferralError(AirflowException): """Raised when a task failed during deferral for some reason.""" + + +class DAGDisallowManual(AirflowException): Review Comment: ```suggestion class ManualDAGRunDisallowed(AirflowException): ``` seems to read better forme. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Remove installation instructions from Breeze's cheatsheet (#22923)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new b8da7a3c8f Remove installation instructions from Breeze's cheatsheet (#22923) b8da7a3c8f is described below commit b8da7a3c8f965dad9f47b569499d33b1698f7b5d Author: Jarek Potiuk AuthorDate: Mon Apr 11 23:51:52 2022 +0200 Remove installation instructions from Breeze's cheatsheet (#22923) The cheatsheet is displayed only after Breeze is installed so it makes no sense to display installation instructions in the cheathsheet. --- dev/breeze/src/airflow_breeze/utils/visuals.py | 13 - 1 file changed, 13 deletions(-) diff --git a/dev/breeze/src/airflow_breeze/utils/visuals.py b/dev/breeze/src/airflow_breeze/utils/visuals.py index a42c250444..e119e83732 100644 --- a/dev/breeze/src/airflow_breeze/utils/visuals.py +++ b/dev/breeze/src/airflow_breeze/utils/visuals.py @@ -76,19 +76,6 @@ CHEATSHEET = f""" [bold][bright_blue]Airflow Breeze Cheatsheet[/][/] -[bright_blue]* Installation[/] - -When you have multiple copies of Airflow, it's better if you use `./breeze` from those -repository as it will have the latest version of breeze and it's dependencies. - -However if you only have one Airflow repository and you have `pipx` installed, you can use -`pipx` to install `breeze` command in your path (`breeze` command is run from this repository then) - -pipx install -e ./dev/breeze --force - -In case you use `pipx`, you might need to occasionally reinstall `breeze` with the `--force` flag -when dependencies change for it. You do not have to do it when you use it via `./breeze` - [bright_blue]* Port forwarding:[/] Ports are forwarded to the running docker containers for webserver and database
[GitHub] [airflow] potiuk merged pull request #22923: Remove installation instructions from Breeze's cheatsheet
potiuk merged PR #22923: URL: https://github.com/apache/airflow/pull/22923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (27b3e31178 -> d1a1fe7b47)
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 27b3e31178 implements #22859 - Add .sql as templatable extension (#22920) add d1a1fe7b47 Fix bug where dynamically mapped tasks got set to REMOVED (#22909) No new revisions were added by this update. Summary of changes: airflow/decorators/base.py | 3 ++ airflow/models/dagrun.py| 52 - airflow/models/mappedoperator.py| 44 ++-- airflow/serialization/serialized_objects.py | 10 +++--- tests/models/test_dagrun.py | 11 +++--- 5 files changed, 101 insertions(+), 19 deletions(-)
[GitHub] [airflow] ashb merged pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb merged PR #22909: URL: https://github.com/apache/airflow/pull/22909 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22483: Priority order tasks even when using pools
github-actions[bot] commented on PR #22483: URL: https://github.com/apache/airflow/pull/22483#issuecomment-1095583680 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22483: Priority order tasks even when using pools
ashb commented on code in PR #22483:
URL: https://github.com/apache/airflow/pull/22483#discussion_r847745645
##
airflow/jobs/scheduler_job.py:
##
@@ -355,141 +355,120 @@ def _executable_task_instances_to_queued(self, max_tis:
int, session: Session =
"%s tasks up for execution:\n\t%s",
len(task_instances_to_examine), task_instance_str
)
-pool_to_task_instances: DefaultDict[str, List[TI]] =
defaultdict(list)
for task_instance in task_instances_to_examine:
-
pool_to_task_instances[task_instance.pool].append(task_instance)
+pool_name = task_instance.pool
-# Go through each pool, and queue up a task for execution if there
are
-# any open slots in the pool.
-
-for pool, task_instances in pool_to_task_instances.items():
-pool_name = pool
-if pool not in pools:
-self.log.warning("Tasks using non-existent pool '%s' will
not be scheduled", pool)
+if pool_name not in pools:
+self.log.warning("Tasks using non-existent pool '%s' will
not be scheduled", pool_name)
starved_pools.add(pool_name)
continue
-pool_total = pools[pool]["total"]
-open_slots = pools[pool]["open"]
+pool_total = pools[pool_name]["total"]
+open_slots = pools[pool_name]["open"]
Review Comment:
```python
pool_stats = pools.get(pool_name)
if not pool_stats:
self.log.warning("Tasks using non-existent pool '%s'
will not be scheduled", pool_name)
starved_pools.add(pool_name)
continue
pool_total = pool_stats["total"]
open_slots = pool_stats["open"]
```
##
airflow/jobs/scheduler_job.py:
##
@@ -355,141 +355,120 @@ def _executable_task_instances_to_queued(self, max_tis:
int, session: Session =
"%s tasks up for execution:\n\t%s",
len(task_instances_to_examine), task_instance_str
)
-pool_to_task_instances: DefaultDict[str, List[TI]] =
defaultdict(list)
for task_instance in task_instances_to_examine:
-
pool_to_task_instances[task_instance.pool].append(task_instance)
+pool_name = task_instance.pool
-# Go through each pool, and queue up a task for execution if there
are
-# any open slots in the pool.
-
-for pool, task_instances in pool_to_task_instances.items():
-pool_name = pool
-if pool not in pools:
-self.log.warning("Tasks using non-existent pool '%s' will
not be scheduled", pool)
+if pool_name not in pools:
+self.log.warning("Tasks using non-existent pool '%s' will
not be scheduled", pool_name)
starved_pools.add(pool_name)
continue
-pool_total = pools[pool]["total"]
-open_slots = pools[pool]["open"]
+pool_total = pools[pool_name]["total"]
+open_slots = pools[pool_name]["open"]
-num_ready = len(task_instances)
-self.log.info(
-"Figuring out tasks to run in Pool(name=%s) with %s open
slots "
-"and %s task instances ready to be queued",
-pool,
-open_slots,
-num_ready,
-)
-
-priority_sorted_task_instances = sorted(
-task_instances, key=lambda ti: (-ti.priority_weight,
ti.execution_date)
-)
+if open_slots <= 0:
+self.log.info(
+"Not scheduling since there are %s open slots in pool
%s", open_slots, pool_name
+)
+# Can't schedule any more since there are no more open
slots.
+pool_num_starving_tasks[pool_name] += 1
+num_starving_tasks_total += 1
+starved_pools.add(pool_name)
+continue
-for current_index, task_instance in
enumerate(priority_sorted_task_instances):
-if open_slots <= 0:
-self.log.info(
-"Not scheduling since there are %s open slots in
pool %s", open_slots, pool
-)
-# Can't schedule any more since there are no more open
slots.
-num_unhandled = len(priority_sorted_task_instances) -
current_index
-pool_num_starving_tasks[pool_name] += num_unhandled
-num_starving_tasks_total += num_unhandled
-
[GitHub] [airflow] uranusjr commented on a diff in pull request #21077: Kubernetes decorator
uranusjr commented on code in PR #21077: URL: https://github.com/apache/airflow/pull/21077#discussion_r847743501 ## airflow/decorators/__init__.pyi: ## @@ -188,8 +188,7 @@ class TaskDecoratorCollection: Dict will unroll to XCom values with keys as XCom keys. Defaults to False. :param use_dill: Whether to use dill or pickle for serialization :param python_command: Python command for executing functions, Default: python3 -:param image: Docker image from which to create the container. -If image tag is omitted, "latest" will be used. +:param image: Docker image from which to create the container.(Required) Review Comment: ```suggestion :param image: Docker image from which to create the container. (Required) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22917: remove stale serialized dags
ashb commented on code in PR #22917: URL: https://github.com/apache/airflow/pull/22917#discussion_r847739746 ## airflow/models/serialized_dag.py: ## @@ -220,6 +220,36 @@ def remove_dag(cls, dag_id: str, session: Session = None): """ session.execute(cls.__table__.delete().where(cls.dag_id == dag_id)) +@classmethod +@provide_session +def remove_unknown_dags_in_dag_file( +cls, active_dag_ids: List[str], dag_file_path: str, session=None +) -> List[str]: +""" +Deletes serialized_dags whose dag_ids are not in the active_dag_ids for a given dag_file_path + +:return: a list of inactive dag ids +:rtype: list[str] +""" +file_path_hash = DagCode.dag_fileloc_hash(dag_file_path) +# there is a db index on fileloc_hash +rows = session.query(cls.dag_id).filter(cls.fileloc_hash == file_path_hash).all() +dag_ids_from_db = [i[0] for i in rows] + +dag_ids_to_remove = [dag_id for dag_id in dag_ids_from_db if dag_id not in active_dag_ids] + +if dag_ids_to_remove: +# delete one by one to avoid locking the table Review Comment: Does this apply to all DBs or just Mysql? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22917: remove stale serialized dags
ashb commented on code in PR #22917: URL: https://github.com/apache/airflow/pull/22917#discussion_r847739271 ## airflow/models/serialized_dag.py: ## @@ -220,6 +220,36 @@ def remove_dag(cls, dag_id: str, session: Session = None): """ session.execute(cls.__table__.delete().where(cls.dag_id == dag_id)) +@classmethod +@provide_session +def remove_unknown_dags_in_dag_file( +cls, active_dag_ids: List[str], dag_file_path: str, session=None +) -> List[str]: +""" +Deletes serialized_dags whose dag_ids are not in the active_dag_ids for a given dag_file_path + +:return: a list of inactive dag ids +:rtype: list[str] +""" +file_path_hash = DagCode.dag_fileloc_hash(dag_file_path) +# there is a db index on fileloc_hash +rows = session.query(cls.dag_id).filter(cls.fileloc_hash == file_path_hash).all() +dag_ids_from_db = [i[0] for i in rows] + +dag_ids_to_remove = [dag_id for dag_id in dag_ids_from_db if dag_id not in active_dag_ids] Review Comment: It's unlikely, but theoretically possible that we could have a hash collision, so we should also check the full path matches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22923: Remove installation instructions from Breeze's cheatsheet
github-actions[bot] commented on PR #22923: URL: https://github.com/apache/airflow/pull/22923#issuecomment-1095567901 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22916: Clarify breeze install message re pipx vs `./breeze`
dstandish commented on PR #22916: URL: https://github.com/apache/airflow/pull/22916#issuecomment-1095567202 haha yeah thought it was bit weird but did not give it much thought -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] edithturn commented on issue #22908: Add nicer error for BREEZE.rst update static check
edithturn commented on issue #22908: URL: https://github.com/apache/airflow/issues/22908#issuecomment-1095556652 @potiuk I reproduced the error. I added breeze installation on static-checks-basic-checks-only, now this step is being skipped, and I don't think it's his purpose to skip it. And static checks are failing with a weird error, looks like everything was fine except after we are adding the images. https://github.com/apache/airflow/runs/5979380037?check_suite_focus=true I will give another shoot modifying something on Breeze.rst. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22916: Clarify breeze install message re pipx vs `./breeze`
potiuk commented on PR #22916: URL: https://github.com/apache/airflow/pull/22916#issuecomment-1095556477 Actuallly - this message should not be there at all :). It's chicken-egg. You won't see it until Breeze is installed, but if you have it installed, you don;t need it any more :) #22923 removes it :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell closed issue #22859: Add .sql template extension rendering for BigQueryInsertJobOperator
josh-fell closed issue #22859: Add .sql template extension rendering for BigQueryInsertJobOperator URL: https://github.com/apache/airflow/issues/22859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #22859: Add .sql template extension rendering for BigQueryInsertJobOperator
josh-fell commented on issue #22859: URL: https://github.com/apache/airflow/issues/22859#issuecomment-109636 Closing via #22920. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #22923: Remove installation instructions from Breeze's cheatsheet
potiuk opened a new pull request, #22923: URL: https://github.com/apache/airflow/pull/22923 The cheatsheet is displayed only after Breeze is installed so it makes no sense to display installation instructions in the cheathsheet. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22922: Reuse reflect_tables helper in db.py
github-actions[bot] commented on PR #22922: URL: https://github.com/apache/airflow/pull/22922#issuecomment-1095552825 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (9c9272de3a -> 27b3e31178)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 9c9272de3a typo in BREEZE.rst (#22919) add 27b3e31178 implements #22859 - Add .sql as templatable extension (#22920) No new revisions were added by this update. Summary of changes: airflow/providers/google/cloud/operators/bigquery.py | 5 - 1 file changed, 4 insertions(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #22920: implements #22859 - Add .sql as templatable extension
potiuk merged PR #22920: URL: https://github.com/apache/airflow/pull/22920 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22920: implements #22859 - Add .sql as templatable extension
boring-cyborg[bot] commented on PR #22920: URL: https://github.com/apache/airflow/pull/22920#issuecomment-1095551963 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish opened a new pull request, #22922: Reuse reflect_tables helper in db.py
dstandish opened a new pull request, #22922: URL: https://github.com/apache/airflow/pull/22922 Just a small code simplification -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847719456 ## tests/models/test_dagrun.py: ## @@ -1036,14 +1036,17 @@ def test_mapped_mixed__literal_not_expanded_at_create(dag_maker, session): mapped = MockOperator.partial(task_id='task_2').expand(arg1=literal, arg2=XComArg(task)) dr = dag_maker.create_dagrun() -indices = ( -session.query(TI.map_index) +query = ( +session.query(TI.map_index, TI.state) .filter_by(task_id=mapped.task_id, dag_id=mapped.dag_id, run_id=dr.run_id) .order_by(TI.map_index) -.all() ) -assert indices == [(-1,)] +assert query.all() == [(-1, None)] + +# Verify_integrity shouldn't change the result now that the TIs exist +dr.verify_integrity() +assert query.all() == [(-1, None)] Review Comment: Oh yes. Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
uranusjr commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847717386 ## tests/models/test_dagrun.py: ## @@ -1036,14 +1036,17 @@ def test_mapped_mixed__literal_not_expanded_at_create(dag_maker, session): mapped = MockOperator.partial(task_id='task_2').expand(arg1=literal, arg2=XComArg(task)) dr = dag_maker.create_dagrun() -indices = ( -session.query(TI.map_index) +query = ( +session.query(TI.map_index, TI.state) .filter_by(task_id=mapped.task_id, dag_id=mapped.dag_id, run_id=dr.run_id) .order_by(TI.map_index) -.all() ) -assert indices == [(-1,)] +assert query.all() == [(-1, None)] + +# Verify_integrity shouldn't change the result now that the TIs exist +dr.verify_integrity() +assert query.all() == [(-1, None)] Review Comment: Passing in `session` should be enough then -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847715879 ## tests/models/test_dagrun.py: ## @@ -1036,14 +1036,17 @@ def test_mapped_mixed__literal_not_expanded_at_create(dag_maker, session): mapped = MockOperator.partial(task_id='task_2').expand(arg1=literal, arg2=XComArg(task)) dr = dag_maker.create_dagrun() -indices = ( -session.query(TI.map_index) +query = ( +session.query(TI.map_index, TI.state) .filter_by(task_id=mapped.task_id, dag_id=mapped.dag_id, run_id=dr.run_id) .order_by(TI.map_index) -.all() ) -assert indices == [(-1,)] +assert query.all() == [(-1, None)] + +# Verify_integrity shouldn't change the result now that the TIs exist +dr.verify_integrity() +assert query.all() == [(-1, None)] Review Comment: It has a flush inside it already https://github.com/apache/airflow/blob/9c9272de3a32d30d831fac1272a07244d5fb8e0b/airflow/models/dagrun.py#L903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
uranusjr commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847713698 ## tests/models/test_dagrun.py: ## @@ -1036,14 +1036,17 @@ def test_mapped_mixed__literal_not_expanded_at_create(dag_maker, session): mapped = MockOperator.partial(task_id='task_2').expand(arg1=literal, arg2=XComArg(task)) dr = dag_maker.create_dagrun() -indices = ( -session.query(TI.map_index) +query = ( +session.query(TI.map_index, TI.state) .filter_by(task_id=mapped.task_id, dag_id=mapped.dag_id, run_id=dr.run_id) .order_by(TI.map_index) -.all() ) -assert indices == [(-1,)] +assert query.all() == [(-1, None)] + +# Verify_integrity shouldn't change the result now that the TIs exist +dr.verify_integrity() +assert query.all() == [(-1, None)] Review Comment: We probably need to do this? ```suggestion # Verify_integrity shouldn't change the result now that the TIs exist dr.verify_integrity(session=session) session.flush() assert query.all() == [(-1, None)] ``` Otherwise `verify_integrity` would use a new session, and the query result won’t change regardless due to transaction isolation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #22846: UUID encoded in CassandraToGCSOperator but not other operators
uranusjr commented on issue #22846: URL: https://github.com/apache/airflow/issues/22846#issuecomment-1095537819 cc @jgao54 in case you know some more details on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #22846: UUID encoded in CassandraToGCSOperator but not other operators
uranusjr commented on issue #22846: URL: https://github.com/apache/airflow/issues/22846#issuecomment-1095537023 You can’t simply revert it since that would introduce a backward incompatibility and break existing usages. Perhaps it’s possible to add a flag on the operator to toggle the format used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22920: implements #22859 - Add .sql as templatable extension
github-actions[bot] commented on PR #22920: URL: https://github.com/apache/airflow/pull/22920#issuecomment-1095535478 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22921: Add revision version map to db.py
dstandish commented on PR #22921: URL: https://github.com/apache/airflow/pull/22921#issuecomment-1095525973 > Do we really need to know it that precisely? Can't we use the `VERSION_REVISION_MAP`and walk the migrations history via alembic api? > > Or could we just "duck-type" the database and check what is or isn't present and make decisions based on that. > > (I'm not a huge fan of having to have a whole load of constants in the code that get updated by hooks) I think the duck typing approach is messy and I'd rather just be precise about it. But, we can look it up when running the upgrade command, that is fine -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #22921: Add revision version map to db.py
dstandish closed pull request #22921: Add revision version map to db.py URL: https://github.com/apache/airflow/pull/22921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847700088 ## airflow/models/mappedoperator.py: ## @@ -559,7 +562,12 @@ def _get_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]: for task_id, length in xcom_query: for mapped_arg_name in mapped_dep_keys[task_id]: map_lengths[mapped_arg_name] += length +return map_lengths +def _resolve_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]: +"""Return dict of argument name to map length, or throw if some are not resolvable""" +expansion_kwargs = self._get_expansion_kwargs() +map_lengths = self._get_map_lengths(run_id, session=session) Review Comment: Done: > """Return dict of argument name to map length. > >If any arguments are not known right now (upstream task not finished) they will not be present in the >dict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #22737: Deprecate S3PrefixSensor
vincbeck commented on PR #22737: URL: https://github.com/apache/airflow/pull/22737#issuecomment-1095518033 I also edited the PR description to add a section to explain the purpose of `default_check_fn` since it might not be very obvious -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #22737: Deprecate S3PrefixSensor
vincbeck commented on PR #22737: URL: https://github.com/apache/airflow/pull/22737#issuecomment-1095517182 I only left tests related to deprecation warnings and default function in `TestS3KeySizeSensor`. I removed the other tests because these use cases are already covered in `TestS3KeySensor` I also moved `default_check_fn` in `S3KeySizeSensor` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
jedcunningham commented on code in PR #22909:
URL: https://github.com/apache/airflow/pull/22909#discussion_r847691219
##
airflow/models/dagrun.py:
##
@@ -697,8 +697,9 @@ def _get_ready_tis(
old_states[schedulable.key] = old_state
continue
-# Expansion of last resort! This is ideally handled in the
mini-scheduler in LocalTaskJob, but if
-# for any reason it wasn't, we need to expand it now
+# This is called in two places: First (and ideally) is from the
mini scheduler at the end of
+# LocalTaskJob, and then as an "expansion of last resort" this is
also called from the scheduler
Review Comment:
```suggestion
# This is called in two places: First (and ideally) is in the
mini scheduler at the end of
# LocalTaskJob, and then as an "expansion of last resort" in the
scheduler
```
nit
##
airflow/models/dagrun.py:
##
@@ -838,14 +839,47 @@ def verify_integrity(self, session: Session =
NEW_SESSION):
ti.state = State.REMOVED
continue
-if task.is_mapped:
-task = cast("MappedOperator", task)
-num_mapped_tis = task.parse_time_mapped_ti_count
-# Check if the number of mapped literals has changed and we
need to mark this TI as removed
-if not num_mapped_tis or ti.map_index >= num_mapped_tis:
+if not task.is_mapped:
+continue
+task = cast("MappedOperator", task)
+num_mapped_tis = task.parse_time_mapped_ti_count
+# Check if the number of mapped literals has changed and we need
to mark this TI as removed
+if num_mapped_tis is not None:
+if ti.map_index >= num_mapped_tis:
+self.log.debug(
+"Removing task '%s' as the map_index is longer than
the literal mapping list (%s)",
+ti,
+num_mapped_tis,
+)
ti.state = State.REMOVED
elif ti.map_index < 0:
+self.log.debug("Removing the unmapped TI '%s' as the
mapping can now be performed", ti)
ti.state = State.REMOVED
+else:
+self.log.info("Restoring mapped task '%s'", ti)
+Stats.incr(f"task_restored_to_dag.{dag.dag_id}", 1, 1)
+ti.state = State.NONE
+else:
+# What if it is _now_ dynamically mapped, but wasn't before?
+total_length = task.run_time_mapped_ti_count(self.run_id,
session=session)
+
+if total_length is None:
+# Not all upstreams finished, so we can't tell what should
be here. Remove everying
Review Comment:
```suggestion
# Not all upstreams finished, so we can't tell what
should be here. Remove everything.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
uranusjr commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847689404 ## airflow/models/mappedoperator.py: ## @@ -559,7 +562,12 @@ def _get_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]: for task_id, length in xcom_query: for mapped_arg_name in mapped_dep_keys[task_id]: map_lengths[mapped_arg_name] += length +return map_lengths +def _resolve_map_lengths(self, run_id: str, *, session: Session) -> Dict[str, int]: +"""Return dict of argument name to map length, or throw if some are not resolvable""" +expansion_kwargs = self._get_expansion_kwargs() +map_lengths = self._get_map_lengths(run_id, session=session) Review Comment: We should document `_get_map_lengths` to explain how it’s different from `_resolve_map_lengths`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847687365 ## airflow/decorators/base.py: ## @@ -406,6 +406,9 @@ class DecoratedMappedOperator(MappedOperator): # in partial_kwargs, and MappedOperator prevents duplication. mapped_op_kwargs: Dict[str, "Mappable"] +def __hash__(self): Review Comment: This was needed for the `@cache` decorator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish opened a new pull request, #22921: Add revision version map to db.py
dstandish opened a new pull request, #22921: URL: https://github.com/apache/airflow/pull/22921 Sometimes we need to be able to get the airflow version for the current state of the airflow metadata database. In order to do this we need to be able to map alembic revision to airflow version, and that is what this mapping does. We maintain it with pre-commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (bc10265077 -> 9c9272de3a)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from bc10265077 Add SmoothOperator (#22813) add 9c9272de3a typo in BREEZE.rst (#22919) No new revisions were added by this update. Summary of changes: BREEZE.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] potiuk merged pull request #22919: typo in BREEZE.rst
potiuk merged PR #22919: URL: https://github.com/apache/airflow/pull/22919 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #22920: implements #22859 - Add .sql as templatable extension
boring-cyborg[bot] commented on PR #22920: URL: https://github.com/apache/airflow/pull/22920#issuecomment-1095495729 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] LennyKLB opened a new pull request, #22920: implements #22859 - Add .sql as templatable extension
LennyKLB opened a new pull request, #22920: URL: https://github.com/apache/airflow/pull/22920 This enables developers to template `.sql` files in Jinja when using `BigQueryInsertJobOperator`. This means that the parameter `configuration.query.query` can now refer to a proper `.sql` template file while keeping the rest of `configuration` in the top-level code of the operator. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22904: Call mapped_dependants only on the original task
github-actions[bot] commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095490525 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] LennyKLB commented on issue #22859: Add .sql template extension rendering for BigQueryInsertJobOperator
LennyKLB commented on issue #22859: URL: https://github.com/apache/airflow/issues/22859#issuecomment-1095481943 The change itself is so small that I refrained from adding/editing tests. I found that the BigQuery-related tests didn't feature any file-based templating. If this is nonetheless still necessary, could someone point me in the right direction in terms of inspiration? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #22916: Clarify breeze install message re pipx vs `./breeze`
dstandish closed pull request #22916: Clarify breeze install message re pipx vs `./breeze` URL: https://github.com/apache/airflow/pull/22916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #22916: Clarify breeze install message re pipx vs `./breeze`
dstandish commented on PR #22916: URL: https://github.com/apache/airflow/pull/22916#issuecomment-1095461058 Ok sounds good, was just making assumptions Will close -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
ashb commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095457156 Oh. https://github.com/apache/airflow/blob/main/scripts/docker/entrypoint_ci.sh#L353-L357 No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847653426 ## airflow/serialization/serialized_objects.py: ## @@ -1081,12 +1081,12 @@ def deserialize_dag(cls, encoded_dag: Dict[str, Any]) -> 'SerializedDAG': setattr(task.subdag, 'parent_dag', dag) if isinstance(task, MappedOperator): -for d in (task.mapped_kwargs, task.partial_kwargs): -for k, v in d.items(): -if not isinstance(v, _XComRef): -continue +expansion_kwargs = task._get_expansion_kwargs() Review Comment: Ah, yeah that makes sense. When the LocalTaskJob checks this it has the real DAG. It's just my change here now makes it be accessed via verify_integrity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22904: Call mapped_dependants only on the original task
uranusjr commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095455487 H… does the CI run long-running tests by default…? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
uranusjr commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847650700 ## airflow/serialization/serialized_objects.py: ## @@ -1081,12 +1081,12 @@ def deserialize_dag(cls, encoded_dag: Dict[str, Any]) -> 'SerializedDAG': setattr(task.subdag, 'parent_dag', dag) if isinstance(task, MappedOperator): -for d in (task.mapped_kwargs, task.partial_kwargs): -for k, v in d.items(): -if not isinstance(v, _XComRef): -continue +expansion_kwargs = task._get_expansion_kwargs() Review Comment: Because those attributes _shouldn’t_ be accessed by the scheduler anyway? So this likely only caused the task to be rendered incorrectly in UI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
ashb commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095450527 ``` FAILED tests/jobs/test_backfill_job.py::TestBackfillJob::test_mapped_dag[test_mapped_classic-DebugExecutor] - RuntimeError: Cannot check for mapped_dependants when not attached to a DAG == 1 failed, 1 passed, 46 deselected, 18 warnings in 19.09s == ~/code/airflow/airflow fix-mapping-removed-tasks-bug* 20s airflow ❯ git di [20:07] [1] !10947 diff --git a/tests/dags/test_mapped_classic.py b/tests/dags/test_mapped_classic.py index 3880cc74f..9759b4c39 100644 --- a/tests/dags/test_mapped_classic.py +++ b/tests/dags/test_mapped_classic.py @@ -32,3 +32,4 @@ def consumer(value): with DAG(dag_id='test_mapped_classic', start_date=days_ago(2)) as dag: PythonOperator.partial(task_id='consumer', python_callable=consumer).expand(op_args=make_arg_lists()) +PythonOperator.partial(task_id='consumer_literal', python_callable=consumer).expand(op_args=[[1],[2],[3]]) diff --git a/tests/dags/test_mapped_taskflow.py b/tests/dags/test_mapped_taskflow.py index 34f6ae3d7..31c132f7c 100644 --- a/tests/dags/test_mapped_taskflow.py +++ b/tests/dags/test_mapped_taskflow.py @@ -29,3 +29,4 @@ with DAG(dag_id='test_mapped_taskflow', start_date=days_ago(2)) as dag: print(repr(value)) consumer.expand(value=make_list()) +consumer.expand(value=[1,2,3]) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] DONDJJ commented on issue #13637: Scheduler takes 100% of CPU without task execution
DONDJJ commented on issue #13637: URL: https://github.com/apache/airflow/issues/13637#issuecomment-1095450372 @[veinkr](https://github.com/veinkr) @[kaxil](https://github.com/kaxil) https://github.com/apache/airflow/blob/2400de2c5ece644cadb870baeea28907fa4dcf58/airflow/dag_processing/processor.py#L641 In my case the problem appears when there are syntax or import errors in dags. I fixed it by just commenting out this string. I don't know why, but every time I use log with "warning" and higher levels in DagFileProcessor class this issue occurs I use airflow 2.2.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22904: Call mapped_dependants only on the original task
ashb commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095449301 Really? Those are failing for me on main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #22904: Call mapped_dependants only on the original task
uranusjr commented on PR #22904: URL: https://github.com/apache/airflow/pull/22904#issuecomment-1095447937 Yeah I just added those two to the test DAG (_without the proposed fix) and CI still passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a diff in pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on code in PR #22909: URL: https://github.com/apache/airflow/pull/22909#discussion_r847645333 ## airflow/serialization/serialized_objects.py: ## @@ -1081,12 +1081,12 @@ def deserialize_dag(cls, encoded_dag: Dict[str, Any]) -> 'SerializedDAG': setattr(task.subdag, 'parent_dag', dag) if isinstance(task, MappedOperator): -for d in (task.mapped_kwargs, task.partial_kwargs): -for k, v in d.items(): -if not isinstance(v, _XComRef): -continue +expansion_kwargs = task._get_expansion_kwargs() Review Comment: This was a bug that I don't know how we didn't hit before? The serialized task as a result had these objects left as _XcomRef, which as it is a named tuple has a `__len__` of 2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on PR #22909: URL: https://github.com/apache/airflow/pull/22909#issuecomment-1095440612 This turned out a bit more complex :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #22909: Fix bug where dynamically mapped tasks got set to REMOVED
ashb commented on PR #22909: URL: https://github.com/apache/airflow/pull/22909#issuecomment-1095441003 And I'm not too happy with how I had to structure the conditionals. Suggestions appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Add SmoothOperator (#22813)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bc10265077 Add SmoothOperator (#22813)
bc10265077 is described below
commit bc10265077a6c1ace46951fccb7c2c85c56cbd15
Author: Tomek Urbaszek
AuthorDate: Mon Apr 11 20:55:58 2022 +0200
Add SmoothOperator (#22813)
Easter is coming so I just came with idea of an easter egg.
---
airflow/operators/smooth.py | 35 +++
docs/spelling_wordlist.txt | 1 +
2 files changed, 36 insertions(+)
diff --git a/airflow/operators/smooth.py b/airflow/operators/smooth.py
new file mode 100644
index 00..9dcbccb9e0
--- /dev/null
+++ b/airflow/operators/smooth.py
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.models.baseoperator import BaseOperator
+from airflow.utils.context import Context
+
+
+class SmoothOperator(BaseOperator):
+"""
+Operator that does literally nothing but it logs YouTube link to
+Sade song "Smooth Operator".
+"""
+
+ui_color = '#e8f7e4'
+yt_link: str = "https://www.youtube.com/watch?v=4TYv2PhG89A;
+
+def __init__(self, **kwargs) -> None:
+super().__init__(**kwargs)
+
+def execute(self, context: Context):
+self.log.info("Enjoy Sade - Smooth Operator: %s", self.yt_link)
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index 744ea2027c..62a32d2624 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -1265,6 +1265,7 @@ runAsUser
runnable
runspace
runtime
+sade
sagemaker
salesforce
samesite
[GitHub] [airflow] potiuk merged pull request #22813: Add SmoothOperator
potiuk merged PR #22813: URL: https://github.com/apache/airflow/pull/22813 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22813: Add SmoothOperator
potiuk commented on PR #22813: URL: https://github.com/apache/airflow/pull/22813#issuecomment-1095436388 Just in time for 2.3.0. We could not wait for SmoothOperator. It's been highly requested feature by many. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22813: Add SmoothOperator
potiuk commented on PR #22813: URL: https://github.com/apache/airflow/pull/22813#issuecomment-1095435812 Not SADE any more and just random failure. Merging :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #22916: Clarify breeze install message re pipx vs `./breeze`
potiuk commented on PR #22916: URL: https://github.com/apache/airflow/pull/22916#issuecomment-1095431205 I completely missed updatinng those instructions after making this decision :blush: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
