[GitHub] [airflow] internetcoffeephone opened a new issue, #28296: Dynamic Task Mapping task does not start because of the error: "Task has been mapped: The task has yet to be mapped!"
internetcoffeephone opened a new issue, #28296:
URL: https://github.com/apache/airflow/issues/28296
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Using Airflow 2.4.2.
I've got a task that retrieves some filenames, which then creates
dynamically mapped tasks to move the files, one per task.
I'm using a similar task across multiple DAGs. However, task mapping fails
on some DAG runs: it inconsistently happens per DAG run, and some DAGs do not
seem to be affected at all. These seem to be the DAGs where no task was ever
mapped, so that the mapped task instance ended up in a Skipped state.
What happens is that multiple files will be found, but only a single
dynamically mapped task will be created. This task never starts and has
map_index of -1. It can be found under the "List instances, all runs" menu, but
says "No Data found." under the "Mapped Tasks" tab.
When I press the "Run" button when the mapped task is selected, the
following error appears:
```Could not queue task instance for execution, dependencies not met:
Previous Dagrun State: depends_on_past is true for this task's DAG, but the
previous task instance has not run yet., Task has been mapped: The task has yet
to be mapped!```
The previous task *has* run however. No errors appeared in my Airflow logs.
### What you think should happen instead
The appropriate amount of task instances should be created, they should
correctly resolve the ```depends_on_past``` check and then proceed to run
correctly.
### How to reproduce
This DAG reliably reproduces the error for me. The first set of mapped tasks
succeeds, the subsequent ones do not.
```
from airflow import DAG
from airflow.decorators import task
import datetime as dt
from airflow.operators.python import PythonOperator
@task
def get_filenames_kwargs():
return [
{
"file_name": i,
}
for i in range(10)
]
def print_filename(file_name):
print(file_name)
with DAG(
dag_id="dtm_test",
start_date=dt.datetime(2022, 12, 10),
default_args={
"owner": "airflow",
"depends_on_past": True,
},
schedule="@daily",
) as dag:
get_filenames_task =
get_filenames_kwargs.override(task_id="get_filenames_task")()
print_filename_task = PythonOperator.partial(
task_id="print_filename_task",
python_callable=print_filename,
).expand(op_kwargs=get_filenames_task)
# Perhaps redundant
get_filenames_task >> print_filename_task
```
### Operating System
Amazon Linux 2
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] ferruzzi closed pull request #28236: Better support for Boto Waiters
ferruzzi closed pull request #28236: Better support for Boto Waiters URL: https://github.com/apache/airflow/pull/28236 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #28256: Include full path to Python files under zip path while clearing import errors.
ephraimbuddy commented on code in PR #28256: URL: https://github.com/apache/airflow/pull/28256#discussion_r1045474431 ## airflow/dag_processing/manager.py: ## @@ -777,8 +777,9 @@ def clear_nonexistent_import_errors(self, session): :param session: session for ORM operations """ query = session.query(errors.ImportError) -if self._file_paths: -query = query.filter(~errors.ImportError.filename.in_(self._file_paths)) +files = list_py_file_paths(self._dag_directory, include_examples=False, include_zip_paths=True) Review Comment: Sadly, `list_py_file_paths` is an expensive operation... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28295: Fundamentals.rst mentions `dag_id` in Instantiate Dag Section but not shown
boring-cyborg[bot] commented on issue #28295: URL: https://github.com/apache/airflow/issues/28295#issuecomment-1346000923 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] datatalking opened a new issue, #28295: Fundamentals.rst mentions `dag_id` in Instantiate Dag Section but not shown
datatalking opened a new issue, #28295: URL: https://github.com/apache/airflow/issues/28295 ### What do you see as an issue? This is my first errata for the docs and under the 'Instantiate a DAG' section it mentions [(https://github.com/apache/airflow/tree/main/docs/apache-airflow/tutorial)/fundamentals.rst](https://github.com/apache/airflow/blob/main/docs/apache-airflow/tutorial/fundamentals.rst#instantiate-a-dag) ''' We'll need a DAG object to nest our tasks into. Here we pass a string that defines the `dag_id`, which serves as a unique identifier for your DAG. We also pass the default argument dictionary that we just defined and define a schedule of 1 day for the DAG ``` It mentions the `dag_id` but never shows it or mentions it again until the bottom section under Tests. Should it be declared or used somehow? I'm doing a fresh install on MacOs and there are so many little flags and issues I'm looking for a more thorough install. I bought the `Data Engineering with Python` book from PacktPub from Paul Crickard and so much has changed the book is becoming less and less helpful. ### Solving the problem Should the `dag_id` object or whatever it is be declared, referenced, shown what it is or how can we be verbose on what it is? A search on that page 'https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html' shows only two mentions of `dag_id` SO I figured this was an oversight and I'm willing to help fill in or be verbose in explaining it. ### Anything else Once I know the purpose or have context on what `dag_id` is or can be explained as I'm willing to submit a PR. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #28076: Update main with 2.5.0 config values
ephraimbuddy commented on code in PR #28076: URL: https://github.com/apache/airflow/pull/28076#discussion_r1045434699 ## airflow/config_templates/config.yml: ## @@ -2377,7 +2377,7 @@ failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed. - version_added: 1.10.11 + version_added: 2.5.0 Review Comment: I have been off for a while... @potiuk what do you think about this? I think the correctness depends on how one looks at it. The section/option is new in 2.5.0 while the old section/option is not new. If we mark it 1.10.11, users would think it'll work in 1.10.11 but that's not true. It will only work in 1.10.11 with the old section name, not the new one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #28076: Update main with 2.5.0 config values
ephraimbuddy commented on code in PR #28076: URL: https://github.com/apache/airflow/pull/28076#discussion_r1040450956 ## airflow/config_templates/config.yml: ## @@ -2377,7 +2377,7 @@ failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed. - version_added: 1.10.11 + version_added: 2.5.0 Review Comment: `AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE` was 1.10.11 while `AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE` started existing in 2.5.0. Makes me wonder if we should have a new field on the config file, `version_renamed` or something because, in some way, this was really added in 2.5.0 but practically works down to 1.10.11 with old section where it was originally added in the old section. cc @ashb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #27829: Improving the release process
ephraimbuddy commented on PR #27829: URL: https://github.com/apache/airflow/pull/27829#issuecomment-1345934132 > cc: @ephraimbuddy Thanks. Have updated it. It seems like `inputimeout` is not installed in the image because it errors out during pre-commit run: ``` File "/Users/ephraimbuddy/Documents/astronomer/airflow/dev/breeze/src/airflow_breeze/commands/main_command.py", line 159, in check_for_rosetta_environment from inputimeout import TimeoutOccurred, inputimeout ModuleNotFoundError: No module named 'inputimeout' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on pull request #28236: Better support for Boto Waiters
ferruzzi commented on PR #28236: URL: https://github.com/apache/airflow/pull/28236#issuecomment-1345793675 Looks like it is all passing but wants a rebase again. I'll give it a nudge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow-checks-action] branch dependabot/npm_and_yarn/qs-6.5.3 created (now 1b3ac69)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/qs-6.5.3 in repository https://gitbox.apache.org/repos/asf/airflow-checks-action.git at 1b3ac69 Bump qs from 6.5.2 to 6.5.3 No new revisions were added by this update.
[GitHub] [airflow-site] chuckcho opened a new pull request, #710: Update _index.md
chuckcho opened a new pull request, #710: URL: https://github.com/apache/airflow-site/pull/710 Duplicate release version (December 2, 2022). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #27317: Dag processor manager queue split (fixes SLAs)
github-actions[bot] commented on PR #27317: URL: https://github.com/apache/airflow/pull/27317#issuecomment-1345701877 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #27305: Add readOnly flag to gitSync section
github-actions[bot] commented on PR #27305: URL: https://github.com/apache/airflow/pull/27305#issuecomment-1345701891 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] tiokim commented on a diff in pull request #28251: task docker documentation added
tiokim commented on code in PR #28251: URL: https://github.com/apache/airflow/pull/28251#discussion_r1045320242 ## docs/apache-airflow-providers-docker/decorators/docker.rst: ## @@ -0,0 +1,130 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + + + +.. _howto/decorator:docker: + +Task Docker Decorator += + +Python callable wrapped within the ``@task.docker`` decorator with args are executed within +the docker container. + +Parameters +-- + +The following parameters are supported in Docker Task decorator. + +multiple_outputs +If set, function return value will be unrolled to multiple XCom values. +Dict will unroll to XCom values with keys as XCom keys. Defaults to False. +use_dill +Whether to use dill or pickle for serialization +python_command +Python command for executing functions, Default python3 +image +Docker image from which to create the container. +If image tag is omitted, "latest" will be used. +api_version +Remote API version. Set to ``auto`` to automatically detect the server's version. +container_name +Name of the container. Optional (templated) +cpus: +Number of CPUs to assign to the container. This value gets multiplied with 1024. +docker_url +URL of the host running the docker daemon. +Default is unix://var/run/docker.sock +environment +Environment variables to set in the container. (templated) +private_environment +Private environment variables to set in the container. +These are not templated, and hidden from the website. +force_pull +Pull the docker image on every run. Default is False. +mem_limit +Maximum amount of memory the container can use. +Either a float value, which represents the limit in bytes, +or a string like ``128m`` or ``1g``. +host_tmp_dir +Specify the location of the temporary directory on the host which will +be mapped to tmp_dir. If not provided defaults to using the standard system temp directory. +network_mode +Network mode for the container. + +It can be one of the following: +bridge +Create new network stack for the container with default docker bridge network +'None' +No networking for this container +container: or +Use the network stack of another container specified via or +host +Use the host network stack. Incompatible with `port_bindings` +'' or '' +Connects the container to user created network(using `docker network create` command) +tls_ca_cert +Path to a PEM-encoded certificate authority to secure the docker connection. +tls_client_cert +Path to the PEM-encoded certificate used to authenticate docker client. +tls_client_key +Path to the PEM-encoded key used to authenticate docker client. +tls_hostname +Hostname to match against the docker server certificate or False to disable the check. +tls_ssl_version +Version of SSL to use when communicating with docker daemon. +tmp_dir +Mount point inside the container to +a temporary directory created on the host by the operator. +The path is also made available via the environment variable +``AIRFLOW_TMP_DIR`` inside the container. +user +Default user inside the docker container. +mounts +List of mounts to mount into the container, e.g. +``['/host/path:/container/path', '/host/path2:/container/path2:ro']``. +working_dir +Working directory to set on the container (equivalent to the -w switch the docker client) +xcom_all +Push all the stdout or just the last line. The default is False (last line). +docker_conn_id +ID of the Airflow connection to use +dns +Docker custom DNS servers +dns_search +Docker custom DNS search domain +auto_remove +Auto-removal of the container on daemon side when the container's process exits. +The default is False. Review Comment: IMHO, [The default](https://github.com/apache/airflow/blob/b52ce7f483d02fe17467ca3fa13cfffdb92ac103/airflow/providers/docker/operators/docker.py#L134) seems `never`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on
[GitHub] [airflow] kolfild26 commented on pull request #27319: XCOM push ORA error code in OracleStoredProcedure
kolfild26 commented on PR #27319: URL: https://github.com/apache/airflow/pull/27319#issuecomment-1345687334 Hi > I'm a bit stuck with the unit test and will update once it's resolved. Finally it's done. Please review. Files modified: `airflow/providers/oracle/operators/oracle.py` `tests/providers/oracle/operators/test_oracle.py` In `test_oracle.py` a new method `test_push_oracle_exit_to_xcom` added. In `oracle.py` added some check that `context` is available. If so, try to pull from xcom. It's needed as other tests don't use `context` when running `task.execute()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28248: Hopefully stabilize quarantined tests for celery hanging
potiuk commented on PR #28248: URL: https://github.com/apache/airflow/pull/28248#issuecomment-1345668746 This one looks good. I saw some other, rare flaky failure after a number of re-reruns with "full tests needed". I think we can merge this one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] miky-schreiber commented on a diff in pull request #27953: Add timezone to dags in main screen
miky-schreiber commented on code in PR #27953: URL: https://github.com/apache/airflow/pull/27953#discussion_r1045292916 ## airflow/models/dag.py: ## @@ -3204,6 +3204,13 @@ def __init__(self, concurrency=None, **kwargs): def __repr__(self): return f"" +@property +def schedule_description(self) -> str | None: Review Comment: @pierrejeambrun the DagModel class doesn't have self.timetable attribute. I used self.timetable_description instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] isaac-florence opened a new issue, #28293: Skipping dag runs on established daily DAG
isaac-florence opened a new issue, #28293:
URL: https://github.com/apache/airflow/issues/28293
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Airflow 2.4.3, running on Kubernetes with High-Availability Scheduler (2
pods). Database is postgres 10.21
I have a dag that has been running for two months with this timetable:
`timetable=CronTriggerTimetable("40 5 * * *", timezone = 'UTC')` (which has not
changed). It has run reliably every day as expected however yesterday it simply
did not run. I have checked for errors and there were none, and checking the
scheduler logs (on both pods) it appears no evaluation of the dag run was done
in either scheduler instance. in the cli inspecting the next 60 run execution
dates indicates there is no problem with the timetable's interpretation
My suspiscion is that a race condition in the `SELECT ... FOR UPDATE` of the
database may have prevented either scheduler executing the DAG. Is this
possible? Otherwise how can this have happened?
### What you think should happen instead
The dag should have run as it always has previously
### How to reproduce
I wish I knew! I can't work it out.
### Operating System
RedHat Enterprise Linux 8
### Versions of Apache Airflow Providers
2.4.3
### Deployment
Other Docker-based deployment
### Deployment details
High availability schedulers (n=2) on OpenShift Kubernetes. Postgres
database (version 10.21
### Anything else
This has happened once before, just under a month ago, on the same DAG.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28293: Skipping dag runs on established daily DAG
boring-cyborg[bot] commented on issue #28293: URL: https://github.com/apache/airflow/issues/28293#issuecomment-1345640631 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on pull request #27405: Add deferrable mode to MLEngineStartTrainingJobOperator
VladaZakharova commented on PR #27405: URL: https://github.com/apache/airflow/pull/27405#issuecomment-1345628311 @potiuk Thank you for rebasing :) Still seems like some errors that are not related to my changes but with Kubernetes executor: https://github.com/apache/airflow/actions/runs/3668592118/jobs/6201917454#step:6:7783 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #27905: listener plugin example added
Bowrna commented on PR #27905: URL: https://github.com/apache/airflow/pull/27905#issuecomment-1345623014 Failing in Postgres-related static check (test related to Kubernetes). I didn't make changes that will fail the test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28292: Move providers
potiuk opened a new pull request, #28292:
URL: https://github.com/apache/airflow/pull/28292
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28289: Add AWS Neptune hook and operators
Taragolis commented on issue #28289: URL: https://github.com/apache/airflow/issues/28289#issuecomment-1345610534 @swapz-z assigned to you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new pull request, #28291: Add script to move providers to a new directory structure
potiuk opened a new pull request, #28291:
URL: https://github.com/apache/airflow/pull/28291
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] swapz-z commented on issue #28289: Add AWS Neptune hook and operators
swapz-z commented on issue #28289: URL: https://github.com/apache/airflow/issues/28289#issuecomment-1345595799 Hello @eladkal I would love to take this up -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (f3f38a1857 -> b52ce7f483)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from f3f38a1857 Remove duplicated GCP Compute IGM system test (#28061) add b52ce7f483 Completed D400 for `airflow/kubernetes/*` (#28212) No new revisions were added by this update. Summary of changes: airflow/kubernetes/k8s_model.py | 9 +++-- airflow/kubernetes/kube_client.py | 11 +++--- airflow/kubernetes/kube_config.py | 2 +- airflow/kubernetes/kubernetes_helper_functions.py | 4 +-- airflow/kubernetes/pod_generator.py | 44 +-- airflow/kubernetes/pod_generator_deprecated.py| 16 ++--- airflow/kubernetes/pod_launcher_deprecated.py | 39 ++-- airflow/kubernetes/secret.py | 17 - 8 files changed, 91 insertions(+), 51 deletions(-)
[GitHub] [airflow] potiuk merged pull request #28212: Completed D400 for `airflow/kubernetes/*`
potiuk merged PR #28212: URL: https://github.com/apache/airflow/pull/28212 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ogremagi4 commented on issue #28178: AttributeError: module 'attr.validators' has no attribute 'min_len'
ogremagi4 commented on issue #28178:
URL: https://github.com/apache/airflow/issues/28178#issuecomment-1345540603
I have the same issue with attrs version 22.1.0 and airflow 2.5
Dagbag check of a task finishes with different results: sometimes it
succeeds, sometimes fails.
Not sure how to debug it
```
[2022-12-11T12:18:47.459+] {logging_mixin.py:137} INFO -
[2022-12-11T12:18:47.459+] {dagbag.py:538} INFO - Filling up the DagBag
from /home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py
[2022-12-11T12:18:48.241+] {processor.py:753} INFO - DAG(s)
dict_keys(['fill_locks']) retrieved from
/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py
[2022-12-11T12:18:48.305+] {logging_mixin.py:137} INFO -
[2022-12-11T12:18:48.304+] {dag.py:2676} INFO - Sync 1 DAGs
[2022-12-11T12:18:48.372+] {logging_mixin.py:137} INFO -
[2022-12-11T12:18:48.372+] {dag.py:3423} INFO - Setting next_dagrun for
fill_locks to 2022-12-11T12:00:00+00:00, run_after=2022-12-11T12:30:00+00:00
[2022-12-11T12:18:48.407+] {processor.py:175} INFO - Processing
/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py took 0.954 seconds
[2022-12-11T12:18:50.383+] {processor.py:153} INFO - Started process
(PID=40112) to work on
/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py
[2022-12-11T12:18:50.384+] {processor.py:640} INFO - Processing file
/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py for tasks to queue
[2022-12-11T12:18:50.384+] {logging_mixin.py:115} INFO -
[2022-12-11T12:18:50.384+] {dagbag.py:508} INFO - Filling up the DagBag
from /home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py
[2022-12-11T12:18:51.098+] {logging_mixin.py:115} INFO -
[2022-12-11T12:18:51.097+] {dagbag.py:321} ERROR - Failed to import:
/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py
Traceback (most recent call last):
File
"/home/airflow/airflow/venv/lib/python3.9/site-packages/airflow/models/dagbag.py",
line 318, in parse
if not self.has_logged:
File "", line 790, in exec_module
File "", line 228, in
_call_with_frames_removed
File "/home/airflow/airflow/dags/etls/new_warehouse/fill_locks.py", line
8, in
from airflow.decorators import task
File
"/home/airflow/airflow/venv/lib/python3.9/site-packages/airflow/decorators/__init__.py",
line 21, in
from airflow.decorators.base import TaskDecorator
File
"/home/airflow/airflow/venv/lib/python3.9/site-packages/airflow/decorators/base.py",
line 41, in
from airflow.datasets import Dataset
File
"/home/airflow/airflow/venv/lib/python3.9/site-packages/airflow/datasets/__init__.py",
line 26, in
class Dataset:
File
"/home/airflow/airflow/venv/lib/python3.9/site-packages/airflow/datasets/__init__.py",
line 29, in Dataset
uri: str = attr.field(validator=[attr.validators.min_len(1),
attr.validators.max_len(3000)])
AttributeError: module 'attr.validators' has no attribute 'min_len'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] swapz-z commented on pull request #28282: Mark stepIds for cancel in EmrAddStepsOperator
swapz-z commented on PR #28282: URL: https://github.com/apache/airflow/pull/28282#issuecomment-1345529940 Thanks [Taragolis](https://github.com/Taragolis) I will incorporate these as suggested > Also this code need to be covered by tests, we want to reduce chance of any regression. Although tests for hooks are added, I was hoping to add more after getting some initial review/thoughts on the functional code, since existing test cases also will have to be modified accordingly, if needed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ayushthe1 commented on pull request #27522: Add example of dockerfile with creating new virtualenv
ayushthe1 commented on PR #27522: URL: https://github.com/apache/airflow/pull/27522#issuecomment-1345525616 > gentle ping :) sorry for the delay ,i will now start working back on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ayushthe1 commented on pull request #27522: Add example of dockerfile with creating new virtualenv
ayushthe1 commented on PR #27522: URL: https://github.com/apache/airflow/pull/27522#issuecomment-1345525488 > gentle ping :) sorry for the delay ,i will now start working back on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pierrejeambrun commented on a diff in pull request #28288: Fix is_local for LocalKubernetesExecutor
pierrejeambrun commented on code in PR #28288:
URL: https://github.com/apache/airflow/pull/28288#discussion_r1045210769
##
airflow/executors/local_kubernetes_executor.py:
##
@@ -43,6 +43,8 @@ class LocalKubernetesExecutor(LoggingMixin):
KUBERNETES_QUEUE = conf.get("local_kubernetes_executor",
"kubernetes_queue")
+is_local: bool = False
Review Comment:
Yep I thought the same. This is the conservative choice, and equivalent to
what was implemented before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated (0db5e46ec1 -> f3f38a1857)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 0db5e46ec1 Proper Python Host output from composite tasks in CI (#28254) add f3f38a1857 Remove duplicated GCP Compute IGM system test (#28061) No new revisions were added by this update. Summary of changes: .../providers/google/cloud/compute_igm/__init__.py | 16 --- .../cloud/compute_igm/example_compute_igm.py | 158 - 2 files changed, 174 deletions(-) delete mode 100644 tests/system/providers/google/cloud/compute_igm/__init__.py delete mode 100644 tests/system/providers/google/cloud/compute_igm/example_compute_igm.py
[GitHub] [airflow] potiuk merged pull request #28061: Remove duplicated GCP Compute IGM system test
potiuk merged PR #28061: URL: https://github.com/apache/airflow/pull/28061 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27405: Add deferrable mode to MLEngineStartTrainingJobOperator
potiuk commented on PR #27405: URL: https://github.com/apache/airflow/pull/27405#issuecomment-1345518123 Rebased after fixing the "public" tests last week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Taragolis opened a new pull request, #28290: Fix Google Cloud async tests
Taragolis opened a new pull request, #28290: URL: https://github.com/apache/airflow/pull/28290 Some async tests not properly mocked return objects - `tests/providers/google/cloud/hooks/test_cloud_composer.py` - `tests/providers/google/cloud/hooks/test_dataproc.py` Within `unittests.TestCase` it do actually nothing: ```console RuntimeWarning: coroutine 'FooBar.spam_egg' was never awaited method() ``` When this tests migrate to `pytest` it raises different errors. This PR: - Move compat (with python 3.7) mockers to separate module - Migrate async tests to `pytest` and fix them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #28248: Hopefully stabilize quarantined tests for celery hanging
potiuk commented on PR #28248: URL: https://github.com/apache/airflow/pull/28248#issuecomment-1345512777 Reopened to rebuild with public runners. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #28248: Hopefully stabilize quarantined tests for celery hanging
potiuk closed pull request #28248: Hopefully stabilize quarantined tests for celery hanging URL: https://github.com/apache/airflow/pull/28248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #28251: task docker documentation added
Bowrna commented on PR #28251: URL: https://github.com/apache/airflow/pull/28251#issuecomment-1345492223 I have fixed the static check issues. @potiuk -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Bowrna closed pull request #28251: task docker documentation added
Bowrna closed pull request #28251: task docker documentation added URL: https://github.com/apache/airflow/pull/28251 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
