[GitHub] [airflow] kristoffern edited a comment on issue #19699: task_instances stuck in "queued" and are missing corresponding celery_taskmeta entries
kristoffern edited a comment on issue #19699: URL: https://github.com/apache/airflow/issues/19699#issuecomment-985293147 @ephraimbuddy I had a task stuck this morning, confirmed that it was stuck as described in this bug by checking the DB. It has been stuck for a few hours when I woke up. However docker had pruned the logs so that the earliest entry was still ~40min after the task had gotten stuck. Will continue to monitor for it if it's important? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kristoffern commented on issue #19699: task_instances stuck in "queued" and are missing corresponding celery_taskmeta entries
kristoffern commented on issue #19699: URL: https://github.com/apache/airflow/issues/19699#issuecomment-985293147 I had a task stuck this morning, confirmed that it was stuck as described in this bug by checking the DB. It has been stuck for a few hours when I woke up. However docker had pruned the logs so that the earliest entry was still ~40min after the task had gotten stuck. Will continue to monitor for it if it's important? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] xelita commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
xelita commented on issue #19962: URL: https://github.com/apache/airflow/issues/19962#issuecomment-985284705 @potiuk agree with you about not using the docker-compose for production deployment. However I just want to test a custom logger implementation locally with docker.compose. In this case, even if I mount the config folder with the appropriate logging file, the stack fails saying it could not find the logger module. Also what is your recommendation regarding Airflow local development (testing custom plugins, ...). I thought docker-compose was suitable for this kind of usecase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hwooson12 commented on pull request #20002: Add hour and minute to time format on x-axis of all charts using nvd3.lineChart
hwooson12 commented on pull request #20002: URL: https://github.com/apache/airflow/pull/20002#issuecomment-985281955 The original format in version 1.7 was like below.  BTW, I am digging into how to break the label into multiple lines. It's wrapped from d3 API so I guess it is going to be more to fix. > I think hour:minute should be placed before (i.e. above) the date, if we break the label into multiple lines I agree with it. If you find how to do it or have some thoughts on it, please leave a comment below! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] alexott commented on issue #19895: Status of testing Providers that were prepared on November 30, 2021
alexott commented on issue #19895: URL: https://github.com/apache/airflow/issues/19895#issuecomment-985279645 I have capacity to test only over the weekend... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #20000: Fix many of the mypy typing issues in airflow.models.dag
uranusjr commented on pull request #2: URL: https://github.com/apache/airflow/pull/2#issuecomment-985279083 It’s probably because Pendulum invents its own timezone class (which is fine) and expects to only receive that timezone class in its own datetime class (which is not fine). I’ll take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19895: Status of testing Providers that were prepared on November 30, 2021
uranusjr commented on issue #19895: URL: https://github.com/apache/airflow/issues/19895#issuecomment-985277834 cc @dimon222 for #19815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19895: Status of testing Providers that were prepared on November 30, 2021
potiuk commented on issue #19895: URL: https://github.com/apache/airflow/issues/19895#issuecomment-985277300 Thanks for those who tested. Today is last day to do some testing :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #20002: Add hour and minute to time format on x-axis of all charts using nvd3.lineChart
uranusjr commented on pull request #20002: URL: https://github.com/apache/airflow/pull/20002#issuecomment-985277474 I think hour:minute should be placed _before_ (i.e. above) the date, if we break the label into multiple lines -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #20005: Addition of FilePatternSFTPSensor Feature
uranusjr commented on pull request #20005: URL: https://github.com/apache/airflow/pull/20005#issuecomment-985274713 Instead of introducing an entirely new sensor, I would combine this to the “regular” sensor. See #17937 for a similar design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #19825: Deprecate passing execution_date to XCom methods
uranusjr commented on pull request #19825: URL: https://github.com/apache/airflow/pull/19825#issuecomment-985268177 I’ve made all tests work and migrated most existing XCom usages to use `run_id`. One (very big) exception is operator link; since `get_link` is publicly subclassable, we can’t just change the argument. The current plan (after discussing with @ashb) is to add an optional flag on the class to toggle the behaviour, but there are much detail (and likely bikeshedding) around that, so I’m going to do it in a separate PR. The depracation warning implemented in this PR on passing `execution_date` to XCom is changed from DeprecationWarning to PendingDeprecationWarning to reflect that—we won’t commit the deprecation until we provide a migration path for operator links (which should still happen in 2.3, but let’s not jump the gun). I also want to extract typing changes I made here into a separate PR, and merge that first before I mark this PR as ready for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] AnudeepKonaboina opened a new pull request #20005: Addition of FilePatternSFTPSensor Feature
AnudeepKonaboina opened a new pull request #20005: URL: https://github.com/apache/airflow/pull/20005 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #20005: Addition of FilePatternSFTPSensor Feature
boring-cyborg[bot] commented on pull request #20005: URL: https://github.com/apache/airflow/pull/20005#issuecomment-985252036 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Acehaidrey commented on pull request #19921: Add dag view to display dagrun history per dag level
Acehaidrey commented on pull request #19921: URL: https://github.com/apache/airflow/pull/19921#issuecomment-985246854 Hi @bbovenzi thanks for the reply - I will do a separate pr for the td_format but do you know which areas is should be applied to? I just want to get a head start from you to verify :) And the audit logs view, I can add something if you feel it would be helpful. But if you all think it's not that is okay with me too - I know all organizations run in their own way. I just did this PR since the poll we sent out had it as a higher priority. If you have screenshots or video clips of how the new tree view will handle this - would love to see that too. I couldn't get a way to see it with the latest version. I agree that if it can be in that view that is even more concise and clear. Want to see if we can leverage that too. Lastly regarding the dagrun history duration, yes think the duration view could use some help. It definitely can be improved but for us, our customers like the table views more vs the graph visualization so we haven't done a whole lot of work there. Just a way to have improved filtering. I can add that, and see if a bar graph would be able to show each day, but for us, seeing the change over day helps see the outliers too. So both would work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] chrisrfu opened a new issue #20004: Airflow UI "stuck" and not showing most recent change
chrisrfu opened a new issue #20004:
URL: https://github.com/apache/airflow/issues/20004
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Mac OS X 10.15.7
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
running it locally with docker and docker-compose version 3
### What happened
1. run _build_me.sh_
- _build_me.sh_ consists of:
```
#!/bin/bash
TAG=$1
if [ -z ${TAG} ]; then
read -p "tag: (latest) " response
TAG=${response:-latest}
fi
docker build --progress=plain --no-cache -t {tag_name}:${TAG} -f Dockerfile .
```
Dockerfile looks like this
```
FROM apache/airflow:2.2.2
USER airflow
WORKDIR /opt/airflow
COPY --chown=airflow dags/ /opt/airflow/dags/
COPY airflow.cfg ./airflow.cfg
COPY requirements.txt ./requirements.txt
COPY devops ./devops
EXPOSE 8082 8080
RUN pip install -r requirements.txt
```
2. run _run_me.sh_
- _run_me.sh_ looks like this:
```
#!/bin/bash
docker-compose --env-file ./config/.env.local up airflow-init
docker-compose --env-file ./config/.env.local up
```
3. go into the airflow UI, log in, and see that what is displayed is not
reflective of the code.
Example: I deliberately create syntax error in my code but it doesn't show
up as a dag import error on the UI. Or more recently there's no tasks on the
UI even though I have dag files.
4. check the webserver container using `docker exec -it {container_id}
/bin/bash` to see if syntax error is there and in fact is there.
### What you expected to happen
I expected the UI to be reflective of what is in the code. After
implementing a change to my code, restarting the docker containers, re-build
the docker image, run docker-compose, I should be able to see new changes (not
something that is "stuck")
It seems there's some sort of error in the scheduler:
ModuleNotFoundError: No module named 'airflow_local_settings'
This is the airflow logs:
[airflow_all_log_files.txt](https://github.com/apache/airflow/files/7646817/airflow_all_log_files.txt)
And this is the airflow scheduler logs:
[airflow_scheduler_log_files.txt](https://github.com/apache/airflow/files/7646818/airflow_scheduler_log_files.txt)
### How to reproduce
You can use the latest docker 4.3.0 (71786) or docker 3.3.3 (64133). I tried
both and the result is the same.
### Anything else
The problem occurs every time I build and run dockerfile and docker-compose.
I've tried using Airflow 2.2.1, 2.2.0, 2.1.4 and so far same result.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] dstandish opened a new issue #20003: Trigger event payload mutually incompatible with deferral kwargs
dstandish opened a new issue #20003:
URL: https://github.com/apache/airflow/issues/20003
### Apache Airflow version
2.2.2 (latest released)
### Operating System
official docker image
### Versions of Apache Airflow Providers
_No response_
### Deployment
Virtualenv installation
### Deployment details
When we defer we can supply kwargs
```
self.defer(
trigger=DateTimeTrigger(moment=target_dttm),
method_name="execute_complete",
kwargs={"other_thing": "hello"},
)
```
If we do, these are passed to `method_name` when we come back out of
deferral.
We can also return a payload from the trigger, and this is also supposed to
be forwarded to `method_name`.
However, when you supply `kwargrs` in deferral, the event payload is ignored.
### What happened
_No response_
### What you expected to happen
I expecte both event payload _and_ deferral kwargs to be passed to the next
method when resuming.
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19922: ShortCircuitOperator does not save to xcom
josh-fell commented on issue #19922: URL: https://github.com/apache/airflow/issues/19922#issuecomment-985211860 +1 on seeing a simple or analogous example. No reason why the `callable` used in the `ShortCircuitOperator` couldn't push an `XCom` and then a downstream `BranchPythonOperator` uses that `XCom` to determine which branch(es) to follow. Of course, you're still up against #7858 being implemented first like you mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch constraints-main updated: Updating constraints. Build id:1533231867
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new c57d3c6 Updating constraints. Build id:1533231867 c57d3c6 is described below commit c57d3c61fda96321986dd0309b19b1d6f7b54b28 Author: Automated GitHub Actions commit AuthorDate: Fri Dec 3 02:14:13 2021 + Updating constraints. Build id:1533231867 This update in constraints is automatically committed by the CI 'constraints-push' step based on HEAD of 'refs/heads/main' in 'apache/airflow' with commit sha 525484388464619832a14d1b28e06e3a097aac97. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.6.txt | 16 constraints-3.7.txt | 18 +- constraints-3.8.txt | 18 +- constraints-3.9.txt | 18 +- constraints-no-providers-3.6.txt | 4 ++-- constraints-no-providers-3.7.txt | 4 ++-- constraints-no-providers-3.8.txt | 4 ++-- constraints-no-providers-3.9.txt | 4 ++-- constraints-source-providers-3.6.txt | 2 +- constraints-source-providers-3.7.txt | 2 +- constraints-source-providers-3.8.txt | 2 +- constraints-source-providers-3.9.txt | 2 +- 12 files changed, 47 insertions(+), 47 deletions(-) diff --git a/constraints-3.6.txt b/constraints-3.6.txt index 80dc456..f0e8ac1 100644 --- a/constraints-3.6.txt +++ b/constraints-3.6.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2021-12-01T14:08:57Z +# This constraints file was automatically generated on 2021-12-03T02:10:33Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -145,7 +145,7 @@ avro-python3==1.9.2.1 aws-xray-sdk==2.8.0 azure-batch==11.0.0 azure-common==1.1.27 -azure-core==1.20.1 +azure-core==1.21.0 azure-cosmos==4.2.0 azure-datalake-store==0.0.52 azure-identity==1.7.1 @@ -253,7 +253,7 @@ google-cloud-audit-log==0.2.0 google-cloud-automl==2.5.2 google-cloud-bigquery-datatransfer==3.4.1 google-cloud-bigquery-storage==2.10.1 -google-cloud-bigquery==2.30.1 +google-cloud-bigquery==2.31.0 google-cloud-bigtable==1.7.0 google-cloud-build==3.7.1 google-cloud-container==1.0.1 @@ -359,7 +359,7 @@ mysqlclient==2.1.0 nbclient==0.5.9 nbformat==5.1.3 neo4j==4.4.0 -nest-asyncio==1.5.1 +nest-asyncio==1.5.4 nodeenv==1.6.0 nox==2020.12.31 ntlm-auth==1.5.0 @@ -441,7 +441,7 @@ pytest-xdist==2.4.0 pytest==6.2.5 python-daemon==2.3.0 python-dateutil==2.8.2 -python-http-client==3.3.3 +python-http-client==3.3.4 python-jenkins==1.7.0 python-jose==3.3.0 python-ldap==3.4.0 @@ -466,14 +466,14 @@ requests-toolbelt==0.9.1 requests==2.26.0 responses==0.16.0 rfc3986==1.5.0 -rich==10.15.1 +rich==10.15.2 rsa==4.8 s3transfer==0.5.0 sasl==0.3.1 scramp==1.4.1 scrapbook==0.5.0 semver==2.13.0 -sendgrid==6.9.1 +sendgrid==6.9.2 sentinels==1.0.0 sentry-sdk==1.5.0 setproctitle==1.2.2 @@ -569,7 +569,7 @@ websocket-client==1.2.1 wrapt==1.13.3 xmltodict==0.12.0 yamllint==1.26.3 -yandexcloud==0.118.0 +yandexcloud==0.119.0 yarl==1.7.2 zdesk==2.7.1 zict==2.0.0 diff --git a/constraints-3.7.txt b/constraints-3.7.txt index d239054..a5083e7 100644 --- a/constraints-3.7.txt +++ b/constraints-3.7.txt @@ -1,5 +1,5 @@ # -# This constraints file was automatically generated on 2021-12-01T14:08:50Z +# This constraints file was automatically generated on 2021-12-03T02:10:27Z # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -144,7 +144,7 @@ avro-python3==1.9.2.1 aws-xray-sdk==2.8.0 azure-batch==11.0.0 azure-common==1.1.27 -azure-core==1.20.1 +azure-core==1.21.0 azure-cosmos==4.2.0 azure-datalake-store==0.0.52 azure-identity==1.7.1 @@ -250,7 +250,7 @@ google-cloud-audit-log==0.2.0 google-cloud-automl==2.5.2 google-cloud-bigquery-datatransfer==3.4.1 google-cloud-bigquery-storage==2.10.1 -google-cloud-bigquery==2.30.1 +google-cloud-bigquery==2.31.0 google-cloud-bigtable==1.7.0 google-cloud-build==3.7.1 google-cloud-container==1.0.1 @@ -306,7 +306,7 @@ influxdb-client==1.24.0 iniconfig==1.1.1 ipdb==0.13.9 ipython-genutils==0.2.0 -ipython==7.30.0 +ipython==7.30.1 iso8601==1.0.2 isodate==0.6.0 itsdangerous==1.1.0 @@ -356,7 +356,7 @@ mysqlclient==2.1.0 nbclient==0.5.9
[GitHub] [airflow] josh-fell commented on issue #12726: Add classic operator in TaskFlow API tutorial
josh-fell commented on issue #12726: URL: https://github.com/apache/airflow/issues/12726#issuecomment-985154006 Does anyone else agree that this can be closed now via existing docs + merging of #19124? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19895: Status of testing Providers that were prepared on November 30, 2021
josh-fell commented on issue #19895: URL: https://github.com/apache/airflow/issues/19895#issuecomment-985152246 Verified #18693, #19431, #19475, #19595, and #19383 Thanks again @potiuk! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] chinwobble commented on issue #19895: Status of testing Providers that were prepared on November 30, 2021
chinwobble commented on issue #19895: URL: https://github.com/apache/airflow/issues/19895#issuecomment-985149251 #19668 tested and working with `apache-airflow-providers-microsoft-azure==3.4.0rc1` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hwooson12 commented on issue #19955: Add hour and minute to time format on x-axis in Landing Times
hwooson12 commented on issue #19955: URL: https://github.com/apache/airflow/issues/19955#issuecomment-985134476 @bbovenzi I created a PR. But I am new to create a PR here so that the PR needs maintainer's approval to run workflows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] hwooson12 opened a new pull request #20002: Add hour and minute to time format on x-axis of all charts using nvd3.lineChart
hwooson12 opened a new pull request #20002: URL: https://github.com/apache/airflow/pull/20002 This PR adds a new date and time format for x axis where all `nvd3.lineChart` classes are initialized in order to see time of execution_date as well as date.  closes: #19955 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #20002: Add hour and minute to time format on x-axis of all charts using nvd3.lineChart
boring-cyborg[bot] commented on pull request #20002: URL: https://github.com/apache/airflow/pull/20002#issuecomment-985125454 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #19409: container_client.create_container() fails when using Container level SAS token (but fine with Storage level SAS token)
josh-fell commented on issue #19409: URL: https://github.com/apache/airflow/issues/19409#issuecomment-985124852 I was thinking we should add a disclaimer in the docs as an explicit "heads up" for others so there is no confusion until a possible fix is implemented. But if you could be so kind as to assign this to me that would be lovely. If you agree on the disclaimer, I can whip that up in the interim. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19993: Update upgrading.rst with detailed code example of how to resolve post-upgrade warning
potiuk commented on pull request #19993: URL: https://github.com/apache/airflow/pull/19993#issuecomment-985117344 Some whitespace problem :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19999: Don't use CREATE TABLE AS SELECT ... with mySQL
github-actions[bot] commented on pull request #1: URL: https://github.com/apache/airflow/pull/1#issuecomment-985115099 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #20000: Fix many of the mypy typing issues in airflow.models.dag
potiuk edited a comment on pull request #2: URL: https://github.com/apache/airflow/pull/2#issuecomment-985110025 AH NOOO!! I MISSED THE #2 one! Bummer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20000: Fix many of the mypy typing issues in airflow.models.dag
potiuk commented on pull request #2: URL: https://github.com/apache/airflow/pull/2#issuecomment-985110025 AH NOOO!! I MISSED THE #2000 one! Bummer! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17329: Split sql statements in DbApi run
github-actions[bot] commented on pull request #17329: URL: https://github.com/apache/airflow/pull/17329#issuecomment-985101679 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19019: Clean up JenkinsJobTriggerOperator
github-actions[bot] commented on pull request #19019: URL: https://github.com/apache/airflow/pull/19019#issuecomment-985101627 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19051: Add process state framework and watermark operator
github-actions[bot] commented on pull request #19051: URL: https://github.com/apache/airflow/pull/19051#issuecomment-985101594 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19039: Allow the user to use dag timezone to create dag_run_id
github-actions[bot] commented on pull request #19039: URL: https://github.com/apache/airflow/pull/19039#issuecomment-985101608 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
potiuk commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985099912 Hmm. And we should really create the venv with the API. that will be even more portable. Rather than running "python -m venv, we could simply use: https://docs.python.org/3/library/venv.html#api -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
potiuk commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985097296 > There is Python310/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory) From the issue: > The sysconfig module provides the programmatic interface to the directory structure - I'd suggest using "sysconfig.get_paths()" to find the locations where files may be written to. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
potiuk commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985096482 > Ah - so maybe on windows we have Scripts instead of bin ? @uranusjr ? Oh yeah: https://bugs.python.org/issue42863 > Also I used python for 64 bits and pyinstaller of 64 bits. ( they have to be similar to build something without that message of virus) Yep. I think we should stick to 64 bits as base platform. 32 is kinda really really old. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jon-fearer opened a new pull request #20001: fix(bigquery): fix big query to mssql transfer issue
jon-fearer opened a new pull request #20001: URL: https://github.com/apache/airflow/pull/20001 closes: [BigQueryToMsSqlOperator uses deprecated method and doesn't use keyword arguments](https://github.com/apache/airflow/issues/19900) related: [Fix BigQuery data extraction in BigQueryToMySqlOperator](https://github.com/apache/airflow/issues/17198) This fixes an issue with the BigQueryToMsSqlOperator that is similar to what was encountered with the BigQueryToMySqlOperator. Specifically, this PR: - Consolidates "_bq_get_data" from both operators into a single function - Updates the deprecated "get_tabledata" function to call "list_rows" using key word arguments instead of positional - Adds a few missing type hints Thanks in advance to maintainers. Please let me know if you have any thoughts or feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #20001: fix(bigquery): fix big query to mssql transfer issue
boring-cyborg[bot] commented on pull request #20001: URL: https://github.com/apache/airflow/pull/20001#issuecomment-985094998 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better . In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #20000: Fix many of the mypy typing issues in airflow.models.dag
ashb commented on pull request #2: URL: https://github.com/apache/airflow/pull/2#issuecomment-985094871 @uranusjr There were a couple of odd ones related to the timetable that I couldn't decipher: ``` airflow/models/dag.py:102: error: Variable "airflow.models.dag.ScheduleIntervalArgNotSet" is not valid as a type ScheduleIntervalArg = Union[ScheduleInterval, None, Type[ScheduleIntervalArgNotSet]] ^ airflow/models/dag.py:102: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases airflow/models/dag.py:159: error: Argument 2 to "CronDataIntervalTimetable" has incompatible type "tzinfo"; expected "Timezone" return CronDataIntervalTimetable(interval, timezone) ^ airflow/models/dag.py:428: error: Incompatible types in assignment (expression has type "Union[Union[str, timedelta, relativedelta], None, Type[ScheduleIntervalArgNotSet?]]", variable has type "Union[str, timedelta, relativedelta]") self.schedule_interval: ScheduleInterval = schedule_interval ^ airflow/models/dag.py:568: error: Argument 2 to "iter_dagrun_infos_between" of "DAG" has incompatible type "Optional[datetime]"; expected "DateTime" for info in self.iter_dagrun_infos_between(start_date, end_date, align=False) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb opened a new pull request #20000: Fix many of the mypy typing issues in airflow.models.dag
ashb opened a new pull request #2: URL: https://github.com/apache/airflow/pull/2 And to fix these, I needed to fix a few other mistakes that are used/called by DAG's methods. Relates to https://github.com/apache/airflow/issues/19891 - it doesn't completely fix any packages, but this change was getting larger. /cc @khalidmammadov -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (12ad64f -> 5254843)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 12ad64f Fixed MyPy issues in tests decorators and hooks (#19996) add 5254843 Fix race condition when starting DagProcessorAgent (#19935) No new revisions were added by this update. Summary of changes: airflow/dag_processing/manager.py | 10 +++ airflow/utils/process_utils.py | 55 -- tests/dag_processing/test_processor.py | 50 +++ 3 files changed, 100 insertions(+), 15 deletions(-)
[GitHub] [airflow] potiuk commented on pull request #19935: Fix race condition when starting DagProcessorAgent
potiuk commented on pull request #19935: URL: https://github.com/apache/airflow/pull/19935#issuecomment-985092064 Woohooo! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19935: Fix race condition when starting DagProcessorAgent
potiuk merged pull request #19935: URL: https://github.com/apache/airflow/pull/19935 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19935: Fix race condition when starting DagProcessorAgent
github-actions[bot] commented on pull request #19935: URL: https://github.com/apache/airflow/pull/19935#issuecomment-985089971 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] edithturn edited a comment on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
edithturn edited a comment on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985087637 > Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ? There is Python310/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory) Also I used python for 64 bits and pyinstaller of 64 bits. ( they have to be similar to build something without that message of virus) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
edithturn commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985087637 > Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ? There is Python/Scripts, where pip is, and I added that path into Variable environments to use pip and python. (not sure if we are speaking about the same directory) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19572: Simplify KubernetesPodOperator
jedcunningham commented on a change in pull request #19572:
URL: https://github.com/apache/airflow/pull/19572#discussion_r761542110
##
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##
@@ -117,79 +130,101 @@ def delete_pod(self, pod: V1Pod) -> None:
reraise=True,
retry=tenacity.retry_if_exception(should_retry_start_pod),
)
-def start_pod(self, pod: V1Pod, startup_timeout: int = 120) -> None:
+def create_pod(self, pod: V1Pod) -> V1Pod:
"""
-Launches the pod synchronously and waits for completion.
+Launches the pod asynchronously.
+
+:param pod:
+:return:
+"""
+return self.run_pod_async(pod)
+
+def await_pod_start(self, pod: V1Pod, startup_timeout: int = 120) -> None:
+"""
+Waits for the pod to reach phase other than ``Pending``
:param pod:
:param startup_timeout: Timeout (in seconds) for startup of the pod
(if pod is pending for too long, fails task)
:return:
"""
-resp = self.run_pod_async(pod)
curr_time = dt.now()
-if resp.status.start_time is None:
-while self.pod_not_started(pod):
-self.log.warning("Pod not yet started: %s", pod.metadata.name)
-delta = dt.now() - curr_time
-if delta.total_seconds() >= startup_timeout:
-msg = (
-f"Pod took longer than {startup_timeout} seconds to
start. "
-"Check the pod events in kubernetes to determine why."
-)
-raise AirflowException(msg)
-time.sleep(1)
-
-def monitor_pod(self, pod: V1Pod, get_logs: bool) -> Tuple[State, V1Pod,
Optional[str]]:
+while True:
+remote_pod = self.read_pod(pod)
+if remote_pod.status.phase != PodStatus.PENDING:
+break
+self.log.warning("Pod not yet started: %s", pod.metadata.name)
+delta = dt.now() - curr_time
+if delta.total_seconds() >= startup_timeout:
+msg = (
+f"Pod took longer than {startup_timeout} seconds to start.
"
+"Check the pod events in kubernetes to determine why."
+)
+raise PodLaunchFailedException(msg)
+time.sleep(1)
+
+def follow_container_logs(self, pod: V1Pod, container_name: str):
Review comment:
```suggestion
def follow_container_logs(self, pod: V1Pod, container_name: str) -> bool:
```
This one feels a little odd. It'll only ever return `True` or it will raise,
right? In that case, does the return value even buy us anything?
##
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##
@@ -51,10 +54,23 @@ def should_retry_start_pod(exception: Exception) -> bool:
class PodStatus:
"""Status of the PODs"""
-PENDING = 'pending'
-RUNNING = 'running'
-FAILED = 'failed'
-SUCCEEDED = 'succeeded'
+PENDING = 'Pending'
+RUNNING = 'Running'
+FAILED = 'Failed'
+SUCCEEDED = 'Succeeded'
+
+terminal_states = {FAILED, SUCCEEDED}
+
+
+def container_is_running(pod: V1Pod, container_name: str):
Review comment:
```suggestion
def container_is_running(pod: V1Pod, container_name: str) -> bool:
```
##
File path: airflow/providers/cncf/kubernetes/utils/pod_launcher.py
##
@@ -117,79 +130,101 @@ def delete_pod(self, pod: V1Pod) -> None:
reraise=True,
retry=tenacity.retry_if_exception(should_retry_start_pod),
)
-def start_pod(self, pod: V1Pod, startup_timeout: int = 120) -> None:
+def create_pod(self, pod: V1Pod) -> V1Pod:
"""
-Launches the pod synchronously and waits for completion.
+Launches the pod asynchronously.
+
+:param pod:
+:return:
+"""
+return self.run_pod_async(pod)
+
+def await_pod_start(self, pod: V1Pod, startup_timeout: int = 120) -> None:
+"""
+Waits for the pod to reach phase other than ``Pending``
:param pod:
:param startup_timeout: Timeout (in seconds) for startup of the pod
(if pod is pending for too long, fails task)
:return:
"""
-resp = self.run_pod_async(pod)
curr_time = dt.now()
-if resp.status.start_time is None:
-while self.pod_not_started(pod):
-self.log.warning("Pod not yet started: %s", pod.metadata.name)
-delta = dt.now() - curr_time
-if delta.total_seconds() >= startup_timeout:
-msg = (
-f"Pod took longer than {startup_timeout} seconds to
start. "
-"Check the pod events in kubernetes to determine why."
-)
-raise AirflowException(msg)
-
[GitHub] [airflow] github-actions[bot] commented on pull request #19997: Allow DockerOperator's image to be templated
github-actions[bot] commented on pull request #19997: URL: https://github.com/apache/airflow/pull/19997#issuecomment-985085656 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19997: Allow DockerOperator's image to be templated
potiuk commented on pull request #19997: URL: https://github.com/apache/airflow/pull/19997#issuecomment-985085484 Ah. So it wasn't ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Fixed MyPy issues in tests decorators and hooks (#19996)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 12ad64f Fixed MyPy issues in tests decorators and hooks (#19996)
12ad64f is described below
commit 12ad64ff00a361562d925a10223dd4bd2c4e63cc
Author: Khalid Mammadov
AuthorDate: Thu Dec 2 23:28:38 2021 +
Fixed MyPy issues in tests decorators and hooks (#19996)
---
tests/decorators/test_python.py | 4 ++--
tests/hooks/test_subprocess.py | 8 +++-
2 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/tests/decorators/test_python.py b/tests/decorators/test_python.py
index 2355568..610fa84 100644
--- a/tests/decorators/test_python.py
+++ b/tests/decorators/test_python.py
@@ -139,8 +139,8 @@ class TestAirflowTaskDecorator(TestPythonBase):
def test_manual_multiple_outputs_false_with_typings(self):
@task_decorator(multiple_outputs=False)
-def identity2(x: int, y: int) -> Dict[int, int]:
-return (x, y)
+def identity2(x: int, y: int) -> Tuple[int, int]:
+return x, y
with self.dag:
res = identity2(8, 4)
diff --git a/tests/hooks/test_subprocess.py b/tests/hooks/test_subprocess.py
index 4693da7..642b219 100644
--- a/tests/hooks/test_subprocess.py
+++ b/tests/hooks/test_subprocess.py
@@ -21,6 +21,7 @@ from pathlib import Path
from subprocess import PIPE, STDOUT
from tempfile import TemporaryDirectory
from unittest import mock
+from unittest.mock import MagicMock
from parameterized import parameterized
@@ -77,14 +78,11 @@ class TestSubprocessHook(unittest.TestCase):
@mock.patch.dict('os.environ', clear=True)
@mock.patch(
"airflow.hooks.subprocess.TemporaryDirectory",
-**{'return_value.__enter__.return_value': '/tmp/airflowtmpcatcat'}, #
type: ignore
+
return_value=MagicMock(__enter__=MagicMock(return_value='/tmp/airflowtmpcatcat')),
)
@mock.patch(
"airflow.hooks.subprocess.Popen",
-**{ # type: ignore
-'return_value.stdout.readline.side_effect': [b'BAR', b'BAZ'],
-'return_value.returncode': 0,
-},
+
return_value=MagicMock(stdout=MagicMock(readline=MagicMock(side_effect=StopIteration),
returncode=0)),
)
def test_should_exec_subprocess(self, mock_popen,
mock_temporary_directory):
hook = SubprocessHook()
[GitHub] [airflow] potiuk merged pull request #19996: Fixed MyPy issues in tests decorators and hooks
potiuk merged pull request #19996: URL: https://github.com/apache/airflow/pull/19996 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] SamWheating opened a new pull request #19999: Don't use CREATE TABLE AS SELECT ... with mySQL
SamWheating opened a new pull request #1: URL: https://github.com/apache/airflow/pull/1 Closes: https://github.com/apache/airflow/issues/19988 Splitting this `CREATE TABLE AS SELECT` query into two queries (`CREATE TABLE LIKE ..` followed by `INSERT INTO`) because the former doesn't play nicely with MySQL. This PR is inherently difficult to test/validate, but I ran the proposed queries against the metadata database in our development environment (where we first ran into this issue) and was able to replicate the issue and verify the fix: ``` // replicating the issue: MySQL [airflow] > create table _airflow_moved__2_2__task_instance as select source.* from task_instance as source; ERROR 1786 (HY000): Statement violates GTID consistency: CREATE TABLE ... SELECT. // Validating that splitting the query fixes the issue MySQL [airflow] > create table _airflow_moved__2_2__task_instance like task_instance; Query OK, 0 rows affected (0.063 sec) MySQL [airflow] > INSERT INTO _airflow_moved__2_2__task_instance select source.* from task_instance as source left join dag_run as dr on (source.dag_id = dr.dag_id and source.execution_date = dr.execution_date) where dr.id is null; Query OK, 22900 rows affected (2.110 sec) ``` Tomorrow I can create a patched image with these changes and validate the fix in our development environment. --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] khalidmammadov opened a new pull request #19998: Fixing MyPy issues in testa/jobs
khalidmammadov opened a new pull request #19998: URL: https://github.com/apache/airflow/pull/19998 Part of https://github.com/apache/airflow/issues/19891 Fixing MyPy issues within tests/jobs --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
potiuk commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985081675 Ah - so maybe on windows we have `Scripts` instead of bin ? @uranusjr ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19987: Remove postgres 9.6 support
jedcunningham commented on a change in pull request #19987:
URL: https://github.com/apache/airflow/pull/19987#discussion_r761538202
##
File path: scripts/ci/libraries/_initialization.sh
##
@@ -115,7 +115,7 @@ function initialization::initialize_base_variables() {
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS
# Currently supported versions of Postgres
-CURRENT_POSTGRES_VERSIONS+=("9.6" "13")
+CURRENT_POSTGRES_VERSIONS+=("10" "13")
Review comment:
Gotcha, cool, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19987: Remove postgres 9.6 support
potiuk edited a comment on pull request #19987: URL: https://github.com/apache/airflow/pull/19987#issuecomment-985079843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19987: Remove postgres 9.6 support
potiuk commented on pull request #19987: URL: https://github.com/apache/airflow/pull/19987#issuecomment-985079843 here -our rules for Python and Kubernetes https://github.com/apache/airflow/discussions/19871 - and we already removed K8S support for older versions without the intention of making it breaking change https://github.com/apache/airflow/commit/34484678bf16128ec7c0a3fd0dbe9f90b2a64ed7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19987: Remove postgres 9.6 support
potiuk commented on pull request #19987: URL: https://github.com/apache/airflow/pull/19987#issuecomment-985077796 > Do we consider it to be a breaking change ? This is a good question. I do not think we should. When software reaches EOL, we remove it. This is what is going to happen very soon with Python 3.6 and we already agreed it's not technically a breaking change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ShakaibKhan commented on issue #17487: Make gantt view to show also retries
ShakaibKhan commented on issue #17487: URL: https://github.com/apache/airflow/issues/17487#issuecomment-985077443 @alexInhert make sense to me and agree that retries should be shown; could you please share the DAG you used to make the above gantt chart -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19987: Remove postgres 9.6 support
potiuk commented on a change in pull request #19987:
URL: https://github.com/apache/airflow/pull/19987#discussion_r761535278
##
File path: scripts/ci/libraries/_initialization.sh
##
@@ -115,7 +115,7 @@ function initialization::initialize_base_variables() {
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS
# Currently supported versions of Postgres
-CURRENT_POSTGRES_VERSIONS+=("9.6" "13")
+CURRENT_POSTGRES_VERSIONS+=("10" "13")
Review comment:
This was deliberate decision.
Those are not consistent because we made a conscious decision that if CI
works for "lowest" and "highest" version of Postgres - it works for those
between.
We NEVER saw postgres related version problem. Not that I am aware of (as
opposed to our Elephant in the room MySQL which keeps on appearing now and
then). this was pure optimization for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19987: Remove postgres 9.6 support
potiuk commented on a change in pull request #19987:
URL: https://github.com/apache/airflow/pull/19987#discussion_r761535278
##
File path: scripts/ci/libraries/_initialization.sh
##
@@ -115,7 +115,7 @@ function initialization::initialize_base_variables() {
export CURRENT_PYTHON_MAJOR_MINOR_VERSIONS
# Currently supported versions of Postgres
-CURRENT_POSTGRES_VERSIONS+=("9.6" "13")
+CURRENT_POSTGRES_VERSIONS+=("10" "13")
Review comment:
This was deliberate decision.
Those are not consistent because we made a conscious decision that if CI
works for "lowest" and "highest" version - it works for those between.
We NEVER saw postgres related version problem. Not that I am aware of (as
opposed to our Elephant in the room MySQL which keeps on appearing now and
then). this was pure optimization for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19994: Fix db downgrades
github-actions[bot] commented on pull request #19994: URL: https://github.com/apache/airflow/pull/19994#issuecomment-985074595 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19993: Update upgrading.rst with detailed code example of how to resolve post-upgrade warning
potiuk commented on pull request #19993: URL: https://github.com/apache/airflow/pull/19993#issuecomment-985073942 Nice! I really like when we give "good" guidance to our users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19993: Update upgrading.rst with detailed code example of how to resolve post-upgrade warning
github-actions[bot] commented on pull request #19993: URL: https://github.com/apache/airflow/pull/19993#issuecomment-985073847 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19990: Order bug report fields by importance
potiuk commented on pull request #19990: URL: https://github.com/apache/airflow/pull/19990#issuecomment-985073034 I like it a lot actually :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19990: Order bug report fields by importance
github-actions[bot] commented on pull request #19990: URL: https://github.com/apache/airflow/pull/19990#issuecomment-985072793 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19990: Order bug report fields by importance
potiuk commented on pull request #19990: URL: https://github.com/apache/airflow/pull/19990#issuecomment-985072388 I like it. I wonder what others think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #19505: Add params dag_id, task_id etc to XCom.serialize_value
dstandish commented on pull request #19505: URL: https://github.com/apache/airflow/pull/19505#issuecomment-985072178 > Is there any work arounds available until this gets merged? you could do it by overriding `set`, which receives all the params -- it just doesn't pass them on to `serialize_value` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19988: Migrations fail due to GTID Consistency Violation when using GTID Replication
potiuk commented on issue #19988: URL: https://github.com/apache/airflow/issues/19988#issuecomment-985064669 naaa. removal is probably out of question for quite a while. But I DO recommend migrating to Postgres. Saves time, money and likely a lot of hair not pulled out of desperation. Yeah. Please Submit PR :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] SamWheating commented on issue #19988: Migrations fail due to GTID Consistency Violation when using GTID Replication
SamWheating commented on issue #19988: URL: https://github.com/apache/airflow/issues/19988#issuecomment-985062585 Ah cool, I hadn't found that report when I searched previously, I'll make a note of checking the discussions tab next time. If it was under our control I would be happy to just disable GTID replication but unfortunately this is a mandatory setting in CloudSQL (the cost of using a managed platform I suppose ) I can submit a PR for this tomorrow, if y'all are in agreement with the proposed fix. Also thanks for the advanced notice of possible MySQL removal - I don't think this would be a huge issue for us as we _could_ switch to Postgres (which would then give us nice features like `CREATE TABLE AS SELECT` and `UPDATE .. RETURNING`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham opened a new pull request #19997: Allow DockerOperator's image to be templated
jedcunningham opened a new pull request #19997: URL: https://github.com/apache/airflow/pull/19997 Users may want to use templating when specifying the image to run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #19988: Migrations fail due to GTID Consistency Violation when using GTID Replication
potiuk edited a comment on issue #19988: URL: https://github.com/apache/airflow/issues/19988#issuecomment-985051118 This has been raised before (https://github.com/apache/airflow/discussions/19871), but since this is CloudSQL limitation, I believe we will need to fix it @uranusjr - wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19988: Migrations fail due to GTID Consistency Violation when using GTID Replication
potiuk commented on issue #19988: URL: https://github.com/apache/airflow/issues/19988#issuecomment-985051978 Oh. Did I forget to mention that MYSQL is our "Elephant in the Room", BTW ? https://lists.apache.org/thread/dp78j0ssyhx62008lbtblrc856nbmlfb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (53b2415 -> cb082d3)
This is an automated email from the ASF dual-hosted git repository. ash pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 53b2415 Better ``pod_template_file`` examples (#19691) add cb082d3 Remove deprecated template_fields from GoogleDriveToGCSOperator (#19991) No new revisions were added by this update. Summary of changes: airflow/providers/google/cloud/transfers/gdrive_to_gcs.py| 2 -- tests/providers/google/cloud/transfers/test_gdrive_to_gcs.py | 2 ++ 2 files changed, 2 insertions(+), 2 deletions(-)
[GitHub] [airflow] ashb closed issue #19977: GoogleDriveToGCSOperator.dry_run() raises AttributeError due to deprecated template_fields
ashb closed issue #19977: URL: https://github.com/apache/airflow/issues/19977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (fae0d97 -> 53b2415)
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from fae0d97 hive provider: minor doc fixes (#19980) add 53b2415 Better ``pod_template_file`` examples (#19691) No new revisions were added by this update. Summary of changes: .../dags_in_image_template.yaml| 22 ++--- .../dags_in_volume_template.yaml | 23 ++ .../git_sync_template.yaml | 17 +--- 3 files changed, 17 insertions(+), 45 deletions(-)
[GitHub] [airflow] ashb commented on issue #19977: GoogleDriveToGCSOperator.dry_run() raises AttributeError due to deprecated template_fields
ashb commented on issue #19977: URL: https://github.com/apache/airflow/issues/19977#issuecomment-985051686 Closed by #19991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb merged pull request #19991: Remove deprecated template_fields and add update tests (#19977)
ashb merged pull request #19991: URL: https://github.com/apache/airflow/pull/19991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #19991: Remove deprecated template_fields and add update tests (#19977)
boring-cyborg[bot] commented on pull request #19991: URL: https://github.com/apache/airflow/pull/19991#issuecomment-985051517 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19988: Migrations fail due to GTID Consistency Violation when using GTID Replication
potiuk commented on issue #19988: URL: https://github.com/apache/airflow/issues/19988#issuecomment-985051118 This has been raised before, but since this is CloudSQL limitation, I believe we will need to fix it @uranusjr - wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #19691: Better ``pod_template_file`` examples
jedcunningham merged pull request #19691: URL: https://github.com/apache/airflow/pull/19691 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] khalidmammadov opened a new pull request #19996: Fixed MyPy issues in tests decorators and hooks
khalidmammadov opened a new pull request #19996: URL: https://github.com/apache/airflow/pull/19996 Part of https://github.com/apache/airflow/issues/19891 Fixed MyPy issues inside tests for decorator and hooks folders --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19984: Add timeout parameter to druid operator so it can be used in hook
potiuk edited a comment on pull request #19984: URL: https://github.com/apache/airflow/pull/19984#issuecomment-985014450 I think you should add/modify unit tests for the hook. Happy to Approve & Run them if you add them @piotrkmita -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19984: Add timeout parameter to druid operator so it can be used in hook
potiuk commented on pull request #19984: URL: https://github.com/apache/airflow/pull/19984#issuecomment-985014450 I think you should add/modify tests for the hook. Happy to Approve & Run them if you add them @piotrkmita -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: hive provider: minor doc fixes (#19980)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new fae0d97 hive provider: minor doc fixes (#19980) fae0d97 is described below commit fae0d97b0a0c21159949e1b0e5b24bb26964bd6e Author: Aakcht AuthorDate: Fri Dec 3 00:23:54 2021 +0300 hive provider: minor doc fixes (#19980) --- .../connections/hive_metastore.rst | 4 ++-- .../connections/hiveserver2.rst | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/apache-airflow-providers-apache-hive/connections/hive_metastore.rst b/docs/apache-airflow-providers-apache-hive/connections/hive_metastore.rst index c5cdb4d..45316d4 100644 --- a/docs/apache-airflow-providers-apache-hive/connections/hive_metastore.rst +++ b/docs/apache-airflow-providers-apache-hive/connections/hive_metastore.rst @@ -52,9 +52,9 @@ Extra (optional) The following parameters are all optional: * ``auth_mechanism`` - Specify the mechanism for authentication the default is ``NOSASL``. + Specify the mechanism for authentication. Default is ``NOSASL``. * ``kerberos_service_name`` - Specify the kerberos service name, the default is ``hive``. + Specify the kerberos service name. Default is ``hive``. When specifying the connection in environment variable you should specify diff --git a/docs/apache-airflow-providers-apache-hive/connections/hiveserver2.rst b/docs/apache-airflow-providers-apache-hive/connections/hiveserver2.rst index 693c295..1d16177 100644 --- a/docs/apache-airflow-providers-apache-hive/connections/hiveserver2.rst +++ b/docs/apache-airflow-providers-apache-hive/connections/hiveserver2.rst @@ -59,11 +59,11 @@ Extra (optional) The following parameters are all optional: * ``auth_mechanism`` - Specify the authentication method for PyHive choose between ``PLAIN``, ``LDAP``, ``KERBEROS`` or, ``Custom`` the default is ``PLAIN``. + Specify the authentication method for PyHive. Choose between ``PLAIN``, ``LDAP``, ``KERBEROS`` or ``Custom``. Default is ``PLAIN``. * ``kerberos_service_name`` - If authenticating with Kerberos specify the Kerberos service name, the default is ``hive``. + If authenticating with Kerberos specify the Kerberos service name. Default is ``hive``. * ``run_set_variable_statements`` - Specify the if you want to run set variable statements the default is ```True``. + Specify if you want to run set variable statements. Default is ``True``. When specifying the connection in environment variable you should specify
[GitHub] [airflow] potiuk merged pull request #19980: hive provider: minor doc fixes
potiuk merged pull request #19980: URL: https://github.com/apache/airflow/pull/19980 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19991: Remove deprecated template_fields and add update tests (#19977)
github-actions[bot] commented on pull request #19991: URL: https://github.com/apache/airflow/pull/19991#issuecomment-985011428 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
potiuk commented on issue #19962: URL: https://github.com/apache/airflow/issues/19962#issuecomment-985011243 > same effect. Also in the initial docker file, dags, logs and plugins folders are all mounted into the container. https://airflow.apache.org/docs/apache-airflow/2.2.2/docker-compose.yaml > > IMHO we need to mount the config folder as well to be able to make the logging customization file available inside the container If you want - yes. You should. Note that the docker-compose quick-start is just that, quick start to get up and running quickly. And if you want to customize it you are on your own there and you should modify the docker compose as you see fit: https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html > DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow. Configuring a Docker-Compose installation that is ready for production requires an intrinsic knowledge of Docker Compose, a lot of customization and possibly even writing the Docker Compose file that will suit your needs from the scratch. It’s probably OK if you want to run Docker Compose-based deployment, but short of becoming a Docker Compose expert, it’s highly unlikely you will get robust deployment with it. So all good. Just modify your docker compose in the way that suits you. No issue for Airflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19962: Unable to load custom logging from log_config.LOGGING_CONFIG due to No module named 'log_config'
potiuk closed issue #19962: URL: https://github.com/apache/airflow/issues/19962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] edithturn commented on pull request #19992: Use `python -m pip` to run pip in the new Breeze2
edithturn commented on pull request #19992: URL: https://github.com/apache/airflow/pull/19992#issuecomment-985010852 I checked that path: .build\breeze2\venv\bin on windows installation, "bin doesn't exist", in my test the installations are not generating that directory. I just got these files/directories instead on the venv directory: - Include - lib - Scripts - pyenv In Linux might be a different behavior to generate that venv directory. I am not sure. The error I was getting was this: "...dist\\Breeze2\\.build\\breze2\\venv returned non-zero exit status [4712] Failed to execute script 'Breeze2' due to unhandled exception" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (a50d2ac -> 347c067)
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from a50d2ac Ensure Spark driver response is valid before setting UNKNOWN status (#19978) add 347c067 Clean up incorrect class names of Google system tests (#19956) No new revisions were added by this update. Summary of changes: tests/providers/google/cloud/operators/test_datacatalog_system.py | 2 +- .../google/cloud/operators/test_gcs_timespan_file_transform_system.py | 2 +- tests/providers/google/cloud/operators/test_speech_to_text_system.py| 2 +- tests/providers/google/cloud/operators/test_stackdriver_system.py | 2 +- tests/providers/google/cloud/operators/test_translate_speech_system.py | 2 +- tests/providers/google/cloud/operators/test_workflows_system.py | 2 +- tests/providers/google/cloud/transfers/test_bigquery_to_gcs_system.py | 2 +- tests/providers/google/cloud/transfers/test_gcs_to_local_system.py | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-)
[GitHub] [airflow] potiuk merged pull request #19956: Clean up incorrect class names of Google system tests
potiuk merged pull request #19956: URL: https://github.com/apache/airflow/pull/19956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated: Ensure Spark driver response is valid before setting UNKNOWN status (#19978)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new a50d2ac Ensure Spark driver response is valid before setting UNKNOWN
status (#19978)
a50d2ac is described below
commit a50d2ac872da7e27d4cb32a2eb12cb75545c4a60
Author: PApostol <50751110+papos...@users.noreply.github.com>
AuthorDate: Thu Dec 2 21:12:25 2021 +
Ensure Spark driver response is valid before setting UNKNOWN status (#19978)
---
airflow/providers/apache/spark/hooks/spark_submit.py| 7 ++-
tests/providers/apache/spark/hooks/test_spark_submit.py | 15 +++
2 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/airflow/providers/apache/spark/hooks/spark_submit.py
b/airflow/providers/apache/spark/hooks/spark_submit.py
index b17eebe..677ea7b 100644
--- a/airflow/providers/apache/spark/hooks/spark_submit.py
+++ b/airflow/providers/apache/spark/hooks/spark_submit.py
@@ -520,10 +520,15 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
:param itr: An iterator which iterates over the input of the subprocess
"""
driver_found = False
+valid_response = False
# Consume the iterator
for line in itr:
line = line.strip()
+# A valid Spark status response should contain a submissionId
+if "submissionId" in line:
+valid_response = True
+
# Check if the log line is about the driver status and extract the
status.
if "driverState" in line:
self._driver_status = line.split(' : ')[1].replace(',',
'').replace('\"', '').strip()
@@ -531,7 +536,7 @@ class SparkSubmitHook(BaseHook, LoggingMixin):
self.log.debug("spark driver status log: %s", line)
-if not driver_found:
+if valid_response and not driver_found:
self._driver_status = "UNKNOWN"
def _start_driver_status_tracking(self) -> None:
diff --git a/tests/providers/apache/spark/hooks/test_spark_submit.py
b/tests/providers/apache/spark/hooks/test_spark_submit.py
index be47bfb..041d038 100644
--- a/tests/providers/apache/spark/hooks/test_spark_submit.py
+++ b/tests/providers/apache/spark/hooks/test_spark_submit.py
@@ -728,6 +728,21 @@ class TestSparkSubmitHook(unittest.TestCase):
assert hook._driver_status == 'RUNNING'
+def test_process_spark_driver_status_log_bad_response(self):
+# Given
+hook = SparkSubmitHook(conn_id='spark_standalone_cluster')
+log_lines = [
+'curl: Failed to connect to http://spark-standalone-master:6066'
+'This is an invalid Spark response',
+'Timed out',
+]
+# When
+hook._process_spark_status_log(log_lines)
+
+# Then
+
+assert hook._driver_status is None
+
@patch('airflow.providers.apache.spark.hooks.spark_submit.renew_from_kt')
@patch('airflow.providers.apache.spark.hooks.spark_submit.subprocess.Popen')
def test_yarn_process_on_kill(self, mock_popen, mock_renew_from_kt):
[GitHub] [airflow] potiuk merged pull request #19978: Ensure Spark driver response is valid before setting UNKNOWN status
potiuk merged pull request #19978: URL: https://github.com/apache/airflow/pull/19978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #19897: Spark driver relaunches if "driverState" is not found in curl response due to transient network issue
potiuk closed issue #19897: URL: https://github.com/apache/airflow/issues/19897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #19977: GoogleDriveToGCSOperator.dry_run() raises AttributeError due to deprecated template_fields
potiuk commented on issue #19977: URL: https://github.com/apache/airflow/issues/19977#issuecomment-985004138 I assigned you to it @bradleybonitatibus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19995: TaskInstance.log_url is uncallable due to execution_date being deprecated
boring-cyborg[bot] commented on issue #19995: URL: https://github.com/apache/airflow/issues/19995#issuecomment-985003529 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] PrincipalsOffice opened a new issue #19995: TaskInstance.log_url is uncallable due to execution_date being deprecated
PrincipalsOffice opened a new issue #19995:
URL: https://github.com/apache/airflow/issues/19995
### Apache Airflow version
2.2.2 (latest released)
### Operating System
Debian GNU/Linux
### Versions of Apache Airflow Providers
N/A
### Deployment
Other Docker-based deployment
### Deployment details
N/A
### What happened
When I created a TaskInstance object and tried to access its `log_url`
property, an `AttributeError: 'NoneType' object has no attribute 'isoformat'`
exception was raised.
### What you expected to happen
`task_instance.log_url` returns the correct url.
### How to reproduce
```
>>> import pendulum
>>> from airflow.models import DAG, TaskInstance
>>> from airflow.operators.dummy import DummyOperator
>>> dag = DAG("validation_dag", start_date=pendulum.parse("20200101"))
>>> task = DummyOperator(task_id="dummy_task", dag=dag,
task_group=dag.task_group)
>>> task_instance = TaskInstance(task, execution_date=execution_date,
run_id="test")
>>> task_instance.log_url
Traceback (most recent call last):
File "", line 1, in
File
"/path_to_my_venv/python3.8/site-packages/airflow/models/taskinstance.py", line
651, in log_url
iso = quote(self.execution_date.isoformat())
AttributeError: 'NoneType' object has no attribute 'isoformat'
```
### Anything else
Not sure if the log_url in 2.2.2 even support the execution_date param
anymore. What's the official expected behavior?
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow-client-python] renovate-ombrea opened a new issue #38: Pass verify_ssl as parameter
renovate-ombrea opened a new issue #38: URL: https://github.com/apache/airflow-client-python/issues/38 In client configuration i would like to pass verify_ssl to False but the value is hard coded. Could you change it to be a param ? Thanks ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18599: datadog parsing error for dagrun.schedule_delay since it is not passed in float type
potiuk commented on issue #18599: URL: https://github.com/apache/airflow/issues/18599#issuecomment-985001092 That's enough if you make a comment, if @coinzerge does not respond in a day or two - just take it over. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
