[GitHub] [airflow] ephraimbuddy commented on a change in pull request #20962: Support generating SQL script for upgrades
ephraimbuddy commented on a change in pull request #20962:
URL: https://github.com/apache/airflow/pull/20962#discussion_r794256824
##
File path: airflow/utils/db.py
##
@@ -1018,8 +1018,42 @@ def upgradedb(session: Session = NEW_SESSION):
exit(1)
with create_global_lock(session=session, lock=DBLocks.MIGRATIONS):
-log.info("Creating tables")
-command.upgrade(config, 'heads')
+log.info("Running migrations")
+rev_2_0_0_head = 'e959f08ac86c'
+if ':' in revision:
+sql = True
+_lowerband, _upperband = revision.split(':')
+head_to_lowerband_range = f"{rev_2_0_0_head}:{_lowerband}"
+head_to_upperband_range = f"{rev_2_0_0_head}:{_upperband}"
+for i in [head_to_lowerband_range, head_to_upperband_range]:
+try:
+command.history(config, rev_range=i)
+except Exception:
+raise AirflowException(
+f"Error while checking history for revision range {i}.
"
+f"Check that {i.split(':')[1]} is a valid revision. "
+f"Supported revision for offline migration is from
{rev_2_0_0_head} "
+f"which is airflow 2.0.0 head"
+)
+log.info("Running offline migrations for revision range %s",
revision)
+elif sql:
+# user supplied a specific SQL revision
+revision = f"{rev_2_0_0_head}:{revision}"
+try:
+command.history(config, rev_range=revision)
Review comment:
Working on suppressing the logs..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] Greetlist commented on issue #19929: Deferrable Operators leave task in strange state on kill
Greetlist commented on issue #19929: URL: https://github.com/apache/airflow/issues/19929#issuecomment-1023933892 > Later on, if the same task is started again, it finishes immediately, like it was continued after being deferred. I have a same problem in [#19612](https://github.com/apache/airflow/issues/19612). Hope it can help you. @eskarimov > When a task was killed while being executed by Triggerer process, there's no log available for this task after kill. Also, on_kill function of Operator isn't called. Triggerer process's log may have the detail log about it, task's ```try_number``` is decreased by ```TaskInstance._defer_task``` so that if this Deferrable Taks is killed, it's try_number is alway 0,so you can not see the log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Greetlist commented on issue #20460: TimeDeltaSensorAsync task is failing occasionally
Greetlist commented on issue #20460: URL: https://github.com/apache/airflow/issues/20460#issuecomment-1023924535 I have a similar problem in [#20308](https://github.com/apache/airflow/issues/20308), I fix it via a tricky way. I have a general solution in the issue, but I do not implement it. Hope it can help you. @andreychernih -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (684fe46 -> cdd9ea6)
This is an automated email from the ASF dual-hosted git repository. jhtimmins pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 684fe46 Static check in Breeze2 (#20848) add cdd9ea6 Add possibility to create user in the Remote User mode (#19963) No new revisions were added by this update. Summary of changes: airflow/www/views.py | 25 + 1 file changed, 25 insertions(+)
[GitHub] [airflow] jhtimmins merged pull request #19963: Add possibility to create user in the Remote User mode Auth.
jhtimmins merged pull request #19963: URL: https://github.com/apache/airflow/pull/19963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #21155: Fix #21096: Support boolean in extra__snowflake__insecure_mode
mik-laj commented on a change in pull request #21155: URL: https://github.com/apache/airflow/pull/21155#discussion_r794167953 ## File path: airflow/providers/snowflake/hooks/snowflake.py ## @@ -32,6 +32,14 @@ from airflow.utils.strings import to_boolean +def _try_to_boolean(value: Any): +if value is None: +return False +if isinstance(value, (str, type(None))): Review comment: Fixed! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on pull request #21175: Fixes Docker xcom functionality
ferruzzi commented on pull request #21175: URL: https://github.com/apache/airflow/pull/21175#issuecomment-1023834665 @uranusjr - You were looking at someone else's attempt at this, which has been sitting for a while. (link in the description) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi opened a new pull request #21175: Fixes Docker xcom functionality
ferruzzi opened a new pull request #21175: URL: https://github.com/apache/airflow/pull/21175 closes: https://github.com/apache/airflow/pull/19027 completes: https://github.com/apache/airflow/pull/19027/ Co-Author: https://github.com/asaf400 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Yeonghobyun edited a comment on issue #21141: Not showing "Last Run" on Airflow Web UI
Yeonghobyun edited a comment on issue #21141: URL: https://github.com/apache/airflow/issues/21141#issuecomment-1023790434 > I have faced the same issue when upgrading from 2.0.2 to 2.1.4, but it was resolved after upgrading again to 2.2.3. Is there any way to fix the source other than upgrading to 2.2.3? We just upgraded airflow 2.1.4 in production environment. Upgrading to 2.2.3 is burdensome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Yeonghobyun commented on issue #21141: Not showing "Last Run" on Airflow Web UI
Yeonghobyun commented on issue #21141: URL: https://github.com/apache/airflow/issues/21141#issuecomment-1023790434 > I have faced the same issue when upgrading from 2.0.2 to 2.1.4, but it was resolved after upgrading again to 2.2.3. Is there any way to fix the source other than upgrading to 2.2.3? We are just upgrading airflow 2.1.4 in production environment. Upgrading to 2.2.3 is burdensome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #20132: Fix type hints in airflow.macros
github-actions[bot] closed pull request #20132: URL: https://github.com/apache/airflow/pull/20132 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19662: BaseOperator.on_kill should optionally accept airflow context
github-actions[bot] commented on pull request #19662: URL: https://github.com/apache/airflow/pull/19662#issuecomment-1023755225 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #20309: Task exited with return code Negsignal.SIGSEGV
github-actions[bot] commented on issue #20309: URL: https://github.com/apache/airflow/issues/20309#issuecomment-1023755186 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #20309: Task exited with return code Negsignal.SIGSEGV
github-actions[bot] closed issue #20309: URL: https://github.com/apache/airflow/issues/20309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19102: Increment Stats counter for task instance SLA misses
github-actions[bot] commented on pull request #19102: URL: https://github.com/apache/airflow/pull/19102#issuecomment-1023755250 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Acehaidrey commented on pull request #20733: Add Audit Log View to Dag View
Acehaidrey commented on pull request #20733: URL: https://github.com/apache/airflow/pull/20733#issuecomment-1023750321 > I was thinking, should we even have any of this in the config? I think a better UX would be to show all possible events on the page and allow a user to check/uncheck each as they wish. Therefore on the page, a user immediately knows which types of events they can see instead of having to look at the config. Hi @bbovenzi on this end, I see what you are alluding to. So for the audit log I actually do not have a good idea of how to incorporate that here. To add a new filter selection or check/uncheck records. Are you open to discussing this further as a downstream action item. I think we can iterate on that. I can work on it right after this one too, if can have some of your help planning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Acehaidrey commented on pull request #20733: Add Audit Log View to Dag View
Acehaidrey commented on pull request #20733: URL: https://github.com/apache/airflow/pull/20733#issuecomment-1023748689 > I wonder if something like the `All Instances` button in the task instance modal can be added as well. The button would link to the audit log browse view with pre-applied filters for the dag / task. > >  Hey @wolfier ! Hope you're well! The case with this button. Right now the sentence links to the audit log view with the filters applied in the tags. I am wondering do you feel there should be multiple buttons here? Because if it is just the single All Instances button, I may say it doesn't seem so great to add just single button for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Acehaidrey commented on pull request #20733: Add Audit Log View to Dag View
Acehaidrey commented on pull request #20733: URL: https://github.com/apache/airflow/pull/20733#issuecomment-1023747908 sorry all - I been dealing with more covid complications but now I am back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] pateash closed pull request #21076: Adding GitHub provider
pateash closed pull request #21076: URL: https://github.com/apache/airflow/pull/21076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch v2-2-test updated (e898e0e -> 5ad965d)
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch v2-2-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard e898e0e Improved instructions for custom image build with docker compose (#21052) discard aae038c Add back legacy .piprc customization for pip (#21124) discard 485e071 Update logging-tasks.rst (#21088) discard 7f592d0 name mismatch (#21055) discard 72ce671 Update v1.yaml (#21024) discard b5da486 Return to the same place when triggering a DAG (#20955) discard bda0a74 Add downgrade to some FAB migrations (#20874) discard 3215d94 Add Roles from Azure OAUTH Response in security manager as it is currently not able map any AD roles to airflow ones (#20707) discard 0337fcb Allow Viewing DagRuns and TIs if a user has DAG "read" perms (#20663) discard 0bf4201 Add documentation and release policy on "latest" constraints (#21093) discard bcc3f7f Logs in to Github Registry when preparing cache (#21069) discard a85b649 Add a link to the DAG model in the Python API reference (#21060) discard 17b7da5 Fix running airflow dags test results in error when run twice (#21031) discard 5eeefc0 Temporary limit Pandas version (#21045) discard c9022f4 Update tutorial.rst (#21043) discard b334800 Add image labels required by ArtifactHub (#21040) discard 1b1bfc5 Fix grammar in ``dags.rst`` (#20988) discard d5870f0 Doc: Added an enum param example (#20841) discard c528166 Python3 requisite start local (#20777) discard 2dbe1e9 Update metric name in documentation (#20764) discard 7cd3fd6 Compare taskgroup and subdag (#20700) discard 111e8c1 Update operators.rst (#20640) discard 543a78b Improve documentation on ``Params`` (#20567) discard 312577e Removes unnecessary --upgrade option from our examples (#20537) discard e0b262c Deprecate smart sensors (#20151) discard e2bb598 Fix grammar and typos in "Logging for Tasks" guide (#20146) discard 8ecdcb9 Fix typo in MySQL Database creation code (Set up DB docs) (#20102) discard 239b1dc Correct set-up-database.rst (#20090) discard 346656f Add requirements.txt description (#20048) discard 4608eaa Fix example code in Doc (#19824) discard 7d37b0e Clean up ``default_args`` usage in docs (#19803) discard ce21b8e Change the name of link to ASF downloads (#19441) discard fbad277 Fix PostgresHook import in tutorial (#19374) discard d46674a docs: reorder imports in tutorials 🎨 (#19035) discard 2cc9ed0 Doc: Improve tutorial documentation and code (#19186) discard 0b45b27 Add docker-compose explanation to conn localhost (#19076) discard d183404 Update CSV ingest code for tutorial (#18960) discard a718042 Adds Pendulum 1.x -> 2.x upgrade documentation (#18955) discard ce341b6 Updating explicit arg example in TaskFlow API tutorial doc (#18907) discard 3714aa3 Adds back documentation about context usage in Python/@task (#18868) discard eb3ded7 Hide version selector for non-versioned packages (#21041) discard f00583e Fix occasional external task sensor tests (#18853) discard 7f6fc19 Restore stability and unquarantine all test_scheduler_job tests (#19860) discard 0ea3608 Fix failing main. (#20094) discard bcf18e5 Add roles to create_user test (#20773) discard 9c6e2d6 Update Celery requirements discard 34fe8dc Update md5 information about image after waiting (#21000) discard 3a165e9 Update refreshing constraints instructions (#21001) discard 995a841 Get rid of upload coverage warnings (#20994) discard 77a82af Add extra sync when adding executable flag to installation scripts (#20987) discard eff9929 Update base python image to be Python 3.7 by default (#20978) discard cd05b1f Fix new buildkit builds on MacOS (#20963) discard ff5ca5b Switch to 'buildkit' to build Airflow images (#20664) discard c133e37 Fix errors thrown by some versions of Bash v4 (#20932) discard e760582 Switch to non-vendored latest connexion library (#20910) discard ec92a35 Uses CI images built in previous step to prepare PROD image (#20889) discard 84b1819 Switch to new MySQL public key (#20912) discard b3e16e2 add entry in release readme to update milestone in Issues (#20890) discard 53a1fc7 Unpin ``argcomplete`` and ``colorlog`` (#20878) discard 1f16854 Unpin ``cattrs`` (#20872) discard d4195a1 Optimize dockerfiles for local rebuilds (#20238) discard f56bc0c Cleaner output for Docker image building (#20747) discard b9d175f Uses airflow user for build segment of docker image (#20744) discard 3fea595 Modernize usage of PIP in Airflow images (#20726) discard 8e2f07b Add color to pytest tests on CI (#20723) discard d23a685 Fix naming convention for sdist provider packages (#20711) discard f4b6d9f Be build -> built, and a stray space (#20703) discard 9e9db3e Cleaner output of docker image building scripts (#20679) discard 4c5661f Remove Python 2 from our images (#20680) discard ec0b7dc Fix incorrect arg
[GitHub] [airflow] lewismc commented on pull request #20755: Augment xcom docs
lewismc commented on pull request #20755: URL: https://github.com/apache/airflow/pull/20755#issuecomment-1023683095 @potiuk finally :) thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] lidalei commented on a change in pull request #21170: Add a note about broken DAG dependency graph
lidalei commented on a change in pull request #21170: URL: https://github.com/apache/airflow/pull/21170#discussion_r794018564 ## File path: docs/apache-airflow/concepts/dags.rst ## @@ -713,7 +713,8 @@ Additional difficulty is that one DAG could wait for or trigger several runs of with different data intervals. The **Dag Dependencies** view ``Menu -> Browse -> DAG Dependencies`` helps visualize dependencies between DAGs. The dependencies are calculated by the scheduler during DAG serialization and the webserver uses them to build -the dependency graph. +the dependency graph. Note that if a DAG depends on another DAG which does not exist (for example a deleted DAG), +the webserver will fail to render the dependency graph. Review comment: Agreed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #18707: Fix mismatch in generated run_id and actual execution_date of dag run
github-actions[bot] commented on pull request #18707: URL: https://github.com/apache/airflow/pull/18707#issuecomment-1023651349 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21171: Airflow schedules tasks early with an MSSQL Metadata DB on a non-UTC server
potiuk edited a comment on issue #21171: URL: https://github.com/apache/airflow/issues/21171#issuecomment-1023643343 I think db-specific case will be better (if simple). We already have ~500 deps in Airflow total (including transitive) and while adding one more seems like no-biggie, adding a 'util' in Airlfow seems to be more "straightforward". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #18707: Fix mismatch in generated run_id and actual execution_date of dag run
jedcunningham closed pull request #18707: URL: https://github.com/apache/airflow/pull/18707 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] davidcaron opened a new pull request #18707: Fix mismatch in generated run_id and actual execution_date of dag run
davidcaron opened a new pull request #18707: URL: https://github.com/apache/airflow/pull/18707 When execution_date is `None`, the generated name for the `run_id` contains the a generated `execution_date` in isoformat. That `execution_date`should be passed to `trigger_dag`, otherwise we have a mismatch between the `execution_date` in `run_id` and the actual `execution_date` of the dag run. This didn't cause any issues per se, but I thought it should be fixed for consistency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21171: Airflow schedules tasks early with an MSSQL Metadata DB on a non-UTC server
potiuk commented on issue #21171: URL: https://github.com/apache/airflow/issues/21171#issuecomment-1023643343 I think db-specific case will be better (if simple) we already have ~500 deps in Airflow total (including transitive) and while adding one more seems like no-biggie, adding a 'util' in Airlfow seems to be more "straightforward". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mattinbits commented on issue #21171: Airflow schedules tasks early with an MSSQL Metadata DB on a non-UTC server
mattinbits commented on issue #21171: URL: https://github.com/apache/airflow/issues/21171#issuecomment-1023636680 @potiuk just to check before I attempt a fix. Are you happy with a solution that uses sqlalchemy-utc for all DB backends, or is it preferrable to have a solution that handles the MSSQL case specifically, and leaves other DB backends unchanged? For example I see there are some DB specific utilities here https://github.com/apache/airflow/blob/aa2cb5545f09d694b9143b323efcd4f6b6c66e60/airflow/utils/sqlalchemy.py -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21171: Airflow schedules tasks early with an MSSQL Metadata DB on a non-UTC server
potiuk commented on issue #21171: URL: https://github.com/apache/airflow/issues/21171#issuecomment-1023627249 Not the first time MySQL bites us. Seems that indeed people even created packages to handle it correctly https://github.com/spoqa/sqlalchemy-utc. I will assign you to this one :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ferruzzi commented on issue #21169: `is_delete_operator_pod=True` and `random_name_suffix=False` can cause KubernetesPodOperator to delete the wrong pod
ferruzzi commented on issue #21169: URL: https://github.com/apache/airflow/issues/21169#issuecomment-1023626676 Interesting combination of events. Another possible solution might be to check the failure cause before calling delete_pod, if it failed because the pod already existed, then don't delete it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mattinbits opened a new issue #21171: Airflow schedules tasks early with an MSSQL Metadata DB on a non-UTC server
mattinbits opened a new issue #21171: URL: https://github.com/apache/airflow/issues/21171 ### Apache Airflow version 2.2.2 ### What happened Airflow schedules a task an hour earlier than expected, when using an MSSQL metadata database where the DB server is set to the CET timezone. The screenshot below shows the DAG starting an hour before the end of the data interval. https://user-images.githubusercontent.com/3765307/151439921-43b2ac5b-3e93-4337-a311-82e694ad3084.png";> ### What you expected to happen Airflow schedules the task at the correct time in UTC. ### How to reproduce It's hard to describe a complete reproducible method since it relies on having an MSSQL Server with particular settings. A relevant DAG would be a simple as: ``` with DAG( dag_id="example_dag", start_date=datetime(2021, 1, 1), schedule_interval="0 9 * * 1-5", ) as dag: task = DummyOperator(task_id="dummy") ``` And Airflow config of: ``` default_timezone = utc ``` This DAG would then be scheduled an hour earlier than expected. ### Operating System Redhat UBI 8 ### Versions of Apache Airflow Providers _No response_ ### Deployment Other Docker-based deployment ### Deployment details Airflow scheduler and webserver each running in a docker container based on Redhat UBI 8. Metadata DB is MSSQL Server running on a Windows Server where the server timezone is CET. ### Anything else In our installation, the problem is happening for any DAG with a UTC based schedule. I believe the root cause is this line of code: https://github.com/apache/airflow/blob/6405d8f804e7cbd1748aa7eed65f2bbf0fcf022e/airflow/models/dag.py#L2872 On MSSQL, `func.now()` appears to correspond to `GETDATE()`, which returns the current time in the timezone of the DB server. But `next_dagrun_create_after` is stored in the database as UTC (in a `datetime2` column, which doesn't include timezone information). So this line of code is equivalent to "Is the current time in CET before the next creation time in UTC?", meaning that a DAG that should start at 09:00 UTC starts at 09:00 CET instead, one hour early. I can verify that `func.now()` returns CET with the SQLAlchemy code `engine.execute(sa.select([sa.func.now()])).fetchall()`. I think the correct way to get the current time in UTC on MSSQL is `GETUTCDATE()`. We ran Airflow 1.10 previously without seeing this problem. From what I can tell, in that version the date comparison is done on the application side rather than in the DB. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #21170: Add a note about broken DAG dependency graph
eladkal commented on a change in pull request #21170: URL: https://github.com/apache/airflow/pull/21170#discussion_r793978951 ## File path: docs/apache-airflow/concepts/dags.rst ## @@ -713,7 +713,8 @@ Additional difficulty is that one DAG could wait for or trigger several runs of with different data intervals. The **Dag Dependencies** view ``Menu -> Browse -> DAG Dependencies`` helps visualize dependencies between DAGs. The dependencies are calculated by the scheduler during DAG serialization and the webserver uses them to build -the dependency graph. +the dependency graph. Note that if a DAG depends on another DAG which does not exist (for example a deleted DAG), +the webserver will fail to render the dependency graph. Review comment: mmm I don't think this is something for documentation. it sounds like a bug. The webserver shouldn't fail - it should explain to the user why the page isn't being rendered -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] lidalei commented on issue #21059: Dag dependency view is not rendering for Postgres backed Airflow
lidalei commented on issue #21059: URL: https://github.com/apache/airflow/issues/21059#issuecomment-1023590394 @potiuk Thanks! I created a PR to explain the case. https://github.com/apache/airflow/pull/21170 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] lidalei opened a new pull request #21170: Add a note about broken DAG dependency graph
lidalei opened a new pull request #21170: URL: https://github.com/apache/airflow/pull/21170 Airflow webserver fails to render dependency graph is a dependency DAG is missing. The issue and cause was described in https://github.com/apache/airflow/issues/21059. related: [Dag dependency view is not rendering for Postgres backed Airflow](https://github.com/apache/airflow/issues/21059) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] SamWheating opened a new issue #21169: `is_delete_operator_pod=True` and `random_name_suffix=False` can cause KubernetesPodOperator to delete the wrong pod
SamWheating opened a new issue #21169: URL: https://github.com/apache/airflow/issues/21169 ### Apache Airflow version 2.2.2 ### What happened When running multiple KubernetesPodOperators with `random_name_suffix=False` and `is_delete_pod_operator=True` the following will happen: 1) The first task will create the Pod `my-pod` 2) The second task will attempt to create the pod, but fail with a 409 response from the API server (this is expected) 3) The second task will delete `my-pod`, because it has `is_delete_pod_operator=True` and the Pod name is consistent between the two tasks. This is unexpected and will cause the first task to fail as well. I understand that this is a rare circumstance, but I think its still worth fixing as anyone using `random_name_suffix=False` in an otherwise default KubernetesPodOperator may result in other pods being killed. As a possible fix, we could [`find_pod`](https://github.com/apache/airflow/blob/684fe46158aa3d6cb2de245d29e20e487d8f2158/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L322) before deleting, to ensure that the pod being deleted has the appropriate `execution_date` label: https://github.com/apache/airflow/blob/ad07923606262ef8a650dcead38183da6bbb5d7b/airflow/providers/cncf/kubernetes/utils/pod_launcher.py#L103-L112 Let me know if you have any other suggestions for how this could be fixed, or if this should just be considered expected behaviour when using fixed Kubernetes Pod IDs. ### What you expected to happen The second task should be able to fail without deleting the pod from the first task. ### How to reproduce Create a DAG with a single KubernetesPodOperator with `random_name_suffix=False` and `is_delete_pod_operator=True` and run it twice in parallel. ### Operating System Debian GNU/Linux 10 (buster) ### Versions of Apache Airflow Providers apache-airflow-providers-cncf-kubernetes=2.2.0 ### Deployment Other 3rd-party Helm chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on a change in pull request #20998: Add dbt Cloud provider
josh-fell commented on a change in pull request #20998:
URL: https://github.com/apache/airflow/pull/20998#discussion_r793949682
##
File path: airflow/providers/dbt/cloud/hooks/dbt.py
##
@@ -0,0 +1,488 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import sys
+import time
+from enum import Enum
+from functools import wraps
+from inspect import signature
+from typing import Any, Callable, Dict, List, Optional, Sequence, Set, Tuple,
Union
+
+from requests import PreparedRequest, Session
+from requests.auth import AuthBase
+from requests.models import Response
+
+from airflow.exceptions import AirflowException
+from airflow.models import Connection
+from airflow.providers.http.hooks.http import HttpHook
+from airflow.typing_compat import TypedDict
+
+if sys.version_info >= (3, 8):
+from functools import cached_property
+else:
+from cached_property import cached_property
+
+
+def fallback_to_default_account(func: Callable) -> Callable:
+"""
+Decorator which provides a fallback value for ``account_id``. If the
``account_id`` is None or not passed
+to the decorated function, the value will be taken from the configured dbt
Cloud Airflow Connection.
+"""
+sig = signature(func)
+
+@wraps(func)
+def wrapper(*args, **kwargs) -> Callable:
+bound_args = sig.bind(*args, **kwargs)
+
+# Check if ``account_id`` was not included in the function signature
or, if it is, the value is not
+# provided.
+if bound_args.arguments.get("account_id") is None:
+self = args[0]
+default_account_id = self.conn.login
+if not default_account_id:
+raise AirflowException("Could not determine the dbt Cloud
account.")
+
+bound_args.arguments["account_id"] = int(default_account_id)
+
+return func(*bound_args.args, **bound_args.kwargs)
+
+return wrapper
+
+
+class TokenAuth(AuthBase):
+"""Helper class for Auth when executing requests."""
+
+def __init__(self, token: str) -> None:
+self.token = token
+
+def __call__(self, request: PreparedRequest) -> PreparedRequest:
+request.headers["Content-Type"] = "application/json"
+request.headers["Authorization"] = f"Token {self.token}"
+
+return request
+
+
+class JobRunInfo(TypedDict):
+"""Type class for the ``job_run_info`` dictionary."""
+
+account_id: int
+run_id: int
+
+
+class DbtCloudJobRunStatus(Enum):
+"""dbt Cloud Job statuses."""
+
+QUEUED = 1
+STARTING = 2
+RUNNING = 3
+SUCCESS = 10
+ERROR = 20
+CANCELLED = 30
+TERMINAL_STATUSES = (SUCCESS, ERROR, CANCELLED)
+
+@classmethod
+def check_is_valid(cls, statuses: Union[int, Sequence[int], Set[int]]):
+"""Validates input statuses are a known value."""
+if isinstance(statuses, (Sequence, Set)):
+for status in statuses:
+cls(status)
+else:
+cls(statuses)
+
+@classmethod
+def is_terminal(cls, status: int) -> bool:
+"""Checks if the input status is that of a terminal type."""
+cls.check_is_valid(statuses=status)
+
+return status in cls.TERMINAL_STATUSES.value
+
+
+class DbtCloudJobRunException(AirflowException):
+"""An exception that indicates a job run failed to complete."""
+
+
+class DbtCloudHook(HttpHook):
+"""
+Interact with dbt Cloud using the V2 API.
+
+:param dbt_cloud_conn_id: The ID of the :ref:`dbt Cloud connection
`.
+"""
+
+conn_name_attr = "dbt_cloud_conn_id"
+default_conn_name = "dbt_cloud_default"
+conn_type = "dbt_cloud"
+hook_name = "dbt Cloud"
+
+@staticmethod
+def get_ui_field_behaviour() -> Dict[str, Any]:
+"""Builds custom field behavior for the dbt Cloud connection form in
the Airflow UI."""
+return {
+"hidden_fields": ["host", "port", "schema", "extra"],
+"relabeling": {"login": "Account ID", "password": "API Token"},
+}
+
+def __init__(self, dbt_cloud_conn_id: str = default_conn_name, *args,
**kwargs) -> None:
+super().__init__(auth_type=TokenAuth)
+self.dbt_cloud_conn_id = dbt_cloud_conn_id
+self
[GitHub] [airflow] SangwanP opened a new issue #21168: Add "last updated timestamp" field to API endpoints
SangwanP opened a new issue #21168: URL: https://github.com/apache/airflow/issues/21168 ### Description _No response_ ### Use case/motivation I would like to have a field added to API endpoints which reflects when a record, that is received in the API response, was last updated. It would be nice to have such a field added to as many endpoints as possible. For example: last_scheduling_decision field can be exposed via the DAG Run API endpoint. This will give the user information on when a given DAGRun record was last updated. I think it's common practice to expose when a record was created and/or last updated, through API's. Particular use case: Our team is running an ETL using the Airflow API as the source system. We would like to load this data incrementally, however we will be unable to do so without a 'last updated' timestamp field exposed through the API endpoints. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] alexhallam commented on issue #21160: Table View
alexhallam commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023564550 I agree! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21165: Update Broken link in api.rst
github-actions[bot] commented on pull request #21165: URL: https://github.com/apache/airflow/pull/21165#issuecomment-1023563500 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21165: Update Broken link in api.rst
potiuk commented on pull request #21165: URL: https://github.com/apache/airflow/pull/21165#issuecomment-1023563283 Static chekcs are failing though :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21160: Table View
potiuk commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023561952 Maybe worth re-evaluation after 2.3.0 is released. When you see how it works and if you are unhappy about it, this might be re-requrested (but in the new context). I do believe the new "not-tree-view-any-more" might change a lot of user patterns there :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21162: Change logging level details of connection info in `get_connection()`
github-actions[bot] commented on pull request #21162: URL: https://github.com/apache/airflow/pull/21162#issuecomment-1023560760 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] alexhallam commented on issue #21160: Table View
alexhallam commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023560089 That makes sense. I still think explicit tables would be nice so statistics show without needing to hover, but it is not a big enough issue to fuss about. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21156: Snowflake connection default value of extras insecure_mode set as string instead of boolean, throws error in Snowflake hook
potiuk commented on issue #21156: URL: https://github.com/apache/airflow/issues/21156#issuecomment-1023559732 Duplicate of #21155 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21156: Snowflake connection default value of extras insecure_mode set as string instead of boolean, throws error in Snowflake hook
potiuk closed issue #21156: URL: https://github.com/apache/airflow/issues/21156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21160: Table View
potiuk commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023558040 Note - It is shown in "graphical" way not only as tooltip. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21160: Table View
potiuk commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023557306 Duplicate of #18675 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21160: Table View
potiuk commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023557116 There is a new tree view coming in Airlfwow 2.3.0 It's already merged. Initial version of it can be seen here https://github.com/apache/airflow/pull/18675 but it already had gone through a number of iterations so it is even better than that. I guess it handles your request pretty nicely. I will close this one as duplicate but if you think your issue is not handled, feel free to comment here and explain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21160: Table View
potiuk closed issue #21160: URL: https://github.com/apache/airflow/issues/21160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21059: Dag dependency view is not rendering for Postgres backed Airflow
potiuk edited a comment on issue #21059: URL: https://github.com/apache/airflow/issues/21059#issuecomment-1023551897 Hmmm maybe it would be a good idea to add it as PR to https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/external_task_sensor.html?highlight=dag%20dependency (I guess this is the right page) I think the two f you @lidalei @zambadruzaman are one of the two best people out there to write such a description tha will be helpful and useful for people like you who will struggle with it :). This is really a good way to give back in return for the free software you get ! You can become one of the more thab 1900 (and approaching 2000 fast) number of contributors to Airflow? This is very simple. Just click "Suggest chnage on this page" at the bottom right and you will get a PR where you will be able to add the change. What say you ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21059: Dag dependency view is not rendering for Postgres backed Airflow
potiuk commented on issue #21059: URL: https://github.com/apache/airflow/issues/21059#issuecomment-1023551897 Hmmm maybe it woudl be a good idea to add it as PR to https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/external_task_sensor.html?highlight=dag%20dependency (I guess this is the right page) I think the two f you @lidalei @zambadruzaman are one of the two best people out there to write such a description tha will be helpful and useful for people like you who will struggle with it :). This is really a good way to give back in return for the free software you get ! You can become one of the more thab 1900 (and approaching 2000 fast) number of contributors to Airflow? This is very simple. Just click "Suggest chnage on this page" at the bottom right and you will get a PR where you will be able to add the change. What say you ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] julienledem commented on pull request #21157: taskinstance: set task on sqlalchemy taskinstance object
julienledem commented on pull request #21157: URL: https://github.com/apache/airflow/pull/21157#issuecomment-1023548597 👍🏻 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] blag opened a new pull request #21167: Add a session backend to store session data in the database
blag opened a new pull request #21167: URL: https://github.com/apache/airflow/pull/21167 This PR re-implements modified portions of [Flask-Session](https://github.com/fengsp/flask-session) to store session data in the database. TODO: * [ ] Tests --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #21155: Fix #21096: Support boolean in extra__snowflake__insecure_mode
potiuk commented on a change in pull request #21155: URL: https://github.com/apache/airflow/pull/21155#discussion_r793872553 ## File path: airflow/providers/snowflake/hooks/snowflake.py ## @@ -32,6 +32,14 @@ from airflow.utils.strings import to_boolean +def _try_to_boolean(value: Any): +if value is None: +return False +if isinstance(value, (str, type(None))): Review comment: isn't type(None) covered by earlier `is None` check? Can there be an object of NoneType that fails "is None" check? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21166: DagFileProcessorManager - ERROR - new DAG tasks are delayed in airflow schedule
boring-cyborg[bot] commented on issue #21166: URL: https://github.com/apache/airflow/issues/21166#issuecomment-1023497727 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ckvtvm-arch opened a new issue #21166: DagFileProcessorManager - ERROR - new DAG tasks are delayed in airflow schedule
ckvtvm-arch opened a new issue #21166:
URL: https://github.com/apache/airflow/issues/21166
### Apache Airflow version
2.2.3 (latest released)
### What happened
new DAG tasks are delayed in scheduling. When a new task is added, it takes
atleast 10 hours to schedule in airflow.
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has
been moved to the hide_sensitive_var_conn_fields option in [core] - the old
setting has been used, but please update your config.
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The default_queue option in [celery] has been moved to the
default_queue option in [operators] - the old setting has been used, but please
update your config.
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The default_queue option in [celery] has been moved to the
default_queue option in [operators] - the old setting has been used, but please
update your config.
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:357
DeprecationWarning: The dag_concurrency option in [core] has been renamed to
max_active_tasks_per_dag - the old setting has been used, but please update
your config.
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The processor_poll_interval option in [scheduler] has been
renamed to scheduler_idle_sleep_time - the old setting has been used, but
please update your config.
_
|__( )_ __/__ / __
/| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_//_//_/ \//|__/
[[34m2022-01-26 19:44:29,624[0m] {[34mscheduler_job.py:[0m596} INFO[0m
- Starting the scheduler[0m
[[34m2022-01-26 19:44:29,625[0m] {[34mscheduler_job.py:[0m601} INFO[0m
- Processing each file at most -1 times[0m
[[34m2022-01-26 19:44:29,743[0m] {[34mmanager.py:[0m163} INFO[0m -
Launched DagFileProcessorManager with pid: 21[0m
[[34m2022-01-26 19:44:29,747[0m] {[34mscheduler_job.py:[0m1114} INFO[0m
- Resetting orphaned tasks for active dag runs[0m
[[34m2022-01-26 19:44:29,749[0m] {[34msettings.py:[0m52} INFO[0m -
Configured default timezone Timezone('America/New_York')[0m
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has
been moved to the hide_sensitive_var_conn_fields option in [core] - the old
setting has been used, but please update your config.
[[34m2022-01-26 19:46:30,004[0m] {[34mmanager.py:[0m334} ERROR[0m -
DagFileProcessorManager (PID=21) last sent a heartbeat 120.26 seconds ago!
Restarting it[0m
[[34m2022-01-26 19:46:30,005[0m] {[34mprocess_utils.py:[0m124} INFO[0m
- Sending Signals.SIGTERM to group 21. PIDs of all processes in the group:
[21][0m
[[34m2022-01-26 19:46:30,005[0m] {[34mprocess_utils.py:[0m75} INFO[0m -
Sending the signal Signals.SIGTERM to group 21[0m
[[34m2022-01-26 19:46:30,137[0m] {[34mprocess_utils.py:[0m70} INFO[0m -
Process psutil.Process(pid=21, status='terminated', exitcode=0,
started='19:44:29') (21) terminated with exit code 0[0m
[[34m2022-01-26 19:46:30,141[0m] {[34mmanager.py:[0m163} INFO[0m -
Launched DagFileProcessorManager with pid: 332[0m
[[34m2022-01-26 19:46:30,148[0m] {[34msettings.py:[0m52} INFO[0m -
Configured default timezone Timezone('America/New_York')[0m
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has
been moved to the hide_sensitive_var_conn_fields option in [core] - the old
setting has been used, but please update your config.
[[34m2022-01-26 19:48:31,121[0m] {[34mmanager.py:[0m334} ERROR[0m -
DagFileProcessorManager (PID=332) last sent a heartbeat 120.98 seconds ago!
Restarting it[0m
[[34m2022-01-26 19:48:31,122[0m] {[34mprocess_utils.py:[0m124} INFO[0m
- Sending Signals.SIGTERM to group 332. PIDs of all processes in the group:
[332][0m
[[34m2022-01-26 19:48:31,122[0m] {[34mprocess_utils.py:[0m75} INFO[0m -
Sending the signal Signals.SIGTERM to group 332[0m
[[34m2022-01-26 19:48:31,294[0m] {[34mprocess_utils.py:[0m70} INFO[0m -
Process psutil.Process(pid=332, status='terminated', exitcode=0,
started='19:46:29') (332) terminated with exit code 0[0m
[[34m2022-01-26 19:48:31,298[0m] {[34mmanager.py:[0m163} INFO[0m -
Launched DagFileProcessorManager with pid: 479[0m
[[34m2022-01-26 19:48:31,305[0m] {[34msettings.py:[0m52} INFO[0m -
Configured default timezone Timezone('America/New_York')[0m
/usr/local/lib/python3.6/site-packages/airflow/configuration.py:361
DeprecationWarning:
[GitHub] [airflow] fritz-astronomer opened a new pull request #21165: Update Broken link in api.rst
fritz-astronomer opened a new pull request #21165: URL: https://github.com/apache/airflow/pull/21165 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new issue #21164: Breeze2 autocomplete requires `click-complete` to be installed
potiuk opened a new issue #21164:
URL: https://github.com/apache/airflow/issues/21164
### Apache Airflow version
main (development)
### What happened
When I setup autocomplete for Breeze2 on a "bare" system when I have no
packages installed, It fails autocomplete with this error:
```
ModuleNotFoundError: No module named 'click_completion'
Traceback (most recent call last):
File "/home/jarek/.pyenv/versions/3.7.9/bin/Breeze2", line 33, in
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts',
'Breeze2')())
File "/home/jarek/.pyenv/versions/3.7.9/bin/Breeze2", line 25, in
importlib_load_entry_point
return next(matches).load()
File
"/home/jarek/.local/lib/python3.7/site-packages/importlib_metadata/__init__.py",
line 194, in load
module = import_module(match.group('module'))
File
"/home/jarek/.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py", line
127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 677, in _load_unlocked
File "", line 728, in exec_module
File "", line 219, in
_call_with_frames_removed
File "/home/jarek/code/airflow/dev/breeze/src/airflow_breeze/breeze.py",
line 24, in
import click_completion
ModuleNotFoundError: No module named 'click_completion'
```
It seems that "autocomplete" feature of Breeze2 requires `click-completion`
to be installed first. This is a small issue an small prerequisite but I think
it is not handled by the currrent `setup-autocomplete`
The same happens if you install Breeze2 with `pipx`.
### What you expected to happen
I expect that `click-completion` package is automatically installled in when
`./Breeze2 setup-autocomplete` is executed.
Also it should be described in the documentation as prerequisite
### How to reproduce
* make sure you have no packages installed in your Python "system
environment" (for example: `pip list | xargs pip uninstall -y` )
* type ./Breeze2
### Operating System
Linux Mint 20.1.3
### Versions of Apache Airflow Providers
Not relevant
### Deployment
Other
### Deployment details
No airflow deployment
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] josh-fell commented on issue #21156: Snowflake connection default value of extras insecure_mode set as string instead of boolean, throws error in Snowflake hook
josh-fell commented on issue #21156: URL: https://github.com/apache/airflow/issues/21156#issuecomment-1023480177 This is a duplicate of #21096 and #21155 has been raised to fix the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (82adce5 -> 684fe46)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git.
from 82adce5 Ensure `on_task_instance_running` listener can get at task
(#21157)
add 684fe46 Static check in Breeze2 (#20848)
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml| 4 +-
dev/breeze/src/airflow_breeze/breeze.py| 45 -
dev/breeze/src/airflow_breeze/global_constants.py | 3 +
dev/breeze/src/airflow_breeze/pre_commit_ids.py| 106
.../pre_commit_ids_TEMPLATE.py.jinja2} | 8 +-
dev/breeze/src/airflow_breeze/utils/run_utils.py | 36 +--
dev/breeze/tests/test_cache.py | 67 +
.../pre_commit_check_pre_commit_hook_names.py | 108 -
8 files changed, 359 insertions(+), 18 deletions(-)
create mode 100644 dev/breeze/src/airflow_breeze/pre_commit_ids.py
copy dev/{provider_packages/get_provider_info_TEMPLATE.py.jinja2 =>
breeze/src/airflow_breeze/pre_commit_ids_TEMPLATE.py.jinja2} (83%)
create mode 100644 dev/breeze/tests/test_cache.py
[GitHub] [airflow] potiuk closed issue #20740: Breeze: Running static checks with Breeze
potiuk closed issue #20740: URL: https://github.com/apache/airflow/issues/20740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #20848: Static check in Breeze2
potiuk merged pull request #20848: URL: https://github.com/apache/airflow/pull/20848 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20848: Static check in Breeze2
potiuk commented on pull request #20848: URL: https://github.com/apache/airflow/pull/20848#issuecomment-1023474381 Lks great! I tested it locally and it works really neatly! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #20962: Support generating SQL script for upgrades
ephraimbuddy commented on a change in pull request #20962:
URL: https://github.com/apache/airflow/pull/20962#discussion_r793840551
##
File path: airflow/utils/db.py
##
@@ -1018,8 +1018,42 @@ def upgradedb(session: Session = NEW_SESSION):
exit(1)
with create_global_lock(session=session, lock=DBLocks.MIGRATIONS):
-log.info("Creating tables")
-command.upgrade(config, 'heads')
+log.info("Running migrations")
+rev_2_0_0_head = 'e959f08ac86c'
+if ':' in revision:
+sql = True
+_lowerband, _upperband = revision.split(':')
+head_to_lowerband_range = f"{rev_2_0_0_head}:{_lowerband}"
+head_to_upperband_range = f"{rev_2_0_0_head}:{_upperband}"
+for i in [head_to_lowerband_range, head_to_upperband_range]:
+try:
+command.history(config, rev_range=i)
+except Exception:
+raise AirflowException(
+f"Error while checking history for revision range {i}.
"
+f"Check that {i.split(':')[1]} is a valid revision. "
+f"Supported revision for offline migration is from
{rev_2_0_0_head} "
+f"which is airflow 2.0.0 head"
+)
+log.info("Running offline migrations for revision range %s",
revision)
+elif sql:
+# user supplied a specific SQL revision
+revision = f"{rev_2_0_0_head}:{revision}"
+try:
+command.history(config, rev_range=revision)
Review comment:
It produces output and I can't think of a way to silence it. There's no
other way I could confirm that we should be good at producing the migration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[airflow] branch main updated: Ensure `on_task_instance_running` listener can get at task (#21157)
This is an automated email from the ASF dual-hosted git repository. ash pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 82adce5 Ensure `on_task_instance_running` listener can get at task (#21157) 82adce5 is described below commit 82adce535eb0c427c230035d648bf3c829824b21 Author: Maciej Obuchowski AuthorDate: Thu Jan 27 18:23:36 2022 +0100 Ensure `on_task_instance_running` listener can get at task (#21157) When we added TaskListener API. It's contract promises to pass TaskInstance object to listener plugin. However, what happens is not 100% true - the object being passed is one that maps to current SQLAlchemy session. `check_and_change_state_before_execution` operates on detached TaskInstance object, then merges it to current session. Since there is no attached object in the SQLAlchemy identity map, SQLAlchemy creates it, and it's this object that's being passed to the SQLAlchemy event listeners. The problem with that is that when creating new SQLAlchemy object, SQLAlchemy takes care about setting only database-mapped fields. The ones that are purely on the python side, like task aren't being set on the new object. This manually sets task on the new SQLAlchemy object, so that `on_task_instance_running` receives a TaskInstance with `task` field set. Signed-off-by: Maciej Obuchowski --- airflow/models/taskinstance.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/airflow/models/taskinstance.py b/airflow/models/taskinstance.py index 6472e74..ed3e9b0 100644 --- a/airflow/models/taskinstance.py +++ b/airflow/models/taskinstance.py @@ -1251,7 +1251,7 @@ class TaskInstance(Base, LoggingMixin): self.external_executor_id = external_executor_id self.end_date = None if not test_mode: -session.merge(self) +session.merge(self).task = task session.commit() # Closing all pooled connections to prevent
[GitHub] [airflow] ashb merged pull request #21157: taskinstance: set task on sqlalchemy taskinstance object
ashb merged pull request #21157: URL: https://github.com/apache/airflow/pull/21157 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk opened a new issue #21163: `setup-autocomplete` in Breeze2 fails with "Permission denied"
potiuk opened a new issue #21163:
URL: https://github.com/apache/airflow/issues/21163
### Apache Airflow version
main (development)
### What happened
When I run "setup-autocomplete" in the new Breeze with zsh it fails with
"permission denied" as it tries to access `/.zshrc`:
```
Installing zsh completion for local user
Activation command scripts are created in this autocompletion path:
/home/jarek/code/airflow/.build/autocomplete/Breeze2-complete.zsh
Do you want to add the above autocompletion scripts to your zsh profile?
[y/N]: y
This will modify the /.zshrc file
Traceback (most recent call last):
File "/home/jarek/code/airflow/.build/breeze2/venv/bin/Breeze2", line 33,
in
sys.exit(load_entry_point('apache-airflow-breeze', 'console_scripts',
'Breeze2')())
File
"/home/jarek/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py",
line 1128, in __call__
return self.main(*args, **kwargs)
File
"/home/jarek/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py",
line 1053, in main
rv = self.invoke(ctx)
File
"/home/jarek/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py",
line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File
"/home/jarek/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py",
line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File
"/home/jarek/code/airflow/.build/breeze2/venv/lib/python3.8/site-packages/click/core.py",
line 754, in invoke
return __callback(*args, **kwargs)
File "/home/jarek/code/airflow/dev/breeze/src/airflow_breeze/breeze.py",
line 248, in setup_autocomplete
write_to_shell(command_to_execute, script_path, breeze_comment)
File "/home/jarek/code/airflow/dev/breeze/src/airflow_breeze/breeze.py",
line 215, in write_to_shell
with open(script_path, 'a') as script_file:
PermissionError: [Errno 13] Permission denied: '/.zshrc'
[jarek:~/code/airflow] static-check-breeze2+ 6s 1 ±
```
### What you expected to happen
I expected the scripts in my `${HOME}` directory to be updated with
auto-complete but apparently it tries to update a file in `root' folder.
### How to reproduce
* Have zsh as your shell
* Run `./Breeze2 setup-autocomplete`
### Operating System
Linux Mint 20.1.3
### Versions of Apache Airflow Providers
- Not relevant -
### Deployment
Other
### Deployment details
No airflow - this is just a development environment.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20848: Static check in Breeze2
potiuk commented on a change in pull request #20848:
URL: https://github.com/apache/airflow/pull/20848#discussion_r793829156
##
File path: dev/breeze/src/airflow_breeze/breeze.py
##
@@ -312,6 +313,47 @@ def build_docs(verbose: bool, docs_only: bool,
spellcheck_only: bool, package_fi
build_documentation.build(verbose, mount_all_flag, airflow_sources,
ci_image_name, doc_builder)
+@option_verbose
+@main.command(
+name="static-check",
+context_settings=dict(
+ignore_unknown_options=True,
+allow_extra_args=True,
+),
+)
+@click.option('--all-files', is_flag=True)
+@click.option('--show-diff-on-failure', is_flag=True)
+@click.option('--last-commit', is_flag=True)
+@click.option('-t', '--type', type=click.Choice(PRE_COMMIT_LIST),
multiple=True)
+@click.option('--files', is_flag=True)
+@click.argument('precommit_args', nargs=-1, type=click.UNPROCESSED)
+def static_check(
+verbose: bool,
+all_files: bool,
+show_diff_on_failure: bool,
+last_commit: bool,
+type: Tuple[str],
+files: bool,
+precommit_args: Tuple,
+):
+if check_package_installed('pre_commit'):
+command_to_execute = ['pre-commit', 'run']
+for single_check in type:
+command_to_execute.append(single_check)
+# command_to_execute.append(EXTRA_STATIC_CHECK_OPTIONS)
Review comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #20848: Static check in Breeze2
potiuk commented on pull request #20848: URL: https://github.com/apache/airflow/pull/20848#issuecomment-1023454736 > @potiuk I think we can include tests for this PR with multiple ways of running the static code checks via breeze so that we can ensure that this code is working for multiple combinations of test conditions and make sure change in code doesn't break the static-check tests. I am not sure about how to implement it though. Could you share your views? This code will actually eventually be run in CI, so I think this does not "really" need more unit testing. I think we are pretty much good as it is now :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20848: Static check in Breeze2
github-actions[bot] commented on pull request #20848: URL: https://github.com/apache/airflow/pull/20848#issuecomment-1023454334 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] josh-fell opened a new pull request #21162: Change logging level details of connection info in `get_connection()`
josh-fell opened a new pull request #21162: URL: https://github.com/apache/airflow/pull/21162 Related: #19883 Currently task logs can contain all of connection details depending on how the associated connection to the task is configured (i.e. if `host` is a provided connection attr). These details are logged at the INFO level but seem more appropriate for debugging. This PR intends to clean up this connection logging a little. The INFO level logging will contain only the connection ID that is used while the details of the connection are changed to the DEBUG level (and still masked). Additionally the connection ID info is logged regardless of the provided connection attrs (i.e. removing the `host` check). Lastly this change also has a small added benefit of not accidentally or unknowingly exposing connection info that users do not want in their logs _first_ rather than the details be exposed and then having to setup configuration to mask them later (assuming the exposure is noticed at all). --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20848: Static check in Breeze2
potiuk commented on a change in pull request #20848:
URL: https://github.com/apache/airflow/pull/20848#discussion_r793826642
##
File path: dev/breeze/src/airflow_breeze/breeze.py
##
@@ -312,6 +313,47 @@ def build_docs(verbose: bool, docs_only: bool,
spellcheck_only: bool, package_fi
build_documentation.build(verbose, mount_all_flag, airflow_sources,
ci_image_name, doc_builder)
+@option_verbose
+@main.command(
+name="static-check",
+context_settings=dict(
+ignore_unknown_options=True,
+allow_extra_args=True,
+),
+)
+@click.option('--all-files', is_flag=True)
+@click.option('--show-diff-on-failure', is_flag=True)
+@click.option('--last-commit', is_flag=True)
+@click.option('-t', '--type', type=click.Choice(PRE_COMMIT_LIST),
multiple=True)
+@click.option('--files', is_flag=True)
+@click.argument('precommit_args', nargs=-1, type=click.UNPROCESSED)
+def static_check(
+verbose: bool,
+all_files: bool,
+show_diff_on_failure: bool,
+last_commit: bool,
+type: Tuple[str],
+files: bool,
+precommit_args: Tuple,
+):
+if check_package_installed('pre_commit'):
+command_to_execute = ['pre-commit', 'run']
+for single_check in type:
+command_to_execute.append(single_check)
+# command_to_execute.append(EXTRA_STATIC_CHECK_OPTIONS)
Review comment:
Only when passed as flag. It is really useful in CI but less useful when
running it locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] edithturn opened a new pull request #21161: added selective ci checks py
edithturn opened a new pull request #21161:
URL: https://github.com/apache/airflow/pull/21161
**Rewrite selective check script in Python**
01. .github/workflows/build-images.yml and .github/workflows/ci.yaml they
are calling the step: **Selective checks**
airflow/scripts/ci/selective_ci_checks.sh wich we are writing in python
- name: Selective checks
id: selective-checks
env:
PR_LABELS: ${{
steps.get-latest-pr-labels.outputs.pullRequestLabels }}
run: |
if [[ ${GITHUB_EVENT_NAME} == "pull_request_target" ]]; then
# Run selective checks
./scripts/ci/selective_ci_checks.sh "${TARGET_COMMIT_SHA}"
else
# Run all checks
./scripts/ci/selective_ci_checks.sh
fi
Close: #19971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21160: Table View
boring-cyborg[bot] commented on issue #21160: URL: https://github.com/apache/airflow/issues/21160#issuecomment-1023437883 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] alexhallam opened a new issue #21160: Table View
alexhallam opened a new issue #21160: URL: https://github.com/apache/airflow/issues/21160 ### Description One of the main things we want to know is task durations. The main way I know how to get this is to hover over the `Gannt` chart or use the `Task Duration` chart. It would be nice to have the data that is in the `Gannt` chart or the `Task Duration` chat in a table view. ### Use case/motivation That plots are making the data harder to find than a table alone. ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch main updated (0ac3b8c -> 362f397)
This is an automated email from the ASF dual-hosted git repository. jedcunningham pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from 0ac3b8c Return slack api call response in slack_hook (#21107) add 362f397 Update `version_added` for `[email] from_email` (#21138) No new revisions were added by this update. Summary of changes: airflow/config_templates/config.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[GitHub] [airflow] jedcunningham merged pull request #21138: Update `version_added` for `[email] from_email`
jedcunningham merged pull request #21138: URL: https://github.com/apache/airflow/pull/21138 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #20847: Added other Opentelemery Instrumentations
potiuk commented on a change in pull request #20847: URL: https://github.com/apache/airflow/pull/20847#discussion_r793799448 ## File path: airflow/www/gunicorn_config.py ## @@ -37,3 +42,18 @@ def on_starting(server): # Load providers before forking workers ProvidersManager().connection_form_widgets + Review comment: Yes. Something like that.should work I think. Needs testing though - for example you could `raise Exception()` in the "if" part and then, if the variable is properly set and passed to the gunicorn, all gunicorns would fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21150: Align Black and blacken-docs config
github-actions[bot] commented on pull request #21150: URL: https://github.com/apache/airflow/pull/21150#issuecomment-1023419779 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21149: Rewrite Airflow documentation home page
potiuk commented on pull request #21149: URL: https://github.com/apache/airflow/pull/21149#issuecomment-1023412406 I see. I love where it is going though :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] zambadruzaman commented on issue #21059: Dag dependency view is not rendering for Postgres backed Airflow
zambadruzaman commented on issue #21059: URL: https://github.com/apache/airflow/issues/21059#issuecomment-1023404868 Hi @lidalei, yes it was the exact same issue, when you delete any dags, make sure there is no other dag depends on the deleted dag. If you have a dag which has external trigger or sensor pointing to the deleted dag, the dependency graph will be broken and the [dagre-d3](https://github.com/dagrejs/dagre-d3) js won't able to render the graph. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #20794: Helm Chart - support for priorityClassName
subkanthi commented on a change in pull request #20794:
URL: https://github.com/apache/airflow/pull/20794#discussion_r793759521
##
File path: chart/templates/pgbouncer/pgbouncer-deployment.yaml
##
@@ -59,6 +59,7 @@ spec:
checksum/pgbouncer-config-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-config-secret.yaml") . | sha256sum }}
checksum/pgbouncer-certificates-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-certificates-secret.yaml") . |
sha256sum }}
spec:
+ priorityClassName: {{ .Values.pgbouncer.priorityClassName }}
Review comment:
sure, will make those changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] subkanthi commented on a change in pull request #20794: Helm Chart - support for priorityClassName
subkanthi commented on a change in pull request #20794:
URL: https://github.com/apache/airflow/pull/20794#discussion_r793759521
##
File path: chart/templates/pgbouncer/pgbouncer-deployment.yaml
##
@@ -59,6 +59,7 @@ spec:
checksum/pgbouncer-config-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-config-secret.yaml") . | sha256sum }}
checksum/pgbouncer-certificates-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-certificates-secret.yaml") . |
sha256sum }}
spec:
+ priorityClassName: {{ .Values.pgbouncer.priorityClassName }}
Review comment:
sure, not done yet, will make those changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21159: [SQSSensor] Add opt-in to disable auto-delete messages on consumption
boring-cyborg[bot] commented on pull request #21159: URL: https://github.com/apache/airflow/pull/21159#issuecomment-1023366571 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] LaPetiteSouris opened a new pull request #21159: [SQSSensor] Add opt-in to disable auto-delete messages on consumption
LaPetiteSouris opened a new pull request #21159: URL: https://github.com/apache/airflow/pull/21159 ## What Add opt-in in `SQSSensor` to disable the behavior of automatically delete messages on consumption. This option is opt-in. By default it is set to `False`, which means the original feature is preserved: `SQS messages are deleted automatically from the queue upon consumption` ## Why In many cases, the `SQSSensor` itselft is not the part of the code base that actually processes the messages. It serves as a short of polling to get messages out for further processing. If the actually code/Aiflow task that processes the messages failed, the message should be put back in the queue for further processing. This is the very basic idea of SQS Polling provided by Amazon. Reference to SQS with AWS Lambda function, [AWS docs](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) states : ``` When your function *successfully* processes a batch, Lambda deletes its messages from the queue. ``` In many case, the `SQSSensor` polls the message and distributes the workload to different task, in a very similar way to AWS Lambda polling.  Thus, there should be option to disable to default behavior, which is to delete the message upon reception. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #20962: Support generating SQL script for upgrades
ashb commented on a change in pull request #20962:
URL: https://github.com/apache/airflow/pull/20962#discussion_r793756208
##
File path: airflow/utils/db.py
##
@@ -1018,8 +1018,42 @@ def upgradedb(session: Session = NEW_SESSION):
exit(1)
with create_global_lock(session=session, lock=DBLocks.MIGRATIONS):
-log.info("Creating tables")
-command.upgrade(config, 'heads')
+log.info("Running migrations")
+rev_2_0_0_head = 'e959f08ac86c'
+if ':' in revision:
+sql = True
+_lowerband, _upperband = revision.split(':')
+head_to_lowerband_range = f"{rev_2_0_0_head}:{_lowerband}"
+head_to_upperband_range = f"{rev_2_0_0_head}:{_upperband}"
+for i in [head_to_lowerband_range, head_to_upperband_range]:
+try:
+command.history(config, rev_range=i)
+except Exception:
+raise AirflowException(
+f"Error while checking history for revision range {i}.
"
+f"Check that {i.split(':')[1]} is a valid revision. "
+f"Supported revision for offline migration is from
{rev_2_0_0_head} "
+f"which is airflow 2.0.0 head"
+)
+log.info("Running offline migrations for revision range %s",
revision)
+elif sql:
+# user supplied a specific SQL revision
+revision = f"{rev_2_0_0_head}:{revision}"
+try:
+command.history(config, rev_range=revision)
Review comment:
Doesn't this produce output? Is that intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21157: taskinstance: set task on sqlalchemy taskinstance object
github-actions[bot] commented on pull request #21157: URL: https://github.com/apache/airflow/pull/21157#issuecomment-1023362633 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] takuti opened a new pull request #21158: Fix documentation errors in apache-airflow/lineage.rst
takuti opened a new pull request #21158: URL: https://github.com/apache/airflow/pull/21158 Noticed two errors in the [Lineage documentation page](https://airflow.apache.org/docs/apache-airflow/stable/lineage.html): - Inline code highlight "task_id=\`\`run_me_first\`\`" doesn't work due to a missing whitespace. - Sample name of custom lineage backend, `ExampleBackend` , is different from the one configured above - `CustomBackend` --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #21158: Fix documentation errors in apache-airflow/lineage.rst
boring-cyborg[bot] commented on pull request #21158: URL: https://github.com/apache/airflow/pull/21158#issuecomment-1023362232 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst) Here are some useful points: - Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that. - In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it. - Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations. - Be patient and persistent. It might take some time to get a review or get the final approval from Committers. - Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack. - Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices). Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: d...@airflow.apache.org Slack: https://s.apache.org/airflow-slack -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mobuchowski commented on pull request #21157: taskinstance: set task on sqlalchemy taskinstance object
mobuchowski commented on pull request #21157: URL: https://github.com/apache/airflow/pull/21157#issuecomment-1023355726 cc @ashb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mobuchowski opened a new pull request #21157: taskinstance: set task on sqlalchemy taskinstance object
mobuchowski opened a new pull request #21157: URL: https://github.com/apache/airflow/pull/21157 In https://github.com/apache/airflow/pull/20443 we added TaskListener API. It's contract promises to pass TaskInstance object to listener plugin. However, what happens is not 100% true - the object being passed is one that maps to current SQLAlchemy session. `check_and_change_state_before_execution` operates on detached TaskInstance object, then merges it to current session. Since there is no attached object in the SQLAlchemy identity map, SQLAlchemy creates it, and it's this object that's being passed to the SQLAlchemy event listeners. The problem with that is that when creating new SQLAlchemy object, SQLAlchemy takes care about setting only database-mapped fields. The ones that are purely on the python side, like `task` aren't being set on the new object. This PR manually sets `task` on the new SQLAlchemy object, so that `on_task_instance_running` receives proper TaskInstance with `task` field set. Signed-off-by: Maciej Obuchowski -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] rsg17 commented on pull request #21084: [wip] Upload data from GCS to Presto
rsg17 commented on pull request #21084: URL: https://github.com/apache/airflow/pull/21084#issuecomment-1023353126 @eladkal : yes! Thank you for checking. Will do it this weekend.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #20794: Helm Chart - support for priorityClassName
jedcunningham commented on a change in pull request #20794:
URL: https://github.com/apache/airflow/pull/20794#discussion_r793733937
##
File path: chart/templates/pgbouncer/pgbouncer-deployment.yaml
##
@@ -59,6 +59,7 @@ spec:
checksum/pgbouncer-config-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-config-secret.yaml") . | sha256sum }}
checksum/pgbouncer-certificates-secret: {{ include (print
$.Template.BasePath "/secrets/pgbouncer-certificates-secret.yaml") . |
sha256sum }}
spec:
+ priorityClassName: {{ .Values.pgbouncer.priorityClassName }}
Review comment:
I meant all instances of it should be conditional, not just flower :)
##
File path: chart/values.yaml
##
@@ -514,6 +514,7 @@ workers:
# Select certain nodes for airflow worker pods.
nodeSelector: {}
+ priorityClassName: ""
Review comment:
You just need to change the type in the schema, e.g:
https://github.com/apache/airflow/blob/0ac3b8c3dd749c59e60cf0169580b9e7c5049d9e/chart/values.schema.json#L431-L434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21156: Snowflake connection default value of extras insecure_mode set as string instead of boolean, throws error in Snowflake hook
boring-cyborg[bot] commented on issue #21156: URL: https://github.com/apache/airflow/issues/21156#issuecomment-1023338370 Thanks for opening your first issue here! Be sure to follow the issue template! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] gregdorval-glf opened a new issue #21156: Snowflake connection default value of extras insecure_mode set as string instead of boolean, throws error in Snowflake hook
gregdorval-glf opened a new issue #21156:
URL: https://github.com/apache/airflow/issues/21156
### Apache Airflow Provider(s)
snowflake
### Versions of Apache Airflow Providers
apache-airflow-providers-snowflake | 2.4.0
### Apache Airflow version
2.2.3 (latest released)
### Operating System
Ubuntu 20.04
### Deployment
Virtualenv installation
### Deployment details
Python 3.8.10
alembic==1.7.5
antiorm==1.2.1
anyio==3.5.0
apache-airflow==2.2.3
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.2
apache-airflow-providers-imap==2.1.0
apache-airflow-providers-snowflake==2.4.0
apache-airflow-providers-sqlite==2.0.1
apispec==3.3.2
argcomplete==1.12.3
asn1crypto==1.4.0
attrs==20.3.0
Babel==2.9.1
blinker==1.4
cattrs==1.5.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.10
click==7.1.2
clickclick==20.10.2
colorama==0.4.4
colorlog==5.0.1
commonmark==0.9.1
croniter==1.0.15
cryptography==3.4.8
db==0.1.1
db-sqlite3==0.0.1
defusedxml==0.7.1
dill==0.3.4
dnspython==2.2.0
docutils==0.16
email-validator==1.1.3
Flask==1.1.4
Flask-AppBuilder==3.4.3
Flask-Babel==2.0.0
Flask-Caching==1.10.1
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OpenID==1.3.0
Flask-SQLAlchemy==2.5.1
Flask-WTF==0.14.3
graphviz==0.19.1
gunicorn==20.1.0
h11==0.12.0
httpcore==0.13.7
httpx==0.19.0
idna==3.3
importlib-metadata==4.10.1
importlib-resources==5.4.0
inflection==0.5.1
iso8601==1.0.2
isodate==0.6.1
itsdangerous==1.1.0
Jinja2==2.11.3
jsonschema==3.2.0
lazy-object-proxy==1.7.1
lockfile==0.12.2
Mako==1.1.6
Markdown==3.3.6
MarkupSafe==2.0.1
marshmallow==3.14.1
marshmallow-enum==1.5.1
marshmallow-oneofschema==3.0.1
marshmallow-sqlalchemy==0.26.1
openapi-schema-validator==0.1.6
openapi-spec-validator==0.3.2
oscrypto==1.2.1
packaging==21.3
pendulum==2.1.2
pkg_resources==0.0.0
prison==0.2.1
psutil==5.9.0
pycparser==2.21
pycryptodomex==3.12.0
Pygments==2.11.2
PyJWT==1.7.1
pyOpenSSL==20.0.1
pyparsing==3.0.7
pyrsistent==0.16.1
python-daemon==2.3.0
python-dateutil==2.8.2
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2021.3
pytzdata==2020.1
PyYAML==6.0
requests==2.27.1
rfc3986==1.5.0
rich==11.0.0
setproctitle==1.2.2
six==1.16.0
sniffio==1.2.0
snowflake-connector-python==2.7.1
snowflake-sqlalchemy==1.2.4
SQLAlchemy==1.3.24
SQLAlchemy-JSONField==1.0.0
SQLAlchemy-Utils==0.38.2
swagger-ui-bundle==0.0.9
tabulate==0.8.9
tenacity==8.0.1
termcolor==1.1.0
text-unidecode==1.3
unicodecsv==0.14.1
urllib3==1.26.8
Werkzeug==1.0.1
WTForms==2.3.3
zipp==3.7.0
### What happened
After creating a new Snowflake connection in the Airflow web portal, the
Extras parameter is automatically set to the boolean value false (see example
below).
{"extra__snowflake__account": "***", "extra__snowflake__database": "",
"extra__snowflake__insecure_mode": false, "extra__snowflake__region":
"canada-central.azure", "extra__snowflake__role": "",
"extra__snowflake__warehouse": "*"}
When running or testing a dag, an error is thrown in
SnowflakeHook._get_conn_params when calling airflow.utils.strings.to_boolean on
the bool value while evaluating insecure_mode. See end of stack trace below.
conn_config = self._get_conn_params()
File
"/home/gdorval/airflow/lib/python3.8/site-packages/airflow/providers/snowflake/hooks/snowflake.py",
line 170, in _get_conn_params
insecure_mode = to_boolean(
File
"/home/gdorval/airflow/lib/python3.8/site-packages/airflow/utils/strings.py",
line 30, in to_boolean
return False if astring is None else astring.lower() in ['true', 't',
'y', 'yes', '1']
AttributeError: 'bool' object has no attribute 'lower'
### What you expected to happen
SnowflakeHook._get_conn_params should expect a boolean value for
"extra__snowflake__insecure_mode". There is no need to convert it from a
string to boolean.
### How to reproduce
1. Create a new Snowflake connection.
2. Create a simple dag with one task using the snowflake connection (example
below).
3. Test the task
```
query_test = """select current_date() as date;"""
with DAG(
'example_snowflake',
start_date=datetime(2021, 1, 1),
default_args=DEFAULT_ARGS,
tags=['snowflake'],
catchup=False,
) as dag:
snowflake_op_sql_str = SnowflakeOperator(
task_id='snowflake_op_sql_str',
dag=dag,
sql=query_test,
snowflake_conn_id=SNOWFLAKE_CONN_ID,
warehouse=SNOWFLAKE_WAREHOUSE,
database=SNOWFLAKE_DATABASE,
schema=SNOWFLAKE_SCHEMA,
[GitHub] [airflow] lidalei commented on issue #21059: Dag dependency view is not rendering for Postgres backed Airflow
lidalei commented on issue #21059: URL: https://github.com/apache/airflow/issues/21059#issuecomment-1023334116 > > Closed this issue, the problem is not due to any bug, it's configuration issue on my side. Hi @zambadruzaman, could u share which configuration issue? We had the same problem after deleting some dags. The browser shows error ``` Uncaught TypeError: Cannot set properties of undefined (setting 'label') at dagre-d3.min.js:103:1930 at Array.forEach () at preProcessGraph (dagre-d3.min.js:103:1833) at Array.fn (dagre-d3.min.js:103:453) at Array.Co.call (d3.min.js:3:15178) at renderGraph (dagDependencies.bd18a52029a9cc9bbe0d.js:265:12) at Object.9 (dagDependencies.bd18a52029a9cc9bbe0d.js:288:1) at __webpack_require__ (dagDependencies.bd18a52029a9cc9bbe0d.js:30:30) at dagDependencies.bd18a52029a9cc9bbe0d.js:94:18 at dagDependencies.bd18a52029a9cc9bbe0d.js:97:10 DevTools failed to load source map: Could not load content for http://airflow.datadott.int/static/appbuilder/css/bootstrap.min.css.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj opened a new pull request #21155: Fix #21096: Support boolean in extra__snowflake__insecure_mode
mik-laj opened a new pull request #21155: URL: https://github.com/apache/airflow/pull/21155 closes: #21096 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #21145: enter the shell breeze2 environment
Bowrna commented on pull request #21145: URL: https://github.com/apache/airflow/pull/21145#issuecomment-1023245481 > > @potiuk i can see several env variables in scripts/ci/libraries/_initialization.sh. > > We definitely should not try to set all of those. They are needed by different parts of "old breeze". The variables that we should pass are exactly those which are "generated" in the "dc_ci" script - just take a look what you have in `./build/dc_ci` and those are the only environment variables that are really needed to run the CI. Another place (we have a few of those) that you can reconcile are in `scripts/ci/docker-compose/_docker.env` and the same variables are repeated in `scripts/ci/docker-compose/base.yaml` - maybe there are some slight differences we will have to reconcile. > > BTW. The reason why we have it in many places is that there are some difference in how "docker-compose 1" and "docker-compose 2" work. This will likely go away soon (Docker-compose 2 has been introduced recently and I think we will soon be able to deprecate docker-compose 1) at which time we will be able to remove one of those lists. > > I think (can't remember) I have a pre-commit that keeps those in-check - but maybe I have not done that, so better to check and reconcile it now. > > Regardless-having yet another list of the variables in python in "global_constants" that will be the "source of truth" would be great. @potiuk ok, I will check the vars in dc_ci script. Also you have mentioned about reconciling vars in `scripts/ci/docker-compose/_docker.env` and `scripts/ci/docker-compose/base.yaml`. I will check that part to understand more about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Melodie97 closed pull request #21154: Melodie97 docsupdate
Melodie97 closed pull request #21154: URL: https://github.com/apache/airflow/pull/21154 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #21145: enter the shell breeze2 environment
potiuk commented on pull request #21145: URL: https://github.com/apache/airflow/pull/21145#issuecomment-1023202231 > @potiuk i can see several env variables in scripts/ci/libraries/_initialization.sh. We definitely should not try to set all of those. They are needed by different parts of "old breeze". The variables that we should pass are exactly those which are "generated" in the "dc_ci" script - just take a look what you have in `./build/dc_ci` and those are the only environment variables that are really needed to run the CI. Another place (we have a few of those) that you can reconcile are in `scripts/ci/docker-compose/_docker.env` and the same variables are repeated in `scripts/ci/docker-compose/base.yaml` - maybe there are some slight differences we will have to reconcile. BTW. The reason why we have it in many places is that there are some difference in how "docker-compose 1" and "docker-compose 2" work. This will likely go away soon (Docker-compose 2 has been introduced recently and I think we will soon be able to deprecate docker-compose 1) at which time we will be able to remove one of those lists. I think (can't remember) I have a pre-commit that keeps those in-check - but maybe I have not done that, so better to check and reconcile it now. Regardless-having yet another list of the variables in python in "global_constants" that will be the "source of truth" would be great. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
