ashb opened a new pull request, #46869:
URL: https://github.com/apache/airflow/pull/46869

   Fixes #45449
   
   To make this work (currently) you have to to do `from airflow.sdk import
   Variable` instead of the `airflow.models` version (that particular wart will 
be
   fixed in a broader back-compat issue.)
   
   Since I want to maintain the property of being able to run a DAG processor
   without the Execution API server running, and since the Dag Processor Manager
   already has a database connection I have chosen to run the FastAPI execution
   server in process.
   
   To achive this I make use of two features:
   
   - The first is the abilty to provide an httpx.Client with a Transport object
     that has an WSGI appliction to not send real requests, but to instead call
     the WSGI app directly to service the request
   - The second is a2wsgi. Since we are making a call from with in a synchronus
     context we have to give httpx a WSGI (if we were making Async requests we
     could give httpx an ASGI app directly), and FastAPI at it's outer layers is
     an async framework (even if it supports running sync routes) we need to
     somehow wrap the async call to return a sync result. a2wsgi does this for 
us
     by using a async loop off the main thread.
   
   I tested this with a simple DAG file initially:
   
   ```python
   import time
   import sys
   from airflow.decorators import dag, task
   from airflow.sdk import Variable
   
   from airflow.utils.session import create_session
   
   if Variable.get("hi", default=None):
       raise RuntimeError("Var hi was defined")
   
   @dag(schedule=None)
   def hi():
       @task()
       def hello():
           print("hello")
           time.sleep(3)
           print("goodbye")
           print("err mesg", file=sys.stderr)
   
       hello()
   
   hi()
   ```
   
   If the variable is defined it results in an import error. If the variable is
   not defined you get the DAG defined.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to