kaxil commented on PR #50300: URL: https://github.com/apache/airflow/pull/50300#issuecomment-2867788090
The "in-process" in the URL does mean that somehow the `use_executor` flag isn't passed True. As an example, this is what you should see <img width="1728" alt="image" src="https://github.com/user-attachments/assets/6d0cfa34-6f71-4a35-8e2a-d27dfb25de35"; /> `dag.test` logs: ``` root@e2e45b18eab2:/opt/airflow# python /files/dags/example_simplest_dag.py [2025-05-09T20:11:07.373+0000] {taskinstance.py:291} WARNING - No serialized dag found for dag 'xample_simplest_dag' [2025-05-09T20:11:08.451+0000] {dag.py:2306} INFO - created dagrun <DagRun xample_simplest_dag @ None: manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF, state:running, queued_at: None. run_type: manual> [2025-05-09T20:11:08.453+0000] {executor_loader.py:269} INFO - Loaded executor: :LocalExecutor: [2025-05-09T20:11:08.487+0000] {local_executor.py:61} INFO - Worker starting up pid=1345 [2025-05-09T20:11:08.519+0000] {dag.py:1159} WARNING - No tasks to run. unrunnable tasks: {<TaskInstance: xample_simplest_dag.my_task manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF [queued]>} 2025-05-09 20:11:08 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1 [2025-05-09T20:11:08.952+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/run "HTTP/1.1 200 OK" 2025-05-09 20:11:08 [debug ] Sending [supervisor] msg=StartupDetails(ti=TaskInstance(id=UUID('0196b6ac-5123-7413-b995-96d80d1a2c58'), task_id='my_task', dag_id='xample_simplest_dag', run_id='manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config={}, parent_context_carrier={}, context_carrier=None, queued_dttm=None), dag_rel_path='/files/dags/example_simplest_dag.py', bundle_info=BundleInfo(name='dags-folder', version=None), requests_fd=23, start_date=datetime.datetime(2025, 5, 9, 20, 11, 8, 567906, tzinfo=datetime.timezone.utc), ti_context=TIRunContext(dag_run=DagRun(dag_id='xample_simplest_dag', run_id='manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF', logical_date=None, data_interval_start=None, data_interval_end=None, run_after=datetime.datetime(2025, 5, 9, 20, 11, 7, 394963, tzinfo=TzInfo(UTC)), start_date=datetime.datetime(2025, 5, 9, 20, 11, 7, 394963, tzinf o=TzInfo(UTC)), end_date=None, clear_number=0, run_type=<DagRunType.MANUAL: 'manual'>, conf={}, consumed_asset_events=[]), task_reschedule_count=0, max_tries=0, variables=[], connections=[], upstream_map_indexes=None, next_method=None, next_kwargs=None, xcom_keys_to_clear=[], should_retry=False), type='StartupDetails') [2025-05-09T20:11:09.565+0000] {dag.py:1159} WARNING - No tasks to run. unrunnable tasks: {<TaskInstance: xample_simplest_dag.my_task manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF [running]>} 2025-05-09 20:11:09 [debug ] Received message from task runner [supervisor] msg=SetRenderedFields(rendered_fields={'templates_dict': None, 'op_args': [], 'op_kwargs': {}}, type='SetRenderedFields') [2025-05-09T20:11:10.605+0000] {dag.py:1159} WARNING - No tasks to run. unrunnable tasks: {<TaskInstance: xample_simplest_dag.my_task manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF [running]>} [2025-05-09T20:11:10.714+0000] {_client.py:1026} INFO - HTTP Request: PUT http://localhost:8080/execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/rtif "HTTP/1.1 201 Created" 2025-05-09 20:11:10 [debug ] Received message from task runner [supervisor] msg=SucceedTask(state='success', end_date=datetime.datetime(2025, 5, 9, 20, 11, 8, 988171, tzinfo=TzInfo(UTC)), task_outlets=[], outlet_events=[], rendered_map_index=None, type='SucceedTask') [2025-05-09T20:11:10.727+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/state "HTTP/1.1 204 No Content" 2025-05-09 20:11:10 [info ] Task finished [supervisor] duration=2.1683323599863797 exit_code=0 final_state=success [2025-05-09T20:11:11.611+0000] {scheduler_job_runner.py:810} INFO - Received executor event with state success for task instance TaskInstanceKey(dag_id='xample_simplest_dag', task_id='my_task', run_id='manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF', try_number=1, map_index=-1) [2025-05-09T20:11:11.644+0000] {scheduler_job_runner.py:852} INFO - TaskInstance Finished: dag_id=xample_simplest_dag, task_id=my_task, run_id=manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF, map_index=-1, run_start_date=2025-05-09 20:11:08.567906+00:00, run_end_date=None, run_duration=None, state=running, executor=LocalExecutor(parallelism=32), executor_state=success, try_number=1, max_tries=0, pool=default_pool, queue=default, priority_weight=1, operator=_PythonDecoratedOperator, queued_dttm=None, scheduled_dttm=2025-05-09 20:11:08.460021+00:00,queued_by_job_id=None, pid=1346 [2025-05-09T20:11:11.673+0000] {dagrun.py:1176} INFO - Marking run <DagRun xample_simplest_dag @ None: manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF, state:running, queued_at: None. run_type: manual> successful Dag run in success state Dag run start:2025-05-09 20:11:07.394963+00:00 end:2025-05-09 20:11:11.673977+00:00 [2025-05-09T20:11:11.679+0000] {dagrun.py:1226} INFO - DagRun Finished: dag_id=xample_simplest_dag, logical_date=None, run_id=manual__2025-05-09T20:11:07.394963+00:00_3s9xiRMF, run_start_date=2025-05-09 20:11:07.394963+00:00, run_end_date=2025-05-09 20:11:11.673977+00:00, run_duration=4.279014, state=success, run_type=manual, data_interval_start=None, data_interval_end=None, [2025-05-09T20:11:11.699+0000] {local_executor.py:223} INFO - Shutting down LocalExecutor; waiting for running tasks to finish. Signal again if you don't want to wait. ``` API Server logs: <img width="1332" alt="image" src="https://github.com/user-attachments/assets/a21af5fa-35de-4526-ba8d-11a13c7eb8e2"; /> Logs: ``` [2025-05-09T20:10:56.842+0000] {api_server_command.py:92} INFO - Running the uvicorn with: Apps: all Workers: 4 Host: 0.0.0.0:8080 Timeout: 120 Logfiles: - ================================================================= INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) INFO: Started parent process [1264] INFO: Started server process [1267] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Started server process [1270] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Started server process [1268] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Started server process [1269] INFO: Waiting for application startup. INFO: Application startup complete. 2025-05-09 20:11:08 [debug ] Starting task instance run hostname=e2e45b18eab2 pid=1346 ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 unixname=root 2025-05-09 20:11:08 [debug ] Retrieved task instance details dag_id=xample_simplest_dag state=queued task_id=my_task ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 2025-05-09 20:11:08 [info ] Task started hostname=e2e45b18eab2 previous_state=queued ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 2025-05-09 20:11:08 [info ] Task instance state updated rows_affected=1 ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 INFO: 127.0.0.1:51544 - "PATCH /execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/run HTTP/1.1" 200 OK 2025-05-09 20:11:09 [info ] Updating RenderedTaskInstanceFields field_count=3 ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 2025-05-09 20:11:10 [debug ] RenderedTaskInstanceFields updated successfully ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 INFO: 127.0.0.1:51544 - "PUT /execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/rtif HTTP/1.1" 201 Created 2025-05-09 20:11:10 [debug ] Updating task instance state new_state=success ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 2025-05-09 20:11:10 [debug ] Retrieved current task instance state max_tries=0 previous_state=running ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 try_number=1 2025-05-09 20:11:10 [info ] Task instance state updated new_state=success rows_affected=1 ti_id=0196b6ac-5123-7413-b995-96d80d1a2c58 INFO: 127.0.0.1:51544 - "PATCH /execution/task-instances/0196b6ac-5123-7413-b995-96d80d1a2c58/state HTTP/1.1" 204 No Content ``` Dag used: ```py from airflow.sdk import Variable, dag, task import os @dag def xample_simplest_dag(): @task def my_task(): print("hellooo") os.environ["AIRFLOW_VAR_MY_VARIABLE"] = "my_value" assert Variable.get("my_variable") == "my_value" my_task() d = xample_simplest_dag() if __name__ == "__main__": d.test(use_executor=True) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
