adh-wonolo commented on PR #32101: URL: https://github.com/apache/airflow/pull/32101#issuecomment-1609576651
@o-nikolas the executor loads and functions normally for everything other than backfills which you can see in the logs too, its just that when executing https://github.com/apache/airflow/blob/45bd9c993ee87789e01aae55f8abb34a0ff1de48/airflow/jobs/backfill_job_runner.py#L543-L549 All that's passed is the name of the Executor (in `NomadExecutor`) rather than the full path which the importer code can only resolve for core executors, hence why this could also be fixed by instead passing the path: `f"{self.job.executor.__class__.__module__}.{self.job.executor_class}"` to that function instead or setting that as the value for `self.job.executor_class` more directly. I'm storing the plugin in `./plugins/executor/nomad_executor.py` Here's the stacktrace: ``` $ airflow dags backfill clean_airflow_metadb --start-date 20230501 --end-date 20230612 --reset-dagruns /Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/cli/commands/dag_command.py:119 RemovedInAirflow3Warning: --ignore-first-depends-on-past is deprecated as the value is always set to True [2023-06-27T09:54:21.489-0400] {dagbag.py:541} INFO - Filling up the DagBag from /Users/adh/repos/internal-tools-airflow/dags You are about to delete these 2 tasks: <TaskInstance: clean_airflow_metadb.clean_xcoms backfill__2023-05-01T00:00:00+00:00 [queued]> <TaskInstance: clean_airflow_metadb.clean_xcoms backfill__2023-06-01T00:00:00+00:00 [scheduled]> Are you sure? [y/n] y [2023-06-27T09:54:29.538-0400] {executor_loader.py:114} INFO - Loaded executor: plugins.executor.nomad_executor.NomadExecutor /Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/plugins_manager.py:258 RemovedInAirflow3Warning: This decorator is deprecated. In previous versions, all subclasses of BaseOperator must use apply_default decorator for the `default_args` feature to work properly. In current version, it is optional. The decorator is applied automatically using the metaclass. 2023-06-27 09:54:29,667 - [bugsnag] WARNING - No API key configured, couldn't notify [2023-06-27T09:54:29.667-0400] {client.py:170} WARNING - No API key configured, couldn't notify Traceback (most recent call last): File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 32, in import_string module_path, class_name = dotted_path.rsplit(".", 1) ValueError: not enough values to unpack (expected 2, got 1) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/adh/.pyenv/versions/airflow310/bin/airflow", line 8, in <module> sys.exit(main()) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/__main__.py", line 48, in main args.func(args) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/cli/cli_config.py", line 52, in command return func(*args, **kwargs) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/cli.py", line 112, in wrapper return f(*args, **kwargs) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/cli/commands/dag_command.py", line 139, in dag_backfill _run_dag_backfill(dags, args) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/cli/commands/dag_command.py", line 92, in _run_dag_backfill dag.run( File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/models/dag.py", line 2490, in run run_job(job=job, execute_callable=job_runner._execute) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/session.py", line 76, in wrapper return func(*args, session=session, **kwargs) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/job.py", line 284, in run_job return execute_job(job, execute_callable=execute_callable) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/job.py", line 313, in execute_job ret = execute_callable() File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/session.py", line 76, in wrapper return func(*args, session=session, **kwargs) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/backfill_job_runner.py", line 914, in _execute self._execute_dagruns( File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/session.py", line 73, in wrapper return func(*args, **kwargs) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/backfill_job_runner.py", line 801, in _execute_dagruns processed_dag_run_dates = self._process_backfill_task_instances( File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/backfill_job_runner.py", line 643, in _process_backfill_task_instances _per_task_process(key, ti, session) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/jobs/backfill_job_runner.py", line 539, in _per_task_process executor_class, _ = ExecutorLoader.import_executor_cls( File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 148, in import_executor_cls return _import_and_validate(executor_name), ConnectorSource.CUSTOM_PATH File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/executors/executor_loader.py", line 129, in _import_and_validate executor = import_string(path) File "/Users/adh/.pyenv/versions/3.10.7/envs/airflow310/lib/python3.10/site-packages/airflow/utils/module_loading.py", line 34, in import_string raise ImportError(f"{dotted_path} doesn't look like a module path") ImportError: NomadExecutor doesn't look like a module path ``` Happy to answer any other questions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
