alvaroserper opened a new issue, #34845: URL: https://github.com/apache/airflow/issues/34845
### Apache Airflow version 2.7.1 ### What happened When running a dag an error ocurred. The error says that there is a metric with an invalid name. This causes that the task of the dag is set up for retry. Then the task executes again and is marked as success. `2023-10-10 13:05:21 [2023-10-10T11:05:21.738+0000] {local_executor.py:135} ERROR - Failed to execute task Invalid stat name: ***.dag.cwf_path_inspector_generator.delete-xcom-task.queued_duration. Please see https://opentelemetry.io/docs/reference/specification/metrics/api/#instrument-name-syntax. 2023-10-10 13:05:21 Traceback (most recent call last): 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/executors/local_executor.py", line 131, in _execute_work_in_fork 2023-10-10 13:05:21 args.func(args) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/cli_config.py", line 49, in command 2023-10-10 13:05:21 return func(*args, **kwargs) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/utils/cli.py", line 113, in wrapper 2023-10-10 13:05:21 return f(*args, **kwargs) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 430, in task_run 2023-10-10 13:05:21 task_return_code = _run_task_by_selected_method(args, _dag, ti) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 208, in _run_task_by_selected_method 2023-10-10 13:05:21 return _run_task_by_local_task_job(args, ti) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/cli/commands/task_command.py", line 270, in _run_task_by_local_task_job 2023-10-10 13:05:21 ret = run_job(job=job_runner.job, execute_callable=job_runner._execute) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/utils/session.py", line 77, in wrapper 2023-10-10 13:05:21 return func(*args, session=session, **kwargs) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/jobs/job.py", line 289, in run_job 2023-10-10 13:05:21 return execute_job(job, execute_callable=execute_callable) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/jobs/job.py", line 318, in execute_job 2023-10-10 13:05:21 ret = execute_callable() 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/jobs/local_task_job_runner.py", line 143, in _execute 2023-10-10 13:05:21 if not self.task_instance.check_and_change_state_before_execution( 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/utils/session.py", line 77, in wrapper 2023-10-10 13:05:21 return func(*args, session=session, **kwargs) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 1366, in check_and_change_state_before_execution 2023-10-10 13:05:21 self.emit_state_change_metric(TaskInstanceState.RUNNING) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/models/taskinstance.py", line 1450, in emit_state_change_metric 2023-10-10 13:05:21 Stats.timing(f"dag.{self.dag_id}.{self.task_id}.{metric_name}", timing) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/metrics/otel_logger.py", line 266, in timing 2023-10-10 13:05:21 if self.metrics_validator.test(stat) and name_is_otel_safe(self.prefix, stat): 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/metrics/otel_logger.py", line 95, in name_is_otel_safe 2023-10-10 13:05:21 return bool(stat_name_otel_handler(prefix, name, max_length=OTEL_NAME_MAX_LENGTH)) 2023-10-10 13:05:21 File "/usr/local/lib/python3.10/dist-packages/airflow/metrics/validators.py", line 142, in stat_name_otel_handler 2023-10-10 13:05:21 raise InvalidStatsNameException( 2023-10-10 13:05:21 airflow.exceptions.InvalidStatsNameException: Invalid stat name: ***.dag.cwf_path_inspector_generator.delete-xcom-task.queued_duration. Please see https://opentelemetry.io/docs/reference/specification/metrics/api/#instrument-name-syntax` ### What you think should happen instead There should not be an error with the name of a default metric causing a task to retry. ### How to reproduce Enable opentelemetry in airflow.cfg: `otel_on = True otel_host = breeze-otel-collector otel_port = 4318 otel_prefix = airflow otel_interval_milliseconds = 30000 # The interval between exports, defaults to 60000 otel_ssl_active = False` Run opentelemetry collector docker: `otel-collector: image: otel/opentelemetry-collector-contrib:0.70.0 container_name: "breeze-otel-collector" command: [--config=/etc/otel-collector-config.yml] volumes: - ./otel-collector/otel-collector-config.yml:/etc/otel-collector-config.yml # - ./otel-collector/keys:/etc/keys ports: - "24318:4318" # OTLP http receiver - "28889:8889" # Prometheus exporter metrics` ### Operating System Ubuntu 22.04.3 LTS ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org