collinmcnulty opened a new issue #20991:
URL: https://github.com/apache/airflow/issues/20991
### Apache Airflow Provider(s)
elasticsearch
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon==1!2.5.0
apache-airflow-providers-cncf-kubernetes==1!2.1.0
apache-airflow-providers-datadog==1!2.0.1
apache-airflow-providers-elasticsearch==1!2.1.0
apache-airflow-providers-ftp==1!2.0.1
apache-airflow-providers-google==1!6.1.0
apache-airflow-providers-http==1!2.0.1
apache-airflow-providers-imap==1!2.0.1
apache-airflow-providers-microsoft-azure==1!3.3.0
apache-airflow-providers-mysql==1!2.1.1
apache-airflow-providers-postgres==1!2.3.0
apache-airflow-providers-redis==1!2.0.1
apache-airflow-providers-slack==1!4.1.0
apache-airflow-providers-sqlite==1!2.0.1
apache-airflow-providers-ssh==1!2.3.0
```
### Apache Airflow version
2.2.3 (latest released)
### Operating System
Debian Bullseye
### Deployment
Astronomer
### Deployment details
_No response_
### What happened
After upgrading to 2.2, task logs from manual dagruns performed before the
upgrade could no longer be retrieved, even though they can still be seen in
Kibana. Scheduled dagruns' tasks and tasks for dagruns begun after the upgrade
are retrieved without issue.
The issue appears to be because these tasks with missing logs all belong to
dagruns that do not have the attribute data_interval_start or data_interval_end
set.
### What you expected to happen
Task logs continue to be fetched after upgrade.
### How to reproduce
Below is how I verified the log fetching process.
I ran the code snippet in a python interpreter in the scheduler to test log
fetching.
```py
from airflow.models import TaskInstance, DagBag, DagRun
from airflow.settings import Session, DAGS_FOLDER
from airflow.configuration import conf
import logging
from dateutil import parser
logger = logging.getLogger('airflow.task')
task_log_reader = conf.get('logging', 'task_log_reader')
handler = next((handler for handler in logger.handlers if handler.name ==
task_log_reader), None)
dag_id = 'pipeline_nile_reconciliation'
task_id = 'nile_overcount_spend_resolution_task'
execution_date = parser.parse('2022-01-10T11:49:57.197933+00:00')
try_number=1
session = Session()
ti = session.query(TaskInstance).filter(
TaskInstance.dag_id == dag_id,
TaskInstance.task_id == task_id,
TaskInstance.execution_date == execution_date).first()
dagrun = session.query(DagRun).filter(
DagRun.dag_id == dag_id,
DagRun.execution_date == execution_date).first()
dagbag = DagBag(DAGS_FOLDER, read_dags_from_db=True)
dag = dagbag.get_dag(dag_id)
ti.task = dag.get_task(ti.task_id)
ti.dagrun = dagrun
handler.read(ti, try_number, {})
```
The following error log indicates errors in the log reading.
```
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/usr/local/lib/python3.9/site-packages/airflow/utils/log/file_task_handler.py",
line 239, in read
log, metadata = self._read(task_instance, try_number_element, metadata)
File
"/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py",
line 168, in _read
log_id = self._render_log_id(ti, try_number)
File
"/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py",
line 107, in _render_log_id
data_interval_start = self._clean_date(dag_run.data_interval_start)
File
"/usr/local/lib/python3.9/site-packages/airflow/providers/elasticsearch/log/es_task_handler.py",
line 134, in _clean_date
return value.strftime("%Y_%m_%dT%H_%M_%S_%f")
AttributeError: 'NoneType' object has no attribute 'strftime'
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
