[
https://issues.apache.org/jira/browse/AIRFLOW-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592997#comment-15592997
]
Dan Davydov commented on AIRFLOW-385:
-------------------------------------
I agree something like this would be very useful, the problem is these logs
contain attempts to parse files for DAGs, so in some cases a DAG might not even
be created due to an error so you couldn't have dag_id/execution_date/...
There is probably a middle-ground, maybe where we have a duplicate log folder
for DAGs that have been successfully parsed with symlinks pointing back to the
original log directory, but this idea scares me (symlinks, and duplication).
Another idea would be a CLI tool like airflow view_scheduler_logs dag_id
task_id execution_date.
In any case I feel this issue is orthogonal to the "latest" symlink (as this
symlink would be useful even in the dag_id/task_id/execution_date model), but
definitely feel free to open another ticket for it.
> Scheduler logs should have a "latest" directory
> -----------------------------------------------
>
> Key: AIRFLOW-385
> URL: https://issues.apache.org/jira/browse/AIRFLOW-385
> Project: Apache Airflow
> Issue Type: Improvement
> Components: logging
> Reporter: Dan Davydov
> Assignee: Vijay Bhat
> Priority: Minor
> Labels: beginner, logging
>
> Right now the scheduler logs for each parsed file look like
> /tmp/airflow/scheduler/logs/2016-07-17/some-dag/...
> The problem is this makes it hard to find the latest logs which is the most
> frequent use case.
> Ideally we would create and have the scheduler keep up to date a symlink to
> the latest date /tmp/airflow/scheduler/logs/latest which would point to e.g.
> /tmp/airflow/scheduler/logs/2016-07-17
> We might also want to consider changing the structure to dag/date instead of
> date/dag too, but that can be done as a separate task if desired.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
