[ https://issues.apache.org/jira/browse/AIRFLOW-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023800#comment-17023800 ]
ASF subversion and git services commented on AIRFLOW-6522: ---------------------------------------------------------- Commit a9190f7b71fd26daf8565ed6aae4107270248469 in airflow's branch refs/heads/v1-10-test from rconroy293 [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a9190f7 ] [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120) The same task instance (including try number) can be run on a worker when using a sensor in "reschedule" mode. Accordingly, this clears the local log file when re-initializing the logger so that the old log lines aren't uploaded again when the logger is closed. (cherry picked from commit 88608caa56bf3621807af860a6a378242220de47) > Sensors in reschedule mode with S3TaskHandler can cause log duplication > ----------------------------------------------------------------------- > > Key: AIRFLOW-6522 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6522 > Project: Apache Airflow > Issue Type: Bug > Components: logging > Affects Versions: 1.10.6 > Reporter: Robert Conroy > Assignee: Robert Conroy > Priority: Minor > Fix For: 1.10.8 > > > With sensors using {{reschedule}} mode and {{S3TaskHandler}} for logging, the > task instance log gets a bunch of duplicate messages. I believe this is > happening because contents of the local log file are appended to what's > already in S3. The local log file may contain log messages that have already > been uploaded to S3 if the task is sent back to a worker that had already > processed a poke for that task instance. -- This message was sent by Atlassian Jira (v8.3.4#803005)