Firstly, you can use other times other than UTC, you set the start date of the dag to a different time zone: https://airflow.apache.org/docs/stable/timezone.html#time-zone-aware-dags . Note that the UI and ExecutionDate will still be in UTC, if you need to get the ExecutionDate in your local timezone this is currently best achieved through Macros: https://stackoverflow.com/questions/44855949/make-custom-airflow-macros-expand-other-macros/54922776#54922776 . I run dozens of DAGs in about more than 5 different timezones this way.
Secondly the execution date of Triggered DAGs has a different logic to Scheduled DAGs (I’m crossing my fingers this inconsistency is resolved in Airflow 2.0 by getting rid of the weird Scheduled DAG logic): * A Triggered DAG has the execution date of when it is actual run or it’s given time to execute. For example say you set it to run at 2020-03-06 12:00:00 then that is it’s execution date. * A Scheduled DAG has an execution date that subscribes to the logic that the DAG is processing a “period” of data. So for example if you schedle it to run at midday every day then the run at 2020-03-06 12:00:00 considers that period of scheduling to be from 2020-03-05 12:00:00 (inclusive) to 2020-03-06 12:00:00 (not inclusive), it then gives you an execution date of 2020-03-05 12:00:00 As you can see even though their actual execution time is the same, their execution date is 1 day apart. A Triggered DAG can not provide the same logic as a Scheduled DAG because it has no schedule to determine the “period”. I have created Macros that gives me the same execution date regardless of whether I am using Scheduled or Triggered DAGs, you can use the above SO answer as a template if you want to do the same, and the thing to remember is that for a Scheduled DAG the next execution date is equivelent to the Triggered DAG execution date. Hope this helps Damian From: Reed Villanueva <[email protected]> Sent: Thursday, March 5, 2020 21:35 To: [email protected] Subject: Dag triggered via TriggerDagRunOperator logs start date as 1 day ahead Seeing a weird problem where when running a dag on cron schedule 0 5 1,3-31 * *<http://airflowetl.ucera.local:8080/admin/dagrun/?flt2_dag_id_equals=hph_clarity_etl_trigger> (note I am still using UTC time (since I do not believe that I can change it in v1.10.7)) that ends by running a TriggerDagRunOperator task. When the "triggering" dag runs, the Last Run date shows the correct date and is reflected in the run_id. Eg. 2020-03-04 05:00 <http://airflowetl.ucera.local:8080/admin/airflow/graph?dag_id=hph_clarity_etl_trigger&execution_date=2020-03-04+05%3A00%3A00%2B00%3A00> However, the dag that gets triggered shows the Last Run date as 1 day ahead and this unexpected behavior is also reflected in the triggered dag's run_id 2020-03-05 05:00 <http://airflowetl.ucera.local:8080/admin/airflow/graph?dag_id=hph_clarity_etl&execution_date=2020-03-05+05%3A00%3A47.975014%2B00%3A00> yet the triggering dag only takes about 1min. to run. Does anyone know what could be happening here? Is there a time difference that I am not recognizing here (note that I am not living in a UTC time zone (I am in HST and the cron is intended to run the triggering dag at 7:00PM each night excluding the 1st of each month))? This electronic message is intended only for the named recipient, and may contain information that is confidential or privileged. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of the contents of this message is strictly prohibited. If you have received this message in error or are not the named recipient, please notify us immediately by contacting the sender at the electronic mail address noted above, and delete and destroy all copies of this message. Thank you. =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
