It seems that this entire file needs to be patched so that the datetime.now() calls use tz=pytz.utc. At some point Python started including timezones in datetime.now() and so this is broken.
I think I can patch it. But one problem I am having is how do I see the log messages of the scheduler when I use for example: self.logger.error(job.latest_heartbeat) ? Russell Jurney @rjurney <http://twitter.com/rjurney> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB <http://facebook.com/jurney> datasyndrome.com On Tue, May 16, 2017 at 3:40 PM, Russell Jurney <russell.jur...@gmail.com> wrote: > I setup conda to run python 3.4 and I still get this when a job is > scheduled and runs: > > [2017-05-16 15:37:54,339] {jobs.py:1171} DagFileProcessor1 INFO - > Processing agl_p2p_api_worker_dag > [2017-05-16 15:37:55,601] {jobs.py:354} DagFileProcessor1 ERROR - Got an > exception! Propagating... > Traceback (most recent call last): > File "/Users/rjurney/Software/incubator-airflow/airflow/jobs.py", line > 346, in helper > pickle_dags) > File "/Users/rjurney/Software/incubator-airflow/airflow/utils/db.py", > line 48, in wrapper > result = func(*args, **kwargs) > File "/Users/rjurney/Software/incubator-airflow/airflow/jobs.py", line > 1584, in process_file > self._process_dags(dagbag, dags, ti_keys_to_schedule) > File "/Users/rjurney/Software/incubator-airflow/airflow/jobs.py", line > 1173, in _process_dags > dag_run = self.create_dag_run(dag) > File "/Users/rjurney/Software/incubator-airflow/airflow/utils/db.py", > line 48, in wrapper > result = func(*args, **kwargs) > File "/Users/rjurney/Software/incubator-airflow/airflow/jobs.py", line > 815, in create_dag_run > if next_run_date > datetime.now(): > TypeError: can't compare offset-naive and offset-aware datetimes > > > What do I do? > > Russell Jurney @rjurney <http://twitter.com/rjurney> > russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB > <http://facebook.com/jurney> datasyndrome.com > > On Tue, May 16, 2017 at 2:18 PM, Russell Jurney <russell.jur...@gmail.com> > wrote: > >> Thanks, we're trying that now! >> >> Russell Jurney @rjurney <http://twitter.com/rjurney> >> russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >> <http://facebook.com/jurney> datasyndrome.com >> >> On Tue, May 16, 2017 at 2:02 PM, Bolke de Bruin <bdbr...@gmail.com> >> wrote: >> >>> Did you try to run this on Py 2.7 / 3.4 as well? I notice you are >>> running on 3.6, which we are not testing against at the moment. >>> >>> Bolke. >>> >>> > On 16 May 2017, at 22:46, Russell Jurney <russell.jur...@gmail.com> >>> wrote: >>> > >>> > We have tasks that run, but we can't get them to run as scheduled or to >>> > un-pause. >>> > >>> > The code for the task looks like this: >>> > >>> > # Run the API worker every 5 minutes >>> > api_worker_dag = DAG( >>> > 'agl_p2p_api_worker_dag', >>> > default_args=default_args, >>> > schedule_interval=timedelta(minutes=5) >>> > ) >>> > >>> > # Run the API worker[ >>> > api_worker_task = BashOperator( >>> > task_id="api_worker_task", >>> > bash_command="""python {{ params.base_path >>> > }}/agl-p2p-api-worker/site/worker.py {{ ds }}""", >>> > params={ >>> > "base_path": project_home >>> > }, >>> > dag=api_worker_dag >>> > ) >>> > >>> > We run this command: airflow unpause agl_p2p_api_worker_dag >>> > >>> > And we see this error: >>> > >>> > [2017-05-16 20:26:48,722] {jobs.py:1408} INFO - Heartbeating the >>> process >>> > manager >>> > [2017-05-16 20:26:48,723] {dag_processing.py:559} INFO - Processor for >>> > /root/airflow/dags/setup.py finished >>> > [2017-05-16 20:26:48,723] {dag_processing.py:578} WARNING - Processor >>> for >>> > /root/airflow/dags/setup.py exited with return code 1. See >>> > /root/airflow/logs/scheduler/2017-05-16/setup.py.log for details. >>> > [2017-05-16 20:26:48,726] {dag_processing.py:627} INFO - Started a >>> process >>> > (PID: 110) to generate tasks for /root/airflow/dags/setup.py - logging >>> into >>> > /root/airflow/logs/scheduler/2017-05-16/setup.py.log >>> > [2017-05-16 20:26:48,727] {jobs.py:1444} INFO - Heartbeating the >>> executor >>> > [2017-05-16 20:26:48,727] {jobs.py:1454} INFO - Heartbeating the >>> scheduler >>> > Process DagFileProcessor19-Process: >>> > Traceback (most recent call last): >>> > File "/opt/conda/lib/python3.6/multiprocessing/process.py", line >>> 249, in >>> > _bootstrap >>> > self.run() >>> > File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, >>> in >>> > run >>> > self._target(*self._args, **self._kwargs) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/jobs.py", line >>> 346, >>> > in helper >>> > pickle_dags) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/utils/db.py", >>> line >>> > 53, in wrapper >>> > result = func(*args, **kwargs) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/jobs.py", line >>> 1585, >>> > in process_file >>> > self._process_dags(dagbag, dags, ti_keys_to_schedule) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/jobs.py", line >>> 1174, >>> > in _process_dags >>> > dag_run = self.create_dag_run(dag) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/utils/db.py", >>> line >>> > 53, in wrapper >>> > result = func(*args, **kwargs) >>> > File "/opt/conda/lib/python3.6/site-packages/airflow/jobs.py", line >>> 807, >>> > in create_dag_run >>> > else max(next_run_date, dag.start_date)) >>> > TypeError: can't compare offset-naive and offset-aware datetimes >>> > >>> > >>> > Note that we can run/schedule the example DAGs from the CLI, and our >>> DAGs >>> > are very closely derived from the examples, so we don't know what to >>> do! >>> > >>> > For instance we can run: >>> > >>> > docker@airflow:/mnt/airflow$ airflow run agl_p2p_api_worker_dag >>> > api_worker_task 2017-01-01 >>> > [2017-05-16 20:44:41,046] {__init__.py:57} INFO - Using executor >>> > SequentialExecutor >>> > Sending to executor. >>> > [2017-05-16 20:44:43,274] {__init__.py:57} INFO - Using executor >>> > SequentialExecutor >>> > Logging into: >>> > /root/airflow/logs/agl_p2p_api_worker_dag/api_worker_task/20 >>> 17-01-01T00:00:00 >>> > >>> > >>> > And there are no errors. We are sure the scheduler is running and the >>> > webserver is running. The airflow run/test commands work, but we get >>> this >>> > same error above when we click the activate button on the web app or >>> > unpause the DAG from the CLI. We have wondered if our start date has >>> to be >>> > in the future maybe? We don't know. >>> > >>> > Any help would be appreciated. Thanks! >>> > >>> > Russell Jurney @rjurney <http://twitter.com/rjurney> >>> > russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB >>> > <http://facebook.com/jurney> datasyndrome.com >>> >>> >> >
