I just discovered, that Flower renders Tasks, as soon as I’m using a
`BranchPythonOperator`. So that does not appear to be a setup problem.
> On 6. May 2020, at 12:11, Andreas Balke <andreas.ba...@now-extern.de> wrote:
>
> Hi Ash,
>
> that appears to be OK:
>
> ● airflow-scheduler.service - Airflow scheduler daemon
> Loaded: loaded (/lib/systemd/system/airflow-scheduler.service; enabled;
> vendor preset: enabled)
> Active: active (running) since Wed 2020-05-06 10:09:20 UTC; 3s ago
> Main PID: 10610 (/usr/bin/python)
> Tasks: 3 (limit: 4667)
> CGroup: /system.slice/airflow-scheduler.service
> ├─10610 /usr/bin/python3 /usr/local/bin/airflow scheduler -n 5
> --pid /run/airflow/scheduler.pid
> └─10641 airflow scheduler -- DagFileProcessorManager
>
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,119]
> {scheduler_job.py:1504} DEBUG - Heartbeating the executor
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {base_executor.py:122} DEBUG - 0 running task instances
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {base_executor.py:123} DEBUG - 0 in queue
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {base_executor.py:124} DEBUG - 32 open slots
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {base_executor.py:133} DEBUG - Calling the <class
> 'airflow.executors.celery_executor.CeleryExecutor'> sync method
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {celery_executor.py:240} DEBUG - No task to query celery, skipping sync
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,120]
> {scheduler_job.py:1459} DEBUG - Ran scheduling loop in 0.00 seconds
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,121]
> {scheduler_job.py:1462} DEBUG - Sleeping for 1.00 seconds
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,153]
> {scheduler_job.py:268} DEBUG - Waiting for
> <Process(DagFileProcessor1-Process, stopped)>
> May 06 10:09:24 ip-10-1-17-115 airflow[10610]: [2020-05-06 10:09:24,234]
> {settings.py:278} DEBUG - Disposing DB connection pool (PID 10646)
>
> # cat /etc/airflow/airflow.cfg | grep -i execut
> # The executor class that airflow should use. Choices include
> # SequentialExecutor, LocalExecutor, CeleryExecutor
> executor = CeleryExecutor
>
> Best, Andreas
>
>
>> On 6. May 2020, at 12:03, Ash Berlin-Taylor <a...@apache.org
>> <mailto:a...@apache.org>> wrote:
>>
>> Your second point there would lead me to believe that the scheduler is
>> still actually running with the default SequentialExecutor.
>>
>> Which config file did you edit?
>>
>> What output is shown when you (re)start the scheduler?
>>
>> Thanks,
>> -ash
>>
>> On May 6 2020, at 11:00 am, Andreas Balke <andreas.ba...@now-extern.de
>> <mailto:andreas.ba...@now-extern.de>> wrote:
>>
>>> Dear Airflow community,
>>>
>>> not sure if I’m targeting the right audience here. Just trying though
>>> :)
>>>
>>> In a basic setup, like described here:
>>> https://www.cloudwalker.io/2019/09/30/airflow-scale-out-with-redis-and-celery/
>>>
>>> <https://www.cloudwalker.io/2019/09/30/airflow-scale-out-with-redis-and-celery/>
>>> <https://www.cloudwalker.io/2019/09/30/airflow-scale-out-with-redis-and-celery/>,
>>> using `airflow_version: 1.10.10`, I discover two issues:
>>>
>>> 1) Even though the `airflow.cfg` is configured to use `CeleryExecutor`
>>> +Redis and all the logs mention to use it, there is no Task visible in
>>> Flower. When a DAG is started, it will only be processed via the DB
>>>
>>> 2) Many times, especially once a DAG is running, in the UI I see the
>>> warning "The scheduler does not appear to be running.”
>>>
>>> Do you have any advice how to solve those?
>>>
>>> Best, Andreas
>
