I haven't done extensive testing as this is a new server but yes, I see the logs in s3 when using Celery.
One thing that helped me troubleshoot was to start in interactive mode: docker-compose -f docker-compose up Without the -d flag. I was able to see some error messages that eventually led me to the solution. Did you upgrade from 1.9 or is this a fresh 1.10 implementation? I think some things changed in the airflow.cfg so in my case, I started from the 1.10 version. On Mon, Sep 17, 2018, 9:37 AM vsr.bhav...@gmail.com <vsr.bhav...@gmail.com> wrote: > > > On 2018/09/17 03:03:06, Pedro Machado <pe...@205datalab.com> wrote: > > Bhavani, > > > > I was able to get it to work. I am using a modified version of > > https://github.com/puckel/docker-airflow > > > > Here is what I had to do: > > > > Environment variables: > > > > I did not change airflow.cfg, but put the following variables in .env and > > modified docker-compose.yml to pass them to the containers: > > > > In .env: > > > > LOGGING_CONFIG_CLASS=log_config.DEFAULT_LOGGING_CONFIG > > REMOTE_BASE_LOG_FOLDER=s3://<bucket>/airflow-logs-dev/ > > REMOTE_LOG_CONN_ID=<my s3 connection> > > > > Then in docker-compose.yml, I passed the following under environment for > > all the airflow containers: > > > > environment: > > - LOAD_EX=n > > - FERNET_KEY=<my key> > > - EXECUTOR=Celery > > - POSTGRES_USER=${POSTGRES_USER} > > - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} > > - POSTGRES_DB=${POSTGRES_DB} > > - POSTGRES_HOST=${POSTGRES_HOST} > > - AIRFLOW__CORE__LOGGING_CONFIG_CLASS=${LOGGING_CONFIG_CLASS} > > - AIRFLOW__CORE__REMOTE_LOGGING=True > > - > > AIRFLOW__CORE__REMOTE_BASE_LOG_FOLDER=${REMOTE_BASE_LOG_FOLDER} > > - AIRFLOW__CORE__REMOTE_LOG_CONN_ID=${REMOTE_LOG_CONN_ID} > > - AIRFLOW__SMTP__SMTP_MAIL_FROM=${SMTP_MAIL_FROM} > > - AIRFLOW__WEBSERVER__BASE_URL=${BASE_URL} > > > > I then created the following files: > > > > __init__.py (empty) > > log_config.py (used code from > > > https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py > > ) > > > > and I set up my Dockerfile to copy them to the ${AIRFLOW_HOME}/config/ > > directory: > > > > COPY config/airflow.cfg ${AIRFLOW_HOME}/airflow.cfg > > COPY config/*.py ${AIRFLOW_HOME}/config/ > > > > After this, the workers were able to log to s3 successfully. > > > > This StackOverfiow answer helped me but I had to make some tweaks: > > > https://stackoverflow.com/questions/50222860/airflow-wont-write-logs-to-s3 > > > > Let me know if this works for you. > > > > Pedro > > > > > > On Sun, Sep 16, 2018 at 1:12 AM Bhavani Ramasamy < > bhavani.ramas...@vydia.com> > > wrote: > > > > > I am facing the same issue.. Did you find any solution for this? > > > > > > On 2018/09/08 01:05:26, Pedro Machado <p...@205datalab.com> wrote: > > > > I am looking at the documentation here> > > > > > > > > https://airflow.incubator.apache.org/howto/write-logs.html#writing-logs-to-amazon-s3 > > > > > > > > > and am wondering if for the s3 configuration it's not necessary to > > > create a> > > > > log configuration file as it's described under the GCP section> > > > > > > > > https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform > > > > > > > > > > > > > I ran a quick test configuring remote_base_log_folder> > > > > and remote_log_conn_id through environment variables and it didn't > > > work.> > > > > > > > > Could someone shed some light on this?> > > > > > > > > Thanks,> > > > > > > > > Pedro> > > > > > > Hello Pedro, > > Thanks for quick response. I am using the similar configurations like you > mentioned. S3 logs are working when i am using LocalExecutor. But when I am > using CeleryExecutor it is not writing to S3. I have the celery result > backend also connected to the same PG database & broker as redis queue. Are > you able to use CeleryExecutor & run airflow in multiple docker containers > with S3 logs? > > Thanks, > Bhavani >
