abhishekbhakat commented on issue #15613: URL: https://github.com/apache/airflow/issues/15613#issuecomment-1319085608
I'm using Astro-runtime image 6.0.4 pip freeze: ``` apache-airflow==2.4.3+astro.1 apache-airflow-providers-amazon==6.0.0 apache-airflow-providers-apache-hive==4.0.1 apache-airflow-providers-apache-livy==3.1.0 apache-airflow-providers-celery==3.0.0 apache-airflow-providers-cncf-kubernetes==4.4.0 apache-airflow-providers-common-sql==1.2.0 apache-airflow-providers-databricks==3.3.0 apache-airflow-providers-dbt-cloud==2.2.0 apache-airflow-providers-elasticsearch==4.2.1 apache-airflow-providers-ftp==3.1.0 apache-airflow-providers-google==8.4.0 apache-airflow-providers-http==4.0.0 apache-airflow-providers-imap==3.0.0 apache-airflow-providers-microsoft-azure==4.3.0 apache-airflow-providers-postgres==5.2.2 apache-airflow-providers-redis==3.0.0 apache-airflow-providers-sftp==4.1.0 apache-airflow-providers-snowflake==3.3.0 apache-airflow-providers-sqlite==3.2.1 apache-airflow-providers-ssh==3.2.0 openlineage-airflow==0.16.1 ``` Relevant Airflow Configs: ``` AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG AIRFLOW__LOGGING__REMOTE_LOGGING=True AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER= AIRFLOW__ELASTICSEARCH__JSON_FORMAT=True AIRFLOW__ELASTICSEARCH_CONFIGS__USE_SSL=False AIRFLOW__ELASTICSEARCH_CONFIGS__VERIFY_CERTS=False AIRFLOW__ELASTICSEARCH__HOST=<username>:<password>@<es_host>:<es_port> AIRFLOW__ELASTICSEARCH__FRONTEND="http://localhost:5601/app/kibana#/discover?_a=(columns:!(message),query:(language:kuery,query:'log_id: \"{log_id}\"'),sort:!(log.offset,asc))" ``` Logstash Conf: ``` # logstash.conf input { file { path => ["/usr/local/airflow/logs/*/*/*/*.log"] codec => "json" } }filter { grok { match => { "path" => "/usr/local/airflow/logs/(?<dag_id>.*?)/(?<task_id>.*?)/(?<execution_date>.*?)/(?<try_number>.*?).log$" } } mutate { add_field => { "log_id" => "%{[dag_id]}-%{[task_id]}-%{[execution_date]}-%{[try_number]}" } } }output { opensearch { hosts => ["172.18.0.1:9200"] index => "index_name-%{+YYYY.MM.dd}" user => "<user>" password => "<password>" ssl => false ssl_certificate_verification => false ecs_compatibility => disabled } stdout { codec => rubydebug } } ``` Have a separate docker image for Logstash sharing logs path with airflow components: ``` FROM opensearchproject/logstash-oss-with-opensearch-output-plugin:latest RUN rm -f /usr/share/logstash/pipeline/logstash.conf COPY logstash.conf /usr/share/logstash/pipeline/logstash.conf USER root RUN mkdir /usr/local/airflow USER logstash ``` For elasticsearch using an Opensearch endpoint. [Link](https://opensearch.org/docs/latest/opensearch/install/docker/#sample-docker-compose-file-for-development) This should do. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
