[GitHub] [airflow] magnusfurugard opened a new issue #19044: Statsd host does not update

[GitBox]

magnusfurugard opened a new issue #19044:
URL: https://github.com/apache/airflow/issues/19044


   ### Official Helm Chart version
   
   1.2.0 (latest released)
   
   ### Apache Airflow version
   
   2.1.4 (latest released)
   
   ### Kubernetes Version
   
   v.1.21.2
   
   ### Helm Chart configuration
   
   ```
   statsd:
     enabled: false
   
   env:
     # Enable statsd here, disable chart default.
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: <service name for statsd>
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: <port for statsd>
   ```
   
   ### Docker Image customisations
   
   None.
   
   ### What happened
   
   Setting `AIRFLOW__METRICS__HOST` seem to have no effect, even if we set 
`statsd.enabled=false` in the chart and enable it through the same config 
(`AIRFLOW__METRICS__STATSD_ON`). Airflow will continue to try to resolve the 
default host, `<name>-statsd`-service, even if set `AIRFLOW__METRICS__HOST`. 
   
   ### What you expected to happen
   
   I expected Airflow to honor the `AIRFLOW__METRICS__HOST` variable and 
resolve the service as specified there.
   
   ### How to reproduce
   
   ###  Step 1 - vanilla install
   Install the helm-chart with the following `values.yml`
   ```yml
   # airflow-values.yml
   
   statsd:
     enabled: true # this is the default
   ```
   ```bash
   helm install -f airflow-values.yml example apache-airflow/airflow --version 
1.2.0 --timeout 1200s
   ```
   
   ### Step 2 - Replace statsd with custom statsd
   We want to disable the chart-built-in statsd and use our own instead. To 
accommodate, set up _any_ statsd deployment in the same namespace. In my case, 
I use the prometheus exporter one but it can be absolutely anything.
   
   ```bash
   # Install statsd. 
   helm repo add prometheus-community 
https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install -f statsd-values.yml statsd 
prometheus-community/prometheus-statsd-exporter --timeout 1200s
   
   # Chart-default is to accept udp/tcp on port 9125. Above config also creates 
a service for us called `statsd-prometheus-statsd-exporter`.
   ```
   ... And update Airflow to communicate with the new statsd
   ```yml
   # airflow-values.yml
   
   # Update Airflow to use the new statsd
   statsd:
     enabled: false
   
   env:
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: statsd-prometheus-statsd-exporter
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: 9125
   ```
   ```bash
   helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 
1.2.0 --timeout 1200s
   ```
   
   ### Step 3 - Airflow fails to resolve the new host
   Run any DAG from within Airflow, or restart the webserver to trigger an 
error, displaying that it's unable to resolve the service; 
   ```
   [2021-10-18 14:22:35,668] {stats.py:359} ERROR - Could not configure 
StatsClient: [Errno -2] Name or service not known, using DummyStatsLogger 
instead.
   ```
   
   ### Step 4 - Rename the new statsd-service to match what Airflow is looking 
for
   We can fix this by overriding the name of the new service to match that 
which we _know_ Airflow is looking for (`<name>-statsd`).
   
   ```yml
   # statsd-values.yml
   
   # Remember we set the name of our helm install to `example` in step 1.
   # This toggle alters the name of the service for our custom statsd to be 
whatever we put below.
   fullnameOverride: example-statsd
   ```
   ```bash
   helm upgrade -f statsd-values.yml statsd 
prometheus-community/prometheus-statsd-exporter --timeout 1200s
   ```
   
   **Airflow is now able to log metrics to our custom statsd-client.**
   
   And just to showcase that these variables are not picked up, let's set them 
to something that we know does not exist. 
   ```yml
   # airflow-values.yml
   
   statsd:
     enabled: false
   
   env:
     - name: AIRFLOW__METRICS__STATSD_ON
       value: true
     - name: AIRFLOW__METRICS__HOST
       value: whatever-you-want-really
     - name: AIRFLOW__METRICS__STATSD_PORT
       value: 3123 # anything numeric
   ```
   ```bash
   helm upgrade -f airflow-values.yml example apache-airflow/airflow --version 
1.2.0 --timeout 1200s
   ```
   
   **Airflow is _still_ able to log metrics to our custom statsd-client.**
   
   
   ### Anything else
   
   I ran this with `executor=KubernetesExecutor`, but I doubt it has any 
effect. Made releases and upgrades faster.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org