[GitHub] [airflow] dondaum commented on issue #17604: Add an option to remove or mask environment variables in the KubernetesPodOperator task instance logs on failure or error events

[GitBox]

dondaum commented on issue #17604:
URL: https://github.com/apache/airflow/issues/17604#issuecomment-899256816


   Thank you for the quick respond.
   
   I think I do not completely understand what you mean. Is it correct that you 
assume that there is another either Airflow variable or connection with the 
name "dbt_warehouse" (lowercase) as key?
   
   If so, I tested it on a fresh local Airflow dev instance with a clean 
environment withouth any other variable than the above mentionend in the 
context section.
   
   But let me test it with another key name that I have not used so far and let 
me list all Airflow variables and connection before.
   
   I guess I am asking this because if the masking works as expected and with 
your latested changes on the exception this feature request, at least from my 
site, is not needed anymore. 
   
   So I run another test, I created a fresh local Airflow instance and listed 
all variables and connections:
   ```
   airflow variables list -o table
   key
   =======
   testvar
   ```
   the variable structure is:
   ```
   airflow variables get testvar
   {
   "THIS_IS_A_RANDOM_SENSITIVE_VAR": "SuperRandomString1"
   }
   ```
   
   and 
   ```
   airflow connections list -o table
   No data found
   ```
   
   I sticked to the same example above with:
   `
   AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES: "THIS_IS_A_RANDOM_SENSITIVE_VAR"
   `
   ```Python
   [2021-08-16 06:09:20,814] {{dbt_k8s_operator.py:67}} INFO - 
{'THIS_IS_A_RANDOM_SENSITIVE_VAR': 'SuperRandomString1'}
   [2021-08-16 06:09:20,814] {{dbt_k8s_operator.py:69}} INFO - Custom 
KubernetesPodOperator runs K8S Task
   [2021-08-16 06:09:20,823] {{taskinstance.py:1501}} ERROR - Task failed with 
exception
   
   ```
   
   Then I only changed the Airflow configuration variable 
AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES and changed the var name to all 
lowercase
   
   `
   AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES: "this_is_a_random_sensitive_var"
   `
   
   ```Python
   [2021-08-16 06:14:19,899] {{dbt_k8s_operator.py:67}} INFO - 
{'THIS_IS_A_RANDOM_SENSITIVE_VAR': '***'}
   [2021-08-16 06:14:19,899] {{dbt_k8s_operator.py:69}} INFO - Custom 
KubernetesPodOperator runs K8S Task
   [2021-08-16 06:14:19,906] {{taskinstance.py:1501}} ERROR - Task failed with 
exception
   
   ```
   
   I get the same behavior.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org