Hi all, We've been using WebHdfsSensor happily to sensor the state of upstream tasks outputting to HDFS except when there is a namenode switch. I've opened https://issues.apache.org/jira/browse/AIRFLOW-2901 to discuss the HDFS HA support.
There are two solutions that I can see, 1. use pyarrow.hdfs which has HA support 2. allow user to configure a list of namenodes WDYT ? Thanks, Manu Zhang
