Hi Manu, You can set up multiple connections with the same conn_id and different host, rather than setting in one single connection.
XD On Thu, Aug 30, 2018 at 11:17 Manu Zhang <owenzhang1...@gmail.com> wrote: > Hi Ben, > > How do you set multiple connections through Web UI (from Connections item > of Admin pull-down list) ? I'm tried setting a comma-separated list to a > conn_id but that doesn't work. > > Thanks, > Manu > > > On Wed, Aug 29, 2018 at 11:31 PM Ben Laird <br.la...@gmail.com> wrote: > > > Hi Manu, > > > > We have the same use case as you, a primary and backup namenode. If I > > understand your issue correctly, the WebHDFSSensor code checks an > iterable > > of Airflow connections to the namenode to find one that is active. > > > > However, my issue (which I've emailed this list about) was that you > cannot > > set multiple connections with the same name (e.g. webhdfs_default) > through > > the CLI, only in the Web interface. I'm planning on submitting a PR soon > to > > remedy this. > > > > Ben > > > > On Wed, Aug 29, 2018 at 2:57 AM Driesprong, Fokko <fo...@driesprong.frl> > > wrote: > > > > > Hi Manu, > > > > > > Thanks for raising this question. There is a PR for moving > > > <https://github.com/apache/incubator-airflow/pull/3560> to hdfs3. > There > > is > > > code in the existing codebase, which support HA > > > < > > > > > > https://github.com/apache/incubator-airflow/blob/53b89b98371c7bb993b242c341d3941e9ce09f9a/airflow/hooks/hdfs_hook.py#L92-L96 > > > >, > > > but this might not be for the sensor. > > > > > > Personally I'm not familiar with pyarrow.hdfs, so I'm not the one to > > judge > > > how mature it is. We need to replace Snakebite for sure since it is > only > > > compatible with Python 2.7. > > > > > > Cheers, Fokko > > > > > > > > > Op wo 29 aug. 2018 om 04:29 schreef Manu Zhang < > owenzhang1...@gmail.com > > >: > > > > > > > Hi all, > > > > > > > > We've been using WebHdfsSensor happily to sensor the state of > upstream > > > > tasks outputting to HDFS except when there is a namenode switch. I've > > > > opened https://issues.apache.org/jira/browse/AIRFLOW-2901 to discuss > > the > > > > HDFS HA support. > > > > > > > > There are two solutions that I can see, > > > > > > > > 1. use pyarrow.hdfs which has HA support > > > > 2. allow user to configure a list of namenodes > > > > > > > > WDYT ? > > > > > > > > Thanks, > > > > Manu Zhang > > > > > > > > > >
