Hi Ash - That'd be perfect. Since conn_id isn't a primary key (since we can add via the UI), I agree that it'd be great to have the same functionality from the CLI with a flag. I can hopefully do this and submit a PR if folks would find it useful!
Ben On Tue, Aug 7, 2018 at 10:28 AM, Ash Berlin-Taylor <a...@apache.org> wrote: > Hmm yes, it appears that the `airflow connections` CLI doesn't let you > create multiple connections of the same conn_id. What the WebUI can do the > CLI should be able to do also! It should allow that in some way (behind a > `--allow-multiple` flag perhaps? I can see an argument for not allowing set > multiple by default as it is often not what people want.) > > -ash > > > On 7 Aug 2018, at 15:17, Deng Xiaodong <xd.den...@gmail.com> wrote: > > > > Hi Ben, > > > > If you would like to set multiple connections with the same *conn_id*, > you > > can only do that in Web UI. That is, you need to build multiple entries > > with different host, but with the same *conn_id*. > > > > The method of setting connection in environment variables can only help > set > > one connection for each *conn_id*. > > > > (Of course Web UI stores your connections in the metadata database. So if > > you really need to avoid manual interventions and need multiple > connections > > for one single conn_id, hacking into the metadata database may be an > > option? even thought not recommended) > > > > Other guys please correct me if I'm wrong in any parts. Thanks. > > > > > > Regards, > > XD > > > > On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br.la...@gmail.com> wrote: > > > >> Hello - > >> > >> I'd like to define multiple connections for the `webhdfs_default` > conn_id, > >> as we have a primary and backup host for the hadoop namenode. This is > for > >> use with the WebHDFSHook, which queries for connections and seemingly > can > >> accept a list of them from the database. > >> > >> ``` > >> nn_connections = self.get_connections(self.webhdfs_conn_id) > >> for nn in nn_connections: > >> ``` > >> > >> https://github.com/apache/incubator-airflow/blob/master/ > airflow/hooks/webhdfs_hook.py#L54-L55 > >> > >> I can manually add 2 entries for `webhdfs_default` in the Web UI, one > for > >> each hostname, however with the CLI and adding the entries to the > >> airflow.cfg I get errors that a connection already exists. > >> > >> It seems WebHDFSHook doesn't support reading multiple hosts from one > entry. > >> Is this a bug? Or is there a different way to accomplish without > manually > >> adding via the UI? > >> > >> Thanks! > >> Ben Laird > >> > >
