Agree that we should have the same functionality over the WebUI and CLI. We
need multiple connections for a single conn-id for load balancing, so +1 for
adding something like `--allow-multiple` or just a warning when a connection
with the same id is created with cli.
On 07/08/2018, 15:29, "Ash Berlin-Taylor" <a...@apache.org> wrote:
Hmm yes, it appears that the `airflow connections` CLI doesn't let you
create multiple connections of the same conn_id. What the WebUI can do the CLI
should be able to do also! It should allow that in some way (behind a
`--allow-multiple` flag perhaps? I can see an argument for not allowing set
multiple by default as it is often not what people want.)
-ash
> On 7 Aug 2018, at 15:17, Deng Xiaodong <xd.den...@gmail.com> wrote:
>
> Hi Ben,
>
> If you would like to set multiple connections with the same *conn_id*, you
> can only do that in Web UI. That is, you need to build multiple entries
> with different host, but with the same *conn_id*.
>
> The method of setting connection in environment variables can only help
set
> one connection for each *conn_id*.
>
> (Of course Web UI stores your connections in the metadata database. So if
> you really need to avoid manual interventions and need multiple
connections
> for one single conn_id, hacking into the metadata database may be an
> option? even thought not recommended)
>
> Other guys please correct me if I'm wrong in any parts. Thanks.
>
>
> Regards,
> XD
>
> On Tue, Aug 7, 2018 at 9:43 PM Ben Laird <br.la...@gmail.com> wrote:
>
>> Hello -
>>
>> I'd like to define multiple connections for the `webhdfs_default`
conn_id,
>> as we have a primary and backup host for the hadoop namenode. This is for
>> use with the WebHDFSHook, which queries for connections and seemingly can
>> accept a list of them from the database.
>>
>> ```
>> nn_connections = self.get_connections(self.webhdfs_conn_id)
>> for nn in nn_connections:
>> ```
>>
>>
https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L55
>>
>> I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
>> each hostname, however with the CLI and adding the entries to the
>> airflow.cfg I get errors that a connection already exists.
>>
>> It seems WebHDFSHook doesn't support reading multiple hosts from one
entry.
>> Is this a bug? Or is there a different way to accomplish without manually
>> adding via the UI?
>>
>> Thanks!
>> Ben Laird
>>
Kaxil Naik
Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com
