Re: Custom (mongodb) hook as a plugin
On 23 September 2016 at 15:06, Rob Goretsky wrote: > More generally, would it make sense to create an enhancement to the plugin > architecture that allows the developer of a new hook to also specify the > name of a new 'connection type' along with what fields would be required > for that connection type? The web UI would then take its existing > hardcoded list and append these plugged-in connection types and fields to > display... Yeah, I filed a ticket a while ago to do so (https://issues.apache.org/jira/browse/AIRFLOW-235) Additionally, it would be good if the connections were not hardcoded so that it were easier to add new connections directly (perhaps through the config file) -Jakob
Re: Custom (mongodb) hook as a plugin
I have been building a plugin with hooks for Redshift and Teradata, and
initially was also confused as to why my I didn't see Redshift and Teradata
appear as "Connection Types" in the Connections admin web interface. I
then realized, as Andrew Phillips pointed out here, that the list of
connection types for the UI is hardcoded, so it is not aware of any hooks
you add via the Plugin interface.
As a workaround, I found that it was perfectly fine to create my
connections in the Web UI as the "MySQL" connection type and then fill in
the host/login/password I needed for connecting to my specific Redshift and
Teradata instances. My custom hooks were then able to read those
host/login/password attributes without any issue - regardless of the fact
that I had set the connection type to "MySQL". So, it seems that the
"connection type" that one chooses ("MySQL", "Postgres", etc) is mostly
irrelevant, since the same fields (host/login/password) are collected by
the UI regardless of the "connection type" (with the exception of the JDBC
and Google Cloud connection types, which show specifically labelled subset
of fields).. Shin, can you try that and see if it works as well in your
case?
More generally, would it make sense to create an enhancement to the plugin
architecture that allows the developer of a new hook to also specify the
name of a new 'connection type' along with what fields would be required
for that connection type? The web UI would then take its existing
hardcoded list and append these plugged-in connection types and fields to
display...
-Rob
On Fri, Sep 23, 2016 at 5:31 PM, Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:
> False, your hook will still read from the centralized connection store,
> even if it lives outside the Airflow package.
>
> Max
>
> On Fri, Sep 23, 2016 at 2:05 PM, Shin Tai wrote:
>
> > Ok - thanks. Sounds like the sort of thing Airflow isn't meant to do
> right
> > now because it wouldn't be possible to configure the connector with
> > usernames, passwords etc
> >
> > I may just have to fork for the time being.
> >
> > Thanks again.
> >
> > On 23 September 2016 at 17:40, Andrew Phillips
> wrote:
> >
> > > 6. Went to UI/Connections but couldn't find my connector/hook
> > >>
> > >
> > > The experts on the list will hopefully set me right I'm my
> understanding
> > > is incorrect, but from what I can see the list of hook types available
> > *in
> > > the UI* is fixed:
> > >
> > > https://github.com/apache/incubator-airflow/blob/f360414774f
> > > 1ecb2e0f4e53ebdd623c7435f9a78/airflow/www/views.py#L2424
> > >
> > > Regards
> > >
> > > ap
> > >
> >
>
Re: Custom (mongodb) hook as a plugin
False, your hook will still read from the centralized connection store, even if it lives outside the Airflow package. Max On Fri, Sep 23, 2016 at 2:05 PM, Shin Tai wrote: > Ok - thanks. Sounds like the sort of thing Airflow isn't meant to do right > now because it wouldn't be possible to configure the connector with > usernames, passwords etc > > I may just have to fork for the time being. > > Thanks again. > > On 23 September 2016 at 17:40, Andrew Phillips wrote: > > > 6. Went to UI/Connections but couldn't find my connector/hook > >> > > > > The experts on the list will hopefully set me right I'm my understanding > > is incorrect, but from what I can see the list of hook types available > *in > > the UI* is fixed: > > > > https://github.com/apache/incubator-airflow/blob/f360414774f > > 1ecb2e0f4e53ebdd623c7435f9a78/airflow/www/views.py#L2424 > > > > Regards > > > > ap > > >
Re: Custom (mongodb) hook as a plugin
Ok - thanks. Sounds like the sort of thing Airflow isn't meant to do right now because it wouldn't be possible to configure the connector with usernames, passwords etc I may just have to fork for the time being. Thanks again. On 23 September 2016 at 17:40, Andrew Phillips wrote: > 6. Went to UI/Connections but couldn't find my connector/hook >> > > The experts on the list will hopefully set me right I'm my understanding > is incorrect, but from what I can see the list of hook types available *in > the UI* is fixed: > > https://github.com/apache/incubator-airflow/blob/f360414774f > 1ecb2e0f4e53ebdd623c7435f9a78/airflow/www/views.py#L2424 > > Regards > > ap >
Re: Custom (mongodb) hook as a plugin
6. Went to UI/Connections but couldn't find my connector/hook The experts on the list will hopefully set me right I'm my understanding is incorrect, but from what I can see the list of hook types available *in the UI* is fixed: https://github.com/apache/incubator-airflow/blob/f360414774f1ecb2e0f4e53ebdd623c7435f9a78/airflow/www/views.py#L2424 Regards ap
Custom (mongodb) hook as a plugin
Hi, I'm attempting to create a custom hook (for mongodb) as a plugin but it's not appearing in the list of connections in the UI. Steps: 1. Ran Airflow via docker (https://github.com/puckel/docker-airflow) 2. Copied the example from https://pythonhosted.org/airflow/plugins.html 3. Copied the postgres hook, renamed a few fields and ended up with https://gist.github.com/shin-nien/2d011972d8631c92675d1de58b000168 (it won't work for mongo yet but I wanted to test that it would be available). 4. Restarted the container and observed pyc in plugins dir was created. 5. Went to UI and noticed the test menus 6. Went to UI/Connections but couldn't find my connector/hook Any tips? Thanks
