I think this sounds like a great idea, though having not had to set up connections in a while it's a little hard for me to picture (though I do remember the pain of figuring out the very first Google Cloud connectors). Could provide a little example?
For example, would I just define a connection as a string, like "mongo://user@password/h.o.s.t:port/database" and trust a plugin to register as a "mongohandler" and turn that URI into a connection? To Alex's point, perhaps more complex needs could be handled as query parameters (today they are just handwritten JSON in "extras", so that's not much of a change): "gcp://project?keyfile=path/to/keyfile" Just shooting out some ideas... Thanks! On Mon, Jan 9, 2017 at 3:44 PM George Leslie-Waksman <[email protected]> wrote: Could registering new types be handled through the plugin infrastructure? On Mon, Jan 9, 2017 at 5:14 AM Alex Van Boxel <[email protected]> wrote: > I was actually going to propose something different with entry-points, but > your requirement beat me to it (but that's ok :-). Actually I think with > this mechanism people would be able to extend Airflow connection mechanism > (and later other stuff) by doing *pip install airflow-sexy-new-connection* > (for example). > > On Mon, Jan 9, 2017 at 1:39 PM Gael Magnan <[email protected]> wrote: > > > Thank you for the read, I'm gonna look at it, it's probably gonna be > better > > that what I have. > > > > Point taken about the URI, I'll see if i can find something generic > enough > > to handle all those cases. > > > > Le lun. 9 janv. 2017 à 13:36, Alex Van Boxel <[email protected]> a écrit > : > > > > > Thanks a lot, yes it clarifies a lot and I do agree you really need to > > hack > > > inside Airflow to add a Connection type. While you're working at this > > could > > > you have a look at the standard python *entry-point mechanism* for > > > registering Connection types/components. > > > > > > A good read on this: > > > > > > > > > http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/advanced_pylons/entry_points_and_plugins.html > > > > > > My first though would be that just by adding an entry to the factory > > method > > > would be enough to register your Connection + ConnectionType and UI. > > > > > > Also note that not everything works with a URI. The Google Cloud > > Connection > > > doesn't have one, it uses a secret key file stored on disk, so don't > > force > > > every connection type to work with URI's. > > > > > > > > > > > > On Mon, Jan 9, 2017 at 1:15 PM Gael Magnan <[email protected]> > wrote: > > > > > > > Yes sure, > > > > > > > > The question was the following: > > > > "I was looking at the code of the connections, and I realized you > can't > > > > easily add a connection type without modifying the airflow code > > source. I > > > > wanted to create a mongodb connection type, but I think the best > > approche > > > > would be to refactor connections first. Thoughts anyone?" > > > > > > > > The answer of Bolke de Bruin was: "making it more generic would be > > > > appreciated" > > > > > > > > So basically the way the code is set up actually every types of > > > connection > > > > existing is defined though a list in the Connection class. It > > implements > > > > exactly the same code for parsing uri to get connections info and > > doesn't > > > > allow for a simple way to get back the uri from the connection infos. > > > > > > > > I need to add a mongodb connection and a way to get it back as a uri, > > so > > > i > > > > could use an other type of connection and play around with that or > > juste > > > > add one more hard coded connection type, but I though this might be > > > > something that comes back regularly and having a simple way to plug > in > > > new > > > > types of connection would make it easier for anyone to contribute a > new > > > > connection type. > > > > > > > > Hope this clarifies my proposal. > > > > > > > > Le lun. 9 janv. 2017 à 12:46, Alex Van Boxel <[email protected]> a > > écrit > > > : > > > > > > > > > Hey Gael, > > > > > > > > > > could you please recap the question here and provide some context. > > Not > > > > > everyone on the mailinglist is actively following Gitter, including > > me. > > > > > With some context it would be easier to give feedback. Thanks. > > > > > > > > > > On Mon, Jan 9, 2017 at 11:15 AM Gael Magnan <[email protected]> > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > following my question on gitter the other day and the response > from > > > > Bolke > > > > > > de Bruin, I've started working on refactoring the connections in > > > > airflow. > > > > > > > > > > > > Before submitting a PR I wanted to share my proposal with you and > > get > > > > > > feedbacks. > > > > > > > > > > > > The idea is quite simple, I've divided the Connection class in > two, > > > > > > Connection and ConnectionType, connection has the same interface > it > > > had > > > > > > before plus a few methods, but the class keeps a reference to a > > > > > dictionary > > > > > > of registered ConnectionType. It delegates the work of parsing > from > > > > URI, > > > > > > formatting to URI (added) and getting the hook to the > > ConnectionType > > > > > > associated with the conn_type. > > > > > > > > > > > > I've thought of two ways of registering new ConnectionTypes, the > > > first > > > > is > > > > > > making the BaseConnectionType use a metaclass that registered any > > new > > > > > > ConnectionType with Connection when the class is declared, it > would > > > > > require > > > > > > the less work to extend the connection module, as just importing > > the > > > > file > > > > > > with the connection would do the trick. > > > > > > The second one is juste to have a function/classmethod that you > > call > > > > > > manually to register your connection. It would be simpler to > > > understand > > > > > but > > > > > > requires more work every time you create a new ConnectionType. > > > > > > > > > > > > Hope this proposal is clear enough, and I'm waiting for feebacks > > and > > > > > > possible improvements. > > > > > > > > > > > > Regards > > > > > > Gael Magnan de Bornier > > > > > > > > > > > -- > > > > > _/ > > > > > _/ Alex Van Boxel > > > > > > > > > > > > -- > > > _/ > > > _/ Alex Van Boxel > > > > > > -- > _/ > _/ Alex Van Boxel >
