Re: RGW blueprint for plugin architecture
On Tue, Aug 20, 2013 at 2:58 AM, Yehuda Sadeh yeh...@inktank.com wrote:
Well, practically I'd like to have such work doing baby steps, rather
than swiping changes. Such changes have higher chances of getting
completed and eventually merged upstream. That's why I prefer the
current model of directly linking the plugins (whether statically or
dynamically), with (relatively) minor internal adjustments.
What current model of directly linking plugins do you refer to exactly?
Maybe start with thinking about the use cases, and then figure what
kind of api that would be. As I said, I'm not sure that an internal
api is the way to go, but rather exposing some lower level
functionality externally. The big difference is that with the former
we tie in the internal architecture, while the latter hides the [gory]
details.
The problem is that right now basically everything is 'lower level
functionality', because a lot of generic stuff depends on S3 stuff,
which in turn depends on generic stuff. Take for example the
following;
class RGWHandler_Usage : public RGWHandler_Auth_S3 { }
class RGWHandler_Auth_S3 : public RGWHandler_ObjStore { }
This basically ties usage statistics collection + authentication
handling + object store all together.
I think this needs to be completely unravelled, but before making all
kinds of use cases (like, usage statistics collection or
authentication in this case) it might be wise to know what the design
decisions were to make the S3 API so very much integrated into
everything else. Or is this just legacy?
Roald
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Tue, Aug 20, 2013 at 4:49 PM, Yehuda Sadeh yeh...@inktank.com wrote: I was referring to your work at wip-rgw-plugin, where the plugin code itself still needs to rely on the rgw utility code. Right. So we can agree on ditching the dynamic loading thing and clean internal API (for now), but at least start separating code into plugins like this? That's not quite a hard dependency. At the moment it's like that, as we made a decision to use the S3 auth for the admin utilities. Switching to a different auth system (atm) would require defining a new auth class and inheriting from it instead. It's not very flexible, but it's not very intrusive. I'd certainly be interested in removing this inheritance relationship and switch to a different pipeline model. I don't know if you looked at it in detail, but for the wip-rgw-plugin work I created a RGWAuthManager / RGWAuthPipeline relation to seggregate authentication specific stuff from the REST handlers. Is that in general a model you like to see discussed in more detail? If so, it would probably be wise to start a separate blueprint for it. As I said, I don't see it as such. We do use it all over the place, but the same way you could just switch these to use RGWHandler_Auth_Swift and it should work (give or take a few tweaks). IMHO, REST handlers should leave authentication/authorization/accounting specific tasks to a separate component (like the aforementioned pipelining system, and maybe integrate that with current RGWUser related code), although this will likely never be purely abstracted (at least for authentication). This just makes the whole system more modular (albeit it just a bit). But for now I propose to implement a small plugin system where plugins are still linked into the rgw core (but code wise as much separated as possible), and keep the auth stuff for later. Roald -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Tue, Aug 20, 2013 at 9:03 AM, Roald van Loon roaldvanl...@gmail.com wrote: On Tue, Aug 20, 2013 at 4:49 PM, Yehuda Sadeh yeh...@inktank.com wrote: I was referring to your work at wip-rgw-plugin, where the plugin code itself still needs to rely on the rgw utility code. Right. So we can agree on ditching the dynamic loading thing and clean internal API (for now), but at least start separating code into plugins like this? That's not quite a hard dependency. At the moment it's like that, as we made a decision to use the S3 auth for the admin utilities. Switching to a different auth system (atm) would require defining a new auth class and inheriting from it instead. It's not very flexible, but it's not very intrusive. I'd certainly be interested in removing this inheritance relationship and switch to a different pipeline model. I don't know if you looked at it in detail, but for the wip-rgw-plugin work I created a RGWAuthManager / RGWAuthPipeline relation to seggregate authentication specific stuff from the REST handlers. Is that in general a model you like to see discussed in more detail? If so, it would probably be wise to start a separate blueprint for it. I didn't look closely at all the details, but yeah, something along those lines. But it'll need to be clearly defined. As I said, I don't see it as such. We do use it all over the place, but the same way you could just switch these to use RGWHandler_Auth_Swift and it should work (give or take a few tweaks). IMHO, REST handlers should leave authentication/authorization/accounting specific tasks to a separate component (like the aforementioned pipelining system, and maybe integrate that with current RGWUser related code), although this will likely never be purely abstracted (at least for authentication). This just makes the whole system more modular (albeit it just a bit). Can't think of examples off the top of my head right now, but the devil's always in the details. Hopefully wrt the auth system there aren't many hidden issues. But for now I propose to implement a small plugin system where plugins are still linked into the rgw core (but code wise as much separated as possible), and keep the auth stuff for later. Sounds good. Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
Hi, I started out by defining the plugin system a bit more (http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture). I'd really appreciate any comments. On Wed, Aug 14, 2013 at 8:40 PM, Yehuda Sadeh yeh...@inktank.com wrote: - AUTH (Authentication backends): Rados/internal, Keystone That's probably the best place to start. I also started working on an auth type of plugin for all Keystone support based on the plugin system currently described in the blueprint (it's an extremely simple and flexible system and IMHO a good way to start). Contrary to the plugin system, the auth plugin itself will move a lot of code around and will therefore have a much greater impact. So it's probably better to separate these projects completely. If we can reach consensus about the way the first version of the plugin system should look like, I will separate these commits into different branches so we can merge it one step at a time. Roald -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Mon, Aug 19, 2013 at 5:34 AM, Roald van Loon roaldvanl...@gmail.com wrote: Hi, I started out by defining the plugin system a bit more (http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture). I'd really appreciate any comments. On Wed, Aug 14, 2013 at 8:40 PM, Yehuda Sadeh yeh...@inktank.com wrote: - AUTH (Authentication backends): Rados/internal, Keystone That's probably the best place to start. I also started working on an auth type of plugin for all Keystone support based on the plugin system currently described in the blueprint (it's an extremely simple and flexible system and IMHO a good way to start). Contrary to the plugin system, the auth plugin itself will move a lot of code around and will therefore have a much greater impact. So it's probably better to separate these projects completely. If we can reach consensus about the way the first version of the plugin system should look like, I will separate these commits into different branches so we can merge it one step at a time. I went over the blueprint, and partially through the code (wip-rgw-plugin). I'm not completely sure about the plugin - core interface that you had in mind. At the moment there's no clean interface, and it just links directly. For example, the keystone plugin needs to call rgw_store_user_info(), the KeystoneClient needs to inherit from RGWHTTPClient, etc. So on one hand we now move this code into external module, but otoh, it still need to link into the monolithic rgw core. Having a clean internal api might be a noble idea, but I'm not sure it's very practical (which is in line with what's there now). Having a linux-kernel like module model makes sense here. The modules/plugins will need to be compiled against the specific version in order for them to be able to load successfully. Also, I'm not sure that we need to bother with dynamic loading. Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Mon, Aug 19, 2013 at 1:55 PM, Roald van Loon roaldvanl...@gmail.com wrote: Hi Yehuda, Thanks for the comments. Yeah, I created the wip-rgw-plugin to get a feeling of how it would look like. It's probably better to see it as a sandbox, but it already clearly shows some of the flaws that you point out. It doesn't need to link into the rgw binary of course, but yes the dl'd .so still expects the (although unexported) symbols to be there. I personally am a *big* fan of a clean internal API and dynamically loadable libraries, especially because I want the librgw to have this feature in the future. For instance, I'd really like a nginx module one day which links to librgw but not all of the APIs or all of the authentication mechanism. The nginx module might want to load/unload specific plugins when it needs to. Same goes for a python extension. An import rgw should not always also be an import rgw.s3 etc. I'd really like these features and mostly I'd really like these features to be slim and not 50+ MB plus Right. However, that doesn't really mean we necessarily need to have tight internal api. It might mean that we'd need to define a different set of apis that will encapsulate these use cases. E.g., rgw data access api, direct user management, etc. By the way, the biggest problem with direct nginx integration is that nginx is event driven, whereas internally the gateway is synchronous (albeit threaded). I agree it's a lot of work (wouldn't call it noble though), but I don't see that as an argument to not do it. However, I also agree that the blueprint/internal API should be extended much more. That's why I asked for comments to begin with :-) Well, practically I'd like to have such work doing baby steps, rather than swiping changes. Such changes have higher chances of getting completed and eventually merged upstream. That's why I prefer the current model of directly linking the plugins (whether statically or dynamically), with (relatively) minor internal adjustments. I'll start thinking about a design for an internal API but I'll definitely need some help with that. I'm not a fulltime Ceph developer :-) Maybe start with thinking about the use cases, and then figure what kind of api that would be. As I said, I'm not sure that an internal api is the way to go, but rather exposing some lower level functionality externally. The big difference is that with the former we tie in the internal architecture, while the latter hides the [gory] details. Roald On Mon, Aug 19, 2013 at 10:05 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Mon, Aug 19, 2013 at 5:34 AM, Roald van Loon roaldvanl...@gmail.com wrote: Hi, I started out by defining the plugin system a bit more (http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture). I'd really appreciate any comments. On Wed, Aug 14, 2013 at 8:40 PM, Yehuda Sadeh yeh...@inktank.com wrote: - AUTH (Authentication backends): Rados/internal, Keystone That's probably the best place to start. I also started working on an auth type of plugin for all Keystone support based on the plugin system currently described in the blueprint (it's an extremely simple and flexible system and IMHO a good way to start). Contrary to the plugin system, the auth plugin itself will move a lot of code around and will therefore have a much greater impact. So it's probably better to separate these projects completely. If we can reach consensus about the way the first version of the plugin system should look like, I will separate these commits into different branches so we can merge it one step at a time. I went over the blueprint, and partially through the code (wip-rgw-plugin). I'm not completely sure about the plugin - core interface that you had in mind. At the moment there's no clean interface, and it just links directly. For example, the keystone plugin needs to call rgw_store_user_info(), the KeystoneClient needs to inherit from RGWHTTPClient, etc. So on one hand we now move this code into external module, but otoh, it still need to link into the monolithic rgw core. Having a clean internal api might be a noble idea, but I'm not sure it's very practical (which is in line with what's there now). Having a linux-kernel like module model makes sense here. The modules/plugins will need to be compiled against the specific version in order for them to be able to load successfully. Also, I'm not sure that we need to bother with dynamic loading. Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RGW blueprint for plugin architecture
Hi all, For those interested; http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture I know a plugin architecture is a sensitive subject for a lot of people (it is for me at least), but I think it might be worthwhile discussing it. Please comment/flame away :-) Roald PS: congrats with the Dumpling release! -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Wed, Aug 14, 2013 at 5:57 AM, Roald van Loon roaldvanl...@gmail.com wrote: Hi all, For those interested; http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture I know a plugin architecture is a sensitive subject for a lot of people (it is for me at least), but I think it might be worthwhile discussing it. Please comment/flame away :-) I'm all for having a modular architecture. In essence this is something we've been aiming at although the interfaces are not as clean and shiny as they should be. OTOH, there's the danger overdoing it and tying our hands behind our backs, so we need to be careful about what to get into a plugin/module, and what should stay as it is now (flexible, albeit monolithic). Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
I most definitely agree, that's why I think a lightweight and extensible solution would be best. No big interface description but just a simple loader function with parameters depending on the type of 'plugin'. If we need another type of plugin, we can add that later on (of course, we have to be extra careful not to make a mess of things and end up having to define another plugin type for every single plugin). That way we can end up having the best of both ways; modular and flexible. What do you think would be the best way to start? Probably to define types of plugins? I can think of two types of plugins right now: - API (REST APIs): S3, Swift, (GS?) etc - AUTH (Authentication backends): Rados/internal, Keystone Implementing these will mostly require moving some code around. That would be doable IMHO. If we have at least the basic plugin framework uprunning, people can start writing their stuff in plugins (instead of keeping a bunch of patches laying around like I do because they make no sense in the current monolithic code base). I think we can do this without tying our hands behind our backs. And as longs as we keep it to simply moving existing code into plugins instead of adding new code until we have a plugin system with which we can live with, we can always go back without breaking functionality. Roald On Wed, Aug 14, 2013 at 7:17 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Wed, Aug 14, 2013 at 5:57 AM, Roald van Loon roaldvanl...@gmail.com wrote: Hi all, For those interested; http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture I know a plugin architecture is a sensitive subject for a lot of people (it is for me at least), but I think it might be worthwhile discussing it. Please comment/flame away :-) I'm all for having a modular architecture. In essence this is something we've been aiming at although the interfaces are not as clean and shiny as they should be. OTOH, there's the danger overdoing it and tying our hands behind our backs, so we need to be careful about what to get into a plugin/module, and what should stay as it is now (flexible, albeit monolithic). Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: RGW blueprint for plugin architecture
On Wed, Aug 14, 2013 at 11:24 AM, Roald van Loon roaldvanl...@gmail.com wrote: I most definitely agree, that's why I think a lightweight and extensible solution would be best. No big interface description but just a simple loader function with parameters depending on the type of 'plugin'. If we need another type of plugin, we can add that later on (of course, we have to be extra careful not to make a mess of things and end up having to define another plugin type for every single plugin). That way we can end up having the best of both ways; modular and flexible. What do you think would be the best way to start? Probably to define types of plugins? I can think of two types of plugins right now: - API (REST APIs): S3, Swift, (GS?) etc I think that for this to actually work out we'd need to provide a stable internal api, which I think that at this point would be quite a big task. - AUTH (Authentication backends): Rados/internal, Keystone That's probably the best place to start. Implementing these will mostly require moving some code around. That would be doable IMHO. If we have at least the basic plugin framework uprunning, people can start writing their stuff in plugins (instead of keeping a bunch of patches laying around like I do because they make no sense in the current monolithic code base). I think we can do this without tying our hands behind our backs. And as longs as we keep it to simply moving existing code into plugins instead of adding new code until we have a plugin system with which we can live with, we can always go back without breaking functionality. Roald On Wed, Aug 14, 2013 at 7:17 PM, Yehuda Sadeh yeh...@inktank.com wrote: On Wed, Aug 14, 2013 at 5:57 AM, Roald van Loon roaldvanl...@gmail.com wrote: Hi all, For those interested; http://wiki.ceph.com/01Planning/02Blueprints/Emperor/rgw:_plugin_architecture I know a plugin architecture is a sensitive subject for a lot of people (it is for me at least), but I think it might be worthwhile discussing it. Please comment/flame away :-) I'm all for having a modular architecture. In essence this is something we've been aiming at although the interfaces are not as clean and shiny as they should be. OTOH, there's the danger overdoing it and tying our hands behind our backs, so we need to be careful about what to get into a plugin/module, and what should stay as it is now (flexible, albeit monolithic). Yehuda -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
