Folks, I think this thread has fallen off the rails a bit, perhaps we could get a tcon/hangout going for this again.
I would be happy to moderate if needed, but it's pretty clear to me to that trying to hash this out in an email thread is becoming couter-productive. Cheers, Tim ----- Original Message ----- > From: "Dominic Hamon" <dha...@twitter.com.INVALID> > To: "dev" <dev@mesos.apache.org> > Sent: Saturday, September 20, 2014 9:21:59 AM > Subject: Re: Mesos Modules Design > > On Fri, Sep 19, 2014 at 2:43 PM, Niklas Nielsen <nik...@mesosphere.io> > wrote: > > > Hi Dominic, > > > > (response inlined) > > > > On 19 September 2014 13:03, Dominic Hamon <dha...@twopensource.com> wrote: > > > > > I'm sorry, but I'm still having a hard time understanding why this needs > > to > > > be dynamic. > > > > > > If the mesos core is split into modules that are built as standalone > > > libraries (static) then at link time the right combination of libraries > > can > > > be bundled together to create the end result. If you want to get even > > > smarter, we can have default versions that are linked in to the mesos > > core > > > as weak symbols so later linked libraries can override the defaults. This > > > may mean that we move to static linking across the board, but frankly > > there > > > are a few benefits to that approach. > > > > > > This works for some, but not all use cases. > > One use-case where it _does_ make sense to statically bake into the image > > could be: https://issues.apache.org/jira/browse/MESOS-1330 > > That be, you probably rarely want to swap network transport implementations > > in and out on per-run basis but will know up front which one to configure > > and use. > > > > On the other hand, having to relink and rebuilding give users a poor > > experience and makes it hard to select (or unselect) custom components. > > Using a prebuilt package and point against libraries is a pretty common > > work-flow: Apache web server relies heavily on modules from dynamic > > libraries. > > > > > I'm going to argue against this, so please bear with me while I work > through it, and please point out anywhere that my assertions are wrong. > > The proposal you are suggesting requires the following: > > - callsites need to be modules aware to use the right factory method to > instantiate the modular object > - mesos-core owners need to be aware of modules and versioning when they > change the abstract API > - module owners need to be aware of the modules API and versioning > - customers (end-users) have to be modules aware and set their runtime > configuration correctly > - errors in versioning or modules will only be caught at runtime > > My alternative proposal requires the following: > > - module owners need to be aware of API changes during core upgrades > - issues are caught at compile time by module owners > - customers (end-users) are unaware of modules and only need to know about > per module configuration (kerberos attributes, for instance) not the > modules system and configuration > > In short, your proposal adds more complexity and shifts the burden of > knowledge and responsibility to every agent, instead of keeping it with the > module owner where it belongs. > > I think the solution that's been put up for review is clever, but I don't > think it solves the problem in the right way for anyone involved. Please > rethink your approach to take that into account. > > > > > TL;DR I don't see the two approaches to be mutually exclusive and we can > > get a lot of leverage with the current design but we want to be able to do > > this with static linking too. (To Tim St Clair's point) > > > > > > > > > > > > > > > > With the approach as defined, does this mean that the default versions > > will > > > also have to be reimplemented as modules? > > > > > > > Not necessary. Mesos default implementations stays as is. See Bernd's > > comment. > > > > > > > > > > Has any effort been put into determining the performance overhead of the > > > approach as specified? > > > > > > > It won't affect Mesos if you are not using modules. The call sites will be > > virtual dispatches no matter whether you are using modules or > > internal/default implementations. > > There is a performance penalty of not being able to do global optimizations > > within the module, but that is the trade-off of implementing a dynamic > > loadable module. > > > > > > > > > > On Fri, Sep 19, 2014 at 11:35 AM, Niklas Nielsen <nik...@mesosphere.io> > > > wrote: > > > > > > > Hi everyone, > > > > > > > > We have been iterating on a design for pluggable modules in Mesos > > lately > > > > and wanted to get a last round of feedback before putting out patch > > sets. > > > > > > > > Tim St Clair, Ben Hindman and I started the discussion (and work) on > > this > > > > subsystem https://issues.apache.org/jira/browse/MESOS-1224 and > > > > https://issues.apache.org/jira/browse/MESOS-1384. > > > > Kapil and Bernd took over the work (shepherded by Ben H and I) and have > > > > expanded on the original design to cope with api/modules/mesos > > versioning > > > > semantics and be extensible enough to cope future changes in the > > modules > > > > subsystem (dealing with modules dependencies, etc). > > > > > > > > The latest description of the modules system has been captured in: > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/MESOS/DRAFT+Design+Doc+-+Mesos+Modules > > > > (for those of you who don't want to read through the JIRA threads). > > > > > > > > We have an implementation ready based on this design and will be > > sharing > > > / > > > > starting review rounds start next week. > > > > > > > > Feel free to use this thread if you have any questions. > > > > > > > > Cheers, > > > > Niklas > > > > > > > > > > > > > > > > -- > > > Dominic Hamon | @mrdo | Twitter > > > *There are no bad ideas; only good ideas that go horribly wrong.* > > > > > > > Hope that helps; if not, we'd be happy to jump on a call. > > > > Niklas > > > > > > -- > Dominic Hamon | @mrdo | Twitter > *There are no bad ideas; only good ideas that go horribly wrong.* > -- Cheers, Timothy St. Clair Red Hat Inc.
