Interesting, this is quite similar to an experiment of mine for a new type of skin that would allow developers to change/extend/reuse/recombine existing skins (I'm calling that composable skins).
Like you, I wanted to untangle layout algorithms from layout containers and control skins, and give them an actual API. In my experiment, I've moved the layout code of HBox to HLayout (an implementation of an interface essentially identical to Layoutable in your proposal), and HBox just delegates its methods to this layout algorithm. A new ComposableSkinBase then accepts a layout algorithm, to which all layout-related methods are delegated (much like with HBox). Where your proposal goes beyond my experiment is the ability to have a tree of layout algorithms, each affecting different nodes. That seems like a very interesting and powerful idea!
