Regarding Docbook... if it can preserve all LyX metadata, including custom insets and such and not lose much in a round-trip conversion, great, but I suspect it can't. Really, you need an XML schema that is as closer an analog of .lyx as possible.
XSLT is not going away. Reproducing its power in C++ is certainly doable -- XSLT and XPath are basically like Lisp's destructuring-bind macro family on steroids, and then some -- but it's be a lot of work.
