Tobias Hunger wrote: > Hello CMake Developers, > > Stephen Kelly and me met last Tuesday to talk about the daemon-mode patch > we both have been working on. It was a very productive meeting: We managed > to resolve almost all the differences of opinion we had.
Thanks Tobias for the summary and everyone else for the discussion. Here are a few additional notes that I took when meeting Tobias last week. These notes don't change anything in the discussion, but expands on a few things to put some meat on the bones and lists some discarded design options. cmMessenger ----------- > * Having cmMessanger would be nice and will make things easier. In particular - a cmMessenger class can have some virtual methods implemented to print to the console as cmake::IssueMessage currently does. It would be inherited by a cmServerMessenger which would override the virtual methods to send them over the server IPC protocol instead of printing them on stdout. This has the advantage of not losing semantic information - a backtrace becomes a JSON array, meaning clients don't have to try to parse the backtrace out of the stdout message as they currently do. That becomes possible with just a IssueMessage virtual. See http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/15607 Additionally, the cmMessenger can have additional high-level virtual methods, such as for find_package results making it easier for user tools and IDEs to get a list of packages searched for and make it possible to give users use-case specific ways of specifying where packages are. Similarly it can have a virtual method for try_compile results (somehow - this might require extending the message type for scripting API) which also give user tools more opportunities to present the user with semantic information and not discard the intent of the person writing the buildsystem. Another place where a virtual method on cmMessenger would make sense is for policy warnings, giving user tools a chance to present them to the user without trying to parse them from stdout. There may be other messages which have semantics whose meaning we currently discard by printing to stdout. All of this is currently blocked by a clean-up in the parser needing further review and perhaps further implementation work: http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/15607/focus=16663 Name ---- Last year before publication I was calling this feature a 'metadata server' (class called cmMetadataServer etc). I probably made a mistake renaming it to a daemon. Now, 'metadata server' might be better and more specific than 'server' alone. Change Notification ------------------- I wrote previously that I think change notification is an essential feature of the first version of the protocol - otherwise the output is no better than a static generated file. http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16589/focus=16648 Tobias was concerned that implementing filesystem watching for all necessary platforms is an unreasonable burden and he would prefer something like QFileSystemWatcher. libuv has a portable abstraction for filesystem change notification similar in a way to QFileSystemWatcher. I implemented a basic use of the libuv API fs notification API last year. Action item for me is to dig out that work or do it again. Then the metadata server can emit a message on the protocol when the source directory content changes. Feature Flags ------------- > 1.2 Protocol versions > > * We need the daemon-mode to support different protocol versions. > > * We want to have the option to mark protocol versions as experimental. As far as I know, Chrome has a development model whereby unfinished features are checked into master immediately and developed there, instead of being developed in a branch. The new/experimental features are disabled by default with 'feature flags' so that people/CI can enable and test features which are candidates to be enabled by default to everyone. We discussed whether this would make sense for the server instead of 'experimental' protocol versions. We decided that such feature flags would make sense if there would be many experimental features in flight from multiple contributors over multiple cmake releases. We don't expect this to be the case, so we consider feature flags to be over-engineered for the server for now. InReplyTo --------- > * Currently a reply (and error and all other messages in response to a > request) contain a "inReplyTo" with the type string from the request > triggering the reply. > > Stephen thinks that is not necessary and argues anything that is not > strictly necessary should not be in the first version of the protocol. Tobias told me that the only reason for this is to make it easier for people developing the actual protocol to paste blocks of text which contain multiple server messages. For example a developer could paste a 'configure' and a 'compute' and a 'codeModel' message over and over while implementing the 'codeModel' message. If the 'configure' or 'compute' fail for some reason with an error Tobias thinks it is confusing to figure out which one had the error after the fact. I don't agree that that is hard or confusing. To be clear: The inReplyTo will never be used by client tools. It is only for the above use-case. My concern is that if we add something only for a use-case like that, client tools will start to use it for something it is not designed for. Then we will regret that we did not design something better for the clients actual need (because inReplyTo isn't exactly what they actually need). So, I am strongly in favor of keeping the protocol content to the absolute minimum - only include things which we know client use-cases for. Downstreams abusing CMake script APIs for things they were not intended for is a long term problem for CMake. We should avoid introducing that problem in the server protocol. Especially as inReplyTo, even in its current form is 'fuzzy' to me: Is it in spontaneous messages which are not in response to anything (message() output, change notifications etc)? Does it have 'special' values? Initialization Granularity -------------------------- We talked about what should happen when a client starts the metadata server and what sequence of protocol commands the client has to send to set things up properly. In my implementation, the initial handshake causes the configure step and the compute step to be executed. The client can't cause only the configure step to be executed. This means that an out-of-process cmake-gui or other cache editor: http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16550/focus=16556 using the metadata server would have to do a full handshake instead of 'just' the configure part for example, making it not analogous to the current cmake-gui. So, a granular approach allows the use-case of partial initialization while also making it possible for a client to do 'full initialization' by requesting the 'configure', 'compute' and 'generate' steps explicitly. As with the cmake-gui, it might make sense to specify the 'generate' command as implying the dependee two. We might benefit from some state-machine diagram telling users (and us) what protocol commands are possible in which state. Capabilities ------------ > * Move generators and other static information into a separate cmake > commandline invocation (http://public.kitware.com/Bug/view.php?id=15462): Previously Tobias wanted this only as part of the protocol, not as something that can be emitted with a regular command line request in the same manner as --help for example. I think such capabilities information should be available easily to anyone writing scripts which invoke cmake. Tobias needs that information for Qt Creator because it needs to know what generators are available for example. That is determined when Creator is started each time. Tobias mentioned that such capability queries should be fast - executing lots of tools to find out their versions and capabilities is something that slows down the start up of QtCreator. He reasoned that he should only have to run one cmake - start the server only - not run cmake first to get the capabilities, then run it a second time in server mode. I noted that starting the server just for a query like this is very much likely to be slower than just getting the information on the command line. Additionally, the server can be started later when loading a project instead of when starting QtCreator itself. Tobias was convinced by that. > TODO: Figure out how to get all the generator information, e.g. which > values the flags some generator taken can have (architecture, etc.). > Stephen volunteered to check into this. This is about toolsets, platforms, architectures which I think can be passed to cmake when executing it with Visual Studio generators. I'm not very clear on that stuff. Brad, do you have an overview? Use-case Driven Design ---------------------- > 2.6) project (Return project structure) > > * Rename to "codemodel" to make the use-case for the data more clear. One of the differences between the protocol proposed by Tobias is that he has one protocol command which returns a structure of all targets, their files and the includes, defines and flags of the files. In contrast, I had implemented an approach which provides multiple protocol commands returning more-granular structures: http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16589/focus=16640 In the discussion we realized that the two approaches satisfy two different use-cases. Tobias needs to get all C++ source files and the includes, defines and flags in order to populate a C++ code model for the project. That allows things like 'find usages of this symbol in the project' etc. A granular way of returning this information to the client would be inconvenient and ineffective. Whereas the use-case for the structure Tobias wants has a use-case of populating a C++ code model, my proposed protocol commands for target_info, file_info etc are for the use-case of understanding the CMake code. That is why the 'codeModel' in Tobias' proposal does not list include directories, compile definitions etc of targets, but only of files. The code model cares about compilations. It is not concerned with targets. The protocol commands I proposed are for helping the user to understand what the effect of CMake code like target_include_directories(MyTarget ${SOME_VAR}) target_compile_definitions(MyTarget ${SOME_OTHER_VAR}) target_link_libraries(MyTarget ${YET_ANOTHER_VAR}) means. The variables (and transitive usage requirements) make the impact of those lines invisible in the code, as my video demonstrates. Those kinds of protocol commands and features are in scope for the protocol, and perhaps also for QtCreator some day, but they are out of scope for now. As a result of that part of the discussion, I think it might make sense to try to use 'use-case driven design' when trying to decide what should be in the protocol. There may be some element of 'redundancy' occurring in the protocol as a result. For example, both the 'codeModel' protocol command and the 'target_info' would return 'stuff (including object sources in both cases) associated with a target'. We don't yet know if that is a problem. Cache ----- > 2.7 cache (report contents of CMakeCache.txt file) > > * Review by other potential users would be appreciated, but no obvious > problems seen. We thought this would be useful for the work of Daniel on his cache-editor tool, and so feedback from him and Clinton (as cmake-gui is potentially another user of this stuff) would be valued. Other Movers ------------ Microsoft just published a JSON protocol for language servers to get information into VSCode, and for other user tools to do the same. https://www.reddit.com/r/programming/comments/4qad14 This could be interesting to learn from. Although they say the protocol is JSON, I can only seem to find TypeScript protocol stuff. Presumably that gets translated to JSON for wire communication, but that's not clear from their docs. https://github.com/google/xi-editor is another editor based on a JSON protocol with a language server. Neovim too: https://github.com/neovim/neovim/wiki/Plugin-UI-architecture These and other examples indicate that user tools are moving towards multi- process with some defined protocols. I don't think the generic protocol from Microsoft is a replacement for the cmake metadata server protocol, but there are certainly going to be lessons to learn from there. C++11 ----- One central difference between the cmake metadata server code and the rest of CMake is that I used cxx_auto_type and cxx_range_for extensively for my sanity and productivity. I don't see any problem with requiring CMake to be built with a recent compiler in order to conditionally support server mode. QtCreator also requires some C++11 features and it is likely the first user of the protocol and the only major one for some time. Thanks, Steve. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers