+1 for using libclang or clangd for parsing. C++ grammars are complicated to create and maintain and I could not find any grammar spec for C++. Note that the cnd implementation has some support for clang backend in the libs.clank package, so I guess LSP implementation could be plugged into existing cnd via this route. Lets use LSP only for getting AST. Rest of cnd builds on top of this to provide indexing, code completion and various types of project, including remote build and debug.
Siddhesh ________________________________ From: antonio <[email protected]> Sent: Saturday, April 10, 2021 6:36 PM To: [email protected] <[email protected]> Subject: Re: [DISCUSS] CND Next Steps Hi Jan, More comments below. El 10/4/21 a las 22:18, Jan Lahoda escribió: > Hi Antonio, > > A few comments below: > > On Sat, Apr 10, 2021 at 7:30 PM antonio <[email protected]> wrote: > >> >> And then recreate the runtime portions of "cnd.antlr" that give support >> to the running compiled grammars. By doing this we'll be able to ship >> CND relatively soon (the time it takes to setup the compilation part and >> to recreate the cnd.antlr runtime classes). >> > > I didn't dig very deep, but is the cnd.antlr really needed at runtime? I > know the project dependencies say yes, but when I was experimenting with > this, I hoped it is only needed at compile time, and the normal antlr > runtime is used at runtime (ide/libs.antlr3.runtime). We hopefully could > simply use the cnd.antlr at compile-time for some time. The grammars themselves import some org.netbeans.modules.cnd.antlr[*|.collections.*|.debug.misc.*], so there's a runtime dependency, yes. https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnetbeans%2Fblob%2Fe6d890a5f0bfff8ccee7c3bb49a821fb5e875900%2Fcnd%2Fcnd.apt%2Fsrc%2Forg%2Fnetbeans%2Fmodules%2Fcnd%2Fapt%2Fimpl%2Fsupport%2Faptexpr.g%23L32&data=04%7C01%7Crane.si%40northeastern.edu%7C3a341ee6d73b41df668708d8fc7115e3%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637536909919163759%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FS%2F0nLYmm8XFztO1smseFAMcWaTkesUDj9EI3H8WNx8%3D&reserved=0 https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fnetbeans%2Fblob%2Fe6d890a5f0bfff8ccee7c3bb49a821fb5e875900%2Fcnd%2Fcnd.apt%2Fsrc%2Forg%2Fnetbeans%2Fmodules%2Fcnd%2Fapt%2Fimpl%2Fsupport%2FaptBigIntegerExpr.g%23L33&data=04%7C01%7Crane.si%40northeastern.edu%7C3a341ee6d73b41df668708d8fc7115e3%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637536909919173753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=avgBcAJW3Ai1OcHd7%2BCJhe5rZ%2Bsxw8RvOAg63fh%2FkuA%3D&reserved=0 It seems there're some Token, TokenStream, TokenBuffer & friends living in those packages also, and that are used elsewhere. Can't tell the difference between these Token* classes and the ones supplied with the antlr runtime. What I mean is that maybe it's worth exploring the difficulty of building a runtime. >> >> O.2 - Option 2: Upgrade the grammars >> >> The second option could be to migrate the [GRAMMARS] to antlrv4. This >> will possibly take more time, but will allow us to have more modern >> grammars that are easier to maintain. Of course many modules (cnd.asm, >> cnd.apt and cnd.kit, possibly others) will have to be recreated to >> accept the change. >> >> O.3 - Option 3: Use clang/libclang/LSP >> >> The third option could be to delegate the lexing/parsing to third >> parties that are more knowledgeable of C/C++/Objective C internals. >> > > Which is more or less what cpplite is trying to do, via LSP using either > ccls or clangd. > > I guess my worry is this: who will be fixing bugs and updating the grammars > as C/C++ evolves? If there's someone willing to maintain the grammars and > features, I am all for it, but it sounds like a lot of work. Using ccls (or > clangd) seemed like something that can be made to work and maintain in a > limited amount of time. We agree here. Maintaining "cnd.antlr" grammars (or antlr4 grammars) is a lot of work, and requires knowledge of both the grammar language itself and of the target language(s). If we're lucky we can get one or two people doing this in Apache NetBeans. Compare that with 10-100 people doing the same thing in clangd/libclang, or any other similar tool. > > The downside of using a third-party tool is that we would be limited to > features supported by the third-party tool, and also it creates less than > ideal out-of-the-box experience. Note that these third-party tools can provide features that could be difficult to achieve with CND, I think. For instance, one could change some compilation flags (#defines, #ifdefs, compiler flags) and see how different portions of the file are highlighted. Also these tools may understand very specific stuff (compiler-specific precompiled headers, such as pch.h in Windows), this is difficult to do in CND (but I've seen some support somewhere). > > Note we can use anything we see fit from the CND donation even if we go > this route (after it goes through IP clearance). > Agreement again. We can take advantage of all the project infrastructure already in CND, etc. > >> It seems "libclang" is a stable API we could use. We could also improve >> the current Cpplite (LSP) and improve it. >> >> Note that if we use LSP or libclang we should start thinking of a good >> solution to build "compilation databases" (possibly from Makefiles too) >> that all clang tools use to do proper parsing. >> > > If we have more complex projects (the cpplite projects are very bare-bones, > and basically only allow the user to say where the compilation database > is), it should be easy to generate the compilation database. It is > basically a list of command lines used to compile the sources. > I can't tell which project-types are most popular in the C++ world nowadays. I think I'd go with Makefile and CMake projects, at least. For Makefiles it seems "Bear" (https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frizsotto%2FBear&data=04%7C01%7Crane.si%40northeastern.edu%7C3a341ee6d73b41df668708d8fc7115e3%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637536909919173753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=u2FZcWihWBatQWtp5YX4bbhbxblLqV6dMWjHSmart68%3D&reserved=0, GPL) is a popular tool to generate compilation databases. CMake latest versions have a flag that generates compilation databases for you. Kind regards, Antonio > Jan > > >> >> And here come the next questions: >> >> Q.3.- Can you think of any other option to O.0...O.3 above? >> >> Q.4.- Which option above (O.0, O.1, O.2, O.3 ... O.N) do you think is >> the best one? >> >> Thanks and apologies for this long email, >> Antonio >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> For further information about the NetBeans mailing lists, visit: >> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FNETBEANS%2FMailing%2Blists&data=04%7C01%7Crane.si%40northeastern.edu%7C3a341ee6d73b41df668708d8fc7115e3%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637536909919173753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8AMtatozDrtCvq%2FtcTVG%2F%2F7l7hOHyyqpI1iHkR2UsPA%3D&reserved=0 >> >> >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] For further information about the NetBeans mailing lists, visit: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FNETBEANS%2FMailing%2Blists&data=04%7C01%7Crane.si%40northeastern.edu%7C3a341ee6d73b41df668708d8fc7115e3%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637536909919173753%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8AMtatozDrtCvq%2FtcTVG%2F%2F7l7hOHyyqpI1iHkR2UsPA%3D&reserved=0
