Hi Antonio, A few comments below:
On Sat, Apr 10, 2021 at 7:30 PM antonio <anto...@vieiro.net> wrote: > Hi all, > > I've been taking a look at the CND donation and I'd like to share with > you some findings, to verify if I'm correct and to discuss what should > be done (I'm afraid this is goning to be a long email). > > Before going any further, you may want to read Jan's instructions for > CND first, available in the README.cnd file in the CND branch: > > [1] > https://github.com/apache/netbeans/blob/cnd/README.cnd > > 1) MISSING SOURCES > > I think we can recreate many of the files missing in the donation listed > in Jan's README.cnd [1] but for these three ones: > > - cnd/cnd.modelimpl ... CPPParserEx.java > - cnd/cnd.modelimpl ... CPPSymbol.java > - cnd/cnd.modelimpl ... cpparser.g > > These three files were probably not donated because their origin could > not be exactly tracked. > > Some research indicates that these files were incorporated to NetBeans a > long time ago, and were originally covered with a very liberal license. > > Further modifications by Sun/Oracle were covered with CDDL/GPL2+CPE, > that we can't use in Apache NetBeans. > > The latest I can track them in NetBeans history are (probably) these > versoions (these don't have latest Sun+Oracle modifications, of course, > and may be poor alternatives to NetBeans 8.2 parsers/lexers): > > [1.1] > > https://github.com/emilianbold/netbeans-releases/blob/ddf7b064d46d371e2d1c69f30176b7ddb11178c1/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/CPPSymbol.java > > [1.2] > > https://github.com/emilianbold/netbeans-releases/blob/ddf7b064d46d371e2d1c69f30176b7ddb11178c1/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/CPPParserEx.java > > [1.3] > > https://github.com/emilianbold/netbeans-releases/blob/ddf7b064d46d371e2d1c69f30176b7ddb11178c1/cnd.modelimpl/src/org/netbeans/modules/cnd/modelimpl/parser/cppparser.g > > And here comes the first question: > > Q1. May we incorporate these three files ([1.1], [1.2], [1.3]) that were > covered by public domain licenses into Apache NetBeans? > > 2) MISSING BINARIES > > Jan's README.cnd [1] also lists three missing binaries required to build > and/or to run CND. > > The first two binaries: > > * [A] 47F0DB803884692F28A26A2FF80A7D6756B091DB-cnd-build-trace-1.0.zip > * [B] E59851B0E49C05D728D5C653E52750FA5B6A8F0E-cnd-rfs-1.0.zip > > Have C/C++ sources that have been donated: > > * [A] https://github.com/apache/netbeans/tree/cnd/cnd/cnd.discovery/tools > * [B] https://github.com/apache/netbeans/tree/cnd/cnd/cnd.remote/tools > > And that can then be built for Windows and Linux with ASF Infrastructure > (MacOS or Solaris builds may require third-party infrastructure). > > The third binary in Jan's README.cnd, > > * [C] 5CAB59D859CAA6598E28131D30DD2E89806DB57F-antlr-3.4.jar > > Is exactly the antlr-3.4 "complete" variant (i.e., the runtime + the > compiler) but can be easily tracked and used (BSD-3 license). > > No questions regarding these binaries, then. > > 3) MISSING MODULE "cnd.antlr" > > Jan's README.cnd [1] also lists a module that was not donated by Oracle: > > * cnd/modules/org-netbeans-modules-cnd-antlr.jar => > cnd/libs.antlr.cnd/external > > This "cnd.antlr" module in NetBeans 8.2 seems to be a fork of Antlr 2 or > Antlr 3. This module is used to _compile_ different grammars in other > modules in CND (this is equivalent to the "antlr.Tool" in antlr > "complete" variants): > > [GRAMMARS] > * cnd.apt (aptexpr.g, aptlexer.g, aptBigIntegerExpr.g) > * cnd.asm (DisScanner.g, IdentScanner.g, ATTSCanner.g) > * cnd.modelimpl (Evaluator.g, CXXParser.g (empty), cppparser.g above, > missing) > > The "cnd.antlr" binary is also used to _run_ the compiled grammars above > (this is equivalent to antlr-runtime libraries) for the cnd.apt, > cnd.asm, cnd.modelimpl _and_ cnd.kit. > > As far as I know we _cannot_ include the binary of NetBeans 8.2 > "cnd.antlr" in Apache NetBeans, neither in the original version nor in > the Oracle's donatin "libs.antlr.cnd", because it's covered by > CDDL/GPLvCE. And here comes the second question: > > Q.2.: We may not include the _binary_ of NetBeans 8.2 "cnd.antlr" in > Apache NetBeans, right? So the current "cnd/libs.antlr.cnd" cannot be > shipped with Apache NetBeans either. > > 4) NEXT STEPS > > Some (I didn't test all) of these [GRAMMARS] don't compile with > antlr.Tool V2.7.2, V3.4 nor V3.5.2. You can try yourself running java > -cp [antlr.jar] antlr.Tool the-grammar-file.g > > This possibly implies that the module "cnd.antlr" has suffered many > variations while inside NetBeans during the years, has significantly > diverged from antlr, and the grammars that once compiled with antlr do > not compile without it right now. > > To clarify: some/all of [GRAMMARS] above can only be compiled with > "cnd.antlr". > > I think we have the following options right now: > > O.0 - Option 0: Keep libs.antlr.cnd > > If the answer of Q.2 above says we can ship with a compiled > "cnd.antlr"/"libs.antlr.cnd" then we can move forward with this "as is". > > O.1 - Option 1: Keep "cnd.antlr" as a compiler and create the runtime > > We could use "cnd.antlr" to _compile_ the grammar files at build time > (even if this is GPL+CPE). > > And then recreate the runtime portions of "cnd.antlr" that give support > to the running compiled grammars. By doing this we'll be able to ship > CND relatively soon (the time it takes to setup the compilation part and > to recreate the cnd.antlr runtime classes). > I didn't dig very deep, but is the cnd.antlr really needed at runtime? I know the project dependencies say yes, but when I was experimenting with this, I hoped it is only needed at compile time, and the normal antlr runtime is used at runtime (ide/libs.antlr3.runtime). We hopefully could simply use the cnd.antlr at compile-time for some time. > > O.2 - Option 2: Upgrade the grammars > > The second option could be to migrate the [GRAMMARS] to antlrv4. This > will possibly take more time, but will allow us to have more modern > grammars that are easier to maintain. Of course many modules (cnd.asm, > cnd.apt and cnd.kit, possibly others) will have to be recreated to > accept the change. > > O.3 - Option 3: Use clang/libclang/LSP > > The third option could be to delegate the lexing/parsing to third > parties that are more knowledgeable of C/C++/Objective C internals. > Which is more or less what cpplite is trying to do, via LSP using either ccls or clangd. I guess my worry is this: who will be fixing bugs and updating the grammars as C/C++ evolves? If there's someone willing to maintain the grammars and features, I am all for it, but it sounds like a lot of work. Using ccls (or clangd) seemed like something that can be made to work and maintain in a limited amount of time. The downside of using a third-party tool is that we would be limited to features supported by the third-party tool, and also it creates less than ideal out-of-the-box experience. Note we can use anything we see fit from the CND donation even if we go this route (after it goes through IP clearance). > It seems "libclang" is a stable API we could use. We could also improve > the current Cpplite (LSP) and improve it. > > Note that if we use LSP or libclang we should start thinking of a good > solution to build "compilation databases" (possibly from Makefiles too) > that all clang tools use to do proper parsing. > If we have more complex projects (the cpplite projects are very bare-bones, and basically only allow the user to say where the compilation database is), it should be easy to generate the compilation database. It is basically a list of command lines used to compile the sources. Jan > > And here come the next questions: > > Q.3.- Can you think of any other option to O.0...O.3 above? > > Q.4.- Which option above (O.0, O.1, O.2, O.3 ... O.N) do you think is > the best one? > > Thanks and apologies for this long email, > Antonio > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@netbeans.apache.org > For additional commands, e-mail: dev-h...@netbeans.apache.org > > For further information about the NetBeans mailing lists, visit: > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > >
