Re: [Vala] lexical analysis using libvala
I am new to Vala, so my remarks might be naive, but: 1) Is there not some kind of source code parser already included in Valadoc that you could reuse (as opposed to writing something from scratch). 2) Is there not and AST parser somewhere in Vala anyway ? 3) Why not parse the vapi files ? 4) Why not derive your analyser from the way the vapi files generator parses the Vala code ? If my remarks are obvious or naive, please ignore them. Serge. On Sat, Jun 18, 2011 at 2:44 PM, Maga Abdurakhmanov maq...@gmail.com wrote: Hi! I'm working on vala plugin for netbeans, and have some questions regarding libvala and parsing (scanning) using libvala. My existing version of plugin is implemented using antlr and it's not easy to maintain and support all the features of vala. So I've come up with the idea to use libvala to parse files and integrate it somehow with java part of plugin. Now i'm trying to parse vala files using libvala, and i've made some progress but got problems with parsing of code comments. Here is my tryout to parse vala files: https://github.com/carbonfx/netbeans-valaproject/blob/libvala/plugin/libvalaproxy/src/valaparser/vala-parser.vala I've made some hacks to get comments from source, and it doesn't looks right for me. For example I'm calling scanner.parse_file_comments(); before each call of read_token method. Is there any way to get comments and correct line and column number for comments using Scanner class? -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Sure I'm looked at the Valadoc (which uses libvala already) and there is AST parser inside vala, which is libvala :) AST doesn't include comments, they are removed as useless for compilation i guess. Also I'm looked into the sources of Scanner class itself and found no way to get all comments without hacks. Sadly i can't derive from Scanner and implement that functionality, there is no overridable virtual methods. On Sat, Jun 18, 2011 at 5:09 PM, Serge Hulne serge.hu...@gmail.com wrote: I am new to Vala, so my remarks might be naive, but: 1) Is there not some kind of source code parser already included in Valadoc that you could reuse (as opposed to writing something from scratch). 2) Is there not and AST parser somewhere in Vala anyway ? 3) Why not parse the vapi files ? 4) Why not derive your analyser from the way the vapi files generator parses the Vala code ? If my remarks are obvious or naive, please ignore them. Serge. On Sat, Jun 18, 2011 at 2:44 PM, Maga Abdurakhmanov maq...@gmail.com wrote: Hi! I'm working on vala plugin for netbeans, and have some questions regarding libvala and parsing (scanning) using libvala. My existing version of plugin is implemented using antlr and it's not easy to maintain and support all the features of vala. So I've come up with the idea to use libvala to parse files and integrate it somehow with java part of plugin. Now i'm trying to parse vala files using libvala, and i've made some progress but got problems with parsing of code comments. Here is my tryout to parse vala files: https://github.com/carbonfx/netbeans-valaproject/blob/libvala/plugin/libvalaproxy/src/valaparser/vala-parser.vala I've made some hacks to get comments from source, and it doesn't looks right for me. For example I'm calling scanner.parse_file_comments(); before each call of read_token method. Is there any way to get comments and correct line and column number for comments using Scanner class? -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
On Sat, Jun 18, 2011 at 05:22:25PM +0400, Maga Abdurakhmanov wrote: Sure I'm looked at the Valadoc (which uses libvala already) and there is AST parser inside vala, which is libvala :) AST doesn't include comments, they are removed as useless for compilation i guess. Also I'm looked into the sources of Scanner class itself and found no way to get all comments without hacks. Sadly i can't derive from Scanner and implement that functionality, there is no overridable virtual methods. Comments are inside ast nodes, look better. -- http://www.debian.org - The Universal Operating System ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Comments are inside ast nodes, look better. There are only some of the comments inside ast nodes (valadoc and first comment in file). To highlight syntax i need to get all of them as tokens. -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
sorry to interfere, i might be naive, but i have a different view here i would advise for an independent from-scratch parser, optimized for a dev tool, for writing code, which is a bit different than for the compiler. the parser has to be permissive, it has to figure out the best it can incomplete and illegal structures, so that the code completion/highlight doesn't brake all the time while you write. On Sat, Jun 18, 2011 at 8:51 PM, Maga Abdurakhmanov maq...@gmail.comwrote: Comments are inside ast nodes, look better. There are only some of the comments inside ast nodes (valadoc and first comment in file). To highlight syntax i need to get all of them as tokens. -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Best regards, Victor Dramba Head of Programming at Cabanova www.cabanova.com www.victordramba.com ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Yes it has, but comment property is returned only for doc comments that begins with /*** and they are bound to symbols. So you'll never get in parser regular comments. To implement syntax highlight i need to have them as tokens at lexical analysis stage (Scanner class in libvala). Comments has meaning only at lexical analysis stage, so we can highlight them. But On Sat, Jun 18, 2011 at 9:55 PM, Luca Bruno lethalma...@gmail.com wrote: On Sat, Jun 18, 2011 at 09:51:10PM +0400, Maga Abdurakhmanov wrote: Comments are inside ast nodes, look better. There are only some of the comments inside ast nodes (valadoc and first comment in file). To highlight syntax i need to get all of them as tokens. Vala.Symbol has a comment property, are you referring to statements? -- http://www.debian.org - The Universal Operating System -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Victor, it was the first way i tried out. I've almost implemented lexer and parser for Vala using antlr rules. And there is a basic plugin for netbeans using that lexer. But there still to much things to fix and do and i'm very limited in my time. libvala has already implemented lexer and parser and evolve with the vala itself. And in the future all new constructions of language will be easier to support. On Sat, Jun 18, 2011 at 10:04 PM, Victor D. victor.dra...@cabanova.infowrote: sorry to interfere, i might be naive, but i have a different view here i would advise for an independent from-scratch parser, optimized for a dev tool, for writing code, which is a bit different than for the compiler. the parser has to be permissive, it has to figure out the best it can incomplete and illegal structures, so that the code completion/highlight doesn't brake all the time while you write. On Sat, Jun 18, 2011 at 8:51 PM, Maga Abdurakhmanov maq...@gmail.com wrote: Comments are inside ast nodes, look better. There are only some of the comments inside ast nodes (valadoc and first comment in file). To highlight syntax i need to get all of them as tokens. -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Best regards, Victor Dramba Head of Programming at Cabanova www.cabanova.com www.victordramba.com ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Antlr is an important project. It features grammar files for most languages. Having one for vala is a great asset for vala, I would say. Also, some effort is needed even for using vala's own parser. Probably, one of the reasons I say this is that even know, I'm not able to install a proper ide for vala on my Ubuntu box. The compiler works fine but neither valide nor the gedit plugin would install correctly. So I still hope that more tools will emerge for this great new language. Having an ANTLR grammar for it would be a boost. Personally, I'm a full-time eclipse user and fan. Hope I didn't say too many useless things, if so, feel free to ignore ;) -- Victor On Jun 18, 2011 10:27 PM, Maga Abdurakhmanov maq...@gmail.com wrote: Victor, it was the first way i tried out. I've almost implemented lexer and parser for Vala using antlr rules. And there is a basic plugin for netbeans using that lexer. But there still to much things to fix and do and i'm very limited in my time. libvala has already implemented lexer and parser and evolve with the vala itself. And in the future all new constructions of language will be easier to support. On Sat, Jun 18, 2011 at 10:04 PM, Victor D. victor.dra...@cabanova.info wrote: sorry to interfere, i might be naive, but i have a different view here i would advise for an independent from-scratch parser, optimized for a dev tool, for writing code, which is a bit different than for the compiler. the parser has to be permissive, it has to figure out the best it can incomplete and illegal structures, so that the code completion/highlight doesn't brake all the time while you write. On Sat, Jun 18, 2011 at 8:51 PM, Maga Abdurakhmanov maq...@gmail.com wrote: Comments are inside ast nodes, look better. There are only some of the comments inside ast nodes (valadoc and first comment in file). To highlight syntax i need to get all of them as tokens. -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Best regards, Victor Dramba Head of Programming at Cabanova www.cabanova.com www.victordramba.com ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
Victor D. wrote: sorry to interfere, i might be naive, but i have a different view here i would advise for an independent from-scratch parser, optimized for a dev tool, for writing code, which is a bit different than for the compiler. the parser has to be permissive, it has to figure out the best it can incomplete and illegal structures, so that the code completion/highlight doesn't brake all the time while you write. This type of feature might also be useful in a compiler, so that it can carry on compiling after an error without generating 100s of spurious 'mismatched parens' type errors. If it doesn't distort the job of libvala too much, maybe it could be extended to cover the (modest) additional requirements of a syntax highlighter, but that means someone taking the time to do patches and get them approved and integrated. I think that a flag that controls whether or not all comments are stored in the AST would not create any significant overhead when turned off for compiling. Jim -- Jim Peters (_)/=\~/_(_) j...@uazu.net (_) /=\ ~/_ (_) Uazú (_)/=\~/_(_)http:// in Peru(_) /=\ ~/_ (_)uazu.net ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
Re: [Vala] lexical analysis using libvala
If it doesn't distort the job of libvala too much, maybe it could be extended to cover the (modest) additional requirements of a syntax highlighter, but that means someone taking the time to do patches and get them approved and integrated. At this point i able to get all comments using some hacks with libvala Scanner class. While doing it i've found some bugs, one of them i posted to bugzilla. I think that a flag that controls whether or not all comments are stored in the AST would not create any significant overhead when turned off for compiling. Jim I agree with you. It doesn't look very hard to add some flag, new lexer types for comments and create a patch for it. I can try to do that work. -- Sincerely, Magomed ___ vala-list mailing list vala-list@gnome.org http://mail.gnome.org/mailman/listinfo/vala-list
