[il-antlr-interest: 34936] [antlr-interest] valid grammar does not compile
Hi,
I build the following grammar with antlrworks:
grammar questionmark;
horef
: '\?' ('a'..'z')
;
antlrworks tells me "check grammar succeeded". However, debugging does not
works because the generated code does not compile !?
My question: how does the rule have to look like so that the code compile?
I want to parse text of the form ?a which should be a trivial task for
antlr.
Thanks,
Didi
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34937] Re: [antlr-interest] valid grammar does not compile
On Thu, Nov 17, 2011 at 9:14 AM, D. Frej wrote:
> Hi,
>
> I build the following grammar with antlrworks:
>
> grammar questionmark;
>
> horef
> :'\?' ('a'..'z')
> ;
>
> antlrworks tells me "check grammar succeeded". However, debugging does not
> works because the generated code does not compile !?
>
>
You should not escape the question mark.
Regards,
Bart.
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34938] Re: [antlr-interest] valid grammar does not compile
and still: the compilation error stays even if I do not quote the question
mark
>
> Original-Nachricht
> Datum: Thu, 17 Nov 2011 09:34:45 +0100
> Von: Bart Kiers
> An: "D. Frej"
> CC: antlr-interest@antlr.org
> Betreff: Re: [antlr-interest] valid grammar does not compile
>
> On Thu, Nov 17, 2011 at 9:14 AM, D. Frej
> wrote:
>
> > Hi,
> >
> > I build the following grammar with antlrworks:
> >
> > grammar questionmark;
> >
> > horef
> > : '\?' ('a'..'z')
> > ;
> >
> > antlrworks tells me "check grammar succeeded". However, debugging does
> > not
> > works because the generated code does not compile !?
> >
> >
>
> You should not escape the question mark.
> Regards,
> Bart.
>
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34939] Re: [antlr-interest] valid grammar does not compile
On Thu, Nov 17, 2011 at 10:14 AM, D. Frej wrote: > and still: the compilation error stays even if I do not quote the > question mark > Ah, hold on, you're using the `..` (range) operator inside a parser rule (horef). Either create a lexer rule matching '?' 'a'..'z': Horef :'?' 'a'..'z' ; , or move 'a'..'z' to a lexer rule: horef :'?' Letter ; Letter : 'a'..'z' ; But still, the question mark should not be escaped. Regards, Bart. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34940] [antlr-interest] C# problem with the gettext()
Hi, I am new at ANTLR and I am having a problem trying to use the C# code generated with the Kuka KRL Grammar... I am using antlrworks version 1.4.3 , Visual Studio 2010. While compiling the C# code, Visual Studio claims that it doesn't recognize the getText() that appears like this: input.LT(1).getText(). I have tried to use different Dotnet libraries but it is still not working. Can anyone please help me? -- View this message in context: http://antlr.1301665.n2.nabble.com/C-problem-with-the-gettext-tp7003942p7003942.html Sent from the ANTLR mailing list archive at Nabble.com. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34941] [antlr-interest] Copyright question concerning the code accompanying the "Language Implementation Patterns" book
Hi, I'm trying to build a language to language translator using ANTLR. I'm also using the "Language Implementation Patterns" book to find my way through this task. For building my symbol tables I would like to use some parts of the source code that accompanies this book. Could anybody clarify if I may use this code or what limitations apply. The code carries always a header like this: /*** * Excerpted from "Language Implementation Patterns", * published by The Pragmatic Bookshelf. * Copyrights apply to this code. It may not be used to create training material, * courses, books, articles, and the like. Contact us if you are in doubt. * We make no guarantees that this code is fit for any purpose. * Visit http://www.pragmaticprogrammer.com/titles/tpdsl for more book information. ***/ The program I build is a commercial program (a contract work) for one of our customers. But it is not something what is mentioned above. You may call it a compiler. Unfortunately the antlr.markmail.org server is not servicing anything for me and a Google search didn't reveal anything specific to this topic. Thanks, Stefan List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34942] Re: [antlr-interest] This should be easy - but I can't figure it out
Am 17.11.2011 02:39:05 schrieb(en) John B. Brodie: > On 11/16/2011 12:02 AM, Voelkel, Andy wrote: > > : : : > > array > > : ( l='[' (f+=FLOAT)+ ']' -> ^(ARRAY_FLOAT ["FLT ARY",$l] $f+) ) > > | ( l='[' (s+=STRING)+ ']' -> ^(ARRAY_STRING["STR ARY",$l] $s+) ) > > ; > > > > [Andy - This causes compiler errors, and I'm really not sure what you are > getting at. > > compiler errors are odd here, maybe the C# target does not support the [ > ] notation for initializing an imaginary token? To be specific on this example only: I believe the order of the parameters (in []) in the rewrite section is wrong. First comes the token reference then follows the text for the imaginary node. array : ( l='[' (f+=FLOAT)+ ']' -> ^(ARRAY_FLOAT [$l,"FLT ARY"] $f+) ) | ( l='[' (s+=STRING)+ ']' -> ^(ARRAY_STRING[$l,"STR ARY"] $s+) ) ; See Chapter 7, p. 176 "Deriving Imaginary Nodes from Real Tokens" in the ANTLR Reference Book (print 02). > > > i think this last form will simplify subsequent processing of the tree. > > note also the proper initialization of the imaginary tokens. > > > > [Andy - what do you mean "proper initialization of the imaginary tokens"] > > the stuff between the [ ] on the right hand side of the -> is > information used to initialize the imaginary token. in Java, they get > translated into parameters to its constructor. i refer you to Dr. Parr's > book to find out more about this feature. Regards, Stefan List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34943] Re: [antlr-interest] C# problem with the gettext()
The C# code uses C# conventions in many (hopefully most) places. input.LT(1).Text Sam -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of karen Sent: Thursday, November 17, 2011 6:25 AM To: antlr-interest@antlr.org Subject: [antlr-interest] C# problem with the gettext() Hi, I am new at ANTLR and I am having a problem trying to use the C# code generated with the Kuka KRL Grammar... I am using antlrworks version 1.4.3 , Visual Studio 2010. While compiling the C# code, Visual Studio claims that it doesn't recognize the getText() that appears like this: input.LT(1).getText(). I have tried to use different Dotnet libraries but it is still not working. Can anyone please help me? -- View this message in context: http://antlr.1301665.n2.nabble.com/C-problem-with-the-gettext-tp7003942p7003 942.html Sent from the ANTLR mailing list archive at Nabble.com. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34945] Re: [antlr-interest] valid grammar does not compile
The range 'a'..'z' does not mean the letters a to z but the token numbers
assigned by ANTLR - you need to read the 5 minute getting started articles
in the WIKI or use the search on the support page.
Jim
> -Original Message-
> From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-
> boun...@antlr.org] On Behalf Of D. Frej
> Sent: Thursday, November 17, 2011 12:15 AM
> To: antlr-interest@antlr.org
> Subject: [antlr-interest] valid grammar does not compile
>
> Hi,
>
> I build the following grammar with antlrworks:
>
> grammar questionmark;
>
> horef
> :'\?' ('a'..'z')
> ;
>
> antlrworks tells me "check grammar succeeded". However, debugging does
> not works because the generated code does not compile !?
>
> My question: how does the rule have to look like so that the code
> compile?
> I want to parse text of the form ?a which should be a trivial task for
> antlr.
>
> Thanks,
>
> Didi
> --
> Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie
> mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
>
> List: http://www.antlr.org/mailman/listinfo/antlr-interest
> Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-
> email-address
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34947] [antlr-interest] confused about rendering all String Templates...
I've gotten a lexer, parser (that outputs AST) and a tree parser work. Currently the tree parser contain little code snippets that include System.out.println to print my translated file. Not very pretty, but it works. My next step was to switch to StringTemplates, because I thought it would help me clean up the tree parser. So far my initial efforts haven't yielded much. After reading TP's books and (this morning) about 40 pages of interest messages about templates I'm beginning to think I might be misunderstanding the use of String Templates... I thought I could include a template and after walking my tree I could call getTemplate on my return object and then toString to have it dump out the results of ALL my templates, but as far as I can tell it only prints out the template of the rule I called. I've debugged the resulting .java file and I can step into the results from each rule and see that a correct template is created. But I can't figure out how to "walk" all the templates and have them dump out my results. >From the reading I've done this morning I'm beginning to believe that the templates aren't expected to do this... Can someone clear this up for me? Point me in the direction of a template example that demonstrates this if it is suppose to work. Any help is appreciated. Morgan List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34948] Re: [antlr-interest] confused about rendering all String Templates...
Hi Morgan, Well, I'm a newbie too, but I can tell you how I understand and use StringTemplates. For me, the end result of walking a tree that produces StringTemplates is a StringTemplate which contains elements which are themselves StringTemplates (or lists of String Templates), which contains elements which are themselves StringTemplates (or lists of String Templates), and so on. At this point no "rendering" to actual strings has been done. After the giant StringTemplate amalgam is created, rendering the top level StringTemplate will render all the sub-StringTemplate in turn, all the way down the tree. In a way you have created a different sort of StringTemplate tree during the tree walk, but it is not the AST itself. So as you are walking a tree, you are taking the StringTemplates returned by rules and putting them into StringTemplates of the containing rules, and the top level rule returns the top level StringTemplate. I hope this is clear, and the experts can correct my misunderstandings, - Andy -Original Message- From: antlr-interest-boun...@antlr.org [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Morgan Jones Sent: Thursday, November 17, 2011 8:39 AM To: antlr-interest@antlr.org Subject: [antlr-interest] confused about rendering all String Templates... I've gotten a lexer, parser (that outputs AST) and a tree parser work. Currently the tree parser contain little code snippets that include System.out.println to print my translated file. Not very pretty, but it works. My next step was to switch to StringTemplates, because I thought it would help me clean up the tree parser. So far my initial efforts haven't yielded much. After reading TP's books and (this morning) about 40 pages of interest messages about templates I'm beginning to think I might be misunderstanding the use of String Templates... I thought I could include a template and after walking my tree I could call getTemplate on my return object and then toString to have it dump out the results of ALL my templates, but as far as I can tell it only prints out the template of the rule I called. I've debugged the resulting .java file and I can step into the results from each rule and see that a correct template is created. But I can't figure out how to "walk" all the templates and have them dump out my results. >From the reading I've done this morning I'm beginning to believe that the templates aren't expected to do this... Can someone clear this up for me? Point me in the direction of a template example that demonstrates this if it is suppose to work. Any help is appreciated. Morgan List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it, may contain information that is confidential and/or legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, please DO NOT disclose the contents to another person, store or copy the information in any medium, or use any of the information contained in or attached to this transmission for any purpose. If you have received this transmission in error, please immediately notify the sender by reply email or at priv...@plantronics.com, and destroy the original transmission and its attachments without reading or saving in any manner. For further information about Plantronics - the Company, its products, brands, partners, please visit our website www.plantronics.com. List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34949] Re: [antlr-interest] confused about rendering all String Templates...
Andy, that does help... thanks for replying. as I read about StringTemplates today I gathered they were capable of much more than I had initially realized... I'll go back and try to track down some examples of the nested String Templates you spoke of. Maybe I'll luck out and someone will point me to a concrete example. ;) Morgan On Thu, Nov 17, 2011 at 12:14 PM, Voelkel, Andy wrote: > Hi Morgan, > > Well, I'm a newbie too, but I can tell you how I understand and use > StringTemplates. For me, the end result of walking a tree that produces > StringTemplates is a StringTemplate which contains elements which are > themselves StringTemplates (or lists of String Templates), which contains > elements which are themselves StringTemplates (or lists of String Templates), > and so on. At this point no "rendering" to actual strings has been done. > After the giant StringTemplate amalgam is created, rendering the top level > StringTemplate will render all the sub-StringTemplate in turn, all the way > down the tree. > > In a way you have created a different sort of StringTemplate tree during the > tree walk, but it is not the AST itself. > > So as you are walking a tree, you are taking the StringTemplates returned by > rules and putting them into StringTemplates of the containing rules, and the > top level rule returns the top level StringTemplate. > > I hope this is clear, and the experts can correct my misunderstandings, > > - Andy > > -Original Message- > From: antlr-interest-boun...@antlr.org > [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Morgan Jones > Sent: Thursday, November 17, 2011 8:39 AM > To: antlr-interest@antlr.org > Subject: [antlr-interest] confused about rendering all String Templates... > > I've gotten a lexer, parser (that outputs AST) and a tree parser work. > > Currently the tree parser contain little code snippets that include > System.out.println to print my translated file. > > Not very pretty, but it works. > > My next step was to switch to StringTemplates, because I thought it would > help me clean up the tree parser. > So far my initial efforts haven't yielded much. > > After reading TP's books and (this morning) about 40 pages of interest > messages about templates I'm beginning to think I might be misunderstanding > the use of String Templates... > > I thought I could include a template and after walking my tree I could call > getTemplate on my return object and then toString to have it dump out the > results of ALL my templates, but as far as I can tell it only prints out the > template of the rule I called. > > I've debugged the resulting .java file and I can step into the results from > each rule and see that a correct template is created. > But I can't figure out how to "walk" all the templates and have them dump out > my results. > > >From the reading I've done this morning I'm beginning to believe that > the templates aren't expected to do this... > > Can someone clear this up for me? > Point me in the direction of a template example that demonstrates this if it > is suppose to work. > > Any help is appreciated. > > Morgan > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > > > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or > previous e-mail messages attached to it, may contain information that is > confidential and/or legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, please DO NOT disclose the contents to another person, store or > copy the information in any medium, or use any of the information contained > in or attached to this transmission for any purpose. If you have received > this transmission in error, please immediately notify the sender by reply > email or at priv...@plantronics.com, and destroy the original transmission > and its attachments without reading or saving in any manner. > > For further information about Plantronics - the Company, its products, > brands, partners, please visit our website www.plantronics.com. > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34950] Re: [antlr-interest] confused about rendering all String Templates...
It's very confusing trying to read the detailed documentation on the StringTemplate site until you get the basic idea, but keep at it. A key thing for me was realizing that the objects embedded as "parameters" in a string template can be just about anything that can be rendered to a string, or collections of things that can be rendered to a string, or StringTemplates themselves. That is what makes the recursion possible, and makes them so powerful. Do you have "The Definitive Guide to Antlr"? That's handy, although it is still a leap from his examples to understanding the terse online documentation. - Andy -Original Message- From: morg...@gmail.com [mailto:morg...@gmail.com] On Behalf Of Morgan Jones Sent: Thursday, November 17, 2011 9:43 AM To: Voelkel, Andy Cc: antlr-interest@antlr.org Subject: Re: [antlr-interest] confused about rendering all String Templates... Andy, that does help... thanks for replying. as I read about StringTemplates today I gathered they were capable of much more than I had initially realized... I'll go back and try to track down some examples of the nested String Templates you spoke of. Maybe I'll luck out and someone will point me to a concrete example. ;) Morgan On Thu, Nov 17, 2011 at 12:14 PM, Voelkel, Andy wrote: > Hi Morgan, > > Well, I'm a newbie too, but I can tell you how I understand and use > StringTemplates. For me, the end result of walking a tree that produces > StringTemplates is a StringTemplate which contains elements which are > themselves StringTemplates (or lists of String Templates), which contains > elements which are themselves StringTemplates (or lists of String Templates), > and so on. At this point no "rendering" to actual strings has been done. > After the giant StringTemplate amalgam is created, rendering the top level > StringTemplate will render all the sub-StringTemplate in turn, all the way > down the tree. > > In a way you have created a different sort of StringTemplate tree during the > tree walk, but it is not the AST itself. > > So as you are walking a tree, you are taking the StringTemplates returned by > rules and putting them into StringTemplates of the containing rules, and the > top level rule returns the top level StringTemplate. > > I hope this is clear, and the experts can correct my > misunderstandings, > > - Andy > > -Original Message- > From: antlr-interest-boun...@antlr.org > [mailto:antlr-interest-boun...@antlr.org] On Behalf Of Morgan Jones > Sent: Thursday, November 17, 2011 8:39 AM > To: antlr-interest@antlr.org > Subject: [antlr-interest] confused about rendering all String Templates... > > I've gotten a lexer, parser (that outputs AST) and a tree parser work. > > Currently the tree parser contain little code snippets that include > System.out.println to print my translated file. > > Not very pretty, but it works. > > My next step was to switch to StringTemplates, because I thought it would > help me clean up the tree parser. > So far my initial efforts haven't yielded much. > > After reading TP's books and (this morning) about 40 pages of interest > messages about templates I'm beginning to think I might be misunderstanding > the use of String Templates... > > I thought I could include a template and after walking my tree I could call > getTemplate on my return object and then toString to have it dump out the > results of ALL my templates, but as far as I can tell it only prints out the > template of the rule I called. > > I've debugged the resulting .java file and I can step into the results from > each rule and see that a correct template is created. > But I can't figure out how to "walk" all the templates and have them dump out > my results. > > >From the reading I've done this morning I'm beginning to believe that > the templates aren't expected to do this... > > Can someone clear this up for me? > Point me in the direction of a template example that demonstrates this if it > is suppose to work. > > Any help is appreciated. > > Morgan > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > > > > > CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or > previous e-mail messages attached to it, may contain information that is > confidential and/or legally privileged. If you are not the intended > recipient, or a person responsible for delivering it to the intended > recipient, please DO NOT disclose the contents to another person, store or > copy the information in any medium, or use any of the information contained > in or attached to this transmission for any purpose. If you have received > this transmission in error, please immediately notify the sender by reply > email or at priv...@plantronics.com, and destroy the original transmission > and its attachments without reading or saving in any manner. > > For
[il-antlr-interest: 34953] Re: [antlr-interest] reuse() methos in 3.4 C runtime
On 6/24/11 7:49 PM, "Jim Idle" wrote:
Hi Jim,
I will send few letters now, but I will start with simple question to this
your letter.
Am I right, that we also in this loop should kill and create again
TreeParser?
Yes?
> Because the documentation is not yet up to date, here is an example of
> reusing the allocated memory in input streams and token streams:
>
>
>
> for (i=0; i{
> // Run the parser.
> //
> psr->start(psr);
>
> // --
> // Now reset everything for the next run.
> // Order of calls is important.
>
> // Input stream can now be reused
> //
> input->reuse(input, sourceCode, sourceLen, sourceName);
>
> // Reset the common token stream so that it will reuse its resources
> //
> tstream->reset(tstream);
>
> // Reset the lexer (new function generated by antlr now)
> //
> lxr->reset(lxr);
>
> // Reset the parser (new function generated by antlr now)
> //
> psr->reset(psr);
> }
--
Best regards,
Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc
Valentina - Joining Worlds of Information
http://www.paradigmasoft.com
[I feel the need: the need for speed]
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34954] Re: [antlr-interest] [C] my v3 Parser no reuse() slower 20% than v2. With reuse() 2GB leaks, oops.
Hi Jim, Thank you for your feedback, and I have update now... 1) I was able remove all .text usage in both Parser and TreeParser. GOOD. 2) BAD ... This have save 500MB, but I still have 1.5GB of allocations in my bench ... And now I see (using Apple Instruments) that all this is eaten by PARSER. Not by Lexer, and not by TreeParser. I just see endless newPool newPool newPool newPool newPool newPool I will send you snapshoot off list so you can see that. And now there is ZERO my code, which affect this. Only ANTLR own logic... This makes me think, that reuse() do not work as expected. As I understand, when we do parser( reset ) It must mark all existed allocations as free in your pool, So next run should reuse all that. Yes? And note, that all my calls to parser, are very similar by size. This is just INSERT INTO( f1, f,2 ... f9 ) VALUES ( v1, v2, ... ) I.e. Pool really should not grow much after first / second iteration of loop. But it grows like crazy. I think you have own test app where you did test this ... May be just increase loop count to million or such To see that RAM on your computer go away ... I very hope you will be able find issue and show how fix it in sources of ANTLR 3.4 ... Please? May be some kind of objects is not marked as free ? == Also interesting fact. v3 without reuse 22.4 sec v3 with reuse and 1.5GB allocation 20.4 sec v2 with reuse 19.7 So if we will be able resolve this 1.5GB "leaks", there is yet hope to be at least not slower of v2 ... === About your hope that V3 C should be much faster of v2 C++ So far I do not see this. I see in profiles, parser 36%RAM only tree parser24%RAM only execute of vdb engine 13% insert recs into disk (!!) db And when I am starting go deep by parser calls ... I just see that deep is big sql -> sql_single -> And each step down just reduce 0.5-0.8% ... This is BODY of each rule of parser ... And nothing really to optimize :( Just a lots of small calls ... NilNodes, LT(), ... === My vision is that this is Nature of ANTLR ... We get many calls of parser funcs ... Deep stack ... Although they are light they eat milliseconds ... And fact that in C you need create structures with huge number of pointers sometimes, then e.g NULL them, in C++ virtual table of methods is created once per class, not once per instances ... This fact can be one of hidden bottleneck IMO. You can workaround this, if also will extract pointers into single separate structure, so instances will have just a single pointer. -- Best regards, Ruslan Zasukhin VP Engineering and New Technology Paradigma Software, Inc Valentina - Joining Worlds of Information http://www.paradigmasoft.com [I feel the need: the need for speed] List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34956] Re: [antlr-interest] reuse() methos in 3.4 C runtime
Hi Jim,
Below are copy-pastes of my class-wrapper around ANTLR3
Lexer/Parser/TreeParser.
So you can see if I made some stupid mistake...
>> On 6/24/11 7:49 PM, "Jim Idle" wrote:
>>
>> Because the documentation is not yet up to date, here is an example of
>> reusing the allocated memory in input streams and token streams:
>>
>> for (i=0; i>{
>> // Run the parser.
>> psr->start(psr);
>>
>> // --
>> // Now reset everything for the next run.
>> // Order of calls is important.
>>
>> // Input stream can now be reused
>> input->reuse(input, sourceCode, sourceLen, sourceName);
>>
>> // Reset the common token stream so that it will reuse its resources
>> tstream->reset(tstream);
>>
>> // Reset the lexer (new function generated by antlr now)
>> lxr->reset(lxr);
>>
>> // Reset the parser (new function generated by antlr now)
>> psr->reset(psr);
>> }
/**
void SqlParser_v3::ResuseParserObjects(
const char*inTextToParse,
vuint32inLength )
{
// ---
// TREE PARSER cannot be reused. Destroy it.
//
if( mpTreeParser )
{
mpTreeParser->free( mpTreeParser );
mpTreeParser = NULL;
}
if( mpNodes )
{
mpNodes->free( mpNodes );
mpNodes = NULL;
}
// ---
// Reuse other objects
//
mpInput->reuse(
mpInput,
(pANTLR3_UINT8) inTextToParse,
(ANTLR3_UINT32) inLength,
(pANTLR3_UINT8) "VSQL" );
mpTokenStream->reset( mpTokenStream );
mpLexer ->reset( mpLexer );
mpParser ->reset( mpParser );
ResetOwnData( mpParser );
}
And few other related methods ...
/**
void SqlParser_v3::Parse_UTF8(
I_SqlDatabaseEx* inDatabase,
const char*inCommand,
const char*inCommandEnd )
{
argused1(inDatabase);
//COMMENT this line to force REUSE() mode ...
//DestroyParserObjects();
if( mpInput )
ResuseParserObjects( inCommand, (inCommandEnd - inCommand) );
else
CreateParserObjects( inCommand, (inCommandEnd - inCommand) );
// -
// Parse the input expression
mAST = mpParser->sql( mpParser );
// IF PARSER have generate some errors,
// then we throw them as VSQL exception.
if( mpParser->pParser->rec->state->errorCount )
{
StToUTF16 cnv( ResultStringBuffer, pErrEnd, GetConverter_UTF8() );
throw VSQL::xVSQLException( ERR_SQL_PARSER_ERROR, cnv.c_str() );
}
}
/**
void SqlParser_v3::CreateParserObjects(
const char*inTextToParse,
vuint32inLength )
{
if( inTextToParse == NULL )
return; // all objects will be still NULLs also.
// --
// Create INPUT object:
// NOTE: SQL strings do not have BOM - first few bytes, which define
endian of UTF16.
// So for UTF16, we must here self specify BE or LE.
mpInput = antlr3StringStreamNew(
(pANTLR3_UINT8) inTextToParse, mEncoding, (ANTLR3_UINT32) inLength,
(pANTLR3_UINT8) "VSQL" );
mpInput->setUcaseLA( mpInput, ANTLR3_TRUE );
// --
// Create LEXER v3 object:
mpLexer = SqlParser_v3LexerNew( mpInput );
mpTokenStream = antlr3CommonTokenStreamSourceNew( ANTLR3_SIZE_HINT,
TOKENSOURCE( mpLexer ) );
// --
// Create PARSER v3 object:
mpParser = SqlParser_v3ParserNew( mpTokenStream ); // is generated by
ANTLR3
mpParser->mDoAllCommands = mDoAllCommandsInitial;
ResetOwnData( mpParser );
/**
void SqlParser_v3::DestroyParserObjects( void )
{
// REVERSE ORDER to construction:
if( mpTreeParser )
{
mpTreeParser->free( mpTreeParser );
mpTreeParser = NULL;
}
if( mpNodes )
{
mpNodes->free( mpNodes );
mpNodes = NULL;
}
if( mpParser )
{
mpParser->mpStartPositions = NULL;
mpParser->free( mpParser );
mpParser = NULL;
}
if( mpTokenStream )
{
mpTokenStream->free( mpTokenStream );
mpTokenStream = NULL;
}
if( mpLexer )
{
mpLexer->free( mpLexer );
mpLexer = NULL;
}
if( mpInput )
{
mpInput->close( mpInput );
mpInput = NULL;
}
}
--
Best regards,
Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc
Valentina - Joining Worlds of Information
http://www.paradigmasoft.com
[I
[il-antlr-interest: 34957] Re: [antlr-interest] reuse() methos in 3.4 C runtime
yes
> -Original Message-
> From: Ruslan Zasukhin [mailto:ruslan_zasuk...@valentina-db.com]
> Sent: Thursday, November 17, 2011 12:21 PM
> To: Jim Idle; antlr-interest@antlr.org
> Subject: Re: [antlr-interest] reuse() methos in 3.4 C runtime
>
> On 6/24/11 7:49 PM, "Jim Idle" wrote:
>
> Hi Jim,
>
> I will send few letters now, but I will start with simple question to
> this your letter.
>
> Am I right, that we also in this loop should kill and create again
> TreeParser?
>
> Yes?
>
>
> > Because the documentation is not yet up to date, here is an example
> of
> > reusing the allocated memory in input streams and token streams:
> >
> >
> >
> > for (i=0; i >{
> > // Run the parser.
> > //
> > psr->start(psr);
> >
> > // --
> > // Now reset everything for the next run.
> > // Order of calls is important.
> >
> > // Input stream can now be reused
> > //
> > input->reuse(input, sourceCode, sourceLen, sourceName);
> >
> > // Reset the common token stream so that it will reuse its
> resources
> > //
> > tstream->reset(tstream);
> >
> > // Reset the lexer (new function generated by antlr now)
> > //
> > lxr->reset(lxr);
> >
> > // Reset the parser (new function generated by antlr now)
> > //
> > psr->reset(psr);
> > }
>
>
>
>
> --
> Best regards,
>
> Ruslan Zasukhin
> VP Engineering and New Technology
> Paradigma Software, Inc
>
> Valentina - Joining Worlds of Information http://www.paradigmasoft.com
>
> [I feel the need: the need for speed]
>
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34958] Re: [antlr-interest] [C] my v3 Parser no reuse() slower 20% than v2. With reuse() 2GB leaks, oops.
You should not be seeing more than a few newPool calls, however, if you are building a tree then this may be affecting it. The reuse stuff was not built for trees, so you may have to debug this because I won't have time to look at new use cases for some time. I will take out the myriad duplication of function pointers over the new year all being well. Jim > -Original Message- > From: Ruslan Zasukhin [mailto:ruslan_zasuk...@valentina-db.com] > Sent: Thursday, November 17, 2011 12:43 PM > To: Jim Idle; antlr-interest@antlr.org > Subject: Re: [antlr-interest] [C] my v3 Parser no reuse() slower 20% > than v2. With reuse() 2GB leaks, oops. > > Hi Jim, > > Thank you for your feedback, and I have update now... > > 1) I was able remove all .text usage in both Parser and TreeParser. > GOOD. > > 2) BAD ... This have save 500MB, >but I still have 1.5GB of allocations in my bench ... > > And now I see (using Apple Instruments) that all this is eaten by > PARSER. > Not by Lexer, and not by TreeParser. > > I just see endless > newPool > newPool > newPool > newPool > newPool > newPool > > I will send you snapshoot off list so you can see that. > > And now there is ZERO my code, which affect this. > Only ANTLR own logic... > > This makes me think, that reuse() do not work as expected. > > As I understand, when we do > parser( reset ) > > It must mark all existed allocations as free in your pool, So next run > should reuse all that. Yes? > > And note, that all my calls to parser, are very similar by size. > This is just > INSERT INTO( f1, f,2 ... f9 ) VALUES ( v1, v2, ... ) > > I.e. Pool really should not grow much after first / second iteration of > loop. But it grows like crazy. > > > I think you have own test app where you did test this ... > May be just increase loop count to million or such To see that RAM on > your computer go away ... > > > I very hope you will be able find issue and show how fix it in sources > of ANTLR 3.4 ... Please? May be some kind of objects is not marked as > free ? > > > == > Also interesting fact. > > v3 without reuse 22.4 sec > v3 with reuse and 1.5GB allocation 20.4 sec > > v2 with reuse 19.7 > > > So if we will be able resolve this 1.5GB "leaks", there is yet hope to > be at least not slower of v2 ... > > > === > About your hope that V3 C should be much faster of v2 C++ So far I do > not see this. > > I see in profiles, > parser 36%RAM only > tree parser24%RAM only > execute of vdb engine 13% insert recs into disk (!!) db > > And when I am starting go deep by parser calls ... I just see that deep > is big > sql -> sql_single -> > > And each step down just reduce 0.5-0.8% ... > > This is BODY of each rule of parser ... > > And nothing really to optimize :( > Just a lots of small calls ... NilNodes, LT(), ... > > > === > My vision is that this is Nature of ANTLR ... We get many calls of > parser funcs ... Deep stack ... Although they are light they eat > milliseconds ... > > > And fact that in C you need create structures with huge number of > pointers sometimes, then e.g NULL them, in C++ virtual table of > methods is created once per class, not once per instances ... This fact > can be one of hidden bottleneck IMO. You can workaround this, if also > will extract pointers into single separate structure, so instances will > have just a single pointer. > > > -- > Best regards, > > Ruslan Zasukhin > VP Engineering and New Technology > Paradigma Software, Inc > > Valentina - Joining Worlds of Information http://www.paradigmasoft.com > > [I feel the need: the need for speed] > List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34959] Re: [antlr-interest] [C] my v3 Parser no reuse() slower 20% than v2. With reuse() 2GB leaks, oops.
On 11/18/11 1:24 AM, "Jim Idle" wrote:
Hi Jim,
> You should not be seeing more than a few newPool calls, however, if you
> are building a tree then this may be affecting it.
You mean
A) my own tree in the parser ?
no, I do not build. Work ANTLR itself to build AST
B) tree parser?
but how this affect?
and everybody need tree parser ...
> The reuse stuff was not built for trees,
Right. This is why in my Reuse() func you can see that I destroy tree
parser, then it is created again later ...
**
void SqlParser_v3::ResuseParserObjects(
const char*inTextToParse,
vuint32inLength )
{
// ---
// TREE PARSER cannot be reused. Destroy it.
//
if( mpTreeParser )
{
mpTreeParser->free( mpTreeParser );
mpTreeParser = NULL;
}
if( mpNodes )
{
mpNodes->free( mpNodes );
mpNodes = NULL;
}
// ---
// Reuse other objects
//
mpInput->reuse(
mpInput,
(pANTLR3_UINT8) inTextToParse,
(ANTLR3_UINT32) inLength,
(pANTLR3_UINT8) "VSQL" );
mpTokenStream->reset( mpTokenStream );
mpLexer ->reset( mpLexer );
mpParser ->reset( mpParser );
ResetOwnData( mpParser );
}
> so you may have to debug this because I won't have time
> to look at new use cases for some time.
Yes, I am going now spend time to see how work parser->reset() and others
here.
But can you at least give me points, what I should see ?
You set some flags for objects of parser?
> I will take out the myriad duplication of function pointers over the new
> year all being well.
This is speed ok. But memory ...
--
Best regards,
Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc
Valentina - Joining Worlds of Information
http://www.paradigmasoft.com
[I feel the need: the need for speed]
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34960] [antlr-interest] Failure to ignore newline
Hi -
This should be a very simple thing - I'm attempting to have my grammar hide
newline, carriage returns, etc. However, every concievable form of a
grammar that attempts to skip over these things or send them to the hidden
channel seems to fail for me. Here's a very basic example:
grammar Test;
prog: ID+;
ID: 'a'..'z'+;
WS: '\n'+ {$channel=HIDDEN;};
// Input: a \n b
// Output: a n b
--
David Riddle
Servoy Developer
(510) 854-6221
www.mcgilly.com
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 34961] Re: [antlr-interest] Failure to ignore newline
Morning David,
On 18/11/11 06:55, David Riddle wrote:
grammar Test;
prog: ID+;
ID: 'a'..'z'+;
WS: '\n'+ {$channel=HIDDEN;};
I just pasted your grammer into ANTLRWorks and it seemed happy. In the
interpreter I entered the following:
a
b
With a real newline between them, it gave the correct output. See
attached - if this list allows attachments, failing that I got:
|
prog
|
---
| |
a b
Are you sure you used a real newline as opposed to the two characters "/n"?
Cheers,
Norm.
Disclaimer: I'm not a compiler writer nor do I play one on TV.
--
Norman Dunbar
Dunbar IT Consultants Ltd
Registered address:
Thorpe House
61 Richardshaw Lane
Pudsey
West Yorkshire
United Kingdom
LS28 7EL
Company Number: 05132767
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
<>--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
