[il-antlr-interest: 33832] Re: [antlr-interest] reuse() methos in 3.4 C runtime
On 6/24/11 7:49 PM, "Jim Idle" wrote: Hi Jim, I do now third attempt switch from ANTLR2 to ANTLR3, And reuse is very important for us, because we do parse SQL strings in the DBMS servers ... I am very happy to read it should work now again in 3.4! But I have got question after this your letter. You say that TreeParser cannot be reused. Then: A) what do method reset() in the generated TreeParser? can/should we use it? If yes when and for what? B) Do you mean that on each loop we must free() Nodes and TreeParser nodes ->free( nodes );nodes = NULL; treePsr ->free( treePsr ); treePsr = NULL; And then create them again nodes = antlr3CommonTreeNodeStreamNewTree(langAST.tree, ANTLR3_SIZE_HINT); treePsr = SqlTreeParser_v3New( nodes ); Or I can just do on each loop treePsr->reset( treePsr ); Thank you in advance for explain > Because the documentation is not yet up to date, here is an example of > reusing the allocated memory in input streams and token streams: > > > > for (i=0; i > > > // Run the parser. > > // > > psr->start(psr); > > > > // -- > > // Now reset everything for the next run. > > // Order of calls is important. > > > > // Input stream can now be reused > > // > > input->reuse(input, sourceCode, sourceLen, sourceName); > > > > // Reset the common token stream so that it will reuse its resources > > // > > tstream->reset(tstream); > > > > // Reset the lexer (new function generated by antlr now) > > // > > lxr->reset(lxr); > > > > // Reset the parser (new function generated by antlr now) > > // > > psr->reset(psr); > > } > > > > Note that tree parsers cannot reuse their allocations but this is rarely an > issue. The input->reuse() will reuse any memory it has allocated, but > requires that you handle the reading of the input files (or otherwise supply > a pointer to them). The input files are assumed to be encoded in the way > that the original input was created, for instance: > > > > input = antlr3FileStreamNew(fname, ANTLR3_ENC_8BIT); > > > > Then all reused input must be 8 bit encoded. > > > > Jim > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: > http://www.antlr.org/mailman/options/antlr-interest/your-email-address > -- Best regards, Ruslan Zasukhin VP Engineering and New Technology Paradigma Software, Inc Valentina - Joining Worlds of Information http://www.paradigmasoft.com [I feel the need: the need for speed] List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33834] Re: [antlr-interest] ANTLR IDE 2.1.2 is out! - Problem...
Hello Edgar, > A new version of ANTLR IDE is available for download. This version > includes: >* ANTLR 3.4 support >* Eclipse Indigo support. yep, really a great tool. Many thanks for it.- I have the problem that no Java-files (lexer/parser) are created after saving a grammar: - using Indigo/64Bit - is it sufficient to use ...\antlr-3.4\lib\antlr-3.4-complete.jar as installed package - ...\antlr-3.4\lib\ contains just that file: antlr-3.4-complete.jar - I created a new Java-project and added ANTLR via File -> New -> Other -> ANTLR -> Combined Grammar On http://antlrv3ide.sourceforge.net/doc/gettingstarted/basictutorial/addingantlrsupport.html ANTLR is shown in the file menu above "Properties". - I can run the grammar with F5 (or via the corresponding menu button), a diagram is shown. But the buttons "Run (Java)" and "Debug (Java)" are grayed (because the lexer/parser-files aren't created?) - I can see ANTLR via Windows -> Preferences, but not via Project -> Properties Any hints? When you need mor infos pls. ask! Concerning your www.pages: When I open an HTML-document from http://antlrv3ide.sourceforge.net/ tab "Documentation" and press the "Back"-button of FF, the tab "Welcome" is then shown. Many thanks and greetings Udo List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33836] [antlr-interest] [C] Crashes if NULL name in antlr3StringStreamNew()
If I call
input = antlr3StringStreamNew(
input_string, Encoding, input_len, NULL );
NULL is name of stream
Then it crashes inside of
newStr8(pANTLR3_STRING_FACTORY factory, pANTLR3_UINT8 ptr)
{
return factory->newPtr8(factory, ptr,
(ANTLR3_UINT32)strlen((const char *)ptr));<< crash
}
Because on mac and linux strlen() crashes on NULL.
--
Best regards,
Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc
Valentina - Joining Worlds of Information
http://www.paradigmasoft.com
[I feel the need: the need for speed]
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33837] [antlr-interest] Multiple newlines question
Hi. I am very new to parser generators, BNF grammars and stuff. I am reading
The definitive ANTLR reference and try things out. Currently I have the
following simple grammar:
grammar Test;
@header {
package test;
}
@lexer::header {
package test;
}
program:
INT (NEWLINE+ INT)* EOF;
INT: '0'..'9'+;
NEWLINE: '\r'?'\n';
WS: (' ' | '\t' | '\r' | '\n')+ {skip();};
and the following test (TestNG):
package test;
import org.antlr.runtime.ANTLRStringStream;
import org.antlr.runtime.CommonTokenStream;
import org.antlr.runtime.TokenStream;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class ANTLRTest {
@DataProvider(name = "data")
public Object[][] getData() {
return new Object[][] {
{ "1\n2\n\n3" }
};
}
@Test(dataProvider = "data")
public void test(String value) throws Exception {
ANTLRStringStream in = new ANTLRStringStream(value);
TestLexer l = new TestLexer(in);
TokenStream ts = new CommonTokenStream(l);
TestParser p = new TestParser(ts);
p.program();
}
}
So, a program is an INT, followed by a sequence of one or many NEWLINES
followed by an INT; at the very end is an EOF (I think I have to do it to
force ANTLR read the whole input?). However, I get the following error
printed out to stderr:
line 4:0 extraneous input '3' expecting EOF
The parser doesn't like the two adjacent newlines. If I leave only one, it
works without errors, but I would like it to be able to deal with multiple
newlines.
Could anybody more experienced with me please take a look and tell me what I
am doing wrong here?
Q2. The same rules (almost - the newlines) are in the NEWLINE and WS tokens
- how does it work that once the newlines are matched and sometimes skipped?
Is the sequence in the grammar file important in this case?
Q3. For some reason, I get the errors printed out on System.err, and the
test passes even with errors. This is not exactly what I would need, I would
need it to fail laudly on any erros (be it lexical or parsing). Any quick
pointer would be nice, but if not, I guess the book will deal cover it
sooner or later.
Best regards,
wujek
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33841] [antlr-interest] [C target] bug in hidden syntactic predicates
Hi,
I've been using antlr for a few months in order to parse logical
formulae, with the C target.
I have a rule in a parser such as
subformula:
atom
| term (x='and' term)+ -> ^($x term+)
| term ('or' term)+-> ^('or' term+)
| term
;
As far as i have understood, antlr generates a syntactic predicate in
order to distinguish between the 'and' and 'or' alternatives.
In the case as presented above, the code of such a predicate is
incorrect, as it begins with :
pANTLR3_COMMON_TOKEN;
= NULL ;
It looks like the name x is omitted in this declaration, though it
appears later in the code.
I don't actually need to name the token 'and', and deleting x fixes the
problem.
Expliciting the syntactic predicate seems to fix it as well.
I also noted that the bug does not appear without the +.
Hope this may help improving this great tool:-)
--
Max
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33842] [antlr-interest] Is this a Maven related bug? syntax error @[1, 9] in ._Antlr3ErrorLog.java
Hi, This is my first time using Maven. My goal was to build the Tool because I needed to flip a debug flag in the analysis code. Yes, I know that I probably only need uber jar, but for first time I stuck to the instructions. I based this on the instructions in BUILD.txt. Yes I did read the comments in ANTLR-3.4/pom.xml. Since I am using snapshots on a Virtual Machine, I did it three times, restoring the snapshot each time, to make sure it was reproducible. So, Maven command works. student@antlr:~$ mvn --version Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500) Maven home: /usr/local/apache-maven/apache-maven-3.0.3 Java version: 1.6.0_18, vendor: Sun Microsystems Inc. Java home: /usr/lib/jvm/java-6-openjdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-5-686", arch: "i386", family: "unix" student@antlr:~$ Maven install worked student@antlr:~/antlr-3.4$ mvn -N install ... [INFO] Installing /home/student/antlr-3.4/pom.xml to /home/student/.m2/repository/org/antlr/antlr-master/3.4/antlr-master-3.4.pom [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1:10.373s [INFO] Finished at: Tue Aug 30 13:29:46 EDT 2011 [INFO] Final Memory: 3M/6M [INFO] student@antlr:~/antlr-3.4$ Build of ANTLR-3.4 failed for Maven plugin. student@antlr:~/antlr-3.4$ mvn -Dmaven.test.skip=true [INFO] Using 'UTF-8' encoding to read mojo metadata. [INFO] Applying mojo extractor for language: java [INFO] [INFO] Reactor Summary: [INFO] [INFO] ANTLR Master build control POM 3.4 SUCCESS [0.391s] [INFO] Antlr 3.4 Runtime . SUCCESS [44.632s] [INFO] ANTLR Grammar Tool v 3.4 .. SUCCESS [31.997s] [INFO] Maven plugin for ANTLR V3.4 ... FAILURE [1:07.349s] [INFO] ANTLR gUnit v3.4 .. SKIPPED [INFO] Maven plugin for gUnit ANTLR V3.4 . SKIPPED [INFO] ANTLR3 Maven Archetype 3.4 SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 3:09.551s [INFO] Finished at: Tue Aug 30 13:46:56 EDT 2011 [INFO] Final Memory: 18M/52M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-plugin-plugin:2.7:descriptor (default-descriptor) on project antlr3-maven-plugin: Execution default-descriptor of goal org.apache.maven.plugins:maven-plugin-plugin:2.7:descriptor failed: syntax error @[1,9] in file:/home/student/antlr-3.4/antlr3-maven-plugin/src/main/java/org/antlr/mojo/antlr3/._Antlr3ErrorLog.java -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :antlr3-maven-plugin student@antlr:~/antlr-3.4$ I took a look at the file ._Antlr3ErrorLog.java and it is not all text. If you need more detail, don't hesitate to ask. Thanks, Eric List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33847] [antlr-interest] Rob Pike on writing a lexer in Go for a template language
Hey All, Slightly off topic post, I thought there might be some interest. Last night I went to a talk by Rob Pike of Google, you can watch the talk at https://www.youtube.com/watch?v=HxaD_trXwRE&feature=player_embedded Before I went my thinking was that this could probably be knocked up in ANTLR in a few minutes, but then All the uncomfort I have with ANTLR lexering came back to me. So I though I'd go to the source and have a look at the lexer for ST, and low and behold ST's lexer is written by hand. Now I'm feeling quite uncomfortable about ANTLR's lexing. I think it basically comes down to the stateless nature of the ANTLR lexing. Not the first time context-sensitive scanning has been mentioned on the list (*). Yes I know that it can be made statefull (*) and/or I can push more onto the parser, but both of these have issues. Statefull ANTLR lexing code I generally find more confusing and harder to write then functionally equivalent code in a target language. Pushing more into the parser in this particular case is inefficient as there are large chunks of text that doesn't need to be tokenized and there is the issue the whitespace tokens might need to behaving differently in different places (hidden verse not). Started off as an off topic post, ended as a rant about lexing regards Gary P.S. I've started on the ANTLR target for Go, still very immature. https://github.com/millergarym/antlr/tree/, * Scott Stanchfield's context-sensitive scanning http://javadude.com/articles/antlr-context-sensitive-scanner.html * a good example of this is Jim's numerical lexing for JavaFX http://www.antlr.org/wiki/display/ANTLR3/Lexer+grammar+for+floating+point,+dot,+range,+time+specs List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33848] [antlr-interest] -debug option with C code
Hi I am using ANTLR v3.4 to generate a parser in C. The generated grammar compiles with no problems on my Linux machine with GCC. When I use the -debug option then ANTLR adds a whole bunch of location tracking information such as DBG->location(DBG, M, N). I am guessing that the M and N are line number and column number respectively. However the very first call to the DBG->location function for every terminal in the grammar is missing the 3rd parameter. The output file contains the call as DBG->location(DBG, M, )// notice that N is not printed in the generated C file. However the function call seems to be correctly printed for nested syntactical constructs on the right hand side of a production/rule. Is the -debug option not fully supported in ANTLR v3.4 with C target? Cheers Ramanand List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.
[il-antlr-interest: 33849] Re: [antlr-interest] [C] Crashes if NULL name in antlr3StringStreamNew()
On 8/30/11 4:17 PM, "Ruslan Zasukhin"
wrote:
>
> If I call
>
> input = antlr3StringStreamNew(
> input_string, Encoding, input_len, NULL );
>
> NULL is name of stream
> Then it crashes inside of
>
>
> newStr8(pANTLR3_STRING_FACTORY factory, pANTLR3_UINT8 ptr)
> {
> return factory->newPtr8(factory, ptr,
> (ANTLR3_UINT32)strlen((const char *)ptr));<< crash
> }
>
>
> Because on mac and linux strlen() crashes on NULL.
In contrast to above, new reuse() method can handle stream name as NULL,
Replacing to ³-memory-² name.
antlr38BitReuse(pANTLR3_INPUT_STREAM input, pANTLR3_UINT8 inString,
ANTLR3_UINT32 size, pANTLR3_UINT8 name)
{
input->isAllocated= ANTLR3_FALSE;
input->data= inString;
input->sizeBuf= size;
// Now we can set up the file name. As we are reusing the stream, there
may already
// be a string that we can reuse for holding the filename.
//
if(input->istream->streamName == NULL)
{
input->istream->streamName=
input->strFactory->newStr(input->strFactory, name == NULL ?
(pANTLR3_UINT8)"-memory-" : name);
input->fileName= input->istream->streamName;
}
else
{
input->istream->streamName->set(input->istream->streamName, (name
== NULL ? (const char *)"-memory-" : (const char *)name));
}
input->reset(input);
}
--
Best regards,
Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc
Valentina - Joining Worlds of Information
http://www.paradigmasoft.com
[I feel the need: the need for speed]
List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe:
http://www.antlr.org/mailman/options/antlr-interest/your-email-address
--
You received this message because you are subscribed to the Google Groups
"il-antlr-interest" group.
To post to this group, send email to il-antlr-inter...@googlegroups.com.
To unsubscribe from this group, send email to
il-antlr-interest+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/il-antlr-interest?hl=en.
