Thanks Jim for feedback. Original run was with a full 3.4 distribution (I do not know precisely what is C library distributed in it)
I made a try with latest C runtime library I know (3c-3.4) and this problem clearly persists. I checked compilation ok and settings properly modified in script to point properly on new library. I use a 3.4 antlr version and 3c-3.4 for C library on 64bits. If you need more details, let me know. Jerome ________________________________ De : Jim Idle <j...@temporal-wave.com> À : antlr-interest@antlr.org Envoyé le : Lundi 21 Novembre 2011 18h30 Objet : Re: [antlr-interest] Re : reuse() methos in 3.4 C runtime/exception report I believe that is fixed in the latest source. Jim > -----Original Message----- > From: antlr-interest-boun...@antlr.org [mailto:antlr-interest- > boun...@antlr.org] On Behalf Of Rampon Jerome > Sent: Monday, November 21, 2011 7:13 AM > To: Ruslan Zasukhin; antlr-interest@antlr.org > Subject: [antlr-interest] Re : reuse() methos in 3.4 C > runtime/exception report > > > > Hi Jim, Ruslan, > > > I experimented some issue in a similar kind of usage (iteration of > multiple inputstream with same lexer & parser) > > I reset the tokenstream, lexer and parser in order specified in Jim's > answer ... > > Unfortunately, for second iteration and up, some tokens in factory do > not see their > > input field properly set (until pools is reallocated). > > Reset of the lexer calls the factory reset but do not clean up the > input field of tokens already in the pool. > > Later on when generating new tokens, their are picked up first in the > pool but setCharIndex is not NULL and input not modified. > > In case of exception (tokenstream) in 2nd or upper files, filename > reported is retrieved from token->input->filename resulting in a > confusing reporting. > > if not clear, let me know and I 'll try to give you more details. > If anything wrong in my analysis, feedbacks are welcome. > > I did not find any simple and clean way with antlr3 API to reset those > input files fields for already-in-the-pool tokens > > > Thanks. > Jerome > PS: Anyway, in term of runtime, this approach in C is not significantly > reducing time but it > > makes huge differences in Java. For my grammar, with a suite of 1800 > input files, it takes around 6 mns for a JAVA 1 file at a time call vs > seconds for a serialisation of input files > > > > ________________________________ > De : Ruslan Zasukhin <ruslan_zasuk...@valentina-db.com> > À : "antlr-interest@antlr.org" <antlr-interest@antlr.org> Envoyé le : > Jeudi 17 Novembre 2011 22h24 Objet : Re: [antlr-interest] reuse() > methos in 3.4 C runtime > > Hi Jim, > > Below are copy-pastes of my class-wrapper around ANTLR3 > Lexer/Parser/TreeParser. > So you can see if I made some stupid mistake... > > > >> On 6/24/11 7:49 PM, "Jim Idle" <j...@temporal-wave.com> wrote: > >> > >> Because the documentation is not yet up to date, here is an example > >> of reusing the allocated memory in input streams and token streams: > >> > >> for (i=0; i<iterations; i++) > >> { > >> // Run the parser. > >> psr->start(psr); > >> > >> // -------------------------------------- > >> // Now reset everything for the next run. > >> // Order of calls is important. > >> > >> // Input stream can now be reused > >> input->reuse(input, sourceCode, sourceLen, sourceName); > >> > >> // Reset the common token stream so that it will reuse its > >>resources > >> tstream->reset(tstream); > >> > >> // Reset the lexer (new function generated by antlr now) > >> lxr->reset(lxr); > >> > >> // Reset the parser (new function generated by antlr now) > >> psr->reset(psr); > >> } > > > /********************************************************************** > **** > void SqlParser_v3::ResuseParserObjects( > const char* inTextToParse, > vuint32 inLength ) > { > // ------------------------------- > // TREE PARSER cannot be reused. Destroy it. > // > if( mpTreeParser ) > { > mpTreeParser->free( mpTreeParser ); > mpTreeParser = NULL; > } > > if( mpNodes ) > { > mpNodes->free( mpNodes ); > mpNodes = NULL; > } > > > // ------------------------------- > // Reuse other objects > // > mpInput->reuse( > mpInput, > (pANTLR3_UINT8) inTextToParse, > (ANTLR3_UINT32) inLength, > (pANTLR3_UINT8) "VSQL" ); > > mpTokenStream->reset( mpTokenStream ); > mpLexer ->reset( mpLexer ); > mpParser ->reset( mpParser ); > > ResetOwnData( mpParser ); > } > > > > And few other related methods ... > > > /********************************************************************** > **** > void SqlParser_v3::Parse_UTF8( > I_SqlDatabaseEx* inDatabase, > const char* inCommand, > const char* inCommandEnd ) { > argused1(inDatabase); > > // COMMENT this line to force REUSE() mode ... > // DestroyParserObjects(); > > if( mpInput ) > ResuseParserObjects( inCommand, (inCommandEnd - inCommand) ); > else > CreateParserObjects( inCommand, (inCommandEnd - inCommand) ); > > > // ------------------------- > // Parse the input expression > mAST = mpParser->sql( mpParser ); > > > // IF PARSER have generate some errors, > // then we throw them as VSQL exception. > if( mpParser->pParser->rec->state->errorCount ) > { > StToUTF16 cnv( ResultStringBuffer, pErrEnd, GetConverter_UTF8() > ); > throw VSQL::xVSQLException( ERR_SQL_PARSER_ERROR, cnv.c_str() > ); > } > } > > > /********************************************************************** > **** > void SqlParser_v3::CreateParserObjects( > const char* inTextToParse, > vuint32 inLength ) > { > if( inTextToParse == NULL ) > return; // all objects will be still NULLs also. > > // ------------------------------ > // Create INPUT object: > > // NOTE: SQL strings do not have BOM - first few bytes, which > define > endian of UTF16. > // So for UTF16, we must here self specify BE or LE. > mpInput = antlr3StringStreamNew( > (pANTLR3_UINT8) inTextToParse, mEncoding, (ANTLR3_UINT32) > inLength, > (pANTLR3_UINT8) "VSQL" ); > > mpInput->setUcaseLA( mpInput, ANTLR3_TRUE ); > > > // ------------------------------ > // Create LEXER v3 object: > > mpLexer = SqlParser_v3LexerNew( mpInput ); > mpTokenStream = antlr3CommonTokenStreamSourceNew( ANTLR3_SIZE_HINT, > TOKENSOURCE( mpLexer ) ); > > > // ------------------------------ > // Create PARSER v3 object: > > mpParser = SqlParser_v3ParserNew( mpTokenStream ); // is generated > by > ANTLR3 > mpParser->mDoAllCommands = mDoAllCommandsInitial; > > ResetOwnData( mpParser ); > > > /********************************************************************** > **** > void SqlParser_v3::DestroyParserObjects( void ) > { > // REVERSE ORDER to construction: > > if( mpTreeParser ) > { > mpTreeParser->free( mpTreeParser ); > mpTreeParser = NULL; > } > > if( mpNodes ) > { > mpNodes->free( mpNodes ); > mpNodes = NULL; > } > > if( mpParser ) > { > mpParser->mpStartPositions = NULL; > > mpParser->free( mpParser ); > mpParser = NULL; > } > > if( mpTokenStream ) > { > mpTokenStream->free( mpTokenStream ); > mpTokenStream = NULL; > } > > if( mpLexer ) > { > mpLexer->free( mpLexer ); > mpLexer = NULL; > } > > if( mpInput ) > { > mpInput->close( mpInput ); > mpInput = NULL; > } > } > > > > > > > -- > Best regards, > > Ruslan Zasukhin > VP Engineering and New Technology > Paradigma Software, Inc > > Valentina - Joining Worlds of Information > http://www.paradigmasoft.com > > [I feel the need: the need for speed] > > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address > > List: http://www.antlr.org/mailman/listinfo/antlr-interest > Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your- > email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address List: http://www.antlr.org/mailman/listinfo/antlr-interest Unsubscribe: http://www.antlr.org/mailman/options/antlr-interest/your-email-address -- You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-inter...@googlegroups.com. To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en.