Le 15/05/2012 21:59, Roman D. Boiko a écrit :
On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
Currently I think about making token a class instead of struct.
...
Could anybody suggest other pros and cons? Which option would you
choose?
Just use a circular buffer of value-type tokens. There is absolutely
no excuse for slow parsing.
I'd like to be able to do efficient token lookup based on its start
index in the file (e.g., for autocompletion and calculating column/line
numbers on demand). I also plan to have ability to update lexer, parser
and semantic analysis results as the user or tool makes edits
(incrementally, not by restarting), and to keep history of changes as
long as it is needed.
To achieve this, I will use Token[N][] for storing tokens, and an
immutable binary tree for efficient lookup and preserving history.
I don't worry much about keeping all tokens in memory, because
information they contain is useful and needed for most scenarios. If, on
the contrary, information is no longer used, it will be garbage collected.
This is a significant redesign of current implementation. It is based on
feedback from this thread and completely satisfies my design goals. It
is also going to be efficient.
Alpha release for community review is planned on May 21, and design
documentation in 10 days after that.
The only information that is usefull over time is the position. This is
the one that you want to calculate lazily.
In other terms, what you want to keep in memory have no interest.
A circular buffer as Timon propose is a better idea.
