On Tuesday, 15 May 2012 at 19:27:26 UTC, Timon Gehr wrote:
On 05/14/2012 05:00 PM, Roman D. Boiko wrote:
Currently I think about making token a class instead of struct.
...
Could anybody suggest other pros and cons? Which option would
you choose?
Just use a circular buffer of value-type tokens. There is
absolutely no excuse for slow parsing.
I'd like to be able to do efficient token lookup based on its
start index in the file (e.g., for autocompletion and calculating
column/line numbers on demand). I also plan to have ability to
update lexer, parser and semantic analysis results as the user or
tool makes edits (incrementally, not by restarting), and to keep
history of changes as long as it is needed.
To achieve this, I will use Token[N][] for storing tokens, and an
immutable binary tree for efficient lookup and preserving history.
I don't worry much about keeping all tokens in memory, because
information they contain is useful and needed for most scenarios.
If, on the contrary, information is no longer used, it will be
garbage collected.
This is a significant redesign of current implementation. It is
based on feedback from this thread and completely satisfies my
design goals. It is also going to be efficient.
Alpha release for community review is planned on May 21, and
design documentation in 10 days after that.
