Adrian, > Le 26 févr. 2020 à 23:58, Adrian Vogelsgesang <[email protected]> a > écrit : > > regarding our discussion on memory management for yyexpected_tokens > -------------------------------------------------------------------------------------------------- > > >> If your use case in incrementally proposing possible keywords, then you want >> them all, and a max of the possible number of tokens is available >> statically: YYNTOKENS. > You are right, I would want to retrieve the whole list of expected tokens. > >> I don't see where this would be useful. > > The code I am having in mind to do so would be > int onstack_tokens[10]; > std::vector<int> heap_tokens; > int cnt_tokens = context.yyexpected_tokens(onstack_tokens, 10); > if (cnt_tokens > 10) { > heap_tokens.resize(cnt_tokens); > context.yyexpected_tokens(heap_tokens.data(), cnt_tokens); > } > int* tokens = (cnt_tokens <= 10) ? onstack_tokens : heap_tokens.data();
Yes, that definitely makes sense when no (reasonable) bound is known. Actually if memory is really an issue, we could also use a bitset. But a list of indexes seems the right approach here. An iterator over the set of expected tokens would also work and meet all these use cases. But maybe that's over-engineering. Hum... We should really think about that. > That way, we can use the stack allocation in the common case (<= 10 expected > tokens) and only fallback to the heap-allocation if we have too many expected > tokens. At least in our grammar, it is common that only very few keywords are > expected, except in a few places, where you can write almost everything. Now > that I think about this, I am not actually sure how well this observation > generalizes to other grammars, though. Preallocating YYNTOKENS (480 in our > case) Wow!!! I believe this is the largest set of tokens I heard of! Congratulations :) How many rules, how many states? > on the stack would cost us ~2KB of stack space. Right. > Probably that would be fine... > >> The point here is efficiency > It's funny how both of us are having performance/efficiency as a goal but are > coming to a different conclusion :) Yep :) > I was not seeing your point about "not spend time counting how many more > there are" on my own. Thanks for pointing that out! With that reasoning, I > can completely agree that the existing implementation is fine. Great, thanks!
