On 21.07.2024 at 13:28, Tim Düsterhus wrote: > On 7/20/24 20:31, Christoph M. Becker wrote: > >> If the tokenizer would tokenize a whole file as a single token, would >> that also be correct? Of course, I'm exaggerating, but > > The function is documented as: > >> token_get_all() parses the given code string into PHP language tokens >> using the Zend engine's lexical scanner. > > So if Zend engine's lexical scanner would tokenize a complete PHP file > as a single token, then the output would be correct, yes.
Apparently we cannot even agree that there is a bug, and as such we cannot fix it in PHP 8.3 according to the release process documentation. >> <https://3v4l.org/qIf2c> doesn't look correct to me – "yield /* comment >> */ from" shouldn't be a single token. > > Looking at this from a user expectation perspective and not a technical > perspective, neither should `yield from` be a single token in: > > var_dump(token_get_all('<?php yield from $foo;')); ACK Cheers, Christoph
