Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-22 Thread Nikita Popov
So, is there any consensus about this? Nikita On Fri, Sep 16, 2011 at 10:07 AM, Ferenc Kovacs tyr...@gmail.com wrote: Wait wait wait. Thats the point here? __COMPILER_HALT_OFFSET__ already tells you where the data starts. -Hannes I didn't sent this message first, but after reading the

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-16 Thread Nicolas Grekas
What I am really hearing in this thread is some people have interest in a way to easily access the __COMPILER_HALT_OFFSET__ Well, not exactly for me : I'm not interested strictly on getting the offset. I'm more interested in doing generic static code analysis, and part of that is getting the

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-16 Thread Ferenc Kovacs
Wait wait wait. Thats the point here? __COMPILER_HALT_OFFSET__ already tells you where the data starts. -Hannes I didn't sent this message first, but after reading the mail from Chris, I think maybe it would clear the confusion: It is about tokenizing a file which has __halt_compiler(); in

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Hannes Magnusson
On Tue, Sep 13, 2011 at 22:08, Ferenc Kovacs tyr...@gmail.com wrote: On Tue, Sep 13, 2011 at 9:56 PM, Christopher Jones christopher.jo...@oracle.com wrote: On 09/13/2011 12:25 PM, Nikita Popov wrote: I changed my previous patch to an SVN patch, so it is easier to apply and added another

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Nikita Popov
Hi Hannes, thanks for your response! On Thu, Sep 15, 2011 at 9:22 AM, Hannes Magnusson hannes.magnus...@gmail.com wrote: the change to ext/tokenizer/tests/token_get_all_variation16.phpt seems really weird? I changed the test to not include a __halt_compiler statement anymore, as it didn't make

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Hannes Magnusson
On Thu, Sep 15, 2011 at 16:51, Nikita Popov nikita@googlemail.com wrote: Hi Hannes, thanks for your response! On Thu, Sep 15, 2011 at 9:22 AM, Hannes Magnusson hannes.magnus...@gmail.com wrote: the change to ext/tokenizer/tests/token_get_all_variation16.phpt seems really weird? I

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Nikita Popov
On Thu, Sep 15, 2011 at 5:05 PM, Hannes Magnusson hannes.magnus...@gmail.com wrote: Wait wait wait. Thats the point here? __COMPILER_HALT_OFFSET__ already tells you where the data starts. __COMPILER_HALT_OFFSET__ tells you the offset if you are running the file. It is only available in the file

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-15 Thread Chris Stockton
Hello, On Thu, Sep 15, 2011 at 8:13 AM, Nikita Popov nikita@googlemail.com wrote: On Thu, Sep 15, 2011 at 5:05 PM, Hannes Magnusson hannes.magnus...@gmail.com wrote: Wait wait wait. Thats the point here? __COMPILER_HALT_OFFSET__ already tells you where the data starts.

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nikita Popov
I just set up an PHP environment and wrote a proper patch (including test changes) to make it collect the next three tokens. It's a git patch and I'm not sure whether it's compatible with SVN patches. I would love it if this would go into 5.4 before beta. I didn't know how one could fetch the rest

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nicolas Grekas
I just set up an PHP environment and wrote a proper patch (including test changes) to make it collect the next three tokens. It's a git patch and I'm not sure whether it's compatible with SVN patches. I would love it if this would go into 5.4 before beta. I didn't know how one could fetch the

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Nikita Popov
I changed my previous patch to an SVN patch, so it is easier to apply and added another patch (the one called tokenizer_patch_full.txt), which additionally fetches the rest into a T_INLINE_HTML. (The _full patch thus contains both changes. I didn't know how I could separate them.) Could somebody

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Christopher Jones
On 09/13/2011 12:25 PM, Nikita Popov wrote: I changed my previous patch to an SVN patch, so it is easier to apply and added another patch (the one called tokenizer_patch_full.txt), which additionally fetches the rest into a T_INLINE_HTML. (The _full patch thus contains both changes. I didn't

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-13 Thread Ferenc Kovacs
On Tue, Sep 13, 2011 at 9:56 PM, Christopher Jones christopher.jo...@oracle.com wrote: On 09/13/2011 12:25 PM, Nikita Popov wrote: I changed my previous patch to an SVN patch, so it is easier to apply and added another patch (the one called tokenizer_patch_full.txt), which additionally

[PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nikita Popov
In Bug #54089 [1] a patch was applied that cuts of token_get_all() output after a T_HALT_COMPILER token. This was done because otherwise PHP would keep on lexing after that and would generate errors because of binary data (which is not valid PHP, mostly.) The problem with the patch is, that there

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Ferenc Kovacs
On Fri, Sep 9, 2011 at 9:15 AM, Nikita Popov nikita@googlemail.com wrote: In Bug #54089 [1] a patch was applied that cuts of token_get_all() output after a T_HALT_COMPILER token. This was done because otherwise PHP would keep on lexing after that and would generate errors because of binary

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nicolas Grekas
Thank you Nikita for take this subject here! On Fri, Sep 9, 2011 at 10:01, Ferenc Kovacs tyr...@gmail.com wrote: don't break there but for the next ';'. You can also just count the number of semantic token after T_HALT_COMPILER (ie excluding whitespace and comments) and once you hit 3, halt.

Re: [PHP-DEV] Revert Tokenizer behavior for 5.4

2011-09-09 Thread Nikita Popov
On Fri, Sep 9, 2011 at 10:46 AM, Nicolas Grekas nicolas.grekas+...@gmail.com wrote: You can also just count the number of semantic token after T_HALT_COMPILER (ie excluding whitespace and comments) and once you hit 3, halt. [...] Maybe the binary data itself, as a big T_INLINE_HTML for example