[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hi,
I've fixed all the issues I found in the original patch (attached).
However then I realized that speed-up was caused by bugs that leaded to
render pages in improper way. After fix the speed-up disappear. :(
So now I don't see any reason to include it into 5.3.
Thanks. Dmitry.
Lukas Kahwe Smith wrote:
On 11.03.2009, at 17:10, Rasmus Lerdorf wrote:
Anyway, it's very good job and 20-30% speedup on some real-life
applications makes sense to include it into 5.3 (from my point of view).
Makes sense to me as well, especially since I don't see any drawbacks
for non-accelerated scripts.
I also think some of those applications that didn't show any gains could
be trivially modified to make use of it. Moving from conditionally
loaded stuff to lazy-loaded is likely to speed things up. Depending of
course on how granular the conditional loading is.
Can we get this patch to release quality by this weekend?
So that people can test it on Monday/Tuesday ahead of RC1?
regards,
Lukas Kahwe Smith
m...@pooteeweet.org
Index: Zend/zend.c
===
RCS file: /repository/ZendEngine2/zend.c,v
retrieving revision 1.308.2.12.2.35.2.29
diff -u -p -d -r1.308.2.12.2.35.2.29 zend.c
--- Zend/zend.c 18 Feb 2009 10:55:08 - 1.308.2.12.2.35.2.29
+++ Zend/zend.c 13 Mar 2009 13:46:07 -
@@ -550,6 +550,10 @@ static void executor_globals_ctor(zend_e
EG(current_module) = NULL;
EG(exit_status) = 0;
EG(active) = 0;
+ EG(lookup_function_hook) = NULL;
+ EG(lookup_class_hook) = NULL;
+ EG(defined_function_hook) = NULL;
+ EG(declared_class_hook) = NULL;
}
/* }}} */
Index: Zend/zend_API.c
===
RCS file: /repository/ZendEngine2/zend_API.c,v
retrieving revision 1.296.2.27.2.34.2.60
diff -u -p -d -r1.296.2.27.2.34.2.60 zend_API.c
--- Zend/zend_API.c 14 Jan 2009 11:56:08 - 1.296.2.27.2.34.2.60
+++ Zend/zend_API.c 13 Mar 2009 13:46:07 -
@@ -2180,7 +2180,7 @@ ZEND_API zend_class_entry *zend_register
if (!parent_ce parent_name) {
zend_class_entry **pce;
- if (zend_hash_find(CG(class_table), parent_name,
strlen(parent_name)+1, (void **) pce)==FAILURE) {
+ if (zend_find_class(CG(class_table), parent_name,
strlen(parent_name)+1, pce TSRMLS_CC)==FAILURE) {
return NULL;
} else {
parent_ce = *pce;
@@ -2228,14 +2228,20 @@ ZEND_API zend_class_entry *zend_register
ZEND_API int zend_register_class_alias_ex(const char *name, int name_len,
zend_class_entry *ce TSRMLS_DC) /* {{{ */
{
char *lcname = zend_str_tolower_dup(name, name_len);
- int ret;
+ ulong hash = zend_inline_hash_func(lcname, name_len+1);
+ zend_class_entry **pce;
- ret = zend_hash_add(CG(class_table), lcname, name_len+1, ce,
sizeof(zend_class_entry *), NULL);
- efree(lcname);
- if (ret == SUCCESS) {
- ce-refcount++;
+ if((EG(lookup_class_hook)
+ /* Check if we have a same-name class already exsiting in our
hook, then fail as we would normally below.
+* This could happen if we have two classes with the same name,
and one of them is being bound at run-time */
+ EG(lookup_class_hook)(lcname, name_len+1, hash, pce) == SUCCESS) ||
+ zend_hash_quick_add(CG(class_table), lcname, name_len+1, hash, ce,
sizeof(zend_class_entry *), NULL) == FAILURE) {
+ efree(lcname);
+ return FAILURE;
}
- return ret;
+ efree(lcname);
+ ce-refcount++;
+ return SUCCESS;
}
/* }}} */
@@ -2412,7 +2418,7 @@ static int zend_is_callable_check_func(i
}
/* Check if function with given name exists.
* This may be a compound name that includes namespace name */
- if (zend_hash_find(EG(function_table), lmname, mlen+1,
(void**)fcc-function_handler) == SUCCESS) {
+ if (zend_find_function(EG(function_table), lmname, mlen+1,
fcc-function_handler TSRMLS_CC) == SUCCESS) {
efree(lmname);
return 1;
}
@@ -3552,6 +3558,86 @@ ZEND_API void zend_restore_error_handlin
}
/* }}} */
+ZEND_API int zend_quick_find_function(HashTable *function_table, const char
*name, int len, zend_uint h, zend_function **fe TSRMLS_DC) /* {{{ */
+{
+ if (zend_hash_quick_find(function_table, name, len, h, (void**)fe) ==
SUCCESS) {
+ return SUCCESS;
+ } else if (EG(lookup_function_hook)) {
+ return EG(lookup_function_hook)(name, len, h, fe);
+ }
+ return FAILURE;
+}
+/* }}} */
+
+ZEND_API int zend_find_function(HashTable *function_table, const char *name,
int len, zend_function **fe TSRMLS_DC) /* {{{ */
+{
+ ulong h = zend_inline_hash_func(name, len);
Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
hi, Then I would rather have a beta2 and not a RC. This change while being great may require more tweaks, and as Marcus suggested, we could push APC in too while being at it. I'm all for it (both :). Cheers, On Wed, Mar 11, 2009 at 5:13 PM, Lukas Kahwe Smith m...@pooteeweet.org wrote: On 11.03.2009, at 17:10, Rasmus Lerdorf wrote: Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Makes sense to me as well, especially since I don't see any drawbacks for non-accelerated scripts. I also think some of those applications that didn't show any gains could be trivially modified to make use of it. Moving from conditionally loaded stuff to lazy-loaded is likely to speed things up. Depending of course on how granular the conditional loading is. Can we get this patch to release quality by this weekend? So that people can test it on Monday/Tuesday ahead of RC1? regards, Lukas Kahwe Smith m...@pooteeweet.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php -- Pierre http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hi 2009/3/13 Pierre Joye pierre@gmail.com: hi, Then I would rather have a beta2 and not a RC. This change while being great may require more tweaks, and as Marcus suggested, we could push APC in too while being at it. I'm all for it (both :). I must admit that I think it would be bad to delay 5.3 more and have another beta because of this, however if possible I would be all for having APC in RC1 next week :) Cheers, -- Kalle Sommer Nielsen ka...@php.net -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Dmitry Stogov wrote: Hi, Personally, I like the patch except for some small possible tweaks, and I believe it can't make any harm with lazy loading disabled. Thanks, what are the tweaks you'd like to see so I can try to include them? Could you provide some benchmark results? I was hoping to solicit some from others on the list, but haven't seen anything yet. My best example of gains is Facebook, I believe these where around 20-30% decrease in CPU usage on a bare-bones page. I did test Joomla and Zend Framework, the gains here aren't much if anything as they seem to be use autoload for most files. (although I would like to see what lazy method loading can do here). I intend to benchmark wordpress as well. I'll post more benchmarks for the above and this once I make some more tweaks that also might give us better results. I anticipate that benchmarks are going to vary pretty drastically depending on code structure, size, autoloading, etc. APC patch to play with it and see advantages/disadvantages? I've gone ahead and checked in the current code for APC, so you'll have the necessary changes if you checkout CVS HEAD. -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
shire wrote: Dmitry Stogov wrote: Hi, Personally, I like the patch except for some small possible tweaks, and I believe it can't make any harm with lazy loading disabled. Thanks, what are the tweaks you'd like to see so I can try to include them? I thought about different semantics for defined_function_hook and declared_class_hook to just load all lazy function/classes into regular ZE tables. It's also possible to pass hash_values into lookup_function_hook and lookup_class_hook to don't calculate them twice. I'll return to you in a day or two (after playing with patch and measuring speed difference) to be more concrete. Thanks. Dmitry. Could you provide some benchmark results? I was hoping to solicit some from others on the list, but haven't seen anything yet. My best example of gains is Facebook, I believe these where around 20-30% decrease in CPU usage on a bare-bones page. I did test Joomla and Zend Framework, the gains here aren't much if anything as they seem to be use autoload for most files. (although I would like to see what lazy method loading can do here). I intend to benchmark wordpress as well. I'll post more benchmarks for the above and this once I make some more tweaks that also might give us better results. I anticipate that benchmarks are going to vary pretty drastically depending on code structure, size, autoloading, etc. APC patch to play with it and see advantages/disadvantages? I've gone ahead and checked in the current code for APC, so you'll have the necessary changes if you checkout CVS HEAD. -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hi Shire, I run patched APC on a number of real-life applications and got more than 30% speedup on XOOPS (99 req/sec instead of 60%) and 20% on ZendFramework (41 req/sec instead of 32), however most applications (drupal, qdig, typo3, wordpress) didn't show significant speed difference. As was expected the patch doesn't affects PHP without APC or with APC and lazy loading disabled. I also got APC warning with Zend Framewoek based blog application, but I didn't try to look deeper. [Wed Mar 11 17:53:02 2009] [apc-warning] [Wed Mar 11 17:53:02 2009] [apc-warning] apc_lookup_class_hook: could not install blogrow in /var/www/html/bench/fw/ZendFramework-1.5.0RC3/library/Zend/Loader.php on line 86 I didn't look careful into APC code, just into PHP patch and I see the following issues: 1) I would prefer to add additional hash_value argument into lookup_function_hook() and lookup_class_hook to prevent multiple calculation. 2) function_exists() should use lookup_function_hook(). 3) interface_exists() and class_alias() should use lookup_class_hook(). 4) ext/soap, ext/reflection, ext/wddx and ext/spl autoload support Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Thanks. Dmitry. shire wrote: Dmitry Stogov wrote: Hi, Personally, I like the patch except for some small possible tweaks, and I believe it can't make any harm with lazy loading disabled. Thanks, what are the tweaks you'd like to see so I can try to include them? Could you provide some benchmark results? I was hoping to solicit some from others on the list, but haven't seen anything yet. My best example of gains is Facebook, I believe these where around 20-30% decrease in CPU usage on a bare-bones page. I did test Joomla and Zend Framework, the gains here aren't much if anything as they seem to be use autoload for most files. (although I would like to see what lazy method loading can do here). I intend to benchmark wordpress as well. I'll post more benchmarks for the above and this once I make some more tweaks that also might give us better results. I anticipate that benchmarks are going to vary pretty drastically depending on code structure, size, autoloading, etc. APC patch to play with it and see advantages/disadvantages? I've gone ahead and checked in the current code for APC, so you'll have the necessary changes if you checkout CVS HEAD. -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Dmitry Stogov wrote: Hi Shire, I run patched APC on a number of real-life applications and got more than 30% speedup on XOOPS (99 req/sec instead of 60%) and 20% on ZendFramework (41 req/sec instead of 32), however most applications (drupal, qdig, typo3, wordpress) didn't show significant speed difference. As was expected the patch doesn't affects PHP without APC or with APC and lazy loading disabled. I also got APC warning with Zend Framewoek based blog application, but I didn't try to look deeper. [Wed Mar 11 17:53:02 2009] [apc-warning] [Wed Mar 11 17:53:02 2009] [apc-warning] apc_lookup_class_hook: could not install blogrow in /var/www/html/bench/fw/ZendFramework-1.5.0RC3/library/Zend/Loader.php on line 86 I didn't look careful into APC code, just into PHP patch and I see the following issues: 1) I would prefer to add additional hash_value argument into lookup_function_hook() and lookup_class_hook to prevent multiple calculation. 2) function_exists() should use lookup_function_hook(). 3) interface_exists() and class_alias() should use lookup_class_hook(). 4) ext/soap, ext/reflection, ext/wddx and ext/spl autoload support Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Makes sense to me as well, especially since I don't see any drawbacks for non-accelerated scripts. I also think some of those applications that didn't show any gains could be trivially modified to make use of it. Moving from conditionally loaded stuff to lazy-loaded is likely to speed things up. Depending of course on how granular the conditional loading is. -Rasmus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On 11.03.2009, at 17:10, Rasmus Lerdorf wrote: Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Makes sense to me as well, especially since I don't see any drawbacks for non-accelerated scripts. I also think some of those applications that didn't show any gains could be trivially modified to make use of it. Moving from conditionally loaded stuff to lazy-loaded is likely to speed things up. Depending of course on how granular the conditional loading is. Can we get this patch to release quality by this weekend? So that people can test it on Monday/Tuesday ahead of RC1? regards, Lukas Kahwe Smith m...@pooteeweet.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Lukas Kahwe Smith wrote: On 11.03.2009, at 17:10, Rasmus Lerdorf wrote: Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Makes sense to me as well, especially since I don't see any drawbacks for non-accelerated scripts. I also think some of those applications that didn't show any gains could be trivially modified to make use of it. Moving from conditionally loaded stuff to lazy-loaded is likely to speed things up. Depending of course on how granular the conditional loading is. Can we get this patch to release quality by this weekend? I think it's possible. I could help with it. Thanks. Dmitry. So that people can test it on Monday/Tuesday ahead of RC1? regards, Lukas Kahwe Smith m...@pooteeweet.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hello Lukas, Wednesday, March 11, 2009, 5:10:57 PM, you wrote: Dmitry Stogov wrote: Hi Shire, I run patched APC on a number of real-life applications and got more than 30% speedup on XOOPS (99 req/sec instead of 60%) and 20% on ZendFramework (41 req/sec instead of 32), however most applications (drupal, qdig, typo3, wordpress) didn't show significant speed difference. As was expected the patch doesn't affects PHP without APC or with APC and lazy loading disabled. I also got APC warning with Zend Framewoek based blog application, but I didn't try to look deeper. [Wed Mar 11 17:53:02 2009] [apc-warning] [Wed Mar 11 17:53:02 2009] [apc-warning] apc_lookup_class_hook: could not install blogrow in /var/www/html/bench/fw/ZendFramework-1.5.0RC3/library/Zend/Loader.php on line 86 I didn't look careful into APC code, just into PHP patch and I see the following issues: 1) I would prefer to add additional hash_value argument into lookup_function_hook() and lookup_class_hook to prevent multiple calculation. 2) function_exists() should use lookup_function_hook(). 3) interface_exists() and class_alias() should use lookup_class_hook(). 4) ext/soap, ext/reflection, ext/wddx and ext/spl autoload support Anyway, it's very good job and 20-30% speedup on some real-life applications makes sense to include it into 5.3 (from my point of view). Makes sense to me as well, especially since I don't see any drawbacks for non-accelerated scripts. I also think some of those applications that didn't show any gains could be trivially modified to make use of it. Moving from conditionally loaded stuff to lazy-loaded is likely to speed things up. Depending of course on how granular the conditional loading is. Right, basically the patch adresses what we say at conferences. Autload can do an amazing job but also harm you. The patch on the other hand somehow combines the advantages of both worlds. You can be explicit because you know it better and for that you do not get any punishment from Autoload execution time. And then again, you don't get punished by including too much. Becuase after all the compiler knows it even better. And of course, given that some people gain over 20%, I don't see why we even need to discuss putting this in. Last but not least, Lukas, what happened, to putting APC into core? marcus Best regards, Marcus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Lukas Kahwe Smith wrote: Can we get this patch to release quality by this weekend? So that people can test it on Monday/Tuesday ahead of RC1? I don't see this being a problem, I do have a few items I'd like to point out for feedback/suggestions: 1) Currently it doesn't support method level lazy loading, it's probably advisable to wait and include this later, but if everyone thinks it's significant enough we could probably add support for that. I'm not sure on all the details this involves, perhaps someone familiar with method calls could suggest difficulty and/or optimized ways to do the lookups. 2) Currently I've inserted these hook calls into everywhere we call/lookup classes. This has the downside that any extension wanting to mess with the function table, lookup entries, etc will need to be aware of the callback hooks. I think a better architecture is to create an API for function table and class table operations. Perhaps this could be done later if we want this likely more stable version in 5.3, and perhaps I'm overstating the significance of other functions doing these sort of things and not being aware of this new feature. (I believe dmitry mentions several extenions that need changes to support this). On the upside not creating an API means existing code will still work if they don't use lazy loading. ;-) 3) The largest portion of time currently is simply adding dummy entries to the lazy hash tables so we can detect name collisions and availability, my next goal is to improve the performance of this by either creating a faster copy of the entries or determine some better method of performing lookups/marking functions as available (something like lookups directly into the APC shared memory segment). Just putting this out there in case anyone has some interesting suggestions. I think all the above 3 items don't necessarily need to be done to have this work in 5.3 (and #3 is pretty much APC/opcode cache centric) and might cause unecessary complication for an initial support of this, but what does everyone else think? Regarding a couple of Dmitry's suggestions: 1) I would prefer to add additional hash_value argument into lookup_function_hook() and lookup_class_hook to prevent multiple calculation. I'm upset I didn't do that in the first place ;-) 2) function_exists() should use lookup_function_hook(). 3) interface_exists() and class_alias() should use lookup_class_hook(). I'm pretty sure I already have support for this, it may have been lost while porting it over, I'll double check. I'll follow up with Dmitry on getting these and any other items taken care of, thanks! -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On 11.03.2009, at 19:55, Marcus Boerger wrote: Last but not least, Lukas, what happened, to putting APC into core? That was planned for PHP 6.0. regards, Lukas Kahwe Smith m...@pooteeweet.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hello Lukas, Wednesday, March 11, 2009, 11:15:23 PM, you wrote: On 11.03.2009, at 19:55, Marcus Boerger wrote: Last but not least, Lukas, what happened, to putting APC into core? That was planned for PHP 6.0. there is no such thing. Let's either do it now or go for 5.4. regards, Lukas Kahwe Smith m...@pooteeweet.org Best regards, Marcus -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hi! there is no such thing. Let's either do it now or go for 5.4. Can we please stop trying new big features each time we approach RC? 5.3 is not last PHP version ever, and it's long overdue. Can't we just release it without putting more and more potentially unstable changes into it, and then discuss other stuff for 5.4, 6.0 and whatever comes in orderly manner? -- Stanislav Malyshev, Zend Software Architect s...@zend.com http://www.zend.com/ (408)253-8829 MSN: s...@zend.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Re: [APC-DEV] Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On 11.03.2009, at 20:58, shire wrote: Lukas Kahwe Smith wrote: Can we get this patch to release quality by this weekend? So that people can test it on Monday/Tuesday ahead of RC1? I don't see this being a problem, I do have a few items I'd like to point out for feedback/suggestions: 1) Currently it doesn't support method level lazy loading, it's probably advisable to wait and include this later, but if everyone thinks it's significant enough we could probably add support for that. I'm not sure on all the details this involves, perhaps someone familiar with method calls could suggest difficulty and/or optimized ways to do the lookups. 2) Currently I've inserted these hook calls into everywhere we call/ lookup classes. This has the downside that any extension wanting to mess with the function table, lookup entries, etc will need to be aware of the callback hooks. I think a better architecture is to create an API for function table and class table operations. Perhaps this could be done later if we want this likely more stable version in 5.3, and perhaps I'm overstating the significance of other functions doing these sort of things and not being aware of this new feature. (I believe dmitry mentions several extenions that need changes to support this). On the upside not creating an API means existing code will still work if they don't use lazy loading. ;-) 3) The largest portion of time currently is simply adding dummy entries to the lazy hash tables so we can detect name collisions and availability, my next goal is to improve the performance of this by either creating a faster copy of the entries or determine some better method of performing lookups/marking functions as available (something like lookups directly into the APC shared memory segment). Just putting this out there in case anyone has some interesting suggestions. I think all the above 3 items don't necessarily need to be done to have this work in 5.3 (and #3 is pretty much APC/opcode cache centric) and might cause unecessary complication for an initial support of this, but what does everyone else think? Hmm, thanks for your open assessment here. What you are saying does make me wonder if we really should push this into 5.3. Even if any changes we need to do can eventually be handled by APC, I do think that there will be other extension authors that would also suffer from us messing around with these internals all too much in every release. So maybe we should wait until the next bigger step before introducing this. Other people can apply the patch themselves if they really need the performance in the mean time .. regards, Lukas Kahwe Smith m...@pooteeweet.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Lukas Kahwe Smith wrote: On 22.02.2009, at 01:10, shire wrote: I've just checked into APC CVS preliminary support for Lazy Loading classes and functions. This means that rather than copying function entries into EG(function_table) and EG(class_table) when an include happen it will mark the functions/classes as available and only actually insert them into the tables when they are called. This is done via hooks added into the various hash table lookups in PHP. I've placed a patch for PHP_5_3 at: http://tekrat.com/downloads/bits/apc_lazy_php53.patch I did not read through the entire thread. As things are close to RC state in 5.3, I would prefer to not do any non bug fixes at this stage. Then again if the benefits are huge and the risk is low it can be considered of course .. Yep, I should have been clearer here. ;-) I don't believe this patch is ready to be committed and I'm not advocating for inclusion in the php53 release. I have some more adjustments I'd like to make to both the APC code and this patch, both for optimization, cleanliness, and to ensure it's the best possible implementation for future enhancements (such as lazy loading methods). I'm more interested in getting constructive feedback and any benchmark results so I can make adjustments to the current code. However, I would like to come back with improved patches and get it included in the next major release as appropriate. I'm all about fixing the remaining php53 todos/bugs and getting this release out! I hope posting this wasn't a distraction from that. Thanks! -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On Thu, Feb 26, 2009 at 1:58 PM, Rodrigo Saboya rodrigo.sab...@bolsademulher.com wrote: snip For the average PHP programmer, the language will simply get faster. That can't be bad in any way. It doesn't encourage you to write bad code, it just doesn't kick you in the nuts when you do. It's probably also worth noting that in a lot of cases, you really *won't* use a lot of what you load, at least initially. If you're working in an OOP fashion, for example, you won't use every method at once on an object, but you're still going to have to pull the whole object in to use it. If you're writing functional code, the same applies: there will often be some generic parts of stuff that you won't use for *every* request that you still won't split out into a seperate file simply because they are *often* used.. I think I'd have to guess that around 40% of the code I load gets used on a single request, but that is because of the aforementioned problem of using classes. It's not that big a worry to me, because I use APC to lessen the hit, and thankfully(? :)) I don't have thousands of r/s, just a few hundred. Seems like a good change to me. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On Feb 26, 2009, at 5:58 AM, Rodrigo Saboya wrote: Ronald, I think you are overreacting a little bit. It may be that proper written could would get no benefit from this patch since it would not load unneeded code and this patch ends up speeding up environments where such correct loading isn't done. I don't think that's a reason to disqualify a feature that brings benefits with no significant drawbacks. For the average PHP programmer, the language will simply get faster. That can't be bad in any way. It doesn't encourage you to write bad code, it just doesn't kick you in the nuts when you do. Sold. (The best arguments are always the short ones). As I said: I'm trying to raise a token flag of discussion/ resistance, but nobody in the discussion seems to have found any really important downsides, yet, with only me raising *any* sort of flag (though I admit it's not much of one). :) -Bop -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Ronald Chmara wrote: On Feb 21, 2009, at 10:55 PM, shire wrote: Hi Ronald, Ronald Chmara wrote: Wait... so if I understand this right, let's envision a code base where, per some random page load, 70 functions are actually called, but, oh, 7,000, or even 700,000, are being included for whatever reason? The speed optimization is in *not* copying a massive amount of things that weren't even needed, or used, in the first place? Essentially, yes, this is probably best summed up by the 80/20 rule where we only use 20% of the code etc... Well, I can see 80% actually *used* code, with 20% in there by accident but 80% unused code? eep! ack! Call the villagers and get the torches and pitchforks!... I should probably be more specific, it's not that you *don't* use the other 80% it's that you use 20% of the code 80% of the time: http://en.wikipedia.org/wiki/Pareto_principle Of course I'm abusing this slightly, but this is the *basic* idea, and it's not based on actual usage statistics. In reality you probably use all 100% of your code in smaller chunks depending on the request. As an illustrative example, if you have HTTP requests for setting, retrieving, deleting items in a database which is supported by corresponding functions. In each request you'll only use one of these functions (30%), but you'll still need to load the entire file/class. The goal of lazy loading is to optimize this and similar situations so you don't have to re-organize your code into unfeasibly small files, or in other ways that might inhibit productivity. One thing I see as quite a beneficial future outcome of your work is the ability to further profile code, and be able to seek out code that marks massive amounts of functions as available without actually ever using them. I think this would be better instrumented through tools actually designed to do this sort of profiling, specifically XDebug, or tools like inclued are very useful. http://pecl.php.net/package/xdebug http://pecl.php.net/package/inclued It's actually likely that only a fraction of the code at Facebook will be used in a request, hence the need for lazy loading. Ouch. Seriously. I can't tell you how to build your code, but I think you might seriously benefit from: Does that make sense? Or did you try it already? :) I'd rather not enter into a public discussion of Facebook's current optimization tactics outside of this patch, but suffice to say we have implemented many optimization techniques, both in PHP code and Internals, many of which we have and will continue to contribute back to the community. Lazy Loading has been a very necessary and optimal solution to this one aspect of our scalability, thus allowing our engineers to focus more of their time and energy on creating new and exciting features for Facebook. Thanks, -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On Sun, Feb 22, 2009 at 6:55 AM, shire sh...@tekrat.com wrote: Thanks for the feedback, and hopefully the above makes sense. I don't want to encourage bad progarmming form and would definitely encourage avoiding this situation if possible, however I think many applications may find this optimization a necessity. Hello, I don't think this patch encourages bad programming any way. It just gives more flexibility for the sites that need to focus on performance. I haven't had the chance to check the patch yet but as a principle this is something I would like to see in future versions of PHP / APC. Good work! -- Mikko Koppanen
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On Feb 21, 2009, at 10:55 PM, shire wrote: Hi Ronald, Ronald Chmara wrote: Wait... so if I understand this right, let's envision a code base where, per some random page load, 70 functions are actually called, but, oh, 7,000, or even 700,000, are being included for whatever reason? The speed optimization is in *not* copying a massive amount of things that weren't even needed, or used, in the first place? Essentially, yes, this is probably best summed up by the 80/20 rule where we only use 20% of the code etc... Well, I can see 80% actually *used* code, with 20% in there by accident but 80% unused code? eep! ack! Call the villagers and get the torches and pitchforks!... ...but environments vary, of course. ;) However, there's still the horribly massive speed hit of semi- loading, and marking, a fairly large amount of unused, un-needed, functions, as available? I don't agree with the description of describing this as a horribly massive speed hit at least in comparison with what was happening without lazy loading. Fair enough. Before the patch, for example, I might describe it (80% unused, 20% used code) as an insanely awful, horribly massive speed hit, and after the patch, as being reduced to a much lesser horribly massive speed hit, but these are just rhetorical, and qualitative, language devices that I used to characterize code issues. In both cases, a large amount of CPU is spent on (effectively) doing nothing, but your patch (as I understand its design) reduces the amount of CPU waste... doing nothing. Also, like I said there's further iterations I plan to make here, one of these being increasing the performance of this marking functions as available. One thing I see as quite a beneficial future outcome of your work is the ability to further profile code, and be able to seek out code that marks massive amounts of functions as available without actually ever using them. I do see the benefit of lazy loading, I'm just not very comfortable with enabling a philosophy of loading up a massive amount of CPU and RAM with just in case they're wanted features and code in the first place. Well I am assuming that this is what a large amount of code does already, except that without lazy loading the situation is significantly worse. Different code bases and philosophies vary. Since much of what I do (enterprise PHP tuning) involves (among many other things) finding, and eliminating, such code, I can say with great confidence that there certainly are bloat-bases out there that load metric hogs-heads of libraries to show a single web email form, but there are also code bases which do *not* rely on endless libraries, frameworks, additional template abstractions and end user libraries, or other pre-determined architectures... to complete the simple task of showing an web email form. To frame the issue another way, you are trying to make huge, complicated, code sets less painful to use, and I am arguing that huge, complicated, code sets are a major part of the problem. but since neither of us can wave a magic wand and reduce the problem to simple, elegant, code sets, you're reducing the magnitude of pain involved. Kudos to you. Your point that we should be sure this does not encourage poor coding practices is well taken, but it's been my experience that code tends to take this form regardless so I'm hoping to make the best of the situation ;-). There will always be bad code, yes. ;-) I'm trying to raise a token flag of discussion/resistance to making bad code practices less painful, as it still enables bad code practices. Also keep in mind that there are cases where you may not know in advance which functions you will/will not call, but it's probably fair to say that the 80/20 rule still holds, so including all the functions you may need is not particularly a misuse of the language, but rather a necessity of a dynamic application and language. It all depends on the use, and environment, I suppose. It certainly can boost an APC code set such as facebook, where many of those files and functions *will* likely be used in the next 20 minutes or so, but I also fear that it will encourage programmers to load everything they have, every time, just in case they need it and 2Gb apache processes (and APC space) can be ugly. I'm not entirely clear on where code being used in the next 20 minutes come into play, what differenc does 100 milliseconds vs. 20 minutes make in APC/lazy loading? FB seems to have a fair bit of traffic, with a semi-patrolled code set, so it's likely that any single APC-loaded function will be invoked *eventually*, within an hour or so. Contrast this with 1,000 different sites hosted on a box, using a less patrolled, fairly unregulated, 1,000 different batches of PHP codesets, where myTotallyCustomDateTime() can have 1,000 different variants,
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
On Feb 21, 2009, at 4:10 PM, shire wrote: I've just checked into APC CVS preliminary support for Lazy Loading classes and functions. This means that rather than copying function entries into EG(function_table) and EG(class_table) when an include happen it will mark the functions/classes as available and only actually insert them into the tables when they are called. Wait... so if I understand this right, let's envision a code base where, per some random page load, 70 functions are actually called, but, oh, 7,000, or even 700,000, are being included for whatever reason? The speed optimization is in *not* copying a massive amount of things that weren't even needed, or used, in the first place? However, there's still the horribly massive speed hit of semi- loading, and marking, a fairly large amount of unused, un-needed, functions, as available? While I do see, and understand, the performance benefit of such a coding direction, it's not exactly encouraging well designed code. For example: ?php include(mega-loader.php); //which kicks off -include(site-mega-loader.php); //which kicks off --include(sub-site-mega-loader.php); //which kicks off ---include(sub-site-application-loader.php); //which kicks off include(sub-site-application-feature-loader.php); //which kicks off -include(sub-site-application-function-loader.php); //which kicks off --include(sub-site-application-function-loader-function.php); //and the actual code which caused the load? my_custom_site_application_echo( Hello World); ? This does not make me happy as a coder. I do see the benefit of lazy loading, I'm just not very comfortable with enabling a philosophy of loading up a massive amount of CPU and RAM with just in case they're wanted features and code in the first place. It *should* hurt to load something. Code should not be ever loaded unless needed. It certainly can boost an APC code set such as facebook, where many of those files and functions *will* likely be used in the next 20 minutes or so, but I also fear that it will encourage programmers to load everything they have, every time, just in case they need it and 2Gb apache processes (and APC space) can be ugly. That being said, I think the idea is good, but the possible mis-uses are potentially tough issues. -Bop -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] [RFC] APC/PHP Lazy Loading
Hi Ronald, Ronald Chmara wrote: Wait... so if I understand this right, let's envision a code base where, per some random page load, 70 functions are actually called, but, oh, 7,000, or even 700,000, are being included for whatever reason? The speed optimization is in *not* copying a massive amount of things that weren't even needed, or used, in the first place? Essentially, yes, this is probably best summed up by the 80/20 rule where we only use 20% of the code etc... However, there's still the horribly massive speed hit of semi-loading, and marking, a fairly large amount of unused, un-needed, functions, as available? I don't agree with the description of describing this as a horribly massive speed hit at least in comparison with what was happening without lazy loading. Also, like I said there's further iterations I plan to make here, one of these being increasing the performance of this marking functions as available. I do see the benefit of lazy loading, I'm just not very comfortable with enabling a philosophy of loading up a massive amount of CPU and RAM with just in case they're wanted features and code in the first place. Well I am assuming that this is what a large amount of code does already, except that without lazy loading the situation is significantly worse. Your point that we should be sure this does not encourage poor coding practices is well taken, but it's been my experience that code tends to take this form regardless so I'm hoping to make the best of the situation ;-). Also keep in mind that there are cases where you may not know in advance which functions you will/will not call, but it's probably fair to say that the 80/20 rule still holds, so including all the functions you may need is not particularly a misuse of the language, but rather a necessity of a dynamic application and language. It certainly can boost an APC code set such as facebook, where many of those files and functions *will* likely be used in the next 20 minutes or so, but I also fear that it will encourage programmers to load everything they have, every time, just in case they need it and 2Gb apache processes (and APC space) can be ugly. I'm not entirely clear on where code being used in the next 20 minutes come into play, what differenc does 100 milliseconds vs. 20 minutes make in APC/lazy loading? It's actually likely that only a fraction of the code at Facebook will be used in a request, hence the need for lazy loading. But we need to make all the functions available due to the dynamic nature of the site and PHP itself as we don't know ahead of time which functions we will need to call. Thanks for the feedback, and hopefully the above makes sense. I don't want to encourage bad progarmming form and would definitely encourage avoiding this situation if possible, however I think many applications may find this optimization a necessity. -shire -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
