Re: [PHP-DEV] Persistent zvals
hi,
On Tue, Apr 10, 2012 at 9:24 AM, Flavius Aspra wrote:
> I also subscribe to the opinion of object persistency not being worth it. In
> an application server, you could probably achieve more by keeping the entire
> phar in RAM.
>
> Currently, you can also simulate that by putting the phar on a ramdisk or
> similar.
That's unrelated. Also having the phar in RAM is also not that useful
as it is exactly what an opcode cache does, while keeping the parsed
("compiled") code in memory cache.
Pre instantiated instances mean than a given app won't have to do all
the include foo;+ new foo(); As it is somehow expensive, relative the
the footprint of a given framework core.
Cheers,
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
--
PHP Internals - PHP Runtime Development Mailing List
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 10, 2012, at 12:24 AM, Flavius Aspra wrote: > On 04/09/2012 10:30 PM, Luke Scott wrote: >> Yeah it would be. He also mentioned something about preloading >> framework classes.. Would like to hear his thoughts on that! > > I also subscribe to the opinion of object persistency not being worth it. In > an application server, you could probably achieve more by keeping the entire > phar in RAM. > > Currently, you can also simulate that by putting the phar on a ramdisk or > similar. This is just the php code though isn't it? It still needs to be converted to opcodes, then executed. Is it possible to execute the phar in a "ready state" (before request, after init of several objects) and make a snapshot of that and use that for every request? (Shouldn't have to be a phar though.) Luke -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On 04/09/2012 10:30 PM, Luke Scott wrote: Yeah it would be. He also mentioned something about preloading framework classes.. Would like to hear his thoughts on that! I also subscribe to the opinion of object persistency not being worth it. In an application server, you could probably achieve more by keeping the entire phar in RAM. Currently, you can also simulate that by putting the phar on a ramdisk or similar. -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
Pierre, On Apr 7, 2012, at 11:14 AM, Pierre Joye wrote: > Something I have been discussed in the past with a couple of persons > is to have a mechanism to automatically instantiate objects on request > start. That's not persistent objects but that could already boost a > little it the base foot print of some frameworks. Would you be able to go into more detail on how this would be implemented? Could something like this be done with a zend extension? Would it involve loading a "loader.php" file and then taking a snapshot of the memory state, or something like that (before loading the actual file index.php file, similar to the prepend feature)? Luke -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 9, 2012, at 7:49 AM, Flavius Aspra wrote: > On 04/07/2012 05:21 AM, Luke Scott wrote: >> From what I've gathered thus far, it is impossible to do without copying the >> >> non-persistent memory into persistent memory, and then back again. > > Hi, glad to see you again StackOverflow user:-) > > I think I've shown you the route by that [1] project, and yes, the emalloc() > call is hardcoded in so many places that you'd have to do it this way. The problem with that is serialization. You can pretty much do the same thing with APC. > > But as Pierre said, an application server for PHP (PHP-FPM?) would be great. > The stub part of a .phar file could play a big role there. Yeah it would be. He also mentioned something about preloading framework classes.. Would like to hear his thoughts on that! Luke > > Regards, > Flavius > > [1] https://github.com/flavius/php-persist > -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On 04/07/2012 05:21 AM, Luke Scott wrote: From what I've gathered thus far, it is impossible to do without copying the non-persistent memory into persistent memory, and then back again. Hi, glad to see you again StackOverflow user:-) I think I've shown you the route by that [1] project, and yes, the emalloc() call is hardcoded in so many places that you'd have to do it this way. But as Pierre said, an application server for PHP (PHP-FPM?) would be great. The stub part of a .phar file could play a big role there. Regards, Flavius [1] https://github.com/flavius/php-persist -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 7, 2012, at 11:14 AM, Pierre Joye wrote: > On Sat, Apr 7, 2012 at 6:51 PM, Luke Scott wrote: > >> I think I understand what you're getting at. So to avoid fragmentation >> you would have to have two independent memory spaces. Making non >> persistent memory persistent would require copying from one space to >> another. Is that correct? > > Yes, that's correct. > >> How is this handled in other platforms where you have an application >> instance (with state, if that's the correct terminology)? > > Application server is something different. PHP is not one, as it could > be possible to do that using the builtin web server (dev only :). An application server would be cool though. I was thinking along the lines of having each process be an instance of the application.. thus the idea about persisting objects. >> Are there any other options for reliably persisting objects/variables >> (without having to recreate)? > > Something I have been discussed in the past with a couple of persons > is to have a mechanism to automatically instantiate objects on request > start. That's not persistent objects but that could already boost a > little it the base foot print of some frameworks. How would that work? I'm intrigued. Luke > > Cheers, > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Sat, Apr 7, 2012 at 6:51 PM, Luke Scott wrote: > I think I understand what you're getting at. So to avoid fragmentation > you would have to have two independent memory spaces. Making non > persistent memory persistent would require copying from one space to > another. Is that correct? Yes, that's correct. > How is this handled in other platforms where you have an application > instance (with state, if that's the correct terminology)? Application server is something different. PHP is not one, as it could be possible to do that using the builtin web server (dev only :). > Are there any other options for reliably persisting objects/variables > (without having to recreate)? Something I have been discussed in the past with a couple of persons is to have a mechanism to automatically instantiate objects on request start. That's not persistent objects but that could already boost a little it the base foot print of some frameworks. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 7, 2012, at 9:45 AM, Pierre Joye wrote: > hi, > > On Sat, Apr 7, 2012 at 4:59 PM, Luke Scott wrote: > >>> The expensive parts here are not the object creation on its own but to >>> get the data they contain (external, calculation, etc.). >> >> With heavy objects, yes. But when you have hundreds of objects? The >> unseralization process in no more efficient than creating the objects >> directly. > > That's what I am saying, yes. Hence that I do not think persistent > objects are that much helpful. I guess it depends on the use case. > >> It depends. It's only cheaper when you have heavy objects. >> >> How do other platforms with a persistent application instance handle this? > > It is horrible in Java. There are some work being made in JS (server > side) too which looks good, but it is basically the same as we discuss > here. Are we talking about v8cgi or Node? Node has a single application instance, but everything has to be asynchronous... > >> I think the bigger issues are how to persist the memory without >> fragmentation. Stanislav talked about that (although those comments >> didn't make it to the list - my fault for not hitting reply all). > > Then post it again here. Done. > >>> The serialization method can be optimized easily, igbinary for example >>> can help to speed up this process. >> >> Optimized, sure. But you can only go so far. You still have the cost >> of recreating those objects. The only way to make it truly zero is by >> not recreating them. > > Right, but the gain is not worth it except in hello world example. > Well, for the large majority of the use cases. In a hello world the difference is probably not measurable. When you start to see it is with hundreds of class instances. > >> igbinary helps. But what about a large binary tree? The only way to >> efficiently keep this in memory (currently) is to write a C/C++ >> extension that persists the memory as a resource. But this takes much >> more time and effort than writing it in PHP. > > I wonder about the opportunity and design correctness to keep large > binary trees in memory, but that's maybe another discussion :) I personally don't, but it was one of the use cases I found where serialization had no benefit. > > >> With a lot of large frameworks you spend more time initializing the >> framework than the request itself (besides querying external resources >> like MySQL). > > In most if not any applications, "(besides querying external resources > like MySQL)" is what takes the largest part of the request time. > That's why I do not think persistent zval is going to bring us that > far. True, for most PHP applications today. But when you hit these limitations you often have to work outside PHP and bind that solution to it. Whereas java, as "horrible" as it is, can handle almost any situation. A classic example of something I couldn't do in PHP: Large file uploads (> 2GB). Our solution was writing an app in java, powered by jetty, to stream the file to the server. It works beautifully , but it had to be done outside PHP. > > >> I think the most frustrating thing for me is knowing this isn't an >> issue with web applications written in Java, which has an always on >> application instance that handles requests. I've always viewed java as >> bloated and slow... But with recent developments of Java being jit >> compiled to native code, these issues are starting to go away. > > It is not really what I see. Many projects I have seen have been > moving away from persistent objects across requests (for the part you > are referring to here), as it adds an insane amount of complexity that > is not worth the effort. I suppose it depends on the industry. In the credit card processing industry (gateways) I see Java a lot. I almost never see PHP in this area. Luke > > Cheers, > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 6, 2012, at 10:41 PM, Stas Malyshev wrote: > Hi! > >> How is memory tracked with emalloc? From my observations anything >> created with emalloc, with or without a zval wrapper, is freed at the >> end of the request. Things created with pemalloc, which seems to be a >> wrapper of malloc, isn't thrown away. > > Zend Engine has its own memory manager, see zend_alloc.c. Unfortunately, > I'm not sure I'll be able to explain in short how it works, but TLDR > version of it is that it's like malloc, but private to Zend Engine, so > everything is freed at the end of the request. > >> From what I've seen this is because the zvals, the values inside, the >> object bucket, etc... is all created by emalloc. Correct me if I'm >> wrong, but even if i could persist the zval (wrapper) the value inside >> would be freed unless both the zval and the value were created with >> pemalloc (alias of malloc). It goes a step further with the objects >> because a zend_object is just a handle ID (pointing to a spot in the >> bucket) and a pointer to object handlers. > > This is correct, all internal memory allocations are done by emalloc. > zend_object, however, is not just a handle - this handle refers to > object structure, which has class pointer, properties, etc. > >> and not much better than serialization. I'd like to avoid copying >> memory if possible. A flag on the zval would be ideal. > > That's what I am trying to explain - flag on one zval would not help > you, since this zval uses dozens, if not hundreds (for bigger object) > other resources, including other zvals, hashtables, object structures, > class structures, function structures, etc. etc. > Add to that that if you allocated something in emalloc arena you can not > just mark it "do not free it" - that would create a huge fragmentation > problem if you had something non-freeable at the middle of freeable > region. So you can not just "flag" random variable as non-freeable > without messing up whole memory manager (btw, not advisable with malloc > either - fragmentation can inflate you memory usage very easily). Not > that it would work even if you could, see above. I think I understand what you're getting at. So to avoid fragmentation you would have to have two independent memory spaces. Making non persistent memory persistent would require copying from one space to another. Is that correct? How is this handled in other platforms where you have an application instance (with state, if that's the correct terminology)? Are there any other options for reliably persisting objects/variables (without having to recreate)? (Re-sending to the list) > -- > Stanislav Malyshev, Software Architect > SugarCRM: http://www.sugarcrm.com/ > (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
hi, On Sat, Apr 7, 2012 at 4:59 PM, Luke Scott wrote: >> The expensive parts here are not the object creation on its own but to >> get the data they contain (external, calculation, etc.). > > With heavy objects, yes. But when you have hundreds of objects? The > unseralization process in no more efficient than creating the objects > directly. That's what I am saying, yes. Hence that I do not think persistent objects are that much helpful. > It depends. It's only cheaper when you have heavy objects. > > How do other platforms with a persistent application instance handle this? It is horrible in Java. There are some work being made in JS (server side) too which looks good, but it is basically the same as we discuss here. > I think the bigger issues are how to persist the memory without > fragmentation. Stanislav talked about that (although those comments > didn't make it to the list - my fault for not hitting reply all). Then post it again here. >> The serialization method can be optimized easily, igbinary for example >> can help to speed up this process. > > Optimized, sure. But you can only go so far. You still have the cost > of recreating those objects. The only way to make it truly zero is by > not recreating them. Right, but the gain is not worth it except in hello world example. Well, for the large majority of the use cases. > igbinary helps. But what about a large binary tree? The only way to > efficiently keep this in memory (currently) is to write a C/C++ > extension that persists the memory as a resource. But this takes much > more time and effort than writing it in PHP. I wonder about the opportunity and design correctness to keep large binary trees in memory, but that's maybe another discussion :) > With a lot of large frameworks you spend more time initializing the > framework than the request itself (besides querying external resources > like MySQL). In most if not any applications, "(besides querying external resources like MySQL)" is what takes the largest part of the request time. That's why I do not think persistent zval is going to bring us that far. > I think the most frustrating thing for me is knowing this isn't an > issue with web applications written in Java, which has an always on > application instance that handles requests. I've always viewed java as > bloated and slow... But with recent developments of Java being jit > compiled to native code, these issues are starting to go away. It is not really what I see. Many projects I have seen have been moving away from persistent objects across requests (for the part you are referring to here), as it adds an insane amount of complexity that is not worth the effort. Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 7, 2012, at 5:22 AM, Pierre Joye wrote: > hi, > > 2012/4/7 Luke Scott : > >> It would be ideal of you could initialize a bunch of objects once and >> carry them over to the next request. No serialization, no copying. A >> lot of framework objects would benefit from this. > > The expensive parts here are not the object creation on its own but to > get the data they contain (external, calculation, etc.). With heavy objects, yes. But when you have hundreds of objects? The unseralization process in no more efficient than creating the objects directly. > That's why we have magic methods like __sleep and __wakeup. Making > objects or values persistent across requests is not possible or very > hard. It is way cleaner to create them on each request and save the > expensive computation or data fetching time (see doctrine for example) > instead of trying to figure out tricks to get rid of the references > and all other ways to mess up a zval between requests. It depends. It's only cheaper when you have heavy objects. How do other platforms with a persistent application instance handle this? I don't believe references are really that big of an issue. You still have scope and refcounts. The difference is now refcounts are ignored and everything is thrown away at the end of the request vs refcounts being observed and memory persisting past the request. I think the bigger issues are how to persist the memory without fragmentation. Stanislav talked about that (although those comments didn't make it to the list - my fault for not hitting reply all). > The serialization method can be optimized easily, igbinary for example > can help to speed up this process. Optimized, sure. But you can only go so far. You still have the cost of recreating those objects. The only way to make it truly zero is by not recreating them. igbinary helps. But what about a large binary tree? The only way to efficiently keep this in memory (currently) is to write a C/C++ extension that persists the memory as a resource. But this takes much more time and effort than writing it in PHP. > It is also important to keep in mind that PHP is not designed to have > persitant user land values and won't be. > There are very good tools to > store these data, like memcached, couchbase, redis and co for the > distributed ones, apc and other for local caches (even like mysqlnd_cc > for caching mysql results :). And these are great solutions to a different set of problems. But they aren't the token solution to every problem. With a lot of large frameworks you spend more time initializing the framework than the request itself (besides querying external resources like MySQL). I think the most frustrating thing for me is knowing this isn't an issue with web applications written in Java, which has an always on application instance that handles requests. I've always viewed java as bloated and slow... But with recent developments of Java being jit compiled to native code, these issues are starting to go away. I have invested 10 years into PHP. Id rather stick with what I know. But it scares me that many enterprises in our sector chose Java over PHP. Luke > > Cheers, > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
hi, 2012/4/7 Luke Scott : > It would be ideal of you could initialize a bunch of objects once and > carry them over to the next request. No serialization, no copying. A > lot of framework objects would benefit from this. The expensive parts here are not the object creation on its own but to get the data they contain (external, calculation, etc.). That's why we have magic methods like __sleep and __wakeup. Making objects or values persistent across requests is not possible or very hard. It is way cleaner to create them on each request and save the expensive computation or data fetching time (see doctrine for example) instead of trying to figure out tricks to get rid of the references and all other ways to mess up a zval between requests. The serialization method can be optimized easily, igbinary for example can help to speed up this process. It is also important to keep in mind that PHP is not designed to have persitant user land values and won't be. There are very good tools to store these data, like memcached, couchbase, redis and co for the distributed ones, apc and other for local caches (even like mysqlnd_cc for caching mysql results :). Cheers, -- Pierre @pierrejoye | http://blog.thepimp.net | http://www.libgd.org -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
Indeed phpdaemon.net appears to cover this ground, although your idea might deliver higher performance. Sent from my iPhone On Apr 6, 2012, at 7:46 PM, Luke Scott wrote: > I've spent the last few days pouring over the Zend engine source code. I > think I have a basic understanding on the memory management. Likely what I > say may be incorrect, so I apologize in advance. > > What I'm trying to do is write an extension to persist PHP variables past > the end of the request in the SAPI to be used on the next request (different > user, same process). I'd like to do this without serialization. The reason > is certain objects in PHP (in a framework, for example) are not request > specific and it would be useful to have these objects already loaded and > usable without having to construct them on every request. (Sample use case: > Dependency injection like Guice without having to write physical containers > - using reflection, written in PHP) > > From what I've gathered thus far, it is impossible to do without copying the > non-persistent memory into persistent memory, and then back again. I'm > assuming this is because all the memory associated with PHP variables use > emalloc, which places it onto a stack that is disposed of at the end of the > request. > > So there seems to only be two ways to do this: > > 1 - Copy non-persistent memory into persistent memory (and back) using a > deep copy. Probably not very efficient. May not be much better than > serialization. > > 2 - Modify the Zend engine to flag objects/zvals as persistent so they > aren¹t thrown away until the process ends. > > #2 seems to be the better way to go. #1 seems to be the only way to do it as > an extension (maintaining its own stack). > > There seems to have been some discussion (7 years ago) of this mentioned > here under 6.9: > > http://www.php.net/~derick/meeting-notes.html > > I've been able to do it somewhat with string zvals, but objects are a > different story (given that a zval contains a handle index referring an > entry in a bucket). The "goal", at least with objects, is the objects > doesn't destruct until the end of the process. With copying memory it looks > like I'd probably have to copy the object into my own bucket, modify the > original in zend's bucket so the destructor isn't called (destructor_callled > = 1), and then at the start of the request copy what's in my bucket into > zend's bucket. > > At this point I feel light a mad scientist. I'm hoping to gain some insight > on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if > it involves modifying the Zend engine. Have you guys had any recent > discussions about doing this? > > Luke > > > -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
Neat idea. I have considered writing persistent php servers that just accept requests serially. Possibly you could write a fastcgi server in pure php. That would still delegate most of the grunt work to apache. Sent from my iPhone On Apr 6, 2012, at 7:46 PM, Luke Scott wrote: > I've spent the last few days pouring over the Zend engine source code. I > think I have a basic understanding on the memory management. Likely what I > say may be incorrect, so I apologize in advance. > > What I'm trying to do is write an extension to persist PHP variables past > the end of the request in the SAPI to be used on the next request (different > user, same process). I'd like to do this without serialization. The reason > is certain objects in PHP (in a framework, for example) are not request > specific and it would be useful to have these objects already loaded and > usable without having to construct them on every request. (Sample use case: > Dependency injection like Guice without having to write physical containers > - using reflection, written in PHP) > > From what I've gathered thus far, it is impossible to do without copying the > non-persistent memory into persistent memory, and then back again. I'm > assuming this is because all the memory associated with PHP variables use > emalloc, which places it onto a stack that is disposed of at the end of the > request. > > So there seems to only be two ways to do this: > > 1 - Copy non-persistent memory into persistent memory (and back) using a > deep copy. Probably not very efficient. May not be much better than > serialization. > > 2 - Modify the Zend engine to flag objects/zvals as persistent so they > aren¹t thrown away until the process ends. > > #2 seems to be the better way to go. #1 seems to be the only way to do it as > an extension (maintaining its own stack). > > There seems to have been some discussion (7 years ago) of this mentioned > here under 6.9: > > http://www.php.net/~derick/meeting-notes.html > > I've been able to do it somewhat with string zvals, but objects are a > different story (given that a zval contains a handle index referring an > entry in a bucket). The "goal", at least with objects, is the objects > doesn't destruct until the end of the process. With copying memory it looks > like I'd probably have to copy the object into my own bucket, modify the > original in zend's bucket so the destructor isn't called (destructor_callled > = 1), and then at the start of the request copy what's in my bucket into > zend's bucket. > > At this point I feel light a mad scientist. I'm hoping to gain some insight > on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if > it involves modifying the Zend engine. Have you guys had any recent > discussions about doing this? > > Luke > > > -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 6, 2012, at 5:23 PM, "Johannes Schlüter" wrote: > Hi, > > On Fri, 2012-04-06 at 16:46 -0700, Luke Scott wrote: >> >> >> From what I've gathered thus far, it is impossible to do without >> copying the non-persistent memory into persistent memory, and then >> back again. I'm assuming this is because all the memory associated >> with PHP variables use emalloc, which places it onto a stack that is >> disposed of at the end of the request. >> >> So there seems to only be two ways to do this: >> >> 1 - Copy non-persistent memory into persistent memory (and back) using >> a deep copy. Probably not very efficient. May not be much better than >> serialization. > > Yes, see apc_store() and friends to see all the small parts needed to > copy it as properly as possible. While this still won't work for all > cases (resources, internal classes, ..) but well, this might be what > you're looking for in a ready state ;-) Apc serializes the data. Unless the data is a cache or session information there is almost no difference between unseralizing a bunch of objects verses and initializing the objects in code. It would be ideal of you could initialize a bunch of objects once and carry them over to the next request. No serialization, no copying. A lot of framework objects would benefit from this. > >> 2 - Modify the Zend engine to flag objects/zvals as persistent so they >> aren¹t thrown away until the process ends. > > Which is a major undertaking, taking copy-on-write and friends into > account. As in a case like > >mark_persistent($persistent); >$persistent['some key'] = function_returning_lots_of_data(); > > would suddenly require to create a copy of all the data. Such things can > quickly cost more than recreating the structures more frequently ... When having to copy data, yes thing yet messy very quickly. But if there were a persistent flag on the zval the new zval would have the persistent flag. The old one would be discarded when it's reference count reached zero. Right now everything is thrown out at the end of the request because it's created with emalloc, regardless of the recount. With a persistent flag (1 or 0) and a recount of 1 or greater it wouldn't be until the end of the process. Easier said than done - for sure. > > johannes > > -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
On Apr 6, 2012, at 5:16 PM, Stas Malyshev wrote: Hi! >From what I've gathered thus far, it is impossible to do without copying the non-persistent memory into persistent memory, and then back again. I'm assuming this is because all the memory associated with PHP variables use emalloc, which places it onto a stack that is disposed of at the end of the request. The stack part is wrong, emalloc does not use stack, but the general sense of it is correct - all memory allocated by emalloc is freed at the end of the request, so anything you would want to preserve would have to be copied to some other memory area. How is memory tracked with emalloc? From my observations anything created with emalloc, with or without a zval wrapper, is freed at the end of the request. Things created with pemalloc, which seems to be a wrapper of malloc, isn't thrown away. I saw some references (off the top if my head - on my phone) to something like G(mm_stack)->_malloc(...), which is called by emalloc. Could you briefly explain how it does work or point be to a link that does? Up until now I've been pouring over the source code - and there is a lot, tracing down macro after macro. 2 - Modify the Zend engine to flag objects/zvals as persistent so they aren¹t thrown away until the process ends. The problem with that is that these variables can contain references to other variables and other things (like class in object, etc.) that will be freed, so the data will point to freed memory. Unless the variable is a simple scalar or you do not allow associating it with any other variables in any way you would have this problem. If it's a simple scalar, you can do better by using existing caches. Not allowing associations with other variables means there's not much reason for this thing to actually be a PHP variable. >From what I've seen this is because the zvals, the values inside, the object bucket, etc... is all created by emalloc. Correct me if I'm wrong, but even if i could persist the zval (wrapper) the value inside would be freed unless both the zval and the value were created with pemalloc (alias of malloc). It goes a step further with the objects because a zend_object is just a handle ID (pointing to a spot in the bucket) and a pointer to object handlers. I've been able to do it somewhat with string zvals, but objects are a different story (given that a zval contains a handle index referring an entry in a bucket). The "goal", at least with objects, is the objects doesn't destruct until the end of the process. With copying memory it looks With objects you have a problem - do you also mean to keep the class? What about the methods code? If you intend to keep all this in memory - and remember you'd also have to guard all of it so no "local" variable gets anywhere into any of object's properties, or sub-values of these properties - I'm not sure you'd do better than serializing. At this point I feel light a mad scientist. I'm hoping to gain some insight on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if it involves modifying the Zend engine. Have you guys had any recent discussions about doing this? Doing it is much harder than you think, since all structures in Zend Engine are assumed to be temporary and you'd need to copy a lot of stuff to make your object work. I wouldn't really advise it. Don't get me wrong - I totally get it. Trying to copy the zvals, the value inside, and the object out of the bucket, and then the descendants is not a good solution. It would probably be buggy at best and not much better than serialization. I'd like to avoid copying memory if possible. A flag on the zval would be ideal. With the continued assumption that memory structures are temporary something like this will never be feasible. In order for something like this to work (at least from my understanding. - I'm probably totally wrong) values inside a zval would have to be created with malloc and the life cycle of that object would have to be determined by the wrapping zval. A zval would have a persistent flag and a recount the value inside would be released when the recount reaches zero or at the end of the request if the persistent flag is zero But this is something totally different than it is now. Luke -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227
Re: [PHP-DEV] Persistent zvals
Hi, On Fri, 2012-04-06 at 16:46 -0700, Luke Scott wrote: > > > From what I've gathered thus far, it is impossible to do without > copying the non-persistent memory into persistent memory, and then > back again. I'm assuming this is because all the memory associated > with PHP variables use emalloc, which places it onto a stack that is > disposed of at the end of the request. > > So there seems to only be two ways to do this: > > 1 - Copy non-persistent memory into persistent memory (and back) using > a deep copy. Probably not very efficient. May not be much better than > serialization. Yes, see apc_store() and friends to see all the small parts needed to copy it as properly as possible. While this still won't work for all cases (resources, internal classes, ..) but well, this might be what you're looking for in a ready state ;-) > 2 - Modify the Zend engine to flag objects/zvals as persistent so they > aren¹t thrown away until the process ends. Which is a major undertaking, taking copy-on-write and friends into account. As in a case like mark_persistent($persistent); $persistent['some key'] = function_returning_lots_of_data(); would suddenly require to create a copy of all the data. Such things can quickly cost more than recreating the structures more frequently ... johannes -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] Persistent zvals
Hi! > From what I've gathered thus far, it is impossible to do without copying the > non-persistent memory into persistent memory, and then back again. I'm > assuming this is because all the memory associated with PHP variables use > emalloc, which places it onto a stack that is disposed of at the end of the > request. The stack part is wrong, emalloc does not use stack, but the general sense of it is correct - all memory allocated by emalloc is freed at the end of the request, so anything you would want to preserve would have to be copied to some other memory area. > 2 - Modify the Zend engine to flag objects/zvals as persistent so they > aren¹t thrown away until the process ends. The problem with that is that these variables can contain references to other variables and other things (like class in object, etc.) that will be freed, so the data will point to freed memory. Unless the variable is a simple scalar or you do not allow associating it with any other variables in any way you would have this problem. If it's a simple scalar, you can do better by using existing caches. Not allowing associations with other variables means there's not much reason for this thing to actually be a PHP variable. > I've been able to do it somewhat with string zvals, but objects are a > different story (given that a zval contains a handle index referring an > entry in a bucket). The "goal", at least with objects, is the objects > doesn't destruct until the end of the process. With copying memory it looks With objects you have a problem - do you also mean to keep the class? What about the methods code? If you intend to keep all this in memory - and remember you'd also have to guard all of it so no "local" variable gets anywhere into any of object's properties, or sub-values of these properties - I'm not sure you'd do better than serializing. > At this point I feel light a mad scientist. I'm hoping to gain some insight > on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if > it involves modifying the Zend engine. Have you guys had any recent > discussions about doing this? Doing it is much harder than you think, since all structures in Zend Engine are assumed to be temporary and you'd need to copy a lot of stuff to make your object work. I wouldn't really advise it. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ (408)454-6900 ext. 227 -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
[PHP-DEV] Persistent zvals
I've spent the last few days pouring over the Zend engine source code. I think I have a basic understanding on the memory management. Likely what I say may be incorrect, so I apologize in advance. What I'm trying to do is write an extension to persist PHP variables past the end of the request in the SAPI to be used on the next request (different user, same process). I'd like to do this without serialization. The reason is certain objects in PHP (in a framework, for example) are not request specific and it would be useful to have these objects already loaded and usable without having to construct them on every request. (Sample use case: Dependency injection like Guice without having to write physical containers - using reflection, written in PHP) >From what I've gathered thus far, it is impossible to do without copying the non-persistent memory into persistent memory, and then back again. I'm assuming this is because all the memory associated with PHP variables use emalloc, which places it onto a stack that is disposed of at the end of the request. So there seems to only be two ways to do this: 1 - Copy non-persistent memory into persistent memory (and back) using a deep copy. Probably not very efficient. May not be much better than serialization. 2 - Modify the Zend engine to flag objects/zvals as persistent so they aren¹t thrown away until the process ends. #2 seems to be the better way to go. #1 seems to be the only way to do it as an extension (maintaining its own stack). There seems to have been some discussion (7 years ago) of this mentioned here under 6.9: http://www.php.net/~derick/meeting-notes.html I've been able to do it somewhat with string zvals, but objects are a different story (given that a zval contains a handle index referring an entry in a bucket). The "goal", at least with objects, is the objects doesn't destruct until the end of the process. With copying memory it looks like I'd probably have to copy the object into my own bucket, modify the original in zend's bucket so the destructor isn't called (destructor_callled = 1), and then at the start of the request copy what's in my bucket into zend's bucket. At this point I feel light a mad scientist. I'm hoping to gain some insight on how this might be done properly with PHP 5.3/5.4 (or just 5.4) - even if it involves modifying the Zend engine. Have you guys had any recent discussions about doing this? Luke
