php-general Digest 2 Sep 2013 08:57:25 -0000 Issue 8352
php-general Digest 2 Sep 2013 08:57:25 - Issue 8352 Topics (messages 322017 through 322020): Re: PHP-5.5.2 +opcache segfaults with Piwik 322017 by: Lester Caine 322018 by: Jan Ehrhardt 322019 by: Jan Ehrhardt 322020 by: Jan Ehrhardt Administrivia: To subscribe to the digest, e-mail: php-general-digest-subscr...@lists.php.net To unsubscribe from the digest, e-mail: php-general-digest-unsubscr...@lists.php.net To post to the list, e-mail: php-gene...@lists.php.net -- ---BeginMessage--- Grant wrote: I've tried php-5.5.2 and 5.5.3 but both segfault with piwik unless the opcache is disabled. Someone filed a piwik bug but was told it's a php bug: http://dev.piwik.org/trac/ticket/4093 Is this a known issue? I'm running my own port of piwik in production with eaccelerator on 5.4 but I've made the decision not to move any of the machines to 5.5 until all of the legacy 5.2 machines have been brought up to 5.4. Not an answer, but explains perhaps why the problem is not being seen by others. -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk ---End Message--- ---BeginMessage--- Lester Caine in php.general (Sun, 01 Sep 2013 12:59:18 +0100): Grant wrote: I've tried php-5.5.2 and 5.5.3 but both segfault with piwik unless the opcache is disabled. Someone filed a piwik bug but was told it's a php bug: http://dev.piwik.org/trac/ticket/4093 Is this a known issue? (...) Not an answer, but explains perhaps why the problem is not being seen by others. It is more likely that users do not notice it. I have Piwik running with PHP 5.3 and php_opcache.dll (Centos 5) and it segfaults: [Sun Sep 01 17:06:53.131410 2013] [core:notice] [pid 25411] AH00052: child pid 25451 exit signal Segmentation fault (11) [Sun Sep 01 17:08:18.561917 2013] [core:notice] [pid 25411] AH00052: child pid 25453 exit signal Segmentation fault (11) [Sun Sep 01 17:08:19.569714 2013] [core:notice] [pid 25411] AH00052: child pid 25450 exit signal Segmentation fault (11) However, nothing special is displaying on the page. You have to grep your Apache error_log to know that it happens. How did you notice? Jan ---End Message--- ---BeginMessage--- Grant in php.general (Sun, 1 Sep 2013 02:13:54 -0700): I've tried php-5.5.2 and 5.5.3 but both segfault with piwik unless the opcache is disabled. Someone filed a piwik bug but was told it's a php bug: http://dev.piwik.org/trac/ticket/4093 Is this a known issue? I changed the PHP on my (dev) Centos5 server to PHP 5.5.3, pulled the latest OPcache from Github and tried to get a segfault. That was clearly visible when I tried to access the Goals tab in the dashboard of one of my websites. The page never got further than 'Loading data...' and my Apache log showed a segfault. So, if it was no known issue it is confirmed now. Jan ---End Message--- ---BeginMessage--- Grant in php.general (Sun, 25 Aug 2013 02:31:29 -0700): I've tried php-5.5.2 and 5.5.3 but both segfault with piwik unless the opcache is disabled. Someone filed a piwik bug but was told it's a php bug: http://dev.piwik.org/trac/ticket/4093 Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This seemed to correct this fatal error on my side: [02-Sep-2013 10:35:40 Europe/Paris] PHP Fatal error: Cannot redeclare _json_encode() (previously declared in /usr/local/lib/php/share/piwik/libs/upgradephp/upgrade.php:109) in /usr/local/lib/php/share/piwik/libs/upgradephp/upgrade.php on line 109 I do not know what opcache has to do with it, although I suspect that Piwik is calling itself a lot of times and that opcache is trailing behind (or something like that). Jan ---End Message---
php-general Digest 3 Sep 2013 01:31:03 -0000 Issue 8353
php-general Digest 3 Sep 2013 01:31:03 - Issue 8353 Topics (messages 322021 through 322023): Re: PHP-5.5.2 +opcache segfaults with Piwik 322021 by: Jan Ehrhardt 322022 by: Lester Caine refernces, arrays, and why does it take up so much memory? 322023 by: Daevid Vincent Administrivia: To subscribe to the digest, e-mail: php-general-digest-subscr...@lists.php.net To unsubscribe from the digest, e-mail: php-general-digest-unsubscr...@lists.php.net To post to the list, e-mail: php-gene...@lists.php.net -- ---BeginMessage--- Jan Ehrhardt in php.general (Mon, 02 Sep 2013 10:57:14 +0200): Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This patch, together with upgrading to the latest OPcache from github solved my segfaults and fatal errors. Jan ---End Message--- ---BeginMessage--- Jan Ehrhardt wrote: Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This patch, together with upgrading to the latest OPcache from github solved my segfaults and fatal errors. Jan - could you post the solution on http://dev.piwik.org/trac/ticket/4093 - better that you get the credit than one of us pinch it ;) -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk ---End Message--- ---BeginMessage--- I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want to do is make the array key the id of the result set row. This is the basic gist of it: private function _normalize_result_set() { foreach($this-tmp_results as $k = $v) { $id = $v['id']; $new_tmp_results[$id] = $v; //2013-08-29 [dv] using a reference here cuts the memory usage in half! unset($this-tmp_results[$k]); /* if ($i++ % 1000 == 0) { gc_enable(); // Enable Garbage Collector var_dump(gc_enabled()); // true var_dump(gc_collect_cycles()); // # of elements cleaned up gc_disable(); // Disable Garbage Collector } */ } $this-tmp_results = $new_tmp_results; //var_dump($this-tmp_results); exit; unset($new_tmp_results); } Without using the = reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552) 6904 = array (size=4) 'id' = string '6904' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '34|' (length=3) 6905 = array (size=4) 'id' = string '6905' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '6|37|' (length=5) However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd EXPECT it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552) 6904 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) 6905 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this-tmp_results as $k = $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All I can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results array in a tricky way to avoid a second array. The concept being I'd add 1 million to the ['id'] (which we want as the new array key), then unset the existing sequential key, then when all done, loop through and shift all the keys by 1
[PHP] Re: PHP-5.5.2 +opcache segfaults with Piwik
Grant in php.general (Sun, 25 Aug 2013 02:31:29 -0700): I've tried php-5.5.2 and 5.5.3 but both segfault with piwik unless the opcache is disabled. Someone filed a piwik bug but was told it's a php bug: http://dev.piwik.org/trac/ticket/4093 Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This seemed to correct this fatal error on my side: [02-Sep-2013 10:35:40 Europe/Paris] PHP Fatal error: Cannot redeclare _json_encode() (previously declared in /usr/local/lib/php/share/piwik/libs/upgradephp/upgrade.php:109) in /usr/local/lib/php/share/piwik/libs/upgradephp/upgrade.php on line 109 I do not know what opcache has to do with it, although I suspect that Piwik is calling itself a lot of times and that opcache is trailing behind (or something like that). Jan -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Re: PHP-5.5.2 +opcache segfaults with Piwik
Jan Ehrhardt in php.general (Mon, 02 Sep 2013 10:57:14 +0200): Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This patch, together with upgrading to the latest OPcache from github solved my segfaults and fatal errors. Jan -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Re: PHP-5.5.2 +opcache segfaults with Piwik
Jan Ehrhardt wrote: Could you try to add a function_exists check to libs/upgradephp/upgrade.php? This at the function declaration of _json_encode: if (!function_exists('_json_encode')) { function _json_encode($var, ... And a extra } at the end. This patch, together with upgrading to the latest OPcache from github solved my segfaults and fatal errors. Jan - could you post the solution on http://dev.piwik.org/trac/ticket/4093 - better that you get the credit than one of us pinch it ;) -- Lester Caine - G8HFL - Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] refernces, arrays, and why does it take up so much memory?
I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want to do is make the array key the id of the result set row. This is the basic gist of it: private function _normalize_result_set() { foreach($this-tmp_results as $k = $v) { $id = $v['id']; $new_tmp_results[$id] = $v; //2013-08-29 [dv] using a reference here cuts the memory usage in half! unset($this-tmp_results[$k]); /* if ($i++ % 1000 == 0) { gc_enable(); // Enable Garbage Collector var_dump(gc_enabled()); // true var_dump(gc_collect_cycles()); // # of elements cleaned up gc_disable(); // Disable Garbage Collector } */ } $this-tmp_results = $new_tmp_results; //var_dump($this-tmp_results); exit; unset($new_tmp_results); } Without using the = reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552) 6904 = array (size=4) 'id' = string '6904' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '34|' (length=3) 6905 = array (size=4) 'id' = string '6905' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '6|37|' (length=5) However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd EXPECT it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552) 6904 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) 6905 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this-tmp_results as $k = $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All I can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results array in a tricky way to avoid a second array. The concept being I'd add 1 million to the ['id'] (which we want as the new array key), then unset the existing sequential key, then when all done, loop through and shift all the keys by 1 million thereby they'd be the right index ID. So add one and unset one immediately after. Clever right? 'cept it too made no difference on memory. Same thing is happening as above where the gc() isn't running or something is holding all that memory until the end. *sigh* Then I tried a different way using array_combine() and noticed something very disturbing. http://www.php.net/manual/en/function.array-combine.php private function _normalize_result_set() { if (!$this-tmp_results || count($this-tmp_results) 1) return; $D_start_mem_usage = memory_get_usage(); foreach($this-tmp_results as $k = $v) { $id = $v['id']; $tmp_keys[] = $id; if ($v['genres']) { $g = explode('|', $v['genres']); $this-tmp_results[$k]['g'] = $g; //this causes a massive spike in memory usage } } //var_dump($tmp_keys, $this-tmp_results); exit; echo \nMEMORY USED BEFORE array_combine: .number_format(memory_get_usage() - $D_start_mem_usage). PEAK: (.number_format(memory_get_peak_usage(true)).)br\n; $this-tmp_results = array_combine($tmp_keys, $this-tmp_results); echo \nMEMORY USED FOR array_combine: .number_format(memory_get_usage() - $D_start_mem_usage). PEAK: (.number_format(memory_get_peak_usage(true)).)br\n; var_dump($tmp_keys, $this-tmp_results); exit; } Just the simple act of adding that 'g' variable element to the array causes a massive change in memory usage. WHAT THE F!? MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264) MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264) And taking out the $this-tmp_results[$k]['g'] = $g; Results in MEMORY USED BEFORE array_combine:
[PHP] Re: refernces, arrays, and why does it take up so much memory?
On 9/2/2013 9:30 PM, Daevid Vincent wrote: I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want to do is make the array key the id of the result set row. This is the basic gist of it: private function _normalize_result_set() { foreach($this-tmp_results as $k = $v) { $id = $v['id']; $new_tmp_results[$id] = $v; //2013-08-29 [dv] using a reference here cuts the memory usage in half! unset($this-tmp_results[$k]); /* if ($i++ % 1000 == 0) { gc_enable(); // Enable Garbage Collector var_dump(gc_enabled()); // true var_dump(gc_collect_cycles()); // # of elements cleaned up gc_disable(); // Disable Garbage Collector } */ } $this-tmp_results = $new_tmp_results; //var_dump($this-tmp_results); exit; unset($new_tmp_results); } Without using the = reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552) 6904 = array (size=4) 'id' = string '6904' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '34|' (length=3) 6905 = array (size=4) 'id' = string '6905' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '6|37|' (length=5) However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd EXPECT it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552) 6904 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) 6905 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this-tmp_results as $k = $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All I can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results array in a tricky way to avoid a second array. The concept being I'd add 1 million to the ['id'] (which we want as the new array key), then unset the existing sequential key, then when all done, loop through and shift all the keys by 1 million thereby they'd be the right index ID. So add one and unset one immediately after. Clever right? 'cept it too made no difference on memory. Same thing is happening as above where the gc() isn't running or something is holding all that memory until the end. *sigh* Then I tried a different way using array_combine() and noticed something very disturbing. http://www.php.net/manual/en/function.array-combine.php private function _normalize_result_set() { if (!$this-tmp_results || count($this-tmp_results) 1) return; $D_start_mem_usage = memory_get_usage(); foreach($this-tmp_results as $k = $v) { $id = $v['id']; $tmp_keys[] = $id; if ($v['genres']) { $g = explode('|', $v['genres']); $this-tmp_results[$k]['g'] = $g; //this causes a massive spike in memory usage } } //var_dump($tmp_keys, $this-tmp_results); exit; echo \nMEMORY USED BEFORE array_combine: .number_format(memory_get_usage() - $D_start_mem_usage). PEAK: (.number_format(memory_get_peak_usage(true)).)br\n; $this-tmp_results = array_combine($tmp_keys, $this-tmp_results); echo \nMEMORY USED FOR array_combine: .number_format(memory_get_usage() - $D_start_mem_usage). PEAK: (.number_format(memory_get_peak_usage(true)).)br\n; var_dump($tmp_keys, $this-tmp_results); exit; } Just the simple act of adding that 'g' variable element to the array causes a massive change in memory usage. WHAT THE F!? MEMORY USED BEFORE array_combine: 105,315,264 PEAK: (224,395,264) MEMORY USED FOR array_combine: 106,573,040 PEAK: (224,395,264) And taking out the
RE: [PHP] Re: refernces, arrays, and why does it take up so much memory?
-Original Message- From: Jim Giner [mailto:jim.gi...@albanyhandball.com] Sent: Monday, September 02, 2013 8:14 PM To: php-general@lists.php.net Subject: [PHP] Re: refernces, arrays, and why does it take up so much memory? On 9/2/2013 9:30 PM, Daevid Vincent wrote: I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want to do is make the array key the id of the result set row. This is the basic gist of it: private function _normalize_result_set() { foreach($this-tmp_results as $k = $v) { $id = $v['id']; $new_tmp_results[$id] = $v; //2013-08-29 [dv] using a reference here cuts the memory usage in half! unset($this-tmp_results[$k]); /* if ($i++ % 1000 == 0) { gc_enable(); // Enable Garbage Collector var_dump(gc_enabled()); // true var_dump(gc_collect_cycles()); // # of elements cleaned up gc_disable(); // Disable Garbage Collector } */ } $this-tmp_results = $new_tmp_results; //var_dump($this-tmp_results); exit; unset($new_tmp_results); } Without using the = reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552) 6904 = array (size=4) 'id' = string '6904' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '34|' (length=3) 6905 = array (size=4) 'id' = string '6905' (length=4) 'studio_id' = string '5' (length=1) 'genres' = string '6|37|' (length=5) However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd EXPECT it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552) 6904 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) 6905 = array (size=4) 'id' = string '86260' (length=5) 'studio_id' = string '210' (length=3) 'genres' = string '8|9|10|29|58|' (length=13) Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this-tmp_results as $k = $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All I can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results array in a tricky way to avoid a second array. The concept being I'd add 1 million to the ['id'] (which we want as the new array key), then unset the existing sequential key, then when all done, loop through and shift all the keys by 1 million thereby they'd be the right index ID. So add one and unset one immediately after. Clever right? 'cept it too made no difference on memory. Same thing is happening as above where the gc() isn't running or something is holding all that memory until the end. *sigh* Then I tried a different way using array_combine() and noticed something very disturbing. http://www.php.net/manual/en/function.array-combine.php private function _normalize_result_set() { if (!$this-tmp_results || count($this-tmp_results) 1) return; $D_start_mem_usage = memory_get_usage(); foreach($this-tmp_results as $k = $v) { $id = $v['id']; $tmp_keys[] = $id; if ($v['genres']) { $g = explode('|', $v['genres']); $this-tmp_results[$k]['g'] = $g; //this causes a massive spike in memory usage } } //var_dump($tmp_keys, $this-tmp_results); exit; echo \nMEMORY USED BEFORE array_combine: .number_format(memory_get_usage() - $D_start_mem_usage). PEAK: (.number_format(memory_get_peak_usage(true)).)br\n; $this-tmp_results = array_combine($tmp_keys, $this-tmp_results); echo \nMEMORY USED FOR