RE: [PHP] Re: refernces, arrays, and why does it take up so much memory?
> -Original Message- > From: Jim Giner [mailto:jim.gi...@albanyhandball.com] > Sent: Monday, September 02, 2013 8:14 PM > To: php-general@lists.php.net > Subject: [PHP] Re: refernces, arrays, and why does it take up so much > memory? > > On 9/2/2013 9:30 PM, Daevid Vincent wrote: > > I'm confused on how a reference works I think. > > > > I have a DB result set in an array I'm looping over. All I simply want to > do > > is make the array key the "id" of the result set row. > > > > This is the basic gist of it: > > > > private function _normalize_result_set() > > { > >foreach($this->tmp_results as $k => $v) > >{ > > $id = $v['id']; > > $new_tmp_results[$id] =& $v; //2013-08-29 [dv] using > a > > reference here cuts the memory usage in half! > > unset($this->tmp_results[$k]); > > > > /* > > if ($i++ % 1000 == 0) > > { > > gc_enable(); // Enable Garbage Collector > > var_dump(gc_enabled()); // true > > var_dump(gc_collect_cycles()); // # of > elements > > cleaned up > > gc_disable(); // Disable Garbage Collector > > } > > */ > >} > >$this->tmp_results = $new_tmp_results; > >//var_dump($this->tmp_results); exit; > >unset($new_tmp_results); > > } > > > > Without using the =& reference, my data works great: > > $new_tmp_results[$id] = $v; > > > > array (size=79552) > >6904 => > > array (size=4) > >'id' => string '6904' (length=4) > >'studio_id' => string '5' (length=1) > >'genres' => string '34|' (length=3) > >6905 => > > array (size=4) > >'id' => string '6905' (length=4) > >'studio_id' => string '5' (length=1) > >'genres' => string '6|37|' (length=5) > > > > However it takes a stupid amount of memory for some unknown reason. > > MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES > > MEMORY PEAK USAGE: 216,530,944 BYTES > > > > When using the reference the memory drastically goes down to what I'd > EXPECT > > it to be (and actually the problem I'm trying to solve). > > MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES > > MEMORY PEAK USAGE: 82,051,072 BYTES > > > > However my array is all kinds of wrong: > > > > array (size=79552) > >6904 => & > > array (size=4) > >'id' => string '86260' (length=5) > >'studio_id' => string '210' (length=3) > >'genres' => string '8|9|10|29|58|' (length=13) > >6905 => & > > array (size=4) > >'id' => string '86260' (length=5) > >'studio_id' => string '210' (length=3) > >'genres' => string '8|9|10|29|58|' (length=13) > > > > Notice that they're all the same values, although the keys seem right. I > > don't understand why that happens because > > foreach($this->tmp_results as $k => $v) > > Should be changing $v each iteration I'd think. > > > > Honestly, I am baffled as to why those unsets() make no difference. All I > > can think is that the garbage collector doesn't run. But then I had also > > tried to force gc() and that still made no difference. *sigh* > > > > I had some other cockamamie idea where I'd use the same tmp_results array > in > > a tricky way to avoid a second array. The concept being I'd add 1 million > > to the ['id'] (which we want as the new array key), then unset the > existing > > sequential key, then when all done, loop through and shift all the keys by > 1 > > million thereby they'd be the right index ID. So add one and unset one > > immediately after. Clever right? 'cept it too made no difference on > memory. > > Same thing is happening as above where the gc() isn't running or something > > is holding all that memory until the end. *sigh* > > > > Then I tried a different way using array_combine() and noticed something > > very disturbing. > > http://www.php.net/manual/en/function.array-combine.php > > > > > > private function _normalize_result_set() > > { > >if (!$this->tmp_results || count($this->tmp_results) < 1) > > return; > > > >$D_start_mem_usage = memory_get_usage(); > >foreach($this->tmp_results as $k => $v) > >{ > > $id = $v['id']; > > $tmp_keys[] = $id; > > > > if ($v['genres']) > > { > > $g = explode('|', $v['genres']); > > $this->tmp_results[$k]['g'] = $g; //this > causes a > > massive spike in memory usage > > } > >} > >//var_dump($tmp_keys, $this->tmp_results); exit; > >echo "\nMEMORY U
Re: [PHP] Re: refernces, arrays, and why does it take up so much memory?
On 9/3/2013 1:09 AM, Daevid Vincent wrote: -Original Message- From: Jim Giner [mailto:jim.gi...@albanyhandball.com] Sent: Monday, September 02, 2013 8:14 PM To: php-general@lists.php.net Subject: [PHP] Re: refernces, arrays, and why does it take up so much memory? On 9/2/2013 9:30 PM, Daevid Vincent wrote: I'm confused on how a reference works I think. I have a DB result set in an array I'm looping over. All I simply want to do is make the array key the "id" of the result set row. This is the basic gist of it: private function _normalize_result_set() { foreach($this->tmp_results as $k => $v) { $id = $v['id']; $new_tmp_results[$id] =& $v; //2013-08-29 [dv] using a reference here cuts the memory usage in half! unset($this->tmp_results[$k]); /* if ($i++ % 1000 == 0) { gc_enable(); // Enable Garbage Collector var_dump(gc_enabled()); // true var_dump(gc_collect_cycles()); // # of elements cleaned up gc_disable(); // Disable Garbage Collector } */ } $this->tmp_results = $new_tmp_results; //var_dump($this->tmp_results); exit; unset($new_tmp_results); } Without using the =& reference, my data works great: $new_tmp_results[$id] = $v; array (size=79552) 6904 => array (size=4) 'id' => string '6904' (length=4) 'studio_id' => string '5' (length=1) 'genres' => string '34|' (length=3) 6905 => array (size=4) 'id' => string '6905' (length=4) 'studio_id' => string '5' (length=1) 'genres' => string '6|37|' (length=5) However it takes a stupid amount of memory for some unknown reason. MEMORY USED @START: 262,144 - @END: 42,729,472 = 42,467,328 BYTES MEMORY PEAK USAGE: 216,530,944 BYTES When using the reference the memory drastically goes down to what I'd EXPECT it to be (and actually the problem I'm trying to solve). MEMORY USED @START: 262,144 - @END: 6,029,312 = 5,767,168 BYTES MEMORY PEAK USAGE: 82,051,072 BYTES However my array is all kinds of wrong: array (size=79552) 6904 => & array (size=4) 'id' => string '86260' (length=5) 'studio_id' => string '210' (length=3) 'genres' => string '8|9|10|29|58|' (length=13) 6905 => & array (size=4) 'id' => string '86260' (length=5) 'studio_id' => string '210' (length=3) 'genres' => string '8|9|10|29|58|' (length=13) Notice that they're all the same values, although the keys seem right. I don't understand why that happens because foreach($this->tmp_results as $k => $v) Should be changing $v each iteration I'd think. Honestly, I am baffled as to why those unsets() make no difference. All I can think is that the garbage collector doesn't run. But then I had also tried to force gc() and that still made no difference. *sigh* I had some other cockamamie idea where I'd use the same tmp_results array in a tricky way to avoid a second array. The concept being I'd add 1 million to the ['id'] (which we want as the new array key), then unset the existing sequential key, then when all done, loop through and shift all the keys by 1 million thereby they'd be the right index ID. So add one and unset one immediately after. Clever right? 'cept it too made no difference on memory. Same thing is happening as above where the gc() isn't running or something is holding all that memory until the end. *sigh* Then I tried a different way using array_combine() and noticed something very disturbing. http://www.php.net/manual/en/function.array-combine.php private function _normalize_result_set() { if (!$this->tmp_results || count($this->tmp_results) < 1) return; $D_start_mem_usage = memory_get_usage(); foreach($this->tmp_results as $k => $v) { $id = $v['id']; $tmp_keys[] = $id; if ($v['genres']) { $g = explode('|', $v['genres']); $this->tmp_results[$k]['g'] = $g; //this causes a massive spike in memory usage } } //var_dump($tmp_keys, $this->tmp_results); exit; echo "\nMEMORY USED BEFORE array_combine: ".number_format(memory_get_usage() - $D_start_mem_usage)." PEAK: (".number_format(memory_get_peak_usage(true)).")\n"; $this->tmp_results = array_combine($tmp_keys, $this->tmp_results); echo "\nMEMORY USED FOR array_combine: ".number_format(memory_get_usage() - $D_start_mem_usag