ID: 40261 Updated by: [EMAIL PROTECTED] Reported By: thuejk at gmail dot com -Status: Open +Status: Assigned Bug Type: Performance problem Operating System: All PHP Version: 5.2.0 Assigned To: dmitry
Previous Comments: ------------------------------------------------------------------------ [2007-01-30 14:53:40] thuejk at gmail dot com The latest PHP snapshot does fix my example, and probably makes my production code work. This will probably fix the problem in far most cases. But it is possible to construct an example which still have the problematic behavior. One example is below. <?php $num = 100000; $a = Array(); for ($i=0; $i<$num; $i++) { $a[$i] = Array(1); } for ($i=0; $i<$num; $i++) { $b[$i] = $a[$i][0]; } unset($a); for ($i=0; $i<$num; $i++) { $b[$i] = "1234567890123456789012345678901234567890123456789012345678901234567\ 8901234567890123456789012345678901234567890123456789012345678901234567890123456\ 7890123456789012345678901234567890123456789012345678901234567890123456789012345\ 6789012345678901234567890123456789012345678901234567890123456789012345678901234\ 5678901234567890"; } ?> ------------------------------------------------------------------------ [2007-01-30 14:03:32] [EMAIL PROTECTED] Please try PHP_5_2 snapshot. It already uses litle bit different "best-fit" implementation and this script takes reasonable time (near the same as 5.1). ------------------------------------------------------------------------ [2007-01-29 13:24:47] thuejk at gmail dot com I added a few printf's and found out that the $num generated "holes" in the memory are 384 big. Zend has a special system for small allocations, but that only works for holes below ZEND_MM_SMALL_SIZE=280. The $num allocations at the end, which cause the problems, and 40 or 88 long. Note that these numbers are from a 64-bit machine, which of course has 8-byte pointers, and so a larger MM overhead. (The bug does also occur on 32bit machines) One temporary solution could be to raise #define ZEND_MM_NUM_BUCKETS 32 from which ZEND_MM_SMALL_SIZE is defined, so that ZEND_MM_SMALL_SIZE is made twice or perhaps 4 times as big. (I got a segfault when I tried that, haven't looked into why) A better solution would be to organize the free blocks in a balanced tree, instead of a linear list which has to be traversed. ------------------------------------------------------------------------ [2007-01-29 01:42:28] thuejk at gmail dot com Ok, after a good deal of work I actually tracked down the problem to fragmentation in your memory management. This understanding enabled me to construct a simpler test case. Note that the problem is unrelated to database function, and this new testcase does not require any database. <?php //According to my calculations, setting $num=18000000 should make this script take ~one year. $num = 100000; $a = Array(); for ($i=0; $i<$num; $i++) { $a[$i] = Array(1); } /* The underlying memory now looks something like * row structure * row value * row structure * ... */ $b = Array(); for ($i=0; $i<$num; $i++) { $b[$i] = $a[$i][0]; } /* The b rows are allocated * * Though I haven't checked it in the code, I have reason to believe * that the values inserted into the b rows are pointers to the same * memory allocated for the values in the a rows */ unset($a); /* The a rows are unallocated, but the values are still references from $b, so the memory map looks something like * row value * free_block * row_value * free_block * ... * * repeated $num times. */ /* Now, for each memory allocation for a row, PHP runs through all * free blocks checking for a best fit. Since there are $num free * blocks, this takes time. This is done by the function * _zend_mm_alloc_int in PHP 5.2.0 Zend/zend_alloc.c : * * end = &heap->free_buckets[0]; * for (p = end->next_free_block; p != end; p = p->next_free_block) { * size_t s = ZEND_MM_FREE_BLOCK_SIZE(p); * if (s > true_size) { * if (s < best_size) { // better fit * best_fit = p; * best_size = s; * } * } else if (s == true_size) { * // Found "big" free block of exactly the same size * best_fit = p; * goto zend_mm_finished_searching_for_block; * } * } */ $c = Array(); for ($i=0; $i<$num; $i++) { $c[$i] = 1; } ?> ------------------------------------------------------------------------ [2007-01-28 01:52:37] thuejk at gmail dot com On second though, black boxes are not good for testing. A simpler timer added... ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/40261 -- Edit this bug report at http://bugs.php.net/?id=40261&edit=1