On Sun, Jan 27, 2013 at 2:57 PM, ammon_lymphater < ammon_lympha...@hotmail.com> wrote:
> > I think I localized the bug - potential query memory use with larger cache > sizes > > Details > - created a database and started in a loop 10 times { import 1m > records; run query} > - for each iteration observe the memory use by SQLITE3 process > using > task manager > - the mimimum use was during the load, the maximum during the query > > Below you will find the observations of memory size for three different > cache sizes (Page size 32768) > > Comments: > - for 2k pages the maximum levels off at d (4m rows) at 220mb, so > the query mem use is 220-60 = 160mb > - for 10k pages the maximum grows by 50mb for each million rows, > query mem is 160 mb initially, but 500mb for 10m rows > - for 30k pages cache the picture is similar to that of 10k pages > cache > > Summary: it looks like for large cache sizes the query has some sort of > memory mis-management. > Run on the full table with 500m records but cache size limited to 2k pages > the same query was fine. > Have you measured SQLite's memory utilization directly using the sqlite3_memory_used() routine described at http://www.sqlite.org/c3ref/memory_highwater.html Do you have any other evidence to suggest that the problem is in SQLite and not in your application? > > The coding is 'f 66 220' means that during loading of the 6-th million of > rows (f) the minimum memory reported > Was 66mb, maximum 220mb) > cache 2k pages > a 90 b 64 166 c 64 ... d 64 219 e 64 220 f 66 220 g 65 220 h 65 > 220 > i 65 220 j 65 220 > > cache 10k pages > a 135 187 b 202 372 c 307 469 d 320 521 e 320 571 f 318 622 g 319 672 h 322 > 722 i 318 772 j 323 824 > > cache 30k pages > > a 187 270 b .. 373 c 363 559 d 421 746 e 562 932 f 711 1175 g 820 1302 h > 974 > 1354 i 960 1403 j 961 1455 > -- D. Richard Hipp d...@sqlite.org _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users