On Sun, Jan 27, 2013 at 2:57 PM, ammon_lymphater <
[email protected]> wrote:
>
> I think I localized the bug - potential query memory use with larger cache
> sizes
>
> Details
> - created a database and started in a loop 10 times { import 1m
> records; run query}
> - for each iteration observe the memory use by SQLITE3 process
> using
> task manager
> - the mimimum use was during the load, the maximum during the query
>
> Below you will find the observations of memory size for three different
> cache sizes (Page size 32768)
>
> Comments:
> - for 2k pages the maximum levels off at d (4m rows) at 220mb, so
> the query mem use is 220-60 = 160mb
> - for 10k pages the maximum grows by 50mb for each million rows,
> query mem is 160 mb initially, but 500mb for 10m rows
> - for 30k pages cache the picture is similar to that of 10k pages
> cache
>
> Summary: it looks like for large cache sizes the query has some sort of
> memory mis-management.
> Run on the full table with 500m records but cache size limited to 2k pages
> the same query was fine.
>
Have you measured SQLite's memory utilization directly using the
sqlite3_memory_used() routine described at
http://www.sqlite.org/c3ref/memory_highwater.html
Do you have any other evidence to suggest that the problem is in SQLite and
not in your application?
>
> The coding is 'f 66 220' means that during loading of the 6-th million of
> rows (f) the minimum memory reported
> Was 66mb, maximum 220mb)
> cache 2k pages
> a 90 b 64 166 c 64 ... d 64 219 e 64 220 f 66 220 g 65 220 h 65
> 220
> i 65 220 j 65 220
>
> cache 10k pages
> a 135 187 b 202 372 c 307 469 d 320 521 e 320 571 f 318 622 g 319 672 h 322
> 722 i 318 772 j 323 824
>
> cache 30k pages
>
> a 187 270 b .. 373 c 363 559 d 421 746 e 562 932 f 711 1175 g 820 1302 h
> 974
> 1354 i 960 1403 j 961 1455
>
--
D. Richard Hipp
[email protected]
_______________________________________________
sqlite-users mailing list
[email protected]
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
