Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[John Drescher]

> What happens if you insert more than your RAM size into an in memory
> database?
> (I'm particularly interested in the Windows context).
>

Are we talking about 32bit windows? I mean under 32 bit windows the
normal address space limit (without the /3GB switch and
LARGEADDRESSAWARE link flag) is 2GB and fragmented so that the largest
single process allocation (without resorting to AWE) is somewhere
around 1.2GB to 1.5GB even if you have a system with 4GB of RAM and
8GB of swap space.

John
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Jay A. Kreibich]

On Tue, Apr 19, 2011 at 03:29:42PM +0100, Simon Slavin scratched on the wall:
> 
> On 19 Apr 2011, at 2:58pm, Adam DeVita wrote:
> 
> > Our application is typically implemented on a standard laptop PC.  It seems
> > that the symptoms displayed are consistent with what this list describes
> > would happen,  so it looks like I can start thinking of how to write a
> > defence.   It does suddenly become very slow.
> > 
> > I think the potential solutions we may implement  are all in application
> > code, so not really an SQLite problem.
> 
> If you are using an in-memory database purely for speed, you might try
> the simple change of not doing that.  Make your database use disk
> space as any normal one would, and delete it after you close it. 
> Windows is pretty good at caching stuff these days and on-disk
> databases aren't as slow as some people would guess.

  You can pass the sqlite3_open*() calls an empty string for the
  filename.  That will create a disk-backed temporary database that
  automatically cleans up after itself.  If you crank the SQLite cache
  up, the performance should be roughly the same as an in-memory
  database, right up until it runs out of memory.  After that, the
  performance should be a bit better, as SQLite's cache is likely to
  be more efficient than paging memory to disk.

   -j

-- 
Jay A. Kreibich < J A Y  @  K R E I B I.C H >

"Intelligence is like underwear: it is important that you have it,
 but showing it to the wrong people has the tendency to make them
 feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Simon Slavin]

On 19 Apr 2011, at 2:58pm, Adam DeVita wrote:

> Our application is typically implemented on a standard laptop PC.  It seems
> that the symptoms displayed are consistent with what this list describes
> would happen,  so it looks like I can start thinking of how to write a
> defence.   It does suddenly become very slow.
> 
> I think the potential solutions we may implement  are all in application
> code, so not really an SQLite problem.

If you are using an in-memory database purely for speed, you might try the 
simple change of not doing that.  Make your database use disk space as any 
normal one would, and delete it after you close it.  Windows is pretty good at 
caching stuff these days and on-disk databases aren't as slow as some people 
would guess.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Adam DeVita]

Thanks for the responses.

Our application is typically implemented on a standard laptop PC.  It seems
that the symptoms displayed are consistent with what this list describes
would happen,  so it looks like I can start thinking of how to write a
defence.   It does suddenly become very slow.

I think the potential solutions we may implement  are all in application
code, so not really an SQLite problem.

Thanks,
Adam



On Mon, Apr 18, 2011 at 10:07 AM, eLaReF  wrote:

> Talking as a Windows user only rather than an SQL expert (I'm not even
> good enough to call myself a beginner!)
>
> Are we talking about a small netbook type with only say 8GB of memory
> and no hard drive.
>
> If a Windows m/c has a hard drive, surely virtual memory
> (drive-swapping) comes into play?
> It would, of course become v-e-r-y slow in comparison.
>
>
> eLaReF
>
>
>
>
> On 18/04/2011 14:46, Pavel Ivanov wrote:
> > You won't be able to insert. The statement will fail.
> >
> > Pavel
> >
> >
> > On Mon, Apr 18, 2011 at 9:44 AM, Adam DeVita
>  wrote:
> >> Good day,
> >>
> >> What happens if you insert more than your RAM size into an in memory
> >> database?
> >> (I'm particularly interested in the Windows context).
> >>
> >> regards,
> >> Adam
> >> ___
> >> sqlite-users mailing list
> >> sqlite-users@sqlite.org
> >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >>
> > ___
> > sqlite-users mailing list
> > sqlite-users@sqlite.org
> > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> >
> >
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
VerifEye Technologies Inc.
905-948-0015x245
151 Whitehall Dr, Unit 2
Markham ON, L3R 9T1
Canada
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[eLaReF]

Talking as a Windows user only rather than an SQL expert (I'm not even 
good enough to call myself a beginner!)

Are we talking about a small netbook type with only say 8GB of memory 
and no hard drive.

If a Windows m/c has a hard drive, surely virtual memory 
(drive-swapping) comes into play?
It would, of course become v-e-r-y slow in comparison.


eLaReF




On 18/04/2011 14:46, Pavel Ivanov wrote:
> You won't be able to insert. The statement will fail.
>
> Pavel
>
>
> On Mon, Apr 18, 2011 at 9:44 AM, Adam DeVita  wrote:
>> Good day,
>>
>> What happens if you insert more than your RAM size into an in memory
>> database?
>> (I'm particularly interested in the Windows context).
>>
>> regards,
>> Adam
>> ___
>> sqlite-users mailing list
>> sqlite-users@sqlite.org
>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>>
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Simon Slavin]

On 18 Apr 2011, at 2:44pm, Adam DeVita wrote:

> What happens if you insert more than your RAM size into an in memory
> database?

Windows doesn't really keep all its memory in actual RAM.  It uses a 'page 
file' to allow it to send data it's not actually using right now to disk, and 
call it back when it needs it.  You generally allow it to use more paging space 
than you have actual space in RAM.  Setting the amount of disk space it can use 
for paging is a system setting.  For instance, this is how it's done in Windows 
XP:



As you can see, you can set a specific size.

So your question becomes '... more than your page space ...'.  But the answer 
is what Pavel wrote: the function you called with INSERT returns a SQLite 
result code indicating failure of some appropriate kind.  You should deal with 
that result just as carefully as you'd deal with any other unexpected non-zero 
result code.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Black, Michael (IS)]

Ummm...are we forgetting about swap space?



If you exceed RAM you hit swap.  If you exceed RAM+SWAP you start failing.



Or does sqlite monitor physical memory usage?



So if you exceed RAM you just start slowing down towards a disk-based equialent 
database.



Michael D. Black

Senior Scientist

NG Information Systems

Advanced Analytics Directorate




From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on 
behalf of Pavel Ivanov [paiva...@gmail.com]
Sent: Monday, April 18, 2011 8:46 AM
To: General Discussion of SQLite Database
Subject: EXT :Re: [sqlite] What happens if you insert more than your RAM size 
into an in memory database?

You won't be able to insert. The statement will fail.

Pavel


On Mon, Apr 18, 2011 at 9:44 AM, Adam DeVita  wrote:
> Good day,
>
> What happens if you insert more than your RAM size into an in memory
> database?
> (I'm particularly interested in the Windows context).
>
> regards,
> Adam
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] What happens if you insert more than your RAM size into an in memory database?

[Pavel Ivanov]

You won't be able to insert. The statement will fail.

Pavel


On Mon, Apr 18, 2011 at 9:44 AM, Adam DeVita  wrote:
> Good day,
>
> What happens if you insert more than your RAM size into an in memory
> database?
> (I'm particularly interested in the Windows context).
>
> regards,
> Adam
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] What happens if you insert more than your RAM size into an in memory database?

[Adam DeVita]

Good day,

What happens if you insert more than your RAM size into an in memory
database?
(I'm particularly interested in the Windows context).

regards,
Adam
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users