[sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Lee Crain]

Is there any reason why this would not be possible? 

Data persistence is not required.

Thanks,

Lee Crain


-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Rich Shepard]

On Mon, 20 Aug 2007, Lee Crain wrote:


Is there any reason why this would not be possible?


Lee,

  Not at all. You can create the databse in memory as long as you have
sufficient storage for it. See the web pages for syntax. If you do not
specify a filename when you invoke sqlite3, it's an in-memory database on
which you can perform all SQL operations.

Rich

--
Richard B. Shepard, Ph.D.   |The Environmental Permitting
Applied Ecosystem Services, Inc.|  Accelerator(TM)
 Voice: 503-667-4517  Fax: 503-667-8863

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[RaghavendraK 70574]

Hi,

But the performance is very bad in comparision with the file based DB,
any suggestion to improve the same.

regards
ragha
**
 This email and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!
 
*

- Original Message -
From: Rich Shepard <[EMAIL PROTECTED]>
Date: Tuesday, August 21, 2007 7:35 am
Subject: Re: [sqlite] A Question About Creating and Accessing a SQLite Database 
in a RAM Drive

> On Mon, 20 Aug 2007, Lee Crain wrote:
> 
> > Is there any reason why this would not be possible?
> 
> Lee,
> 
>   Not at all. You can create the databse in memory as long as you 
> havesufficient storage for it. See the web pages for syntax. If 
> you do not
> specify a filename when you invoke sqlite3, it's an in-memory 
> database on
> which you can perform all SQL operations.
> 
> Rich
> 
> -- 
> Richard B. Shepard, Ph.D.   |The Environmental 
> PermittingApplied Ecosystem Services, Inc.|  
> Accelerator(TM)<http://www.appl-ecosys.com> Voice: 503-667-
> 4517  Fax: 503-667-8863
> 
> ---
> --
> To unsubscribe, send email to [EMAIL PROTECTED]
> ---
> --
> 
> 

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Lee Crain]

Rich,

The approach I planned was a little different than what you proposed.

I've done some experimentation with updating a SQLite database by
overwriting it with an updated copy of the database. It's a simple file
overwrite, it's fast, and the database, with its new data contents, is
fully useable.

My plan:
1. Create a RAM drive.
1. Copy a hard drive based SQLite database from its hard drive to a RAM
drive. 
2. Perform updates to the SQLite database in the RAM drive (for speed). 
3. Copy the RAM based SQLite database back to the hard drive and overwrite
the original SQLite database. 
4. Delete the RAM drive.

This technique for performing database updates offline and then updating
the original database via a file copy operation has worked very well on
hard drives. I am only considering using the RAM drive to improve the
speed of the database updates.

Lee Crain




-Original Message-
From: Rich Shepard [mailto:[EMAIL PROTECTED] 
Sent: Monday, August 20, 2007 5:36 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] A Question About Creating and Accessing a SQLite
Database in a RAM Drive

On Mon, 20 Aug 2007, Lee Crain wrote:

> Is there any reason why this would not be possible?

Lee,

   Not at all. You can create the databse in memory as long as you have
sufficient storage for it. See the web pages for syntax. If you do not
specify a filename when you invoke sqlite3, it's an in-memory database on
which you can perform all SQL operations.

Rich

-- 
Richard B. Shepard, Ph.D.   |The Environmental Permitting
Applied Ecosystem Services, Inc.|  Accelerator(TM)
<http://www.appl-ecosys.com> Voice: 503-667-4517  Fax:
503-667-8863

--
---
To unsubscribe, send email to [EMAIL PROTECTED]
--
---



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Rich Shepard]

On Tue, 21 Aug 2007, Lee Crain wrote:


The approach I planned was a little different than what you proposed.


  That's fine, Lee.


This technique for performing database updates offline and then updating
the original database via a file copy operation has worked very well on
hard drives. I am only considering using the RAM drive to improve the
speed of the database updates.


  This was common in the early 1980s when drives and other hardware were
slow. I've not seen a situation any time recently when this was necessary
with modern hardware and fast memory. When I was capturing real-time data
(lat/lon from the GPS receiver and depth from the sonar), I'd write both to
memory buffers, then write to disk on a regular basis. This let me use
slower hardware (compared to the data flow) while writing to disk in chunks
and ensuring that no data were lost.

  I'm confident that you can tune your database for speed in other ways, but
-- of course -- it's your choice.

Good luck with it,

Rich

--
Richard B. Shepard, Ph.D.   |The Environmental Permitting
Applied Ecosystem Services, Inc.|  Accelerator(TM)
 Voice: 503-667-4517  Fax: 503-667-8863

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

RE: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Lee Crain]

Rich,

We're going to delete and rewrite ~109,369 records in 5 tables every week.


Hard drives are a minimum of 10,000 times slower than RAM. I'll let you
know if this process is not a lot faster than writing the records,
individually, to a hard drive. 

Lee Crain

_


-Original Message-
From: Rich Shepard [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, August 21, 2007 11:15 AM
To: sqlite-users@sqlite.org
Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite
Database in a RAM Drive

On Tue, 21 Aug 2007, Lee Crain wrote:

> The approach I planned was a little different than what you proposed.

   That's fine, Lee.

> This technique for performing database updates offline and then updating
> the original database via a file copy operation has worked very well on
> hard drives. I am only considering using the RAM drive to improve the
> speed of the database updates.

   This was common in the early 1980s when drives and other hardware were
slow. I've not seen a situation any time recently when this was necessary
with modern hardware and fast memory. When I was capturing real-time data
(lat/lon from the GPS receiver and depth from the sonar), I'd write both
to
memory buffers, then write to disk on a regular basis. This let me use
slower hardware (compared to the data flow) while writing to disk in
chunks
and ensuring that no data were lost.

   I'm confident that you can tune your database for speed in other ways,
but
-- of course -- it's your choice.

Good luck with it,

Rich

-- 
Richard B. Shepard, Ph.D.   |The Environmental Permitting
Applied Ecosystem Services, Inc.|  Accelerator(TM)
<http://www.appl-ecosys.com> Voice: 503-667-4517  Fax:
503-667-8863

--
---
To unsubscribe, send email to [EMAIL PROTECTED]
--
---



-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[Scott Hess]

If you're already willing to use a ram drive, you should benchmark
SQLite against a disk-backed file, with 'PRAGMA synchronous = off;'.
For most modern operating systems, this shouldn't provide
significantly different performance than using a RAM disk.

Note that you said 'Data persistence is not required.'  My suggestion
would be to treat such a database as described above exactly as you'd
treat a database on a RAM disk.  With synchronous off, the OS is free
to reorder things pretty arbitrarily, and data may not be written out
for a surprisingly long period, and it may not be written in any
internally-consistent order.

-scott


On 8/21/07, Lee Crain <[EMAIL PROTECTED]> wrote:
> Rich,
>
> We're going to delete and rewrite ~109,369 records in 5 tables every week.
>
>
> Hard drives are a minimum of 10,000 times slower than RAM. I'll let you
> know if this process is not a lot faster than writing the records,
> individually, to a hard drive.
>
> Lee Crain
>
> _
>
>
> -Original Message-
> From: Rich Shepard [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, August 21, 2007 11:15 AM
> To: sqlite-users@sqlite.org
> Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite
> Database in a RAM Drive
>
> On Tue, 21 Aug 2007, Lee Crain wrote:
>
> > The approach I planned was a little different than what you proposed.
>
>That's fine, Lee.
>
> > This technique for performing database updates offline and then updating
> > the original database via a file copy operation has worked very well on
> > hard drives. I am only considering using the RAM drive to improve the
> > speed of the database updates.
>
>This was common in the early 1980s when drives and other hardware were
> slow. I've not seen a situation any time recently when this was necessary
> with modern hardware and fast memory. When I was capturing real-time data
> (lat/lon from the GPS receiver and depth from the sonar), I'd write both
> to
> memory buffers, then write to disk on a regular basis. This let me use
> slower hardware (compared to the data flow) while writing to disk in
> chunks
> and ensuring that no data were lost.
>
>I'm confident that you can tune your database for speed in other ways,
> but
> -- of course -- it's your choice.
>
> Good luck with it,
>
> Rich
>
> --
> Richard B. Shepard, Ph.D.   |The Environmental Permitting
> Applied Ecosystem Services, Inc.|  Accelerator(TM)
> <http://www.appl-ecosys.com> Voice: 503-667-4517  Fax:
> 503-667-8863
>
> --
> ---
> To unsubscribe, send email to [EMAIL PROTECTED]
> --
> ---
>
>
>
> -
> To unsubscribe, send email to [EMAIL PROTECTED]
> -
>
>

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[John Stanton]

Sqlite caches its data using available memory.  Using a memory resident 
database will consequently give you little advantage.  If you dod not 
require the psersistence of disk storage you can relax the ACID 
environment by using the non-synchronous pragma and get the best of both 
wolrds, memory resident speed and disk storage.


Lee Crain wrote:

Rich,

We're going to delete and rewrite ~109,369 records in 5 tables every week.


Hard drives are a minimum of 10,000 times slower than RAM. I'll let you
know if this process is not a lot faster than writing the records,
individually, to a hard drive. 


Lee Crain

_


-Original Message-
From: Rich Shepard [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, August 21, 2007 11:15 AM

To: sqlite-users@sqlite.org
Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite
Database in a RAM Drive

On Tue, 21 Aug 2007, Lee Crain wrote:



The approach I planned was a little different than what you proposed.



   That's fine, Lee.



This technique for performing database updates offline and then updating
the original database via a file copy operation has worked very well on
hard drives. I am only considering using the RAM drive to improve the
speed of the database updates.



   This was common in the early 1980s when drives and other hardware were
slow. I've not seen a situation any time recently when this was necessary
with modern hardware and fast memory. When I was capturing real-time data
(lat/lon from the GPS receiver and depth from the sonar), I'd write both
to
memory buffers, then write to disk on a regular basis. This let me use
slower hardware (compared to the data flow) while writing to disk in
chunks
and ensuring that no data were lost.

   I'm confident that you can tune your database for speed in other ways,
but
-- of course -- it's your choice.

Good luck with it,

Rich




-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: RE: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[RaghavendraK 70574]

Hi,

I have tried with a ram disk and ramfs. Read Performance is same as that of 
disk based system.
How did i evaluate?
wrote simple program
  create file on disk,ramfs
  do pread(...)
In fact sometimes the performance is low when using ramdisk or ramfs or dev/shm.

FileSize: 80MB
read at once: 20bytes
type of read: sequential,meaning in the do loop just increment the offset by 
20bytes.

Pls guide me how pread can be replaced for ramfs which is much more efficent.
Is it possible to avoid this pread system as this consumes significant cpu 
cycles when it switches from user to kernel mode.

regards
ragha


**
 This email and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!
 
*

- Original Message -
From: Rich Shepard <[EMAIL PROTECTED]>
Date: Wednesday, August 22, 2007 1:14 am
Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite Database 
in a RAM Drive

> On Tue, 21 Aug 2007, Lee Crain wrote:
> 
> > The approach I planned was a little different than what you 
> proposed.
>   That's fine, Lee.
> 
> > This technique for performing database updates offline and then 
> updating> the original database via a file copy operation has 
> worked very well on
> > hard drives. I am only considering using the RAM drive to 
> improve the
> > speed of the database updates.
> 
>   This was common in the early 1980s when drives and other 
> hardware were
> slow. I've not seen a situation any time recently when this was 
> necessarywith modern hardware and fast memory. When I was 
> capturing real-time data
> (lat/lon from the GPS receiver and depth from the sonar), I'd 
> write both to
> memory buffers, then write to disk on a regular basis. This let me use
> slower hardware (compared to the data flow) while writing to disk 
> in chunks
> and ensuring that no data were lost.
> 
>   I'm confident that you can tune your database for speed in 
> other ways, but
> -- of course -- it's your choice.
> 
> Good luck with it,
> 
> Rich
> 
> -- 
> Richard B. Shepard, Ph.D.   |The Environmental 
> PermittingApplied Ecosystem Services, Inc.|  
> Accelerator(TM)<http://www.appl-ecosys.com> Voice: 503-667-
> 4517  Fax: 503-667-8863
> 
> ---
> --
> To unsubscribe, send email to [EMAIL PROTECTED]
> ---
> --
> 
> 

-
To unsubscribe, send email to [EMAIL PROTECTED]
-

Re: RE: [sqlite] A Question About Creating and Accessing a SQLite Database in a RAM Drive

[RaghavendraK 70574]

Hi,

Sqlite uses pread and the performance with ramdisk and disk is the same
on Linux SuSE9.

regards
ragha

**
 This email and its attachments contain confidential information from HUAWEI, 
which is intended only for the person or entity whose address is listed above. 
Any use of the information contained herein in any way (including, but not 
limited to, total or partial disclosure, reproduction, or dissemination) by 
persons other than the intended recipient(s) is prohibited. If you receive this 
e-mail in error, please notify the sender by phone or email immediately and 
delete it!
 
*

- Original Message -
From: Lee Crain <[EMAIL PROTECTED]>
Date: Wednesday, August 22, 2007 3:31 am
Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite Database 
in a RAM Drive

> Rich,
> 
> We're going to delete and rewrite ~109,369 records in 5 tables 
> every week.
> 
> 
> Hard drives are a minimum of 10,000 times slower than RAM. I'll 
> let you
> know if this process is not a lot faster than writing the records,
> individually, to a hard drive. 
> 
> Lee Crain
> 
> _
> 
> 
> -Original Message-
> From: Rich Shepard [mailto:[EMAIL PROTECTED] 
> Sent: Tuesday, August 21, 2007 11:15 AM
> To: sqlite-users@sqlite.org
> Subject: RE: [sqlite] A Question About Creating and Accessing a SQLite
> Database in a RAM Drive
> 
> On Tue, 21 Aug 2007, Lee Crain wrote:
> 
> > The approach I planned was a little different than what you 
> proposed.
>   That's fine, Lee.
> 
> > This technique for performing database updates offline and then 
> updating> the original database via a file copy operation has 
> worked very well on
> > hard drives. I am only considering using the RAM drive to 
> improve the
> > speed of the database updates.
> 
>   This was common in the early 1980s when drives and other 
> hardware were
> slow. I've not seen a situation any time recently when this was 
> necessarywith modern hardware and fast memory. When I was 
> capturing real-time data
> (lat/lon from the GPS receiver and depth from the sonar), I'd 
> write both
> to
> memory buffers, then write to disk on a regular basis. This let me use
> slower hardware (compared to the data flow) while writing to disk in
> chunks
> and ensuring that no data were lost.
> 
>   I'm confident that you can tune your database for speed in 
> other ways,
> but
> -- of course -- it's your choice.
> 
> Good luck with it,
> 
> Rich
> 
> -- 
> Richard B. Shepard, Ph.D.   |The Environmental 
> PermittingApplied Ecosystem Services, Inc.|  
> Accelerator(TM)<http://www.appl-ecosys.com> Voice: 503-667-
> 4517  Fax:
> 503-667-8863
> 
> ---
> ---
> ---
> To unsubscribe, send email to [EMAIL PROTECTED]
> ---
> ---
> ---
> 
> 
> 
> ---
> --
> To unsubscribe, send email to [EMAIL PROTECTED]
> ---
> --
> 
> 

-
To unsubscribe, send email to [EMAIL PROTECTED]
-