Re: [sqlalchemy] SQLAlchemy hot-copy capability

[Richard Gerd Kuesters]

*src* can be an in memory database, it works too :)

on backup, I just use:

*        destdb = apsw.Connection(__DATABASE_FILE__)**
**        with destdb.backup("main", _connection, "main") as backup:**
**            while not backup.done:**
**                backup.step(100)*


it's a bit faster then using *time.sleep(...)* :)

cheers,
Richard.

On 09/19/2013 05:08 AM, pr64 wrote:

HI everyone,

First, i would like to thank you for your comments.

Then, for those who run linux, the following works:

 1- copy your db.sqlite to /dev/shm/db.sqlite (thus in ram fs)
 2- play with it through sqlalchemy
 3- backup it to disk when needed

                    src = apsw.Connection(ram_db_file)
                    dest = apsw.Connection(disk_db_file)

                    backup = dest.backup("main", src, "main")
                    try:
                        while not backup.done:

time.sleep(0.1) # let other processes access the source db backup.step(5) # number of pages (1 kbyte each)

                    finally:
                        backup.finish()
                        subprocess.call(["sync"])
HTH,

Thanks again,

Pierre

On Friday, August 30, 2013 6:22:06 PM UTC+2, Richard Gerd Kuesters wrote:

    Well, what Jeff wrote is also true.

    I do hot-copy of databases because we have a set of products that
    have full automated builds and, to increase performance, I made
    the build generate the SQLite database on memory and then dump it
    to the filesystem.


    Cheers,
    Richard.

    Hi,

    I'm currently running several python applications (each app using
    sqlalchemy) accessing (read/write) a single SQLite database
    stored on disk.

    For performance reasons, I would like to store this db file in
    RAM memory (ie, in my /dev/shm)

    The applications would then access a shared in-memory db through
    SQLAlchemy and a backup application would periodically make a hot
    copy of the in-memory db to disk. Then, on a [power off, power
    on] sequence, this backup app would copy the backed up db file
    from disk to RAM before launching the other apps.

    Before starting, I would like to know if you think it is
    feaseable. My questions are:

       1- Has SQLALchemy an API to do a hot copy? (based on
    http://sqlite.org/backup.htmlfor this particular db type)

       2- If so, is this an actual hot copy, ie: the other apps will
    still run without waiting for the backup app to finish he backup?

       3- Is there a particular configuration in SQLAlchemy that
    enables sharing an in-momory db from different apps (python
    processes)?

    Thanks a lot for your feedback,

    Pierre

--

    Pierre,

    While I do think this is feasible, I would discourage going down
    this path unless you have a really good reason. It sounds like
    you need a real database engine like postgres here. It basically
    does everything you describe out of the box, is easy to set up,
    and will likely be more reliable than anything you could come up
    with on your own. It will also perform better when dealing with
    many simultaneous transactions.

    If you still run into performance problems you could look into
    introducing a caching layer such as memcached, but I wouldn’t
    cross that bridge until I had thoroughly tweaked my db settings
    and identified real bottlenecks in my application(s).

    Jeff Peck

-- You received this message because you are subscribed to the

    Google Groups "sqlalchemy" group.
    To unsubscribe from this group and stop receiving emails from it,
    send an email to sqlalchemy+...@googlegroups.com <javascript:>.
    To post to this group, send email to sqlal...@googlegroups.com
    <javascript:>.
    Visit this group at http://groups.google.com/group/sqlalchemy
    <http://groups.google.com/group/sqlalchemy>.
    For more options, visit https://groups.google.com/groups/opt_out
    <https://groups.google.com/groups/opt_out>.

--

You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com.

To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/groups/opt_out.