Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Arjun Ranade
I'm working on a project that requires on-demand creation of a fresh database as quick as possible (seconds). Essentially, this is a build server that will require a cloned instance of Postgres to run unit tests on. So the pattern of use would be: - Build is triggered - New postgres insta

Re: Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Kevin Wilkinson
if you are able/willing to use ZFS (rather than ext4, xfs, ...) to store your database, then it might work for you. ZFS is copy-on-write so it can very quickly clone a database. kevin On 3/1/2019 12:08 PM, Arjun Ranade wrote: I'm working on a project that requires on-demand creation of a fresh

Re: Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Kenneth Marshall
On Fri, Mar 01, 2019 at 11:57:30AM -0800, Kevin Wilkinson wrote: > if you are able/willing to use ZFS (rather than ext4, xfs, ...) to > store your database, then it might work for you. ZFS is > copy-on-write so it can very quickly clone a database. > > kevin Hi Arjun Redhat 7 does have LVM snaps

Re: Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Jerry Sievers
Kenneth Marshall writes: > On Fri, Mar 01, 2019 at 11:57:30AM -0800, Kevin Wilkinson wrote: > >> if you are able/willing to use ZFS (rather than ext4, xfs, ...) to >> store your database, then it might work for you. ZFS is >> copy-on-write so it can very quickly clone a database. >> >> kevin > >

Re: Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Bruce Klein
Apologies for the low tech suggestion, but if this really is a clone of a previously existing template, could the clone operation just be done ahead of time? I.e., have the build server keep X copies ready for use and generate additional copies as those are consumed, so that the cloning is no longe

Re: Methods to quickly spin up copies of an existing databases

2019-03-01 Thread Arjun Ranade
Pre-copying is not really an option since we could potentially need 1-X instances so it needs to be scalable. XFS also allows for "cp --reflink" which I could do on a PGDATA directory and then change the port number. That's probably the method I'll try first. We do use barman, but again a barman

Re: Methods to quickly spin up copies of an existing databases

2019-03-02 Thread Karsten Hilbert
On Fri, Mar 01, 2019 at 04:51:32PM -0500, Arjun Ranade wrote: > Pre-copying is not really an option since we could potentially need 1-X > instances so it needs to be scalable. Would it work to have a PG instance with a ready-to-use cluster/DB inside a VM, and make a copy of that as needed ? The