Re: pgbackrest - hiding the encryption password

2021-05-19 Thread David Steele
On 5/19/21 2:48 PM, Ron wrote: On 5/19/21 1:34 PM, David Steele wrote: On 5/19/21 1:49 PM, Ron wrote: Currently on our RHEL 7.8 system, /etc/pgbackrest.conf is root:root and 633 perms.  Normally, that's ok, but is a horrible idea when it's a plaintext file, and stores the

Re: pgbackrest - hiding the encryption password

2021-05-19 Thread David Steele
On 5/19/21 1:49 PM, Ron wrote: Currently on our RHEL 7.8 system, /etc/pgbackrest.conf is root:root and 633 perms.  Normally, that's ok, but is a horrible idea when it's a plaintext file, and stores the pgbackrest encryption password. Would pgbackrest (or something else) break if I change it

Re: pgbackrest info of encrypted seems broken

2021-05-19 Thread David Steele
On 5/19/21 1:42 PM, Ron wrote: On 5/19/21 12:08 PM, David Steele wrote: On 5/19/21 9:47 AM, Ron wrote: This is on Postgresql 9.6, if it matters.  Attached is a text file showing the pgbackrest version, config, backup log and "pgbackrest info" output. I followed the instructions

Re: pgbackrest info of encrypted seems broken

2021-05-19 Thread David Steele
On 5/19/21 9:47 AM, Ron wrote: This is on Postgresql 9.6, if it matters.  Attached is a text file showing the pgbackrest version, config, backup log and "pgbackrest info" output. I followed the instructions in https://pgbackrest.org/user-guide.html#quickstart/configure-encryption, and succ

Re: Pgbackrest restore options

2020-05-01 Thread David Steele
On 4/17/20 2:38 AM, Konireddy Rajashekar wrote: Doubt regarding the backrest restore options, is there any way to ensure the backups are good as expected, instead of restoring the whole cluster or database is there any other possibility to ensure it by restoring a single file?  or any other s

Re: It is possible to force periodically switch xlog?

2020-04-01 Thread David Steele
On 4/1/20 7:19 PM, Edson Richter wrote: IIRC archive_timeout does precisely that. I would assume that "archive_mode" must be "on" or "always" - but I do have "archive_mode=off". I'm not sure what the point of switching the WAL segment would be with archive_mode=off. What are you tryin

Re: pgbackrest: ERROR: [029]: unable to convert base 10 string '0000000B' to unsigned int

2020-01-23 Thread David Steele
On 1/21/20 8:45 AM, David Steele wrote: On 1/21/20 12:55 AM, Eric Veldhuyzen wrote: Ron wrote: On 1/21/20 1:10 AM, Eric Veldhuyzen wrote: Hi, We are using pgbackrest (2.21) to backup out postgresql (11) clusters. Last night our nightly diff backup gave me the ERROR: unable to convert base 10

Re: pgbackrest: ERROR: [029]: unable to convert base 10 string '0000000B' to unsigned int

2020-01-21 Thread David Steele
On 1/21/20 12:55 AM, Eric Veldhuyzen wrote: Ron wrote: On 1/21/20 1:10 AM, Eric Veldhuyzen wrote: Hi, We are using pgbackrest (2.21) to backup out postgresql (11) clusters. Last night our nightly diff backup gave me the ERROR: unable to convert base 10 string '000B' to unsigned int. I trie

Re: pgbackrest concerns and doubts.

2019-12-02 Thread David Steele
Hi Ajay, On 12/2/19 1:52 AM, Ajay Pratap wrote: I am using Postgresql 10 and pgbackrest 2.18 version on centos 7.6 I have few doubts regard pgbackrest. a) what is the log rotation policy for pgbackrest logs. I see it logs on default path /var/log/pgbackrest/-.log  what is the log rotation pol

Re: Postgres Point in time Recovery (PITR),

2019-10-18 Thread David Steele
On 10/18/19 11:29 AM, Luca Ferrari wrote: On Fri, Oct 18, 2019 at 10:30 AM Andreas Joseph Krogh wrote: We use barman (https://www.pgbarman.org/) for continuous streaming backup and I had to restore from it once, and it went like this: Just for the records, here's an example of restore with p

Re: pgbackrest with PAF(corosync and pacmaker)

2019-10-17 Thread David Steele
On 10/17/19 9:48 AM, Ajay Pratap wrote: I am using pacemaker and corosync to setup two nodes High availability cluster for postgreSQL 10. In the scenario I want to setup timely backup with pgbackrest. Using the _backup from standby_ feature I could able to take backup from my secondary  postg

Re: pgbackrest - question about restoring cluster to a new cluster on same server

2019-09-18 Thread David Steele
On 9/18/19 10:18 PM, Jerry Sievers wrote: > David Steele writes: > >> This is not an issue unless you seriously game the system. When a > > And/or your recovery system is running archive_mode=always :-) > > I don't know how popular that setting value is

Re: pgbackrest - question about restoring cluster to a new cluster on same server

2019-09-18 Thread David Steele
On 9/18/19 9:40 PM, Ron wrote: > > I'm concerned with one pgbackrest process stepping over another one and > the restore (or the "pg_ctl start" recovery phase) accidentally > corrupting the production database by writing WAL files to the original > cluster. This is not an issue unless you serious

Re: pgbackrest - question about restoring cluster to a new cluster on same server

2019-09-18 Thread David Steele
On 9/18/19 6:59 PM, Ron wrote: > > Scenario: there's data corruption on production server, so we need to do > a PITR restore from "a few days ago" of the cluster holding the prod > databases to a second cluster on that same VM in order to try and find > the missing data and load it back into the p

Re: pgbackrest restore to new location?

2019-09-18 Thread David Steele
On 9/17/19 10:03 PM, Stephen Frost wrote: > > That said- it brings up a pretty serious issue that should be discussed, > and that's nuking this: > > HINT: If you are not restoring from a backup, try removing the file > ".../backup_label". > > That hint is absolutely wrong these days when many

Re: pgbackrest restore to new location?

2019-09-17 Thread David Steele
On 9/17/19 7:23 AM, Luca Ferrari wrote: > On Tue, Sep 17, 2019 at 12:00 PM Ron wrote: >> The real problem is that after doing that, "pg_ctl start -D >> /path/to/new/data" fails with "PANIC: could not locate a valid checkpoint >> record". Sounds like backup_label is missing or has been deleted. T

Re: PGbackrest capacity related question

2019-07-22 Thread David Steele
On 7/22/19 5:08 PM, chiru r wrote: Hi  All, I am using pgbackrest on my production workloads and it is working as expected. However our DB is growing more and more . We have currently two disks and each one is 4 TB size. However 3.8TB size data got loaded into DB currently and our /pgback mou

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 12:58 PM, Prakash Ramakrishnan wrote: > > I did the installation for using postgresql 10.4 version edb standard > edition.so again need to run the yum command or please advise me . It really depends on how you've installed EDB, packages or their customer installer. The custom installer

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 12:58 PM, Adrian Klaver wrote: > On 5/9/19 9:54 AM, David Steele wrote: >> On 5/9/19 12:51 PM, Prakash Ramakrishnan wrote: >>> Hi Adrian, >>> >>> I don't know about that client libraries can you please explain or share >>> me the

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 12:51 PM, Prakash Ramakrishnan wrote: > Hi Adrian, > > I don't know about that client libraries can you please explain or share > me the command I will show you the output and we need the solution for > this pgbackrest issue . yum install postgresql10 should do it, but I thought pqlib

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 11:12 AM, Prakash Ramakrishnan wrote: > Yes, I did.  OK, then I would say something is wrong with your package configuration, but I don't know what it might be. Regards, -- -David da...@pgmasters.net

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 10:57 AM, Prakash Ramakrishnan wrote: > > David am trying to restore the backup in dev server please find the > below details , libpq should be installed with Postgres. Have you installed Postgres yet? -- -David da...@pgmasters.net

Re: perl path issue

2019-05-09 Thread David Steele
On 5/9/19 9:39 AM, Adrian Klaver wrote: > On 5/9/19 4:53 AM, Prakash Ramakrishnan wrote: >> >> We having perl issue for while during restore operation for using >> pgbackrest backup tool can you please do the needful. > > Looks to me like the Postgres client library libpq is not installed or > can

Re: Regarding pgaudit log_directory

2019-03-31 Thread David Steele
On 3/31/19 8:01 PM, Durgamahesh Manne wrote: On Saturday, March 30, 2019, David Steele <mailto:da...@pgmasters.net>> wrote: On 3/29/19 3:32 PM, Durgamahesh Manne wrote:    I could not find parameter related to pgaudit log_directory  . pgAudit does not suppor

Re: Regarding pgaudit log_directory

2019-03-30 Thread David Steele
On 3/29/19 3:32 PM, Durgamahesh Manne wrote:   I could not find parameter related to pgaudit log_directory  . pgAudit does not support logging outside the standard PostgreSQL logging facility and there are no plans for such a feature. The general solution is to use Splunk, ELK, etc. to do

Re: Barman disaster recovery solution

2019-03-01 Thread David Steele
Achilleas, On 2/27/19 11:39 AM, Achilleas Mantzios wrote: On 21/2/19 9:28 π.μ., Achilleas Mantzios wrote: On 21/2/19 9:17 π.μ., Julie Nishimura wrote: Does anyone use this solution? any recommenations? Thanks! Barman will fit most requirements. PgBackRest excels when WAL traffic goes on 10

Re: Pgbackrest Comparability issue

2019-03-01 Thread David Steele
On 3/1/19 1:37 AM, Adrian Klaver wrote: On 2/28/19 12:28 PM, chiru r wrote: The below steps we followed. yum -y install perl-parent yum -y install perl-Time-HiRes yum -y install perl-JSON yum -y install perl-Digest-SHA yum -y install perl-Digest yum -y install perl-DBD-Pg Downloaded pgbackrest

Re: Barman disaster recovery solution

2019-02-28 Thread David Steele
On 2/28/19 9:21 AM, Achilleas Mantzios wrote: On 28/2/19 1:08 π.μ., Ahmed, Nawaz wrote: Hi, I believe the "file copy" method (listed in the table) in pgbackrest is based on pg_basebackup, so i think it should be "pg_basebackup over ssh" as pgbackrest internally calls

Re: Barman disaster recovery solution

2019-02-27 Thread David Steele
On 2/27/19 4:48 PM, Achilleas Mantzios wrote: On 27/2/19 4:16 μ.μ., David Steele wrote: On 2/27/19 2:31 PM, Achilleas Mantzios wrote: On 27/2/19 1:58 μ.μ., rich...@simkorp.com.br wrote: Just to notice, I d o use backup from standby and WAL archive from standby. It is possible. But you have to

Re: Barman disaster recovery solution

2019-02-27 Thread David Steele
On 2/27/19 2:31 PM, Achilleas Mantzios wrote: On 27/2/19 1:58 μ.μ., rich...@simkorp.com.br wrote: Just to notice, I d o use backup from standby and WAL archive from standby. It is possible. But you have to configure standby with option of wal archive "always". I guess there are issues with it

Re: [pgbackrest] Expiring the last backup?

2019-01-24 Thread David Steele
On 1/25/19 8:02 AM, Ron wrote: On 1/24/19 11:22 PM, David Steele wrote: On 1/24/19 5:07 PM, Ron wrote: Are these the steps, or am I missing something? $ pgbackrest stop $ pgbackrest stanza-delete --stanza=mystanza--force That looks right but no need for --force.  That's what the `sto

Re: [pgbackrest] Expiring the last backup?

2019-01-24 Thread David Steele
On 1/24/19 5:07 PM, Ron wrote: Are these the steps, or am I missing something? $ pgbackrest stop $ pgbackrest stanza-delete --stanza=mystanza--force That looks right but no need for --force. That's what the `stop` is for -- to let pgBackRest know you really mean to do this. See documentat

Re: Setting up continuous archiving

2018-10-15 Thread David Steele
On 10/15/18 5:09 PM, Benoit Lobréau wrote: By the way, do/can they both use streaming to receive WAL records? Or streaming is only for standby servers. For backups you have only file-based log shipping? barman supports streaming but it's not as magical as one might think. See pgbarm

Re: Advice on logging strategy

2018-10-12 Thread David Steele
On 10/11/18 11:26 AM, Mike Martin wrote: This results in enormous log files which take ages to import using copy becuase each execute statement is logged with the parameters chosen Is there any way around this? I cant find any way to filter dml statements pgAudit (https://github.com/pgaudi

Re: Setting up continuous archiving

2018-09-26 Thread David Steele
On 9/26/18 8:20 AM, Yuri Kanivetsky wrote: I'm trying to compile a basic set of instruction needed to set up continuous archiving and to recover from a backup. I'm running PostgreSQL 9.3 on Debian Stretch system. This is an incredibly complex topic and it is very difficult to implement correc

Re: Volume partitioning (was Re: pgbackrest when data/base is symlinked to another volume)

2018-09-10 Thread David Steele
Hi Ron On 9/8/18 4:26 PM, Ron wrote: Googled "postgresql disk partitioning" and "postgresql volume partitioning" without much success. Is the best practice volume partitioning: /Database/9.6/data /Database/9.6/data/pg_log /Database/9.6/data/pg_xlog where /var/lib/pgsql/9.6 (on RHEL6) is a s

Re: pgbackrest when data/base is symlinked to another volume

2018-09-08 Thread David Steele
On 9/7/18 8:47 PM, Ron wrote: On 09/07/2018 05:22 PM, David Steele wrote: On 9/6/18 11:21 PM, Ron wrote: Will pgbackrest properly backup and restore the cluster if data/base, data/pg_xlog and data/pg_log are symlinks? PGDATA=/var/lib/pgsql/9.6/data $PGDATA/base -> /Database/9.6/b

Re: pgbackrest when data/base is symlinked to another volume

2018-09-07 Thread David Steele
Hi Ron, On 9/6/18 11:21 PM, Ron wrote: Will pgbackrest properly backup and restore the cluster if data/base, data/pg_xlog and data/pg_log are symlinks? PGDATA=/var/lib/pgsql/9.6/data $PGDATA/base -> /Database/9.6/base $PGDATA/pg_log -> /Database/9.6/pg_log $PGDATA/pg_xlog -> /Database/9.6/pg

Re: Barman versus pgBackRest

2018-09-04 Thread David Steele
On 9/4/18 7:41 AM, Ron wrote: > On 03/09/2018 08:56 AM, David Steele wrote: > >>> - use replication slots for backingup wal on the backup server. >> Another good feature.  We have not added it yet because pgBackRest was >> originally written for very high-volume clust

Re: Barman versus pgBackRest

2018-09-04 Thread David Steele
Hi Ron, On 9/4/18 7:41 AM, Ron wrote: > On 03/09/2018 08:56 AM, David Steele wrote: > [snip] >>> About pgBarman, I like : >>> - be able restore on a remote server from the backup server >> This a good feature, and one that has been requested for pgBackRest. You &

Re: Barman versus pgBackRest

2018-09-04 Thread David Steele
On 9/4/18 11:55 AM, Ron wrote: > On 09/04/2018 10:51 AM, David Steele wrote: > [snip] >> This will work, but I don't think it's what Ron is getting at. >> >> To be clear, it is not possible to restore a database into an *existing* >> cluster using

Re: Barman versus pgBackRest

2018-09-04 Thread David Steele
On 9/4/18 11:24 AM, Joshua D. Drake wrote: > On 09/04/2018 07:52 AM, Ron wrote: >> On 09/04/2018 09:24 AM, Joshua D. Drake wrote: >>> On 09/04/2018 07:14 AM, Ron wrote: That was about barman, in the barman group.  This is asking about pgbackrest...  :) So: does pgbackrest h

Re: During promotion, new master tries to archive same segment twice

2018-08-16 Thread David Steele
On 8/16/18 4:37 AM, Phil Endecott wrote: > David Steele wrote: >> On 8/15/18 4:25 PM, Phil Endecott wrote: >>> - Should my archive_command detect the case where it is asked to >>> write the same file again with the same contents, and report success >>> in tha

Re: During promotion, new master tries to archive same segment twice

2018-08-15 Thread David Steele
Hi Phil, On 8/15/18 4:25 PM, Phil Endecott wrote: > > Questions: > > - Can all of this be blamed on the failure of the first attempt > to promote due to the wrong permissions on recovery.conf? Yes, it looks that way. > - Should my archive_command detect the case where it is asked to > write

Re: Barman versus pgBackRest

2018-04-13 Thread David Steele
Hi Thomas, On 4/11/18 3:14 AM, Thomas Poty wrote: Sorry for answering only now but I just saw you answer only now. > To be clear, I'm the original author and primary maintainer of pgBackRest. I am very happy to see guys like you to take time to answer me. Thank you You are welcome. Users

Re: Barman versus pgBackRest

2018-03-09 Thread David Steele
Hi Thomas, On 3/6/18 2:53 PM, Thomas Poty wrote: > Hello Community, > > I hesitate to use barman or pgBackRest.  I have found a lot of common > points between them and a few differences: To be clear, I'm the original author and primary maintainer of pgBackRest. I'll let the Barman folks speak t

Re: Is there a continuous backup for pg ?

2018-03-02 Thread David Steele
Hi Gary, On 3/2/18 2:05 PM, Gary M wrote: > Hi, > > I have an unusual requirement for schema based, live backup of a 24/7 > database processing 100K inserts/updates per hour. The data store is > around 100TB. > > The requirement is supporting an incremental backup of 10 minute > windows.  Replic

Re: initdb when data/ folder has mount points

2018-02-22 Thread David Steele
On 2/22/18 1:16 AM, Michael Paquier wrote: > On Wed, Feb 21, 2018 at 07:56:38PM -0500, David Steele wrote: >> On 2/21/18 7:01 PM, Tom Lane wrote: >>> For pg_log, just put it somewhere else and set the appropriate >>> configuration option to say where to write the postm

Re: initdb when data/ folder has mount points

2018-02-21 Thread David Steele
On 2/21/18 7:01 PM, Tom Lane wrote: Ron Johnson writes: Apparently, initdb assumes that data/ is one big mount point. However, we have four mount points: /var/lib/pgsql/9.6/data/backup /var/lib/pgsql/9.6/data/base /var/lib/pgsql/9.6/data/pg_log /var/lib/pgsql/9.6/data/pg_xlog Don't do that.

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
On 2/19/18 3:41 PM, Don Seiler wrote: > On Mon, Feb 19, 2018 at 10:17 AM, David Steele <mailto:da...@pgmasters.net>> wrote: > > If pg_log is symlinked to PGDATA it will be copied.  pg_xlog is not > copied in any backup. > > > So an external pg_log directo

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
On 2/19/18 2:05 PM, Don Seiler wrote: > On Mon, Feb 19, 2018 at 12:39 PM, David Steele <mailto:da...@pgmasters.net>> wrote: > > > I read "open it for testing (or backups in this case)" as letting > recovery complete and promoting the cluster to a mas

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
On 2/19/18 11:29 AM, Simon Riggs wrote: > On 19 February 2018 at 16:17, David Steele wrote: >>> > I did come up with a sort of Rube Goldberg-esque workaround for now >>> > involving using a clone of the prod standby VM from Veeam backup to >>> use &

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
On 2/19/18 10:32 AM, Don Seiler wrote: > On Mon, Feb 19, 2018 at 9:21 AM, David Steele <mailto:da...@pgmasters.net>> wrote: > > > Yes, they are typically very small.  The general exception to this rule > is if logs are stored in pg_log.  I recommend storing logs

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
Hi Don, On 2/19/18 10:01 AM, Don Seiler wrote: > On Mon, Feb 19, 2018 at 8:53 AM, David Steele <mailto:da...@pgmasters.net>> wrote: > > Anything *not* in global (except pg_control), base, pg_tblspc, > pg_xact/pg_clog, and pg_multixact are copied from the primary.

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
Hi Don, On 2/19/18 9:25 AM, Don Seiler wrote: > On Mon, Feb 19, 2018 at 8:18 AM, David Steele <mailto:da...@pgmasters.net>> wrote: > > It copies files that are not replicated from the primary so that a > primary-style backup is created. Anything that is replicated (

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
On 2/18/18 10:20 PM, Michael Paquier wrote: > On Sun, Feb 18, 2018 at 06:48:30PM -0600, Don Seiler wrote: >> On Sun, Feb 18, 2018 at 6:43 PM, Michael Paquier wrote: >>> You may want to contact the maintainers directly through github where >>> the project is maintained: >>> https://github.com/pgbac

Re: pgBackRest backup from standby

2018-02-19 Thread David Steele
Hi Don, On 2/18/18 7:34 PM, Don Seiler wrote: > > Looking to use pgBackRest to take a backup from a hot standby. I'm > reading that pgBackRest still needs to connect to the primary and copy > some files. My questions are: > > 1. What files does it need to copy? Config files? WAL files? It copi

Re: PgBackRest question?

2017-12-29 Thread David Steele
On 12/29/17 1:53 PM, Stephen Frost wrote: > Greetings, > > * David Steele (da...@pgmasters.net) wrote: >> On 12/28/17 6:15 PM, chiru r wrote: >>> I am unable to copy the complete backup.manifest file due to security >>> reasons . please find the below contents. &g

Re: PgBackRest question?

2017-12-29 Thread David Steele
On 12/28/17 6:15 PM, chiru r wrote: > > I am unable to copy the complete backup.manifest file due to security > reasons . please find the below contents. > > [backup:target] > pg_data={"path":"/u02/pgdata01/9.5/data","type":"path"} > pg_tblspc/721349={"path":"/u02/pgdata02/report1","tablespace-id

Re: PgBackRest question?

2017-12-28 Thread David Steele
On 12/28/17 5:15 PM, chiru r wrote: Please find the below details. postgres=# select datname, oid from pg_database;   datname  |  oid ---+  template0 |  13289  postgres  |  13294  template1 |      1  db1       | 770161  db2       | 770162  db3       | 770169 (6 rows) That

Re: PgBackRest question?

2017-12-28 Thread David Steele
On 12/28/17 3:38 PM, chiru r wrote: Thank you for the details  David and Stephen.. I am unable to recover the database which associated with user table space . Please see the below test case and suggest me,if any one has the  issues while recovering single database. *Test case: * 1) created

Re: Does PostgreSQL check database integrity at startup?

2017-12-27 Thread David Steele
On 12/26/17 4:58 PM, Alvaro Herrera wrote: pgBackRest will validate all page checksums (including indexes, etc.) in the cluster during backup. Full backups check everything, incr/differential backups check only the files that have changed. If a table or index file is of zero length when back

Re: Does PostgreSQL check database integrity at startup?

2017-12-26 Thread David Steele
On 12/26/17 10:57 AM, Pavel Stehule wrote: 2017-12-26 16:50 GMT+01:00 Edson Carlos Ericksson Richter I'm rebuilding the standby server for two days already, with 23% of completion status... If lost the database and backups because of that failure, it would be a giant disaster.

Re: PgBackRest question?

2017-12-18 Thread David Steele
On 12/17/17 7:10 PM, chiru r wrote: > Thanks David for the replay. > > we are not interested to use replication/Standby configuration at this > moment with pgbackrest. Database restores all work the same way -- the only difference is a few configuration parameters. > We are looking to restore th

Re: PgBackRest question?

2017-12-16 Thread David Steele
On 12/15/17 4:36 PM, chiru r wrote: > > Thanks,I am thinking about a specific recovery case. > > Lets assume Heavy transactional system we configured. > It is generating WAL 2000/hr and recycling automatically in pg_wal > directory.  > > QA : > > Sunday -- 11 PM  -- Full backup done. > Monday -