Re: documenting the backup manifest file format

2020-05-15 Thread David Steele
On 5/15/20 10:17 AM, Tom Lane wrote: David Steele writes: On 5/15/20 9:34 AM, Tom Lane wrote: I vote for following the backup_label precedent; that's stood for quite some years now. Of course, my actual preference is to use epoch time which is easy to work with and eliminates the possibilit

Re: documenting the backup manifest file format

2020-05-15 Thread Tom Lane
David Steele writes: > On 5/15/20 9:34 AM, Tom Lane wrote: >> I vote for following the backup_label precedent; that's stood for quite >> some years now. > Of course, my actual preference is to use epoch time which is easy to > work with and eliminates the possibility of conversion errors. It is

Re: documenting the backup manifest file format

2020-05-15 Thread David Steele
On 5/15/20 9:34 AM, Tom Lane wrote: Robert Haas writes: It's a good question. My inclination was to think that GMT would be the clearest thing, but I also didn't realize that the result would thus be inconsistent with backup_label. Not sure what's best here. I vote for following the backup_la

Re: documenting the backup manifest file format

2020-05-15 Thread Tom Lane
Robert Haas writes: > It's a good question. My inclination was to think that GMT would be > the clearest thing, but I also didn't realize that the result would > thus be inconsistent with backup_label. Not sure what's best here. I vote for following the backup_label precedent; that's stood for qu

Re: documenting the backup manifest file format

2020-05-15 Thread Robert Haas
On Fri, May 15, 2020 at 2:10 AM Fujii Masao wrote: > I have one question related to this; Why don't we use log_timezone, > like backup_label? log_timezone is used for "START TIME" field in > backup_label. Sorry if this was already discussed. > > /* Use the log timezone here, not th

Re: documenting the backup manifest file format

2020-05-14 Thread Fujii Masao
On 2020/04/15 5:33, Andrew Dunstan wrote: On 4/14/20 4:09 PM, Alvaro Herrera wrote: On 2020-Apr-14, Andrew Dunstan wrote: OK, but I think if we're putting a timestamp string in ISO-8601 format in the manifest it should be in UTC / Zulu time, precisely to avoid these issues. If that's too m

Re: documenting the backup manifest file format

2020-04-17 Thread Fujii Masao
On 2020/04/15 22:24, Robert Haas wrote: On Tue, Apr 14, 2020 at 11:49 PM Fujii Masao wrote: While reading the document that you pushed, I thought that it's better to define index term for backup manifest, so that we can easily reach this document from the index page. Thought? Patch attached.

Re: documenting the backup manifest file format

2020-04-16 Thread Jehan-Guillaume de Rorthais
On Wed, 15 Apr 2020 18:54:14 -0400 David Steele wrote: > On 4/15/20 6:43 PM, Jehan-Guillaume de Rorthais wrote: > > On Wed, 15 Apr 2020 12:03:28 -0400 > > Robert Haas wrote: > > > >> On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais > >> wrote: > >>> But for backup_manifest, it'

Re: documenting the backup manifest file format

2020-04-15 Thread David Steele
On 4/15/20 6:43 PM, Jehan-Guillaume de Rorthais wrote: On Wed, 15 Apr 2020 12:03:28 -0400 Robert Haas wrote: On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais wrote: But for backup_manifest, it's kind of shame we have to check the checksum against an transformed version of the fil

Re: documenting the backup manifest file format

2020-04-15 Thread Jehan-Guillaume de Rorthais
On Wed, 15 Apr 2020 12:03:28 -0400 Robert Haas wrote: > On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais > wrote: > > But for backup_manifest, it's kind of shame we have to check the checksum > > against an transformed version of the file. Did you consider creating eg. a > > separate

Re: documenting the backup manifest file format

2020-04-15 Thread Robert Haas
On Wed, Apr 15, 2020 at 11:23 AM Jehan-Guillaume de Rorthais wrote: > But for backup_manifest, it's kind of shame we have to check the checksum > against an transformed version of the file. Did you consider creating eg. a > separate backup_manifest.sha256 file? > > I'm very sorry in advance if thi

Re: documenting the backup manifest file format

2020-04-15 Thread Jehan-Guillaume de Rorthais
On Tue, 14 Apr 2020 12:56:49 -0400 Robert Haas wrote: > On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera > wrote: > > Yeah, I guess I'm just saying that it feels brittle to have a file > > format that's supposed to be good for data exchange and then make it > > itself depend on representation deta

Re: documenting the backup manifest file format

2020-04-15 Thread Robert Haas
On Tue, Apr 14, 2020 at 11:49 PM Fujii Masao wrote: > While reading the document that you pushed, I thought that it's better > to define index term for backup manifest, so that we can easily reach > this document from the index page. Thought? Patch attached. Fine with me. I tend not to think abou

Re: documenting the backup manifest file format

2020-04-14 Thread Fujii Masao
On 2020/04/14 2:40, Robert Haas wrote: On Fri, Mar 27, 2020 at 4:32 PM Andres Freund wrote: I don't like having a file format that's intended to be used by external tools too that's undocumented except for code that assembles it in a piecemeal fashion. Do you mean in a follow-on patch this r

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 4:11 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: Happily ISO-8601 is always UTC. Uh, it is not -- https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators Whoops, you are correct. I've just never seen non-UTC in the wild yet. -- -David da...@pgmasters.net

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 4:09 PM, Alvaro Herrera wrote: > On 2020-Apr-14, Andrew Dunstan wrote: > >> OK, but I think if we're putting a timestamp string in ISO-8601 format >> in the manifest it should be in UTC / Zulu time, precisely to avoid >> these issues. If that's too much trouble then yes an epoch time w

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, David Steele wrote: > Happily ISO-8601 is always UTC. Uh, it is not -- https://en.wikipedia.org/wiki/ISO_8601#Time_zone_designators -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, Andrew Dunstan wrote: > OK, but I think if we're putting a timestamp string in ISO-8601 format > in the manifest it should be in UTC / Zulu time, precisely to avoid > these issues. If that's too much trouble then yes an epoch time will > probably do. The timestamp is always specif

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 3:55 PM, Andrew Dunstan wrote: OK, but I think if we're putting a timestamp string in ISO-8601 format in the manifest it should be in UTC / Zulu time, precisely to avoid these issues. If that's too much trouble then yes an epoch time will probably do. Happily ISO-8601 is always UTC.

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 3:19 PM, David Steele wrote: > On 4/14/20 3:03 PM, Andrew Dunstan wrote: >> >> On 4/14/20 1:33 PM, David Steele wrote: >>> On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: > On 4/14/20 12:56 PM, Robert Haas wrote: > >> Hmm, did David s

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 3:03 PM, Andrew Dunstan wrote: On 4/14/20 1:33 PM, David Steele wrote: On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: On 4/14/20 12:56 PM, Robert Haas wrote: Hmm, did David suggest that before? I don't recall for sure. I think he had some suggestio

Re: documenting the backup manifest file format

2020-04-14 Thread Andrew Dunstan
On 4/14/20 1:33 PM, David Steele wrote: > On 4/14/20 1:27 PM, Alvaro Herrera wrote: >> On 2020-Apr-14, David Steele wrote: >> >>> On 4/14/20 12:56 PM, Robert Haas wrote: >>> Hmm, did David suggest that before? I don't recall for sure. I think he had some suggestion, but I'm not sure if

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 1:27 PM, Alvaro Herrera wrote: On 2020-Apr-14, David Steele wrote: On 4/14/20 12:56 PM, Robert Haas wrote: Hmm, did David suggest that before? I don't recall for sure. I think he had some suggestion, but I'm not sure if it was the same one. "I'm also partial to using epoch time i

Re: documenting the backup manifest file format

2020-04-14 Thread Alvaro Herrera
On 2020-Apr-14, David Steele wrote: > On 4/14/20 12:56 PM, Robert Haas wrote: > > > Hmm, did David suggest that before? I don't recall for sure. I think > > he had some suggestion, but I'm not sure if it was the same one. > > "I'm also partial to using epoch time in the manifest because it is > g

Re: documenting the backup manifest file format

2020-04-14 Thread David Steele
On 4/14/20 12:56 PM, Robert Haas wrote: On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera wrote: Yeah, I guess I'm just saying that it feels brittle to have a file format that's supposed to be good for data exchange and then make it itself depend on representation details such as the order that fi

Re: documenting the backup manifest file format

2020-04-14 Thread Robert Haas
On Mon, Apr 13, 2020 at 5:43 PM Alvaro Herrera wrote: > Yeah, I guess I'm just saying that it feels brittle to have a file > format that's supposed to be good for data exchange and then make it > itself depend on representation details such as the order that fields > appear in, the letter case, or

Re: documenting the backup manifest file format

2020-04-13 Thread Alvaro Herrera
On 2020-Apr-13, Robert Haas wrote: > On Mon, Apr 13, 2020 at 3:34 PM Alvaro Herrera > wrote: > > Are these hex figures upper or lower case? No leading zeroes? This > > would normally not matter, but the toplevel checksum will care. > > Not really. You just feed the whole file except for the l

Re: documenting the backup manifest file format

2020-04-13 Thread David Steele
On 4/13/20 4:14 PM, Robert Haas wrote: On Mon, Apr 13, 2020 at 3:34 PM Alvaro Herrera wrote: Also, I see no mention of prettification-chars such as newlines or indentation. I suppose if I pass a manifest file through prettification (or Windows newline conversion), the checksum may break. It

Re: documenting the backup manifest file format

2020-04-13 Thread Robert Haas
On Mon, Apr 13, 2020 at 4:10 PM Andrew Dunstan wrote: > Seems ok. A tiny example, or an excerpt, might be nice. An empty database produces a manifest about 1200 lines long, so a full example seems like too much to include in the documentation. An excerpt could be included, I suppose. -- Robert

Re: documenting the backup manifest file format

2020-04-13 Thread Robert Haas
On Mon, Apr 13, 2020 at 3:34 PM Alvaro Herrera wrote: > Are these hex figures upper or lower case? No leading zeroes? This > would normally not matter, but the toplevel checksum will care. Not really. You just feed the whole file except for the last line through shasum and you get the answer.

Re: documenting the backup manifest file format

2020-04-13 Thread Andrew Dunstan
On 4/13/20 1:40 PM, Robert Haas wrote: > On Fri, Mar 27, 2020 at 4:32 PM Andres Freund wrote: >> I don't like having a file format that's intended to be used by external >> tools too that's undocumented except for code that assembles it in a >> piecemeal fashion. Do you mean in a follow-on patc

Re: documenting the backup manifest file format

2020-04-13 Thread Robert Haas
On Mon, Apr 13, 2020 at 2:28 PM Erik Rijkers wrote: > Can you double check this sentence? Seems strange to me but I don't > know why; it may well be that my english is not good enough. Maybe a > comma after 'required' makes reading easier? > > The timeline from which this range of WAL record

Re: documenting the backup manifest file format

2020-04-13 Thread Alvaro Herrera
+ The LSN at which replay must begin on the indicated timeline in order to + make use of this backup. The LSN is stored in the format normally used + by PostgreSQL; that is, it is a string + consisting of two strings of hexademical characters, each with a length + of betwe

Re: documenting the backup manifest file format

2020-04-13 Thread Erik Rijkers
On 2020-04-13 20:08, Robert Haas wrote: [v2-0001-Document-the-backup-manifest-file-format.patch] Can you double check this sentence? Seems strange to me but I don't know why; it may well be that my english is not good enough. Maybe a comma after 'required' makes reading easier? The tim

Re: documenting the backup manifest file format

2020-04-13 Thread Robert Haas
On Mon, Apr 13, 2020 at 1:55 PM Justin Pryzby wrote: > typos: > manifes > hexademical (twice) Thanks. v2 attached. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company v2-0001-Document-the-backup-manifest-file-format.patch Description: Binary data

Re: documenting the backup manifest file format

2020-04-13 Thread Justin Pryzby
On Mon, Apr 13, 2020 at 01:40:56PM -0400, Robert Haas wrote: > On Fri, Mar 27, 2020 at 4:32 PM Andres Freund wrote: > > I don't like having a file format that's intended to be used by external > > tools too that's undocumented except for code that assembles it in a > > piecemeal fashion. Do you m