I wrote:
> While I've been poking at the pg_dump issues, it's occurred to me that
> changing the default would be a great forcing function for finding out
> any lurking problems. What I'm inclined to do now is to commit it
> *with* the change of default, and let it be that way at least for a
> few
Bernd Helmle writes:
> --On Samstag, Juli 11, 2009 13:40:44 +0300 Peter Eisentraut
> wrote:
>> OK, here is an updated patch. It has the setting as enum, completed
>> documentation, and libpq support. I'll add it to the commit fest in the
>> hope that someone else can look it over in detail.
--On Dienstag, August 04, 2009 10:28:48 -0400 Tom Lane
wrote:
While I've been poking at the pg_dump issues, it's occurred to me that
changing the default would be a great forcing function for finding out
any lurking problems.
+1
--
Thanks
Bernd
--
Sent via pgsql-hacker
Robert Haas writes:
> On Tue, Aug 4, 2009 at 10:28 AM, Tom Lane wrote:
>> If this seems reasonable, I can make a note of the point in the commit
>> message, so that we won't forget when the time comes.
> Or, what we could do is start an open items for 8.5 list similar to
> the one we made for 8.4
On Tue, Aug 4, 2009 at 16:31, Robert Haas wrote:
> On Tue, Aug 4, 2009 at 10:28 AM, Tom Lane wrote:
>> Peter Eisentraut writes:
>>> On Monday 03 August 2009 22:11:08 Tom Lane wrote:
I'm starting to look at this patch. I observe that it's setting the
default output format to HEX. If cha
On Tue, Aug 4, 2009 at 10:28 AM, Tom Lane wrote:
> Peter Eisentraut writes:
>> On Monday 03 August 2009 22:11:08 Tom Lane wrote:
>>> I'm starting to look at this patch. I observe that it's setting the
>>> default output format to HEX. If changing the default behavior was
>>> agreed to, or even d
Peter Eisentraut writes:
> On Monday 03 August 2009 22:11:08 Tom Lane wrote:
>> I'm starting to look at this patch. I observe that it's setting the
>> default output format to HEX. If changing the default behavior was
>> agreed to, or even discussed, I do not remember where. Shouldn't the
>> de
On Monday 03 August 2009 22:11:08 Tom Lane wrote:
> I'm starting to look at this patch. I observe that it's setting the
> default output format to HEX. If changing the default behavior was
> agreed to, or even discussed, I do not remember where. Shouldn't the
> default stay the same?
I did pose
Alvaro Herrera writes:
> Tom Lane wrote:
>> Well, unless you want to leave *all* the bytea functions in builtins.h
>> there will still be some risk there. I'd actually sooner break calls
>> of byteaout than other things, because in reality every caller of
>> byteaout is going to need to be inspec
Tom Lane wrote:
> Alvaro Herrera writes:
> > I vote for a new bytea.h file that does not slurp in byteain/byteaout,
> > to avoid breaking 3rd party code. miscadmin.h seems the worst solution,
> > since it's already included in 210 other files.
>
> Well, unless you want to leave *all* the bytea f
Alvaro Herrera writes:
> I vote for a new bytea.h file that does not slurp in byteain/byteaout,
> to avoid breaking 3rd party code. miscadmin.h seems the worst solution,
> since it's already included in 210 other files.
Well, unless you want to leave *all* the bytea functions in builtins.h
there
Tom Lane wrote:
> Greg Stark writes:
> > On Tue, Aug 4, 2009 at 12:18 AM, Tom Lane wrote:
> >> One other stylistic gripe: I don't much like inserting a GUC variable
> >> definition into builtins.h --- that file has traditionally only
> >> contained function extern declarations. �The best alternati
Greg Stark writes:
> On Tue, Aug 4, 2009 at 12:18 AM, Tom Lane wrote:
>> One other stylistic gripe: I don't much like inserting a GUC variable
>> definition into builtins.h --- that file has traditionally only
>> contained function extern declarations. The best alternative I can
>> think of is to
On Tue, Aug 4, 2009 at 12:18 AM, Tom Lane wrote:
> One other stylistic gripe: I don't much like inserting a GUC variable
> definition into builtins.h --- that file has traditionally only
> contained function extern declarations. The best alternative I can
> think of is to move the bytea-related st
One other stylistic gripe: I don't much like inserting a GUC variable
definition into builtins.h --- that file has traditionally only
contained function extern declarations. The best alternative I can
think of is to move the bytea-related stuff into a new include file
include/utils/bytea.h. Has a
Bernd Helmle writes:
> --On Montag, August 03, 2009 15:11:08 -0400 Tom Lane
> wrote:
>> I'm starting to look at this patch. I observe that it's setting the
>> default output format to HEX. If changing the default behavior was
>> agreed to, or even discussed, I do not remember where. Shouldn't
--On Montag, August 03, 2009 15:11:08 -0400 Tom Lane
wrote:
I'm starting to look at this patch. I observe that it's setting the
default output format to HEX. If changing the default behavior was
agreed to, or even discussed, I do not remember where. Shouldn't the
default stay the same?
I
Bernd Helmle writes:
> --On Samstag, Juli 11, 2009 13:40:44 +0300 Peter Eisentraut
> wrote:
>> OK, here is an updated patch. It has the setting as enum, completed
>> documentation, and libpq support. I'll add it to the commit fest in the
>> hope that someone else can look it over in detail.
--On Freitag, Juli 24, 2009 20:50:16 +0200 Bernd Helmle
wrote:
I don't believe i can do very much this weekend...
I have to delay that until sunday, but will get my hands on some
performance and function tests again, since i have access on the customer
machine then.
--
Thanks
--On Freitag, Juli 24, 2009 11:38:06 -0400 Tom Lane
wrote:
Bernd, are you done reviewing this or did you intend to do more?
It's still marked as "needs review" on the commitfest page.
I hoped to get more profiling data like Andrew suggested, but haven't
enough time to do it :( The customer
Bernd Helmle writes:
> I've attached a slightly edited patch which fixes a compiler warning in
> encode.c, too.
Bernd, are you done reviewing this or did you intend to do more?
It's still marked as "needs review" on the commitfest page.
regards, tom lane
--
Sent via pg
Peter Eisentraut writes:
> OK, here is an updated patch. It has the setting as enum, completed
> documentation, and libpq support. I'll add it to the commit fest in the hope
> that someone else can look it over in detail.
I found that there is another issue that should be addressed, maybe not
Bernd Helmle wrote:
--On Dienstag, Juli 21, 2009 16:49:45 -0400 Andrew Dunstan
wrote:
You just tested COPY, not pg_dump, right? Some pg_dump numbers would be
interesting, both for text and custom formats.
Plain COPY, yes. I planned testing pg_dump for this round of my review
but ran out
--On Dienstag, Juli 21, 2009 16:49:45 -0400 Andrew Dunstan
wrote:
You just tested COPY, not pg_dump, right? Some pg_dump numbers would be
interesting, both for text and custom formats.
Plain COPY, yes. I planned testing pg_dump for this round of my review but
ran out of time unfortunately.
Bernd Helmle wrote:
--On Samstag, Juli 11, 2009 13:40:44 +0300 Peter Eisentraut
wrote:
OK, here is an updated patch. It has the setting as enum, completed
documentation, and libpq support. I'll add it to the commit fest in the
hope that someone else can look it over in detail.
I've sta
--On Samstag, Juli 11, 2009 13:40:44 +0300 Peter Eisentraut
wrote:
OK, here is an updated patch. It has the setting as enum, completed
documentation, and libpq support. I'll add it to the commit fest in the
hope that someone else can look it over in detail.
I've started looking at this an
On Wednesday 08 July 2009 01:07:08 Tom Lane wrote:
> Peter Eisentraut writes:
> > Here is a first cut at a new hex bytea input and output format. Example:
> > ...
> > SET bytea_output_hex = true;
> >
> > Should the configuration parameter be a boolean or an enum, opening
> > possibilities for oth
Hello, Bernd.
You wrote:
BH> --On Dienstag, Juli 07, 2009 18:07:08 -0400 Tom Lane
BH> wrote:
>> Enum. If we do this then it seems entirely fair that someone might
>> want other settings someday. Also, it seems silly to pick a format
>> partly on the grounds that it's expansible, and then not
2009/7/8 Bernd Helmle :
> --On Dienstag, Juli 07, 2009 18:07:08 -0400 Tom Lane
> wrote:
>
>> Enum. If we do this then it seems entirely fair that someone might
>> want other settings someday. Also, it seems silly to pick a format
>> partly on the grounds that it's expansible, and then not make t
--On Dienstag, Juli 07, 2009 18:07:08 -0400 Tom Lane
wrote:
Enum. If we do this then it seems entirely fair that someone might
want other settings someday. Also, it seems silly to pick a format
partly on the grounds that it's expansible, and then not make the
control GUC expansible. Perhaps
Peter Eisentraut writes:
> Here is a first cut at a new hex bytea input and output format. Example:
> ...
> SET bytea_output_hex = true;
> Should the configuration parameter be a boolean or an enum, opening
> possibilities for other formats?
Enum. If we do this then it seems entirely fair tha
On Wednesday 06 May 2009 18:47:57 Tom Lane wrote:
> So the ambiguous-input problem is solved if we define the new format(s)
> to be started by backslash and something that the old code would reject.
> I'd keep it short, like "\x", but there's still room for multiple
> formats if anyone really wants
--On Freitag, Mai 29, 2009 11:06:28 +0300 Peter Eisentraut
wrote:
Btw., I have started to write some code for that.
Cool. Let me know if i can help out somewhere.
--
Thanks
Bernd
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to
--On Samstag, Mai 30, 2009 00:47:16 +0300 Hannu Krosing
wrote:
And we can also escape the need to uncompress TOAST'ed fields - just
markup the compression as another \c at the beginning of data.
Hmm i thought about that, but that seems only to make sense if there is an
easy way to "bypass"
On Fri, 2009-05-29 at 11:06 +0300, Peter Eisentraut wrote:
> On Friday 29 May 2009 04:26:35 Bruce Momjian wrote:
> > Added to TODO:
> > |Improve bytea COPY format
> >
> > * http://archives.postgresql.org/pgsql-hackers/2009-05/msg00192.php
>
> Btw., I have started to write some code for tha
On Wed, 2009-05-06 at 18:33 +0300, Peter Eisentraut wrote:
> On Tuesday 05 May 2009 17:38:33 Tom Lane wrote:
> > "Kevin Grittner" writes:
> > > Bernd Helmle wrote:
> > >> Another approach would be to just dump bytea columns in binary
> > >> format only (not sure how doable that is, though).
> > >
On Friday 29 May 2009 04:26:35 Bruce Momjian wrote:
> Added to TODO:
> |Improve bytea COPY format
>
> * http://archives.postgresql.org/pgsql-hackers/2009-05/msg00192.php
Btw., I have started to write some code for that.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresq
Added to TODO:
|Improve bytea COPY format
* http://archives.postgresql.org/pgsql-hackers/2009-05/msg00192.php
---
Merlin Moncure wrote:
> On Sat, May 16, 2009 at 11:23 AM, Stefan Kaltenbrunner
> wrote:
> >
On Sat, May 16, 2009 at 11:23 AM, Stefan Kaltenbrunner
wrote:
> Bernd Helmle wrote:
>>
>> --On Mittwoch, Mai 06, 2009 19:04:21 -0400 Tom Lane
>> wrote:
>>
>>> So I'm now persuaded that a better textual representation for bytea
>>> should indeed make things noticeably better here. It would be
>>>
Bernd Helmle wrote:
--On Mittwoch, Mai 06, 2009 19:04:21 -0400 Tom Lane
wrote:
So I'm now persuaded that a better textual representation for bytea
should indeed make things noticeably better here. It would be
useful though to cross-check this thought by profiling a case that
dumps a comparab
--On Mittwoch, Mai 06, 2009 19:04:21 -0400 Tom Lane
wrote:
So I'm now persuaded that a better textual representation for bytea
should indeed make things noticeably better here. It would be
useful though to cross-check this thought by profiling a case that
dumps a comparable volume of text dat
--On Mittwoch, Mai 06, 2009 19:04:21 -0400 Tom Lane
wrote:
So I'm now persuaded that a better textual representation for bytea
should indeed make things noticeably better here. It would be
useful though to cross-check this thought by profiling a case that
dumps a comparable volume of text dat
Bernd Helmle writes:
> --On Dienstag, Mai 05, 2009 10:00:37 -0400 Tom Lane
> wrote:
>> Seems like the right response might be some micro-optimization effort on
>> byteaout.
> Hmm looking into profiler statistics seems to second your suspicion:
> Normal COPY shows:
> % cumulative self
Peter Eisentraut writes:
> For distinguishing various input formats, we could use the backslash
> to escape the format specification without breaking backward
> compatibilty, e.g.,
Oh, you're right! I had been thinking that byteain treats \x as
just meaning x if x isn't an octal digit, but actua
On Tuesday 05 May 2009 17:38:33 Tom Lane wrote:
> "Kevin Grittner" writes:
> > Bernd Helmle wrote:
> >> Another approach would be to just dump bytea columns in binary
> >> format only (not sure how doable that is, though).
> >
> > If that's not doable, perhaps a base64 option for bytea COPY?
>
>
On Wed, May 6, 2009 at 8:02 AM, Andrew Dunstan wrote:
>
>
> Merlin Moncure wrote:
>>
>> On Tue, May 5, 2009 at 4:14 PM, Tom Lane wrote:
>>
>>>
>>> Heikki Linnakangas writes:
>>>
Tom Lane wrote:
>
> I'm thinking plain old pairs-of-hex-digits might be the best
> tradeoff
Andrew Dunstan wrote:
another nit with base64 is that properly encoded data requires
newlines according to the standard.
er, no, not as I read rfc 3548 s 2.1.
cheers
andrew
Why does encode('my text', 'base64') include newlines in its output? I
think MIME requires text to be broken i
Andrew Dunstan writes:
> Bernd Helmle wrote:
>> I'm dumb: I don't understand why a hex conversion would be
>> significantly faster than what we have now?
> Quite apart from anything else you would not need the current loop over
> the bytea input to calculate the result length - in hex it would
Merlin Moncure wrote:
On Tue, May 5, 2009 at 4:14 PM, Tom Lane wrote:
Heikki Linnakangas writes:
Tom Lane wrote:
I'm thinking plain old pairs-of-hex-digits might be the best
tradeoff if conversion speed is the criterion.
That's a lot less space-efficient than base6
On Tue, May 5, 2009 at 4:14 PM, Tom Lane wrote:
> Heikki Linnakangas writes:
>> Tom Lane wrote:
>>> I'm thinking plain old pairs-of-hex-digits might be the best
>>> tradeoff if conversion speed is the criterion.
>
>> That's a lot less space-efficient than base64, though.
>
> Well, base64 could gi
Bernd Helmle wrote:
--On Dienstag, Mai 05, 2009 16:57:50 -0400 Andrew Dunstan
wrote:
Hex will already provide some space savings over our current encoding
method for most byteas anyway. It's not like we'd be making things less
efficient space-wise. And in compressed archives the space diffe
--On Dienstag, Mai 05, 2009 16:57:50 -0400 Andrew Dunstan
wrote:
Hex will already provide some space savings over our current encoding
method for most byteas anyway. It's not like we'd be making things less
efficient space-wise. And in compressed archives the space difference is
likely to diss
Tom Lane wrote:
"Kevin Grittner" writes:
Bernd Helmle wrote:
Another approach would be to just dump bytea columns in binary
format only (not sure how doable that is, though).
If that's not doable, perhaps a base64 option for bytea COPY?
I'm thinking plain old pairs-of-hex-digits might be
Tom Lane wrote:
Heikki Linnakangas writes:
Tom Lane wrote:
I'm thinking plain old pairs-of-hex-digits might be the best
tradeoff if conversion speed is the criterion.
That's a lot less space-efficient than base64, though.
Well, base64 could give a 33% savings, but
Heikki Linnakangas writes:
> Tom Lane wrote:
>> I'm thinking plain old pairs-of-hex-digits might be the best
>> tradeoff if conversion speed is the criterion.
> That's a lot less space-efficient than base64, though.
Well, base64 could give a 33% savings, but it's significantly harder
to encode/d
Tom Lane wrote:
> Unless we can think of a more bulletproof format selection mechanism
Would it make any sense to have an option on the COPY command to tell
it to use base64 for bytea columns?
-Kevin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to
Sorry got top-posting -- stupid iphone mail client.
We could eliminate the problem with old dumps by doing something like
\x to indicate a new-style hex dump.
That doesn't make us 100% safe against arbitrary user input but should
be pretty low risk.
--
Greg
On 5 May 2009, at 18:51, Tom
Andrew Dunstan writes:
> Tom Lane wrote:
>> I'm thinking plain old pairs-of-hex-digits might be the best
>> tradeoff if conversion speed is the criterion. The main problem
>> in any case would be to decide how to control the format option.
> Yeah. Any ideas on how to do that? I can't think of a
Tom Lane wrote:
"Kevin Grittner" writes:
Bernd Helmle wrote:
Another approach would be to just dump bytea columns in binary
format only (not sure how doable that is, though).
If that's not doable, perhaps a base64 option for bytea COPY?
I'm thinking plain old pai
--On Dienstag, Mai 05, 2009 10:00:37 -0400 Tom Lane
wrote:
Seems like the right response might be some micro-optimization effort on
byteaout.
Hmm looking into profiler statistics seems to second your suspicion:
Normal COPY shows:
% cumulative self self total
time s
Tom Lane wrote:
> "Kevin Grittner" writes:
> > Bernd Helmle wrote:
> >> Another approach would be to just dump bytea columns in binary
> >> format only (not sure how doable that is, though).
>
> > If that's not doable, perhaps a base64 option for bytea COPY?
>
> I'm thinking plain old pairs-of
"Kevin Grittner" writes:
> Bernd Helmle wrote:
>> Another approach would be to just dump bytea columns in binary
>> format only (not sure how doable that is, though).
> If that's not doable, perhaps a base64 option for bytea COPY?
I'm thinking plain old pairs-of-hex-digits might be the best
tr
Tom Lane writes:
> It seems rather antithetical to one of the main goals of pg_dump,
> which is to provide a dump that can reliably be loaded onto other
> machines or newer versions of Postgres.
You're calling for a pg_export/pg_import tool suite, or I have to learn
to read again :)
> I don't t
Dimitri Fontaine writes:
> Still, apart from lack of interest from developpers and/or resources, is
> there some reason we don't have a pg_dump --binary option?
It seems rather antithetical to one of the main goals of pg_dump,
which is to provide a dump that can reliably be loaded onto other
mach
Tom Lane writes:
> Bernd Helmle writes:
>> That latter occurred recently to me, a customer would like to dump large
>> tables (approx. 12G in size) with pg_dump, but he was annoyed about the
>> performance. Using COPY BINARY reduced the time (unsurprisingly) to a
>> fraction (from 12 minutes
Bernd Helmle wrote:
> Another approach would be to just dump bytea columns in binary
> format only (not sure how doable that is, though).
If that's not doable, perhaps a base64 option for bytea COPY?
-Kevin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make chang
Bernd Helmle writes:
> From time to time we had complains about slow dump of large tables with
> bytea columns, people often complaining about a) size and b) duration of
> the dump.
> That latter occurred recently to me, a customer would like to dump large
> tables (approx. 12G in size) with p
From time to time we had complains about slow dump of large tables with
bytea columns, people often complaining about a) size and b) duration of
the dump.
That latter occurred recently to me, a customer would like to dump large
tables (approx. 12G in size) with pg_dump, but he was annoyed ab
68 matches
Mail list logo
