Greg Stark <[EMAIL PROTECTED]> writes:
> It occurs to me that at least on some OSes the WAL logs are being synced with
> O_SYNC or its ilk. In those cases the writes should be guaranteed to be
> written out in the order postgres wrote them. So if the tail end of the WAL
> entry is there (is there a
It occurs to me that at least on some OSes the WAL logs are being synced with
O_SYNC or its ilk. In those cases the writes should be guaranteed to be
written out in the order postgres wrote them. So if the tail end of the WAL
entry is there (is there any sort of footer?) then the entire entry must
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> I'm still a little nervous about dropping down to CRC32 from CRC64 and so
> was just wondering what the net saving would be using one CRC64 across the
> whole WAL record?
None to speak of; the startup/teardown time is trivial. It's the
per-byte cos
Simon Riggs <[EMAIL PROTECTED]> writes:
> If I/O is a problem, then surely you will agree that PreAllocXLog is
> still required and should not be run by a backend?
It is still required, but it isn't run by backends --- it's fired off
during checkpoints. I think there was some discussion recently
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 31 May 2005 17:27
> To: Greg Stark
> Cc: Mark Cave-Ayland (External); 'Manfred Koizar'; 'Bruce
> Momjian'; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogI
On Tue, 2005-05-31 at 22:36 -0400, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
> > Hmmm. I seem to recall asking myself why xl_prev existed if it wasn't
> > used, but passed that by. Damn.
>
> I couldn't believe it'd been overlooked this long, either. It's the
> sort of thing that y
Simon Riggs <[EMAIL PROTECTED]> writes:
> Hmmm. I seem to recall asking myself why xl_prev existed if it wasn't
> used, but passed that by. Damn.
I couldn't believe it'd been overlooked this long, either. It's the
sort of thing that you assume got done the first time :-(
> PreAllocXLog was alrea
On Tue, 2005-05-31 at 12:27 -0400, Tom Lane wrote:
> Greg Stark <[EMAIL PROTECTED]> writes:
> > Tom Lane <[EMAIL PROTECTED]> writes:
> >>> Is the random WAL data really the concern? It seems like a more reliable
> >>> way
> >>> of dealing with that would be to just accompany every WAL entry with a
On Tue, 31 May 2005 12:07:53 +0100, "Mark Cave-Ayland"
<[EMAIL PROTECTED]> wrote:
>Perhaps Manfred can tell us the generator
>polynomial that was used to create the lookup tables?
32 26 23 22 16 12 11 10 8 7 5 4 2 1
X + X + X + X + X + X + X + X + X + X + X + X +
Greg Stark <[EMAIL PROTECTED]> writes:
> Tom Lane <[EMAIL PROTECTED]> writes:
>>> Is the random WAL data really the concern? It seems like a more reliable way
>>> of dealing with that would be to just accompany every WAL entry with a
>>> sequential id and stop when the next id isn't the correct one
Tom Lane <[EMAIL PROTECTED]> writes:
> > Is the random WAL data really the concern? It seems like a more reliable way
> > of dealing with that would be to just accompany every WAL entry with a
> > sequential id and stop when the next id isn't the correct one.
>
> We do that, too (the xl_prev link
Greg Stark <[EMAIL PROTECTED]> writes:
> Tom Lane <[EMAIL PROTECTED]> writes:
>> It's not really a matter of backstopping the hardware's error detection;
>> if we were trying to do that, we'd keep a CRC for every data page, which
>> we don't. The real reason for the WAL CRCs is as a reliable metho
Tom Lane <[EMAIL PROTECTED]> writes:
> It's not really a matter of backstopping the hardware's error detection;
> if we were trying to do that, we'd keep a CRC for every data page, which
> we don't. The real reason for the WAL CRCs is as a reliable method of
> identifying the end of WAL: when th
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
>> Alternatively, we might say that 64-bit CRC was overkill from
>> day one, and we'd rather get the additional 10% or 20% or so
>> speedup. I'm kinda leaning in that direction, but only weakly.
> What would you need to persuade you either way? I b
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 27 May 2005 17:49
> To: Mark Cave-Ayland (External)
> Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
> pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] C
Tom Lane wrote:
> Alternatively, we might say that 64-bit CRC was overkill from day one,
> and we'd rather get the additional 10% or 20% or so speedup. I'm kinda
> leaning in that direction, but only weakly.
Yes, I lean in that direction too since the CRC calculation is showing
up in our profilin
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> I don't know whether gcc is just producing an inefficient CRC32 compared to
> 2x32 but the results seem very odd There must be something else we are
> missing?
I went back and looked at the code, and see that I was misled by
terminology: what we
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 27 May 2005 15:00
> To: Mark Cave-Ayland (External)
> Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
> pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cos
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 27 May 2005 15:00
> To: Mark Cave-Ayland (External)
> Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
> pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> Opt 32 32a32b2x32 64 64a 64b
>
> O14.91 4.86 5.43 6.00 11.4 11.3911.39
> O24.96 4.94 4.69 5.18 15.86 18.7524.73
> O34.82 4.83
> -Original Message-
> From: Manfred Koizar [mailto:[EMAIL PROTECTED]
> Sent: 25 May 2005 20:25
> To: Manfred Koizar
> Cc: Tom Lane; Greg Stark; Bruce Momjian; Mark Cave-Ayland
> (External); pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert C
> -Original Message-
> From: Manfred Koizar [mailto:[EMAIL PROTECTED]
> Sent: 25 May 2005 20:25
> To: Manfred Koizar
> Cc: Tom Lane; Greg Stark; Bruce Momjian; Mark Cave-Ayland
> (External); pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogIn
On Wed, 18 May 2005 13:50:22 +0200, I wrote:
>The most important figure is, that at MaxSpeed (/O2) 2x32 is almost
>twice as fast as CRC32 while only being marginally slower than CRC32.
^
Silly typo! That should have been:
The most important figure is, that at MaxSpeed (/O2) 2
On Wed, 18 May 2005 01:12:26 -0400, Tom Lane <[EMAIL PROTECTED]> wrote:
>> Wait, par for 32-bit CRCs? Or for 64-bit CRCs calculated using 32-bit ints?
>
>Right, the latter. We haven't actually tried to measure the cost of
>plain 32bit CRCs... although I seem to recall that when we originally
>deci
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 18 May 2005 06:12
> To: Greg Stark
> Cc: Bruce Momjian; Manfred Koizar; Mark Cave-Ayland
> (External); pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert CRC calculatio
On K, 2005-05-18 at 10:24 +0300, Hannu Krosing wrote:
> On T, 2005-05-17 at 22:37 -0400, Tom Lane wrote:
> > Bruce Momjian writes:
> > > I don't understand why we are testing 64-bit CRC when I thought we
> > > agreed that 32-bit was good enough for our purposes.
> >
> > Well, we need to understan
On T, 2005-05-17 at 22:37 -0400, Tom Lane wrote:
> Bruce Momjian writes:
> > I don't understand why we are testing 64-bit CRC when I thought we
> > agreed that 32-bit was good enough for our purposes.
>
> Well, we need to understand exactly what is going on here. I'd not
> like to think that we
On E, 2005-05-16 at 12:35 -0400, Tom Lane wrote:
> Given the lack of impressive advantage to the 64-bit code even on 64-bit
> architectures, it might be best to go with the 32-bit code everywhere,
> but I also think we have grounds to file a gcc bug report.
Maybe on other platforms , but 20% on P
"Dann Corbit" <[EMAIL PROTECTED]> writes:
> Probably you already knew that, and probably the birthday paradox does
> not apply.
>
> I generally use 64 bit CRCs (UMAC) for just about anything that needs a
> CRC.
> http://www.cs.ucdavis.edu/~rogaway/umac/
The birthday paradox doesn't come up here.
Tom Lane wrote:
> Greg Stark <[EMAIL PROTECTED]> writes:
> > Tom Lane <[EMAIL PROTECTED]> writes:
> >> Do we know that? The results I showed put at least one fundamentally
> >> 32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
> >> and 64bit CRCs.
>
> > Wait, par for 32-bi
Greg Stark <[EMAIL PROTECTED]> writes:
> Tom Lane <[EMAIL PROTECTED]> writes:
>> Do we know that? The results I showed put at least one fundamentally
>> 32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
>> and 64bit CRCs.
> Wait, par for 32-bit CRCs? Or for 64-bit CRCs cal
Tom Lane <[EMAIL PROTECTED]> writes:
> Do we know that? The results I showed put at least one fundamentally
> 32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
> and 64bit CRCs.
Wait, par for 32-bit CRCs? Or for 64-bit CRCs calculated using 32-bit ints?
--
greg
-
May 17, 2005 9:26 PM
> To: Bruce Momjian
> Cc: Manfred Koizar; Mark Cave-Ayland; pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
>
> Bruce Momjian writes:
> > Tom Lane wrote:
> >> Well, we need to understand exactly what is goi
Bruce Momjian writes:
> Tom Lane wrote:
>> Well, we need to understand exactly what is going on here. I'd not
>> like to think that we dropped back from 64 to 32 bit because of one
>> possibly-minor optimization bug in one compiler on one platform.
>> Even if that compiler+platform is 90% of the
Tom Lane wrote:
> Bruce Momjian writes:
> > I don't understand why we are testing 64-bit CRC when I thought we
> > agreed that 32-bit was good enough for our purposes.
>
> Well, we need to understand exactly what is going on here. I'd not
> like to think that we dropped back from 64 to 32 bit be
Bruce Momjian writes:
> I don't understand why we are testing 64-bit CRC when I thought we
> agreed that 32-bit was good enough for our purposes.
Well, we need to understand exactly what is going on here. I'd not
like to think that we dropped back from 64 to 32 bit because of one
possibly-minor
Manfred Koizar wrote:
> On Mon, 16 May 2005 12:35:35 -0400, Tom Lane <[EMAIL PROTECTED]> wrote:
> >Anyone want to try it with non-gcc compilers?
>
> MS VC++ 6.0 with various predefined optimizer settings
>
> 2x3264
> Default (without any /O) 0.828125
On Mon, 16 May 2005 12:35:35 -0400, Tom Lane <[EMAIL PROTECTED]> wrote:
>Anyone want to try it with non-gcc compilers?
MS VC++ 6.0 with various predefined optimizer settings
2x3264
Default (without any /O) 0.828125 0.906250
MinSize (contains /O1)
On Mon, 16 May 2005, Tom Lane wrote:
> I did some experimentation and concluded that gcc is screwing up
> big-time on optimizing the CRC64 code for 32-bit Intel. It does much
> better on every other architecture though.
>
> Anyone want to try it with non-gcc compilers?
Solaris 9 x86 - Sun Wor
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 16 May 2005 17:36
> To: Mark Cave-Ayland (External)
> Cc: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
> I did some experimentation and
(BI tested crctest in two machines and two versions of gcc.
(B
(BUltraSPARC III, gcc 2.95.3:
(Bgcc -O1 crctest.c 1.321517 s
(Bgcc -O2 crctest.c 1.099186 s
(Bgcc -O3 crctest.c 1.099330 s
(Bgcc -O1 crctest64.c
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> Sigh, so it would help if I had added the offset to the data pointer... ;)
Would you post the corrected program so people can try it on a few other
architectures? No point in reinventing the wheel, even if it is a
pretty trivial wheel.
e'; 'Bruce Momjian';
> > 'pgsql-hackers@postgresql.org'
> > Subject: RE: [HACKERS] Cost of XLogInsert CRC calculations
>
>
> Win32 UINT64: 0x782104059a01660 (crc0) ~158us
> Win32 UINT32: 0x78210405 (crc1), 0x59a01660 (crc0) ~58us
> FC
On Mon, 2005-05-16 at 12:12 +0100, Mark Cave-Ayland wrote:
> This now gives the following (correct) result on both platforms:
> Win32: 1.8GHz P4, WinXP
> Linux: 2.8GHz Xeon, FC1
>
>
> Win32 UINT64: 0x782104059a01660 (crc0)
> ~158us
> Win32 UINT32: 0x78210405 (crc
: pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
>
>
> "Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> > Sigh, so it would help if I had added the offset to the data
> > pointer... ;)
>
> Would you post the corre
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> I didn't post the sources to the list originally as I wasn't sure if the
> topic were of enough interest to warrant a larger email. I've attached the
> two corrected programs as a .tar.gz - crctest.c uses uint32, whereas
> crctest64.c uses uint64.
I
> -Original Message-
> From: Mark Cave-Ayland [mailto:[EMAIL PROTECTED]
> Sent: 16 May 2005 09:04
> To: 'Simon Riggs'
> Cc: 'Christopher Kings-Lynne'; 'Tom Lane'; 'Bruce Momjian';
> 'pgsql-hackers@postgresql.org'
> S
> -Original Message-
> From: Simon Riggs [mailto:[EMAIL PROTECTED]
> Sent: 12 May 2005 16:52
> To: Mark Cave-Ayland (External)
> Cc: 'Christopher Kings-Lynne'; 'Tom Lane'; 'Bruce Momjian';
> pgsql-hackers@postgresql.org
> Subject: RE:
On Thu, 2005-05-12 at 13:48 +0100, Mark Cave-Ayland wrote:
> > From: Christopher Kings-Lynne [mailto:[EMAIL PROTECTED]
> >
> > >>perhaps the CRC-32 routines could be written in in-line assembler
> > >
> > >
> > > If you can do this, step right up. :-)
> > >
> > > Best Regards, Simon Riggs
> >
> -Original Message-
> From: Christopher Kings-Lynne [mailto:[EMAIL PROTECTED]
> Sent: 12 May 2005 02:46
> To: Simon Riggs
> Cc: Mark Cave-Ayland (External); 'Tom Lane'; 'Bruce Momjian';
> pgsql-hackers@postgresql.org
> Subject: Re: [HAC
perhaps the CRC-32 routines could be written in in-line assembler
If you can do this, step right up. :-)
Best Regards, Simon Riggs
Surely there's an open source code floating around somewhere?
Chris
---(end of broadcast)---
TIP 4: Don't 'kill -9' the
On Wed, 2005-05-11 at 13:40 +0100, Mark Cave-Ayland wrote:
> So unless we can guarantee a minimum of 1k data per Xlog record then
> Adler-32 won't be suitable.
Most records are either 8KB or much less than 1KB. Is the benefit gained
from the 8KB records worth the loss on the more frequent smaller
> -Original Message-
> From: Tom Lane [mailto:[EMAIL PROTECTED]
> Sent: 10 May 2005 23:22
> To: Simon Riggs
> Cc: Bruce Momjian; Mark Cave-Ayland (External);
> pgsql-hackers@postgresql.org
> Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
&
On Tue, 2005-05-10 at 18:22 -0400, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
> > The cause of the performance problem has been attributed to it being a
> > 64-bit rather than 32-bit calculation. That is certainly part of it, but
> > I have seen evidence that there is an Intel proces
Simon Riggs <[EMAIL PROTECTED]> writes:
> The cause of the performance problem has been attributed to it being a
> 64-bit rather than 32-bit calculation. That is certainly part of it, but
> I have seen evidence that there is an Intel processor stall associated
> with the use of a single byte consta
On Tue, 2005-05-10 at 10:34 -0400, Bruce Momjian wrote:
> Tom Lane wrote:
> > "Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> > > I was just researching some articles on compression (zlib) and I saw
> > > mention
> > > of the Adler-32 algorithm which is supposed to be slightly less accurate
> > >
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> I was just researching some articles on compression (zlib) and I saw mention
> of the Adler-32 algorithm which is supposed to be slightly less accurate
> than an equivalent CRC calculation but significantly faster to compute. I
> haven't located a go
Tom Lane wrote:
> "Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> > I was just researching some articles on compression (zlib) and I saw mention
> > of the Adler-32 algorithm which is supposed to be slightly less accurate
> > than an equivalent CRC calculation but significantly faster to compute.
> -Original Message-
> From: Mark Cave-Ayland [mailto:[EMAIL PROTECTED]
> Sent: 07 March 2005 11:04
> To: '[EMAIL PROTECTED]'
> Cc: 'pgsql-hackers@postgreSQL.org'
> Subject: Re: Cost of XLogInsert CRC calculations
(cut)
> > I suppose that the bulk of the CPU cycles being attributed to
>
On Fri, 2005-03-11 at 19:31 +0100, Hans-JÃrgen SchÃnig wrote:
> > One of the things I was thinking about was whether we could use up those
> > cycles more effectively. If we were to include a compression routine
> > before we calculated the CRC that would
> > - reduce the size of the blocks to be
One of the things I was thinking about was whether we could use up those
cycles more effectively. If we were to include a compression routine
before we calculated the CRC that would
- reduce the size of the blocks to be written, hence reduce size of xlog
- reduce the following CRC calculation
I
On Mon, 2005-03-07 at 20:50 -0500, Tom Lane wrote:
> Simon Riggs <[EMAIL PROTECTED]> writes:
> > Well, we're using the CRC in 3 separate places...
> > (1) for xlog records
> > (2) for complete blocks copied to xlog
> > (3) for control files
>
> > For (1), records are so short that probably CRC16 w
Tom Lane <[EMAIL PROTECTED]> writes:
> Simon Riggs <[EMAIL PROTECTED]> writes:
>
> > For (1), records are so short that probably CRC16 would be sufficient
> > without increasing the error rate noticeably.
>
> The control files are so short that CRC16 would be plenty.
It's not really the size o
Simon Riggs <[EMAIL PROTECTED]> writes:
> Well, we're using the CRC in 3 separate places...
> (1) for xlog records
> (2) for complete blocks copied to xlog
> (3) for control files
> For (1), records are so short that probably CRC16 would be sufficient
> without increasing the error rate noticeably
On Mon, 2005-03-07 at 09:39 -0500, Tom Lane wrote:
> "Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> > Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
> > days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
> > sure there are some error rates so
Tom Lane wrote:
> "Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
>
>>Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
>>days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
>>sure there are some error rates somewhere dependent upon the polynomial
"Mark Cave-Ayland" <[EMAIL PROTECTED]> writes:
> Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
> days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
> sure there are some error rates somewhere dependent upon the polynomial and
> the types of er
Hi Tom,
> I was profiling a case involving UPDATEs into a table with too many
indexes (brought to > mind by mysql's sql-bench, about which more later) and
got this rather surprising result > for routines costing more than 1% of the
total runtime:
(cut)
> I suppose that the bulk of the CPU cycles
On Sun, 2005-03-06 at 00:17 -0500, Tom Lane wrote:
> I was profiling a case involving UPDATEs into a table with too many
> indexes (brought to mind by mysql's sql-bench, about which more later)
> and got this rather surprising result for routines costing more than
> 1% of the total runtime:
>
> Ea
On Sun, 6 Mar 2005, Tom Lane wrote:
I suppose that the bulk of the CPU cycles being attributed to XLogInsert
are going into the inlined CRC calculations. Maybe we need to think
twice about the cost/benefit ratio of using 64-bit CRCs to protect xlog
records that are often only a few dozen bytes.
Is
70 matches
Mail list logo
