> > > Putting xl_prev to the end helps only for direct IO WAL sync
> > > methods, else we would need it on every page.
> >
> > [There is already an XLogRecPtr on each 8k page.]
>
> Given that hardware sector size is still 512 bytes, should
> there be a way of detecting a missing 512 byte block
On Fri, Oct 27, 2006 at 10:11:08AM +0100, Simon Riggs wrote:
> > Putting xl_prev to the end helps only for direct IO WAL sync methods,
> > else we would need it on every page.
>
> [There is already an XLogRecPtr on each 8k page.]
Given that hardware sector size is still 512 bytes, should there be
On Fri, 2006-10-27 at 10:54 +0200, Zeugswetter Andreas ADI SD wrote:
> >> In the WAL we just need to be able to detect torn pages and stop
> >> reading WAL at that point. That's easier and doesn't really need a
> >> CRC. We could just adopt the Sybase strategy of storing a unique id
> >> number
>> In the WAL we just need to be able to detect torn pages and stop
>> reading WAL at that point. That's easier and doesn't really need a
>> CRC. We could just adopt the Sybase strategy of storing a unique id
>> number every 512 bytes throughout the WAL page. If those numbers
don't
>> match th
On Mon, Oct 23, 2006 at 05:23:27PM -0400, Tom Lane wrote:
> Mark Kirkwood <[EMAIL PROTECTED]> writes:
> > Right - I think the regression is caused by libc and kernel being built
> > with gcc 3.4.6 and the test program being built with gcc 4.1.2.
>
> Why do you think that? The performance of the
"Simon Riggs" <[EMAIL PROTECTED]> writes:
> I've looked into this in more depth following your suggestion: I think
> it seems straightforward to move the xl_prev field from being a header
> to a trailer. That way when we do the test on the back pointer we will
> be assured that there is no torn p
On Tue, 2006-10-24 at 15:42 -0400, Gregory Stark wrote:
> Tom Lane <[EMAIL PROTECTED]> writes:
>
> > Simon is essentially arguing that if we are willing to assume no
> > incomplete write() we may as well assume it for WAL too. This seems
> > to me to be raising the risk significantly, but I admit
"Gurjeet Singh" <[EMAIL PROTECTED]> writes:
> On 10/23/06, Tom Lane <[EMAIL PROTECTED] > wrote:
>> I didn't particularly trust the timing calculations in your benchmark
>> program,
> Any particular reason? (why and what did you doubt in it?).
Well, the specific thing that set off my bogometer
On Tue, 2006-10-24 at 14:07 -0400, Tom Lane wrote:
> "Gregory Maxwell" <[EMAIL PROTECTED]> writes:
> > I'm not aware of any other system which can guaranteed the atomicity
> > of 8k writes.
>
> The reasoning for supporting full_page_writes = off is that if you have
> a stable kernel and suitable b
Tom Lane <[EMAIL PROTECTED]> writes:
> Simon is essentially arguing that if we are willing to assume no
> incomplete write() we may as well assume it for WAL too. This seems
> to me to be raising the risk significantly, but I admit that I can't
> put my finger on why exactly.
Actually I think we
Sorry for getting into the conversation so late... It was a long weekend in India.On 10/23/06, Tom Lane <
[EMAIL PROTECTED]
> wrote:I didn't particularly trust the timing calculations in your benchmark
program, Any particular reason? (why and what did you doubt in it?). I designed the prog.
"Gregory Maxwell" <[EMAIL PROTECTED]> writes:
> I'm not aware of any other system which can guaranteed the atomicity
> of 8k writes.
The reasoning for supporting full_page_writes = off is that if you have
a stable kernel and suitable backup power, battery backed write cache,
etc, your risk of a pa
On 10/24/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
I wasn't aware that a system could protect against this. :-)
I write 8 Kbytes - how can I guarantee that the underlying disk writes
all 8 Kbytes before it loses power? And why isn't the CRC a valid means
of dealing with this? :-)
[snip]
[EMAIL PROTECTED] wrote:
> I'm on wrong on one of these assumptions, I'm open to being educated.
> My opinion as of a few seconds ago, is that a write to a single disk
> sector is safe, but that a write that extends across several sectors
> is not.
Unless it's fsync'ed, which is what we do at CHE
On Tue, 2006-10-24 at 14:52 +0100, Simon Riggs wrote:
> On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
> > "Simon Riggs" <[EMAIL PROTECTED]> writes:
> > > On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
> > >> There are actually three checks used to detect end of WAL: zero record
> > >> leng
On Tue, 2006-10-24 at 12:47 -0400, [EMAIL PROTECTED] wrote:
> On Tue, Oct 24, 2006 at 05:05:58PM +0100, Simon Riggs wrote:
> > On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
> > > On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
> > > > On Tue, 2006-10-24 at 09:37 -0400, To
On Tue, Oct 24, 2006 at 05:05:58PM +0100, Simon Riggs wrote:
> On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
> > On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
> > > On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
> > > > No, because unlike tuples, WAL records can and
On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
> On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
> > On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
> > > No, because unlike tuples, WAL records can and do cross page boundaries.
>
> > But not that often, with full_page_
[EMAIL PROTECTED] writes:
> The 5 year claim seems
> decades too short unless they are talking about a newer technology.
I think what Simon is on about is CRCs being routinely used on the cable
between the disk drive and the CPU. When I was involved in this stuff
you usually only got a parity bit
On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
> On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
> > "Simon Riggs" <[EMAIL PROTECTED]> writes:
> > > On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
> > >> There are actually three checks used to detect end of WAL: zero record
> >
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
> "Simon Riggs" <[EMAIL PROTECTED]> writes:
> > On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
> >> There are actually three checks used to detect end of WAL: zero record
> >> length, invalid checksum, and incorrect back-pointer. Zero length i
"Simon Riggs" <[EMAIL PROTECTED]> writes:
> On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
>> There are actually three checks used to detect end of WAL: zero record
>> length, invalid checksum, and incorrect back-pointer. Zero length is
>> the first and cleanest-looking test, but AFAICS we hav
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
> "Simon Riggs" <[EMAIL PROTECTED]> writes:
> > On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
> >> No can do --- we rely on the checksums to be able to tell when we've hit
> >> the end of WAL during replay.
>
> > No we don't: Zero length re
Tom Lane wrote:
Mark Kirkwood <[EMAIL PROTECTED]> writes:
Right - I think the regression is caused by libc and kernel being built
with gcc 3.4.6 and the test program being built with gcc 4.1.2.
Why do you think that? The performance of the CRC loop shouldn't depend
at all on either libc or t
Mark Kirkwood <[EMAIL PROTECTED]> writes:
> Right - I think the regression is caused by libc and kernel being built
> with gcc 3.4.6 and the test program being built with gcc 4.1.2.
Why do you think that? The performance of the CRC loop shouldn't depend
at all on either libc or the kernel, beca
Benny Amorsen wrote:
"MK" == Mark Kirkwood <[EMAIL PROTECTED]> writes:
MK> Here are the results after building gcc 4.1.2 (repeating results
MK> for gcc 3.4.6 for comparison). I suspect that performance is
MK> probably impacted because gcc 4.1.2 (and also the rest of the
MK> tool-chain) is built
On Mon, 23 Oct 2006, Tom Lane wrote:
> Hmm. Maybe store the CRCs into a global array somewhere?
>
> uint32 results[NTESTS];
>
> for ...
> {
> INIT/COMP/FIN_CRC32...
> results[j] = mycrc;
> }
>
> This still adds a bit of overhead to the outer loo
Jeremy Drake <[EMAIL PROTECTED]> writes:
> On Mon, 23 Oct 2006, Tom Lane wrote:
>> That's not a good workaround, because making mycrc expensive to access
>> means your inner loop timing isn't credible at all. Instead try making the
>> buffer array nonlocal --- malloc it, perhaps.
> That did not m
On Mon, 23 Oct 2006, Tom Lane wrote:
> Jeremy Drake <[EMAIL PROTECTED]> writes:
> > So at this point I realize that intel's compiler is optimizing the loop
> > away, at least for the std crc and probably for both. So I make mycrc an
> > array of 2, and substript mycrc[j&1] in the loop.
>
> That's
"Simon Riggs" <[EMAIL PROTECTED]> writes:
> On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
>> No can do --- we rely on the checksums to be able to tell when we've hit
>> the end of WAL during replay.
> No we don't: Zero length records are the trigger for EOF.
Only if the file happens to be
> In most cases, it would be foolish to avoid: but there are cases where
> the data is CRC checked by the hardware/system already, so I'd like to
> make an option to turn this off, defaulting to on, for safety.
How would we know? What are those cases?
Sounds like a foot gun to me.
Sincerely,
J
On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
> "Simon Riggs" <[EMAIL PROTECTED]> writes:
> > Instead, I'd like to include a parameter to turn off CRC altogether, for
> > heavily CPU bound operations and the WAL drive on trustworthy hardware.
>
> No can do --- we rely on the checksums to be a
Bruce, Tom, All:
> > Given the numbers I posted earlier today, the proposal is dead in the
> > water anyway, quite aside from any legal considerations.
>
> Agreed. I just wanted to point out we have other sharks in the water.
*IF* Slice-by-8 turned out to be a winner, I could get the legal issue
On 10/23/06, Tom Lane <[EMAIL PROTECTED]> wrote:
It's not so much that I don't trust Intel as that a CRC algorithm is
exactly the sort of nice little self-contained thing that people love
to try to patent these days. What I am really afraid of is that someone
else has already invented this same
"Simon Riggs" <[EMAIL PROTECTED]> writes:
> Instead, I'd like to include a parameter to turn off CRC altogether, for
> heavily CPU bound operations and the WAL drive on trustworthy hardware.
No can do --- we rely on the checksums to be able to tell when we've hit
the end of WAL during replay. You
On Sun, 2006-10-22 at 18:06 -0400, Tom Lane wrote:
> These numbers are um, not impressive. Considering that a large fraction
> of our WAL records are pretty short, the fact that slice8 consistently
> loses at short buffer lengths is especially discouraging. Much of that
> ground could be made up
Jeremy Drake <[EMAIL PROTECTED]> writes:
> So at this point I realize that intel's compiler is optimizing the loop
> away, at least for the std crc and probably for both. So I make mycrc an
> array of 2, and substript mycrc[j&1] in the loop.
That's not a good workaround, because making mycrc expe
> "MK" == Mark Kirkwood <[EMAIL PROTECTED]> writes:
MK> Here are the results after building gcc 4.1.2 (repeating results
MK> for gcc 3.4.6 for comparison). I suspect that performance is
MK> probably impacted because gcc 4.1.2 (and also the rest of the
MK> tool-chain) is built with gcc 3.4.6 -
On Mon, 2006-10-23 at 05:22 -0400, Gregory Stark wrote:
> "Simon Riggs" <[EMAIL PROTECTED]> writes:
>
> > Slice-By-8 was first mentioned here:
>
> Are you sure?
>
> US patent 7,047,479 filed in 2002 sounds like it may be relevant:
>
> http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2
Mark Kirkwood wrote:
Tom Lane wrote:
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
Ah - good point, FreeBSD is using an older compiler:
FreeBSD: gcc (GCC) 3.4.6 [FreeBSD] 20060305
Linux: gcc (GCC) 4.1.1 (Gentoo 4.1.1)
Hmm - there is a Free
"Simon Riggs" <[EMAIL PROTECTED]> writes:
> Slice-By-8 was first mentioned here:
Are you sure?
US patent 7,047,479 filed in 2002 sounds like it may be relevant:
http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=7047
On Mon, 23 Oct 2006, Mark Kirkwood wrote:
> Tom Lane wrote:
> >
> >
> > Yah, I checked. Several times... but if anyone else wants to repeat
> > the experiment, please do. Or look for bugs in either my test case
> > or Gurjeet's.
>
>
Just for fun, I tried it out with both GCC and with Intel's C
Tom Lane wrote:
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc version 4.0.1 (Apple Computer, Inc. build 5363)
both Intels: gcc version 4.1.1 20060525 (Red Hat 4.1.1-1)
Interesting that it's o
On Mon, 23 Oct 2006 07:35:08 +0200, Tom Lane <[EMAIL PROTECTED]> wrote:
I realize I forgot to document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc version 4.0.1 (Apple Computer, Inc. build 5363)
both Intels: gcc version 4.1.1 20060525 (Red Hat 4.1.1-1)
Interesting th
On Sun, 2006-10-22 at 17:18 +0200, Martijn van Oosterhout wrote:
> On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
> > Hi all,
> >
> >Michael Kounavis has given a green signal (please refer the forwarded
> > message).
>
> I don't think that helps. The publishing date of this ar
Tom Lane wrote:
Mark Kirkwood <[EMAIL PROTECTED]> writes:
Interesting that the slice-8 algorithm seems to work noticeably better
on FreeBSD than Linux -
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
Mark Kirkwood <[EMAIL PROTECTED]> writes:
> Interesting that the slice-8 algorithm seems to work noticeably better
> on FreeBSD than Linux -
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc versio
Tom Lane wrote:
Yah, I checked. Several times... but if anyone else wants to repeat
the experiment, please do. Or look for bugs in either my test case
or Gurjeet's.
FWIW - FreeBSD and Linux results using Tom's test program on almost
identical hardware[1]:
Std crc Slice-8
[EMAIL PROTECTED] writes:
> ... My first speculation is
> that you did your Intel numbers backwards. My second speculation is
> that you already thought of that and confirmed before posting. :-)
Yah, I checked. Several times... but if anyone else wants to repeat
the experiment, please do. Or loo
On Sun, Oct 22, 2006 at 06:06:13PM -0400, Tom Lane wrote:
> Intel Xeon EM64T (Fedora Core 5)
>
> 8192 bytes4.4208797.633120
> 1024 bytes0.5717940.819372
> 64 bytes 0.0473540.071906
>
> Intel Pentium 4 (Fedora Core 5)
>
> 8192 bytes6.94232428.84857
Tom Lane wrote:
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > Tom Lane wrote:
> >> Bruce Momjian <[EMAIL PROTECTED]> writes:
> >>> Also, why are we more critical of an Intel-provided
> >>> idea than any other idea we get from the community?
> >>
> >> Bitter experience with other companies.
>
>
Bruce Momjian <[EMAIL PROTECTED]> writes:
> Tom Lane wrote:
>> Bruce Momjian <[EMAIL PROTECTED]> writes:
>>> Also, why are we more critical of an Intel-provided
>>> idea than any other idea we get from the community?
>>
>> Bitter experience with other companies.
> The problem is we have lots of c
Tom Lane wrote:
> Bruce Momjian <[EMAIL PROTECTED]> writes:
> > However, I am not sure getting a clarification from the author even
> > helps us legally. Also, why are we more critical of an Intel-provided
> > idea than any other idea we get from the community?
>
> Bitter experience with other co
"Gurjeet Singh" <[EMAIL PROTECTED]> writes:
>> If possible, people should test it on different platforms, so as to
>> ensure that it doesn't perform any worse than older implementation on any
>> supported platform (please post the results, if you do test it).
I didn't particularly trust the timing
On 10/22/06, Martijn van Oosterhout wrote:
On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
> Hi all,
>
>Michael Kounavis has given a green signal (please refer the forwarded
> message).
I don't think that helps. The publishing date of this article was March
2006. If this is r
On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
> Hi all,
>
>Michael Kounavis has given a green signal (please refer the forwarded
> message).
I don't think that helps. The publishing date of this article was March
2006. If this is really the first time this algorithm was publi
On 10/22/06, Gregory Maxwell <[EMAIL PROTECTED]> wrote:
On Opteron 265 his test code shows SB8 (the intel alg) is 2.48x fasterfor checksum and 1.95x faster for verify for the 800 * 1024 blocks of1 KB each workload. For 10 blocks of 8k I got simmlar results aswell.
I think you meant800*1024 bloc
Hi all, Michael Kounavis has given a green signal (please refer the forwarded message).Best regards,-- [EMAIL PROTECTED][EMAIL PROTECTED] gmail | hotmail | yahoo }.com
-- Forwarded message --From: Kounavis, Michael E <[EMAIL PROTECTED]
>Date: Oct 20, 2006 10:43 PMSubject: RE: CRC
On 10/21/06, Tom Lane <[EMAIL PROTECTED]> wrote:
[snip]
It hasn't even been tested. One thing I'd want to know about is the
performance effect on non-Intel machines.
On Opteron 265 his test code shows SB8 (the intel alg) is 2.48x faster
for checksum and 1.95x faster for verify for the 800 * 10
Bruce Momjian <[EMAIL PROTECTED]> writes:
> However, I am not sure getting a clarification from the author even
> helps us legally. Also, why are we more critical of an Intel-provided
> idea than any other idea we get from the community?
Bitter experience with other companies.
> So unless we hea
Gurjeet Singh wrote:
> On 10/20/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote:
> >
> > It doesn't say anything about
> > being free of patents though.
> >
> > The Sourceforge project referred to in the article (but for which no
> > link is given) seems to be this one:
> >
> > http://sourceforge.net/
On 10/20/06, Alvaro Herrera <[EMAIL PROTECTED]> wrote:
It doesn't say anything aboutbeing free of patents though.The Sourceforge project referred to in the article (but for which nolink is given) seems to be this one:
http://sourceforge.net/projects/slicing-by-8Yes. I had already mentioned that lin
Tom Lane wrote:
> "Gurjeet Singh" <[EMAIL PROTECTED]> writes:
> >> Please refer to the following link for a new algorithm to calculate
> >> CRC, developed by Intel.
> >>
> >> http://www.intel.com/technology/magazine/communications/slicing-by-8-0306.htm
>
> I can't help wondering what the patent s
"Gurjeet Singh" <[EMAIL PROTECTED]> writes:
>> Please refer to the following link for a new algorithm to calculate
>> CRC, developed by Intel.
>>
>> http://www.intel.com/technology/magazine/communications/slicing-by-8-0306.htm
I can't help wondering what the patent status of this algorithm is.
(
Resending... The previous 3 attempt haven't succeeded, probably because of size restrictions; so sending in two parts.On 10/18/06, Gurjeet Singh <
[EMAIL PROTECTED]> wrote:Hi all, Please refer to the following link for a new algorithm to calculate CRC, developed by Intel.
http://www.intel.co
65 matches
Mail list logo