Re: [HACKERS] Nearing beta?
On Thu, 2013-04-11 at 12:22 -0400, Tom Lane wrote: > We went around on whether we liked this or not, but it seemed to me > that the discussion came out at the same place Peter had submitted > to start with. I don't know why he's not committed it, but I have > no objection to him doing so, as long as it happens PDQ. Committed and commit fest closed. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PROPOSAL: tracking aggregated numbers from pg_stat_database
Tomas Vondra wrote: > On 13.4.2013 15:01, Peter Eisentraut wrote: > > long-term analysis? Maybe you could even use event triggers to have > > DROP DATABASE do that automatically. > > I don't think event triggers are a good solution, although I'm wondering > how that's supposed to work. It doesn't, because event triggers do not support shared objects (databases, tablespaces, roles). -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On Sat, Apr 13, 2013 at 06:14:28PM +0300, Ants Aasma wrote: > > CRCs are known to be good for that sort of thing; it's what they were > > designed for. I'd like to see some evidence that any substitute > > algorithm has similar properties. Without that, I'm going to vote > > against this idea. > > Sorry for creating confusion here by playing fast and loose with the > terminology. We are not talking about that hash function at all. What > we are talking about here is Fowler-Noll-Vo-ish > (http://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) > hash function that is restructured to be parallelisable with SIMD > instructions with the explicit goal of being as fast as possible. The > resulting hash function is roughly two orders of magnitude faster than > 1-byte-at-a-time CRC32 currently in use. Performance is about > comparable with optimized fixed size memcpy running in cache. > > Based on current analysis, it is particularly good at detecting single > bit errors, as good at detecting burst errors as can be expected from > 16 bits and not horrible at detecting burst writes of zeroes. It is > quite bad at detecting multiple uncorrelated single bit errors and > extremely bad at detecting repeating patterns of errors in low order > bits. > > All in all I would say that the performance is worth the loss in > detection capability as we are not talking about using the checksum to > prove correctness. Agreed. It would be good to get these details into the patch so others are not confused in the future. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Process title for autovac
On Sat, Apr 13, 2013 at 5:56 AM, Peter Eisentraut wrote: > On Sat, 2013-04-06 at 13:20 -0700, Jeff Janes wrote: > > I've often wanted to know what the autovacuum worker was doing. The > > process title seems like the best place to get this information, but > > the process title tells me what database it is in, but not what table > > it is working on. > > Because the process title is publicly visible, you shouldn't put any > "interesting" information in it. > OK. I did not think that the existence of a table name would be interesting, but I can see that some would consider it so. > I think what you want might be better kept in pg_stat_activity. > > And in fact it is already there. I had just never thought of looking there for background process stuff. It even includes the notice "(to prevent wraparound)" when applicable. Thanks! I'd still like it in the process title, because I'm not worried about exposing table names and I always have 'top' running while I don't monitor pg_stat_activity as a matter of routine. But I guess the security concern wins. I'll mark it as rejected. Thanks, Jeff
Re: [HACKERS] Enabling Checksums
On Sat, Apr 13, 2013 at 6:26 PM, Andres Freund wrote: >> All in all I would say that the performance is worth the loss in >> detection capability as we are not talking about using the checksum to >> prove correctness. > > Is it actually a loss compared to our 16bit flavor of crc32 we now use? > I didn't think so far from the properties you described? I would have to run the testsuite I made to see now much but I would presume so. The algorithm relies on multiplication for bit diffusion and multiply has lousy diffusion on low order bits, exactly no diffusion for the lowest bit. And for 16bit values low order bits is quite a large fraction of the total hash. If we allow for operations that are not in SSE2 then there are a few things that we could do to make the hash quality better without affecting performance. pmulld instruction (SSE4.1) would allow for 32bit values in the intermediate state. And pshufb (SSE3) would allow us to swap high and low bytes introducing additional mixing. On Intel Sandy Bridge, if I understand the microarchitecture correctly, either change would be basically free, but not both because pshufb and paddw use execution ports 0 and 5, while pmulld needs port 0 and pmullw needs port 1. Currently the main loop takes 1 cycle per 16byte chunk, any changes introducing conflicts there would cut the performance in half. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PROPOSAL: tracking aggregated numbers from pg_stat_database
On 13.4.2013 15:01, Peter Eisentraut wrote: > On Sat, 2013-04-06 at 21:51 +0200, Tomas Vondra wrote: >> This more or less works in stable environments, but once you start >> dropping databases (think of hosting with shared DB server) it gets >> unusable because after DROP DATABASE the database suddenly disappears >> from the sum. >> >> Therefore I do propose tracking the aggregated stats, similar to the >> pg_stat_bgwriter view. > > It seems like this will open a can of worms. Maybe someone wants > aggregated stats for pg_stat_user_tables? Or maybe instead of the sum, > someone wants to track the average? etc. I don't think we should turn What I propose is a simple cumulative counter, just like the other counters we do have right now. I don't think tracking an average (or any other statistics) makes much sense here. And as the number of objects changes over time (e.g. dbs may be created/dropped), I'm wondering what would be the definition of average? BTW I've proposed tracking aggregated table/index stats in my second message in this thread. > the statistics collector into a poor man's data warehouse or statistics > engine. Couldn't you transfer the data to some other system for I certainly don't want to overcomplicate the stats system, and I don;t think I'm turning it into a DWH or statistics engine. And even with these aggregated counters, it still requires snapshotting and additional processing. It's just a bunch of counters. I'm currently struggling with (quite uncommon) deployments where databases are created/dropped regularly (not to mention tables and indexes), and it's surprisingly difficult to process such stats to get reasonable values. The point is this allows tracking data that are otherwise effectively lost. With the current stats you have to discard intervals where databases were dropped (because well, the data disappear so you don't know what is the actual snapshot diff). Depending on the number of DB drops and snapshot interval, this may very well be most of the time. > long-term analysis? Maybe you could even use event triggers to have > DROP DATABASE do that automatically. I don't think event triggers are a good solution, although I'm wondering how that's supposed to work. Tomas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On 2013-04-13 18:14:28 +0300, Ants Aasma wrote: > On Sat, Apr 13, 2013 at 5:58 PM, Tom Lane wrote: > > Andres Freund writes: > >> On 2013-04-13 09:14:26 -0400, Bruce Momjian wrote: > >>> As I understand it, SIMD is just a CPU-optimized method for producing a > >>> CRC checksum. Is that right? Does it produce the same result as a > >>> non-CPU-optimized CRC calculation? > > > >> No we are talking about a different algorithm that results in different > >> results, thats why its important to choose now since we can't change it > >> later without breaking pg_upgrade in further releases. > >> http://en.wikipedia.org/wiki/SIMD_%28hash_function%29 > > > > [ squint... ] We're talking about a *cryptographic* hash function? > > Why in the world was this considered a good idea for page checksums? > > > > In the first place, it's probably not very fast compared to some > > alternatives, and in the second place, the criteria by which people > > would consider it a good crypto hash function have approximately nothing > > to do with what we need for a checksum function. What we want for a > > checksum function is high probability of detection of common hardware > > failure modes, such as burst errors and all-zeroes. This is > > particularly critical when we're going with only a 16-bit checksum --- > > the probabilities need to be skewed in the right direction, or it's not > > going to be all that terribly useful. > > > > CRCs are known to be good for that sort of thing; it's what they were > > designed for. I'd like to see some evidence that any substitute > > algorithm has similar properties. Without that, I'm going to vote > > against this idea. > > Sorry for creating confusion here by playing fast and loose with the > terminology. We are not talking about that hash function at all. What > we are talking about here is Fowler-Noll-Vo-ish > (http://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) > hash function that is restructured to be parallelisable with SIMD > instructions with the explicit goal of being as fast as possible. The > resulting hash function is roughly two orders of magnitude faster than > 1-byte-at-a-time CRC32 currently in use. Performance is about > comparable with optimized fixed size memcpy running in cache. Gah, one shouldn't look to quick for a reference, sorry. > Based on current analysis, it is particularly good at detecting single > bit errors, as good at detecting burst errors as can be expected from > 16 bits and not horrible at detecting burst writes of zeroes. It is > quite bad at detecting multiple uncorrelated single bit errors and > extremely bad at detecting repeating patterns of errors in low order > bits. > All in all I would say that the performance is worth the loss in > detection capability as we are not talking about using the checksum to > prove correctness. Is it actually a loss compared to our 16bit flavor of crc32 we now use? I didn't think so far from the properties you described? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On Sat, Apr 13, 2013 at 5:58 PM, Tom Lane wrote: > Andres Freund writes: >> On 2013-04-13 09:14:26 -0400, Bruce Momjian wrote: >>> As I understand it, SIMD is just a CPU-optimized method for producing a >>> CRC checksum. Is that right? Does it produce the same result as a >>> non-CPU-optimized CRC calculation? > >> No we are talking about a different algorithm that results in different >> results, thats why its important to choose now since we can't change it >> later without breaking pg_upgrade in further releases. >> http://en.wikipedia.org/wiki/SIMD_%28hash_function%29 > > [ squint... ] We're talking about a *cryptographic* hash function? > Why in the world was this considered a good idea for page checksums? > > In the first place, it's probably not very fast compared to some > alternatives, and in the second place, the criteria by which people > would consider it a good crypto hash function have approximately nothing > to do with what we need for a checksum function. What we want for a > checksum function is high probability of detection of common hardware > failure modes, such as burst errors and all-zeroes. This is > particularly critical when we're going with only a 16-bit checksum --- > the probabilities need to be skewed in the right direction, or it's not > going to be all that terribly useful. > > CRCs are known to be good for that sort of thing; it's what they were > designed for. I'd like to see some evidence that any substitute > algorithm has similar properties. Without that, I'm going to vote > against this idea. Sorry for creating confusion here by playing fast and loose with the terminology. We are not talking about that hash function at all. What we are talking about here is Fowler-Noll-Vo-ish (http://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) hash function that is restructured to be parallelisable with SIMD instructions with the explicit goal of being as fast as possible. The resulting hash function is roughly two orders of magnitude faster than 1-byte-at-a-time CRC32 currently in use. Performance is about comparable with optimized fixed size memcpy running in cache. Based on current analysis, it is particularly good at detecting single bit errors, as good at detecting burst errors as can be expected from 16 bits and not horrible at detecting burst writes of zeroes. It is quite bad at detecting multiple uncorrelated single bit errors and extremely bad at detecting repeating patterns of errors in low order bits. All in all I would say that the performance is worth the loss in detection capability as we are not talking about using the checksum to prove correctness. Regards, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On 2013-04-13 10:58:53 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2013-04-13 09:14:26 -0400, Bruce Momjian wrote: > >> As I understand it, SIMD is just a CPU-optimized method for producing a > >> CRC checksum. Is that right? Does it produce the same result as a > >> non-CPU-optimized CRC calculation? > > > No we are talking about a different algorithm that results in different > > results, thats why its important to choose now since we can't change it > > later without breaking pg_upgrade in further releases. > > http://en.wikipedia.org/wiki/SIMD_%28hash_function%29 > > [ squint... ] We're talking about a *cryptographic* hash function? > Why in the world was this considered a good idea for page checksums? In Ants' implementation its heck of a lot of faster than any CRC implementation we have seen so far on relatively large blocks (like pages). pgbench results: ca+csw_uxo-frkuzl0yzs0wsdl8dipzv-ugmvyn-yv45sgub...@mail.gmail.com byte/cycle comparison: CA+CSw_su1fopLNBz1NAfkSNw4_=gv+5pf0kdlqmpvukw1q4...@mail.gmail.com > In the first place, it's probably not very fast compared to some > alternatives, and in the second place, the criteria by which people > would consider it a good crypto hash function have approximately nothing > to do with what we need for a checksum function. What we want for a > checksum function is high probability of detection of common hardware > failure modes, such as burst errors and all-zeroes. This is > particularly critical when we're going with only a 16-bit checksum --- > the probabilities need to be skewed in the right direction, or it's not > going to be all that terribly useful. > > CRCs are known to be good for that sort of thing; it's what they were > designed for. I'd like to see some evidence that any substitute > algorithm has similar properties. Without that, I'm going to vote > against this idea. Ants has dome some analysis on this, like CA+CSw_tMoA85e=1vs4omjzjg2mr_hulikovpd80dp5rurds...@mail.gmail.com . That doesn't look bad to me and unless I am missing something its better than our CRC with 16bit. So while I would say its not 100% researched there has been a rather detailed investigation by Ants - I am rather impressed. My biggest doubt so far is the reliance on inline assembly for the top performance on x86-64 and a generic implementation otherwise that only is really fast with appropriate compiler flags.. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
Andres Freund writes: > On 2013-04-13 09:14:26 -0400, Bruce Momjian wrote: >> As I understand it, SIMD is just a CPU-optimized method for producing a >> CRC checksum. Is that right? Does it produce the same result as a >> non-CPU-optimized CRC calculation? > No we are talking about a different algorithm that results in different > results, thats why its important to choose now since we can't change it > later without breaking pg_upgrade in further releases. > http://en.wikipedia.org/wiki/SIMD_%28hash_function%29 [ squint... ] We're talking about a *cryptographic* hash function? Why in the world was this considered a good idea for page checksums? In the first place, it's probably not very fast compared to some alternatives, and in the second place, the criteria by which people would consider it a good crypto hash function have approximately nothing to do with what we need for a checksum function. What we want for a checksum function is high probability of detection of common hardware failure modes, such as burst errors and all-zeroes. This is particularly critical when we're going with only a 16-bit checksum --- the probabilities need to be skewed in the right direction, or it's not going to be all that terribly useful. CRCs are known to be good for that sort of thing; it's what they were designed for. I'd like to see some evidence that any substitute algorithm has similar properties. Without that, I'm going to vote against this idea. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On 2013-04-13 09:14:26 -0400, Bruce Momjian wrote: > On Fri, Apr 12, 2013 at 02:38:27PM -0700, Jeff Davis wrote: > > In general, we have more flexibility with WAL because there is no > > upgrade issue. It would be nice to share code with the data page > > checksum algorithm; but really we should just use whatever offers the > > best trade-off in terms of complexity, performance, and error detection > > rate. > > > > I don't think we need to decide all of this right now. Personally, I'm > > satisfied having SIMD checksums on data pages now and leaving WAL > > optimization until later. > > As I understand it, SIMD is just a CPU-optimized method for producing a > CRC checksum. Is that right? Does it produce the same result as a > non-CPU-optimized CRC calculation? No we are talking about a different algorithm that results in different results, thats why its important to choose now since we can't change it later without breaking pg_upgrade in further releases. http://en.wikipedia.org/wiki/SIMD_%28hash_function%29 Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On Fri, Apr 12, 2013 at 02:38:27PM -0700, Jeff Davis wrote: > In general, we have more flexibility with WAL because there is no > upgrade issue. It would be nice to share code with the data page > checksum algorithm; but really we should just use whatever offers the > best trade-off in terms of complexity, performance, and error detection > rate. > > I don't think we need to decide all of this right now. Personally, I'm > satisfied having SIMD checksums on data pages now and leaving WAL > optimization until later. As I understand it, SIMD is just a CPU-optimized method for producing a CRC checksum. Is that right? Does it produce the same result as a non-CPU-optimized CRC calculation? -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PROPOSAL: tracking aggregated numbers from pg_stat_database
On Sat, 2013-04-06 at 21:51 +0200, Tomas Vondra wrote: > This more or less works in stable environments, but once you start > dropping databases (think of hosting with shared DB server) it gets > unusable because after DROP DATABASE the database suddenly disappears > from the sum. > > Therefore I do propose tracking the aggregated stats, similar to the > pg_stat_bgwriter view. It seems like this will open a can of worms. Maybe someone wants aggregated stats for pg_stat_user_tables? Or maybe instead of the sum, someone wants to track the average? etc. I don't think we should turn the statistics collector into a poor man's data warehouse or statistics engine. Couldn't you transfer the data to some other system for long-term analysis? Maybe you could even use event triggers to have DROP DATABASE do that automatically. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Process title for autovac
On Sat, 2013-04-06 at 13:20 -0700, Jeff Janes wrote: > I've often wanted to know what the autovacuum worker was doing. The > process title seems like the best place to get this information, but > the process title tells me what database it is in, but not what table > it is working on. Because the process title is publicly visible, you shouldn't put any "interesting" information in it. I think what you want might be better kept in pg_stat_activity. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
On 12 April 2013 23:21, Ants Aasma wrote: >> In general, we have more flexibility with WAL because there is no >> upgrade issue. It would be nice to share code with the data page >> checksum algorithm; but really we should just use whatever offers the >> best trade-off in terms of complexity, performance, and error detection >> rate. >> >> I don't think we need to decide all of this right now. Personally, I'm >> satisfied having SIMD checksums on data pages now and leaving WAL >> optimization until later. > > +1 OK, lets drop that idea then. SIMD checksums for 16-bit page checksums only in this release. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
