Re: [HACKERS] tolower() identifier downcasing versus multibyte encodings
On Sat, Mar 19, 2011 at 6:10 AM, Tom Lane t...@sss.pgh.pa.us wrote: Or we could bite the bullet and start using str_tolower(), but the performance implications of that are unpleasant; not to mention that we really don't want to re-introduce the Turkish problem with unexpected handling of i/I in identifiers. How about first pass with 'a' - 'A' and if highbit is found then str_tolower()? You will still confuse turks, but at least nothing should break. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Indent authentication overloading
On Mon, Mar 14, 2011 at 16:26, Magnus Hagander mag...@hagander.net wrote: On Mon, Mar 14, 2011 at 16:17, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Mon, Mar 14, 2011 at 14:43, Robert Haas robertmh...@gmail.com wrote: Also, the text is not accurate: nothing has been automatically changed to anything. The pg_hba.conf file is just as it was. You could say something like ident authentication on local socket treated as peer, but I think a better idea would be to just remove this message altogether. The idea being to let people know it's been deprecated, nothing else. But sure, we can just remove the message - at elast for now, and maybe add it $n releases down the road when people are expected to have changed over. I'm with Robert on this one --- the first reaction I had to your description of the patch was why do we need a log message for that? If there were some real reason to push people away from use of the non-preferred term, maybe it'd be worth nagging them to change; but there isn't. Ok. fair enough, I'll take that part out. Here's an updated patch that removes this log message, and adds a few lines to initdb to create a combination of ident/peer rows. And finally, adds docs. Are people in general in favor of making the change provided I do that, then? Comments? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml index c05805b..3f4631e 100644 --- a/doc/src/sgml/client-auth.sgml +++ b/doc/src/sgml/client-auth.sgml @@ -457,17 +457,28 @@ hostnossl replaceabledatabase/replaceable replaceableuser/replaceable termliteralident//term listitem para - Obtain the operating system user name of the client (for - TCP/IP connections by contacting the ident server on the - client, for local connections by getting it from the - operating system) and check if it matches the requested - database user name. + Obtain the operating system user name of the client + by contacting the ident server on the client + and check if it matches the requested database user name. + This is only available for TCP/IP connections. See xref linkend=auth-ident for details. /para /listitem /varlistentry varlistentry +termliteralpeer//term +listitem + para + Obtain the operating system user name from the operating system + and check if it matches the requested database user name. + This is only available for local connections. + See xref linkend=auth-peer for details. + /para +/listitem + /varlistentry + + varlistentry termliteralldap//term listitem para @@ -1200,7 +1211,7 @@ omicron bryanh guest1 /sect2 sect2 id=auth-ident - titleIdent-based Authentication/title + titleIdent Authentication/title indexterm primaryident/primary @@ -1208,11 +1219,9 @@ omicron bryanh guest1 para The ident authentication method works by obtaining the client's -operating system user name and using it as the allowed database user -name (with an optional user name mapping). -The determination of the client's -user name is the security-critical point, and it works differently -depending on the connection type, as described below. +operating system user name from an ident server and using it as +the allowed database user name (with an optional user name mapping). +This is only supported on TCP/IP connections. /para para @@ -1230,9 +1239,6 @@ omicron bryanh guest1 /variablelist /para - sect3 -titleIdent Authentication Over TCP/IP/title - para The quoteIdentification Protocol/quote is described in RFC 1413. Virtually every Unix-like @@ -1275,36 +1281,48 @@ omicron bryanh guest1 since productnamePostgreSQL/ does not have any way to decrypt the returned string to determine the actual user name. /para - /sect3 + /sect2 + + sect2 id=auth-peer + titlePeer Authentication/title + + indexterm +primarypeer/primary + /indexterm + + para +The peer authentication method works by obtaining the client's +operating system user name from the kernel and using it as the +allowed database user name (with optional user name mapping). This +is only supported on local connections. + /para - sect3 -titleIdent Authentication Over Local Sockets/title + para +The following configuration options are supported for productnamepeer/productname: +variablelist + varlistentry + termliteralmap/literal/term + listitem + para +Allows for mapping between system and
Re: [HACKERS] VACUUM FULL deadlock with backend startup
On Fri, Mar 18, 2011 at 11:25 PM, Nikhil Sontakke nikhil.sonta...@enterprisedb.com wrote: The fix is similar to the earlier commit by Tom. I tested this fix against 8.3.13. We lock the parent catalog now before calling index_open. Patch against git HEAD attached with this mail. I guess we will backpatch this? Tom's last commit was backpatched till 8.2 I think. Is it really worth slowing down the startup sequence for every connection to avoid deadlocking against a very rare maintenance operation? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_last_xact_replay_timestamp meaning
Robert Haas wrote: On Fri, Mar 18, 2011 at 7:23 PM, Bruce Momjian br...@momjian.us wrote: I just applied a doc patch for pg_last_xact_replay_timestamp, and the text now says: ? ? ? entryGet timestamp of last transaction replayed during recovery. ? ? ? ?This is the time at which the commit or abort WAL record for that ? ? ? ?transaction was generated on the primary. ? ? ? ?If no transactions have been replayed during recovery, this function ? ? ? ?returns NULL. ?Otherwise, if recovery is still in progress this will ? ? ? ?increase monotonically. ?If recovery has completed then this value will ? ? ? ?remain static at the value of the last transaction applied during that ? ? ? ?recovery. ?When the server has been started normally without recovery ? ? ? ?the function returns NULL. Is this really the last commit/abort record or the last WAL record? What should it be? ?Is the name of this function correct? ?Do we care only about commit/abort records? Why? Commit and abort records have a timestamp. Other WAL records don't. OK, thanks. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Sync Rep and shutdown Re: [HACKERS] Sync Rep v19
On 2011-03-18 18:25, Robert Haas wrote: On Fri, Mar 18, 2011 at 1:15 PM, Simon Riggssi...@2ndquadrant.com wrote: On Thu, 2011-03-17 at 09:33 -0400, Robert Haas wrote: Thanks for the review! Lets have a look here... You've added a test inside the lock to see if there is a standby, which I took out for performance reasons. Maybe there's another way, I know that code is fiddly. You've also added back in the lock acquisition at wakeup with very little justification, which was a major performance hit. Together that's about a20% hit in performance in Yeb's tests. I think you should spend a little time thinking how to retune that. Ouch. Do you have a link that describes his testing methodology? I will look at it. Testing 'methodology' sounds a bit heavy. I tested a number of patch versions over time, with 30 second, hourly and nightly pgbench runs. The nightly more for durability/memory leak testing than tps numbers, since I gradually got the impression that pg_bench tests on syncrep setups somehow suffer less from big differences between tests. postgres and recovery.conf I used to test v17 is listed here http://archives.postgresql.org/pgsql-hackers/2011-02/msg02364.php After the tests on v17 I played a bit with small memory changes in the postgres.conf to see if the tps would go up. It went up a little but not enough to mention on the lists. All tests after v17 were done with the postgres.conf that I've copy pasted below. I mentioned a performance regression in http://archives.postgresql.org/pgsql-hackers/2011-03/msg00298.php And performance improvement in http://archives.postgresql.org/pgsql-hackers/2011-03/msg00464.php All three servers (el cheapo consumer grade) the same: triple core AMD's, 16GB ECC, raid 0 over 2 SATA disks, XFS, nobarrier, separated data and xlog partitions. NB: there is no BBU controller in these servers. They don't run production stuff, it's just for testing. 1Gbit ethernet on non-blocking HP switch. No other load. ./configure --enable-depend --with-ossp-uuid --with-libxml --prefix=/mgrid/postgres regards, Yeb Havinga Here's the postgresql.conf non-default I used after each new initdb. (synchronous_replication is off since it prevented me from adding a replication user, so after a initial basebackup I needed to turn it on) #-- # CUSTOMIZED OPTIONS #-- #custom_variable_classes = '' # list of custom variable class names #shared_preload_libraries = 'pg_stat_statements' #custom_variable_classes = 'pg_stat_statements' #pg_stat_statements.max = 100 #pg_stat_statements.track = all syslog_ident = relay autovacuum = off #debug_print_parse = on #debug_print_rewritten = on #debug_print_plan = on #debug_pretty_print = on log_error_verbosity = verbose log_min_messages = warning log_min_error_statement = warning listen_addresses = '*'# what IP address(es) to listen on; search_path='\$user\, public, hl7' archive_mode = on archive_command = 'cd .' checkpoint_completion_target = 0.9 checkpoint_segments = 16 default_statistics_target = 500 constraint_exclusion = on max_connections = 100 maintenance_work_mem = 528MB effective_cache_size = 5GB work_mem = 144MB wal_buffers = 8MB shared_buffers = 528MB wal_level = 'archive' max_wal_senders = 10 wal_keep_segments = 100 # 1600MB (for production increase this) synchronous_standby_names = 'standby1,standby2,standby3' #synchronous_replication = on -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] VACUUM FULL deadlock with backend startup
Hi, The fix is similar to the earlier commit by Tom. I tested this fix against 8.3.13. We lock the parent catalog now before calling index_open. Patch against git HEAD attached with this mail. I guess we will backpatch this? Tom's last commit was backpatched till 8.2 I think. Is it really worth slowing down the startup sequence for every connection to avoid deadlocking against a very rare maintenance operation? Not really a performance issue AFAICS. If the relcache init file exists, then the phase2 of the catalog cache which eventually calls the above code path is avoided. Regards, Nikhils
Re: [HACKERS] tolower() identifier downcasing versus multibyte encodings
Marko Kreen mark...@gmail.com writes: On Sat, Mar 19, 2011 at 6:10 AM, Tom Lane t...@sss.pgh.pa.us wrote: Or we could bite the bullet and start using str_tolower(), but the performance implications of that are unpleasant; not to mention that we really don't want to re-introduce the Turkish problem with unexpected handling of i/I in identifiers. How about first pass with 'a' - 'A' and if highbit is found then str_tolower()? Hm, maybe. There's still the problem of what to do in src/port/pgstrcasecmp.c, which won't have the infrastructure needed to do that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Indent authentication overloading
Magnus Hagander mag...@hagander.net writes: Here's an updated patch that removes this log message, and adds a few lines to initdb to create a combination of ident/peer rows. And finally, adds docs. Comments? As near as I can tell (I hate reading u-style diffs) you've documented the ident and peer keywords as being mutually exclusive, ie, the docs say that the correct keyword for the connection type *must* be used in pg_hba.conf. Which is not how the code behaves, and shouldn't be how the code behaves, for backwards-compatibility reasons. The docs need to state the truth, namely that ident is still allowed as a synonym for peer on local connections. Otherwise people will get confused as to why their pg_hba files still work. The code changes look sane in a quick scan, though I didn't read them in detail. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sync rep fsync=off
On 18 Mar 2011, at 21:12, Robert Haas wrote: While investigating Simon's complaint about my patch of a few days ago, I discovered that synchronous replication appears to slow to a crawl if fsync is turned off on the standby. I'm not sure why this is happening or what the right behavior is in this case, but I think some kind of adjustment is needed because the current behavior is quite surprising. We have few servers here running 8.3. And few weeks ago I had to populate one database with quite a number of entries. I have script that does that, but it takes a while. I decided to turn fsck to off. Oddly enough, the server started to crawl quite badly, load was very high. That was 8.3 on rhel 5.4. My point is, it is sometimes bad combination of disks and controllers that does that. Not necessarily software. fsync off doesn't always mean that things are going to fly, it can cause it to expose hardware bottlenecks much quicker. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] odd output of pg_basebackup
On Sun, Feb 13, 2011 at 17:44, Magnus Hagander mag...@hagander.net wrote: On Mon, Feb 7, 2011 at 13:43, Fujii Masao masao.fu...@gmail.com wrote: Hi, When I ran pg_basebackup with -x, -P and -v options, I encountered the following odd output. $ pg_basebackup -D hoge -x -P -v xlog start point: 0/220 33708/17322 kB (100%) 1/1 tablespaces ( )02) 02) is a part of the previously reported message, it looks odd that the subsequent progress report doesn't reset that. We should reset the output buffer with each progress report? Yes. I thought I had fixed that already.. Hmm. I'll stick it on my list.. I had fixed it, it was sitting in the patch with functionality for streaming the xlog during base backup over a second connection istead of in the main one. Since it looks like I won't be able to finish that one in time, I have extracted the fix for this problem from it and applied it as a separate patch. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Indent authentication overloading
On Sat, Mar 19, 2011 at 16:24, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: Here's an updated patch that removes this log message, and adds a few lines to initdb to create a combination of ident/peer rows. And finally, adds docs. Comments? As near as I can tell (I hate reading u-style diffs) you've documented Ah, apologies. I made a -c diff, found a typo, remade it as -u.. pfft. the ident and peer keywords as being mutually exclusive, ie, the docs say that the correct keyword for the connection type *must* be used in pg_hba.conf. Which is not how the code behaves, and shouldn't be how the code behaves, for backwards-compatibility reasons. The docs need to state the truth, namely that ident is still allowed as a synonym for peer on local connections. Otherwise people will get confused as to why their pg_hba files still work. Hmm: Good point, update attached. The code changes look sane in a quick scan, though I didn't read them in detail. Thx. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ *** a/doc/src/sgml/client-auth.sgml --- b/doc/src/sgml/client-auth.sgml *** *** 457,473 hostnossl replaceabledatabase/replaceable replaceableuser/replaceable termliteralident//term listitem para ! Obtain the operating system user name of the client (for ! TCP/IP connections by contacting the ident server on the ! client, for local connections by getting it from the ! operating system) and check if it matches the requested ! database user name. See xref linkend=auth-ident for details. /para /listitem /varlistentry varlistentry termliteralldap//term listitem para --- 457,486 termliteralident//term listitem para ! Obtain the operating system user name of the client ! by contacting the ident server on the client ! and check if it matches the requested database user name. ! Ident authentication can only be used on TCP/IP ! connections. When specified for local connections, peer ! authentication will be used instead. See xref linkend=auth-ident for details. /para /listitem /varlistentry varlistentry + termliteralpeer//term + listitem + para + Obtain the operating system user name from the operating system + and check if it matches the requested database user name. + This is only available for local connections. + See xref linkend=auth-peer for details. + /para + /listitem +/varlistentry + +varlistentry termliteralldap//term listitem para *** *** 1200,1206 omicron bryanh guest1 /sect2 sect2 id=auth-ident !titleIdent-based Authentication/title indexterm primaryident/primary --- 1213,1219 /sect2 sect2 id=auth-ident !titleIdent Authentication/title indexterm primaryident/primary *** *** 1208,1220 omicron bryanh guest1 para The ident authentication method works by obtaining the client's ! operating system user name and using it as the allowed database user ! name (with an optional user name mapping). ! The determination of the client's ! user name is the security-critical point, and it works differently ! depending on the connection type, as described below. /para para The following configuration options are supported for productnameident/productname: variablelist --- 1221,1239 para The ident authentication method works by obtaining the client's ! operating system user name from an ident server and using it as ! the allowed database user name (with an optional user name mapping). ! This is only supported on TCP/IP connections. /para +note + para + When ident is specified for a local (non-TCP/IP) connection, + peer authentication (see xref linkend=auth-peer) will be + used instead. + /para +/note + para The following configuration options are supported for productnameident/productname: variablelist *** *** 1230,1238 omicron bryanh guest1 /variablelist /para -sect3 - titleIdent Authentication Over TCP/IP/title - para The quoteIdentification Protocol/quote is described in RFC 1413. Virtually every Unix-like --- 1249,1254 *** *** 1275,1310 omicron bryanh guest1 since productnamePostgreSQL/ does not have any way to decrypt the returned string to
Re: [HACKERS] Indent authentication overloading
Magnus Hagander mag...@hagander.net writes: On Sat, Mar 19, 2011 at 16:24, Tom Lane t...@sss.pgh.pa.us wrote: ... The docs need to state the truth, namely that ident is still allowed as a synonym for peer on local connections. Otherwise people will get confused as to why their pg_hba files still work. Hmm: Good point, update attached. That looks OK to me. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Collations versus record-returning functions
I wrote: ISTM there are basically three things we might do about this: (1) Decide that the patch's behavior is correct and what's embodied in the regression expected file is wrong. (2) Allow collations to propagate up through nodes that deliver noncollatable outputs. (3) Decide that composite types are collatable. I thought of another possibility, which is to special-case field selection from a function-returning-composite, ie make it look through the function node and use the function's input collation. FieldSelect needs to be a special case in the collation assignment code anyway because of the possibility of taking the collation from the field declaration instead of the input, so this is not *quite* as ugly as it first sounds. It's still ugly, but it makes that regression test pass with only a very localized change. So I will do it like that for now until someone comes up with an argument for another choice. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] tolower() identifier downcasing versus multibyte encodings
On Sat, Mar 19, 2011 at 5:05 PM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Sat, Mar 19, 2011 at 6:10 AM, Tom Lane t...@sss.pgh.pa.us wrote: Or we could bite the bullet and start using str_tolower(), but the performance implications of that are unpleasant; not to mention that we really don't want to re-introduce the Turkish problem with unexpected handling of i/I in identifiers. How about first pass with 'a' - 'A' and if highbit is found then str_tolower()? Hm, maybe. There's still the problem of what to do in src/port/pgstrcasecmp.c, which won't have the infrastructure needed to do that. You mean client-side? Could we have a str_tolower without xxx_l branch that always does wide-char conversion if high-bit is set? Custom locale there won't make sense there anyway? -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Indent authentication overloading
On Sat, Mar 19, 2011 at 17:39, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: On Sat, Mar 19, 2011 at 16:24, Tom Lane t...@sss.pgh.pa.us wrote: ... The docs need to state the truth, namely that ident is still allowed as a synonym for peer on local connections. Otherwise people will get confused as to why their pg_hba files still work. Hmm: Good point, update attached. That looks OK to me. Applied. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Accepted into GSoC 2011, student proposals start March 28
Hi! The PostgreSQL project has been accepted into the Google Summer of Code 2011. Students may begin submitting proposals starting March 28, concluding on April 8. Development work runs from May 23 through August 15. For students, suggested projects, ideas and details are at: http://wiki.postgresql.org/wiki/GSoC_2011 Our GSoC landing page is at: http://www.google-melange.com/gsoc/org/show/google/gsoc2011/postgresql We encourage students to contact project admins - me, Josh Berkus and Robert Treat this year - if they have questions. Once students have a proposal in mind, we will encourage them to engage with pgsql-hackers to flesh out their proposals and get feedback the same way that all contributors do. For those of you who have been around for previous GSoCs, this should be familiar to you. :) Many thanks to the 15 volunteer mentors and admins this year (in no particular order): Dave Page - Former mentor - pgAdmin, Windows, Packaging, Infrastructure Heikki Linnakangas - Postgres Committer Magnus Hagander - Postgres Committer, Windows, pgAdmin Guillaume Lelarge - pgAdmin Jehan-Guillaume de Rorthais - phpPgAdmin Joe Abbate - Python-related, catalog-related projects David E. Wheeler - Perl-related, extensions, PGXN Mark Wong - benchmarking, monitoring, performance Tatsuo Ishii - Postgres Committer, pgpool-II Stephen Frost - Postgres contributor Devrim Gündüz - Administration related software (dashboard) Josh Berkus - auto-configuration, performance testing Selena Deckelmann - configuration, testing Andreas Scherbaum - performance, configuration, testing Robert Treat - Past mentor 2x, co-admin, Mentor Summit attendee. We can always accept more mentors! Actual assignment to projects depends greatly on the proposals from students. Please contact me if you are interested. Thanks! -selena -- http://chesnok.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.
On 03/18/2011 10:48 PM, Kevin Grittner wrote: the least of the evils. I guess we should document it, though, so nobody has a false expectation that seeing something on the replica means that a connection looking at the master will see something that current. Agreed. Note, however, that even if there's no such guarantee, it's highly unlikely for a user (or application) to ever notice this during normal operation. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sync rep fsync=off
On Sat, Mar 19, 2011 at 11:34 AM, Grzegorz Jaskiewicz g...@pointblue.com.pl wrote: On 18 Mar 2011, at 21:12, Robert Haas wrote: While investigating Simon's complaint about my patch of a few days ago, I discovered that synchronous replication appears to slow to a crawl if fsync is turned off on the standby. I'm not sure why this is happening or what the right behavior is in this case, but I think some kind of adjustment is needed because the current behavior is quite surprising. We have few servers here running 8.3. And few weeks ago I had to populate one database with quite a number of entries. I have script that does that, but it takes a while. I decided to turn fsck to off. Oddly enough, the server started to crawl quite badly, load was very high. That was 8.3 on rhel 5.4. My point is, it is sometimes bad combination of disks and controllers that does that. Not necessarily software. fsync off doesn't always mean that things are going to fly, it can cause it to expose hardware bottlenecks much quicker. Well, it's possible. But I think it'd be worth a look at the code to see if there's some bad interaction there between the no-fsync code and the sync-rep code - like, if we don't actually fsync, does the flush pointer ever get updated? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] VACUUM FULL deadlock with backend startup
On Sat, Mar 19, 2011 at 10:46 AM, Nikhil Sontakke nikhil.sonta...@enterprisedb.com wrote: Not really a performance issue AFAICS. If the relcache init file exists, then the phase2 of the catalog cache which eventually calls the above code path is avoided. Oh, that doesn't sound so bad, then. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] I am confused after reading codes of PostgreSQL three week
2011/3/19 Vaibhav Kaushal vaibhavkaushal...@gmail.com: Hello hom, Frankly I am a learner as well. The experts here are almost always ready to help and would be a better source of information. Moreover I am also using eclipse but I do not use it for building the source. I use it only as a source code browser (its easy in GUI; isn't it? ). I am trying to learn about the executor so can't say much about the parser. However I suppose that you must be knowing the rules of the tools flex and bison to understand the parser. And why are you into scan.c? It is created by flex dear. Read the scan.l and gram.y instead. It is these files which are responsible for the major work done by the parser. If you are keen about the parser, go learn lex and yacc (or flex and bison ... they are almost the same) and then go through the scan.l and gram.y files. It is actually an _extremely_ tough job to read the generated files. Once again, do turn off the Search for duplicate source files option. There are no duplicate files in the source tree. Also, if you are using the copy of source tree which was built once in the workspace, things can be a little different. @others: Well, I do know that there are a few books in the market written by the devs but how much does it help when I am already banging my head into source since last 5 months? Regards, Vaibhav Thanks Vaibhav. I trace into scan.c because I want to known how the paser tree is built and I debug the source step by step. Then the eclipse pick up the scan.I and the excute order does not match the code. Actually, I have no idea which module of the source I should read first. I have a quick glance at the source and I known a litter about how a query excutes. But the modules are so connected. I don't known what part I should be deep in. Now, I plan to study deep in mmgr. Will it be suitable? -- Best Wishes! hom -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Sync Rep and shutdown Re: [HACKERS] Sync Rep v19
On Sat, Mar 19, 2011 at 10:32 AM, Yeb Havinga yebhavi...@gmail.com wrote: Testing 'methodology' sounds a bit heavy. I tested a number of patch versions over time, with 30 second, hourly and nightly pgbench runs. The nightly more for durability/memory leak testing than tps numbers, since I gradually got the impression that pg_bench tests on syncrep setups somehow suffer less from big differences between tests. postgres and recovery.conf I used to test v17 is listed here http://archives.postgresql.org/pgsql-hackers/2011-02/msg02364.php After the tests on v17 I played a bit with small memory changes in the postgres.conf to see if the tps would go up. It went up a little but not enough to mention on the lists. All tests after v17 were done with the postgres.conf that I've copy pasted below. Hmm, I'm not going to be able to reproduce this here, and my test setup didn't show a clear regression. I can try beating on it some more, but... Any chance you could rerun your test with the latest master-branch code, and perhaps also with the patch I proposed upthread to remove a branch from the section protection by SyncRepLock? I can't really tell from reading the emails you linked what was responsible for the slowdowns and speedups, and it is unclear to me how much impact my recent changes actually had. I would have expected the dominant cost to be waiting for the slave to complete its fsync, with network overhead as runner-up. And indeed this appears to be the case on my ext4-based system. I would not have expected contention on SyncRepLock to be much of a factor. It strikes me that if contention on SyncRepLock IS the dominating factor, the whole approach to queue management is pretty well busted. *Every* walsender takes SyncRepLock in exclusive mode on receipt of *every* standby reply message. That seems rather inefficient. To make matters worse, every time a walsender grabs SyncRepLock, it redoes the whole computation of who the synchronous standby is from scratch. It strikes me that it ought to be possible to rejigger things so that when a walsender exits, it signals any other walsenders that exist to recheck whether they need to take over the role of synchronous standby; then, each walsender needs to take the lock and recheck only when it first connects, and each time it's signalled thereafter. When a walsender does decide that a change in the synchronous standby is needed, it should store the ID of the new walsender in shared memory, in a variable protected by SyncRepLock, so that the current synchronous standby can notice the change with a simple integer comparison. It also strikes me that we ought to be able to rejigger things so that the backends being woken up never need to touch shared memory at all - i.e. eliminate syncRepState - thus reducing cache line contention. If WaitLatch() returns true, then the latch was set, presumably by walsender. My recent patch added a couple of places where the latch can be set by the waiting process itself in response to an interrupt, but that case can be handled by adding a backend-local flag variable indicating whether we set the latch ourselves. If we determine that the latch is set and the did-it-myself flag is clear, we can conclude that we were awakened by a walsender and call it good. If the latch is set and the did-it-myself flag is also set, then we were interrupted, and we MAY also have been awakened by walsender at around the same time. We grab SyncRepLock to remove ourselves from the queue, and if we find we're already removed, then we know we were interrupted just as walsender awakened us; otherwise, it's a pure interrupt. It'd be interesting to see the results of some testing with LWLOCK_STATS defined, to see whether SyncRepLock actually is contended and if so to what degree. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
