Re: [HACKERS] Serializable Isolation without blocking
Hi, Kevin Grittner wrote: We're very much on the same page. My goal was to get predicate locking that didn't miss anything, even though it was ridiculously coarse, then implement the simplest possible SSI on top of it, without worrying about optimizations, then incrementally move toward production quality. I clearly didn't communicate that as well as I'd hoped. :-( I understood it now :-) And I think it generally is a good plan. I sort of missed the predicate locking step from the wiki's Development Path. I now see it has its own section above. Is there any such plan of development for predicate locking? To me that seems to be the harder problem (due to being more general). Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 1:51 AM, Tom Lane t...@sss.pgh.pa.us wrote: This strikes me as a completely bad idea. We need get no farther than the point that it assumes nobody can have a database named replication Though I might misunderstand your point. My proposal would force the users who have a database named replication to change their .pgpass file and enclose the replication database field in double quote when they upgrade the Postgres to v8.5. For example; 192.168.1.50:5432:replication:foo:foopass The same problem also exists in pg_hba.conf. It's because I introduced new keyword replication in pg_hba.conf to authenticate the standby server. This restriction is not acceptable? If so, I'd need to consider an authentication configuration for replication again: introduce new configuration file? just change the keyword name to unpopular one?... (although I notice the patch also appears to assume that libpq knows internally that the connection is for replication --- I thought we were going to avoid libpq changes for SR?) Yes, but I changed the libpq just a bit; if the conninfo string including replication=1 is given to PQconnectdb(), the libpq determines that this connection is for replication, and puts the replication-request in a startup packet. This is for a backend to switch to walsender mode when the startup packet arrives. Otherwise, we would have to authenticate such backend twice on different context, i.e., a normal backend and walsender. So the settings for each context would be required in pg_hba.conf. This is odd, I think. I don't see any real strong reason why a .pgpass entry for this purpose couldn't depend on having * in the database field. Oh, you are right. Since a role cannot use a different password per database, * in the database field seems to be enough. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: I: [HACKERS] TODO: Allow substring/replace() to get/set bit values
Is anybody interested? Otherwise the entry could be removed from the TODO list... Even if not, you can still submit a patch. There are a lot more users of PG than there are people who read -hackers. Ok, I'll try and submit a patch. Thank you very much. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buffer statistics for pg_stat_statements
Tom Lane t...@sss.pgh.pa.us wrote: I don't necessarily know what the right thing to do with the new ones is, but I am pretty sure that pg_indent will revert any changes you make to the existing ones. That it will. The proposed changes to the existing lines are an exercise in uselessness; and to the extent that you format the added lines with this layout in mind, the final result could be worse than what you'd get if you adapt to pg_indent's rules to start with. Here is the proposed patch to adjust white spaces. It does not indent variables, but indents comments of the variables to adjust other fields. Are those changes ok? Regards, --- Takahiro Itagaki NTT Open Source Software Center pg_stat_statements_bufusage_20100107.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 07:19, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Jan 6, 2010 at 11:11 PM, Magnus Hagander mag...@hagander.net wrote: I haven't read up on the rest of the patch, but where do we put the rest of the information about the replication master? Like which IP and port to connect to? Perhaps it could/should go there? Such information are supplied in the parameter 'primary_conninfo' of recovery.conf. For example; primary_conninfo = 'host=192.168.1.50 port=5432 user=foo' So the password can just go there, no? If an untrusted user has direct read access to your PGDATA directory (where recovery.conf goes), you've lost already... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 09:26, Fujii Masao masao.fu...@gmail.com wrote: The same problem also exists in pg_hba.conf. It's because I introduced new keyword replication in pg_hba.conf to authenticate the standby server. This restriction is not acceptable? If so, I'd need to consider an authentication configuration for replication again: introduce new configuration file? just change the keyword name to unpopular one?... I certainly think there are a lot of installations out there with a database named replication. That doesn't mean it's unacceptable though. However, wouldn't it make more logical sense to replace host/hostssl with replication/replicationssl rather than overload the database field? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Hot standby documentation
Having concluded I really need to start playing with hot standby, I started looking for documentation on the subject. I found what I was looking for; I also found this page[1], which, it seems, ought to mention hot standby. Comments? [1] http://developer.postgresql.org/pgdocs/postgres/high-availability.html -- Joshua Tolley / eggyknap End Point Corporation http://www.endpoint.com signature.asc Description: Digital signature
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 5:44 PM, Magnus Hagander mag...@hagander.net wrote: Such information are supplied in the parameter 'primary_conninfo' of recovery.conf. For example; primary_conninfo = 'host=192.168.1.50 port=5432 user=foo' So the password can just go there, no? Yeah, the password can be supplied in primary_conninfo. primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass' Also you can use the environment variable PGPASSWORD. Is this enough? I was thinking that some people would complain that only the password for replication cannot be supplied in .pgpass. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] true serializability and predicate locking
Kevin Grittner wrote: Another interesting thing which crossed my mind (and I should probably add a section for such things in the wiki) is that, given the overhead and conflict implications of using table scans in serializable transactions, we should perhaps try to discourage table scans from being chosen for those using serializable transactions. I figure we can probably fudge this to a workable degree by adjusting tuple cost GUCs, but if you wanted to think about this issue in more depth, it might be useful. I don't know if that's a good idea. It's an attempt to guess what the user really wanted, and one reason why I dislike Microsoft is that their software does exactly that. Messing with the optimizer might result in bad plans, much to the surprise of the user who changed nothing. What have you won if you take out fewer locks, but your query runs forever? I'd opt for good documentation that tells you about the pitfalls of this serializable implementation and counsels you. That also helps to keep it simple. Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 5:46 PM, Magnus Hagander mag...@hagander.net wrote: However, wouldn't it make more logical sense to replace host/hostssl with replication/replicationssl rather than overload the database field? Seems good. How about the following formats? replication user CIDR-address auth-method [auth-options] replicationssl user CIDR-address auth-method [auth-options] replication user IP-address IP-mask auth-method [auth-options] replicationssl user IP-address IP-mask auth-method [auth-options] Note that database field has been removed since it's useless for replication. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Serializable Isolation without blocking
2010/1/7 Markus Wanner mar...@bluegap.ch: (It's interesting that with database page level granularity, he states that predicate locking would not be necessary. Instead any page can be locked at any time. For this to work, according to my reasoning, you'd have to know in advance on which page potentially accessible (but not yet visible) new tuples would end up. This is close to impossible for Postgres, however, it seems to work for Berkeley DB). The specifics of relation databases can be entirely ignored in case serializability is provided on the page layer level. This also implies that pages that are part of indexes and such must be treated in exactly the same way as the pages that constitute the heap. Predicate locking is only needed to provide the same semantics in a context where the page layer doesn't provide serializability, but other layers (that understand that we are talking about a relational database) are supposed to provide it. I am not suggesting that Postgres should start supporting page layer level concurrency control, but rather indicating that most literature should be interpreted this way: concurrency control can be viewed as orthogonal to relational databases, so that is the way it was originally modeled. As this seems to be an optimization of predicate locking, don't we need to implement that first? Whole-table locking is a trivial implementation of predicate locking. Nicolas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] true serializability and predicate locking
Robert Haas wrote:
Jeff Davis wrote:
I have a question regarding true serializability and predicate locking.
There's some context on the wiki page:
If you have the following DDL:
create table mytable(mycircle circle);
create index mytable_mycircle_idx on mytable
using gist (mycircle);
and two transactions:
T1:
BEGIN;
SELECT * FROM mytable WHERE mycircle '(0, 0), 10';
-- if any rows are returned, ROLLBACK
INSERT INTO mytable(mycircle) VALUES('(0, 0), 10');
COMMIT;
T2:
BEGIN;
SELECT * FROM mytable WHERE mycircle '(5, 5), 5';
-- if any rows are returned, ROLLBACK
INSERT INTO mytable(mycircle) VALUES('(5, 5), 5');
COMMIT;
Clearly one of those transactions should abort, because that will happen
in either serialized order. But I don't see where any lock is stored,
nor how the conflict is detected.
There has been a lot of theoretical discussion on this matter, but I'd
like to know how it will work in this specific case. You can't merely
lock a few index pages, because the INSERT might put the tuple in
another page.
I'm still trying to catch up on this discussion as well as relevant
papers, but this question has been on my mind.
One approach that might work for GiST is to get some kind of lock
(SIREAD?) on the predicates for the pages that the search does not
match. That way, the conflict can be detected if an INSERT tries to
update the predicate of a page to something that the search may have
matched.
If the index was GIN instead of GiST, I think the fastupdate feature
would cause a problem, though (as Greg brought up). Fastupdate may need
to be disabled when using truly serializable transactions.
It seems to me that we shouldn't pre-judge too much about how
predicate locking will ultimately be implemented. Suppose the first
query in your example were:
SELECT * FROM mytable WHERE [some incredibly complex condition
involving all sorts of strange magic]
It seems to me that in a case like this our only realistic option is
to lock the entire table until T1 commits.
I don't know if such a thing would be easy to implement in
PostgreSQL, but I had thought that the standard approach to
implement predicate locking is like this:
Whenever you touch (read) a data structure, you tag it with a lock
that prevents anybody else from modifying the data structure in a way
that would change your result if you perform the same operation again
(or with SIREAD locks, it will not prevent the modification, but
may lead to aborted transactions later).
So no matter how complex the index condition is, it will boil down
to accessing and hence locking certain parts of a table or index
(in the case of a B*-tree, you'll probably end up locking gaps in
the index).
I'd say that the index should know what exactly should be locked
if a certain operation must become repeatable.
Yours,
Laurenz Albe
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby documentation
On Thu, Jan 7, 2010 at 6:09 PM, Joshua Tolley eggyk...@gmail.com wrote: Having concluded I really need to start playing with hot standby, I started looking for documentation on the subject. I found what I was looking for; I also found this page[1], which, it seems, ought to mention hot standby. Comments? [1] http://developer.postgresql.org/pgdocs/postgres/high-availability.html +1 At least, it should be mentioned that the slave can answer read-only queries in Warm Standby Using Point-In-Time Recovery. And so Table 25-1 should be changed. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 7/01/2010 9:15 AM, Robert Haas wrote: On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: David E. Wheelerda...@kineticode.com wrote: Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections (perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg, neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? At this point, I'm personally wondering if it's worth putting together a simple (ish) C program that reads a file describing command interleavings on n connections. It fires up one thread per connection required, then begins queuing commands up for the threads to execute in per-thread fifo queues. The input file may contain synchronization points where two or more explicitly specified threads (or just all threads) must finish all their queued work before they may be given more. Yes, it requires wrangling low-level threading ( pthreads, or the practically equivalent for simple purposes but differently spelled win32 threading ) so it's not going to be beautiful. But it'd permit a declarative form for tests and a single, probably fairly maintainable, test runner. I reluctantly suspect that XML would be a good way to describe the tests - first a block declaring your connections and their conn strings, then a sequence of statements (each of which is associated with a named connection) and synchronization points. Though, come to think of it, a custom plaintext format would be pretty trivial too. CONN conn1: dbname=regress, user=regress CONN conn2: dbname=regress, user=regress STMT conn1: SELECT blah blah; STMT conn2: UPDATE blah blah; SYNC conn1, conn2 etc. Or alternately one-file-per-connection (which would be handy if one connection has *lots* of commands and others only occasional ones) - the only trouble there being how to conveniently specify synchronization points. Anyway: If Java were acceptable I'd put one together now - but somehow I don't think requiring Java would be popular if Perl is an issue ;-) My C/pthreads is more than a little bit rusty (ie: practially nonexistent) and mostly confined to exception-controlled C++ code with RAII for lock management. If C++ is OK, I can write and post a tool for evaluation, but if it must be plain C ... well, I'll avoid scarring you with the sight of what I'd produce. I suspect that a custom tool for this job could actually be fairly simple. A lot simpler than a proper, clean and usable parallel psql, anyway. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] true serializability and predicate locking
2010/1/7 Albe Laurenz laurenz.a...@wien.gv.at: I don't know if such a thing would be easy to implement in PostgreSQL, but I had thought that the standard approach to implement predicate locking is like this: Whenever you touch (read) a data structure, you tag it with a lock that prevents anybody else from modifying the data structure in a way that would change your result if you perform the same operation again (or with SIREAD locks, it will not prevent the modification, but may lead to aborted transactions later). So no matter how complex the index condition is, it will boil down to accessing and hence locking certain parts of a table or index (in the case of a B*-tree, you'll probably end up locking gaps in the index). That would be an access layer based version of predicate locking (of which a typical implementation is the already-mentioned next-key locking). The most pure version of predicate locking literally locks predicates (i.e., conditions on rows). Conflicts are detected by checking for overlap between predicates: is there a (possibly non-existent) row that matches the same predicate? If so, and the locks are incompatible, then a conflict arises (this would be different in the SIREAD case, of course; I am talking about traditional 2PL + predicate locking). In such a pure implementation of predicate locking, the overlap testing is be done using the algebraic properties of the conditions, which is of course extremely difficult (if not impossible) to implement perfectly in a system that allows conditions of arbitrary complexity. Therefore, the conditions are typically simplified in such a way that they become true for more rows, but are easier to check for overlap. Nicolas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] true serializability and predicate locking
Nicolas Barbier wrote: I don't know if such a thing would be easy to implement in PostgreSQL, but I had thought that the standard approach to implement predicate locking is like this: Whenever you touch (read) a data structure, you tag it with a lock that prevents anybody else from modifying the data structure in a way that would change your result if you perform the same operation again (or with SIREAD locks, it will not prevent the modification, but may lead to aborted transactions later). So no matter how complex the index condition is, it will boil down to accessing and hence locking certain parts of a table or index (in the case of a B*-tree, you'll probably end up locking gaps in the index). That would be an access layer based version of predicate locking (of which a typical implementation is the already-mentioned next-key locking). The most pure version of predicate locking literally locks predicates (i.e., conditions on rows). Conflicts are detected by checking for overlap between predicates: is there a (possibly non-existent) row that matches the same predicate? If so, and the locks are incompatible, then a conflict arises (this would be different in the SIREAD case, of course; I am talking about traditional 2PL + predicate locking). In such a pure implementation of predicate locking, the overlap testing is be done using the algebraic properties of the conditions, which is of course extremely difficult (if not impossible) to implement perfectly in a system that allows conditions of arbitrary complexity. Therefore, the conditions are typically simplified in such a way that they become true for more rows, but are easier to check for overlap. That sounds like major AI to me. What do you do if the condition involves user defined functions? Are there any implementations taking such an approach? Yours, Laurenz Albe -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unresolved bugs
On Wed, Jan 06, 2010 at 03:36:39PM -0500, Tom Lane wrote: BUG #5236: Aparent bug in ecpg http://archives.postgresql.org/pgsql-bugs/2009-12/msg00078.php Michael needs to look at that one. I'm waiting for a reproducable test case. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org ICQ: 179140304, AIM/Yahoo/Skype: michaelmeskes, Jabber: mes...@jabber.org VfL Borussia! Forca Barca! Go SF 49ers! Use: Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq naming on Win64
On Thu, Jan 7, 2010 at 3:58 AM, Hiroshi Inoue in...@tpf.co.jp wrote: Maybe I'm missing the point and have a question. For example, do 32bit psql and the 64bit one have the same name? If so, where will they be installed? I'm only talking about libpq. I see no reason to have 32 64 bit versions of other utilities installed in parallel. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug with PATHs having non-ASCII characters
On Thu, Jan 7, 2010 at 02:37, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Chuck McDevitt cmcdev...@greenplum.com wrote: Just an FYI regarding this bug: http://archives.postgresql.org/pgsql-bugs/2009-12/msg00267.php The wide-char version of any WIN32 API call will accept or return data in UTF-16 encoded Unicode, regardless of the local environment's single-byte (MBCS) encoding settings (codepage). I have a Windows-specific patch for open(), attached for reference. But we need to consider about other issues: - We need to consider about not only only open(), but also opendir(), stat() and symlink(). - An entirely-different fix is needed for non-Windows platforms. Probably we will convert encodings from GetDatabaseEncoding() to GetPlatformEncoding() in MBCS, but this is not needed on Windows. We should consider avoiding random ifdef blocks for the switching. Shouldn't we develop this with multi-platform in mind from the start, instead of doing a Windows specific patch? It may be that we end up with two completely different codepaths, but more likely we can share some of it between them? - Those conversions are not free. We might need to avoid conversions for paths under $PGDATA because we only use ascii names in the server. I used a test with IS_HIGHBIT_SET in the attached patch, but I'm not sure whether it is the best method. If we're going to end up with our own wrapper anyway, we can just pass an extra parameter to it saying if we want conversion or not? That way we can avoid doing it for cases where we know it's safe, but do it when user-input is included? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing plperl-plperlu interaction
On Wed, Jan 06, 2010 at 07:07:12PM -0500, Andrew Dunstan wrote: Alvaro Herrera wrote: decibel wrote: We've actually run into similar issues. Alvaro came up with a patch that fixes our specific issue, but I think he said there were some other cases that needed to be fixed as well. Anyone looking to fix this should ping Alvaro first. No, what I fixed was the contrib/xml2 crasher that I still haven't submitted here (sorry). The plperl fix came from Alexey Klyukin, and AFAIK the fix is in 8.4.2. Yes. [thinks] Could we have a regression test for it with an alternative result file for the case where plperl and plperlu are not allowed to run together? Or perhaps put the tests that require multiplicity into a plperl_multi.sql file and add logic to the GNUmakefile to add that file to REGRESS if perl -V:usemultiplicity returns usemultiplicity='define'; Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Github mirror
By popular request, I've set up a job that will push a mirror of the master branch of our git repository (git.postgresql.og/git/postgresql.git) to github. The main reason is visibility, and the ability for github folks to work with their tools. (Trivial job, literally two lines in an existing shellscript) I've disabled all additional features like wikis and issue trackers. I've also set up an autoresponder so that if anybody tries to use the github pull request method of submitting a patch they're told to send their patch to pgsql-hackers. You can find the repo at http://github.com/postgres/postgres. Note that if you use githubs fork feature into your own account to work on something, the PostgreSQL repository is big enough to use up more than half the space given out to free/opensource github accounts (currently about 200Mb (which shows up as 0.1Gb though), account space is 300Mb). But given some of thee nice browsing features at github, some people may find it worth trying out anyway. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Status of plperl inter-sp calling
On Wed, Jan 06, 2010 at 08:46:11PM -0500, Tom Lane wrote: Tim Bunce tim.bu...@pobox.com writes: On Wed, Jan 06, 2010 at 01:45:45PM -0800, David E. Wheeler wrote: On Jan 6, 2010, at 12:20 PM, Tom Lane wrote: One of the things on my to-do list for today is to make configure reject Perl versions less than $TBD. I thought we had agreed a week or so back that 5.8 was the lowest safe version because of utf8 and other considerations. +1, and 5.8.3 at a minimum for utf8 stuff, 5.8.8 much much better. I think we said 5.8.1 at the time, but 5.8.3 sounds good to me. There would be _very_ few places using 5.8.6. I went with 5.8 as the cutoff, for a couple of reasons: we're not in the business of telling people they ought to be up-to-date, but only of rejecting versions that demonstrably fail badly; I think 5.8.0 will fail badly, possibly demonstrably but more likely in subtle ways relating to utf8 that are hard to debug. and I found out that older versions of awk are not sufficiently competent with and || to code a more complex test properly :-(. A version check that doesn't actually do what it claims to is worse than useless, and old buggy awk is exactly what you'd expect to find on a box with old buggy perl. Either of these approaches should work back to perl 5.0... perl -we 'use 5.008001' 2/dev/null echo ok or perl -we 'exit($] 5.008001)' echo ok (It's also worth noting that the perl version seen at configure time is not necessarily that seen at runtime, anyway, so there's not a lot of point in getting too finicky here.) A simple use 5.008001; at the start of src/pl/plperl/plc_perlboot.pl would address that. I believe Andrew is planing to commit my plperl refactor patch soon. He could add it then, or I could add it to my feature patch (which I plan to reissue soon, with very minor changes, and add to commitfest). Tim. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] memory context debugging
On Wed, Jan 6, 2010 at 11:14 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, Jan 6, 2010 at 10:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: What tools do we have for identifying memory leaks? User complaints :-( YGTBFKM. Not really. Given the memory context architecture, leaks are simply not a big deal in 99% of the system. We just need a few coding rules like don't run random code in CacheMemoryContext ;-) It seems like we should have a tool that dumps out every memory context in the system, with the number of allocations and frees and number of bytes allocated and freed since the last reset. Maybe the time of the last reset. You could run that before and after doing whatever it is that might leak and compare. Once you've identified a place that might leak and a test case that would exercise it, you've already done most of the work. What you're describing sounds to me like a lot of work for not much return. Furthermore, if you do have a leaking test case and you don't know exactly where the leak is coming from, numbers about how big the leak is aren't any help in finding the cause. What you really want is numbers that are per palloc call site, which would not be simple to get. I have occasionally wondered about hooking up something similar to valgrind for this; but the problem is that it would drown you in false positives because of the number of places where we intentionally leave stuff to be cleaned up at context reset. About 10 years ago I worked on a C++ project and they had a tool, whose details I no longer remember, that dumped out memory allocation data and it was an invaluable debugging tool not only for detecting leaks but also for figuring out which parts of the code were memory-intensive. With what we have today, it sounds like you can't even do something like run the regression tests and then check whether anything leaked into a context that doesn't get reset, which IMO ought to be routine testing. It's true that detecting leaks into statement or tuple level contexts is probably a little more challenging because of the reliance on context resets, but without a tool it's REALLY hard. Saying that once you've identified a place that might leak and a test case you've already done most of the work does not seem true to me. What is the next step, at that point? Visual inspection of the code? Even for someone who knows the code inside out, that's only feasible if you're pretty sure that there is a leak there. If you just want to test for leaks, it's a poor way to do it. And if you aren't familiar with every detail of the code (such as, ahem, the points where cache invalidations can happen) then it's even harder. Getting information by palloc call site doesn't see all that difficult, actually, though it would require some rejiggering of our macros. Basically you just need to set things up so that when memory-context debugging is enabled, the actual allocator (MemoryContextAlloc in our case, I think) gets __FILE__ and __LINE__ as arguments. Then you just make some sort of very simple data structure (like an array of structs) where you record data for each new combination that comes in. This would be associated with the context, not global, so that you can clean them up easily when the context is reset. You also need the same thing for free. Then you write a function that prints out the contents of the array; it's useful to sort it by bytes allocated. So then if you want to see if you have any statement-lifetime memory leaks, for example, you dump the per-statement context just before it gets reset and look at how many bytes/allocations you have from each call site. And then you say... wait a minute, that call site shouldn't have been allocating in this context... or wow, i didn't realize that got so big... or whatever the case may be. It doesn't remove the need for manual analysis, but it gives you a big jump on where to start analyzing. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Serializable Isolation without blocking
Hi, Kevin Grittner wrote: I'm probably not quite as clueless as you think on this; I realize that keeping SIREAD locks in memory will require many more slots for locks, escalation from tuple level to page or coarser when there are many on a table, or (most likely) both. ..oh, there's the dynamic adjustment idea again. Cool! What I do think is that the structures for regular locks seem usable to track the information I need without having to burden read operations with disk output, which I see as absolutely necessary for adequate performance. I certainly agree that SIREAD locks should never hit the disk. However, thinking of SIREAD more as marks or hints, I'm not so sure the current locking structures are appropriate. If we are talking about table level locks, those are not fine grained enough and have different semantics (i.e. a user can acquire these locks, which certainly doesn't make any sense for SIREAD). My gut feeling is that adjusting them for use with SIREAD is more work than implementing your own shared memory area for SIREAD marks. Row level locks are very fine grained, but those are spilled to disk in its current implementation. So those are an even worse fit for the needs of SIREAD. It also gives me somewhere to store locks related to search ranges where data doesn't currently exist, and the ability to store read locks from many transactions against the same data. Isn't a hash table in shared memory enough to store the SIREAD locks (and a lot simpler to manage)? An open question in my mind is whether I might need to keep write locks for serializable transactions in the shared memory lock hash table until commit or rollback IIUC row level locks are kept in the tuple header until the end of the transaction (and even beyond). Is this really a problem? I suppose these more persistent write locks should be kept out of the DEFAULT lock method, too I fail to understand that part. What's the DEFAULT lock method? Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Patch: Allow substring/replace() to get/set bit values
Hi all,
attached a patch that adds the following functions for bit string:
- overlay
- get_bit
- set_bit
Some info:
1) overlay is implemented as calls to substring; given the different way
substring behaves when used with strings vs bit strings:
test=# SELECT substring(B'0001' from 1 for -1);
substring
--
0001
(1 row)
test=# SELECT substring('0001' from 1 for -1);
ERROR: negative substring length not allowed
I don't think that this overlay implementation is what we want?
Example:
test=# SELECT overlay(B'' placing B'01' from 1 for 2);
overlay
-
0111
(1 row)
(looks ok)
test=# SELECT overlay(B'' placing B'01' from 0 for 2);
overlay
---
0
(1 row)
This happens because substring(bit, pos, -1) means substring(bit, pos,
length(bit string)),
and -1 values for bit substring parameters are allowed: is this a bug in bit
substring???
2) I tried implementing bit_get and bit_set as calls to overlay/substring:
DATA(insert OID = 3032 ( get_bit PGNSP PGUID 14 1 0 0 f f f t
f i 2 0 23 1560 23 _null_ _null_ _null_ _null_ select
(pg_catalog.substring($1, $2, 1))::int4 _null_ _null_ _null_ ));
DESCR(get bit);
DATA(insert OID = 3033 ( set_bit PGNSP PGUID 14 1 0 0 f f f t
f i 3 0 1560 1560 23 23 _null_ _null_ _null_ _null_ select
pg_catalog.overlay($1, $3::bit, $2, 1) _null_ _null_ _null_ ));
DESCR(set bit);
but this doesn't give any check on the values provided:
that the bit looked for is in fact in the right range, and that the bit in
set_bit is in fact a bit
(I don't like the idea of writing select set_bit(B'01010111', B'1') instead
of
select set_bit(B'01010111', 1) ).
So I coded them in proper C internal functions.
Leonardo
patch_bit.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On m??n, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote: Yea, I am not excited about having vacuumdb do only analyze, but it seems the most minimal solution. I spelled it --only-analyze and just posted the reason and patch. I can't find the patch and the reason, but note that we already have other options like --data-only, --schema-only, --tuples-only. I personally don't like the spelling of --only-analyze. In particular note that pg_dump has options --schema and --schema-only, and nobody has complained about that. I concur with Peter that this spelling is gratuitously unlike everyplace else. Oh, interesting about pg_dump. Let's just go with --analyze-only. --only-analyze is feeling odd to me too. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq naming on Win64
Dave Page wrote: On Thu, Jan 7, 2010 at 3:58 AM, Hiroshi Inoue in...@tpf.co.jp wrote: Maybe I'm missing the point and have a question. For example, do 32bit psql and the 64bit one have the same name? If so, where will they be installed? I'm only talking about libpq. I see no reason to have 32 64 bit versions of other utilities installed in parallel. Well what about the libraries (e.g openssl) libpq links? regards, Hiroshi Inoue -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq naming on Win64
On Thu, Jan 7, 2010 at 12:21 PM, Hiroshi Inoue in...@tpf.co.jp wrote: Dave Page wrote: On Thu, Jan 7, 2010 at 3:58 AM, Hiroshi Inoue in...@tpf.co.jp wrote: Maybe I'm missing the point and have a question. For example, do 32bit psql and the 64bit one have the same name? If so, where will they be installed? I'm only talking about libpq. I see no reason to have 32 64 bit versions of other utilities installed in parallel. Well what about the libraries (e.g openssl) libpq links? Yes, that is an issue as I noted earlier. We probably would still have to put the 32 bit and 64 bit libraries in a different directory because of their naming conventions. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
Fujii Masao wrote: On Thu, Jan 7, 2010 at 5:44 PM, Magnus Hagander mag...@hagander.net wrote: Such information are supplied in the parameter 'primary_conninfo' of recovery.conf. For example; primary_conninfo = 'host=192.168.1.50 port=5432 user=foo' So the password can just go there, no? Yeah, the password can be supplied in primary_conninfo. primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass' Also you can use the environment variable PGPASSWORD. Is this enough? I was thinking that some people would complain that only the password for replication cannot be supplied in .pgpass. That seems enough to me. BTW, how do you set up authentication using an SSL certificate? ISTM that's the way we should be encouraging people to configure authentication between a master and standby, rather than type a password to a file. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] advantage of new vacuum
Hello can somebody explain advantages of new vacuum? Thank you Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
Tom Lane wrote: Fujii Masao masao.fu...@gmail.com writes: The attached patch supports new keyword 'replication' on .pgpass file. This keyword is used to specify the password for the standby server to connect to the primary server. This strikes me as a completely bad idea. We need get no farther than the point that it assumes nobody can have a database named replication (although I notice the patch also appears to assume that libpq knows internally that the connection is for replication --- I thought we were going to avoid libpq changes for SR?) As the patch stands there's an extra startup packet, similar to SSLRequest, in the connection handshake. But I think we should get rid of that too, and just send an extra replication=on option along with username and other options in the startup packet. (as Fujii pointed out already, you can have a database called replication. But you need to quote it in pg_hba.conf, like you need to quote all and sameuser/group/role.) -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: On mC3A5n, 2010-01-04 at 13:07 -0500, Bruce Momjian wrote: Yea, I am not excited about having vacuumdb do only analyze, but it seems the most minimal solution. I spelled it --only-analyze and just posted the reason and patch. I can't find the patch and the reason, but note that we already have other options like --data-only, --schema-only, --tuples-only. I personally don't like the spelling of --only-analyze. In particular note that pg_dump has options --schema and --schema-only, and nobody has complained about that. I concur with Peter that this spelling is gratuitously unlike everyplace else. Oh, interesting about pg_dump. Let's just go with --analyze-only. --only-analyze is feeling odd to me too. Done, attached and applied. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. + Index: doc/src/sgml/ref/vacuumdb.sgml === RCS file: /cvsroot/pgsql/doc/src/sgml/ref/vacuumdb.sgml,v retrieving revision 1.47 diff -c -c -r1.47 vacuumdb.sgml *** doc/src/sgml/ref/vacuumdb.sgml 6 Jan 2010 05:31:13 - 1.47 --- doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 12:37:08 - *** *** 28,34 grouparg--freeze/argarg-F/arg/group grouparg--verbose/argarg-v/arg/group grouparg--analyze/argarg-z/arg/group !grouparg--only-analyze/argarg-o/arg/group arg--table | -t replaceabletable/replaceable arg( replaceable class=parametercolumn/replaceable [,...] )/arg /arg --- 28,34 grouparg--freeze/argarg-F/arg/group grouparg--verbose/argarg-v/arg/group grouparg--analyze/argarg-z/arg/group !grouparg--analyze-only/argarg-o/arg/group arg--table | -t replaceabletable/replaceable arg( replaceable class=parametercolumn/replaceable [,...] )/arg /arg *** *** 42,48 grouparg--freeze/argarg-F/arg/group grouparg--verbose/argarg-v/arg/group grouparg--analyze/argarg-z/arg/group !grouparg--only-analyze/argarg-o/arg/group /cmdsynopsis /refsynopsisdiv --- 42,48 grouparg--freeze/argarg-F/arg/group grouparg--verbose/argarg-v/arg/group grouparg--analyze/argarg-z/arg/group !grouparg--analyze-only/argarg-o/arg/group /cmdsynopsis /refsynopsisdiv *** *** 143,149 varlistentry termoption-o/option/term ! termoption--only-analyze/option/term listitem para Only calculate statistics for use by the optimizer (no vacuum). --- 143,149 varlistentry termoption-o/option/term ! termoption--analyze-only/option/term listitem para Only calculate statistics for use by the optimizer (no vacuum). *** *** 168,174 para Clean or analyze replaceable class=parametertable/replaceable only. Column names can be specified only in conjunction with ! the option--analyze/option or option--only-analyze/option options. /para tip para --- 168,174 para Clean or analyze replaceable class=parametertable/replaceable only. Column names can be specified only in conjunction with ! the option--analyze/option or option--analyze-only/option options. /para tip para Index: src/bin/scripts/vacuumdb.c === RCS file: /cvsroot/pgsql/src/bin/scripts/vacuumdb.c,v retrieving revision 1.31 diff -c -c -r1.31 vacuumdb.c *** src/bin/scripts/vacuumdb.c 6 Jan 2010 16:04:05 - 1.31 --- src/bin/scripts/vacuumdb.c 7 Jan 2010 12:37:08 - *** *** 15,26 static void vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose, ! bool and_analyze, bool only_analyze, bool freeze, const char *table, const char *host, const char *port, const char *username, enum trivalue prompt_password, const char *progname, bool echo); static void vacuum_all_databases(bool full, bool inplace, bool verbose, bool and_analyze, ! bool only_analyze, bool freeze, const char *host, const char *port, const char *username, enum trivalue prompt_password, const char *progname, bool echo, bool quiet); --- 15,26 static void vacuum_one_database(const char *dbname, bool full, bool inplace, bool verbose, ! bool and_analyze, bool analyze_only, bool freeze, const char *table, const char *host, const char *port, const char *username, enum trivalue prompt_password, const char *progname, bool echo); static void vacuum_all_databases(bool full, bool inplace, bool verbose, bool and_analyze, ! bool analyze_only, bool
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
Magnus Hagander wrote: However, wouldn't it make more logical sense to replace host/hostssl with replication/replicationssl rather than overload the database field? It makes more sense to me to overload the database field. When you connect for replication, you're not connecting to any particular database, but to a special walsender mode. Note that 'local' makes sense for replication too; you can connect for replication via a Unix-domain socket. Replication isn't a connection method like host/hostssl/local are. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 10:21, Fujii Masao masao.fu...@gmail.com wrote: On Thu, Jan 7, 2010 at 5:46 PM, Magnus Hagander mag...@hagander.net wrote: However, wouldn't it make more logical sense to replace host/hostssl with replication/replicationssl rather than overload the database field? Seems good. How about the following formats? replication user CIDR-address auth-method [auth-options] replicationssl user CIDR-address auth-method [auth-options] replication user IP-address IP-mask auth-method [auth-options] replicationssl user IP-address IP-mask auth-method [auth-options] Note that database field has been removed since it's useless for replication. Hm, no, I think I withdraw my comment about pg_hba.conf. It seems better to overload the database name here. I'm not particularly keen on yet another different set of columns, which is what happens when you remove the database field. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
On Thu, Jan 7, 2010 at 13:34, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: On Thu, Jan 7, 2010 at 5:44 PM, Magnus Hagander mag...@hagander.net wrote: Such information are supplied in the parameter 'primary_conninfo' of recovery.conf. For example; primary_conninfo = 'host=192.168.1.50 port=5432 user=foo' So the password can just go there, no? Yeah, the password can be supplied in primary_conninfo. primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass' Also you can use the environment variable PGPASSWORD. Is this enough? I was thinking that some people would complain that only the password for replication cannot be supplied in .pgpass. That seems enough to me. BTW, how do you set up authentication using an SSL certificate? ISTM that's the way we should be encouraging people to configure authentication between a master and standby, rather than type a password to a file. Connectoin parameters: sslcert, sslkey and sslrootcert in most cases. Or just put the key in ~/.postgresql/postgresql.key. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Serializable Isolation without blocking
Hi, Nicolas Barbier wrote: The specifics of relation databases can be entirely ignored in case serializability is provided on the page layer level. Aha, I now see very vaguely how that could work, yes. Thank you for elaborating on this. I agree that this isn't the best way forward for Postgres. As this seems to be an optimization of predicate locking, don't we need to implement that first? Whole-table locking is a trivial implementation of predicate locking. ..and whole-database locking is a trivial implementation of true serializability. In a way, both are optimizations of that trivial implementation. My point is that due to that dependency, the conceptual design of a solution for predicate locking (with acceptable performance) should at least be considered before going into details with true serializability. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unresolved bugs
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: *sigh* - that was mostly ment as a joke and not a really serious comment. However the idea I actually had with BZ back in the days was not to use it as a full fledged tracker(in the sense of exposing it to users or developers) Instead I would just use it as the background engine that does nothing more than being subscribed to -bugs, tracks the stuff there and provides an summary export about (not)replied to reports. If somebody later on wants to annotate the emails/reports there (as in solved,open,moved to todo,not a pg core question,whatever) fine - if not fine as well :) bugzilla doesn't really interface to email well enough to do that. I gather that debbugs might work better, but I have no personal experience with it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, Dec 31, 2009 at 6:40 PM, Simon Riggs si...@2ndquadrant.com wrote:
Building racy infrastructure when it can be avoided with a little care
still seems not to be the best path to me.
Doing that will add more complexity in an area that is hard to test
effectively. I think the risk of introducing further bugs while trying
to fix this rare condition is high. Right now the conflict processing
needs more work and is often much less precise than this, so improving
this aspect of it would not be a priority. I've added it to the TODO
though. Thank you for your research.
Patch implements recovery conflict signalling using SIGUSR1
multiplexing, then uses a SessionCancelPending mode similar to Joachim's
TransactionCancelPending.
I have reworked Simon's patch a bit and attach the result.
Quick facts:
- Hot Standby only uses SIGUSR1
- SIGINT behaves as it did before: it only cancels running statements
- pg_cancel_backend() continues to use SIGINT
- I added pg_cancel_idle_transaction() to cancel an idle transaction via
SIGUSR1
- One central function HandleCancelAction() sets the flags before calling
ProcessInterrupts(), it is called from the different signal handlers and
receives parameters about what it should do
- If a SIGUSR1 reason is used that will cancel something, ProcArrayLock is
acquired until the signal has been sent to make sure that we won't signal the
wrong backend. Does this sufficiently cover the concerns of Andres Freund
discussed upthread?
As there were so many boolean SomethingCancelPending variables I changed them
to be bitmasks and merged all of them into a single variable. However I am not
sure if we can change the name and type of especially InterruptPending that
easily as it was marked with PGDLLIMPORT...
@Simon: Is there a reason why you have not yet removed recoveryConflictMode
from PGPROC?
Joachim
diff -cr cvs/src/backend/access/transam/xact.c cvs.build/src/backend/access/transam/xact.c
*** cvs/src/backend/access/transam/xact.c 2009-12-24 13:55:12.0 +0100
--- cvs.build/src/backend/access/transam/xact.c 2010-01-05 11:25:17.0 +0100
***
*** 313,320
/*
* IsAbortedTransactionBlockState
*
! * This returns true if we are currently running a query
! * within an aborted transaction block.
*/
bool
IsAbortedTransactionBlockState(void)
--- 313,319
/*
* IsAbortedTransactionBlockState
*
! * This returns true if we are within an aborted transaction block.
*/
bool
IsAbortedTransactionBlockState(void)
***
*** 2692,2697
--- 2691,2738
}
/*
+ * AbortTransactionAndAnySubtransactions
+ *
+ * Similar to AbortCurrentTransaction but if any subtransactions
+ * in progress we abort them and all of their parents. So this is
+ * used when the caller wishes to make the abort untrappable by the user.
+ */
+ void
+ AbortTransactionAndAnySubtransactions(void)
+ {
+ TransactionState s = CurrentTransactionState;
+
+ switch (s-blockState)
+ {
+ case TBLOCK_DEFAULT:
+ case TBLOCK_STARTED:
+ case TBLOCK_BEGIN:
+ case TBLOCK_INPROGRESS:
+ case TBLOCK_END:
+ case TBLOCK_ABORT:
+ case TBLOCK_SUBABORT:
+ case TBLOCK_ABORT_END:
+ case TBLOCK_ABORT_PENDING:
+ case TBLOCK_PREPARE:
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ AbortCurrentTransaction();
+ break;
+
+ case TBLOCK_SUBINPROGRESS:
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBEND:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ AbortTransactionAndAnySubtransactions();
+ break;
+ }
+ }
+
+
+ /*
* PreventTransactionChain
*
* This routine is to be called by statements that must not run inside
diff -cr cvs/src/backend/commands/vacuum.c cvs.build/src/backend/commands/vacuum.c
*** cvs/src/backend/commands/vacuum.c 2009-12-24 13:55:13.0 +0100
--- cvs.build/src/backend/commands/vacuum.c 2010-01-03 20:24:26.0 +0100
***
*** 3936,3942
CHECK_FOR_INTERRUPTS();
/* Nap if appropriate */
! if (VacuumCostActive !InterruptPending
VacuumCostBalance = VacuumCostLimit)
{
int msec;
--- 3936,3942
CHECK_FOR_INTERRUPTS();
/* Nap if appropriate */
! if (VacuumCostActive !IsInterruptEventPending
VacuumCostBalance = VacuumCostLimit)
{
int msec;
diff -cr cvs/src/backend/postmaster/autovacuum.c cvs.build/src/backend/postmaster/autovacuum.c
*** cvs/src/backend/postmaster/autovacuum.c 2009-12-09 11:24:41.0 +0100
--- cvs.build/src/backend/postmaster/autovacuum.c 2010-01-07 00:22:21.0 +0100
***
*** 479,488
/* Prevents interrupts while cleaning up */
HOLD_INTERRUPTS();
! /* Forget any pending QueryCancel request */
! QueryCancelPending = false;
disable_sig_alarm(true);
! QueryCancelPending = false; /* again in case timeout occurred */
/* Report the error to the server log */
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote: On Thu, Dec 31, 2009 at 6:40 PM, Simon Riggs si...@2ndquadrant.com wrote: Building racy infrastructure when it can be avoided with a little care still seems not to be the best path to me. Doing that will add more complexity in an area that is hard to test effectively. I think the risk of introducing further bugs while trying to fix this rare condition is high. Right now the conflict processing needs more work and is often much less precise than this, so improving this aspect of it would not be a priority. I've added it to the TODO though. Thank you for your research. Patch implements recovery conflict signalling using SIGUSR1 multiplexing, then uses a SessionCancelPending mode similar to Joachim's TransactionCancelPending. I have reworked Simon's patch a bit and attach the result. Oh dear, this is exactly what I've been working on... Quick facts: - Hot Standby only uses SIGUSR1 - SIGINT behaves as it did before: it only cancels running statements - pg_cancel_backend() continues to use SIGINT - I added pg_cancel_idle_transaction() to cancel an idle transaction via SIGUSR1 - One central function HandleCancelAction() sets the flags before calling ProcessInterrupts(), it is called from the different signal handlers and receives parameters about what it should do - If a SIGUSR1 reason is used that will cancel something, ProcArrayLock is acquired until the signal has been sent to make sure that we won't signal the wrong backend. Does this sufficiently cover the concerns of Andres Freund discussed upthread? As there were so many boolean SomethingCancelPending variables I changed them to be bitmasks and merged all of them into a single variable. However I am not sure if we can change the name and type of especially InterruptPending that easily as it was marked with PGDLLIMPORT... @Simon: Is there a reason why you have not yet removed recoveryConflictMode from PGPROC? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unresolved bugs
Tom Lane wrote: bugzilla doesn't really interface to email well enough to do that. I gather that debbugs might work better, but I have no personal experience with it. 1. My recollection is that last time we looked the debbugs people themselves said they didn't think it was suitable for us. 2. IIRC there are at least some facilities (possibly using extra modules) for bugzilla - email integration. I haven't looked at it lately, however, so I don't know what the state of play is. I think Stefan got some good stuff working a couple of years ago. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator issues
Bruce Momjian escribió: Oh, interesting about pg_dump. Let's just go with --analyze-only. --only-analyze is feeling odd to me too. Done, attached and applied. bikeshedding Why -o and not, say, -Z? I imagine you picked -o for only but it seems strange. / -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] true serializability and predicate locking
2010/1/7 Albe Laurenz laurenz.a...@wien.gv.at: Nicolas Barbier wrote: In such a pure implementation of predicate locking, the overlap testing is be done using the algebraic properties of the conditions, which is of course extremely difficult (if not impossible) to implement perfectly in a system that allows conditions of arbitrary complexity. Therefore, the conditions are typically simplified in such a way that they become true for more rows, but are easier to check for overlap. That sounds like major AI to me. It is, but only if you want to approach perfection, which is probably not the way to go. What do you do if the condition involves user defined functions? Then you have to become conservative, I guess: if you don't know otherwise, you assume it *might* do the bad thing: the predicate might match any inputs (i.e., you could convert such a condition to a whole-table lock if there are no other restrictions ANDed to it). Are there any implementations taking such an approach? I personally don't know of any production-ready DBMSs that use any predicate locking approach other than the access layer based one (i.e., locking index ranges and falling back to whole-table locks for table scans); but then, I am not an expert in this matter. I think that it is a good thing in itself to remove plan dependencies (which would be accomplished by locking on the algebraic level), but it seems that none other the other DBMSs were able to implement it like that. Nicolas -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unresolved bugs
Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: *sigh* - that was mostly ment as a joke and not a really serious comment. However the idea I actually had with BZ back in the days was not to use it as a full fledged tracker(in the sense of exposing it to users or developers) Instead I would just use it as the background engine that does nothing more than being subscribed to -bugs, tracks the stuff there and provides an summary export about (not)replied to reports. If somebody later on wants to annotate the emails/reports there (as in solved,open,moved to todo,not a pg core question,whatever) fine - if not fine as well :) bugzilla doesn't really interface to email well enough to do that. I gather that debbugs might work better, but I have no personal experience with it. well recent bugzilla versions do have both an email interface (an early version of that was used in my prototype - the newer ones are much more powerful an can do) and an remote XML-RPC Interface (which I used for the bugform integration). As a pure email tracker with no real external apps debbugs is probably more powerful though (but not really used a lot outside of the debian project). Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-extending table partitions?
Adding on to this use case: what do we do when we reach end of year? Probably auto-archive as per weekly, monthly , quarterly or yearly tables? On Thu, Jan 7, 2010 at 1:14 AM, Josh Berkus j...@agliodbs.com wrote: On 1/6/10 9:13 AM, Robert Haas wrote: On Wed, Jan 6, 2010 at 12:06 PM, David Fetter da...@fetter.org wrote: On Tue, Jan 05, 2010 at 08:50:25PM -0700, u235sentinel wrote: Robert Haas wrote: Getting full? ...Robert Ok. Bad analogy. We have the tables setup to write data according to the month it was loaded. We have a December table, a January table and so on. Basically following the examples given on the 8.4 web site. FWIW, our roadmap is to add a 2nd type or partitioning which would be on the sub-table level and much more automated for that reason. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Chetan Sutrave http://www.enterprisedb.com
Re: [HACKERS] pg_migrator issues
Alvaro Herrera wrote:
Bruce Momjian escribi?:
Oh, interesting about pg_dump. Let's just go with --analyze-only.
--only-analyze is feeling odd to me too.
Done, attached and applied.
bikeshedding
Why -o and not, say, -Z? I imagine you picked -o for only but it
seems strange.
/
Hmmm, sure -Z makes sense. Change applied with attached patch. Thanks.
--
Bruce Momjian br...@momjian.ushttp://momjian.us
EnterpriseDB http://enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
Index: doc/src/sgml/ref/vacuumdb.sgml
===
RCS file: /cvsroot/pgsql/doc/src/sgml/ref/vacuumdb.sgml,v
retrieving revision 1.48
diff -c -c -r1.48 vacuumdb.sgml
*** doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 12:38:55 - 1.48
--- doc/src/sgml/ref/vacuumdb.sgml 7 Jan 2010 14:34:39 -
***
*** 28,34
grouparg--freeze/argarg-F/arg/group
grouparg--verbose/argarg-v/arg/group
grouparg--analyze/argarg-z/arg/group
!grouparg--analyze-only/argarg-o/arg/group
arg--table | -t replaceabletable/replaceable
arg( replaceable class=parametercolumn/replaceable [,...] )/arg
/arg
--- 28,34
grouparg--freeze/argarg-F/arg/group
grouparg--verbose/argarg-v/arg/group
grouparg--analyze/argarg-z/arg/group
!grouparg--analyze-only/argarg-Z/arg/group
arg--table | -t replaceabletable/replaceable
arg( replaceable class=parametercolumn/replaceable [,...] )/arg
/arg
***
*** 42,48
grouparg--freeze/argarg-F/arg/group
grouparg--verbose/argarg-v/arg/group
grouparg--analyze/argarg-z/arg/group
!grouparg--analyze-only/argarg-o/arg/group
/cmdsynopsis
/refsynopsisdiv
--- 42,48
grouparg--freeze/argarg-F/arg/group
grouparg--verbose/argarg-v/arg/group
grouparg--analyze/argarg-z/arg/group
!grouparg--analyze-only/argarg-Z/arg/group
/cmdsynopsis
/refsynopsisdiv
***
*** 142,148
/varlistentry
varlistentry
! termoption-o/option/term
termoption--analyze-only/option/term
listitem
para
--- 142,148
/varlistentry
varlistentry
! termoption-Z/option/term
termoption--analyze-only/option/term
listitem
para
Index: src/bin/scripts/vacuumdb.c
===
RCS file: /cvsroot/pgsql/src/bin/scripts/vacuumdb.c,v
retrieving revision 1.32
diff -c -c -r1.32 vacuumdb.c
*** src/bin/scripts/vacuumdb.c 7 Jan 2010 12:38:55 - 1.32
--- src/bin/scripts/vacuumdb.c 7 Jan 2010 14:34:44 -
***
*** 41,47
{quiet, no_argument, NULL, 'q'},
{dbname, required_argument, NULL, 'd'},
{analyze, no_argument, NULL, 'z'},
! {analyze-only, no_argument, NULL, 'o'},
{freeze, no_argument, NULL, 'F'},
{all, no_argument, NULL, 'a'},
{table, required_argument, NULL, 't'},
--- 41,47
{quiet, no_argument, NULL, 'q'},
{dbname, required_argument, NULL, 'd'},
{analyze, no_argument, NULL, 'z'},
! {analyze-only, no_argument, NULL, 'Z'},
{freeze, no_argument, NULL, 'F'},
{all, no_argument, NULL, 'a'},
{table, required_argument, NULL, 't'},
***
*** 107,113
case 'z':
and_analyze = true;
break;
! case 'o':
analyze_only = true;
break;
case 'F':
--- 107,113
case 'z':
and_analyze = true;
break;
! case 'Z':
analyze_only = true;
break;
case 'F':
***
*** 351,361
printf(_( -f, --full do full vacuuming\n));
printf(_( -F, --freezefreeze row transaction information\n));
printf(_( -i, --inplace do full inplace vacuuming\n));
- printf(_( -o, --analyze-only only update optimizer hints\n));
printf(_( -q, --quiet don't write any messages\n));
printf(_( -t, --table='TABLE[(COLUMNS)]' vacuum specific table only\n));
printf(_( -v, --verbose write a lot of output\n));
printf(_( -z, --analyze update optimizer hints\n));
printf(_( --help show this help, then exit\n));
printf(_( --version output version information, then exit\n));
printf(_(\nConnection options:\n));
--- 351,361
printf(_( -f, --full do full vacuuming\n));
printf(_( -F, --freezefreeze row transaction information\n));
printf(_( -i, --inplace do full inplace vacuuming\n));
printf(_( -q, --quiet don't write any messages\n));
printf(_( -t, --table='TABLE[(COLUMNS)]' vacuum specific table only\n));
printf(_( -v, --verbose write a lot of output\n));
printf(_( -z, --analyze
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
Fujii Masao masao.fu...@gmail.com writes: On Thu, Jan 7, 2010 at 5:46 PM, Magnus Hagander mag...@hagander.net wrote: However, wouldn't it make more logical sense to replace host/hostssl with replication/replicationssl rather than overload the database field? Seems good. How about the following formats? replication user CIDR-address auth-method [auth-options] replicationssl user CIDR-address auth-method [auth-options] replication user IP-address IP-mask auth-method [auth-options] replicationssl user IP-address IP-mask auth-method [auth-options] Note that database field has been removed since it's useless for replication. I'm getting more and more confused here. I thought we were talking about client-side .pgpass. This seems to be talking about pg_hba.conf. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 'replication' keyword on .pgpass (Streaming Replication)
Tom Lane wrote: I'm getting more and more confused here. I thought we were talking about client-side .pgpass. This seems to be talking about pg_hba.conf. Yeah, the topic was covertly changed. It seems we have consensus to not change .pgpass, and to leave pg_hba.conf as it is now in the patch as well. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Auto-extending table partitions?
On Thu, Jan 07, 2010 at 08:01:16PM +0530, Chetan Suttraway wrote: Adding on to this use case: what do we do when we reach end of year? Probably auto-archive as per weekly, monthly , quarterly or yearly tables? Because such requirements are so specific to each place, it's easier to do this in your own code. While managing partitions may get simpler than our current table inheritance, it's unlikely that automated tools will ever be able to handle all the cases for it, so that coding is likely to be part of your partitioning strategy for the foreseeable future. Cheers, David. P.S. Please do trim, as I just did, and don't top post. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thursday 07 January 2010 14:45:55 Joachim Wieland wrote: On Thu, Dec 31, 2009 at 6:40 PM, Simon Riggs si...@2ndquadrant.com wrote: Building racy infrastructure when it can be avoided with a little care still seems not to be the best path to me. Doing that will add more complexity in an area that is hard to test effectively. I think the risk of introducing further bugs while trying to fix this rare condition is high. Right now the conflict processing needs more work and is often much less precise than this, so improving this aspect of it would not be a priority. I've added it to the TODO though. Thank you for your research. Patch implements recovery conflict signalling using SIGUSR1 multiplexing, then uses a SessionCancelPending mode similar to Joachim's TransactionCancelPending. I have reworked Simon's patch a bit and attach the result. Quick facts: - Hot Standby only uses SIGUSR1 - SIGINT behaves as it did before: it only cancels running statements - pg_cancel_backend() continues to use SIGINT - I added pg_cancel_idle_transaction() to cancel an idle transaction via SIGUSR1 - One central function HandleCancelAction() sets the flags before calling ProcessInterrupts(), it is called from the different signal handlers and receives parameters about what it should do - If a SIGUSR1 reason is used that will cancel something, ProcArrayLock is acquired until the signal has been sent to make sure that we won't signal the wrong backend. Does this sufficiently cover the concerns of Andres Freund discussed upthread? I think it solves the major concern (which I btw could easily reproduce using software that is in production) but unfortunately not completely. The avoided situation is: C(Client): BEGIN; SELECT WHATEVER; S(Standby): conflict with C S: Starting to cancel C C: COMMIT S: Sending Signal to C C: Wrong transaction is aborted The situation not avoided is: C: BEGIN; SELECT ... S: conflict with C, lock procarray, sending signal(thats asynchronous), unlock procarray C: COMMIT; BEGIN C: Signal arrives C: Wrong txn is killled It should be easy to fix this by having a cancel_localTransactionId field in the procarray which gets cleaned uppon transaction/backend start and gets checked in the signal handler (should be casted to sig_atomic_t) Will cookup a patch if nobody speaks against something like that. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that it should do what you would do yourself by hand. And adding DBI as a requirement for running a buildfarm member would be a significant extra barrier to entry, ISTM. (I am very fond of DBI, and use it frequently, BTW) What about something less than a requirement then? If you have it great, you can run these extra tests. If you don't have it, no harm, no foul. We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. All the prerequisites should be in place for 99% of the machines: a C compiler and Perl are the biggies, and I can't see any buildfarm members running without those. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071014 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktF+ucACgkQvJuQZxSWSsjYOgCglyLIyGCr60og+iQSnyRgkowd +lYAnRDjPe/XxC7gb9OBPdpZlqU0wncK =kPIR -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
Joachim Wieland j...@mcknight.de writes: As there were so many boolean SomethingCancelPending variables I changed them to be bitmasks and merged all of them into a single variable. This seems like a truly horrid idea, because those variables are set by signal handlers. A bitmask cannot be manipulated atomically, so you have almost certainly introduced race-condition bugs. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Buffer statistics for pg_stat_statements
On Thu, Jan 7, 2010 at 3:31 AM, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: Tom Lane t...@sss.pgh.pa.us wrote: I don't necessarily know what the right thing to do with the new ones is, but I am pretty sure that pg_indent will revert any changes you make to the existing ones. That it will. The proposed changes to the existing lines are an exercise in uselessness; and to the extent that you format the added lines with this layout in mind, the final result could be worse than what you'd get if you adapt to pg_indent's rules to start with. Here is the proposed patch to adjust white spaces. It does not indent variables, but indents comments of the variables to adjust other fields. Are those changes ok? I think so. I'm not sure if it will push out the comment that is immediately adjacent to the trailing semicolon, but I don't think it will decrease the indent on the ones you've indented more. I think this is close enough for now and you should go ahead and commit it. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v3
Le 04/01/2010 22:36, Guillaume Lelarge a écrit : Le 29/12/2009 14:12, Guillaume Lelarge a écrit : Le 29/12/2009 00:03, Guillaume Lelarge a écrit : Le 28/12/2009 22:59, Tom Lane a écrit : Guillaume Lelarge guilla...@lelarge.info writes: Le 28/12/2009 17:06, Tom Lane a écrit : I think we were stalled on the question of whether to use one array or two parallel arrays. Do you want to try coding up a sample usage of each possibility so we can see which one seems more useful? I'm interested in working on this. But I don't find the thread that talk about this. Try here http://archives.postgresql.org/message-id/4aae8ccf.9070...@esilo.com Thanks. I've read all the new version of PQconnectdb and Determining client_encoding from client locale threads. I think I understand the goal. Still need to re-read this one (http://archives.postgresql.org/message-id/6222.1253734...@sss.pgh.pa.us) and completely understand it (will probably need to look at the code, at least the PQconnectdb one). But I'm definitely working on this. If I try to sum up my readings so far, this is what we still have to do: 1. try the one-array approach PGconn *PQconnectParams(const char **params) 2. try the two-arrays approach PGconn *PQconnectParams(const char **keywords, const char **values) Instead of doing a wrapper around PQconnectdb, we need to refactor the whole function, so that we can get rid of the parsing of the conninfo string (which is quite complicated). Using psql as an example would be a good idea, AFAICT. Am I right? did I misunderstand or forget something? I supposed I was right since noone yell at me :) I worked on this tonight. You'll find two patches attached, one for the one-array approach, one for the two-arrays approach. I know some more factoring can be done (at least, the get the fallback resources... part). I'm OK to do them. I just need to know if I'm on the right track. Hmmm... sorry but... can i have some comments on these two patches, please? -- Guillaume. http://www.postgresqlfr.org http://dalibo.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: Allow substring/replace() to get/set bit values
On Thu, Jan 7, 2010 at 7:02 AM, Leonardo F m_li...@yahoo.it wrote: attached a patch that adds the following functions for bit string: Thanks! Please add your patch here: https://commitfest.postgresql.org/action/commitfest_view/open The next CommitFest starts January 15th. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 8.5 Open Items
On Fri, Dec 25, 2009 at 3:09 PM, Robert Haas robertmh...@gmail.com wrote: I'm not sure whether we ever posted this schedule anywhere official - if so, I can't find it - but my understanding is that we have consensus on the release schedule described here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01251.php Under this schedule, the last CommitFest for the 8.5 release will start 3 weeks from today. As I think many of us remember, the last CommitFest for 8.4 was extremely long and was followed by some amount of further delay before we were able to go to beta. I believe that we have made a lot of progress in how we manage CommitFests this release cycle, and I am fairly confident that we will be able to wrap up the last CommitFest in a timely fashion. I am personally committed to doing whatever I can to make sure that happens and hereby volunteer to manage the last CommitFest, unless someone else would like to take a crack at it, in which case by all means feel free. The open items list is a source of somewhat more concern for me, because while patches can be bounced if we run out of time, open items don't just go away. I've added two items I know about here: http://wiki.postgresql.org/wiki/PostgreSQL_8.5_Open_Items I would encourage anyone else who is aware of open items to begin adding them to this list, adding sections as necessary. Last time around, we ended up assembling our initial list from a dump of one of Bruce's mail folders. I'm not sure whether that's the best way to go about it, but if we're going to do it that way, we should do it SOON, so that we start to get an idea what the must-fix issues are before we get down to the wire. We have a little time on any issues that are must-fix for the final release, but the scheduled gap between alpha4 and beta1 is quite short, so anything that is must-fix for beta is something we need to start thinking about soon. We can't do that, though, until we first have a list of items and a decision about which ones fall into that category. I was slightly surprised not to get any responses to this email. Alvaro added one or two things to the page mentioned above, and I added the bugs I posted about yesterday, but that's it. Are there really no other open issues for 8.5? That would be great, but I am concerned that there may be other things that people just haven't gotten around to mentioning yet. Also, should I assume that the silence in response to this email indicates that everyone likes the plan where I manage the next CommitFest? Thanks, ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 8.5 Open Items
On Thu, Jan 7, 2010 at 16:47, Robert Haas robertmh...@gmail.com wrote: On Fri, Dec 25, 2009 at 3:09 PM, Robert Haas robertmh...@gmail.com wrote: I'm not sure whether we ever posted this schedule anywhere official - if so, I can't find it - but my understanding is that we have consensus on the release schedule described here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01251.php Under this schedule, the last CommitFest for the 8.5 release will start 3 weeks from today. As I think many of us remember, the last CommitFest for 8.4 was extremely long and was followed by some amount of further delay before we were able to go to beta. I believe that we have made a lot of progress in how we manage CommitFests this release cycle, and I am fairly confident that we will be able to wrap up the last CommitFest in a timely fashion. I am personally committed to doing whatever I can to make sure that happens and hereby volunteer to manage the last CommitFest, unless someone else would like to take a crack at it, in which case by all means feel free. The open items list is a source of somewhat more concern for me, because while patches can be bounced if we run out of time, open items don't just go away. I've added two items I know about here: http://wiki.postgresql.org/wiki/PostgreSQL_8.5_Open_Items I would encourage anyone else who is aware of open items to begin adding them to this list, adding sections as necessary. Last time around, we ended up assembling our initial list from a dump of one of Bruce's mail folders. I'm not sure whether that's the best way to go about it, but if we're going to do it that way, we should do it SOON, so that we start to get an idea what the must-fix issues are before we get down to the wire. We have a little time on any issues that are must-fix for the final release, but the scheduled gap between alpha4 and beta1 is quite short, so anything that is must-fix for beta is something we need to start thinking about soon. We can't do that, though, until we first have a list of items and a decision about which ones fall into that category. I was slightly surprised not to get any responses to this email. Alvaro added one or two things to the page mentioned above, and I added the bugs I posted about yesterday, but that's it. Are there really no other open issues for 8.5? That would be great, but I am concerned that there may be other things that people just haven't gotten around to mentioning yet. Well, I have a couple of things in the win64 stuff still to go, but that patch is active on a CF page still, so I haven't added it to the list yet... Also, should I assume that the silence in response to this email indicates that everyone likes the plan where I manage the next CommitFest? Certainly +n from me :-) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PostgreSQL 8.5 Open Items
On Thu, Jan 7, 2010 at 10:50 AM, Magnus Hagander mag...@hagander.net wrote: On Thu, Jan 7, 2010 at 16:47, Robert Haas robertmh...@gmail.com wrote: On Fri, Dec 25, 2009 at 3:09 PM, Robert Haas robertmh...@gmail.com wrote: I'm not sure whether we ever posted this schedule anywhere official - if so, I can't find it - but my understanding is that we have consensus on the release schedule described here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01251.php Under this schedule, the last CommitFest for the 8.5 release will start 3 weeks from today. As I think many of us remember, the last CommitFest for 8.4 was extremely long and was followed by some amount of further delay before we were able to go to beta. I believe that we have made a lot of progress in how we manage CommitFests this release cycle, and I am fairly confident that we will be able to wrap up the last CommitFest in a timely fashion. I am personally committed to doing whatever I can to make sure that happens and hereby volunteer to manage the last CommitFest, unless someone else would like to take a crack at it, in which case by all means feel free. The open items list is a source of somewhat more concern for me, because while patches can be bounced if we run out of time, open items don't just go away. I've added two items I know about here: http://wiki.postgresql.org/wiki/PostgreSQL_8.5_Open_Items I would encourage anyone else who is aware of open items to begin adding them to this list, adding sections as necessary. Last time around, we ended up assembling our initial list from a dump of one of Bruce's mail folders. I'm not sure whether that's the best way to go about it, but if we're going to do it that way, we should do it SOON, so that we start to get an idea what the must-fix issues are before we get down to the wire. We have a little time on any issues that are must-fix for the final release, but the scheduled gap between alpha4 and beta1 is quite short, so anything that is must-fix for beta is something we need to start thinking about soon. We can't do that, though, until we first have a list of items and a decision about which ones fall into that category. I was slightly surprised not to get any responses to this email. Alvaro added one or two things to the page mentioned above, and I added the bugs I posted about yesterday, but that's it. Are there really no other open issues for 8.5? That would be great, but I am concerned that there may be other things that people just haven't gotten around to mentioning yet. Well, I have a couple of things in the win64 stuff still to go, but that patch is active on a CF page still, so I haven't added it to the list yet... Yeah, I'm not worried about the patches on the CF page. I have confidence we'll be able to kill all those off - we can commit them, postpone them, or reject them. What I'm worried about is people showing up at the last minute and saying we can't release until we decide what to do about these 20 issues. I don't want to wait until late February to have that conversation. Also, should I assume that the silence in response to this email indicates that everyone likes the plan where I manage the next CommitFest? Certainly +n from me :-) Hopefully n0. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] unresolved bugs
On Wed, Jan 6, 2010 at 21:36, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: The first of these in particular is a fairly detailed report of what looks might be a fairly serious problem. pg_listener entries deleted under heavy NOTIFY load only on Windows http://archives.postgresql.org/pgsql-bugs/2009-12/msg00274.php Yeah, that needs to be looked at by a Windows expert (ie not me). I missed that one. Will add to my list. BUG #5245: Full Server Certificate Chain Not Sent to client http://archives.postgresql.org/pgsql-bugs/2009-12/msg00145.php Not clear to me whether this is a bug, or at least our bug. It's not a bug, it's potentially a feature request :) Isn't that behavior entirely openssl's to determine? No, I think we can control that somehow. Let's leave it on the open items list, and I'll try to find some time to investigate it. Since it's behavior change, it's probably 8.5 material, and not backpatch. Oh, and if somebody else has the time to investigate it, please be my guest :) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: Allow substring/replace() to get/set bit values
Thanks! Please add your patch here:
https://commitfest.postgresql.org/action/commitfest_view/open
Ok; but what about what I said about the difference between bit/string
substring?
That affects overlay behaviour for bit...
I've even got
ERROR: invalid memory alloc request size 4244635647
with:
SELECT substring(B'0001' from 5 for -2);
(this is 8.4.2, windows version, not modified...)
test=# SELECT substring(B'0001' from 1 for -1);
substring
--
0001
(1 row)
test=# SELECT substring('0001' from 1 for -1);
ERROR: negative substring length not allowed
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 2010-01-07 11:50 +0200, Craig Ringer wrote: On 7/01/2010 9:15 AM, Robert Haas wrote: Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? At this point, I'm personally wondering if it's worth putting together a simple (ish) C program that reads a file describing command interleavings on n connections. It fires up one thread per connection required, then begins queuing commands up for the threads to execute in per-thread fifo queues. The input file may contain synchronization points where two or more explicitly specified threads (or just all threads) must finish all their queued work before they may be given more. CONN conn1: dbname=regress, user=regress CONN conn2: dbname=regress, user=regress STMT conn1: SELECT blah blah; STMT conn2: UPDATE blah blah; SYNC conn1, conn2 etc. Or alternately one-file-per-connection (which would be handy if one connection has *lots* of commands and others only occasional ones) - the only trouble there being how to conveniently specify synchronization points. I had a similar syntax in mind, but instead of using threads, just execute the file in order using asynchronous connections. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On 2010-01-07 18:13 +0200, Marko Tiikkaja wrote: I had a similar syntax in mind, but instead of using threads, just execute the file in order using asynchronous connections. I completely failed to make the point here which was to somehow mark which statements will (or, should) block. So here we go: A= BEGIN; B= BEGIN; A= UPDATE foo ..; B= UPDATE foo ..; -- this will block A= COMMIT; B= SELECT * FROM foo; B= COMMIT; [expected output here] Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: Allow substring/replace() to get/set bit values
On Thu, Jan 7, 2010 at 11:05 AM, Leonardo F m_li...@yahoo.it wrote:
Thanks! Please add your patch here:
https://commitfest.postgresql.org/action/commitfest_view/open
Ok; but what about what I said about the difference between bit/string
substring?
That affects overlay behaviour for bit...
I've even got
ERROR: invalid memory alloc request size 4244635647
with:
SELECT substring(B'0001' from 5 for -2);
(this is 8.4.2, windows version, not modified...)
test=# SELECT substring(B'0001' from 1 for -1);
substring
--
0001
(1 row)
test=# SELECT substring('0001' from 1 for -1);
ERROR: negative substring length not allowed
I haven't tried to reproduce it, but that sounds like a bug.
Robert
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Greg Sabino Mullane g...@turnstep.com writes: We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. As a packager, my reaction to that is over my dead body. We have enough trouble keeping our own software up to date, and pretty much every external component that we've started to bundle has been a disaster from a maintenance standpoint. (Examples: the zic database is constant work and yet almost never up to date; the snowball stemmer never gets updated.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler wrote: On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: We have not yet fully accepted the notion that you must have Perl to build (and, in fact, I am right now trying to verify that you don't). I don't think that requiring Perl to test is going to fly. I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. Unless I am mistaken, Perl is required in any case to build from CVS, although not from a tarball. DBI/DBD::pg is quite another issue, however. I have been deliberately very conservative about what modules to require for the buildfarm, and we have similarly (and I think wisely) been conservative about what modules to require for Perl programs in the build process. Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. cheers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, Jan 7, 2010 at 4:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: As there were so many boolean SomethingCancelPending variables I changed them to be bitmasks and merged all of them into a single variable. This seems like a truly horrid idea, because those variables are set by signal handlers. A bitmask cannot be manipulated atomically, so you have almost certainly introduced race-condition bugs. True... However there are just a few places where the patch uses bitmasks for modification of the variable. As Simon seems to be working on this currently anyway, I'll leave it to him to either keep the 5 boolean variables or do some mutual exclusion as in HandleNotifyInterrupt() (or do something completely different). Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, Jan 7, 2010 at 3:03 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote: I have reworked Simon's patch a bit and attach the result. Oh dear, this is exactly what I've been working on... Sorry, as you have posted a first patch some days ago I thought you were waiting for feedback... Just tried to help ;-) Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
On Wed, Jan 6, 2010 at 3:03 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Fujii Masao wrote: I've done that in my git branch. Could you push that git branch to a public place? Ahh, sorry, forgot that again. It's there now, at git://git.postgresql.org/git/users/heikki/postgres.git, branch 'replication'. I'm feeling like we're running out of time to get this committed. Committing large patches late in the release cycle is a recipe for a buggy beta, possibly a long beta, and a buggy release, and we're now down to 8 days before the start of the final CommitFest, after which our schedule indicates that we expect to put out an alpha and a beta relatively quickly. If this isn't ready to go, maybe we need to postpone it to 8.6. We've already had a bunch of bug reports (some of which have been fixed) as a result of HS, and I don't see any reason to believe that this isn't going to have the same problem. Personally, I would rather have a release without SR in June or July than a release with SR in August or September. We already have too many good features in the tree to hold up the whole process for patches that aren't ready yet - though like everyone else, I think this is a killer feature. Thoughts? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Thu, Jan 7, 2010 at 11:46 AM, Andrew Dunstan and...@dunslane.net wrote: Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. I completely agree. As between those two options, count me as +1 for the thin veneer. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Patch: Allow substring/replace() to get/set bit values
Leonardo F m_li...@yahoo.it writes: I've even got ERROR: invalid memory alloc request size 4244635647 with: SELECT substring(B'0001' from 5 for -2); Hm, yeah, somebody was sloppy about exposing the three-argument form of varbit substring and using -1 to represent the two-argument form. What we can do in the back branches is make the code treat any negative value as meaning two-arg form. To throw an error we'd need to refactor the pg_proc representation ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Wed, Jan 06, 2010 at 08:40:28PM -0500, Andrew Dunstan wrote: A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. Greg Stark had a patch a couple of years ago. Dunno what happened to it since then. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, 2010-01-07 at 17:50 +0100, Joachim Wieland wrote: On Thu, Jan 7, 2010 at 3:03 PM, Simon Riggs si...@2ndquadrant.com wrote: On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote: I have reworked Simon's patch a bit and attach the result. Oh dear, this is exactly what I've been working on... Sorry, as you have posted a first patch some days ago I thought you were waiting for feedback... Just tried to help ;-) Well, you have been very helpful, its just this last little part that is duplicated. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
On Thu, 2010-01-07 at 11:55 -0500, Robert Haas wrote: Personally, I would rather have a release without SR in June or July than a release with SR in August or September. If SR will be ready until then, I'd like to see a release in September which has SR in it. We already postponed SR a lot. Many of advocacy people including me already mentioned about SR, and many people are lookig after it. BTW, July probably won't be a good time for a new release, because of people's holidays. ...and maybe then we can start 8.5 - 9.0 thread. Regards, -- Devrim GÜNDÜZ, RHCE Command Prompt - http://www.CommandPrompt.com devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz signature.asc Description: This is a digitally signed message part
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote: @Simon: Is there a reason why you have not yet removed recoveryConflictMode from PGPROC? Unfortunately we still need a mechanism to mark which backends have been cancelled already. Transaction state for virtual transactions isn't visible on the procarray, so we need something there to indicate that a backend has been sent a conflict. Otherwise we'd end up waiting for it endlessly. The name will be changing though. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 6, 2010, at 6:31 PM, Kevin Grittner wrote:
As far as I've been able to determine so far, to call psql in a
relatively portable way would require something like this:
http://perldoc.perl.org/perlfork.html
Here's an example using IPC::Open3:
#!/usr/local/bin/perl -w
use strict;
use warnings;
use IPC::Open3;
use Symbol 'gensym';
use constant EOC = __DONE__\n;
my ($in1, $out1, $err1) = (gensym, gensym, gensym);
my ($in2, $out2, $err2) = (gensym, gensym, gensym);
my $pid1 = open3 $in1, $out1, $err1, 'bash';
my $pid2 = open3 $in2, $out2, $err2, 'bash';
print $in1 cd ~/dev/postgresql\n;
print $in1 ls doc\n;
print $in1 echo , EOC;
while ((my $line = $out1)) {
last if $line eq EOC;
print LS: $line;
}
print Finished file listing\n\n;
print $in2 cd ~/dev/postgresql\n;
print $in2 head -4 README\n;
print $in2 echo , EOC;
while (defined( my $line = $out2 )) {
last if $line eq EOC;
print HEAD: $line;
}
print Finished reading README\n\n;
print $in1 exit\n;
print $in2 exit\n;
waitpid $pid2, 0;
print All done!\n;
With that, I get:
LS: KNOWN_BUGS
LS: MISSING_FEATURES
LS: Makefile
LS: README.mb.big5
LS: README.mb.jp
LS: TODO
LS: bug.template
LS: src
Finished file listing
HEAD: PostgreSQL Database Management System
HEAD: =
HEAD:
HEAD: This directory contains the source code distribution of the
PostgreSQL
Finished reading README
All done!
I could easily write a very simple module to abstract all that stuff for you,
then you could just do something like:
my $psql1 = Shell::Pipe-new(qw(psql -U postgres));
my $psql2 = Shell::Pipe-new(qw(psql -U postgres));
$psql1-print('SELECT * from pg_class');
while (my $line = $psql1-readln) { print Output: $line\n }
$psql1-close;
All I'd need is some more reliable way than echo DONE__\n to be able to tell
when a particular command has finished outputting.
Thoughts?
Best,
David
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
Andrew Dunstan and...@dunslane.net writes: Unless I am mistaken, Perl is required in any case to build from CVS, although not from a tarball. Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
2010/1/7 Devrim GÜNDÜZ dev...@gunduz.org: On Thu, 2010-01-07 at 11:55 -0500, Robert Haas wrote: Personally, I would rather have a release without SR in June or July than a release with SR in August or September. June, yes. July, frankly, no, because July == September, when it comes to any such scheduling. At least in the countries where my clients are :) If SR will be ready until then, I'd like to see a release in September which has SR in it. We already postponed SR a lot. Many of advocacy people including me already mentioned about SR, and many people are lookig after it. BTW, July probably won't be a good time for a new release, because of people's holidays. -1. Frankly, if advocacy people said it would be there, they didn't tell the truth, and that's their problem. If they said hopefully it will be there, but we don't know yet, then they don't have a problem either way. Not having our release schedule driven by marketing is a *strength* of our project! We made the mistake last time to delay the release significantly for a single feature. It turned out said feature didn't make it *anyway*. Let's not repeat that mistake. ...and maybe then we can start 8.5 - 9.0 thread. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 7, 2010, at 9:08 AM, Tom Lane wrote: Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. In that case, there's nothing for it except concurrent psql. Or else some sort of shell environment that's available on all platforms. do we require bash on Windows? Oh, wait, the Windows build requires Perl… Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
Simon Riggs si...@2ndquadrant.com writes: On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote: @Simon: Is there a reason why you have not yet removed recoveryConflictMode from PGPROC? Unfortunately we still need a mechanism to mark which backends have been cancelled already. Transaction state for virtual transactions isn't visible on the procarray, so we need something there to indicate that a backend has been sent a conflict. Otherwise we'd end up waiting for it endlessly. The name will be changing though. While we're discussing this: the current coding with AbortOutOfAnyTransaction within ProcessInterrupts is *utterly* unsafe. I realize that's just a toy placeholder, but getting rid of it has to be on the list of stop-ship items. Right at the moment I'd prefer to see CONFLICT_MODE_ERROR always turned into CONFLICT_MODE_FATAL than to imagine this is going to work. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
On Thursday 07 January 2010 18:10:43 Magnus Hagander wrote: Not having our release schedule driven by marketing is a *strength* of our project! Yes. We made the mistake last time to delay the release significantly for a single feature. It turned out said feature didn't make it *anyway*. Let's not repeat that mistake. I would consider SR to be significantly less complex than HS though. What about giving it two weeks from now on to be in a comittable state? Last time the main discussion started a good while *after* the last commitfest... Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
David E. Wheeler da...@kineticode.com writes: On Jan 7, 2010, at 9:08 AM, Tom Lane wrote: Right, but to my mind building from a tarball needs to include the ability to run the regression tests on what you built. So injecting Perl into that is moving the goalposts on build requirements. In that case, there's nothing for it except concurrent psql. Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
On Jan 7, 2010, at 9:19 AM, Tom Lane wrote: In that case, there's nothing for it except concurrent psql. Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. I was rather assuming that was what we were talking about here, since we have in the past discussed testing things like dump and restore, which would require something like Perl to handle multiple processes, and wouldn't work very well for a regular release. I think if we have the ability to add tests that are not distributed, it gives us a lot more freedom and opportunity to test things that are not currently well-tested. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
2010/1/7 Magnus Hagander mag...@hagander.net: 2010/1/7 Devrim GÜNDÜZ dev...@gunduz.org: On Thu, 2010-01-07 at 11:55 -0500, Robert Haas wrote: Personally, I would rather have a release without SR in June or July than a release with SR in August or September. June, yes. July, frankly, no, because July == September, when it comes to any such scheduling. At least in the countries where my clients are :) In terms of when the release comes out, maybe. In terms of the NEXT release, it still matters. If the release is delayed, the first CommitFest of the next release will be that much later. If we put out a release by July 1 of this year, we can repeat the same schedule for the next release that we are using for this release and I will be happy with that. If we don't put out a release until September, our first CommitFest will be at least 2 months later than it was for the last one, which means that (1) we will have a gap of 8 months without a CommitFest and (2) 8.6 will have no chance of coming out before September 2011, and may end up being more like Thanksgiving if that one also slips. I really don't want to go 8 months with no CommitFest. That leads to too many patches in the queue, too many merge conflicts, too many patch authors who just plain give up, and no feedback to anyone for a very, very long time. If SR will be ready until then, I'd like to see a release in September which has SR in it. We already postponed SR a lot. Many of advocacy people including me already mentioned about SR, and many people are lookig after it. BTW, July probably won't be a good time for a new release, because of people's holidays. -1. Frankly, if advocacy people said it would be there, they didn't tell the truth, and that's their problem. If they said hopefully it will be there, but we don't know yet, then they don't have a problem either way. Not having our release schedule driven by marketing is a *strength* of our project! We made the mistake last time to delay the release significantly for a single feature. It turned out said feature didn't make it *anyway*. Let's not repeat that mistake. Indeed. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
Magnus Hagander mag...@hagander.net writes: We made the mistake last time to delay the release significantly for a single feature. It turned out said feature didn't make it *anyway*. Let's not repeat that mistake. Yeah, we've certainly learned that lesson often enough, or should I say failed to learn that lesson? However, HS is already in the tree, and HS without SR is a whole lot less compelling than HS with SR. So it's going to be pretty unsatisfying if we can't get SR in there. I read Robert's original question not so much as a proposal to slip the schedule to accommodate SR as a question about whether SR could still meet the current schedule. I think we ought to get that answered before we start debating schedule changes. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 However, HS is already in the tree, and HS without SR is a whole lot less compelling than HS with SR. So it's going to be pretty unsatisfying if we can't get SR in there. I don't think that's the case. Having HS alone would be a huge win, and the sooner we can get it out there the better. Those that are waiting for SR might have to wait one more version, but my intuition tells me that's a small minority compared to those waiting for HS. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071231 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGGoMACgkQvJuQZxSWSsj74ACgmjeQgRIAncQiCeQ5aaEeWI3y UHMAoOFWsCldiRzC0GJygwDdYXLGjE4O =YAwK -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Unless we are prepared to define concurrency testing as something separate from the basic regression tests. Which is kind of annoying but perhaps less so than the alternatives. It certainly seems to me to be the kind of thing you wouldn't need to test in order to have confidence in a local build. I thought we were leaning towards something separate. - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071234 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGGxYACgkQvJuQZxSWSsgG0gCfZ4eTpXd/97p3zSJpLqGhKMh6 8nMAoJ2lQUaCWNVeSPDU8fq7VnkO0s4C =xBOo -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thu, 2010-01-07 at 12:14 -0500, Tom Lane wrote:
Simon Riggs si...@2ndquadrant.com writes:
On Thu, 2010-01-07 at 14:45 +0100, Joachim Wieland wrote:
@Simon: Is there a reason why you have not yet removed recoveryConflictMode
from PGPROC?
Unfortunately we still need a mechanism to mark which backends have been
cancelled already. Transaction state for virtual transactions isn't
visible on the procarray, so we need something there to indicate that a
backend has been sent a conflict. Otherwise we'd end up waiting for it
endlessly. The name will be changing though.
While we're discussing this: the current coding with
AbortOutOfAnyTransaction within ProcessInterrupts is *utterly* unsafe.
I realize that's just a toy placeholder, but getting rid of it has to be
on the list of stop-ship items. Right at the moment I'd prefer to see
CONFLICT_MODE_ERROR always turned into CONFLICT_MODE_FATAL than to
imagine this is going to work.
Hmmm. Can you check my current attempt? This may suffer this problem.
If, so can you explain a little more for me? Thanks.
--
Simon Riggs www.2ndQuadrant.com
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index ea40420..e05792e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -313,8 +313,7 @@ IsTransactionState(void)
/*
* IsAbortedTransactionBlockState
*
- * This returns true if we are currently running a query
- * within an aborted transaction block.
+ * This returns true if we are within an aborted transaction block.
*/
bool
IsAbortedTransactionBlockState(void)
@@ -2692,6 +2691,48 @@ AbortCurrentTransaction(void)
}
/*
+ * AbortTransactionAndAnySubtransactions
+ *
+ * Similar to AbortCurrentTransaction but if any subtransactions
+ * in progress we abort them *and* all of their parents. So this is
+ * used when the caller wishes to make the abort untrappable by the user.
+ * After this has run IsAbortedTransactionBlockState() will be true.
+ */
+void
+AbortTransactionAndAnySubtransactions(void)
+{
+ TransactionState s = CurrentTransactionState;
+
+ switch (s-blockState)
+ {
+ case TBLOCK_DEFAULT:
+ case TBLOCK_STARTED:
+ case TBLOCK_BEGIN:
+ case TBLOCK_INPROGRESS:
+ case TBLOCK_END:
+ case TBLOCK_ABORT:
+ case TBLOCK_SUBABORT:
+ case TBLOCK_ABORT_END:
+ case TBLOCK_ABORT_PENDING:
+ case TBLOCK_PREPARE:
+ case TBLOCK_SUBABORT_END:
+ case TBLOCK_SUBABORT_RESTART:
+ AbortCurrentTransaction();
+ break;
+
+ case TBLOCK_SUBINPROGRESS:
+ case TBLOCK_SUBBEGIN:
+ case TBLOCK_SUBEND:
+ case TBLOCK_SUBABORT_PENDING:
+ case TBLOCK_SUBRESTART:
+ AbortSubTransaction();
+ CleanupSubTransaction();
+ AbortTransactionAndAnySubtransactions();
+ break;
+ }
+}
+
+/*
* PreventTransactionChain
*
* This routine is to be called by statements that must not run inside
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index 85f14f6..e2e64dd 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -324,6 +324,7 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
/* must be cleared with xid/xmin: */
proc-vacuumFlags = ~PROC_VACUUM_STATE_MASK;
proc-inCommit = false; /* be sure this is cleared in abort */
+ proc-recoveryConflictPending = false;
/* Clear the subtransaction-XID cache too while holding the lock */
proc-subxids.nxids = 0;
@@ -350,6 +351,7 @@ ProcArrayEndTransaction(PGPROC *proc, TransactionId latestXid)
/* must be cleared with xid/xmin: */
proc-vacuumFlags = ~PROC_VACUUM_STATE_MASK;
proc-inCommit = false; /* be sure this is cleared in abort */
+ proc-recoveryConflictPending = false;
Assert(proc-subxids.nxids == 0);
Assert(proc-subxids.overflowed == false);
@@ -377,7 +379,7 @@ ProcArrayClearTransaction(PGPROC *proc)
proc-xid = InvalidTransactionId;
proc-lxid = InvalidLocalTransactionId;
proc-xmin = InvalidTransactionId;
- proc-recoveryConflictMode = 0;
+ proc-recoveryConflictPending = false;
/* redundant, but just in case */
proc-vacuumFlags = ~PROC_VACUUM_STATE_MASK;
@@ -1665,7 +1667,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid,
if (proc-pid == 0)
continue;
- if (skipExistingConflicts proc-recoveryConflictMode 0)
+ if (skipExistingConflicts proc-recoveryConflictPending)
continue;
if (!OidIsValid(dbOid) ||
@@ -1704,7 +1706,7 @@ GetConflictingVirtualXIDs(TransactionId limitXmin, Oid dbOid,
* Returns pid of the process signaled, or 0 if not found.
*/
pid_t
-CancelVirtualTransaction(VirtualTransactionId vxid, int cancel_mode)
+CancelVirtualTransaction(VirtualTransactionId vxid, ProcSignalReason sigmode)
{
ProcArrayStruct *arrayP = procArray;
int index;
@@ -1722,28 +1724,22 @@ CancelVirtualTransaction(VirtualTransactionId vxid, int cancel_mode)
if (procvxid.backendId == vxid.backendId
procvxid.localTransactionId == vxid.localTransactionId)
{
- /*
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. Er...why not? That's what psql uses. As for those advocating using a custom C program written using libpq - that's basically what DBI/DBD::Pg ends up being! Only with a shiny Perl outside and years of real-world testing and usage. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. I'm still *very* interested in making a libpq-less pure perl driver, if anyone feels like funding it, let me know! :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071236 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGHF0ACgkQvJuQZxSWSsiFmACfUJbRDUJGvDTJNjgj/dyQKVCA tZwAn2fiXKNWbWzYXobrHZjeE8aSSiVv =sGzK -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
Greg Sabino Mullane wrote: -BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 However, HS is already in the tree, and HS without SR is a whole lot less compelling than HS with SR. So it's going to be pretty unsatisfying if we can't get SR in there. I don't think that's the case. Having HS alone would be a huge win, and the sooner we can get it out there the better. Those that are waiting for SR might have to wait one more version, but my intuition tells me that's a small minority compared to those waiting for HS. while I agree that HS is very useful without SR, I think that it's mostly the well known powerusers inthe community are actively waiting for HS and not so much for SR. For the typical user outside of -hackers or even -general I'm not so sure about that... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
On Thu, Jan 7, 2010 at 12:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: We made the mistake last time to delay the release significantly for a single feature. It turned out said feature didn't make it *anyway*. Let's not repeat that mistake. Yeah, we've certainly learned that lesson often enough, or should I say failed to learn that lesson? I think the latter phrasing is more accurate. However, HS is already in the tree, and HS without SR is a whole lot less compelling than HS with SR. So it's going to be pretty unsatisfying if we can't get SR in there. I read Robert's original question not so much as a proposal to slip the schedule to accommodate SR as a question about whether SR could still meet the current schedule. I think we ought to get that answered before we start debating schedule changes. Unfortunately, we've also discovered from hard experience that the timing of commits is difficult to predict unless the answer is something like today or tomorrow. I'm not terribly interested in an estimate of when this will be committed if it's much more distant than that because experience indicates that such estimates are typically inaccurate, usually on the optimistic side. I seem to recall Heikki estimating two weeks for SR about this time last year, and of course it took a lot longer than that, even if you subtract out the breaks in the action. That's not because Heikki is a bad estimator; it's just that estimating how long a particular piece of code will take to finish is extremely difficult and almost no one can do it with any degree of accuracy. It is the things the programmer can't foresee that push out the end date, and of course you can't know how many of those there will be. I like Andres' suggestion upthread of setting a deadline and determining to bounce the patch if it's not committed by that date. If it turns out we have to bounce it, that stinks, but I don't think it makes sense to go to beta with a huge, barely-tested pile of code in the tree. Not that the testing Heikki and Fujii Masao have been doing until now hasn't been good, but it's not nearly as rigorous as what we will get when all of our users start banging on it. The problem with even TALKING about changing the schedule is that we will have no idea what to change it TO. If we add two months to the schedule today, that will probably increase the chances of SR getting committed within that time frame (unless, of course, Heikki's employer uses that as an excuse to take him off the project for two months...) but we don't know how much because we can't predict how long it's going to take to be ready. If someone could show us a curve with probability on one axis and commit date on the other axis we could probably make a good decision about where to slice it off, but that isn't possible. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 while I agree that HS is very useful without SR, I think that it's mostly the well known powerusers inthe community are actively waiting for HS and not so much for SR. For the typical user outside of -hackers or even -general I'm not so sure about that... Well, I can state that we have plenty of clients that would be very interested in HS, but none that would really care if it came without SR. This power user knows a lot of people outside of -hackers and - -general and they are what I'm basing my opinion on. :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201001071303 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGIjEACgkQvJuQZxSWSsgT8gCgsfgjp+1ND312KXtExdqtlDRy tcYAnigTw1L+m4hFeT+qQ8mPHRitn78V =b+Vn -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug with PATHs having non-ASCII characters
Magnus Hagander wrote: On Thu, Jan 7, 2010 at 02:37, Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp wrote: I have a Windows-specific patch for open(), attached for reference. But we need to consider about other issues: - We need to consider about not only only open(), but also opendir(), stat() and symlink(). - An entirely-different fix is needed for non-Windows platforms. Probably we will convert encodings from GetDatabaseEncoding() to GetPlatformEncoding() in MBCS, but this is not needed on Windows. We should consider avoiding random ifdef blocks for the switching. Shouldn't we develop this with multi-platform in mind from the start, instead of doing a Windows specific patch? It may be that we end up with two completely different codepaths, but more likely we can share some of it between them? I'm not sure there's a lot you can do in platforms other than Windows. On Windows the filenames are all Unicode and you know you can get them using UTF16. On other platforms it could be anything, and it is certainly not server_encoding; even different filesystems can use different encodings. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Serializable Isolation without blocking
Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git. serializable sounds more like the branch name. I'll wait a bit for other comments before taking any action. Robert's advice being the last (and only) offered on the topic, I'm taking the silence as agreement and have dropped the request for a serializable repository and added one for /users/kgrittn/postgres instead. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
Simon Riggs si...@2ndquadrant.com writes: On Thu, 2010-01-07 at 12:14 -0500, Tom Lane wrote: While we're discussing this: the current coding with AbortOutOfAnyTransaction within ProcessInterrupts is *utterly* unsafe. I realize that's just a toy placeholder, but getting rid of it has to be on the list of stop-ship items. Right at the moment I'd prefer to see CONFLICT_MODE_ERROR always turned into CONFLICT_MODE_FATAL than to imagine this is going to work. Hmmm. Can you check my current attempt? This may suffer this problem. This looks like a mess. You've duplicated a whole lot of code and not fixed the fundamental problem. If, so can you explain a little more for me? Thanks. You can not do this from inside an interrupt service routine. Period. No amount of deck-chair-rearrangement will fix that. As far as I can think at the moment, the best you can do is throw the elog(ERROR), and if control gets out to the error recovery block in PostgresMain, you can force a transaction abort there. In other words, pretty much the same logic that was there before; the only addition that I think is safe is to allow this to happen while DoingCommandRead, so that you can cancel an idle transaction. Now of course the problem with this approach, if you choose to see it as a problem, is that somebody could trap the error and try to continue processing. The only way you can positively guarantee that the backend will give up whatever locks it's holding is if you elog(FATAL) instead of trying to do normal error processing. So maybe the right thing is to forget about CONFLICT_MODE_ERROR altogether. How critical is it that an HS-requested query cancel be any more likely to do anything than a regular query cancel is? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Application name patch - v3
On Thu, Jan 7, 2010 at 10:33 AM, Guillaume Lelarge guilla...@lelarge.info wrote: Le 04/01/2010 22:36, Guillaume Lelarge a écrit : Le 29/12/2009 14:12, Guillaume Lelarge a écrit : Le 29/12/2009 00:03, Guillaume Lelarge a écrit : Le 28/12/2009 22:59, Tom Lane a écrit : Guillaume Lelarge guilla...@lelarge.info writes: Le 28/12/2009 17:06, Tom Lane a écrit : I think we were stalled on the question of whether to use one array or two parallel arrays. Do you want to try coding up a sample usage of each possibility so we can see which one seems more useful? I'm interested in working on this. But I don't find the thread that talk about this. Try here http://archives.postgresql.org/message-id/4aae8ccf.9070...@esilo.com Thanks. I've read all the new version of PQconnectdb and Determining client_encoding from client locale threads. I think I understand the goal. Still need to re-read this one (http://archives.postgresql.org/message-id/6222.1253734...@sss.pgh.pa.us) and completely understand it (will probably need to look at the code, at least the PQconnectdb one). But I'm definitely working on this. If I try to sum up my readings so far, this is what we still have to do: 1. try the one-array approach PGconn *PQconnectParams(const char **params) 2. try the two-arrays approach PGconn *PQconnectParams(const char **keywords, const char **values) Instead of doing a wrapper around PQconnectdb, we need to refactor the whole function, so that we can get rid of the parsing of the conninfo string (which is quite complicated). Using psql as an example would be a good idea, AFAICT. Am I right? did I misunderstand or forget something? I supposed I was right since noone yell at me :) I worked on this tonight. You'll find two patches attached, one for the one-array approach, one for the two-arrays approach. I know some more factoring can be done (at least, the get the fallback resources... part). I'm OK to do them. I just need to know if I'm on the right track. Hmmm... sorry but... can i have some comments on these two patches, please? I would suggest adding your patch(es) to: https://commitfest.postgresql.org/action/commitfest_view/open Probably just one entry for the two of them would be most appropriate. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Serializable Isolation without blocking
On Thu, Jan 7, 2010 at 19:08, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: I think you should have users/kgrittner/postgres.git rather than serializable.git. serializable sounds more like the branch name. I'll wait a bit for other comments before taking any action. Robert's advice being the last (and only) offered on the topic, I'm taking the silence as agreement and have dropped the request for a serializable repository and added one for /users/kgrittn/postgres instead. ... and i've approved it :-) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Testing with concurrent sessions
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. As a packager, my reaction to that is over my dead body. We have enough trouble keeping our own software up to date, and pretty much every external component that we've started to bundle has been a disaster from a maintenance standpoint. (Examples: the zic database is constant work and yet almost never up to date; the snowball stemmer never gets updated.) As a counterargument, I'll point out that this won't be as critical as zic, especially if we're talking about an additional/optional set of tests. Also, Tim Bunce and I are right here, so the maintenance should not be that bad (and I'd hazard that a lot more people in the community know Perl/DBI than zic or stemmers). - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201001071315 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAktGJNQACgkQvJuQZxSWSshoHwCg9urTTT19m55soiIjuYKKuB5L PjYAoJbDAe7j4xsxsSFfVEkYDFpTjKE9 =48oW -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Small locking bugs in hs
On Sun, Dec 27, 2009 at 2:12 PM, Andres Freund and...@anarazel.de wrote: While unlikely to cause issues two new LWLockAcquire calls use the wrong locking mode. Patch attached. Does it make sense to add this to the 2010-01 CommitFest so we don't lose track of it? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication and postmaster signaling
Robert Haas robertmh...@gmail.com writes: I like Andres' suggestion upthread of setting a deadline and determining to bounce the patch if it's not committed by that date. If it turns out we have to bounce it, that stinks, but I don't think it makes sense to go to beta with a huge, barely-tested pile of code in the tree. Not that the testing Heikki and Fujii Masao have been doing until now hasn't been good, but it's not nearly as rigorous as what we will get when all of our users start banging on it. This argument would hold more water if there weren't *already* a huge, barely-tested pile of code in the tree, namely HS. If you think that's anywhere near ready to go to beta, I'm afraid I'd better disillusion you immediately. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standy introduced problem with query cancel behavior
On Thursday 07 January 2010 19:12:31 Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Thu, 2010-01-07 at 12:14 -0500, Tom Lane wrote: While we're discussing this: the current coding with AbortOutOfAnyTransaction within ProcessInterrupts is *utterly* unsafe. I realize that's just a toy placeholder, but getting rid of it has to be on the list of stop-ship items. Right at the moment I'd prefer to see CONFLICT_MODE_ERROR always turned into CONFLICT_MODE_FATAL than to imagine this is going to work. Hmmm. Can you check my current attempt? This may suffer this problem. This looks like a mess. You've duplicated a whole lot of code and not fixed the fundamental problem. If, so can you explain a little more for me? Thanks. You can not do this from inside an interrupt service routine. Period. No amount of deck-chair-rearrangement will fix that. As far as I can think at the moment, the best you can do is throw the elog(ERROR), and if control gets out to the error recovery block in PostgresMain, you can force a transaction abort there. In other words, pretty much the same logic that was there before; the only addition that I think is safe is to allow this to happen while DoingCommandRead, so that you can cancel an idle transaction. Stupid question: Why dont we add CHECK_FOR_INTERRUPTS (or something similar) to everything calling recv (especially in the EINTR) case? That way we can simply abort in the normal context without the constraint of an interrupt handler because recv() will finish after having serviced a signal handler. Sure there will be some parts calling CHECK_FOR_INTERRUPTS not often enough but thats surely not that critical - and after some time using a bit more force is ok I guess. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
