Re: [HACKERS] switch UNLOGGED to LOGGED
On Fri, Apr 22, 2011 at 4:13 AM, Leonardo Francalanci m_li...@yahoo.it
wrote:
Maybe you should change xl_act_commit to have a separate list of rels to
drop the init fork for (instead of mixing those with the list of files
to
drop as a whole).
I tried to follow your suggestion, thank you very much.
I have to admit I don't like this approach very much. I can't see
adding 4 bytes to every commit record for this feature.
I understand.
What if, in xl_xact_commit, instead of
RelFileNode xnodes
I use another struct for xnodes, something like:
{
RelFileNode xnode;
boolonlyInitFork;
}
That would increase the commit record size only when there are
RelFileNode(s) to drop at commit. So, instead of 4 bytes in
every commit, there are wasted bytes when the commit record
contains deleted permanent relations (that should happen much
less). I'm open to suggestions here...
3) Should we have a cascade option? I don't know if I have to handle
inherited tables and other dependent objects
Look at the way ALTER TABLE [ONLY] works for other action types, and copy it.
Ok
Thank you very much
Leonardo
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch for new feature: Buffer Cache Hibernation
Mitsuru IWASAKI wrote: the patch is available at: http://people.freebsd.org/~iwasaki/postgres/buffer-cache-hibernation-postgresql-20110508.patch We can't accept patches just based on a pointer to a web site. Please e-mail this to the mailing list so that it can be considered a submission under the project's licensing terms. I hope this would be committable and the final version. PostgreSQL has high standards for code submissions. Extremely few submissions are committed without significant revisions to them based on code review. So far you've gotten a first round of high-level design review, there's several additional steps before something is considered for a commit. The whole process is outlined at http://wiki.postgresql.org/wiki/Submitting_a_Patch From a couple of minutes of reading the patch, the first things that pop out as problems are: -All of the ControlFile - controlFile renaming has add a larger difference to ReadControlFile than I would consider ideal. -Touching StrategyControl is not something this patch should be doing. -I don't think your justification (debugging or portability) for keeping around your original code in here is going to be sufficient to do so. -This should not be named enable_buffer_cache_hibernation. That very large diff you ended up with in the regression tests is because all of the settings named enable_* are optimizer control settings. Using the name buffer_cache_hibernation instead would make a better starting point. From a bigger picture perspective, this really hasn't addressed any of my comments about shared_buffers only being the beginning of the useful cache state to worry about here. I'd at least like the solution to the buffer cache save/restore to have a plan for how it might address that too one day. This project is also picky about only committing code that fits into the long-term picture for desired features. Having a working example of a server-side feature doing cache storage and restoration is helpful though. Don't think your work here is unappreciated--it is. Getting this feature added is just a harder problem than what you've done so far. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why not install pgstattuple by default?
On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote: On 05/07/2011 05:26 PM, Peter Eisentraut wrote: On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote: pg_config is useful quite apart from its use in building things, as was discussed upthread. Link please. http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php That thread just asserts that it might be useful, and I responded by asking for what. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why not install pgstattuple by default?
On 05/08/2011 05:24 AM, Peter Eisentraut wrote: On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote: On 05/07/2011 05:26 PM, Peter Eisentraut wrote: On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote: pg_config is useful quite apart from its use in building things, as was discussed upthread. Link please. http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php That thread just asserts that it might be useful, and I responded by asking for what. As I said there: to see how the libraries are configured, for example. Just the other day I wanted to know what compilation options had been used for a particular installation. pg_config wasn't installed because the -devel package wasn't installed, and it would have saved me quite some time if pg_config had been available. Another example is to find out what the installation is using for shares, the service directory and so on. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Questions about the internal of fastpath function call
Hi, I am reading the source code of fastpath to understand the internal of fastpath. I can not understand how does it send result to the client, I hope somebody can help me on this. I see it call it invoke the function in HandleFunctionRequest() - retval = FunctionCallInvoke(fcinfo); - SendFunctionResult(retval, fcinfo.isnull, fip-rettype, rformat); -OidOutputFunctionCall() - OutputFunctionCall() - result = DatumGetCString(FunctionCall1(flinfo, val)); - result = FunctionCallInvoke(fcinfo); but I can not see in where it send the tuples to the client, instead it invoke FunctionCallInvoke again. Can anyone tell me how it copy the tuples and send it to the client. Thanks a lot. Clipper
Re: [HACKERS] Questions about the internal of fastpath function call
On Sun, May 8, 2011 at 8:01 AM, lee Richard clipper.ken...@gmail.com wrote: Hi, I am reading the source code of fastpath to understand the internal of fastpath. I can not understand how does it send result to the client, I hope somebody can help me on this. I see it call it invoke the function in HandleFunctionRequest() - retval = FunctionCallInvoke(fcinfo); - SendFunctionResult(retval, fcinfo.isnull, fip-rettype, rformat); -OidOutputFunctionCall() - OutputFunctionCall() - result = DatumGetCString(FunctionCall1(flinfo, val)); - result = FunctionCallInvoke(fcinfo); but I can not see in where it send the tuples to the client, instead it invoke FunctionCallInvoke again. Can anyone tell me how it copy the tuples and send it to the client. It can't: the fastpath function can only return a scalar. See the client side docs. The whole system is a kludge more or less. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Why not install pgstattuple by default?
On sön, 2011-05-08 at 07:21 -0400, Andrew Dunstan wrote: As I said there: to see how the libraries are configured, for example. Just the other day I wanted to know what compilation options had been used for a particular installation. pg_config wasn't installed because the -devel package wasn't installed, and it would have saved me quite some time if pg_config had been available. Another example is to find out what the installation is using for shares, the service directory and so on. Yeah, those are decent reasons. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Questions about the internal of fastpath function call
Merlin, Oh, I didnt realized that it does not support to return scalar, thanks a lot. When it returns a single value, I see it use the following function, SendFunctionCall result = DatumGetByteaP(FunctionCall1(flinfo, val)); I still can not see how it return a single return value to the client, and why it call FunctionCall1() again when it want to send the result. Regards Clipper 2011/5/8 Merlin Moncure mmonc...@gmail.com On Sun, May 8, 2011 at 8:01 AM, lee Richard clipper.ken...@gmail.com wrote: Hi, I am reading the source code of fastpath to understand the internal of fastpath. I can not understand how does it send result to the client, I hope somebody can help me on this. I see it call it invoke the function in HandleFunctionRequest() - retval = FunctionCallInvoke(fcinfo); - SendFunctionResult(retval, fcinfo.isnull, fip-rettype, rformat); -OidOutputFunctionCall() - OutputFunctionCall() - result = DatumGetCString(FunctionCall1(flinfo, val)); - result = FunctionCallInvoke(fcinfo); but I can not see in where it send the tuples to the client, instead it invoke FunctionCallInvoke again. Can anyone tell me how it copy the tuples and send it to the client. It can't: the fastpath function can only return a scalar. See the client side docs. The whole system is a kludge more or less. merlin
Re: [HACKERS] Why not install pgstattuple by default?
My example is of doing self-discovery to see if all needful database components seem to be properly installed. E.g. - the app needs pgcrypto, intarray, and a custom data type. The install script can consequently inform the production folk either looks good, or, alternately, seems problematic! Actually, I haven't coded a sample of the look for custom SPI types part, but it's a natural extension of what I have. Of course, it only provides a legitimate test when run on the database server, which isn't always how production folk want to do it, but that's part of a different argument...
[HACKERS] could not write block xlog flush request 3FD/0 is not satisfied
Hi again pgsql-hackers, my experimental index which I am developing still has some problems. Perhaps the list has some advices or hints where this error might occur: WARNING: could not write block 6 of base/459204/483963 DETAIL: Multiple failures --- write error might be permanent. directly followed by an: ERROR: xlog flush request 3FD/0 is not satisfied --- flushed only to 0/20E2DC4 CONTEXT: writing block 6 of relation base/459204/483963 So far my index worked for small relations, up to 1.000 tuples, now with a relation of 1.000.000 tuples I am running into these problems. I think I might be lacking some knowledge to really solve the problem. I am happy about any suggestions or advices you can make. Best wishes, Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] could not write block xlog flush request 3FD/0 is not satisfied
Yves Weißig wrote: my experimental index which I am developing still has some problems. Perhaps the list has some advices or hints where this error might occur: WARNING: could not write block 6 of base/459204/483963 DETAIL: Multiple failures --- write error might be permanent. directly followed by an: ERROR: xlog flush request 3FD/0 is not satisfied --- flushed only to 0/20E2DC4 CONTEXT: writing block 6 of relation base/459204/483963 So far my index worked for small relations, up to 1.000 tuples, now with a relation of 1.000.000 tuples I am running into these problems. Is there any chance you're running out of disk space? If not, what OS is this? What sort of storage system do you have? -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] could not write block xlog flush request 3FD/0 is not satisfied
Am 08.05.2011 17:38, schrieb Kevin Grittner: Yves Weißig wrote: my experimental index which I am developing still has some problems. Perhaps the list has some advices or hints where this error might occur: WARNING: could not write block 6 of base/459204/483963 DETAIL: Multiple failures --- write error might be permanent. directly followed by an: ERROR: xlog flush request 3FD/0 is not satisfied --- flushed only to 0/20E2DC4 CONTEXT: writing block 6 of relation base/459204/483963 So far my index worked for small relations, up to 1.000 tuples, now with a relation of 1.000.000 tuples I am running into these problems. Is there any chance you're running out of disk space? If not, what OS is this? What sort of storage system do you have? -Kevin Thanks for the reply. I think disk space shouldn't be the problem: postgres@farnsworth:/usr/local/pgsql/data/pg_log$ df -h FilesystemSize Used Avail Use% Mounted on /dev/sda1 7.5G 4.1G 3.1G 58% / none 496M 228K 496M 1% /dev none 501M 140K 501M 1% /dev/shm none 501M 100K 501M 1% /var/run none 501M 0 501M 0% /var/lock It is an Ubuntu 10.10 running in a VirtualBox. The storage is located locally on my HDD, no SAN or network access. I'm afraid it is an error in my code... Yves -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] could not write block xlog flush request 3FD/0 is not satisfied
On Sun, May 8, 2011 at 4:18 PM, Yves Weißig weis...@rbg.informatik.tu-darmstadt.de wrote: ERROR: xlog flush request 3FD/0 is not satisfied --- flushed only to 0/20E2DC4 That's a pretty big difference in log positions. It seems likely you've overwritten the block header writing garbage to the LSN. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] improvements to pgtune
Shiv wrote: So my exams are over now and am fully committed to the project in terms of time. I have started compiling a sort of personal todo for myself. I agree with your advice to start the project with small steps first. (I have a copy of the code and am trying to glean as much of it as I can) I just fixed a couple of bugs in the program that were easier to correct than explain. The code changes have been pushed to the github repo. I've also revised the output format to be a lot nicer. There's a UI shortcut you may find useful too; the program now takes a single input parameter as the input file, outputting to standard out. So a sample run might look like this now: $ ./pgtune postgresql.conf.sample [old settings] #-- # pgtune wizard run on 2011-05-08 # Based on 2060728 KB RAM in the server #-- default_statistics_target = 100 maintenance_work_mem = 120MB checkpoint_completion_target = 0.9 effective_cache_size = 1408MB work_mem = 12MB wal_buffers = 8MB checkpoint_segments = 16 shared_buffers = 480MB max_connections = 80 I would really appreciate your reply to Josh's thoughts. It would help me understand the variety of tasks and a possible ordering for me to attempt them. Josh's comments :/ What would you list as the main things pgtune doesn't cover right now? I have my own list, but I suspect that yours is somewhat different./ / / /I do think that autotuning based on interrogating the database is possible. However, I think the way to make it not be a tar baby is to tackle it one setting at a time, and start with ones we have the most information for. One of the real challenges there is that some data can be gleaned from pg_* views, but a *lot* of useful performance data only shows up in the activity log, and then only if certain settings are enabled./ I just revised the entire TODO file (which is now TODO.rst, formatted in ReST markup: http://docutils.sourceforge.net/rst.html ; test with rst2html TODO.rst TODO.html and look at the result). It should be easier to follow the flow of now, and it's organized in approximately the order I think things need to get finished in. There are few major areas for expansion that might happen on this program to choose from. I was thinking about doing them in this order: 1) Fix the settings validation and limits. I consider this a good place to start on hacking the code. it's really necessary work eventually, and it's easier to get started with than the other ideas. 2) Improve internals related to tracking things like memory and connections so they're easier to pass around the program. Adding a platform class is what I was thinking of. See the Estimating shared memory usage section of the TODO for more information. Add PostgreSQL version as another input to that. 3) Improve the settings model used for existing parameters. Right now people have reported that the work_mem settings suggested in particular are too high for many servers. Ideas about why that is are in the TODO. (This really requires the platform change be done first, or the code will be too hard to write/maintain) 4) Estimate memory used by the configuration and output sysctl.conf files. (Needs platform change too) 5) Add tuning suggestions for new parameters. The most obvious ideas all involve adding common logging changes. 6) Create some new UIs for running the program. A text-based program that asked questions (a 'wizard') or a GUI program doing the same are two common suggestions. The ideas Josh was talking about for interrogating the database for things are all a long ways off from the current state of the code being able to support them. If (1) through (3) here were done, that whole direction starts with (5) and then runs further that way. That might be a valid direction to move next instead of the (4), (6) I've listed here. You'd have finished something that taught enough about how the existing program works to be able to make some more difficult design decisions about fitting new features into it. If you really want to get right into live server analysis, there's no way for that to fit into the current program yet. And I don't think you'll get enough practice to see how it would without doing some more basic work first. You might as well write something new if that's your goal, and expect that you may not finish anything useful by the end of the summer. If you want to complete a project that results in code that people absolutely will use, the more boring plan I've outlined goes that way. One of the secrets to software development is that ideas for complicated features rarely result in software that gets released, while working on simpler programs that don't aim so high leads to software that ships to the world and finds users. The only
Re: [HACKERS] Questions about the internal of fastpath function call
lee Richard clipper.ken...@gmail.com writes: I still can not see how it return a single return value to the client, and why it call FunctionCall1() again when it want to send the result. pq_endmessage is where the bytes actually get pushed out to the client, if that's what you're looking for. The rest of the activity in SendFunctionResult is just concerned with constructing the message to be sent within buf. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch for new feature: Buffer Cache Hibernation
On 08.05.2011 07:58, Mitsuru IWASAKI wrote: I'll do more testing tomorrow, and hopefully finalize my patch. Done! the patch is available at: http://people.freebsd.org/~iwasaki/postgres/buffer-cache-hibernation-postgresql-20110508.patch I'd suggest doing this as an extension module. All the changes to existing server code seem superficial. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] XML with invalid chars
On 04/27/2011 11:41 PM, Noah Misch wrote:
On Wed, Apr 27, 2011 at 11:22:37PM -0400, Andrew Dunstan wrote:
On 04/27/2011 05:30 PM, Noah Misch wrote:
To make things worse, the dump/reload problems seems to depend on your version
of libxml2, or something. With git master, a CentOS 5 system with
2.6.26-2.1.2.8.el5_5.1 accepts the ^A byte, but an Ubuntu 8.04 LTS system with
2.6.31.dfsg-2ubuntu rejects it. Even with a patch like this, systems with a
lenient libxml2 will be liable to store XML data that won't restore on a system
with a strict libxml2. Perhaps we should emit a build-time warning if the local
libxml2 is lenient?
No, I think we need to be strict ourselves.
Then I suppose we'd also scan for invalid characters in xml_parse()? Or, at
least, do so when linked to a libxml2 that neglects to do so itself?
Yep.
Injecting the check here aids xmlelement and xmlforest , but xmlcomment
and xmlpi still let the invalid byte through. You can also still inject the
byte into an attribute value via xmlelement. I wonder if it wouldn't make
more sense to just pass any XML that we generate from scratch through libxml2.
There are a lot of holes to plug, otherwise.
Maybe there are, but I'd want lots of convincing that we should do that
at this stage. Maybe for 9.2. I think we can plug the holes fairly
simply for xmlpi and xmlcomment, and catch the attributes by moving this
check up into map_sql_value_to_xml_value().
I don't have much convincing to offer -- hunting down the holes seem fine, too.
I think I've done that. Here's the patch I have now. It looks like we
can catch pretty much everything by putting checks in four places, which
isn't too bad.
Please review and try to break.
cheers
andrew
diff --git a/src/backend/utils/adt/xml.c b/src/backend/utils/adt/xml.c
index ee82d46..12cfd56 100644
--- a/src/backend/utils/adt/xml.c
+++ b/src/backend/utils/adt/xml.c
@@ -142,6 +142,20 @@ static void SPI_sql_row_to_xmlelement(int rownum, StringInfo result,
#define NAMESPACE_XSI http://www.w3.org/2001/XMLSchema-instance;
#define NAMESPACE_SQLXML http://standards.iso.org/iso/9075/2003/sqlxml;
+/* forbidden C0 control chars */
+#define FORBIDDEN_C0 \
+ \x01\x02\x03\x04\x05\x06\x07\x08\x0B\x0C\x0E\x0F\x10\x11 \
+ \x12\x13\x14\x15\x16\x17\x18\x19\x1A\x1B\x1C\x1D\x1E\x1F
+
+static inline void
+check_forbidden_c0(char * str)
+{
+ if (strpbrk(str,FORBIDDEN_C0) != NULL)
+ ereport(ERROR,
+(errcode(ERRCODE_INVALID_TEXT_REPRESENTATION),
+ errmsg(character out of range),
+ errdetail(XML does not support control characters.)));
+}
#ifdef USE_LIBXML
@@ -411,6 +425,8 @@ xmlcomment(PG_FUNCTION_ARGS)
appendStringInfoText(buf, arg);
appendStringInfo(buf, --);
+ check_forbidden_c0(buf.data);
+
PG_RETURN_XML_P(stringinfo_to_xmltype(buf));
#else
NO_XML_SUPPORT();
@@ -718,6 +734,8 @@ xmlpi(char *target, text *arg, bool arg_is_null, bool *result_is_null)
}
appendStringInfoString(buf, ?);
+ check_forbidden_c0(buf.data);
+
result = stringinfo_to_xmltype(buf);
pfree(buf.data);
return result;
@@ -1184,6 +1202,9 @@ xml_parse(text *data, XmlOptionType xmloption_arg, bool preserve_whitespace,
encoding,
PG_UTF8);
+ /* check for illegal XML chars */
+ check_forbidden_c0((char *) utf8string);
+
/* Start up libxml and its parser (no-ops if already done) */
pg_xml_init();
xmlInitParser();
@@ -1804,6 +1825,9 @@ map_sql_value_to_xml_value(Datum value, Oid type, bool xml_escape_strings)
getTypeOutputInfo(type, typeOut, isvarlena);
str = OidOutputFunctionCall(typeOut, value);
+ /* check for illegal XML chars */
+ check_forbidden_c0(str);
+
/* ... exactly as-is for XML, and when escaping is not wanted */
if (type == XMLOID || !xml_escape_strings)
return str;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Questions about the internal of fastpath function call
Tom, Thanks a lot. The part I cant understand is I cant see where SendFunctionResult construct the return value into the buf. I expect it should copy something in retval to buf, but I cant find any of this in the code, the main logic of SendFunctionResult is: 00159 getTypeOutputInfo http://doxygen.postgresql.org/lsyscache_8c.html#48d637426d06bfdf607eb29da3be558a(rettype http://doxygen.postgresql.org/structfp__info.html#2b1f02caad79fae010cdc5836a2fcb83, typoutput, typisvarlena);00160 outputstr = OidOutputFunctionCall http://doxygen.postgresql.org/fmgr_8c.html#9b747dd35f098ab369b44e94f4708210(typoutput, retval);00161 pq_sendcountedtext http://doxygen.postgresql.org/pqformat_8c.html#e6acc43bbc9bee686de201de82a26db5(buf, outputstr, strlen(outputstr), false);00162 pfree http://doxygen.postgresql.org/mcxt_8c.html#4de9741ca04b2f01a82d3de16a1d6bf2(outputstr); I think it should copy the real result in OidOutputFunctionCall, but unfortunately I can only find it call FunctionCall1() again. Do I miss something? If I copy the tuples returnned from pl module here, is it possible to send a vector result from fastpath function call? 2011/5/9 Tom Lane t...@sss.pgh.pa.us lee Richard clipper.ken...@gmail.com writes: I still can not see how it return a single return value to the client, and why it call FunctionCall1() again when it want to send the result. pq_endmessage is where the bytes actually get pushed out to the client, if that's what you're looking for. The rest of the activity in SendFunctionResult is just concerned with constructing the message to be sent within buf. regards, tom lane
[HACKERS] make check in src/test/isolation is unworkable
I believe that the make check target in src/test/isolation is fundamentally unportable, as is illustrated by the fact that buildfarm member coypu is currently choking on it. The reason is that the pg_isolation_regress program depends on libpq, and in particular it depends on having an *installed* libpq. Anyplace where it appears to work, it's because you already installed Postgres, or at least libpq. Apparently coypu is the only buildfarm member that hasn't got a reasonably recent libpq already installed in system directories; or maybe it's just the first such that's tried to run the isolation-test script step. While we could maybe hack this to the point where it works (on some platforms) by dynamically linking libpq from the source tree, I don't think it's worth the trouble. Recommendations: 1. Modify the buildfarm script to run make installcheck in the isolation-test step, and of course move that to after doing the install step. 2. Get rid of the check target in src/test/isolation/Makefile. We don't need to be dealing with bug reports from people who try to use it and get either a link failure (easily diagnosed) or a libpq version compatibility problem (not so easily diagnosed). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make check in src/test/isolation is unworkable
On Mon, May 9, 2011 at 12:35 AM, Tom Lane t...@sss.pgh.pa.us wrote: While we could maybe hack this to the point where it works (on some platforms) by dynamically linking libpq from the source tree, I don't think it's worth the trouble. How is this different from the regular case with pg_regress? -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] make check in src/test/isolation is unworkable
Greg Stark gsst...@mit.edu writes: How is this different from the regular case with pg_regress? pg_regress doesn't link in libpq. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] patch: fix race in SSI's CheckTargetForConflictsIn
On Fri, May 06, 2011 at 10:49:22PM -0400, Dan Ports wrote:
Will update the patch.
Updated patch (in response to Robert's comments) attached.
Dan
--
Dan R. K. Ports MIT CSAILhttp://drkp.net/
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 419e0d9..9d51f08 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -3846,6 +3846,8 @@ CheckTargetForConflictsIn(PREDICATELOCKTARGETTAG *targettag)
LWLockId partitionLock;
PREDICATELOCKTARGET *target;
PREDICATELOCK *predlock;
+ PREDICATELOCK *mypredlock = NULL;
+ PREDICATELOCKTAG mypredlocktag;
Assert(MySerializableXact != InvalidSerializableXact);
@@ -3891,139 +3893,21 @@ CheckTargetForConflictsIn(PREDICATELOCKTARGETTAG *targettag)
if (sxact == MySerializableXact)
{
/*
- * If we're getting a write lock on the tuple and we're not in a
- * subtransaction, we don't need a predicate (SIREAD) lock. We
- * can't use this optimization within a subtransaction because the
- * subtransaction could be rolled back, and we would be left
- * without any lock at the top level.
+ * If we're getting a write lock on a tuple, we don't need
+ * a predicate (SIREAD) lock on the same tuple. We can
+ * safely remove our SIREAD lock, but we'll defer doing so
+ * until after the loop because that requires upgrading to
+ * an exclusive partition lock.
*
- * At this point our transaction already has an ExclusiveRowLock
- * on the relation, so we are OK to drop the predicate lock on the
- * tuple, if found, without fearing that another write against the
- * tuple will occur before the MVCC information makes it to the
- * buffer.
+ * We can't use this optimization within a subtransaction
+ * because the subtransaction could roll back, and we
+ * would be left without any lock at the top level.
*/
if (!IsSubTransaction()
GET_PREDICATELOCKTARGETTAG_OFFSET(*targettag))
{
-uint32 predlockhashcode;
-PREDICATELOCKTARGET *rmtarget = NULL;
-PREDICATELOCK *rmpredlock;
-LOCALPREDICATELOCK *locallock,
- *rmlocallock;
-
-/*
- * This is a tuple on which we have a tuple predicate lock. We
- * only have shared LW locks now; release those, and get
- * exclusive locks only while we modify things.
- */
-LWLockRelease(SerializableXactHashLock);
-LWLockRelease(partitionLock);
-LWLockAcquire(SerializablePredicateLockListLock, LW_SHARED);
-LWLockAcquire(partitionLock, LW_EXCLUSIVE);
-LWLockAcquire(SerializableXactHashLock, LW_EXCLUSIVE);
-
-/*
- * Remove the predicate lock from shared memory, if it wasn't
- * removed while the locks were released. One way that could
- * happen is from autovacuum cleaning up an index.
- */
-predlockhashcode = PredicateLockHashCodeFromTargetHashCode
- ((predlock-tag), targettaghash);
-rmpredlock = (PREDICATELOCK *)
- hash_search_with_hash_value(PredicateLockHash,
-(predlock-tag),
-predlockhashcode,
-HASH_FIND, NULL);
-if (rmpredlock)
-{
- Assert(rmpredlock == predlock);
-
- SHMQueueDelete(predlocktargetlink);
- SHMQueueDelete((predlock-xactLink));
-
- rmpredlock = (PREDICATELOCK *)
- hash_search_with_hash_value(PredicateLockHash,
- (predlock-tag),
- predlockhashcode,
- HASH_REMOVE, NULL);
- Assert(rmpredlock == predlock);
-
- RemoveTargetIfNoLongerUsed(target, targettaghash);
-
- LWLockRelease(SerializableXactHashLock);
- LWLockRelease(partitionLock);
- LWLockRelease(SerializablePredicateLockListLock);
-
- locallock = (LOCALPREDICATELOCK *)
- hash_search_with_hash_value(LocalPredicateLockHash,
- targettag, targettaghash,
- HASH_FIND, NULL);
-
- /*
- * Remove entry in local lock table if it exists and has
- * no children. It's OK if it doesn't exist; that means
- * the lock was transferred to a new target by a different
- * backend.
- */
- if (locallock != NULL)
- {
- locallock-held = false;
-
- if (locallock-childLocks == 0)
- {
- rmlocallock = (LOCALPREDICATELOCK *)
-hash_search_with_hash_value(LocalPredicateLockHash,
- targettag, targettaghash,
- HASH_REMOVE, NULL);
- Assert(rmlocallock == locallock);
- }
- }
-
- DecrementParentLocks(targettag);
-
- /*
- * If we've cleaned up the last of the predicate locks for
- * the target, bail out before re-acquiring the locks.
- */
- if (rmtarget)
- return;
-
- /*
- * The list has been altered. Start over at the front.
- */
- LWLockAcquire(partitionLock, LW_SHARED);
- nextpredlock = (PREDICATELOCK *)
- SHMQueueNext((target-predicateLocks),
- (target-predicateLocks),
- offsetof(PREDICATELOCK,
Re: [HACKERS] make check in src/test/isolation is unworkable
On 05/08/2011 07:35 PM, Tom Lane wrote: I believe that the make check target in src/test/isolation is fundamentally unportable, as is illustrated by the fact that buildfarm member coypu is currently choking on it. The reason is that the pg_isolation_regress program depends on libpq, and in particular it depends on having an *installed* libpq. Anyplace where it appears to work, it's because you already installed Postgres, or at least libpq. darn, you're right. Apparently coypu is the only buildfarm member that hasn't got a reasonably recent libpq already installed in system directories; or maybe it's just the first such that's tried to run the isolation-test script step. Most aren't running the test because they aren't updated yet. There are six machines running the tests: pgbfprod=# select distinct sysname from build_status_log where log_stage ~ 'isolation' and snapshot now() - interval '2 months'; sysname - anchovy coypu crake bobcat chough kite chough is doing the wrong thing anyway, because I got distracted and forgot to fill in the MSVC piece of the puzzle. While we could maybe hack this to the point where it works (on some platforms) by dynamically linking libpq from the source tree, I don't think it's worth the trouble. Recommendations: 1. Modify the buildfarm script to run make installcheck in the isolation-test step, and of course move that to after doing the install step. working on that. I have pushed a quick fix disabling the test for now until I come up with proper coding for this tomorrow. See https://github.com/PGBuildFarm/client-code/commit/bb1d2f972205d0a8f438bfde86a0fc99ffdeb24a 2. Get rid of the check target in src/test/isolation/Makefile. We don't need to be dealing with bug reports from people who try to use it and get either a link failure (easily diagnosed) or a libpq version compatibility problem (not so easily diagnosed). +1. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prefered Types
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom Lane wrote: BTW, not to rain on the parade or anything, but I'll bet that rejiggering anything at all here will result in whining that puts the 8.3-era removal of a few implicit casts to shame. I'll take that bet, as it's really hard to imagine anything being worse than the pain caused by 8.3 to many people using Postgres. But if this is anything at all like that (e.g. requiring rewriting tons of SQL queries or modifying system catalogs), then a big fat -1. I know, probably a moot point by now, but 8.3 is a sore spot for me. :) - -- Greg Sabino Mullane g...@turnstep.com PGP Key: 0x14964AC8 201105082230 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3HU0IACgkQvJuQZxSWSshp2gCeLzjdXPQ0NkwDzby0f8DzUErz FUEAoLNkIzJ5jWxVP2Ck3BZgxhd6HUhq =yALY -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prefered Types
Greg Sabino Mullane g...@turnstep.com writes: Tom Lane wrote: BTW, not to rain on the parade or anything, but I'll bet that rejiggering anything at all here will result in whining that puts the 8.3-era removal of a few implicit casts to shame. I'll take that bet, as it's really hard to imagine anything being worse than the pain caused by 8.3 to many people using Postgres. You think? At least the 8.3 changes resulted in easily-diagnosed parser errors. The folks who complained about it were complaining because they couldn't be bothered to fix anything about their applications, not because it was difficult to understand or to fix. It seems likely to me that any changes in function resolution behavior will result in failures that are *much* harder to diagnose. The actual fix might be the same (ie, insert an explicit cast or two) but back-tracking from the observed problem to that fix could be an order of magnitude more difficult. For example, if you start noticing an occasional integer overflow that didn't happen before, it might be pretty darn difficult to figure out that the problem is that an operation that was formerly resolved as int4 + int4 is now resolved as int2 + int2. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Prefered Types
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tom and I: BTW, not to rain on the parade or anything, but I'll bet that rejiggering anything at all here will result in whining that puts the 8.3-era removal of a few implicit casts to shame. I'll take that bet, as it's really hard to imagine anything being worse than the pain caused by 8.3 to many people using Postgres. You think? At least the 8.3 changes resulted in easily-diagnosed parser errors. The folks who complained about it were complaining because they couldn't be bothered to fix anything about their applications, not because it was difficult to understand or to fix. Those of us in the trenches saw things a little differently. There's a difference between couldn't be bothered and the sometimes herculean task of changing an existing complicated code base, including finding all the problems, fixing, writing tests, doing QA, etc. It was also difficult to explain all this to clients: why their code worked just fine on all previous versions, what the exact theoretical dangers involved are (and agreeing that, yes, it doesn't really apply to their particular code), and the sheer man-hours it was going to take to get their application over the 8.3 hump. (Granted, there's the system catalog hacks, but a) they introduce other problems and b) it's dangerous to reapply constantly when pg_dumping or moving across versions) It seems likely to me that any changes in function resolution behavior will result in failures that are *much* harder to diagnose. The actual fix might be the same (ie, insert an explicit cast or two) but back-tracking from the observed problem to that fix could be an order of magnitude more difficult. For example, if you start noticing an occasional integer overflow that didn't happen before, it might be pretty darn difficult to figure out that the problem is that an operation that was formerly resolved as int4 + int4 is now resolved as int2 + int2. Have I mentioned I'm already a big -1 on the whole idea? :) Yes, this will be a more subtle problem to diagnose, but I also think it will affect less code and thus not elicit as much whining. Besides, I never recommend clients use SMALLINT anyway. (That type you are using: I do not think it's as efficient as you think it is) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201105082312 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAk3HYk4ACgkQvJuQZxSWSshQ+ACePUFS++9q4lhsdWSolIqDuI+r LY4AoOBsEszt1goBe73GBuSW+dt0DfWF =gycE -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
