Re: [HACKERS] How to get SE-PostgreSQL acceptable
KaiGai, * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: I don't provide both of security_label and security_acl system columns for system/user tables. I didn't write it explicitly, it might make you confusing. User cannot see what security label is assigned to them due to lack of system column, so new sepgsql_xxx_getcon() functions are provided an interface to see security label. In this patch, I don't touch new system columns. I think Bruce's question was where you stored the security_acl and security_label columns. Based on your response (and a bit of purusal through the code.google site), it looks like you still have security_acl and security_label defined as internal columns and being included for at least system tables (or is it everywhere?). I think what people are looking for, instead, is either additional columns to just the existing system tables that need them (eg: pg_class, pg_attribute) or included in the existing ACL structure (the aclitem structure). Another option might be a seperate set of tables. This would further reduce the patch pretty significantly, I suspect, though you will need to rework some things. In terms of minimally invasive, I would guess modifying the existing ACL structure for the ACL info, and a seperate table to track the labels for different objects/sub-objects (similar to pg_depend) would be your best approach. That would require no changes to existing system tables, but a few changes in places where the ACL is handled, and then the hooks in the right places to do the permission checking. Just my 2c. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] Space reservation v02
On Tue, 2009-01-27 at 13:06 +0100, Zdenek Kotala wrote: Space reservation MUST TO be implemented if we want to have 8.4-8.5 upgrade. Why not just add a few dummy columns onto each catalog table? If we need space to expand a row we can just drop one of the dummy columns from the new catalog definition. That's an old DBA trick to avoid having to rewrite a table when you want to add a column. Seems much simpler to add columns than write special code, especially when we might find the new code has bugs and hasn't done what we thought it might. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
On Fri, 2009-01-30 at 13:15 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I'm thinking to add a new function that will allow crash testing easier. pg_crash_standby() will issue a new xlog record, XLOG_CRASH_STANDBY, which when replayed will just throw a FATAL error and crash Startup process. We won't be adding that to the user docs... This will allow us to produce tests that crash the server at specific places, rather than trying to trap those points manually. Heh, talk about a footgun ;-). I don't think including that in CVS is a good idea. Thought you'd like it. I'd have preferred it in a plugin... :-( Not really sure why its a problem though. We support pg_ctl stop -m immediate, which is the equivalent, yet we don't regard that as a footgun. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
On Fri, 2009-01-30 at 13:25 +0200, Heikki Linnakangas wrote: That whole area was something I was leaving until last, since immediate shutdown doesn't work either, even in HEAD. (Fujii-san and I discussed this before Christmas, briefly). We must handle shutdown gracefully, can't just leave bgwriter running after postmaster exit. Nobody was suggesting we should. -- Simon Riggs www.2ndQuadrant.com PostgreSQL Training, Services and Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
On Fri, 2009-01-30 at 16:55 +0200, Heikki Linnakangas wrote:
Ok, here's an attempt to make shutdown work gracefully.
Startup process now signals postmaster three times during startup: first
when it has done all the initialization, and starts redo. At that point.
postmaster launches bgwriter, which starts to perform restartpoints when
it deems appropriate. The 2nd time signals when we've reached consistent
recovery state. As the patch stands, that's not significant, but it will
be with all the rest of the hot standby stuff. The 3rd signal is sent
when startup process has finished recovery. Postmaster used to wait for
the startup process to exit, and check the return code to determine
that, but now that we support shutdown, startup process also returns
with 0 exit code when it has been requested to terminate.
Yeh, seems much cleaner.
Slightly bizarre though cos now we're pretty much back to my originally
proposed design. C'est la vie.
I like this way because it means we might in the future get Startup
process to perform post-recovery actions also.
The startup process now catches SIGTERM, and calls proc_exit() at the
next WAL record. That's what will happen in a fast shutdown. Unexpected
death of the startup process is treated the same as a backend/auxiliary
process crash.
Good. Like your re-arrangement of StartupProcessMain also.
Your call to PMSIGNAL_RECOVERY_COMPLETED needs to be if
(IsUnderPostmaster), or at least a comment to explain why not or perhaps
an Assert.
Think you need to just throw away this chunk
@@ -5253,7 +5386,7 @@ StartupXLOG(void)
* Complain if we did not roll forward far enough to render the
backup
* dump consistent.
*/
- if (XLByteLT(EndOfLog, ControlFile-minRecoveryPoint))
+ if (InRecovery !reachedSafeStartPoint)
{
if (reachedStopPoint) /* stopped because of stop
request */
ereport(FATAL,
--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
Stephen Frost wrote: KaiGai, * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: I don't provide both of security_label and security_acl system columns for system/user tables. I didn't write it explicitly, it might make you confusing. User cannot see what security label is assigned to them due to lack of system column, so new sepgsql_xxx_getcon() functions are provided an interface to see security label. In this patch, I don't touch new system columns. I think Bruce's question was where you stored the security_acl and security_label columns. Based on your response (and a bit of purusal through the code.google site), it looks like you still have security_acl and security_label defined as internal columns and being included for at least system tables (or is it everywhere?). In the current working tree, it (security id) is stored within padding field of HeapTupleHeader, so internal facility can read it via HeapTupleGetSecLabel(), but I already removed security_acl and security_label definition. Its basic structure is unchanged, the text form of security label is stored within pg_security system catalog. I think what people are looking for, instead, is either additional columns to just the existing system tables that need them (eg: pg_class, pg_attribute) or included in the existing ACL structure (the aclitem structure). Another option might be a seperate set of tables. It should not be held in text form within each tuples. Please remind why I rework for the feature again. Its facilities are a bit large to get included at a time, so its development should be done step-by-step approach and separatable ones (row-level, largeobjects) are postponed. This change is extreamly large, almost same as implementing from zero. I think it is far from step-by-step. This would further reduce the patch pretty significantly, I suspect, though you will need to rework some things. No, it need to rework *any* things. :( Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Commitfest infrastructure (was Re: [HACKERS] 8.4 release planning)
Peter Eisentraut wrote: On Thursday 29 January 2009 11:40:48 Stefan Kaltenbrunner wrote: well from a quick glance there is the bugzilla demo install as well as pieces of reviewboard and patchwork on the trackerdemo jail. So what's the URL and where can we sign up? resurrected the install and subscribed it to pgsql-hackers: http://trackerdemo.postgresql.org however it seems that it won't deal with patches that just have Content-Type: text/plain (like: http://archives.postgresql.org/pgsql-hackers/2009-01/msg02586.php) - seems not to hard to fix from a quick glance at the code however. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
On Sat, Jan 31, 2009 at 4:21 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2009-01-27 at 13:06 +0100, Zdenek Kotala wrote: Space reservation MUST TO be implemented if we want to have 8.4-8.5 upgrade. Why not just add a few dummy columns onto each catalog table? If we need space to expand a row we can just drop one of the dummy columns from the new catalog definition. That's an old DBA trick to avoid having to rewrite a table when you want to add a column. Seems much simpler to add columns than write special code, especially when we might find the new code has bugs and hasn't done what we thought it might. Wow, that is really sneaky. I like it! Actually, this even handles cases that the patch won't. For example, suppose there's an ARRAY[] of 2-byte objects and in PG 8.5 we make all of them 4 bytes. A fixed amount of space reservation per tuple is useless, but your idea works fine. Just add a new column of the same type and set them equal. As long as you can roughly predict how much extra crap to stuff in there, you're good. Can some variant of this be made to work if the index tuples expand? What if we're expanding the page header? Even if it can, space reservation (whether through adding dummy columns or an explicit facility) seems like an effort to avoid the hard problems of moving stuff around between pages during the upgrade process itself. I'm not sure that any such effort can be 100% successful, but then again if we don't need any special code changes to support it, maybe that doesn't matter right now. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] parallel restore
Tom Lane wrote: Okay, another question --- there are two places in pg_backup_custom.c where the patch #ifdef's out hasSeek tests on WIN32. Why is that? If checkSeek() is wrong on Windows, wouldn't it be better to fix it? Oh, dear. That's a hangover from before that got fixed earlier this month. checkSeek() should now work. I will make sure it does and let you know. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
KaiGai, * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: Stephen Frost wrote: I think Bruce's question was where you stored the security_acl and security_label columns. Based on your response (and a bit of purusal through the code.google site), it looks like you still have security_acl and security_label defined as internal columns and being included for at least system tables (or is it everywhere?). In the current working tree, it (security id) is stored within padding field of HeapTupleHeader, so internal facility can read it via HeapTupleGetSecLabel(), but I already removed security_acl and security_label definition. Its basic structure is unchanged, the text form of security label is stored within pg_security system catalog. I'm pretty sure that structure is part of what people were unhappy about though, and what makes it a much more invasive change that violates certain levels in the system by requiring information at much lower levels than it had before. I think what people are looking for, instead, is either additional columns to just the existing system tables that need them (eg: pg_class, pg_attribute) or included in the existing ACL structure (the aclitem structure). Another option might be a seperate set of tables. It should not be held in text form within each tuples. Please remind why I rework for the feature again. Its facilities are a bit large to get included at a time, so its development should be done step-by-step approach and separatable ones (row-level, largeobjects) are postponed. This change is extreamly large, almost same as implementing from zero. I think it is far from step-by-step. Yes, the development should be step-by-step, with minimally invasive changes each time. The first step is just hooking into SELinux for access levels that PostgreSQL already supports (database, schema, table, column). Requiring that the HeapTupleHeader be changed and that additional internal system columns be added, along with SELinux checks at levels that aren't appropriate, is alot more than this first patch should really need. Those kinds of changes would be with the patch to add non-SELinux row-level security. Following that, SELinux hooks would be added to the RLS checks. I think you need to look at it from the point of view of the committers and consider the whole patch, not just the quantity of code under security/. While that is important, the changes to the code *outside* of security/ is what the committers are going to be most concerned about because that gets into the overall structure and design and are areas which are much more likely to affect users who don't use this feature, performance, etc. Or at least, that's what I'd expect. Getting some commentary from core would be nice.. Thanks, Stephen signature.asc Description: Digital signature
[HACKERS] pgevent warnings on mingw
I just noticed these warnings on pgevent/mingw. I guess we've probably been living with it for years, but it would be nice to get it clean. (There are also some similar warnings earlier in the build regarding RegisterWaitForSingleObject.) dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
On Sat, Jan 31, 2009 at 8:32 AM, Stephen Frost sfr...@snowman.net wrote: * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: Stephen Frost wrote: I think Bruce's question was where you stored the security_acl and security_label columns. Based on your response (and a bit of purusal through the code.google site), it looks like you still have security_acl and security_label defined as internal columns and being included for at least system tables (or is it everywhere?). In the current working tree, it (security id) is stored within padding field of HeapTupleHeader, so internal facility can read it via HeapTupleGetSecLabel(), but I already removed security_acl and security_label definition. Its basic structure is unchanged, the text form of security label is stored within pg_security system catalog. I'm pretty sure that structure is part of what people were unhappy about though, and what makes it a much more invasive change that violates certain levels in the system by requiring information at much lower levels than it had before. IANAC, but that's my impression too. The simplified patch shouldn't assume that row-level security in its current form is going to end up getting put back in. AFAICS, there's no reason why the security ID for tables can't be a regular attribute in pg_class, or why the security attribute for columns can't be a regular attribute in pg_attribute. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Andrew Dunstan wrote: I just noticed these warnings on pgevent/mingw. I guess we've probably been living with it for years, but it would be nice to get it clean. (There are also some similar warnings earlier in the build regarding RegisterWaitForSingleObject.) dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? Yeah, they've been there forever. This happened, IIRC, when we changed the files to actually do what the documentation says. The commit message is: 2) Change the pgevent DLL file so it doens't specify ordinal for the functions. You're not supposed to do that. You're actually supposed to declare them as PRIVATE as well, but mingw doesn't support that. VC++ will throw a warning and not an error though, so we can live with it. The change was required to build at all with MSVC. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
I think what people are looking for, instead, is either additional columns to just the existing system tables that need them (eg: pg_class, pg_attribute) or included in the existing ACL structure (the aclitem structure). Another option might be a seperate set of tables. It should not be held in text form within each tuples. Please remind why I rework for the feature again. Its facilities are a bit large to get included at a time, so its development should be done step-by-step approach and separatable ones (row-level, largeobjects) are postponed. This change is extreamly large, almost same as implementing from zero. I think it is far from step-by-step. Yes, the development should be step-by-step, with minimally invasive changes each time. The first step is just hooking into SELinux for access levels that PostgreSQL already supports (database, schema, table, column). Requiring that the HeapTupleHeader be changed and that additional internal system columns be added, along with SELinux checks at levels that aren't appropriate, is alot more than this first patch should really need. Those kinds of changes would be with the patch to add non-SELinux row-level security. Following that, SELinux hooks would be added to the RLS checks. A concetn is current implementation assumes objects to be checked (tables, columns, ..., now) have its security identifier. Whole of the access control facility makes its decision based on security identifier. When the text form is used? It is used only when SE-PostgreSQL communicate with in-kernel SELinux because it is protocol. The result of decision making is cached in userspace to reduce kernel invocation, and all the entries are tagged by security identifier. So, any other facility does not need to handle text form. Removing security id also means replacement of whole of the current SE-PostgreSQL facilities! BTW, I had tried an approach which have security label by text, but its performance pain (due to string comparison) is unacceptable. (IIRC, about 50% of transaction per sec by pg_bench.) Indeed, we don't apply MAC on row-level in v8.4, so it does not make serious matter on table/column level, but is surely has to be scrapped once and implementaed from zero again. I think you need to look at it from the point of view of the committers and consider the whole patch, not just the quantity of code under security/. While that is important, the changes to the code *outside* of security/ is what the committers are going to be most concerned about because that gets into the overall structure and design and are areas which are much more likely to affect users who don't use this feature, performance, etc. IIRC, we recently discussed that we should not afraid to apply core code if it tries to get merged, so, I scraped all the PGACE facilities. Keeping independency under security/* is a concept of PGACE. It enabled to implement a new security feature with minimum impact to the core code, but, we concluded it is not necessary a few days ago, then it is just before to finish most of works... Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
Robert Haas wrote: On Sat, Jan 31, 2009 at 8:32 AM, Stephen Frost sfr...@snowman.net wrote: * KaiGai Kohei (kai...@kaigai.gr.jp) wrote: Stephen Frost wrote: I think Bruce's question was where you stored the security_acl and security_label columns. Based on your response (and a bit of purusal through the code.google site), it looks like you still have security_acl and security_label defined as internal columns and being included for at least system tables (or is it everywhere?). In the current working tree, it (security id) is stored within padding field of HeapTupleHeader, so internal facility can read it via HeapTupleGetSecLabel(), but I already removed security_acl and security_label definition. Its basic structure is unchanged, the text form of security label is stored within pg_security system catalog. I'm pretty sure that structure is part of what people were unhappy about though, and what makes it a much more invasive change that violates certain levels in the system by requiring information at much lower levels than it had before. IANAC, but that's my impression too. The simplified patch shouldn't assume that row-level security in its current form is going to end up getting put back in. AFAICS, there's no reason why the security ID for tables can't be a regular attribute in pg_class, or why the security attribute for columns can't be a regular attribute in pg_attribute. If it is identifier, it can be compoundable. I dislike it is held as text. It fundamentaly breaks SE-PostgreSQL's architecture, and requires to scrap near future. -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Hi. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? I was solved for Marko-san at the time of the work of pgbouncer, and obtained the solution. The patch is this. == Before == $ make gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -g -I../../../src/include -I/C/work/MinGW_WORK/include -I./src/include/port/win32 -DEXEC_BACKEND -I../../../src/include/port/win32 -c -o pgevent.o pgevent.c sed -e 's;FILEDESC;Eventlog message formatter;' -e 's;VFT_APP;VFT_DLL;' -e 's;_ICO_;;' -e 's;\(VERSION.*\),0 *$;\1,'`date '+%y%j' | sed 's/^0*//'`';' ../../../src/port/win32ver.rc win32ver.rc windres pgmsgevent.rc -o pgmsgevent.o --include-dir=../../../src/include --include-dir=../../../src/include --include-dir=. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 == After patch. == $ make gcc -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv -g -I../../../src/include -I/C/work/MinGW_WORK/include -I./src/include/port/win32 -DEXEC_BACKEND -I../../../src/include/port/win32 -c -o pgevent.o pgevent.c sed -e 's;FILEDESC;Eventlog message formatter;' -e 's;VFT_APP;VFT_DLL;' -e 's;_ICO_;;' -e 's;\(VERSION.*\),0 *$;\1,'`date '+%y%j' | sed 's/^0*//'`';' ../../../src/port/win32ver.rc win32ver.rc windres pgmsgevent.rc -o pgmsgevent.o --include-dir=../../../src/include --include-dir=../../../src/include --include-dir=. dlltool --export-all-symbols -A --output-def pgevent.def pgevent.o pgmsgevent.o dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o == It will be great if this is taken into consideration. Regards, Hiroshi Saito pgevent_patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Hiroshi Saito wrote: Hi. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? I was solved for Marko-san at the time of the work of pgbouncer, and obtained the solution. The patch is this. This appears to be exactly what you are *not* supposed to do. Which is assign an ordinal. See: http://msdn.microsoft.com/en-us/library/8e705t74(VS.71).aspx (that's just about the warning, there is a page somewhere with more details, but I can't find it right now) //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] expected collation and ctype on windows?
What is expected for collation and ctype settings on Windows when we specify utf8 as the encoding and en_US.utf8 as the locale? I am seeing them set to English_United States.1252. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
Simon Riggs escreveu: On Tue, 2009-01-27 at 13:06 +0100, Zdenek Kotala wrote: Space reservation MUST TO be implemented if we want to have 8.4-8.5 upgrade. Why not just add a few dummy columns onto each catalog table? If we need space to expand a row we can just drop one of the dummy columns from the new catalog definition. The problem is: how much space do we need at each catalog table? Suppose that after 2 releases we need x + 1 bytes per tuple but we only foresaw x bytes; so we need to deprecate support for this version and advise DBA to run pg_upgrade two times. I prefer a solution where we know beforehand the size per tuple so when we're rewriting we could use this information to upgrade. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Andrew Dunstan wrote: Hiroshi Inoue wrote: Eventually does the crash come from the call SetEnvironemntVariable (.., NULL) on mingw-XP(or older?)? I'm also interested in this issue and want to know the cause. The debugger shows that we actually fail on a popen() call in intdb. However, if we replace the calls to SetEnvironmentVariable(foo,NULL) with calls to SetEnvironmentVariable(foo,) then there is no failure. My theory is that on XP somehow the former is corrupting the environment such that when popen() tries to copy the environment for the new child process, it barfs. Well, XP only does it when it's built with mingw! Or is this actually dependent on if the binary is run under msys or cmd? //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Hiroshi Inoue wrote: Andrew Dunstan wrote: Andrew Dunstan wrote: Magnus Hagander wrote: Andrew Dunstan wrote: Magnus Hagander wrote: Are we *sure*, btw, that this is actually a mingw issue, and not something else in the environment? Could you try a MSVC compiled binary on the same machine? My MSVC buildfarm animal runs on the same machine, and does not suffer the same problem. Meh. Stupid mingw :-) So how about we #ifdef out that NULL setting based on WIN32_ONLY_COMPILER, does that seem reasonable? The odd thing is that it doesn't seem to affect Vista, only XP. Anyway, yes, I think that would be OK. How do we then test to see if the original problem is still fixed? Further proof that this is a Windows version issue: I took the problem build from my XP and put it on my Vista box: the same build that causes a problem on XP runs perfectly on Vista. Go figure. Maybe we need a version check at runtime? That would be icky. Eventually does the crash come from the call SetEnvironemntVariable (.., NULL) on mingw-XP(or older?)? I'm also interested in this issue and want to know the cause. However is it necessary to call SetEnvironmentVariable() in the first place? My original patch doesn't contain SetEnvironmentVariable call in pg_unsetenv() because _putenv() seems to call SetEnvironmentVariable internally. It's because I factored in another place where we *did* call it explicitly. Perhaps this code was put in for compatibility with some old version of mingw or something? If everything works if we remove that call in both msvc and mingw, we can just do that, yes. It still doesn't really explain *why* it crashes though. //Magnus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Hi. Sorry, I was not able to find the point to which you point. This appears to be exactly what you are *not* supposed to do. Which is assign an ordinal. See: http://msdn.microsoft.com/en-us/library/8e705t74(VS.71).aspx (that's just about the warning, there is a page somewhere with more details, but I can't find it right now) Please see this, if the point which you suggest becomes about output image size. == Before == -rwxr-xr-x1 HIROSHI Administ 425977 Feb 1 00:33 pgevent.dll == After patch. == -rwxr-xr-x1 HIROSHI Administ 425955 Feb 1 00:34 pgevent.dll Then, new pgevent.def is this. ; C:\MinGW\bin\dlltool.exe --export-all-symbols -A --output-def pgevent.def pgevent.o pgmsgevent.o EXPORTS dllregisterser...@0 @ 1 DllUnregisterServer = dllunregisterser...@0 @ 2 dllunregisterser...@0 @ 3 g_module @ 4 DATA DllRegisterServer = dllregisterser...@0 @ 5 Regards, Hiroshi Saito -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
On Sat, Jan 31, 2009 at 01:28:58PM -0200, Euler Taveira de Oliveira wrote: The problem is: how much space do we need at each catalog table? Suppose that after 2 releases we need x + 1 bytes per tuple but we only foresaw x bytes; so we need to deprecate support for this version and advise DBA to run pg_upgrade two times. I prefer a solution where we know beforehand the size per tuple so when we're rewriting we could use this information to upgrade. Just add one column of a variable length type, let pg_upgrade fill it with the required amount of padding prior to upgrade and you're done. Ofcourse, the simplest way to me for handling type changes seems to be to keep the old type OID reserved and have the new version of the type with a new OID. Then the entire problem vanishes. But it was decided a long time ago not to do that. Have a nice day, -- Martijn van Oosterhout klep...@svana.org http://svana.org/kleptog/ Please line up in a tree and maintain the heap invariant while boarding. Thank you for flying nlogn airlines. signature.asc Description: Digital signature
Re: [HACKERS] mingw check hung
Hi. Well, XP only does it when it's built with mingw! Or is this actually dependent on if the binary is run under msys or cmd? Both they look at a problem. http://winpg.jp/~saito/pg_bug/20090124/ Then, If SetEnvironmentVariable of Andrew-san point is removed, a problem will clearvery strange... Regards, Hiroshi Saito -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] parallel restore
Andrew Dunstan wrote: Tom Lane wrote: Okay, another question --- there are two places in pg_backup_custom.c where the patch #ifdef's out hasSeek tests on WIN32. Why is that? If checkSeek() is wrong on Windows, wouldn't it be better to fix it? Oh, dear. That's a hangover from before that got fixed earlier this month. checkSeek() should now work. I will make sure it does and let you know. Here is a new patch. Changes: * above #ifdefs removed * fixed declaration of DeClone() * brought up to date with CVS. successfully tested on Windows. cheers andrew parallel_restore_16.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Magnus Hagander wrote: Andrew Dunstan wrote: Hiroshi Inoue wrote: Eventually does the crash come from the call SetEnvironemntVariable (.., NULL) on mingw-XP(or older?)? I'm also interested in this issue and want to know the cause. The debugger shows that we actually fail on a popen() call in intdb. However, if we replace the calls to SetEnvironmentVariable(foo,NULL) with calls to SetEnvironmentVariable(foo,) then there is no failure. My theory is that on XP somehow the former is corrupting the environment such that when popen() tries to copy the environment for the new child process, it barfs. Well, XP only does it when it's built with mingw! Or is this actually dependent on if the binary is run under msys or cmd? Even weirder. It has now started working. For no apparent reason. I am seriously confused. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
IANAC, but that's my impression too. The simplified patch shouldn't assume that row-level security in its current form is going to end up getting put back in. AFAICS, there's no reason why the security ID for tables can't be a regular attribute in pg_class, or why the security attribute for columns can't be a regular attribute in pg_attribute. If it is identifier, it can be compoundable. I dislike it is held as text. It fundamentaly breaks SE-PostgreSQL's architecture, and requires to scrap near future. I think the column in pg_attribute and pg_class can and should be an OID. The issue is whether it's a regular OID column or a new system column. Stephen and I are saying it should be a regular column. pg_security can stick around to map OIDs to text labels. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] adding stuff to parser, question
Hey folks,
I am trying to add GRANT SELECT ON ALL TABLES to postgres, as it has
been quite few times now - where I had to write a procedure for that.
I kind of looked around, and more or less know how to approach it. But
I am stuck on parser :), yes - parser.
Can someone walk me through adding new rules to parser ?
so far I have this:
Index: parser/gram.y
===
RCS file: /projects/cvsroot/pgsql/src/backend/parser/gram.y,v
retrieving revision 2.656
diff -u -r2.656 gram.y
--- parser/gram.y 22 Jan 2009 20:16:05 - 2.656
+++ parser/gram.y 31 Jan 2009 16:44:57 -
@@ -494,7 +494,7 @@
STATISTICS STDIN STDOUT STORAGE STRICT_P STRIP_P SUBSTRING
SUPERUSER_P
SYMMETRIC SYSID SYSTEM_P
- TABLE TABLESPACE TEMP TEMPLATE TEMPORARY TEXT_P THEN TIME TIMESTAMP
+ TABLE TABLES TABLESPACE TEMP TEMPLATE TEMPORARY TEXT_P THEN TIME
TIMESTAMP
TO TRAILING TRANSACTION TREAT TRIGGER TRIM TRUE_P
TRUNCATE TRUSTED TYPE_P
@@ -4301,6 +4301,13 @@
n-objs = $2;
$$ = n;
}
+ | ALL TABLES
+ {
+ PrivTarget *n = (PrivTarget *)
palloc(sizeof(PrivTarget));
+ n-objtype = ACL_OBJECT_RELATION;
+ n-objs = NULL;
+ $$ = n;
+ }
| SEQUENCE qualified_name_list
{
PrivTarget *n = (PrivTarget *)
palloc(sizeof(PrivTarget));
Index: parser/keywords.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/parser/keywords.c,v
retrieving revision 1.209
diff -u -r1.209 keywords.c
--- parser/keywords.c 1 Jan 2009 17:23:45 - 1.209
+++ parser/keywords.c 31 Jan 2009 16:44:57 -
@@ -373,6 +373,7 @@
{sysid, SYSID, UNRESERVED_KEYWORD},
{system, SYSTEM_P, UNRESERVED_KEYWORD},
{table, TABLE, RESERVED_KEYWORD},
+ {table, TABLES, RESERVED_KEYWORD},
{tablespace, TABLESPACE, UNRESERVED_KEYWORD},
{temp, TEMP, UNRESERVED_KEYWORD},
{template, TEMPLATE, UNRESERVED_KEYWORD},
But that seems to be not nearly enough, for psql to recognize GRANT
SELECT ON ALL TABLES TO foo.
Please help me out, with stuff I am missing here. I never had any
expierence with bison, or any other lexical parsers for that matter.
Thanks. :)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
Ofcourse, the simplest way to me for handling type changes seems to be to keep the old type OID reserved and have the new version of the type with a new OID. Then the entire problem vanishes. But it was decided a long time ago not to do that. Why was that decision made? Suppose you have a type called widget and you decide it sucks and you want to reimplement it. So in release N+1, you rename the old type to old_shitty_widget and leave it with the same OID, add the new type under the name widget with a different oid, and document that old_shitty_widget should not be used. Then in release N+2 you remove old_shitty_widget altogether. People who upgrade via pg_dump will automatically get the new and improved widget type because that is what is now called widget. But people who in-place upgrade will end up with the old_shitty_widget type. Then you just run some dead simple postupdate script that goes through and issues ALTER TABLE commands to change each old_shitty_widget column to a widget column. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 16:46, Grzegorz Jaskiewicz wrote:
+ {table, TABLES, RESERVED_KEYWORD},
Gaaah, a typo...
:(
(another useless post to -hackers, by myself).
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
Grzegorz Jaskiewicz g...@pointblue.com.pl writes:
You're going to kick yourself, but:
{table, TABLE, RESERVED_KEYWORD},
+ {table, TABLES, RESERVED_KEYWORD},
^
I don't see any reason offhand why it should have to be a reserved word
though. You should be able to make it an UNRESERVED_KEYWORD. Oh, and you'll
want to add it to the list of tokens in unreserved_keyword in gram.y as well.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL
training!
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On Sat, Jan 31, 2009 at 04:46:26PM +, Grzegorz Jaskiewicz wrote: Hey folks, I am trying to add GRANT SELECT ON ALL TABLES to postgres, Is this part of the SQL:2008? If not, is there something else that is? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 17:17, Gregory Stark wrote:
Grzegorz Jaskiewicz g...@pointblue.com.pl writes:
You're going to kick yourself, but:
{table, TABLE, RESERVED_KEYWORD},
+ {table, TABLES, RESERVED_KEYWORD},
^
I don't see any reason offhand why it should have to be a reserved
word
though. You should be able to make it an UNRESERVED_KEYWORD. Oh, and
you'll
want to add it to the list of tokens in unreserved_keyword in gram.y
as well.
I am really novice with parsers here, so - I felt like I have to do
it, in order to make it work. It just wasn't working without that bit
in keywords.c.
I shall try your way, thanks :)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
from http://wiki.postgresql.org/wiki/Todo: [E] inline: 10px-UntickedTick.svg.pngAllow GRANT/REVOKE permissions to be applied to all schema objects with one command The proposed syntax is: GRANT SELECT ON ALL TABLES IN public TO phpuser; GRANT SELECT ON NEW TABLES IN public TO phpuser; -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On Sat, Jan 31, 2009 at 05:24:15PM +, Grzegorz Jaskiewicz wrote: from http://wiki.postgresql.org/wiki/Todo: I see the TODO item, but I don't see anything in the SQL standard, which I think is something we should explore before making a potentially incompatible change. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
Grzegorz Jaskiewicz wrote: from http://wiki.postgresql.org/wiki/Todo: [E] Allow GRANT/REVOKE permissions to be applied to all schema objects with one command The proposed syntax is: GRANT SELECT ON ALL TABLES IN public TO phpuser; GRANT SELECT ON NEW TABLES IN public TO phpuser; But the syntax you posted does not do this at all. Where does it restrict the grant to a single schema, like the syntax above? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] mingw check hung
Andrew Dunstan wrote: Even weirder. It has now started working. For no apparent reason. I am seriously confused. I spoke too soon :-( http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=dawn_batdt=2009-01-31%2016:28:16 cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 17:30, Andrew Dunstan wrote: But the syntax you posted does not do this at all. Where does it restrict the grant to a single schema, like the syntax above? I am just starting the attempt here, obviously since I admit that my parser skills are next to none - I didn't address such issue. So far, testing this change - I can do issue GRANT SELECT ON ALL TABLES TO xxx, and it parses well. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 17:28, David Fetter wrote: On Sat, Jan 31, 2009 at 05:24:15PM +, Grzegorz Jaskiewicz wrote: from http://wiki.postgresql.org/wiki/Todo: I see the TODO item, but I don't see anything in the SQL standard, which I think is something we should explore before making a potentially incompatible change. Personally, I don't think we should follow SQL 2008 to the letter. I am sure, many ppl miss that ability - I know I do, so I wanted to implement that right in the core. Wether SQL standard has something of same functionality but different syntax or not, well - I would love to know too - but I never read SQL standard - to be honest. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
Robert Haas wrote: People who upgrade via pg_dump will automatically get the new and improved widget type because that is what is now called widget. But people who in-place upgrade will end up with the old_shitty_widget type. Then you just run some dead simple postupdate script that goes through and issues ALTER TABLE commands to change each old_shitty_widget column to a widget column. Altering column type with ALTER TABLE needs to exclusively-lock and rewrite the whole table, so you might as well pg_dump+restore that table. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
Robert Haas wrote: Ofcourse, the simplest way to me for handling type changes seems to be to keep the old type OID reserved and have the new version of the type with a new OID. Then the entire problem vanishes. But it was decided a long time ago not to do that. Why was that decision made? Suppose you have a type called widget and you decide it sucks and you want to reimplement it. So in release N+1, you rename the old type to old_shitty_widget and leave it with the same OID, add the new type under the name widget with a different oid, and document that old_shitty_widget should not be used. Then in release N+2 you remove old_shitty_widget altogether. Yeah, that works. The other approach is to convert the data types along with the new page format. That works too, and avoids having to keep around the old type and all that deprecation and stuff. I don't remember any hard decision on that, and we're not facing any data type changes in this release IIRC. It is something we should consider on a case-by-case basis when we get there. There might be reasons to do it like that, if for example the old format can't be converted to new format in a non-lossy fashion (e.g. float-timestamps - integer-timestamps). And authors of 3rd party data types are naturally free to do what they want. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
On 1/31/09, Magnus Hagander mag...@hagander.net wrote: Hiroshi Saito wrote: Hi. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? I was solved for Marko-san at the time of the work of pgbouncer, and obtained the solution. The patch is this. This appears to be exactly what you are *not* supposed to do. Which is assign an ordinal. See: http://msdn.microsoft.com/en-us/library/8e705t74(VS.71).aspx AFAICS by default the mingw assigns the ordinal anyway? So whats your point? (that's just about the warning, there is a page somewhere with more details, but I can't find it right now) I found this: http://www.geocities.com/yongweiwu/stdcall.htm which definitely has too much detail... -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
Simon Riggs wrote: On Fri, 2009-01-30 at 13:15 +0200, Heikki Linnakangas wrote: Simon Riggs wrote: I'm thinking to add a new function that will allow crash testing easier. pg_crash_standby() will issue a new xlog record, XLOG_CRASH_STANDBY, which when replayed will just throw a FATAL error and crash Startup process. We won't be adding that to the user docs... This will allow us to produce tests that crash the server at specific places, rather than trying to trap those points manually. Heh, talk about a footgun ;-). I don't think including that in CVS is a good idea. Thought you'd like it. I'd have preferred it in a plugin... :-( Not really sure why its a problem though. We support pg_ctl stop -m immediate, which is the equivalent, yet we don't regard that as a footgun. If you poison your WAL archive with a XLOG_CRASH_RECOVERY record, recovery will never be able to proceed over that point. There would have to be a switch to ignore those records, at the very least. pg_ctl stop -m immediate has some use in a production system, while this would be a pure development aid. For that purpose, you might as use a patched version. Presumably you want to test different kind of crashes and at different points. FATAL, PANIC, segfault etc. A single crash mechanism like that will only test one path. You don't really need to do it with a new WAL record. You could just add a GUC or recovery.conf option along the lines of recovery_target: crash_target=0/123456, and check for that in ReadRecord or wherever you want the crash to occur. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot standby, recovery infra
Simon Riggs wrote: On Fri, 2009-01-30 at 16:55 +0200, Heikki Linnakangas wrote: Ok, here's an attempt to make shutdown work gracefully. Startup process now signals postmaster three times during startup: first when it has done all the initialization, and starts redo. At that point. postmaster launches bgwriter, which starts to perform restartpoints when it deems appropriate. The 2nd time signals when we've reached consistent recovery state. As the patch stands, that's not significant, but it will be with all the rest of the hot standby stuff. The 3rd signal is sent when startup process has finished recovery. Postmaster used to wait for the startup process to exit, and check the return code to determine that, but now that we support shutdown, startup process also returns with 0 exit code when it has been requested to terminate. Yeh, seems much cleaner. Slightly bizarre though cos now we're pretty much back to my originally proposed design. C'est la vie. Yep. I didn't see any objections to that approach in the archives. There was other problems in the early versions of the patch, but nothing related to this arrangement. I like this way because it means we might in the future get Startup process to perform post-recovery actions also. Yeah, it does. Do you have something in mind already? Your call to PMSIGNAL_RECOVERY_COMPLETED needs to be if (IsUnderPostmaster), or at least a comment to explain why not or perhaps an Assert. Nah, StartupProcessMain is only run under postmaster; you don't want to install signal handlers in a stand-along backend. Stand-alone backend calls StartupXLOG directly. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] Space reservation v02
On Jan 31, 2009, at 2:44 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Robert Haas wrote: People who upgrade via pg_dump will automatically get the new and improved widget type because that is what is now called widget. But people who in-place upgrade will end up with the old_shitty_widget type. Then you just run some dead simple postupdate script that goes through and issues ALTER TABLE commands to change each old_shitty_widget column to a widget column. Altering column type with ALTER TABLE needs to exclusively-lock and rewrite the whole table, so you might as well pg_dump+restore that table. Not at all. Lots of DDL operations take table locks, but they're still a lot more convenient than dump+restore. Think dependencies. A stickier wicket is how to handle functions and views that depend on the old type, so maybe this isn't quite as clean as I thought. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] foreign_data test fails with non-C locale
Zdenek Kotala wrote: PL-check gives the diff below on PLTCL tests under en_US locale. I guess the simplest answer is to add an alternative result file. Yes, I thought about add locale suffix for alternative result file, but it could be useless overhead. But some tests can be modified. For example select * from T_pkey1 order by key1 using @, key2; can be rewritten as select * from T_pkey1 order by key1 using @, key2::name; Is that the preferred solution? I want to fix this so I can re-enable building with TCL in dungbeetle. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] LIMIT NULL
Howdy, Is it intentional that `LIMIT NULL` means the same as `LIMIT ALL`? If so, I'd like to submit a patch to document it, because I've found it useful in SQL functions: http://justatheory.com/computers/databases/postgresql/dynamic-limit.html Thanks, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
On 1/31/09, Marko Kreen mark...@gmail.com wrote: On 1/31/09, Magnus Hagander mag...@hagander.net wrote: Hiroshi Saito wrote: Hi. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? I was solved for Marko-san at the time of the work of pgbouncer, and obtained the solution. The patch is this. This appears to be exactly what you are *not* supposed to do. Which is assign an ordinal. See: http://msdn.microsoft.com/en-us/library/8e705t74(VS.71).aspx AFAICS by default the mingw assigns the ordinal anyway? So whats your point? FWIW, I experimented with various approaches on DLL generation, and my theory is now that numeric slot is mandatory for export, name slot optional. And what the above link above warns about is that if you manually assign a numeric slot, you may create gaps in slot table, thus resulting in larger image. But this assumes you already have slots that are manually assigned, which should not happen with pgevent.dll. Does it sound sane? With pgbouncer I used dlltool --export-all-symbols -A to generate the .def file, thus the slots are actually assigned automatically by dlltool, so no gap problems should arise. As Postgres does not generate .def automatically, thus Hiroshi simply posted the resulting .def file, which should not have gap problems. But such detailed .def avoids the confusing warnings from dllwrap. If you are worried about gap problems I suggest instead doing the same and making the .def file auto-generated. -- marko ps. I suggest also tagging the g_module as static. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On Sat, Jan 31, 2009 at 05:40:57PM +, Grzegorz Jaskiewicz wrote:
On 31 Jan 2009, at 17:30, Andrew Dunstan wrote:
But the syntax you posted does not do this at all. Where does it
restrict the grant to a single schema, like the syntax above?
I am just starting the attempt here, obviously since I admit that my
parser skills are next to none - I didn't address such issue.
So far, testing this change - I can do issue GRANT SELECT ON ALL TABLES
TO xxx, and it parses well.
Your desire to figure out how to add stuff to the parser is certainly
commendable, and you might consider continuing down the road you've started on
for that purpose alone. But the desire of others on this list to remain close
to the SQL standard is equally commendable, and unlikely to go away :) If your
main purpose is to get the feature implemented, whether or not you learn how
to add new syntax, you might consider writing a function instead. This
function might take parameters such as the privilege to grant and the user to
grant it to, and be called something like this:
SELECT my_grant_function('someuser', 'someprivilege');
This would do what you need, and in no way conflict with the standard if one
day it covers the feature in question.
- Josh / eggyknap
signature.asc
Description: Digital signature
Re: [HACKERS] adding stuff to parser, question
On 1 Feb 2009, at 00:05, Joshua Tolley wrote:
to add new syntax, you might consider writing a function instead. This
function might take parameters such as the privilege to grant and
the user to
grant it to, and be called something like this:
SELECT my_grant_function('someuser', 'someprivilege');
Well, if you read my first post - I did wrote such a function, and it
works just fine. But still - since that was in TODO, I figured I might
give it a go.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 17:17, Gregory Stark wrote: I don't see any reason offhand why it should have to be a reserved word though. You should be able to make it an UNRESERVED_KEYWORD. Oh, and you'll want to add it to the list of tokens in unreserved_keyword in gram.y as well. removing it from keywords.c and adding it to unserserved_keywords crowd didn't make it... so I'll stick with keywords.c for timebeing. So far I got mostly critique here, even tho - I haven't started much, which is quite sad in a way - because it is not very pro-creative, but I'll still continue on with the patch - whatever the outcome. ta. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
On 31 Jan 2009, at 17:22, David Fetter wrote: Is this part of the SQL:2008? If not, is there something else that is? As far as I can see in the 'free' draft, no. Which is quite funny, cos 'DATABASE' keyword isn't there too.. Anyway. Looks like m$'s sybase clone got it, so I figure - at least some ppl figured it would be a nice feature ;) Can someone lead me into, how should I grab all Oid's for tables (in all user defined schemas, public. and others in db - except for system ones) in /src/backend/catalog/namespace.c please ? (which would probably be the best place to do it.) so far I figured, that adding another ACL_OBJECT define would be the best option, instead of passing NIL in objnames, for which there's an assert at begin of objectNamesToOids(). Reason I am asking about the backend/catalog approach, is because of all structures, and existing code (which I only got to go through Today for first time), I don't see any existing example, that would enumerate 'everything' in specific category. Thanks. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Hi. I am checking that consider sufficient test as Marko-san and it is satisfactory. However, That there is a portion which does not suit the solution of MSVC also understands. Therefore, How is this proposal? 1. )remove pgevent.def It is always generated. 2.) improvement of the Makefile and Solution.pm What do you think? Regards, Hiroshi Saito - Original Message - From: Marko Kreen mark...@gmail.com On 1/31/09, Marko Kreen mark...@gmail.com wrote: On 1/31/09, Magnus Hagander mag...@hagander.net wrote: Hiroshi Saito wrote: Hi. dllwrap --def pgevent.def -o pgevent.dll pgevent.o pgmsgevent.o Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 Warning: resolving _DllUnregisterServer by linking to _dllunregisterser...@0 Use --enable-stdcall-fixup to disable these warnings Use --disable-stdcall-fixup to disable these fixups Warning: resolving _DllRegisterServer by linking to _dllregisterser...@0 What do we have to do to clean this stuff up? I was solved for Marko-san at the time of the work of pgbouncer, and obtained the solution. The patch is this. This appears to be exactly what you are *not* supposed to do. Which is assign an ordinal. See: http://msdn.microsoft.com/en-us/library/8e705t74(VS.71).aspx AFAICS by default the mingw assigns the ordinal anyway? So whats your point? FWIW, I experimented with various approaches on DLL generation, and my theory is now that numeric slot is mandatory for export, name slot optional. And what the above link above warns about is that if you manually assign a numeric slot, you may create gaps in slot table, thus resulting in larger image. But this assumes you already have slots that are manually assigned, which should not happen with pgevent.dll. Does it sound sane? With pgbouncer I used dlltool --export-all-symbols -A to generate the .def file, thus the slots are actually assigned automatically by dlltool, so no gap problems should arise. As Postgres does not generate .def automatically, thus Hiroshi simply posted the resulting .def file, which should not have gap problems. But such detailed .def avoids the confusing warnings from dllwrap. If you are worried about gap problems I suggest instead doing the same and making the .def file auto-generated. -- marko ps. I suggest also tagging the g_module as static. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers pgevent_def_patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] adding stuff to parser, question
Grzegorz Jaskiewicz wrote: On 31 Jan 2009, at 17:22, David Fetter wrote: Is this part of the SQL:2008? If not, is there something else that is? As far as I can see in the 'free' draft, no. Which is quite funny, cos 'DATABASE' keyword isn't there too.. Anyway. Looks like m$'s sybase clone got it, so I figure - at least some ppl figured it would be a nice feature ;) The standard doesn't have the concept of a database. In Postgres, a database is essentially the same as the standard's concept of a catalog. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
Robert Haas wrote: IANAC, but that's my impression too. The simplified patch shouldn't assume that row-level security in its current form is going to end up getting put back in. AFAICS, there's no reason why the security ID for tables can't be a regular attribute in pg_class, or why the security attribute for columns can't be a regular attribute in pg_attribute. If it is identifier, it can be compoundable. I dislike it is held as text. It fundamentaly breaks SE-PostgreSQL's architecture, and requires to scrap near future. I think the column in pg_attribute and pg_class can and should be an OID. The issue is whether it's a regular OID column or a new system column. Stephen and I are saying it should be a regular column. pg_security can stick around to map OIDs to text labels. OK, I accept to omit a facility to save security id on padding field of HeapTupleHeader *in this step*, if is has no other matter unexpected. One melancholic thing is adding a member into pg_proc. It defines more than 2000 of entries which I have to modify correctly. :( Is there any script to help it? Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Ooops, sorry., One mistake:-( Please this. - Original Message - From: Hiroshi Saito z-sa...@guitar.ocn.ne.jp Hi. I am checking that consider sufficient test as Marko-san and it is satisfactory. However, That there is a portion which does not suit the solution of MSVC also understands. Therefore, How is this proposal? 1. )remove pgevent.def It is always generated. 2.) improvement of the Makefile and Solution.pm What do you think? Regards, Hiroshi Saito pgevent_def_patch_v2 Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
* KaiGai Kohei (kai...@kaigai.gr.jp) wrote: One melancholic thing is adding a member into pg_proc. It defines more than 2000 of entries which I have to modify correctly. :( Is there any script to help it? Not that I'm aware of.. You might be able to create an appropriate regex though, esp. if you're adding it at the end.. Stephen signature.asc Description: Digital signature
Re: [HACKERS] How to get SE-PostgreSQL acceptable
KaiGai Kohei wrote: One melancholic thing is adding a member into pg_proc. It defines more than 2000 of entries which I have to modify correctly. :( Is there any script to help it? Last time I added a column to a large catalog, I used a perl script to help me, IIRC, but I haven't kept it. Could be a useful tool if somebody could generalize such a thing. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] How to get SE-PostgreSQL acceptable
Andrew Dunstan wrote: KaiGai Kohei wrote: One melancholic thing is adding a member into pg_proc. It defines more than 2000 of entries which I have to modify correctly. :( Is there any script to help it? Last time I added a column to a large catalog, I used a perl script to help me, IIRC, but I haven't kept it. Could be a useful tool if somebody could generalize such a thing. My preference is sed awk. :) OK, I'll also do it by myself. -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgevent warnings on mingw
Ahh, sorry..like the spam again! I thought over that the existing msvc the did not have uneasines. so, I wish to make it this as correspondence with worried Magnus-san. It is after sufficient test. Regards, Hiroshi Saito - Original Message - From: Hiroshi Saito z-sa...@guitar.ocn.ne.jp Ooops, sorry., One mistake:-( Please this. - Original Message - From: Hiroshi Saito z-sa...@guitar.ocn.ne.jp Hi. I am checking that consider sufficient test as Marko-san and it is satisfactory. However, That there is a portion which does not suit the solution of MSVC also understands. Therefore, How is this proposal? 1. )remove pgevent.def It is always generated. 2.) improvement of the Makefile and Solution.pm What do you think? Regards, Hiroshi Saito -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers pgevent_def_patch_v3 Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] add_path optimization
I've been doing some benchmarking and profiling on the PostgreSQL
query analyzer, and it seems that (at least for the sorts of queries
that I typically run) the dominant cost is add_path(). I've been able
to find two optimizations that seem to help significantly:
1. add_path() often calls compare_fuzzy_path_costs() twice on the same
pair of paths, and when the paths compare equal on one criterion, some
comparisons are duplicated. I've refactored this function to return
the results of both calculations without repeating any floating-point
arithmetic.
2. match_unsorted_outer() adds as many as 5 nested loop joins at a
time with the same set of pathkeys. In my tests, it tended to be ~3 -
cheapest inner, cheapest inner materialized, and cheapest inner index.
Since these all have the same pathkeys, clearly only the one with the
cheapest total cost is in the running for cheapest total cost for that
set of pathkeys, and likewise for startup cost (and the two may be the
same). Yet we compare all of them against the whole pathlist, one
after the other, including (for the most part) the rather expensive
pathkey comparison. I've added a function add_similar_paths() and
refactored match_unsorted_outer() to use it.
On a couple of complex (and proprietary) queries with 12+ joins each,
I measure a planning time improvement of 8-12% with the attached patch
applied. It would be interesting to try to replicate this on a
publicly available data set, but I don't know of a good one to use.
Suggestions welcome - results of performance testing on your own
favorite big queries even more welcome. Simple test harness also
attached. I took the approach of dropping caches, starting the
server, and then running this 5 times each on several queries,
dropping top and bottom results.
...Robert
*** a/src/backend/optimizer/path/joinpath.c
--- b/src/backend/optimizer/path/joinpath.c
***
*** 473,478 match_unsorted_outer(PlannerInfo *root,
--- 473,481
if (nestjoinOK)
{
+ Path *paths[5];
+ int npath = 0;
+
/*
* Always consider a nestloop join with this outer and
* cheapest-total-cost inner. When appropriate, also consider
***
*** 480,486 match_unsorted_outer(PlannerInfo *root,
* cheapest-startup-cost inner path, and the cheapest innerjoin
* indexpaths.
*/
! add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
jointype,
--- 483,489
* cheapest-startup-cost inner path, and the cheapest innerjoin
* indexpaths.
*/
! paths[npath++] = (Path *)
create_nestloop_path(root,
joinrel,
jointype,
***
*** 488,496 match_unsorted_outer(PlannerInfo *root,
outerpath,
inner_cheapest_total,
restrictlist,
! merge_pathkeys));
if (matpath != NULL)
! add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
jointype,
--- 491,499
outerpath,
inner_cheapest_total,
restrictlist,
! merge_pathkeys);
if (matpath != NULL)
! paths[npath++] = (Path *)
create_nestloop_path(root,
joinrel,
jointype,
***
*** 498,506 match_unsorted_outer(PlannerInfo *root,
outerpath,
matpath,
restrictlist,
! merge_pathkeys));
if (inner_cheapest_startup != inner_cheapest_total)
! add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
jointype,
--- 501,509
outerpath,
matpath,
restrictlist,
! merge_pathkeys);
if (inner_cheapest_startup != inner_cheapest_total)
! paths[npath++] = (Path *)
create_nestloop_path(root,
joinrel,
jointype,
***
*** 508,516 match_unsorted_outer(PlannerInfo *root,
outerpath,
inner_cheapest_startup,
restrictlist,
! merge_pathkeys));
if (index_cheapest_total != NULL)
! add_path(joinrel, (Path *)
create_nestloop_path(root,
joinrel,
jointype,
--- 511,519
outerpath,
inner_cheapest_startup,
restrictlist,
! merge_pathkeys);
if (index_cheapest_total != NULL)
! paths[npath++] = (Path *)
create_nestloop_path(root,
joinrel,
jointype,
***
*** 518,527 match_unsorted_outer(PlannerInfo *root,
outerpath,
index_cheapest_total,
restrictlist,
! merge_pathkeys));
if (index_cheapest_startup != NULL
index_cheapest_startup != index_cheapest_total)
! add_path(joinrel, (Path *)
Re: [HACKERS] Updated backslash consistency patch
On 23 Jan 2009, at 00:03, Tom Lane wrote: Stephen Frost sfr...@snowman.net writes: Seeing this list reminded me of a pet-peeve.. \du and \dg actually show the same info, that's fine, but neither of them show the rolcanlogin value. +1 for fixing that. was it that easy, or I got it wrong? :) psql_du_conlogin.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
