Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
I have similar patch and it works. There is two isues: * we missing column in pg_proc about state (not all procedures are obfuscated), I solved it for plpgsl with using probin. * decrypt is expensive on language handler level. Every session have to do it again and again, better decrypt in system cache or somewhere there. Regards Pavel Stehule ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pgwin32_open returning EINVAL
On Fri, Dec 21, 2007 at 01:30:07AM -0500, Jaime Casanova wrote: On Dec 20, 2007 9:31 PM, Alvaro Herrera [EMAIL PROTECTED] wrote: Jaime Casanova escribió: it doesn't compile on current head on mingw 5.1 and msys 1.0.10; of course, it doesn't compile on 8.2 neither in order to Alvaro's contact to test it I think this patch should fix it. I think win32.mak needs to be similarly patched. ok. it's solved now. i added the lines in the patch and pgsleep.o in (circa line 34 in src/interfaces/libpq) OBJS= fe-auth.o fe-connect.o fe-exec.o fe-misc.o fe-print.o fe-lobj.o \ fe-protocol2.o fe-protocol3.o pqexpbuffer.o pqsignal.o fe-secure.o \ md5.o ip.o wchar.o encnames.o noblock.o pgstrcasecmp.o thread.o \ $(filter crypt.o getaddrinfo.o inet_aton.o open.o snprintf.o strerror.o strlcpy.o win32error.o pgsleep.o, $(LIBOBJS)) That's the wrong place - it should be added for win32 only. Proper patch committed. Sorry 'bout that one. *grumble* //Magnus ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pgwin32_open returning EINVAL
Andrew Dunstan escribió: Alvaro Herrera wrote: I think this patch should fix it. I think win32.mak needs to be similarly patched. Don't you also need to add pgsleep.o to $(OBJS) in the win32 stanza? Hmm. Wow, that's silly. I introduced a hack in a Replicator's Makefile to avoid having to duplicate the file list. I'll look into submitting it into libpq's for 8.4. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Sorting Improvements for 8.4
Brian Hurt wrote: While we're blue skying things, I've had an idea for a sorting algorithm kicking around for a couple of years that might be interesting. It's a variation on heapsort to make it significantly more block-friendly. I have no idea if the idea would work, or how well it'd work, but it might be worthwhile kicking around. Now, the core idea of heapsort is that the array is put into heap order- basically, that a[i] = a[2i+1] and a[i] = a[2i+2] (doing the 0-based array version here). The problem is that, assuming that the length of a is larger than memory, then a[2i+1] is likely going to be on a different page or block than a[i]. That means every time you have to bubble down a new element, you end up reading O(log N) blocks- this is *per element*. The variation is to instead work with blocks, so you have a block of entries b[i], and you change the definition of heap order, so that min(b[i]) = max(b[2i+1]) and min(b[i]) = max(b[2i+2]). Also, during bubble down, you need to be carefull to only change the minimum value of one of the two child blocks b[2i+1] and b[2i+2]. Other than that, the algorithm works as normal. The advantage of doing it this way is that while each bubble down still takes O(log N) blocks being touched, you get a entire block worth of results for your effort. Make your blocks large enough (say, 1/4 the size of workmem) and you greatly reduce N, the number of blocks you have to deal with, and get much better I/O (when you're reading, you're reading megabytes at a shot). Now, there are boatloads of complexities I'm glossing over here. This is more of a sketch of the idea. But it's something to consider. Following up to myself (my apologies), but it's occurred to me that there are three advantages to this proposal that I've since thought of: 1) The two child blocks b[2i+1] and b[2i+2]- the one with the larger minimum element is the one we might replace. In other words, if min(b[2i+1]) min(b[2i+2]) and min(b[i]) min(b[2i+1]), then we know we're going to want the blocks b[4i+3] and b[4i+4]- before we're done with blocks b[2i+1] and b[2i+2]. The point here is that this would work wonders with the posix_fadvise/asyncio ideas kicking around. It'd be easy for the code to keep 2 large writes and 2 large reads going pretty constantly. 2) There is some easy parallelization available. I'm not sure how much worth this is, but the bubble down code is fairly easy to parallelize. If we have two bubble-downs going on in parallel, once they go down different branches (one thread goes to block b[2i+1] while the other goes to b[2i+2]) they no longer interact. Blocks near the root of the heap would be contended over, and multiple threads means smaller blocks to keep the total memory foot print the same. Personally, I think the asyncio idea above is more likely to be worthwhile. 3) It's possible to perform the sort lazily. You have the initial O(N) pass over the list, but then each block is only O(log N) cost. If it's likely that only the first part of the result is needed, then much of the work can be avoided. Brian ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] pgindent issue with EXEC_BACKEND-only typedefs
Alvaro Herrera wrote: I don't know how to make it output the symbol names like it seems to do for you. Having the typedef list in the script itself seems like a barrier for other people to contribute to this thing. I wonder if that can be changed so that the typedef is on a separate list. Done and committed. (Why are we still distributing pgjindent anyway?) Removed. I am also wondering if current GNU indent is now better suited to the task. Perhaps the bugs that it had on earlier versions have since been fixed? I remember checking the source code size a couple of years ago and it had grown by an order of magnitude or something like that. Yes, we should give that another try. I will keep this as an 8.4 item. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Dec 21, 2007 3:18 AM, Pavel Stehule [EMAIL PROTECTED] wrote: I have similar patch and it works. There is two isues: * we missing column in pg_proc about state (not all procedures are obfuscated), I solved it for plpgsl with using probin. I was hoping to avoid making any catalog or other changes to support encryption specifically. Maybe your patch stands on its own merits...I missed the original discussion. Do you think the code you wrote can be adapted to do other things besides encryption? * decrypt is expensive on language handler level. Every session have to do it again and again, better decrypt in system cache or somewhere there. Doesn't bother me in the least...and caching unencrypted data is scary. Also, aes256 is pretty fast for what it gives you and function bodies are normally short. The real issue as I see it is where to keep the key. How did you handle that? merlin ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] pgindent issue with EXEC_BACKEND-only typedefs
Alvaro Herrera wrote: Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: I don't know how to make it output the symbol names like it seems to do for you. I dislike the object-file-based approach altogether, not least because it appears to depend on unportable aspects of someBSD's objdump. Surely there's some code out there that can find typedef names from the source files? Why does pgindent even need to be told this? indent needs the typedef list. Maybe we can hack something based on typedefs in the source code, instead of object files. I am worried it will be too hard to find typedefs in complex cases where the typedef name is embedded: typedef void (*ClosePtr) (struct _archiveHandle * AH); The only think of is to grab typedefs from the object file and then also try to get them from the souce too and combine them and remove duplicates. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] pgindent issue with EXEC_BACKEND-only typedefs
Bruce Momjian [EMAIL PROTECTED] writes:
Alvaro Herrera wrote:
indent needs the typedef list. Maybe we can hack something based on
typedefs in the source code, instead of object files.
The only think of is to grab typedefs from the object file and then also
try to get them from the souce too and combine them and remove duplicates.
Something I noticed the other day is that pgindent doesn't seem to treat
struct foo or union foo as a type name, which is pretty silly
because no context is needed at all to recognize that. We tend not to
do that too much --- the project style is to use a typedef name --- but
there are some places that do it, particularly the regex code. For
instance there are extra spaces here:
static void
cmtreefree(struct colormap * cm,
union tree * tree,
int level)/* level number (top == 0) of this block */
{
Fixable?
regards, tom lane
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On 21/12/2007, Merlin Moncure [EMAIL PROTECTED] wrote: On Dec 21, 2007 3:18 AM, Pavel Stehule [EMAIL PROTECTED] wrote: I have similar patch and it works. There is two isues: * we missing column in pg_proc about state (not all procedures are obfuscated), I solved it for plpgsl with using probin. I was hoping to avoid making any catalog or other changes to support encryption specifically. Maybe your patch stands on its own merits...I missed the original discussion. Do you think the code you wrote can be adapted to do other things besides encryption? I don't know. It was fast hack that just works. It hat to do obfuscation, and it do it well. * decrypt is expensive on language handler level. Every session have to do it again and again, better decrypt in system cache or somewhere there. Doesn't bother me in the least...and caching unencrypted data is scary. Also, aes256 is pretty fast for what it gives you and function bodies are normally short. The real issue as I see it is where to keep the key. How did you handle that? merlin Simply. I use for password some random plpgsql message text and compile it. I though about GUC, and about storing password in postgresql.conf. It's equal to protection level. We cannot protect code on 100%. If you have admin or superuser account and if you know some internal, you can simply get code. http://blog.pgsql.cz/index.php?/archives/10-Obfuscator-PLpgSQL-procedur.html#extended sorry for czech desc Pavel ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Fri, Dec 21, 2007 at 12:09:28AM -0500, Merlin Moncure wrote: Maybe a key management solution isn't required. If, instead of strictly wrapping a language with an encryption layer, we provide hooks (actors) that have the ability to operate on the function body when it arrives and leaves pg_proc, we may sidestep the key problem (leaving it to the user) and open up the doors to new functionality at the same time. I like this idea much better, because the same basic mechanism can be used for more than one thing, and it doesn't build in a system that is fundamentally weak. Of course, you _can_ build a weak system this way, but there's an important difference between building a fundamentally weak system and making weak systems possible. A ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
Pavel Stehule [EMAIL PROTECTED] writes: On 21/12/2007, Merlin Moncure [EMAIL PROTECTED] wrote: ... The real issue as I see it is where to keep the key. How did you handle that? Simply. I use for password some random plpgsql message text and compile it. I though about GUC, and about storing password in postgresql.conf. It's equal to protection level. We cannot protect code on 100%. If you have admin or superuser account and if you know some internal, you can simply get code. Yeah. There is no defense against someone who is prepared to go in there with a debugger and pull the post-decryption code out of memory. So what we need to think about is what sorts of threats we *can* or should defend against. A couple of goals that seem like they might be reasonable are: * Even a superuser can't get the code at the SQL level, ie, it's secure if you rule out debugger-level attacks. (For example, this might prevent someone who had remotely breached the superuser account from getting the code.) * Code not available if you just look at what's on-disk, ie, you can't get it by stealing a backup tape. Any other threats we could consider defending against? BTW, this thread definitely doesn't belong on -performance anymore. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Fri, Dec 21, 2007 at 12:40:05AM -0500, Tom Lane wrote: whether there is a useful policy for it to implement. Andrew Sullivan argued upthread that we cannot get anywhere with both keys and encrypted function bodies stored in the same database (I hope that's an adequate summary of his point). It is. I'm not a security expert, but I've been spending some time listening to some of them lately. The fundamental problem with a system that stores the keys online in the same repository is not just its potential for compromise, but its brittle failure mode: once the key is recovered, you're hosed. And there's no outside check of key validity, which means attackers have a nicely-contained target to hit. I'm not convinced that he's right, but that has to be the first issue we think about. The whole thing is a dead end if there's no way to do meaningful encryption --- punting an insoluble problem to the user doesn't make it better. Well, one thing you could do with the proposal is build a PKCS#11 actor, that could talk to an HSM. Not everyone needs HSMs, of course, but they do make online key storage much less risky (because correctly designed ones make key recovery practically impossible). So the mechanism can be made effectively secure even for very strong cryptographic uses. Weaker cases might use a two-level key approach, with a data-signing key online all the time to do the basic encryption and validation, but a key-signing key that is always offline or otherwise unavailable from within the system. The key signing key only authenticates (and doesn't encrypt) the data signing key. You could use a different actor for this, to provide an interface to one-way functions or something. This gives you a way to revoke a data-signing key. You couldn't protect already compromised data this way, but at least you could prevent new disclosures. Yes, I'm being hand-wavy now, but I can at least see how these different approaches are possible under the suggestion, so it seems like a possibly fruitful avenue to explore. The more I think about it, actually, the more I like it. A ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
Andrew Sullivan [EMAIL PROTECTED] writes: On Fri, Dec 21, 2007 at 12:09:28AM -0500, Merlin Moncure wrote: Maybe a key management solution isn't required. I like this idea much better, because the same basic mechanism can be used for more than one thing, and it doesn't build in a system that is fundamentally weak. Of course, you _can_ build a weak system this way, but there's an important difference between building a fundamentally weak system and making weak systems possible. I find myself unconvinced by this argument. The main problem is: how do we know that it's possible to build a strong system atop this mechanism? Just leaving it to non-security-savvy users seems to me to be a great way to guarantee a lot of weak systems in the field. ISTM our minimum responsibility would be to design and document how to build a strong protection system using the feature ... and at that point why not build it in? I've certainly got no objection to making a mechanism that can be used for more than one purpose; but not offering a complete security solution is abdicating our responsibility. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Dec 21, 2007 11:48 AM, Andrew Sullivan [EMAIL PROTECTED] wrote: On Fri, Dec 21, 2007 at 12:40:05AM -0500, Tom Lane wrote: whether there is a useful policy for it to implement. Andrew Sullivan argued upthread that we cannot get anywhere with both keys and encrypted function bodies stored in the same database (I hope that's an adequate summary of his point). It is. I'm not a security expert, but I've been spending some time listening to some of them lately. The fundamental problem with a system that stores the keys online in the same repository is not just its potential for compromise, but its brittle failure mode: once the key is recovered, you're hosed. And there's no outside check of key validity, which means attackers have a nicely-contained target to hit. I'm not convinced that he's right, but that has to be the first issue we think about. The whole thing is a dead end if there's no way to do meaningful encryption --- punting an insoluble problem to the user doesn't make it better. Well, one thing you could do with the proposal is build a PKCS#11 actor, that could talk to an HSM. Not everyone needs HSMs, of course, but they do make online key storage much less risky (because correctly designed ones make key recovery practically impossible). So the mechanism can be made effectively secure even for very strong cryptographic uses. ISTM the main issue is how exactly the authenticated user interacts with the actor to give it the information it needs to get the real key. This is significant because we don't want to be boxed into an actor implementation that doesn't allow that interaction. If simply calling out via a function is enough (which, to be perfectly honest, I don't know), then we can implement the actor system and let actor implementations spring to life in contrib, pgfoundry, etc. as the community presents them. merlin ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[HACKERS] Postgres.bki
Hello, I am changing the file pg_index, it is necessary to generate a new file postgres.bki? How to generate? Using the command: ./genbki.sh Thanks -- Pedro Belmino. # Ciência da Computação - UNIFOR # [EMAIL PROTECTED]
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
Merlin Moncure [EMAIL PROTECTED] writes: ISTM the main issue is how exactly the authenticated user interacts with the actor to give it the information it needs to get the real key. This is significant because we don't want to be boxed into an actor implementation that doesn't allow that interaction. We don't? What purpose would such a setup serve? I would think that for the applications we have in mind, the *last* thing you want is for the end user to hold the key. The whole point of this is to keep him from seeing the function source code, remember? Andrew's suggestion of an outside-the-database key server is apropos, but I think it would end up being a situation where the key server is under the control of whoever wrote the function and wants to guard it against the end user. The key server would want some kind of authentication token but I think that could perfectly well be an ID for the database server, not the individual end user. There's no need for anything as awkward as an interactive sign-on, AFAICS. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] Postgres.bki
Pedro Belmino [EMAIL PROTECTED] writes: I am changing the file pg_index, it is necessary to generate a new file postgres.bki? How to generate? make will take care of it. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Fri, Dec 21, 2007 at 01:57:44PM -0500, Tom Lane wrote: Merlin Moncure [EMAIL PROTECTED] writes: ISTM the main issue is how exactly the authenticated user interacts with the actor to give it the information it needs to get the real key. This is significant because we don't want to be boxed into an actor implementation that doesn't allow that interaction. We don't? What purpose would such a setup serve? I would think that for the applications we have in mind, the *last* thing you want is for the end user to hold the key. The whole point of this is to keep him from seeing the function source code, remember? Hmm; this may be exactly part of the problem, though. It seems there are two possible cases in play: 1. Protect the content in the database (in this case, function bodies) from _all_ users on a given server. This is a case where you want to protect (say) your function body from your users, because you have a closed-source application. 2. Protect the content of a field from _some_ users on a given system, based on the permissions they hold. This is roughly analagous to others not being able to look in the table I created, because I haven't GRANTed them permission. (2) is really a case for column-level access controls, I guess. But if we're trying to solve this problem too, then user passwords or something make sense. A ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
Andrew Sullivan [EMAIL PROTECTED] writes: Hmm; this may be exactly part of the problem, though. It seems there are two possible cases in play: 1.Protect the content in the database (in this case, function bodies) from _all_ users on a given server. This is a case where you want to protect (say) your function body from your users, because you have a closed-source application. 2.Protect the content of a field from _some_ users on a given system, based on the permissions they hold. This is roughly analagous to others not being able to look in the table I created, because I haven't GRANTed them permission. I would argue that (2) is reasonably well served today by setting up separate databases for separate users. The people who are complaining seem to want to send out a set of functions into a hostile environment, which is surely case (1). regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 21 Dec 2007 16:47:46 -0500 Andrew Sullivan [EMAIL PROTECTED] wrote: On Fri, Dec 21, 2007 at 04:19:51PM -0500, Tom Lane wrote: 2.Protect the content of a field from _some_ users on a given system, I would argue that (2) is reasonably well served today by setting up separate databases for separate users. I thought actually this was one of the use-cases we were hearing. Different people using the same database (because the same data), with rules about the different staff being able to see this or that function body. I can easily imagine such a case, for instance, in a large organization with different departments and different responsibilities. It seems a shame that the only answer we have there is, Give them different databases. I think there is a fundamental disconnect here. The Give them different databases argument is essentially useless. Consider a organization that has Sales and HR. You don't give both a separate database. They need access to some of each others information. Just not all. Sincerely, Joshua D. Drake - -- The PostgreSQL Company: Since 1997, http://www.commandprompt.com/ Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240 Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate SELECT 'Training', 'Consulting' FROM vendor WHERE name = 'CMD' -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFHbDkOATb/zqfZUUQRAqzmAJ9VhNXYtr/N7px3/iUenUJN+7r9jQCfQtj+ Hyfo4fLGrBGUN4jJcSgZEh0= =xeLs -END PGP SIGNATURE- ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] pgindent issue with EXEC_BACKEND-only typedefs
Tom Lane wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
Alvaro Herrera wrote:
indent needs the typedef list. Maybe we can hack something based on
typedefs in the source code, instead of object files.
The only think of is to grab typedefs from the object file and then also
try to get them from the souce too and combine them and remove duplicates.
Something I noticed the other day is that pgindent doesn't seem to treat
struct foo or union foo as a type name, which is pretty silly
because no context is needed at all to recognize that. We tend not to
do that too much --- the project style is to use a typedef name --- but
there are some places that do it, particularly the regex code. For
instance there are extra spaces here:
static void
cmtreefree(struct colormap * cm,
union tree * tree,
int level)/* level number (top == 0) of this block */
{
Fixable?
Yes, I found those are 't' STABS rather than T which are used in cases
where you do typedef struct {} name. The next pgindent will have those
typedefs you want.
--
Bruce Momjian [EMAIL PROTECTED]http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com
+ If your life is a hard drive, Christ can be your backup. +
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Fri, Dec 21, 2007 at 04:19:51PM -0500, Tom Lane wrote: 2. Protect the content of a field from _some_ users on a given system, I would argue that (2) is reasonably well served today by setting up separate databases for separate users. I thought actually this was one of the use-cases we were hearing. Different people using the same database (because the same data), with rules about the different staff being able to see this or that function body. I can easily imagine such a case, for instance, in a large organization with different departments and different responsibilities. It seems a shame that the only answer we have there is, Give them different databases. I actually think organizations that think keeping function bodies secret like this to be a good idea are organizations that will eventually make really stupid mistakes. But that doesn't mean they're not under the legal requirement to do this. For instance, my current employer has (externally-mandated) organizational conflict of interest rules that require all disclosure to be done exclusively as need to know. Under the right (!) legal guidance, such a requirement could easily lead to rules about function-body disclosure. From my point of view, such a use case is way more compelling than function-body encryption (although I understand that one too). A ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Proposal for Null Bitmap Optimization(for TrailingNULLs)
On Dec 20, 2007, at 2:36 AM, Gokulakannan Somasundaram wrote: I checked it by creating a table with 10 columns on a 32 bit machine. i inserted 100,000 rows with trailing nulls and i observed savings of 400Kbytes. That doesn't really tell us anything... how big was the table originally? Also, testing on 64 bit would be interesting. -- Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED] Give your computer some brain candy! www.distributed.net Team #1828 smime.p7s Description: S/MIME cryptographic signature
Re: [HACKERS] function body actors (was: [PERFORM] viewing source code)
On Fri, 2007-21-12 at 18:05 -0400, Andrew Sullivan [EMAIL PROTECTED] wrote: 2. Protect the content of a field from _some_ users on a given system, I would argue that (2) is reasonably well served today by setting up separate databases for separate users. I thought actually this was one of the use-cases we were hearing. Different people using the same database (because the same data), with rules about the different staff being able to see this or that function body. I can easily imagine such a case, for instance, in a large organization with different departments and different responsibilities. It seems a shame that the only answer we have there is, Give them different databases. There is also Veil: http://veil.projects.postgresql.org/curdocs/index.html __ Marc signature.asc Description: This is a digitally signed message part
Re: [HACKERS] New style of hash join proposal
Decibel! wrote: I fear the real complexity would be (as always) in the planner rather than the executor. I haven't really looked into what it would take to arrange this or how to decide when to do it. TODO? This email was added to the 8.4 queue: http://momjian.postgresql.org/cgi-bin/pgpatches_hold Once we are in 8.4 development we can decide if we are going to implement it or add it to the TODO list. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] timetz range check issue
When inserting a timetz in binary mode, there are no range checks on the time
value (nor on the zone). In text mode, things are fine:
postgres=# insert into t values ('24:00:00.01-05'::timetz);
ERROR: date/time field value out of range: 24:00:00.01-05
// 24:00:00.01-05
double d = ((24 * MINS_PER_HOUR + 0) * SECS_PER_MINUTE) + 0 + 0.01);
postgres=# select * from t;
val
24:00:00.01-05
It displays it correctly, but it doesn't seem like proper behavior. If I insert
25:00:00.00-05, I get nothing:
postgres=# select * from t;
val
-
(1 row)
QUERY-LOG:
LOG: execute unnamed: INSERT INTO timetz_test VALUES ($1)
DETAIL: parameters: $1 = 'p£ '
I get no error from libpq. I think a range check is needed in timetz_recv
time_recv.
version: (interger_datetimes=off)
PostgreSQL 8.3devel on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.1.1
20070105 (Red Hat 4.1.1-52)
Andrew
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] pgsql: Improve wording.
Martijn van Oosterhout wrote: -- Start of PGP signed section. On Tue, Dec 18, 2007 at 09:32:51AM -0300, Alvaro Herrera wrote: Martijn van Oosterhout wrote: On Mon, Dec 17, 2007 at 01:48:31PM +, Alvaro Herrera wrote: Log Message: --- Improve wording. I agree that the parenthised phrase should be removed. http://developer.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/runtime.sgml?r1=1.387r2=1.391 That's fine with me. It's just that I figured that if we're going to add something, we should at least add something factually accurate. I think your proposed wording is removed enough from what the complainant was saying that it is not worth to stick it in. The point here is, to what extent do we want to spoon-feed careless sysadmins? Not sure what the complaint was. The only useful advice I can think of is that before turning it on you need to makes sure you have lots of swap. For the rest, I think the wording was fine as it was. We are back to the old wording, but a link to an OOM article was added: http://lwn.net/Articles/104179/ -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] pgsql: Improve wording.
Alvaro Herrera wrote: Martijn van Oosterhout wrote: On Mon, Dec 17, 2007 at 01:48:31PM +, Alvaro Herrera wrote: Log Message: --- Improve wording. I'd suggest removing everything between the parentheses, or perhaps something like: By tracking allocated memory rather than used memory it removes the possibilty of failing in a page fault. This option also require you allocate enoghh swap to cover all possible eventualities, though in production it will not be used. I agree that the parenthised phrase should be removed. http://developer.postgresql.org/cvsweb.cgi/pgsql/doc/src/sgml/runtime.sgml?r1=1.387r2=1.391 I think your proposed wording is removed enough from what the complainant was saying that it is not worth to stick it in. The point here is, to what extent do we want to spoon-feed careless sysadmins? OK, I have removed the paratheses paragraph about fork() and added link to an article about the OOM: http://lwn.net/Articles/104179/ On Linux 2.6 and later, an additional measure is to modify the kernel's behavior so that it will not quoteovercommit/ memory. Although this setting will not prevent the ulink url=http://lwn.net/Articles/104179/;OOM killer/ from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via commandsysctl/command: -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] timetz range check issue
Andrew Chernow [EMAIL PROTECTED] writes: I think a range check is needed in timetz_recv time_recv. I think that the design philosophy for the binary I/O code is to be as fast as safely possible, and accordingly range-checks are present only where needed for the backend to defend itself. Is there anything that goes horribly wrong if a client shoves a bad zone offset at us? (If we want to change this philosophy, I won't necessarily quibble, but I don't think these two recv routines are the only ones that would need to be tightened up.) regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [HACKERS] pgsql: Improve wording.
bruce wrote: I think your proposed wording is removed enough from what the complainant was saying that it is not worth to stick it in. The point here is, to what extent do we want to spoon-feed careless sysadmins? OK, I have removed the paratheses paragraph about fork() and added link to an article about the OOM: http://lwn.net/Articles/104179/ On Linux 2.6 and later, an additional measure is to modify the kernel's behavior so that it will not quoteovercommit/ memory. Although this setting will not prevent the ulink url=http://lwn.net/Articles/104179/;OOM killer/ from being invoked altogether, it will lower the chances significantly and will therefore lead to more robust system behavior. This is done by selecting strict overcommit mode via commandsysctl/command: I just read that article in full and it does a better job of explaining OOM than we ever could: :-) Not everybody agrees that this (OOM) is a fruitful use of developer time. Andries Brouwer came up with this analogy: An aircraft company discovered that it was cheaper to fly its planes with less fuel on board. The planes would be lighter and use less fuel and money was saved. On rare occasions however the amount of fuel was insufficient, and the plane would crash. This problem was solved by the engineers of the company by the development of a special OOF (out-of-fuel) mechanism. In emergency cases a passenger was selected and thrown out of the plane. (When necessary, the procedure was repeated.) A large body of theory was developed and many publications were devoted to the problem of properly selecting the victim to be ejected. Should the victim be chosen at random? Or should one choose the heaviest person? Or the oldest? Should passengers pay in order not to be ejected, so that the victim would be the poorest on board? And if for example the heaviest person was chosen, should there be a special exception in case that was the pilot? Should first class passengers be exempted? Now that the OOF mechanism existed, it would be activated every now and then, and eject passengers even when there was no fuel shortage. The engineers are still studying precisely how this malfunction is caused. -- Bruce Momjian [EMAIL PROTECTED]http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] binary decode
I'm working on a decoder to take a raw main/base file and given table format parameters to pull out relevant data. My question is whether anyone has developed such a tool. Something that takes the raw file and table format as input and creates an ascii dump (similar to pgdump). The purpose is to pull valid data out of raw postregsql files in order to work on a recovery from a bad disk corruption which resulted in the database being irreparable. ---(end of broadcast)--- TIP 6: explain analyze is your friend
