Re: [HACKERS] why was libpq.so's version number bumped?
Neil Conway [EMAIL PROTECTED] writes: Christopher Kings-Lynne said: There have been HEAPS of security fixes between 7.2 and 7.3. That's only the case if your definition of a security fix is pretty fast and loose -- as yours seems to be. Hmm? On 7.2, an unpriviliged database user can read the more or less the whole memory image of the server process. On 7.3, this has been fixed. -- Florian Weimer[EMAIL PROTECTED] University of Stuttgart http://CERT.Uni-Stuttgart.DE/people/fw/ RUS-CERT fax +49-711-685-5898 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] why was libpq.so's version number bumped?
Palle Girgensohn [EMAIL PROTECTED] writes: One of the reasons I ask is, if it is a good reason, like say security, maybe I can persuade the FreeBSD port responsible guys to bring the port into the upcoming FreeBSD 5.0 release. 7.3 is not completely compatible with 7.2 at the SQL level, and quite a few things break. I wouldn't force users to switch right now. -- Florian Weimer[EMAIL PROTECTED] University of Stuttgart http://CERT.Uni-Stuttgart.DE/people/fw/ RUS-CERT fax +49-711-685-5898 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] python interface
On Tuesday 17 December 2002 14:10, Bruce Momjian wrote:
I think the python interface in /interfaces/python should be moved to
gborg. It already has its own web site:
http://www.druid.net/pygresql/
I would love to add PyGreSQL to gborg but no matter how often I log in it
tells me that I need to be logged in to create a new project.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Upgrading rant.
Tom Lane wrote: Well, this is exactly the issue: someone would have to put substantial amounts of time into update mechanisms and/or maintenance of obsolete versions, as opposed to features, performance improvements, or bug fixes. Personally, I feel that if we weren't working as hard as we could on features/performance/bugfixes, the upgrade issue would be moot because there wouldn't *be* any reason to upgrade. So I'm not planning to redirect my priorities. But this is an open source project and every developer gets to set their own priorities. If you can persuade someone to put their time into that, go for it. Do not under estimate the upgrade issue. I think it is huge and a LOT of people have problems with it. Personally, I never understood why the dump/restore needed to happen in the first place. Can't the data and index file format be more rigidly defined and stuck too? Can't there just be some BKI process to add new data entries? I had the same issues with 7.1 and 7.2, ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Upgrading rant.
On Fri, 2003-01-03 at 13:45, mlw wrote: Tom Lane wrote: Personally, I feel that if we weren't working as hard as we could on features/performance/bugfixes, the upgrade issue would be moot because there wouldn't *be* any reason to upgrade. What about the standard Microsoft reason for upgrades - the bug fixes ;) So I'm not planning to redirect my priorities. But this is an open source project and every developer gets to set their own priorities. If you can persuade someone to put their time into that, go for it. Do not under estimate the upgrade issue. Very true! If upgrading is hard, users will surely expect us to keep maintaining all non-upgradable old versions for the foreseeable future ;( I think it is huge and a LOT of people have problems with it. Personally, I never understood why the dump/restore needed to happen in the first place. Can't the data and index file format be more rigidly defined and stuck too? I don't think the main issues are with file _formats_ but rather with system file structures - AFAIK it is a fundamental design decision (arguably a design flaw ;( ) that we use system tables straight from page cache via C structure pointers, even though there seems to be a layer called storage Manager which should hide the on-disk format completely. Can't there just be some BKI process to add new data entries? I had the same issues with 7.1 and 7.2, -- Hannu Krosing [EMAIL PROTECTED] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Threads
Hi all,
I am sure, many of you would like to delete this message before reading, hold
on. :-)
There is much talk about threading on this list and the idea is always
deferred for want of robust thread models across all supported platforms and
feasibility of gains v/s efforts required.
I think threads are useful in difference situations namely parallelising
blocking conditions and using multiple CPUs.
Attached is a framework that I ported to C from a C++ server I have written.
It has threadpool and threads implementation based on pthreads.
This code expects minimum pthreads implementation and does not assume anything
on threads part (e.g kernel threads or not etc.)
I request hackers on this list to take a look at it. It should be easily
pluggable in any source code and is released without any strings for any use.
This framework allows to plug-in the worker function and argument on the fly.
The threads created are sleeping by default and can be woken up s and when
required.
I propose to use it incrementally in postgresql. Let's start with I/O. When a
block of data is being read, rather than blocking for read, we can set up
creator-consumer link between two threads That we way can utilize that I/O
time in a overlapped fashion.
Further threads can be useful when the server has more CPUs. It can spread CPU
intensive work to different threads such as index creation or sorting. This
way we can utilise idle CPU which we can not as of now.
There are many advantages that I can see.
1)Threads can be optionally turned on/off depending upon the configuration. So
we can entirely keep existing functionality and convert them one-by-one to
threaded application.
2)For each functionality we can have two code branches, one that do not use
threads i.e. current code base and one that can use threads. Agreed the
binary will be bit bloated but that would give enormous flexibility. If we
find a thread implementation buggy, we simply switch it off either in
compilation or inconfiguration.
3) Not much efforts should be required to plug code into this model. The idea
of using threads is to assign exclusive work to each thread. So that should
not require much of a locking.
In case of using multiple CPUs, separate functions need be written that can
handle the things in a thread-safe fashion. Also a merger function would be
required which would merge results of worker threads. That would be totally
additional.
I would say two threads per CPU per back-end should be a reasonable default as
that would cover I/O blocking well. Of course unless threading is turned off
in build or in configuration.
Please note that I have tested the code in C++ and my C is rusty. Quite likely
there are bugs in the code. I will stress test the code on monday but I would
like to seek an opinion on this as soon as possible. ( Hey but it compiles
clean..)
If required I can post example usage of this code, but I don't think that
should be necessary.:-)
Bye
Shridhar
#define _REENTRANT
#include stdio.h
#include stdlib.h
#include pthread.h
#include unistd.h
//typedefs
typedef void* (*function)(void *);
typedef void* argtype;
typedef struct
{
pthread_mutex_t lock;
pthread_cond_t cond;
unsigned short freeCount,n,count;
void *pool;
} threadPool;
typedef struct
{
pthread_t t;
pthread_attr_t tattr;
pthread_mutex_t lock;
pthread_cond_t cond;
argtype arg;
function f;
unsigned short quit;
threadPool *p;
} thread;
/*Thread functions*/
void initThread(thread **t,threadPool *pool);
void deleteThread(thread **t);
void stop(thread *thr);
void wakeForWork(thread *thr,function func,argtype a);
argtype runner(void *ptr);
/*thread pool functions*/
void initPool(threadPool **pool,unsigned short numthreads);
void deletePool(threadPool **p);
void putThread(threadPool *p,thread *t);
thread *getThread(threadPool *p);
#include thread.h
void initThread(thread **t,threadPool *pool)
{
thread *thr=(thread *)malloc(sizeof(thread));
if(!thr)
{
fprintf(stderr,\nCan not allocate memory for thread. Quitting...\n);
exit(1);
}
*t=thr;
pthread_attr_init((thr-tattr));
pthread_mutex_init((thr-lock), NULL);
pthread_cond_init((thr-cond), NULL);
pthread_attr_setdetachstate((thr-tattr),PTHREAD_CREATE_DETACHED);
thr-quit=0;
thr-p=pool;
//Create the thread
int ret=pthread_create((thr-t),(thr-tattr),runner,(void *)thr);
if(ret!=0)
{
fprintf(stderr,\nCan not create thread. Quitting...\n);
exit(1);
}
}
void deleteThread(thread **t)
{
thread *thr=*t;
if(!t) return;
stop(thr);
pthread_attr_destroy((thr-tattr));
pthread_cond_destroy((thr-cond));
pthread_mutex_destroy((thr-lock));
free(thr);
}
void stop(thread *thr)
{
unsigned short i;
thr-quit=1;
pthread_cond_signal((thr-cond));
for(i=0;thr-quit i10;i++)
{
if(i=10)
{
pthread_kill(thr-t,9);
break;
}
usleep(400);
}
}
void wakeForWork(thread *thr,function func,argtype a)
{
thr-f=func;
thr-arg=a;
[HACKERS] complie error on windows
Hi, I am trying to install postgresql-7.3 on windows and I keep getting the following error despite having downloaded a compiler. Can anyone tell me what I am not doing right? I am a newbie to postgres and development. My ultimate goal is to create a data driven application utilizing the J2EE architecture. $ ./configure checking build system type... i686-pc-cygwin checking host system type... i686-pc-cygwin checking which template to use... win checking whether to build with 64-bit integer date/time support... no checking whether to build with recode support... no checking whether NLS is wanted... no checking for default port number... 5432 checking for default soft limit on number of connections... 32 checking for gcc... no checking for cc... no configure: error: no acceptable C compiler found in $PATH Thanks, Al ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] complie error on windows
If you run, gcc, at the prompt (preferably the one you're trying to run configure from), do you get something like, gcc: No input files or do you get, gcc: command not found? If you get the later (or something like it), you need to include it in your path, just as it's telling you to do. If you get the former, something would appear to be off. Greg On Fri, 2003-01-03 at 09:43, Claiborne, Aldemaco Earl (Al) wrote: Hi, I am trying to install postgresql-7.3 on windows and I keep getting the following error despite having downloaded a compiler. Can anyone tell me what I am not doing right? I am a newbie to postgres and development. My ultimate goal is to create a data driven application utilizing the J2EE architecture. $ ./configure checking build system type... i686-pc-cygwin checking host system type... i686-pc-cygwin checking which template to use... win checking whether to build with 64-bit integer date/time support... no checking whether to build with recode support... no checking whether NLS is wanted... no checking for default port number... 5432 checking for default soft limit on number of connections... 32 checking for gcc... no checking for cc... no configure: error: no acceptable C compiler found in $PATH Thanks, Al ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] PostgreSQL Password Cracker
On Thu, 2003-01-02 at 19:33, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Also, does anyone know why the development docs are 7.3.1? Because it was pointed to that branch during the 7.3 beta cycle. It needs to be repointed to CVS tip. I dunno how to do so, however. Is someone working to get 7.3.1 announced on our main web site? I would like to think that someone in either the -advocacy or webmaster groups will get around to that sometime soon ;-) I don't want to speak for anyone else, but honestly I don't think anyone is working on it (I mean how long could it take to do it and how long has 7.3.1 been released?). I'd be happy to do it if someone would explain to me how to do it. I have a lot of access to a bunch of stuff but I don't think I have access to the cvs that the main site is stored in. Anonymous access and an email of who to send patches to would be enough if people are worried about me trashing things :-) Robert Treat ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] PostgreSQL Password Cracker
-Original Message- From: Robert Treat [mailto:[EMAIL PROTECTED]] Sent: 03 January 2003 15:36 To: Tom Lane Cc: Bruce Momjian; Justin Clift; [EMAIL PROTECTED]; Dave Page Subject: Re: [HACKERS] PostgreSQL Password Cracker On Thu, 2003-01-02 at 19:33, Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Also, does anyone know why the development docs are 7.3.1? Because it was pointed to that branch during the 7.3 beta cycle. It needs to be repointed to CVS tip. I dunno how to do so, however. Is someone working to get 7.3.1 announced on our main web site? I would like to think that someone in either the -advocacy or webmaster groups will get around to that sometime soon ;-) I don't want to speak for anyone else, but honestly I don't think anyone is working on it (I mean how long could it take to do it and how long has 7.3.1 been released?). I'd be happy to do it if someone would explain to me how to do it. I have a lot of access to a bunch of stuff but I don't think I have access to the cvs that the main site is stored in. Anonymous access and an email of who to send patches to would be enough if people are worried about me trashing things :-) www.postgresql.com - with a little luck should become www.postgresql.org rsn. Regards, Dave. ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL Password Cracker
Tom Lane writes: separate out the parts that are only interesting to a programmer using libpq from the parts that are interesting to a user of a libpq-based program (for example, all the info about environment variables, conninfo string syntax, and .pgpass). The sections on environment variables and the .pgpass file are at the sect1 level, which is about as prominent as I think we can make them. Certainly they are not a subject matter that warrants a whole chapter of their own. What's missing are some cross-references. -- Peter Eisentraut [EMAIL PROTECTED] ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] complie error on windows
- Original Message - From: Greg Copeland [EMAIL PROTECTED] Sent: January 03, 2003 10:49 AM If you run, gcc, at the prompt (preferably the one you're trying to run configure from), do you get something like, gcc: No input files or do you get, gcc: command not found? If you get the later (or something like it), you need to include it in your path, ... or install gcc at the first place. By default gcc doesn't get installed on cygwin, you have to explicitly mark it to intall next time you run setup.exe. -s ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
On Thursday 02 January 2003 19:26, Tom Lane wrote: Lamar Owen [EMAIL PROTECTED] writes: So I figured I'd roll a 7.1.3 RPMset for him to install onto Red Hat 8. It was very bad. It simply would not build -- I guess it's the gcc 3 stuff. If you don't know *exactly* why it doesn't build, I don't think you should be blaming us for it. Who did I blame? (no one). I just made a statement -- which you pretty much agreed with later (gcc --version being multiline IS a gcc 3 problem, no? :-)). No blame at all. None of the message was a blame game of any kind, although you seem to take it personally every time I bring up upgrading. But, then again, I take it personally when someone e-mails me ranting (and sometimes cussing) about my stupid program's not upgrading (and PostgreSQL isn't 'my' program!). THIS DOESN'T HAPPEN WITH MySQL. Oh? Do they have a crystal ball that lets them predict incompatible future platform changes? No, they just allow for the old format, while making a new format. At least that's the way it looks from reading the docs and a cursory install run. The Linux Magazine article states it in a quite pointed way -- I'll grab that magazine when I get back home and post a quote if you'd like. (I'm not very sure I believe your assertion above, anyway. We are painfully aware of our own compatibility issues, but I wonder how many people on this list pay close enough attention to MySQL to know what their version-to-version compatibility track record *really* is.) I pay close enough attention that I was asked by a publisher to do a technical review of a MySQL reference book. And I'd love to see someone who has the time to do so (not me) grab this bull by the horns and make it happen. Well, this is exactly the issue: someone would have to put substantial amounts of time into update mechanisms and/or maintenance of obsolete versions, as opposed to features, performance improvements, or bug fixes. Fixing the upgrade 'bug' is a bugfix, IMHO. A _big_ bugfix. Personally, I feel that if we weren't working as hard as we could on features/performance/bugfixes, the upgrade issue would be moot because there wouldn't *be* any reason to upgrade. However, I'm sure there are people who hesitate to upgrade BECAUSE of the difficulty, thereby causing them to miss features. And potentially bugfixes, too. So I'm not planning to redirect my priorities. But this is an open source project and every developer gets to set their own priorities. If you can persuade someone to put their time into that, go for it. Which is why I shake the branches periodically, to see if I can persuade someone who can do this (in practice this means they either have a great deal of free time, or they are paid fulltime to work on PostgreSQL). I think you're wasting your time trying to hold us to a higher standard of backwards-compatibility than is maintained by the OSes and tools we must sit on top of. I think PostgreSQL already sets a higher standard in many ways. Particularly in the release cycle (we don't 'release early and release often'). -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Threads
Please no threading threads!!! Has anyone calculated the interval and period of PostgreSQL needs threads posts? The *ONLY* advantage threading has over multiple processes is the time and resources used in creating new processes. That being said, I admit that creating a threaded program is easier than one with multiple processes, but PostgreSQL is already there and working. Drawbacks to a threaded model: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) (3) Stack space. In a threaded application they are more limits to stack usage. I'm not sure, but I bet PostgreSQL would have a problem with a fixed size stack, I know the old ODBC driver did. (4) Lock Contention. The various single points of access in a process have to be serialized for multiple threads. heap allocation, deallocation, etc all have to be managed. In a multple process model, these resources would be separated by process contexts. (5) Lastly, why bother? Seriously? Process creation time is an issue true, but its an issue with threads as well, just not as bad. Anyone who is looking for performance should be using a connection pooling mechanism as is done in things like PHP. I have done both threaded and process servers. The threaded servers are easier to write. The process based severs are more robust. From an operational point of view, a select foo from bar where x y will take he same amount of time. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Upgrading rant.
Hannu Krosing wrote: I don't think the main issues are with file _formats_ but rather with system file structures - AFAIK it is a fundamental design decision (arguably a design flaw ;( ) that we use system tables straight from page cache via C structure pointers, even though there seems to be a layer called storage Manager which should hide the on-disk format completely. I don't think that is a big issue, no one is saying the file format should change or be used any differently, just that the structures be more rigid, and anyone wishing to make a change, had better also have an upgrade strategy. Perhaps there could be a review/document phase for 7.4 where the structures are cleaned up, checked, and perhaps have a couple reserved entries added. (As governed by efficiecy) and have one last speak now, or forever hold your peace and cast them in stone. I think, as professional standards go, this is probably LONG over due. ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] python interface
D'Arcy J.M. Cain [EMAIL PROTECTED] writes: I would love to add PyGreSQL to gborg but no matter how often I log in it tells me that I need to be logged in to create a new project. Weird. Maybe you're blocking cookies, or something like that? regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Upgrading rant.
Hannu Krosing [EMAIL PROTECTED] writes: I don't think the main issues are with file _formats_ but rather with system file structures - AFAIK it is a fundamental design decision (arguably a design flaw ;( ) that we use system tables straight from page cache via C structure pointers, The system tables are not the problem. pg_upgrade has shown how we can have cross-version upgrades no matter how much the system catalogs change (a good thing too, because we cannot freeze the system catalog layout without bringing development to a standstill). A schema-only dump and restore is cheap enough that there's no real reason to look for any other solution. Changes in the on-disk representation of user tables would be harder to deal with, but they are also much rarer (AFAIR we've only done that twice: WAL required additions to page and tuple headers, and then there were Manfred's space-saving changes in 7.3). And as of 7.3 there is a version field in page headers, which would in theory allow for a page-at-a-time update process to work. There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. The point is that *someone would have to do the work*. Unless someone wants to step up and volunteer, there's little value in discussing it. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] python interface
On Friday 03 January 2003 15:24, Tom Lane wrote:
D'Arcy J.M. Cain [EMAIL PROTECTED] writes:
I would love to add PyGreSQL to gborg but no matter how often I log in it
tells me that I need to be logged in to create a new project.
Weird. Maybe you're blocking cookies, or something like that?
Hmmm. I am using Opera and I have noticed cookie related strangeness before
but cookies do work. I use them with mailman all the time.
I'll try another browser tonight.
--
D'Arcy J.M. Cain darcy@{druid|vex}.net | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Threads
-Original Message- From: mlw [mailto:[EMAIL PROTECTED]] Sent: Friday, January 03, 2003 12:47 PM To: Shridhar Daithankar Cc: PGHackers Subject: Re: [HACKERS] Threads Please no threading threads!!! Has anyone calculated the interval and period of PostgreSQL needs threads posts? The *ONLY* advantage threading has over multiple processes is the time and resources used in creating new processes. Threading is absurdly easier to do portably than fork(). Will you fork() successfully on MVS, VMS, OS/2, Win32? On some operating systems, thread creation is absurdly faster than process creation (many orders of magnitude). That being said, I admit that creating a threaded program is easier than one with multiple processes, but PostgreSQL is already there and working. Drawbacks to a threaded model: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. If you use C++ you can try/catch and nothing bad happens to anything but the naughty thread. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) Poorly written applications leak memory. Fragmentation is a legitimate concern. (3) Stack space. In a threaded application they are more limits to stack usage. I'm not sure, but I bet PostgreSQL would have a problem with a fixed size stack, I know the old ODBC driver did. A single server with 20 threads will consume less total free store memory and automatic memory than 20 servers. You have to decide how much stack to give a thread, that's true. (4) Lock Contention. The various single points of access in a process have to be serialized for multiple threads. heap allocation, deallocation, etc all have to be managed. In a multple process model, these resources would be separated by process contexts. Semaphores are more complicated than critical sections. If anything, a shared memory approach is more problematic and fragile, especially when porting to multiple operating systems. (5) Lastly, why bother? Seriously? Process creation time is an issue true, but its an issue with threads as well, just not as bad. Anyone who is looking for performance should be using a connection pooling mechanism as is done in things like PHP. I have done both threaded and process servers. The threaded servers are easier to write. The process based severs are more robust. From an operational point of view, a select foo from bar where x y will take he same amount of time. Probably true. I think a better solution is a server that can start threads or processes or both. But that's neither here nor there and I'm certainly not volunteering to write it. Here is a solution to the dilemma. Make the one who suggests the feature be the first volunteer on the team that writes it. Is it a FAQ? If not, it ought to be. ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 14:47, mlw wrote: Please no threading threads!!! Ya, I'm very pro threads but I've long since been sold on no threads for PostgreSQL. AIO on the other hand... ;) Your summary so accurately addresses the issue it should be a whole FAQ entry on threads and PostgreSQL. :) Drawbacks to a threaded model: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) These are things that can't be stressed enough. IMO, these are some of the many reasons why applications running on MS platforms tend to have much lower application and system up times (that and resources leaks which are inherent to the platform). BTW, if you do much in the way of threaded coding, there is libHorde which is a heap library for heavily threaded, memory hungry applications. It excels in performance, reduces heap lock contention (maintains multiple heaps in a very thread smart manner), and goes a long way toward reducing heap fragmentation which is common for heavily memory based, threaded applications. (3) Stack space. In a threaded application they are more limits to stack usage. I'm not sure, but I bet PostgreSQL would have a problem with a fixed size stack, I know the old ODBC driver did. Most modern thread implementations use a page guard on the stack to determine if it needs to grow or not. Generally speaking, for most modern platforms which support threading, stack considerations rarely become an issue. (5) Lastly, why bother? Seriously? Process creation time is an issue true, but its an issue with threads as well, just not as bad. Anyone who is looking for performance should be using a connection pooling mechanism as is done in things like PHP. I have done both threaded and process servers. The threaded servers are easier to write. The process based severs are more robust. From an operational point of view, a select foo from bar where x y will take he same amount of time. I agree with this, however, using threads does open the door for things like splitting queries and sorts across multiple CPUs. Something the current process model, which was previously agreed on, would not be able to address because of cost. Example: select foo from bar where x y order by foo ;, could be run on multiple CPUs if the sort were large enough to justify. After it's all said and done, I do agree that threading just doesn't seem like a good fit for PostgreSQL. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 14:52, Dann Corbit wrote: -Original Message- (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. If you use C++ you can try/catch and nothing bad happens to anything but the naughty thread. That doesn't protect against the type of issues he's talking about. Invalid pointer reference is a very common snafu which really hoses threaded applications. Not to mention resource leaks AND LOCKED resources which are inherently an issue on Win32. Besides, it's doubtful that PostgreSQL is going to be rewritten in C++ so bringing up try/catch is pretty much an invalid argument. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) Poorly written applications leak memory. Fragmentation is a legitimate concern. And well written applications which attempt to safely handle segfaults, etc., often leak memory and lock resources like crazy. On Win32, depending on the nature of the resources, once this happens, even process termination will not free/unlock the resources. (4) Lock Contention. The various single points of access in a process have to be serialized for multiple threads. heap allocation, deallocation, etc all have to be managed. In a multple process model, these resources would be separated by process contexts. Semaphores are more complicated than critical sections. If anything, a shared memory approach is more problematic and fragile, especially when porting to multiple operating systems. And critical sections lead to low performance on SMP systems for Win32 platforms. No task can switch on ANY CPU for the duration of the critical section. It's highly recommend by MS as the majority of Win32 applications expect uniprocessor systems and they are VERY fast. As soon as multiple processors come into the mix, critical sections become a HORRIBLE idea if any soft of scalability is desired. Is it a FAQ? If not, it ought to be. I agree. I think mlw's list of reasons should be added to a faq. It terse yet says it all! -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Threads
I am sure, many of you would like to delete this message before reading, hold on. :-) I'm afraid most posters did not read the message. Those who replied Why bother? did not address your challenge: I think threads are useful in difference situations namely parallelising blocking conditions and using multiple CPUs. This is indeed one of the few good reasons for threads. Indeed, large/robust systems use a mix. The consensus of the group is that those who do the work are not ready for threads. Which is fine. Looking into my crystal ball, I see that it will happen, though it appears so far away. bbaker ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Upgrading rant.
On Friday 03 January 2003 15:16, Lamar Owen wrote: On Thursday 02 January 2003 19:26, Tom Lane wrote: Lamar Owen Wrote THIS DOESN'T HAPPEN WITH MySQL. Oh? Do they have a crystal ball that lets them predict incompatible future platform changes? No, they just allow for the old format, while making a new format. At least that's the way it looks from reading the docs and a cursory install run. The Linux Magazine article states it in a quite pointed way -- I'll grab that magazine when I get back home and post a quote if you'd like. Linux Magazine, January 2003, page 28. Article by Jeremy Zawodny. Too long to quote directly. Relevant section is headed 'Compatability'. The short of it: the on-disk format didn't change between MySQL 3.23 and MySQL 4.0, so you don't need to do a massive export and import process. You will, however, need to run mysql_fix_privilege_tables to update the grant tables to use the new privileges features in 4.0. Also, and I'll quote this one: Moving to MySQL 4.1 will require you to either dump and re-import your data or run various ALTER commands to take advantage of its new features. However, if you don't need those features, your existing tables will work fine. It's that last sentence that has me perturbed. It is just so obvious a feature... -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] why was libpq.so's version number bumped?
Florian Weimer wrote: Neil Conway [EMAIL PROTECTED] writes: Christopher Kings-Lynne said: There have been HEAPS of security fixes between 7.2 and 7.3. That's only the case if your definition of a security fix is pretty fast and loose -- as yours seems to be. Hmm? On 7.2, an unpriviliged database user can read the more or less the whole memory image of the server process. On 7.3, this has been fixed. Huh. Never heard that before. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] why was libpq.so's version number bumped?
Bruce Momjian [EMAIL PROTECTED] writes: Hmm? On 7.2, an unpriviliged database user can read the more or less the whole memory image of the server process. On 7.3, this has been fixed. Huh. Never heard that before. It's the cash_out(2) crash, but with some other conversion function: rusfw= SELECT byteain(134512640); byteain - \177ELF\001\001\001 (1 row) rusfw= (Of course, the address varies from machine to machine.) -- Florian Weimer[EMAIL PROTECTED] University of Stuttgart http://CERT.Uni-Stuttgart.DE/people/fw/ RUS-CERT fax +49-711-685-5898 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
Tom Lane wrote: The system tables are not the problem. pg_upgrade has shown how we can have cross-version upgrades no matter how much the system catalogs change (a good thing too, because we cannot freeze the system catalog layout without bringing development to a standstill). A schema-only dump and restore is cheap enough that there's no real reason to look for any other solution. Changes in the on-disk representation of user tables would be harder to deal with, but they are also much rarer (AFAIR we've only done that twice: WAL required additions to page and tuple headers, and then there were Manfred's space-saving changes in 7.3). And as of 7.3 there is a version field in page headers, which would in theory allow for a page-at-a-time update process to work. There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. The point is that *someone would have to do the work*. Unless someone wants to step up and volunteer, there's little value in discussing it. pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. Is pg_upgrade too hard to run? Is no one really interested in it? -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] pg_dump.options.diff -- Take III
hmmm... hate to resend it, but i have never seen this passing through...
re-attempt
- Original Message -
From: Serguei Mokhov [EMAIL PROTECTED]
Sent: January 02, 2003 6:37 PM
Subject: Re: [HACKERS] pg_dump.options.diff -- Take III
- Original Message -
From: Tom Lane [EMAIL PROTECTED]
Sent: January 02, 2003 6:16 PM
static void
usage(void)
{
#if defined(HAVE_GETOPT_LONG)
#define xo(longOption,shortOption,desc) printf(%s %s\n, longOption, desc)
#else
#define xo(longOption,shortOption,desc) printf(%s %s\n, shortOption, desc)
#endif
... lots of xo() calls ...
#undef xo
}
This gives us the convenience of a very short name within the usage()
subroutines, while not polluting the namespace for everyplace else in
these utilities.
Revised patch attached as per above.
-s
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Upgrading rant.
- Original Message - From: Bruce Momjian [EMAIL PROTECTED] Sent: January 03, 2003 6:31 PM Tom Lane wrote: The system tables are not the problem. pg_upgrade has shown how we can have cross-version upgrades no matter how much the system catalogs change (a good thing too, because we cannot freeze the system catalog layout without bringing development to a standstill). A schema-only dump and restore is cheap enough that there's no real reason to look for any other solution. Changes in the on-disk representation of user tables would be harder to deal with, but they are also much rarer (AFAIR we've only done that twice: WAL required additions to page and tuple headers, and then there were Manfred's space-saving changes in 7.3). And as of 7.3 there is a version field in page headers, which would in theory allow for a page-at-a-time update process to work. There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. The point is that *someone would have to do the work*. Unless someone wants to step up and volunteer, there's little value in discussing it. pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. I guess the main point here was to have co-existing versions of data in the /data dierectory and the sever be able to the talk to all of them, with an option to upgrade the data to the most recent format at some point. If it's done that way, there won't be much of a pain if at all to upgrade the server only and benifit from the general bug-fixes and general new features. The specific features that require more recent data models, will have to be tracked down, and dealt with if attempted for the older data model. The other main point was to find smb to do the dirty job of implementing, maintaining, and tracking down format changes of that concurrent data storage model. pg_upgrade would be the tool to envetually upgrade the data to the recent model in that case. Is pg_upgrade too hard to run? Is no one really interested in it? Would still be nice to have some GUI to it in Java/Web/*Admin/TclTk etc. It pg_upgrade documented anywhere besided man pages and the script itself? -s ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL Password Cracker
Peter Eisentraut wrote: Tom Lane writes: separate out the parts that are only interesting to a programmer using libpq from the parts that are interesting to a user of a libpq-based program (for example, all the info about environment variables, conninfo string syntax, and .pgpass). The sections on environment variables and the .pgpass file are at the sect1 level, which is about as prominent as I think we can make them. Certainly they are not a subject matter that warrants a whole chapter of their own. What's missing are some cross-references. But it is a sect1 in libpq. It should be a sect1 somewhere that makes more sense. Right now it appears here: Table of Contents 1. libpq - C Library 1.1. Introduction 1.2. Database Connection Functions 1.3. Command Execution Functions 1.4. Asynchronous Query Processing 1.5. The Fast-Path Interface 1.6. Asynchronous Notification 1.7. Functions Associated with the COPY Command 1.8. libpq Tracing Functions 1.9. libpq Control Functions 1.10. Environment Variables ^ 1.11. Files 1.12. Threading Behavior 1.13. Building Libpq Programs 1.14. Example Programs It doesn't belong in libpq, and it doesn't belong in the Programmer's Guide. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Upgrading rant.
Serguei Mokhov wrote: pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. I guess the main point here was to have co-existing versions of data in the /data dierectory and the sever be able to the talk to all of them, with an option to upgrade the data to the most recent format at some point. If it's done that way, there won't be much of a pain if at all to upgrade the server only and benifit from the general bug-fixes and general new features. The specific features that require more recent data models, will have to be tracked down, and dealt with if attempted for the older data model. We have to update the system tables for each release, and that is going to require something like pg_upgrade. I don't see how we can avoid that. We could enable reading of old heap/index file formats if we tried hard, but that still requires pg_upgrade to update the system catalogs. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] PostgreSQL Password Cracker
Bruce Momjian [EMAIL PROTECTED] writes: But it is a sect1 in libpq. It should be a sect1 somewhere that makes more sense. ... It doesn't belong in libpq, and it doesn't belong in the Programmer's Guide. How could it not belong in libpq? But you are right that the Programmer's Guide seems the wrong place for information that is important to end-users. Perhaps libpq needs to have a chapter in the User's Guide as well as a chapter in the Programmer's Guide? Or maybe we could put the relevant information into a reference page under PostgreSQL Client Applications (titled something along the line of common behavior of all libpq-based client applications). regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Upgrading rant.
Bruce Momjian [EMAIL PROTECTED] writes: There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. pg_upgrade does work, assuming there are no changes to the index or heap file formats. Does it really work? I had thought that there were some issues associated with adjusting transaction numbers that couldn't be solved by pg_upgrade in its present shell-script form; I was anticipating that pg_upgrade would have to be rewritten as a C program so that it could get at stuff at the necessary low level. I cannot recall the details right at the moment though. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. This may be what I was recalling. Did you find a bulletproof fix? regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] PostgreSQL Password Cracker
Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: But it is a sect1 in libpq. It should be a sect1 somewhere that makes more sense. ... It doesn't belong in libpq, and it doesn't belong in the Programmer's Guide. How could it not belong in libpq? But you are right that the Programmer's Guide seems the wrong place for information that is important to end-users. Some people don't even know libpq exists. If someone uses only psql, how would they know? Perhaps libpq needs to have a chapter in the User's Guide as well as a chapter in the Programmer's Guide? Or maybe we could put the relevant information into a reference page under PostgreSQL Client Applications (titled something along the line of common behavior of all libpq-based client applications). Yep, that's what I would do. -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Threads
Serguei Mokhov [EMAIL PROTECTED] writes: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (1) is an issue only for user-level threads. Uh, what other kind of thread have you got in mind here? I suppose the lack-of-cross-thread-protection issue would go away if our objective was only to use threads for internal parallelism in each backend instance (ie, you still have one process per connection, but internally it would use multiple threads to process subqueries in parallel). Of course that gives up the hope of faster connection startup that has always been touted as a major reason to want Postgres to be threaded... regards, tom lane ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/users-lounge/docs/faq.html
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 19:34, Tom Lane wrote: Serguei Mokhov [EMAIL PROTECTED] writes: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (1) is an issue only for user-level threads. Umm. No. User or system level threads, the statement is true. If a thread kills over, the process goes with it. Furthermore, on Win32 platforms, it opens a whole can of worms no matter how you care to address it. Uh, what other kind of thread have you got in mind here? I suppose the lack-of-cross-thread-protection issue would go away if our objective was only to use threads for internal parallelism in each backend instance (ie, you still have one process per connection, but internally it would use multiple threads to process subqueries in parallel). Several have previously spoken about a hybrid approach (ala Apache). IIRC, it was never ruled out but it was simply stated that no one had the energy to put into such a concept. Of course that gives up the hope of faster connection startup that has always been touted as a major reason to want Postgres to be threaded... regards, tom lane Faster startup, should never be the primary reason as there are many ways to address that issue already. Connection pooling and caching are by far, the most common way to address this issue. Not only that, but by definition, it's almost an oxymoron. If you really need high performance, you shouldn't be using transient connections, no matter how fast they are. This, in turn, brings you back to persistent connections or connection pools/caches. -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Upgrading rant.
OK, taking up the pg_upgrade banner, I think there are two things missing from the current code: 1) schema awareness -- easily fixed with some code 2) need to creat clog files to match incremented xid I can do 1, and I think Tom can help me with 2. Then folks can test it and see how it works. It is that last part that made me stop around 7.2, and 7.3 had a heap format change that guarantteed it wouldn't work. Also, I think we make index format changes more frequently that Tom recollects. Tom? --- Lamar Owen wrote: On Friday 03 January 2003 18:31, Bruce Momjian wrote: Tom Lane wrote: There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. The point is that *someone would have to do the work*. Unless someone wants to step up and volunteer, there's little value in discussing it. pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. Is pg_upgrade too hard to run? Is no one really interested in it? It has been considered 'experimental' in the past, Bruce. It needs more credibility from the development group, as in 'We recommend you try to use pg_upgrade (after making a backup), then attempt to do a dump/restore if pg_upgrade doesn't work (and pg_upgrade needs to be more robust in its failure modes). I am very interested in pg_upgrade, as I believe I mentioned the last go through this topic. But it's the 'red-headed stepchild' utility here. (I'm red-headed, my mother's red-headed, so no slight meant to those of fiery folicles.) But it's also all or nothing -- you go the whole way through. It's again our tremendous dependence upon the contents of the system catalogs that does us in. That is one of our greatest strengths, until you have to upgrade for some reason. Then it becomes our biggest liability. And unlike Tom I think it is worthwhile to discuss it periodically, to remind the group as a whole (which composition and membership changes frequently) that there's a problem waiting to be solved. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Upgrading rant.
On Friday 03 January 2003 18:31, Bruce Momjian wrote: Tom Lane wrote: There isn't any fundamental reason why we cannot have a pg_upgrade utility; claiming that there is something wrong with how we handle catalog changes misses the point. The point is that *someone would have to do the work*. Unless someone wants to step up and volunteer, there's little value in discussing it. pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. Is pg_upgrade too hard to run? Is no one really interested in it? It has been considered 'experimental' in the past, Bruce. It needs more credibility from the development group, as in 'We recommend you try to use pg_upgrade (after making a backup), then attempt to do a dump/restore if pg_upgrade doesn't work (and pg_upgrade needs to be more robust in its failure modes). I am very interested in pg_upgrade, as I believe I mentioned the last go through this topic. But it's the 'red-headed stepchild' utility here. (I'm red-headed, my mother's red-headed, so no slight meant to those of fiery folicles.) But it's also all or nothing -- you go the whole way through. It's again our tremendous dependence upon the contents of the system catalogs that does us in. That is one of our greatest strengths, until you have to upgrade for some reason. Then it becomes our biggest liability. And unlike Tom I think it is worthwhile to discuss it periodically, to remind the group as a whole (which composition and membership changes frequently) that there's a problem waiting to be solved. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11 ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
Bruce Momjian [EMAIL PROTECTED] writes: OK, taking up the pg_upgrade banner, I think there are two things missing from the current code: 1) schema awareness -- easily fixed with some code 2) need to creat clog files to match incremented xid I can do 1, and I think Tom can help me with 2. I was just now wondering whether we really need to do that at all. We're already vacuuming the user tables before we bring 'em over. What if we VACUUM FREEZE them instead? Then there are *no* xids of interest in the tables being brought over, and no need to screw around with the xid counter in the new installation. That in turn would mean no need to mess with its pg_clog files. I think we'd still need to advance the xlog position past the old installation's xlog end, but we have the tool for that (pg_resetxlog) already. Also, I think we make index format changes more frequently that Tom recollects. Tom? Oh? Name one... not that they'd be a critical problem anyway, as we could easily reconstruct indexes via REINDEX rather than moving them over, any time we made such a change. regards, tom lane ---(end of broadcast)--- TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
Re: [HACKERS] Upgrading rant.
Bruce Momjian [EMAIL PROTECTED] writes: Is pg_upgrade too hard to run? It is hard and dangerous to run (IMHO). One thing I would like to see that would make it much safer to run is to recast it to operate through standalone backends. That is, the idea is bring down old postmaster install new version run new version's pg_upgrade -- all work done without a postmaster start new postmaster It's much harder to get this wrong than it is to mess up with the current situation (where pg_upgrade talks to live postmasters). There isn't any simple way to lock *everyone* out of the DB and still allow pg_upgrade to connect via the postmaster, and even if there were, the DBA could too easily forget to do it. This would require a nontrivial amount of work (notably, we'd have to be able to get pg_dump to run against a standalone backend) but I don't think I'd trust pg_upgrade as a production-grade tool until its invocation method looks like the above. regards, tom lane ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Threads
Greg Copeland wrote: Of course that gives up the hope of faster connection startup that has always been touted as a major reason to want Postgres to be threaded... regards, tom lane Faster startup, should never be the primary reason as there are many ways to address that issue already. Connection pooling and caching are by far, the most common way to address this issue. Not only that, but by definition, it's almost an oxymoron. If you really need high performance, you shouldn't be using transient connections, no matter how fast they are. This, in turn, brings you back to persistent connections or connection pools/caches. Connection time should *never* be in the critical path. There, I've said it!! People who complain about connection time are barking up the wrong tree. Regardless of the methodology, EVERY OS has issues with thread creation, process creation, the memory allocation, and system manipulation required to manage it. Under load this is ALWAYS slower. I think that if there is ever a choice, "do I make startup time faster?" or "Do I make PostgreSQL not need a dump/restore for upgrade" the upgrade problem has a much higher impact to real PostgreSQL sites.
Re: [HACKERS] Threads
On Fri, 2003-01-03 at 21:39, mlw wrote: Connection time should *never* be in the critical path. There, I've said it!! People who complain about connection time are barking up the wrong tree. Regardless of the methodology, EVERY OS has issues with thread creation, process creation, the memory allocation, and system manipulation required to manage it. Under load this is ALWAYS slower. I think that if there is ever a choice, do I make startup time faster? or Do I make PostgreSQL not need a dump/restore for upgrade the upgrade problem has a much higher impact to real PostgreSQL sites. Exactly. Trying to speed up something that shouldn't be in the critical path is exactly what I'm talking about. I completely agree with you! -- Greg Copeland [EMAIL PROTECTED] Copeland Computer Consulting ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] Upgrading rant.
On Sat, 2003-01-04 at 02:17, Tom Lane wrote: There isn't any simple way to lock *everyone* out of the DB and still allow pg_upgrade to connect via the postmaster, and even if there were, the DBA could too easily forget to do it. I tackled this issue in the Debian upgrade scripts. I close the running postmaster and open a new postmaster using a different port, so that normal connection attempts will fail because there is no postmaster running on the normal port. -- Oliver Elphick[EMAIL PROTECTED] Isle of Wight, UK http://www.lfix.co.uk/oliver GPG: 1024D/3E1D0C1C: CA12 09E0 E8D5 8870 5839 932A 614D 4C34 3E1D 0C1C But because of his great love for us, God, who is rich in mercy, made us alive with Christ even when we were dead in transgressions-it is by grace you have been saved.Ephesians 2:4,5 ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
Re: [HACKERS] Upgrading rant.
On Fri, 3 Jan 2003, Bruce Momjian wrote: Is pg_upgrade too hard to run? Is no one really interested in it? All of my boxes are still on 7.2.3. Does that represent a viable test base? ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] why was libpq.so's version number bumped?
Between 7.2 and 7.3 there was an API change to the pgnotify(?) struct or something. When 7.3 was released, we forgot to bump the version number, so we're doing it now. It was something of an oversight, but it really needed to be done. There have been HEAPS of security fixes between 7.2 and 7.3. Depending on your definition of security. eg. Going 'select cash_out(2);' on any 7.2 server and below will crash the backend. It will not do that in 7.3. Most of the buffer problems were fixed in 7.2.3. Since going from 7.2 to 7.3 is a significant upgrade, the FreeBSD guys would probablyu be right tho to refuse such a major upgrade... Still, it's a pity though. Postgres 7.3 has been tested and works fine on FreeBSD 5. The databases/postgresql7 port has been updated and 7.3.1 should appear in the 5.0 release. ;) Not all postgresql ports have been updated though since some of the PRs are assigned to another committer. :-/ Let me know if there are any requests from BSD DBAs that'd like to see a port that they highly depend on updated. ps. Why is Postgres 7.3 still in ports/databases/postgresql-devel ?? Actually, maybe it was a good thing since if 7.3.1 becomes the new standard port people won't be bitten so much by the library version bump. The -devel port has been out for a few months, those that are serious DBAs likely have a staging/test environment to work with. If anyone runs across any serious problems on FreeBSD, let me know ASAP. -sc -- Sean Chittenden ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Threads
Also remember that in even well developed OS's like FreeBSD, all a process's threads will execute only on one CPU. This might change in FreeBSD 5.0, but still a threaded app (such as MySQL) cannot use mutliple CPUs on a FreeBSD system. Chris On Fri, 3 Jan 2003, mlw wrote: Please no threading threads!!! Has anyone calculated the interval and period of PostgreSQL needs threads posts? The *ONLY* advantage threading has over multiple processes is the time and resources used in creating new processes. That being said, I admit that creating a threaded program is easier than one with multiple processes, but PostgreSQL is already there and working. Drawbacks to a threaded model: (1) One thread screws up, the whole process dies. In a multiple process application this is not too much of an issue. (2) Heap fragmentation. In a long uptime application, such as a database, heap fragmentation is an important consideration. With multiple processes, each process manages its own heap and what ever fragmentation that exists goes away when the connection is closed. A threaded server is far more vulnerable because the heap has to manage many threads and the heap has to stay active and unfragmented in perpetuity. This is why Windows applications usually end up using 2G of memory after 3 months of use. (Well, this AND memory leaks) (3) Stack space. In a threaded application they are more limits to stack usage. I'm not sure, but I bet PostgreSQL would have a problem with a fixed size stack, I know the old ODBC driver did. (4) Lock Contention. The various single points of access in a process have to be serialized for multiple threads. heap allocation, deallocation, etc all have to be managed. In a multple process model, these resources would be separated by process contexts. (5) Lastly, why bother? Seriously? Process creation time is an issue true, but its an issue with threads as well, just not as bad. Anyone who is looking for performance should be using a connection pooling mechanism as is done in things like PHP. I have done both threaded and process servers. The threaded servers are easier to write. The process based severs are more robust. From an operational point of view, a select foo from bar where x y will take he same amount of time. ---(end of broadcast)--- TIP 6: Have you searched our list archives? http://archives.postgresql.org ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Upgrading rant.
pg_upgrade does work, assuming there are no changes to the index or heap file formats. (However, I now need to update it for schemas.) However, the last time I worked on it for 7.2, no one was really interested in testing it, so it never got done. In fact, there was a bug in the handling of clog or wal files, but I didn't find out about it until long after 7.2 because no one was using it. Is pg_upgrade too hard to run? Is no one really interested in it? I'm interested. I might look into it a bit more this release cycle as it seems that I don't really have the time for major features like I did last cycle. Still looking for something straightforward to work on... Chris ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
