[HACKERS] A few questions
Hi people, I'm new here, and I would like to know a few things about how developers works. Could be posible that two developers works on the same item, but in isolation? What happens with the different patchs they produced? I would like to do my PFC in semantic optimization, what is now implemented about this in potgresql? Thank's to all. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] A few questions
On Tue, Oct 18, 2005 at 09:57:03AM +0200, Sebastián Ávila wrote: I'm new here, and I would like to know a few things about how developers works. Could be posible that two developers works on the same item, but in isolation? What happens with the different patchs they produced? I would like to do my PFC in semantic optimization, what is now implemented about this in potgresql? There is no formal process control. Though usually most people developing for PostgreSQL are subscribed to this list so if you mention here what you're doing here you can avoid duplicate work. If two people end up with a patch for the same thing, well, may be best patch win. Actually though, they usually get compared and merged to produce the best possible result. We're generally more concerned about clean maintainable code than who did what when. Semantic optimisation huh? I don't think anybody is working on that in the general sense, though specific fixes are applied now and then. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. pgpRUM3zeLmvP.pgp Description: PGP signature
[HACKERS] Problem in ExecEvalExpr function
Hi, I have some doubts related to ExecEvalExpr functions which is used to executing the expression. Here we are finding the subexpression recursively. But I am not able to find it out that where exactly the comparison taking place. For eg. relation A id name age relation B name age status if query is select id , name , age , status from A , B where A.name = B.name and A.age = B.age. Then parser find this and condition and divide this into two expression. But I am not getting in which function exactly these comparison between tupleslot taking place. -- -- | Sandeep Satpal | | M.Tech Student | | Lab 212 KReSIT | -- ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] PostgreSQL roadmap for 8.2 and beyond.
Tom Lane wrote: Patrick Bakker [EMAIL PROTECTED] writes: It would be useful if you could toggle SQL statement logging without restarting PostgreSQL and additionally if you could turn on selective SQL logging. Additionally, it would be great if you could log SQL statements to a separate file from the main log and in such a way that the statements are seperated in an easily parsed form. The runtime toggling of statement logging would also be most useful if you could toggle it outside of a particular session. This is where the selective SQL logging would be useful I think. Some selectable options that could be useful: - by database - by user - by IP address You do realize that the majority of that can be done today? You need to study the available mechanism for setting GUC parameters a little more. Not quite, if you want to set it on databases you're not connected to, i.e. if you want to 'tap' a backend. Regards, Andreas ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Problem in ExecEvalExpr function
On Tue, Oct 18, 2005 at 02:38:39PM +0530, sandeep satpal wrote: Hi, I have some doubts related to ExecEvalExpr functions which is used to executing the expression. Since they are operators, they go through ExecEvalOper. However, after the first call they go through ExecMakeFunctionResult. The operator has an oid which is used to find the function and it is eventually called using FunctionCallInvoke. Note, ExecEvalExpr is a macro, so you can't break on it. Here we are finding the subexpression recursively. But I am not able to find it out that where exactly the comparison taking place. For eg. relation A id name age relation B name age status if query is select id , name , age , status from A , B where A.name = B.name and A.age = B.age. Note that if the operator appears in the OPERATOR CLASS and the optimizer has determined that it can use an index, it won't go through this code at all, instead using the compare function in _bt_compare. Then parser find this and condition and divide this into two expression. But I am not getting in which function exactly these comparison between tupleslot taking place. FunctionCallInvoke will be calling the function for operator =. Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. pgp8ldqdWg3IL.pgp Description: PGP signature
[HACKERS] SIGSEGV in Postgres 8.0.3 (libpq4)
Hi, I have a set of perl scripts which invoke each other (via system()); eventually I found that they were crashing and ultimately causing Perl to SIGSEGV. I am using Debian testing and have recompiled postgres-8.0.3-15 to include debugging symbols. This is a dual-CPU dual-stacked IPv4/IPv6 host running Linux 2.4.22. From what I have been able to determine the problem is in libpq4: Starting program: /usr/bin/perl ./add_address 265 2001:502:d399:0:0:0:0:44 [Thread debugging using libthread_db enabled] [New Thread 16384 (LWP 8523)] (no debugging symbols found) Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 16384 (LWP 8523)] r0x4038275b in pqGetc ( result=0xbfffee0b @[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]/[EMAIL PROTECTED]@X���7\0178@, conn=0x401c13d4) at fe-misc.c:85 85 *result = conn-inBuffer[conn-inCursor++]; (gdb) bt #0 0x4038275b in pqGetc ( result=0xbfffee0b @[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]/[EMAIL PROTECTED]@X���7\0178@, conn=0x401c13d4) at fe-misc.c:85 #1 0x40388543 in pqParseInput3 (conn=0x401c13d4) at fe-protocol3.c:80 #2 0x403805f4 in parseInput (conn=Variable conn is not available. ) at fe-exec.c:1097 #3 0x40380f37 in PQgetResult (conn=0x401c13d4) at fe-exec.c:1171 #4 0x403812e4 in PQexecStart (conn=0x401c13d4) at fe-exec.c:1315 #5 0x40381532 in PQexec (conn=0x401c13d4, query=0x82fada8 DEALLOCATE dbdpg_2) at fe-exec.c:1223 #6 0x40367459 in _result () from /usr/lib/perl5/auto/DBD/Pg/Pg.so #7 0x4036da35 in dbd_st_deallocate_statement () from /usr/lib/perl5/auto/DBD/Pg/Pg.so #8 0x4036dfe8 in dbd_st_destroy () from /usr/lib/perl5/auto/DBD/Pg/Pg.so #9 0x40360784 in XS_DBD__Pg__st_DESTROY () from /usr/lib/perl5/auto/DBD/Pg/Pg.so #10 0x40342e65 in XS_DBI_dispatch () from /usr/lib/perl5/auto/DBI/DBI.so #11 0x080c4386 in Perl_pp_entersub () #12 0x080644dc in Perl_call_sv () #13 0x080642b1 in Perl_call_sv () #14 0x080ccc55 in Perl_sv_clear () #15 0x080cd420 in Perl_sv_free () #16 0x080cce19 in Perl_sv_clear () #17 0x080cd420 in Perl_sv_free () #18 0x080b0a1f in Perl_mg_free () #19 0x080cd1ba in Perl_sv_clear () #20 0x080cd420 in Perl_sv_free () #21 0x080c56f4 in Perl_sv_add_arena () #22 0x080c5876 in Perl_sv_clean_objs () #23 0x0806202d in perl_destruct () #24 0x0805fde0 in main () (gdb) I am happy to try out any patches, etc., however I am not on the mailing list, so please CC: any replies to me. Thanks, Anand -- `When any government, or any church for that matter, undertakes to say to its subjects, This you may not read, this you must not see, this you are forbidden to know, the end result is tyranny and oppression no matter how holy the motives' -- Robert A Heinlein, If this goes on -- ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Problem in ExecEvalExpr function
Sandeep, You seem to be having a lot of trouble understanding the evaluation code which isn't that complex. Are you still trying to do that case-insensitive comparison? If so, I seriously recommend that you just look at the citext project on gborg and stop messing around in the backend, it's just going to get ya into trouble. On 10/18/05, Martijn van Oosterhout kleptog@svana.org wrote: On Tue, Oct 18, 2005 at 02:38:39PM +0530, sandeep satpal wrote: Hi, I have some doubts related to ExecEvalExpr functions which is used to executing the _expression_.Since they are operators, they go through ExecEvalOper. However, after the first call they go through ExecMakeFunctionResult. The operator hasan oid which is used to find the function and it is eventually calledusing FunctionCallInvoke.Note, ExecEvalExpr is a macro, so you can't break on it. Here we are finding the subexpression recursively. But I am not able to find it out that where exactly the comparison taking place. For eg. relation A id name age relation B name age status if query is select id , name , age , status from A , B where A.name = B.name and A.age = B.age .Note that if the operator appears in the OPERATOR CLASS and theoptimizer has determined that it can use an index, it won't go throughthis code at all, instead using the compare function in _bt_compare. Then parser find this and condition and divide this into two _expression_. But I am not getting in which function exactly these comparison between tupleslot taking place.FunctionCallInvoke will be calling the function for operator =. Have a nice day,--Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them.-- Respectfully,Jonah H. Harris, Database Internals Architect EnterpriseDB Corporationhttp://www.enterprisedb.com/
[HACKERS] Call for translators
As the release of PostgreSQL 8.1 draws near, it is once again time to update the translations of the program messages. To check the status of your language, check out this web site: http://developer.postgresql.org/~petere/nlsstatus/ To participate in the translation effort, please follow the instructions set out on that page, or contact me directly if you have further questions. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Seeing context switch storm with 10/13 snapshot of
On Tue, 18 Oct 2005 00:25:25 +0100 Simon Riggs [EMAIL PROTECTED] wrote: Please try this patch and see if it reduces the CS storm: http://archives.postgresql.org/pgsql-patches/2005-10/msg00091.php Yes, I will. I'd been trying to figure out what triggered it, as I was unable to reproduce it for a while, but it gets there eventually. I had just re-started with auto vacuum off, as that might be the trigger? I'll apply the patch, re-build and re-start with auto vacuum back on, they way I'm running when I know I'll see it. Do you have access to another similar machine to do comparative testing? Do you have access to another machine with different CPU arch? It would be good to firmly isolate this to a CPU architecture interaction issue. I do have access to a windows version of the DL-380 but it's only a single CPU. I've a dual AMD server at home, but I cannot run against the db there (reverse VPN?). I'd need that reproducibility Tom wanted that I have not gotten to yet. Are the Xeons particularly old models? How new is the server? It is a HP Proliant (post Compaq merger) DL-380. A couple years old. Thanks, Rob ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Seeing context switch storm with 10/13 snapshot of
On Tue, 18 Oct 2005 10:29:43 -0600 Robert Creager [EMAIL PROTECTED] wrote: On Tue, 18 Oct 2005 00:25:25 +0100 Simon Riggs [EMAIL PROTECTED] wrote: Please try this patch and see if it reduces the CS storm: Sorry, didn't work. Took about an hour, and now it's now at the CS storm (averaging 94k). I've now disabled auto vacuum, just to see if that is a trigger, and am re-running. Thanks, Rob ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] libpq's pollution of application namespace
On Mon, 2005-17-10 at 13:32 -0400, Tom Lane wrote: I dislike portability approaches that try to enumerate supported cases, rather than being general in the first place. Do we need to have this on every platform we support? The symbols we want to hide are internal by convention anyway -- using a linker script or similar technique just improves upon this by preventing applications from misbehaving (and it also improves performance slightly). If no one has bothered to add support for a particular platform's linker they won't get these benefits, but that doesn't seem like a disaster. -Neil ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] Alpha: HEAD: Failure
I just tried(!) to run the buildfarm script on a Alpha DS10L/NetBSD 2.0 box. It died in make check. I have a tarball of the last-logs, to which I added a backtrace from the coredump(!). It's at: http://www.lerctr.org/~ler/alphadeath.tar.gz $ gtar tzvf alphadeath.tar.gz drwxr-xr-x ler/users 0 2005-10-18 14:21 lastrun-logs -rw-r--r-- ler/users 22708 2005-10-18 13:21 lastrun-logs/CVS.log -rw-r--r-- ler/users 11889 2005-10-18 13:27 lastrun-logs/configure.log -rw-r--r-- ler/users228889 2005-10-18 13:27 lastrun-logs/config.log -rw-r--r-- ler/users156157 2005-10-18 13:56 lastrun-logs/make.log -rw-r--r-- ler/users 1048443 2005-10-18 14:01 lastrun-logs/check.log -rw-r--r-- ler/users 1058671 2005-10-18 14:01 lastrun-logs/web-txn.data -rw-r--r-- ler/users964295 2005-10-18 14:19 lastrun-logs/bt.out $ Would someone pull it down, and let me know how I can help fix it? Thanks! LER -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 US ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[HACKERS] Character Conversions Handling
Hi, I'm trying to understand the schema laying behind backend/utils/adt/like.c to downcase letters [1]. When I look at the other tolower() implementations, there're lots of them spread around. (In interfaces/libpq, backend/regex, backend/utils/adt/like and etc.) For example, despite having pg_wc_tolower() function in regc_locale.c, achieving same with manually in iwchareq() of like.c. I'd so appreciated if somebody can point me the places where I should start to look at to understand the character handling with different encodings. Also, I wonder why didn't we use any btow/mbsrtowc/wctomb like functions. Is this for portability with other compilers? [1] iwchareq() is using pg_mb2wchar_with_len() which decides the right mb2wchar function from pg_wchar_table. When I look at backend/mb/wchar.c there're some other specific to locale mblen and mb2wchar routines. For example, EUC_KR is handled with pg_euc2wchar_with_len() function, but LATIN5 is handled with pg_latin12wchar_with_len() function. Will we write a new function for latin5 like pg_latin52wchar_with_len() if we'd encounter with a new problem with latin5? Regards. ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] 2nd try @NetBSD/2.0 Alpha
Upped the stack to 8Mb. Now it dies in Plcheck. Logs/bt.out in: http://www.lerctr.org/~ler/alphadeath2.tar.gz $ tar tzvf alphadeath2.tar.gz drwxr-xr-x 2 ler users 0 Oct 18 16:01 lastrun-logs -rw-r--r-- 1 ler users 22708 Oct 18 14:50 lastrun-logs/CVS.log -rw-r--r-- 1 ler users 11889 Oct 18 14:56 lastrun-logs/configure.log -rw-r--r-- 1 ler users 228889 Oct 18 14:56 lastrun-logs/config.log -rw-r--r-- 1 ler users 156157 Oct 18 15:32 lastrun-logs/make.log -rw-r--r-- 1 ler users 201363 Oct 18 15:38 lastrun-logs/check.log -rw-r--r-- 1 ler users 40733 Oct 18 15:45 lastrun-logs/make-contrib.log -rw-r--r-- 1 ler users 49672 Oct 18 15:48 lastrun-logs/make-install.log -rw-r--r-- 1 ler users 1531 Oct 18 15:49 lastrun-logs/initdb.log -rw-r--r-- 1 ler users 61 Oct 18 15:49 lastrun-logs/startdb-1.log -rw-r--r-- 1 ler users 128491 Oct 18 15:55 lastrun-logs/install-check.log -rw-r--r-- 1 ler users 68 Oct 18 15:55 lastrun-logs/stopdb-1.log -rw-r--r-- 1 ler users 61 Oct 18 15:55 lastrun-logs/startdb-2.log -rw-r--r-- 1 ler users 2929 Oct 18 15:56 lastrun-logs/pl-install-check.log -rw-r--r-- 1 ler users 6996 Oct 18 15:56 lastrun-logs/web-txn.data -rw-r--r-- 1 ler users 3844 Oct 18 16:01 lastrun-logs/bt.out tar: ustar vol 1, 16 files, 870400 bytes read, 0 bytes written in 1 secs (870400 bytes/sec) $ -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 US ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] Seeing context switch storm with 10/13 snapshot of
On Tue, 18 Oct 2005 12:24:03 -0600 Robert Creager [EMAIL PROTECTED] wrote: On Tue, 18 Oct 2005 10:29:43 -0600 Robert Creager [EMAIL PROTECTED] wrote: On Tue, 18 Oct 2005 00:25:25 +0100 Simon Riggs [EMAIL PROTECTED] wrote: Please try this patch and see if it reduces the CS storm: Sorry, didn't work. Took about an hour, and now it's now at the CS storm (averaging 94k). I've now disabled auto vacuum, just to see if that is a trigger, and am re-running. Vacuum (cron or auto) has no impact on the problem for 8.1beta3. Tom had mentioned running oprofile, but I'm unable to get either 0.9.1 or 0.9.0 versions of oprofile working, so looks like I'll have to get a reproducible test case... Anyone know of a script that can replay a PostgreSQL log file? Then I could log all queries, wait till the problem hits, and then replay to see if that reproduces it... Cheers, Rob ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[HACKERS] sort_mem statistics ...
do we maintain anything anywhere for this? mainly, some way of determining # of 'sorts to disk' vs 'sort in memory', to determine whether or not sort_mem is set to a good value? I don't think there is currently, but wondering how hard it would be to get something like this added ... ? thanks .. Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
[HACKERS] 8.04 and RedHat/CentOS init script issue
Hi, I installed 8.04 via RPM on Centos 4.2 which is the same as RedHat 4.2 and while booting the init script reports that the daemon [FAILED], but after I logon it shows the postmaster running and I am able to connect from any client remotely. I made not modifcations to the script and there is nothing out of the ordinary in the log. Thanks, Tony ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] Alpha: HEAD: Failure
Larry Rosenman ler@lerctr.org writes: Tom Lane wrote: Given that the failure was in infinite_recurse, I'm guessing the problem is that the stack limit is less than our default expectation (4MB I think). What would you like me to set it to, 4mb will do. and how do we make sure that the buildfarm stuff Sets it appropriately? This regression test is how ;-) Come to think of it, I should add an entry to the interpreting the regession tests doco about this. Will do it in a bit. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] Alpha: HEAD: Failure
Larry Rosenman ler@lerctr.org writes: I just tried(!) to run the buildfarm script on a Alpha DS10L/NetBSD 2.0 box. It died in make check. Given that the failure was in infinite_recurse, I'm guessing the problem is that the stack limit is less than our default expectation (4MB I think). regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] Alpha: HEAD: Failure
Tom Lane wrote: Larry Rosenman ler@lerctr.org writes: I just tried(!) to run the buildfarm script on a Alpha DS10L/NetBSD 2.0 box. It died in make check. Given that the failure was in infinite_recurse, I'm guessing the problem is that the stack limit is less than our default expectation (4MB I think). regards, tom lane What would you like me to set it to, and how do we make sure that the buildfarm stuff Sets it appropriately? LER -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 US ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] Question about Ctrl-C and less
I run into this problem sometimes, especially when I realize that the query I've just started is going to run for a very long time and not really provide anything useful. I find that I have to close the shell window to get out of it, and I'm always a bit uncomforatble doing that. -Kevin Martijn van Oosterhout kleptog@svana.org At the moment I'm just looking for a concensus that it's a problem to be solved. ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] sort_mem statistics ...
Marc G. Fournier [EMAIL PROTECTED] writes: do we maintain anything anywhere for this? mainly, some way of determining # of 'sorts to disk' vs 'sort in memory', to determine whether or not sort_mem is set to a good value? As of 8.1 you could turn on trace_sort to collect some data about this. Looking at the code, I notice that the messages are all emitted at level NOTICE. Perhaps that was not such a good idea --- it'd be pretty much in-your-face if it were on all the time. Does anyone think it'd be a good idea to emit the trace_sort messages at level LOG, instead? regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] A costing analysis tool
Thanks to all who have been offering suggestions. I have been
reading them and will try to incorporate as much as possible.
I have already reworked that little brain-dead python script into
something which uses a regular expression to pick off all of the
data from each cost/timing line (including the first one), and
tracks the heirarchy. I'll put all of these into the analysis
database.
Due to client operational problems I've been called in on, I
haven't done much more than that so far. I'll try to firm up a
proposed schema for the data soon, and some idea of what a
test case definition will look like. Then I'll probably have to
set it aside for two or three weeks.
I'll attach the current plan scanner for review, comment,
improvement. Also, someone may want to look at the results
from a few queries to get ideas going on how they want to use
the data.
Regarding the idea of a site where results could be posted
and loaded into a database which would be available for
public access -- I agree that would be great; however, my
client is not willing to take that on. If anyone wants to
volunteer, that wuold be fantastic.
-Kevin
Jim C. Nasby [EMAIL PROTECTED]
On Fri, Oct 14, 2005 at 03:34:43PM -0500, Kevin Grittner wrote:
of the two times as a reliability factor. Unfortunately, that
means doubling the number of cache flushes, which is likely
to be the most time-consuming part of running the tests. On
the bright side, we would capture the top level runtimes you
want.
Actually, if you shut down the database and run this bit of code with a
high enough number you should have a nicely cleaned cache.
int main(int argc, char *argv[]) {
if (!calloc(atoi(argv[1]), 1024*1024)) { printf(Error allocating
memory.\n); }
}
Running that on a dual Opteron (842's, I think) gives:
[EMAIL PROTECTED]:35]~:10time ./a.out 3300
3.142u 8.940s 0:40.62 29.7% 5+4302498k 0+0io 2pf+0w
That was on http://stats.distributed.net and resulted in about 100MB
being paged to disk. With 3000 it only took 20 seconds, but might not
have cleared 100% of memory.
parse_plan.py
Description: Binary data
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
Tom Lane wrote: Martijn van Oosterhout kleptog@svana.org writes: On Tue, Oct 18, 2005 at 04:04:42PM -0500, Larry Rosenman wrote: Upped the stack to 8Mb. Now it dies in Plcheck. Wierd, it's dying in malloc() because the C library called kill() from __libc_mutex_unlock(). I wonder if this is related to the threaded libpython doesn't work problem we've seen on some BSDen. Does this platform have separate implementations of libc for threaded and unthreaded applications? If so, and if libperl is trying to pull in a threaded libc along with itself, maybe this is the symptom you'd see. It's reasonably probable that this is the first call to malloc() after libperl has been loaded into the backend ... regards, tom lane Doesn't appear to have a separate libc, HOWEVER, -lpthread may be screwing us: $ ldd perl perl: -lm.0 = /usr/lib/libm.so.0 -lcrypt.0 = /usr/lib/libcrypt.so.0 -lpthread.0 = /usr/lib/libpthread.so.0 -lperl = /usr/pkg/lib/perl5/5.8.0/alpha-netbsd-thread-multi/CORE/libperl.so -lc.12 = /usr/lib/libc.so.12 $ I'm not the machines owner, but I can ask if we can get a NON-threaded PERL. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 US ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
Martijn van Oosterhout kleptog@svana.org writes: On Tue, Oct 18, 2005 at 04:04:42PM -0500, Larry Rosenman wrote: Upped the stack to 8Mb. Now it dies in Plcheck. Wierd, it's dying in malloc() because the C library called kill() from __libc_mutex_unlock(). I wonder if this is related to the threaded libpython doesn't work problem we've seen on some BSDen. Does this platform have separate implementations of libc for threaded and unthreaded applications? If so, and if libperl is trying to pull in a threaded libc along with itself, maybe this is the symptom you'd see. It's reasonably probable that this is the first call to malloc() after libperl has been loaded into the backend ... regards, tom lane ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Tue, Oct 18, 2005 at 04:04:42PM -0500, Larry Rosenman wrote: Upped the stack to 8Mb. Now it dies in Plcheck. Logs/bt.out in: http://www.lerctr.org/~ler/alphadeath2.tar.gz Wierd, it's dying in malloc() because the C library called kill() from __libc_mutex_unlock(). You're not the only one though: http://archive.netbsd.se/?ml=netbsd-usersa=2004-01m=18027 No-one answered that one either... -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. pgp2iUhqV0cV1.pgp Description: PGP signature
Re: [HACKERS] Character Conversions Handling
On Tue, Oct 18, 2005 at 10:29:30PM +0300, Volkan YAZICI wrote: Hi, I'm trying to understand the schema laying behind backend/utils/adt/like.c to downcase letters [1]. When I look at the other tolower() implementations, there're lots of them spread around. (In interfaces/libpq, backend/regex, backend/utils/adt/like and etc.) For example, despite having pg_wc_tolower() function in regc_locale.c, achieving same with manually in iwchareq() of like.c. I'd so appreciated if somebody can point me the places where I should start to look at to understand the character handling with different encodings. Also, I wonder why didn't we use any btow/mbsrtowc/wctomb like functions. Is this for portability with other compilers? PostgreSQL has to be compatable across many platforms, including those that don't have any multibyte support, and there are a few of those. Just like PostgreSQL includes a complete copy of the timezone library, so various bits usually handled by system libraries have been incorporated into the backend. This include encoding support. [1] iwchareq() is using pg_mb2wchar_with_len() which decides the right mb2wchar function from pg_wchar_table. When I look at backend/mb/wchar.c there're some other specific to locale mblen and mb2wchar routines. For example, EUC_KR is handled with pg_euc2wchar_with_len() function, but LATIN5 is handled with pg_latin12wchar_with_len() function. Will we write a new function for latin5 like pg_latin52wchar_with_len() if we'd encounter with a new problem with latin5? In this particular case it's not an issue since all the Latin-N encodings are all single byte encodings, they don't have to be handled seperately. But yes, this means that PostgreSQL's behaviour may vary from that of the surrounding system. The current planning is to use a cross-platform library (ICU) to handle all the locale and encoding related issues. This is a large task and I wouldn't be surprised if it takes a release or two. Hopefully it will clean all these issues up... Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. pgpQ5IdfNikO5.pgp Description: PGP signature
Re: [HACKERS] Seeing context switch storm with 10/13 snapshot of
Robert, Anyone know of a script that can replay a PostgreSQL log file? Then I could log all queries, wait till the problem hits, and then replay to see if that reproduces it... log_statement=true in your postgresql.conf file. The trick is weeding out all the other non-query stuff. -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] sort_mem statistics ...
On Tue, 18 Oct 2005, Tom Lane wrote: Marc G. Fournier [EMAIL PROTECTED] writes: On Tue, 18 Oct 2005, Tom Lane wrote: Looking at the code, I notice that the messages are all emitted at level NOTICE. Perhaps that was not such a good idea --- it'd be pretty much in-your-face if it were on all the time. Does anyone think it'd be a good idea to emit the trace_sort messages at level LOG, instead? If someone sets trace_sort, does it matter what level its emit'd at? Well, yeah. It depends whether you are thinking of the trace feature as being used interactively, or as something turned on to gather data over time in a production installation. In the second case you'd want the info to go to the postmaster log, but not want to see it dumped on your terminal all the time ... Oops, sorry, I was thinking in terms of syslog log levels ... :( Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] sort_mem statistics ...
Marc G. Fournier [EMAIL PROTECTED] writes: On Tue, 18 Oct 2005, Tom Lane wrote: Looking at the code, I notice that the messages are all emitted at level NOTICE. Perhaps that was not such a good idea --- it'd be pretty much in-your-face if it were on all the time. Does anyone think it'd be a good idea to emit the trace_sort messages at level LOG, instead? If someone sets trace_sort, does it matter what level its emit'd at? Well, yeah. It depends whether you are thinking of the trace feature as being used interactively, or as something turned on to gather data over time in a production installation. In the second case you'd want the info to go to the postmaster log, but not want to see it dumped on your terminal all the time ... regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
I think in this case you need to try to install the PL manually and see what happens - run psql, attach the debugger to the backend, and issue create language plperl ... Having it die at this stage is rather strange. cheers andrew Larry Rosenman wrote: Upped the stack to 8Mb. Now it dies in Plcheck. ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] sort_mem statistics ...
On Tue, 18 Oct 2005, Tom Lane wrote: Marc G. Fournier [EMAIL PROTECTED] writes: do we maintain anything anywhere for this? mainly, some way of determining # of 'sorts to disk' vs 'sort in memory', to determine whether or not sort_mem is set to a good value? As of 8.1 you could turn on trace_sort to collect some data about this. Looking at the code, I notice that the messages are all emitted at level NOTICE. Perhaps that was not such a good idea --- it'd be pretty much in-your-face if it were on all the time. Does anyone think it'd be a good idea to emit the trace_sort messages at level LOG, instead? If someone sets trace_sort, does it matter what level its emit'd at? Its not on by default, at least :) Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Question about Ctrl-C and less
On Tue, Oct 18, 2005 at 05:15:20PM -0500, Kevin Grittner wrote: I run into this problem sometimes, especially when I realize that the query I've just started is going to run for a very long time and not really provide anything useful. I find that I have to close the shell window to get out of it, and I'm always a bit uncomforatble doing that. Hmm, I'm glad it isn't just me experiencing this on a regular basis. It's possibly also because Debian sets the default pager to less, which makes it happen on every machine I use. The simple patch I posted seems unlikely to make it into 8.1 so I'll add a more comprehensive patch to my list hopefully for 8.2... Have a nice day, -- Martijn van Oosterhout kleptog@svana.org http://svana.org/kleptog/ Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a tool for doing 5% of the work and then sitting around waiting for someone else to do the other 95% so you can sue them. pgpaHwIYul74p.pgp Description: PGP signature
Re: [HACKERS] sort_mem statistics ...
Satoshi, And I want to get statistic info through system views, like pg_statio_*. I don't think anyone disagrees with that. It's just a little too late to get in for 8.1. -- --Josh Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] sort_mem statistics ...
Josh Berkus wrote: And I want to get statistic info through system views, like pg_statio_*. I don't think anyone disagrees with that. It's just a little too late to get in for 8.1. Thanks for comment. I hope 8.2 will get it. -- NAGAYASU Satoshi [EMAIL PROTECTED] ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Oct 18, 2005, at 5:11 PM, Martijn van Oosterhout wrote: On Tue, Oct 18, 2005 at 05:03:35PM -0500, Larry Rosenman wrote: Doesn't appear to have a separate libc, HOWEVER, -lpthread may be screwing us: snip If it is that, does it work if you compile postgres with -lpthread. Sure, we don't use the functions but maybe it's a prerequisite to be able to dlopen() thread libs. Should be quicker to test, just rerun the final link command by hand with the extra option... I added a LIBS += -lpthread to the end of src/makefiles/ Makefile.netbsd and got a LOOP on the make check :( More ideas? LER -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-351-4152 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 PGP.sig Description: This is a digitally signed message part
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Oct 18, 2005, at 8:49 PM, Larry Rosenman wrote: On Oct 18, 2005, at 5:11 PM, Martijn van Oosterhout wrote: On Tue, Oct 18, 2005 at 05:03:35PM -0500, Larry Rosenman wrote: Doesn't appear to have a separate libc, HOWEVER, -lpthread may be screwing us: snip If it is that, does it work if you compile postgres with -lpthread. Sure, we don't use the functions but maybe it's a prerequisite to be able to dlopen() thread libs. Should be quicker to test, just rerun the final link command by hand with the extra option... I added a LIBS += -lpthread to the end of src/makefiles/ Makefile.netbsd and got a LOOP on the make check :( More ideas? LER I've removed the --with-perl from the config for now, and am re- running it yet again :) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-351-4152 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 PGP.sig Description: This is a digitally signed message part
Re: [HACKERS] sort_mem statistics ...
Isn't that what pg_stat_database reports with its xact_commit and xact_rollback values? Ah yes. Doh :) Chris ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
Larry Rosenman ler@lerctr.org writes: I added a LIBS += -lpthread to the end of src/makefiles/ Makefile.netbsd and got a LOOP on the make check :( Er ... define LOOP? regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Oct 18, 2005, at 9:39 PM, Tom Lane wrote: Larry Rosenman ler@lerctr.org writes: I added a LIBS += -lpthread to the end of src/makefiles/ Makefile.netbsd and got a LOOP on the make check :( Er ... define LOOP? postgres master process sitting with 98%+ cpu for 1hour and NO progress being made. I could not find a truss/strace binary on the box :( regards, tom lane -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-351-4152 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Tue, Oct 18, 2005 at 09:41:21PM -0500, Larry Rosenman wrote: I could not find a truss/strace binary on the box :( In BSD land try ktrace. -- Michael Fuhr ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [HACKERS] 2nd try @NetBSD/2.0 Alpha
On Oct 18, 2005, at 10:03 PM, Michael Fuhr wrote: On Tue, Oct 18, 2005 at 08:59:23PM -0600, Michael Fuhr wrote: On Tue, Oct 18, 2005 at 09:41:21PM -0500, Larry Rosenman wrote: I could not find a truss/strace binary on the box :( In BSD land try ktrace. ...or attach with a debugger like gdb. d'oh. I go stupid occasionally :) If someone wants me to, I can try that. As to the without perl build, it dies in contribcheck. Logs in: http://www.lerctr.org/~ler/alphacontribdeath.tar.gz $ tar tzvf alphacontribdeath.tar.gz drwxr-xr-x 2 ler users 0 Oct 18 22:00 lastrun-logs -rw-r--r-- 1 ler users 22708 Oct 18 21:25 lastrun-logs/ CVS.log -rw-r--r-- 1 ler users 11453 Oct 18 21:29 lastrun-logs/ configure.log -rw-r--r-- 1 ler users 227987 Oct 18 21:29 lastrun-logs/ config.log -rw-r--r-- 1 ler users 154407 Oct 18 21:47 lastrun-logs/ make.log -rw-r--r-- 1 ler users 201363 Oct 18 21:50 lastrun-logs/ check.log -rw-r--r-- 1 ler users 40733 Oct 18 21:54 lastrun-logs/ make-contrib.log -rw-r--r-- 1 ler users 49358 Oct 18 21:54 lastrun-logs/ make-install.log -rw-r--r-- 1 ler users 1531 Oct 18 21:55 lastrun-logs/ initdb.log -rw-r--r-- 1 ler users 60 Oct 18 21:55 lastrun-logs/ startdb-1.log -rw-r--r-- 1 ler users 128491 Oct 18 21:57 lastrun-logs/ install-check.log -rw-r--r-- 1 ler users 65 Oct 18 21:57 lastrun-logs/ stopdb-1.log -rw-r--r-- 1 ler users 60 Oct 18 21:57 lastrun-logs/ startdb-2.log -rw-r--r-- 1 ler users711 Oct 18 21:57 lastrun-logs/pl- install-check.log -rw-r--r-- 1 ler users 64 Oct 18 21:57 lastrun-logs/ stopdb-2.log -rw-r--r-- 1 ler users 60 Oct 18 21:57 lastrun-logs/ startdb-3.log -rw-r--r-- 1 ler users 18425 Oct 18 21:57 lastrun-logs/ install-contrib.log -rw-r--r-- 1 ler users 41443 Oct 18 22:00 lastrun-logs/ contrib-install-check.log -rw-r--r-- 1 ler users 45714 Oct 18 22:00 lastrun-logs/web- txn.data tar: ustar vol 1, 19 files, 962560 bytes read, 0 bytes written in 1 secs (962560 bytes/sec) $ -- Michael Fuhr -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-351-4152 E-Mail: ler@lerctr.org US Mail: 3535 Gaspar Drive, Dallas, TX 75220-3611 ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [HACKERS] Question about Ctrl-C and less
Martijn van Oosterhout wrote: Very well, patch attached. It's quite simple actually. However, there seems to be some push back from people suggesting that jumping back to the main loop before the pager has quit is not buggy behaviour. Assuming that a ^C will kill the pager is just folly. Making assumptions about what the pager will do upon receipt of SIGINT is folly as well. Setting up SIGINT to be ignored may be the right answer (I don't believe it is -- see below), but if so then it needs to be done properly. If it gets ignored prior to the popen(), then the child will also end up ignoring it by default, because signal disposition is inherited by child processes. If we ignore SIGINT, it should be after the popen(), not before. When the user sends SIGINT, he means to interrupt whatever processing is currently occurring. He expects to regain control of the terminal. If psql is in the process of sending data to the pager, then a SIGINT should cause psql to stop doing so. So I think the right answer here is for psql to handle SIGINT internally by doing a pclose() first (and at this point, it probably should ignore SIGINT altogether), then returning to the main loop (and, of course, cleaning up anything that needs it along the way). If the child hasn't exited then pclose() will block until it has. The end result should be the semantics you want: if psql is in the middle of sending a bunch of rows of output to the pager, this will interrupt that process. If the pager remains running then it will hopefully give the user the ability to scroll through whatever results were sent to it. Tom asked if we should be blocking SIGQUIT and SIGHUP too. Standard procedure for spawning external interactive processes includes blocking SIGQUIT too (see system() manpage). SIGQUIT has a different standard meaning in Unix than SIGINT: it causes the process to drop core. We should not be blocking it -- we should be leaving it alone. The reason is that it's quite possible that the user wants to have psql generate a core file while it's writing output to the pager. Logically speaking, when the user sends an interrupt from the keyboard they expect to interrupt the currently active *interaxtive* process. They expect to interrupt the currently active processing. Not quite the same thing. Hence, once psql has spawned the pager, it should ignore such interrupts until control is returned (after pclose). So yes, I would suggest blocking SIGQUIT also, if only to prevent terminal corruption problems. Interactive programs like less trap SIGQUIT to restore the terminal settings on exit, but the exit anyway. They should be dropping core upon receipt of SIGQUIT. It might be nice if they cleaned up the terminal first, but receipt of a SIGQUIT generally means that the user wants to run the resulting core file through a debugger, and trapping the signal could alter the stack such that the resulting core would be less useful. I'd rather have to clean up the terminal manually than have an unusable core file. -- Kevin Brown [EMAIL PROTECTED] ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] sort_mem statistics ...
Isn't that what pg_stat_database reports with its xact_commit and xact_rollback values? On Wed, 19 Oct 2005, Christopher Kings-Lynne wrote: do we maintain anything anywhere for this? mainly, some way of determining # of 'sorts to disk' vs 'sort in memory', to determine whether or not sort_mem is set to a good value? I don't think there is currently, but wondering how hard it would be to get something like this added ... ? While on the subject of stats - is there any way to count the total transactions that have occurred since the last stats reset? Do we track that single number somewhere? Chris Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: [EMAIL PROTECTED] Yahoo!: yscrappy ICQ: 7615664 ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
