Re: [PATCHES] pgbench on mingw needs fflush
Tatsuo Ishii [EMAIL PROTECTED] writes: Can we distinguish mingw case from others so that we could ifdef out the extra fflush()? Isn't the right fix a single-spot patch to force stderr into unbuffered mode? It's supposed to be that way already per spec. I could see a one-or-two-line patch to remind mingw of the buffer mode it's supposed to be using ... I'm not excited about making an ongoing commitment to remember to fflush() after every write call, especially not when no standard-compliant platform needs it. regards, tom lane ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] guc patch: Make variables fall back to default values
Peter Eisentraut [EMAIL PROTECTED] writes: Gregory Stark wrote: This is a pretty major user-visible change. While I'm strongly in favour of it it seems like there ought to be some documentation file touched by this, no? Or have I missed it? In my opinion, and possibly that of others who have worked on this issue, the old behavior was a pretty much a bug and now it works as expected. Not sure how to document that. It's a release-note item ... assuming that it doesn't get reverted in the near future. Could we have some attention to the all-red buildfarm? regards, tom lane ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] guc patch: Make variables fall back to default values
Tom Lane [EMAIL PROTECTED] writes: It's a release-note item ... assuming that it doesn't get reverted in the near future. Could we have some attention to the all-red buildfarm? It's not just a bug. There's code missing. The code seems to assume that all custom variables are strings. There are about half a dozen Assert(variable-vartype == PGC_STRING) throughout the patch. That's not true, plperl's use_strict is a boolean and we have DefineCustome*Variable functions for each type of variable. Perl bombs because plperl.use_strict is a boolean. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] pgbench on mingw needs fflush
On Tue, Mar 13, 2007 at 05:09:15PM +0900, ITAGAKI Takahiro wrote: Tatsuo Ishii [EMAIL PROTECTED] wrote: Can we distinguish mingw case from others so that we could ifdef out the extra fflush()? The buffered stderr might be a bug of mingw After a little research, I found that MSDN says the buffered stderr is a specifications on Windows somehow, not a bug. setvbuf() is better solution for the problem. This is more simple and no need to use ifdef. I was just going to suggest this, because this is what we already use in the backend (src/backend/main/main.c). Applied, but with the #ifdefs, because that's cleaner and that's also how we do it in the backend. //Magnus ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] guc patch: Make variables fall back to default values
On Tue, Mar 13, 2007 at 08:22:17AM +, Gregory Stark wrote:
The code seems to assume that all custom variables are strings. There are
about half a dozen Assert(variable-vartype == PGC_STRING) throughout the
patch. That's not true, plperl's use_strict is a boolean and we have
DefineCustome*Variable functions for each type of variable. Perl bombs
because plperl.use_strict is a boolean.
The attached patch removes those Asserts.
But this is not the whole story. I wonder why setting plperl.use_strict
is supposed to work at all? Where is the corresponding definition of
plperl as a custom variable class? I can add it manually to
postgresql.conf and make the regression tests work but is this the intended
way?
Joachim
Index: src/backend/utils/misc/guc.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/utils/misc/guc.c,v
retrieving revision 1.380
diff -c -r1.380 guc.c
*** src/backend/utils/misc/guc.c 12 Mar 2007 22:09:28 - 1.380
--- src/backend/utils/misc/guc.c 13 Mar 2007 10:03:26 -
***
*** 2658,2678
gen = guc_variables[idx];
- /*
- * Even though this function could delete other types of variables as well,
- * at the moment we only call it for custom variables that always have type
- * string.
- */
Assert(gen-group == CUSTOM_OPTIONS);
- Assert(gen-vartype == PGC_STRING);
! conf = (struct config_string *) gen;
! set_string_field(conf, conf-reset_val, NULL);
! set_string_field(conf, conf-tentative_val, NULL);
! for (stack = conf-gen.stack; stack; stack = prev)
{
! set_string_field(conf, stack-tentative_val.stringval, NULL);
! set_string_field(conf, stack-value.stringval, NULL);
prev = stack-prev;
pfree(stack);
}
--- 2658,2678
gen = guc_variables[idx];
Assert(gen-group == CUSTOM_OPTIONS);
! if (gen-vartype == PGC_STRING)
! {
! conf = (struct config_string *) gen;
! set_string_field(conf, conf-reset_val, NULL);
! set_string_field(conf, conf-tentative_val, NULL);
! }
! for (stack = gen-stack; stack; stack = prev)
{
! if (gen-vartype == PGC_STRING)
! {
! set_string_field(conf, stack-tentative_val.stringval, NULL);
! set_string_field(conf, stack-value.stringval, NULL);
! }
prev = stack-prev;
pfree(stack);
}
***
*** 2698,2706
gen = guc_variables[idx];
Assert(gen-group == CUSTOM_OPTIONS);
- Assert(gen-vartype == PGC_STRING);
-
- conf = (struct config_string *) gen;
/*
* Here we check whether it is safe to really delete the variable
--- 2698,2703
***
*** 2723,2731
* then been deleted from the configuration file should behave
* as if it had been introduced in the session.
*/
- Assert(gen-vartype == PGC_STRING);
gen-reset_source = PGC_S_DEFAULT;
! set_string_field(conf, conf-reset_val, NULL);
}
else
guc_delete_variable(name);
--- 2720,2731
* then been deleted from the configuration file should behave
* as if it had been introduced in the session.
*/
gen-reset_source = PGC_S_DEFAULT;
! if (gen-vartype == PGC_STRING)
! {
! conf = (struct config_string *) gen;
! set_string_field(conf, conf-reset_val, NULL);
! }
}
else
guc_delete_variable(name);
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatypes do not
match
Re: [PATCHES] Recalculating OldestXmin in a long-running vacuum
Heikki Linnakangas wrote: Alvaro Herrera wrote: Was this revisited? Arranging the tests has taken me longer than I thought, but I now finally have the hardware and DBT-2 set up. I just finished a pair of 2h tests with autovacuum off, and continuous vacuum of the stock table. I'm trying to get the results uploaded on some public website so we can all see and discuss them. I finally got around looking at this again. I ran two 24h test runs with DBT-2, one with the patch and one without. To get comparable, predictable results, I turned autovacuum off and run a manual vacuum in a loop on the stock-table alone. As expected, the steady-state of the stock table is smaller with the patch. But only by ~2%, that's slightly less than I expected. But what surprises me is that response times went up a with the patch. I don't know why. The full test results are here: http://community.enterprisedb.com/oldestxmin/ 92 was run with the patch, 93 without it. BTW: The iostat chart clearly shows the vacuum WAL flush problem. The WAL utilization jumps from ~20% to ~40% during the 2nd vacuum pass. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Recalculating OldestXmin in a long-running vacuum
Heikki Linnakangas wrote: I ran two 24h test runs with DBT-2, one with the patch and one without. To get comparable, predictable results, I turned autovacuum off and run a manual vacuum in a loop on the stock-table alone. As expected, the steady-state of the stock table is smaller with the patch. But only by ~2%, that's slightly less than I expected. But what surprises me is that response times went up a with the patch. I don't know why. Maybe because of increased contention of ProcArrayLock? (I assume you are using that, althought I haven't seen the patch) -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Recalculating OldestXmin in a long-running vacuum
Alvaro Herrera wrote: Heikki Linnakangas wrote: I ran two 24h test runs with DBT-2, one with the patch and one without. To get comparable, predictable results, I turned autovacuum off and run a manual vacuum in a loop on the stock-table alone. As expected, the steady-state of the stock table is smaller with the patch. But only by ~2%, that's slightly less than I expected. But what surprises me is that response times went up a with the patch. I don't know why. Maybe because of increased contention of ProcArrayLock? (I assume you are using that, althought I haven't seen the patch) I am, but I doubt that's it. The response times are dominated by I/O, so any increase in lock contention would hardly show up. And the patch is only adding one GetOldestXmin call every 1000 scanned pages, which is nothing compared to the thousands of GetSnapshot calls the normal transactions are issuing per minute. The patch must have changed the I/O pattern in some subtle way. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] guc patch: Make variables fall back to default values
Gregory Stark [EMAIL PROTECTED] writes: It's not just a bug. There's code missing. The code seems to assume that all custom variables are strings. There are about half a dozen Assert(variable-vartype == PGC_STRING) throughout the patch. That's not true, plperl's use_strict is a boolean and we have DefineCustome*Variable functions for each type of variable. Well, they *are* strings as long as they're custom. Once a DefineCustomFoo has been executed, there (should be) no difference between a custom variable and a hard-wired one. The thing that I was wondering about is the same Joachim mentioned: how is it that the regression test ever worked? The answer is that it's not really testing custom variables, because it doesn't try to set plperl.use_strict until after plperl has been loaded into the current session. So by that time the variable exists and should look like a perfectly ordinary boolean GUC variable. The fact that it doesn't look like that says to me that there's something wrong with the patch logic, over and above the question of what it should be Asserting. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] guc patch: Make variables fall back to default values
Tom Lane wrote: The thing that I was wondering about is the same Joachim mentioned: how is it that the regression test ever worked? The answer is that it's not really testing custom variables, because it doesn't try to set plperl.use_strict until after plperl has been loaded into the current session. I think that the sole purpose of c_v_c is to allow custom variables in the configuration file, because that is possibly read before modules are loaded. Basically it just means that prefix.* is not rejected. In a session, it doesn't make a difference what c_v_c is set to; the variable needs to be registered period. However, if the registration code runs only when the module is invoked for the first time rather than at the start of the session (as in the case of plperl), then it's apparently impossible to set a variable in a session before the first call. It's all very weird. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] guc patch: Make variables fall back to default values
On Tue, Mar 13, 2007 at 10:19:54AM -0400, Tom Lane wrote: Gregory Stark [EMAIL PROTECTED] writes: It's not just a bug. There's code missing. The code seems to assume that all custom variables are strings. There are about half a dozen Assert(variable-vartype == PGC_STRING) throughout the patch. That's not true, plperl's use_strict is a boolean and we have DefineCustome*Variable functions for each type of variable. Well, they *are* strings as long as they're custom. Once a DefineCustomFoo has been executed, there (should be) no difference between a custom variable and a hard-wired one. The code in question is the only place that calls one of the DefineCustom*Variable functions. But those functions set var-group = CUSTOM_OPTIONS what makes variables look like custom variables defined via SQL or the config file but in reality they aren't. Hence the confusion of the type assertion. The thing that I was wondering about is the same Joachim mentioned: how is it that the regression test ever worked? The answer is that it's not really testing custom variables, because it doesn't try to set plperl.use_strict until after plperl has been loaded into the current session. So by that time the variable exists and should look like a perfectly ordinary boolean GUC variable. The fact that it doesn't look like that says to me that there's something wrong with the patch logic, over and above the question of what it should be Asserting. What is wrong is that plperl defines a variable that is a mix of a guc variable and a custom variable. It claims being a custom variable by setting var-group = CUSTOM_OPTIONS but it does not set the respective custom_variable_class and so by definition it can't be a custom variable. Joachim ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] guc patch: Make variables fall back to default values
Joachim Wieland [EMAIL PROTECTED] writes: On Tue, Mar 13, 2007 at 10:19:54AM -0400, Tom Lane wrote: Well, they *are* strings as long as they're custom. Once a DefineCustomFoo has been executed, there (should be) no difference between a custom variable and a hard-wired one. The code in question is the only place that calls one of the DefineCustom*Variable functions. But those functions set var-group = CUSTOM_OPTIONS what makes variables look like custom variables defined via SQL or the config file but in reality they aren't. Hence the confusion of the type assertion. My point here that you shouldn't be using var-group to make any semantic choices. That's supposed to be a label for user convenience, nothing else. regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] guc patch: Make variables fall back to default values
On Tue, Mar 13, 2007 at 11:08:52AM -0400, Tom Lane wrote: Joachim Wieland [EMAIL PROTECTED] writes: On Tue, Mar 13, 2007 at 10:19:54AM -0400, Tom Lane wrote: Well, they *are* strings as long as they're custom. Once a DefineCustomFoo has been executed, there (should be) no difference between a custom variable and a hard-wired one. The code in question is the only place that calls one of the DefineCustom*Variable functions. But those functions set var-group = CUSTOM_OPTIONS what makes variables look like custom variables defined via SQL or the config file but in reality they aren't. Hence the confusion of the type assertion. My point here that you shouldn't be using var-group to make any semantic choices. That's supposed to be a label for user convenience, nothing else. Then what is the criterion to tell what is a custom variable and what isn't? If it contains a dot in the name it is? This wouldn't resolve the problem at hand either... :-( We might have to think about custom variables as a whole, what we have now seems like a very unclear definition and everybody has his own opinion about what it is and how it works (and I'm not excluding myself here :-)). Joachim ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] guc patch: Make variables fall back to default values
Joachim Wieland [EMAIL PROTECTED] writes: On Tue, Mar 13, 2007 at 11:08:52AM -0400, Tom Lane wrote: My point here that you shouldn't be using var-group to make any semantic choices. That's supposed to be a label for user convenience, nothing else. Then what is the criterion to tell what is a custom variable and what isn't? Why do you need to tell that? IMHO, once the DefineCustomFoo function has been executed, it should be exactly like any other variable (other than having a funny name). regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] guc patch: Make variables fall back to default values
On Tue, Mar 13, 2007 at 11:52:38AM -0400, Tom Lane wrote: Then what is the criterion to tell what is a custom variable and what isn't? Why do you need to tell that? IMHO, once the DefineCustomFoo function has been executed, it should be exactly like any other variable (other than having a funny name). For example for the fall-back-to-default patch. I might not need to tell if it has been introduced by one of the DefineCustomFoo functions but for the other custom variables. Imagine that we have defined a custom variable via the configuration file, remove it and send SIGHUP. My understanding is that this variable should then be deleted from the pool of valid variables because it falls back to its default value and the default value of a custom variable is its non-existance. Joachim ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] guc patch: Make variables fall back to default values
Joachim Wieland [EMAIL PROTECTED] writes: On Tue, Mar 13, 2007 at 11:52:38AM -0400, Tom Lane wrote: Why do you need to tell that? IMHO, once the DefineCustomFoo function has been executed, it should be exactly like any other variable (other than having a funny name). For example for the fall-back-to-default patch. I might not need to tell if it has been introduced by one of the DefineCustomFoo functions but for the other custom variables. Imagine that we have defined a custom variable via the configuration file, remove it and send SIGHUP. My understanding is that this variable should then be deleted from the pool of valid variables because it falls back to its default value and the default value of a custom variable is its non-existance. Once DefineCustomFoo has been executed, you have a reset value to fall back to. I think what you really want is to recognize variables that are in the placeholder state, and have them go away as above. For that you check the GUC_CUSTOM_PLACEHOLDER flag. In any case there must never be any use of var-group for decision making; that's simply wrong. However, ISTM that forcing variables to go away is useless extra code. What purpose does it serve? Not error checking, because you already accepted the variable before. Surely you wouldn't argue that, say, reverting to a prior setting of check_function_bodies should cause the system to go back and validate a CREATE FUNCTION command it has already accepted. Moreover, while you could perhaps argue that the principle of least surprise cuts either way here, it seems to me there's a good argument for not throwing away variables: you might be discarding data the user needed. So I'd vote for just leaving them there. regards, tom lane ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
[PATCHES] Updated Packed Varlena patch v20 (final?)
Updated patch: I went through the high traffic areas and levelled them up to using the new macros to avoid detoasting smaller arguments. They key areas are really btree operators since they have a noticeable effect on sorts, especially index builds, when the data being sorted fits in memory. There is a bit of a name-game here. The macros I made are called VARDATA_ANY(datum) VARSIZE_ANY(datum) AND VARSIZE_ANY_EXHDR(datum). And the datatype macros are, for example, PG_GETARG_TEXT_PP() and DatumGetTextPP() -- short for packed pointer. Maybe not the prettiest names in the world but clear and I've found them pretty easy to spot when I'm looking for inconsistent use of VARSIZE_ANY-VARDATA for example. I thought of PVARSIZE/PVARDATA (for packed) but I'm afraid it would camouflage such cases. Anyone have any better ideas? If not I'm satisfied with them... Except maybe VARSIZE_ANY_EXHDR() which seems too long. I got to almost everything in varlena.c and varchar.c so that includes text, bpchar, and bytea as well as varchar's few procedures. That includes probably more than I really needed to, but as long as the datatypes are working with bytes it's convenient enough. Also, I replaced my tweaks to hstore and pg_trgm with Teodore's. And of course updated CVS. http://community.enterprisedb.com/varlena/varvarlena-20.patch.gz I'm going to be putting this aside for a little while. I think it's really done. There's more that can be done of course, hit inet and numeric with the new macros, for instance. But I want to see what happens when it gets reviewed before I do that kind of bookkeeping. One thing that I've left in there again is the htonl/ntohl macros in the big-endian case. It really makes sense to either remove them or remove the #ifdef. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
[PATCHES] HOT WIP Patch - Version 4.4
Please see the attached version 4.4 of HOT WIP patch. I have fixed couple of bugs in the earlier version posted. Other than that there are not any significant changes in the patch. The row-level fragmentation had a bug where we were unintentionally sorting the line pointers array more than once. Also, the defragmented lengths were computed wrongly and was a source of many errors. Another bug fix was in the heap_hot_fetch() code path where we were asserting on finding a LP_DELETEd tuple in the hot chain. I later realized that this is not required and we should rather just assume that the chain is broken, something very similar to the xmax/xmin checks. Thanks, Pavan -- EnterpriseDBhttp://www.enterprisedb.com NewHOT-v4.4.patch.gz Description: GNU Zip compressed data ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
