Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-24 Thread Robert Haas
On Thu, Nov 20, 2014 at 11:00 AM, Robert Haas wrote: > On Thu, Nov 20, 2014 at 7:30 AM, Amit Kapila wrote: >> Few compilation errors in the patch: >> 1>contrib\postgres_fdw\postgres_fdw.c(2107): error C2198: >> 'set_config_option' : too few arguments for call >> 1>contrib\postgres_fdw\postgres_fd

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-20 Thread Robert Haas
On Thu, Nov 20, 2014 at 7:30 AM, Amit Kapila wrote: > Few compilation errors in the patch: > 1>contrib\postgres_fdw\postgres_fdw.c(2107): error C2198: > 'set_config_option' : too few arguments for call > 1>contrib\postgres_fdw\postgres_fdw.c(2111): error C2198: > 'set_config_option' : too few argu

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-20 Thread Amit Kapila
On Wed, Nov 12, 2014 at 11:09 PM, Robert Haas wrote: > > On Wed, Nov 12, 2014 at 11:36 AM, Robert Haas wrote: > > On Wed, Nov 12, 2014 at 11:19 AM, Andres Freund wrote: > >> The question is whether the library is actually loaded in that case? > >> Because that normally only happens early during

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-12 Thread Robert Haas
On Wed, Nov 12, 2014 at 11:36 AM, Robert Haas wrote: > On Wed, Nov 12, 2014 at 11:19 AM, Andres Freund > wrote: >> The question is whether the library is actually loaded in that case? >> Because that normally only happens early during startup - which is why >> it's a PGC_BACKEND guc. > > It look

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-12 Thread Andres Freund
On 2014-11-12 11:36:14 -0500, Robert Haas wrote: > On Wed, Nov 12, 2014 at 11:19 AM, Andres Freund > wrote: > > The question is whether the library is actually loaded in that case? > > Because that normally only happens early during startup - which is why > > it's a PGC_BACKEND guc. > > It looks

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-12 Thread Robert Haas
On Wed, Nov 12, 2014 at 11:19 AM, Andres Freund wrote: > The question is whether the library is actually loaded in that case? > Because that normally only happens early during startup - which is why > it's a PGC_BACKEND guc. It looks like that does not work. [rhaas pgsql]$ PGOPTIONS='-c local_pr

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-12 Thread Andres Freund
On 2014-11-12 11:15:26 -0500, Robert Haas wrote: > On Mon, Nov 10, 2014 at 4:58 PM, Andres Freund wrote: > > No, that's not what I was thinking of. Consider: > > > > export pg_libdir=$(pg_config --libdir) > > mkdir $pg_libdir/plugins > > ln -s $pg_libdir/auto_explain.so $pg_libdir/plugins/ > > PG_

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-12 Thread Robert Haas
On Mon, Nov 10, 2014 at 4:58 PM, Andres Freund wrote: > No, that's not what I was thinking of. Consider: > > export pg_libdir=$(pg_config --libdir) > mkdir $pg_libdir/plugins > ln -s $pg_libdir/auto_explain.so $pg_libdir/plugins/ > PG_OPTIONS='-c local_preload_libraries=auto_explain' psql > > and

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Andres Freund
On 2014-11-10 14:54:15 -0500, Robert Haas wrote: > On Mon, Nov 10, 2014 at 1:29 PM, Andres Freund wrote: > >> That's an issue to which we may need to engineer some solution, but > >> not in this patch. Overall, the patch's architecture is modeled > >> closely on the way we synchronize GUCs to new

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Stephen Frost wrote: > > at least for 9.5, which is coming up awful fast at this point, if > > we're going to stick with the 'normal' schedule and freeze in the > > spring. > > https://wiki.postgresql.org/wiki/PgCon_2014_Developer_Meeting#9.5_Sc

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Robert Haas
On Mon, Nov 10, 2014 at 1:29 PM, Andres Freund wrote: >> That's an issue to which we may need to engineer some solution, but >> not in this patch. Overall, the patch's architecture is modeled >> closely on the way we synchronize GUCs to new backends when using >> EXEC_BACKEND, and I don't think w

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Alvaro Herrera
Stephen Frost wrote: > at least for 9.5, which is coming up awful fast at this point, if > we're going to stick with the 'normal' schedule and freeze in the > spring. https://wiki.postgresql.org/wiki/PgCon_2014_Developer_Meeting#9.5_Schedule Doesn't look all that "normal" to me, with that commit

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > > 1. Any other opinions for or against pg_background as a concept? I > > thought the ability to kick of vacuums (or other stuff with > > PreventTransactionChain) asynchronously was pretty cool, as we as the > > ability to use it as an authenticatio

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Andres Freund
On 2014-11-10 12:10:49 -0500, Robert Haas wrote: > - Patch 4 has had some review and really, as far as I can tell, the > only really major issue that has cropped up is the possibility that > the GUC settings that are legal in backend A aren't legal in backend B > for some reason; in that case, rest

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-11-10 Thread Robert Haas
So, summarizing the state of this patch set: - Patches 1, 2, 3, and 5 are committed. - Patch 4 has had some review and really, as far as I can tell, the only really major issue that has cropped up is the possibility that the GUC settings that are legal in backend A aren't legal in backend B for s

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-31 Thread Jim Nasby
On 10/24/14, 6:17 PM, Jim Nasby wrote: - Does anyone have a tangible suggestion for how to reduce the code duplication in patch #6? Between execute_sql_string() and tcop/exec_simple_query()? Is there stuff in exec_simple that's not safe for bgwriter? I'm not seeing why we can't use exec_simple.

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-30 Thread Andres Freund
Hi, On 2014-10-30 18:03:27 -0400, Robert Haas wrote: > On Wed, Oct 29, 2014 at 4:58 PM, Andres Freund wrote: > >> That's true. I don't know what to do about it. I'm somewhat inclined > >> to think that, if this remains in contrib, it's OK to ignore those > >> issues until such time as people co

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-30 Thread Robert Haas
On Wed, Oct 29, 2014 at 4:58 PM, Andres Freund wrote: >> That's true. I don't know what to do about it. I'm somewhat inclined >> to think that, if this remains in contrib, it's OK to ignore those >> issues until such time as people complain about them, because anybody >> who dislikes the things

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Andres Freund
On 2014-10-29 16:38:59 -0400, Robert Haas wrote: > On Wed, Oct 29, 2014 at 3:28 PM, Andres Freund wrote: > >> > Hm. So every user can do this once the extension is created as the > >> > functions are most likely to be PUBLIC. Is this a good idea? > >> > >> Why not? If they can log in, they could

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Robert Haas
On Wed, Oct 29, 2014 at 3:28 PM, Andres Freund wrote: >> > Hm. So every user can do this once the extension is created as the >> > functions are most likely to be PUBLIC. Is this a good idea? >> >> Why not? If they can log in, they could start separate sessions with >> similar effect. > > Connect

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Petr Jelinek
On 29/10/14 20:00, Robert Haas wrote: After reviewing all of those possibilities with the sort of laser-like focus the situation demands, I'm inclined to endorse Alvaro's proposal to rename the existing dsm_keep_mapping() function to dsm_pin_mapping() and the existing dsm_keep_segment() function

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Andres Freund
On 2014-10-22 19:03:28 -0400, Robert Haas wrote: > On Wed, Oct 8, 2014 at 6:32 PM, Andres Freund wrote: > > I got to ask: Why is it helpful that we have this in contrib? I have a > > good share of blame to bear for that, but I think we need to stop > > dilluting contrib evermore with test programs

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Andres Freund
On 2014-10-29 15:00:36 -0400, Robert Haas wrote: > 1. Does anyone strongly object to that course of action? I don't. > 2. Does anyone wish to argue for or against back-patching the name > changes to 9.4? +1. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadran

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-29 Thread Robert Haas
On Sat, Oct 25, 2014 at 7:01 AM, Alvaro Herrera wrote: > I do think that dsm_keep_mapping is a strange name for what it does. OK, so let me see if I can summarize the votes so far on this (highly important?) naming issue: - Andres doesn't like "unkeep". He suggests dsm_manage_mapping(), dsm_en

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-25 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Oct 24, 2014 at 4:46 PM, Jim Nasby wrote: > > On 10/24/14, 12:21 PM, Robert Haas wrote: > >> - What should we call dsm_unkeep_mapping, if not that? > > > > Only option I can think of beyond unkeep would be > > dsm_(un)register_keep_mapping. Dunno that it's worth it. >

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Petr Jelinek
On 24/10/14 23:07, Robert Haas wrote: On Fri, Oct 24, 2014 at 4:53 PM, Jim Nasby wrote: The only case I can think of would be actually connecting to a remote database; in that case would we even want something as raw as this? I suspect not, in which case I don't see an issue. On the other hand,

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/24/14, 4:03 PM, Robert Haas wrote: On Fri, Oct 24, 2014 at 4:46 PM, Jim Nasby wrote: On 10/24/14, 12:21 PM, Robert Haas wrote: - What should we call dsm_unkeep_mapping, if not that? Only option I can think of beyond unkeep would be dsm_(un)register_keep_mapping. Dunno that it's worth i

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Petr Jelinek
On 24/10/14 23:03, Robert Haas wrote: On Fri, Oct 24, 2014 at 4:46 PM, Jim Nasby wrote: On 10/24/14, 12:21 PM, Robert Haas wrote: - What should we call dsm_unkeep_mapping, if not that? Only option I can think of beyond unkeep would be dsm_(un)register_keep_mapping. Dunno that it's worth it.

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Robert Haas
On Fri, Oct 24, 2014 at 4:53 PM, Jim Nasby wrote: > The only case I can think of would be actually connecting to a remote > database; in that case would we even want something as raw as this? I > suspect not, in which case I don't see an issue. On the other hand, if we > ever think we might want t

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Robert Haas
On Fri, Oct 24, 2014 at 4:46 PM, Jim Nasby wrote: > On 10/24/14, 12:21 PM, Robert Haas wrote: >> - What should we call dsm_unkeep_mapping, if not that? > > Only option I can think of beyond unkeep would be > dsm_(un)register_keep_mapping. Dunno that it's worth it. Hmm, we could rename dsm_keep_ma

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/24/14, 3:26 PM, Robert Haas wrote: On Fri, Oct 24, 2014 at 3:30 PM, Jim Nasby wrote: On 10/24/14, 2:23 PM, Jim Nasby wrote: On the serialization structure itself, should we be worried about a mismatch between available GUCs on the sender vs the receiver? Presumably if the sender outputs

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/24/14, 12:21 PM, Robert Haas wrote: - What should we call dsm_unkeep_mapping, if not that? Only option I can think of beyond unkeep would be dsm_(un)register_keep_mapping. Dunno that it's worth it. - Does anyone have a tangible suggestion for how to reduce the code duplication in patch

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Robert Haas
On Fri, Oct 24, 2014 at 3:30 PM, Jim Nasby wrote: > On 10/24/14, 2:23 PM, Jim Nasby wrote: >> On the serialization structure itself, should we be worried about a >> mismatch between available GUCs on the sender vs the receiver? Presumably if >> the sender outputs a GUC that the receiver doesn't kn

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Robert Haas
On Fri, Oct 24, 2014 at 3:23 PM, Jim Nasby wrote: > Here's a review of patch 4. Thanks! > Perhaps it would be good to document the serialization format. I at least > would have found it helpful, especially when reading > estimate_variable_size(). I can take a stab at that if you'd rather not be

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/24/14, 2:23 PM, Jim Nasby wrote: On the serialization structure itself, should we be worried about a mismatch between available GUCs on the sender vs the receiver? Presumably if the sender outputs a GUC that the receiver doesn't know about we'll get an error, but what if the sender didn'

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/24/14, 12:21 PM, Robert Haas wrote: On Fri, Oct 24, 2014 at 1:13 PM, Jim Nasby wrote: It's a valid concern, but I think the way to handle it if needed is to limit the number of connections a user can open. Or perhaps another option would be to change the permissions on the related functio

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Robert Haas
On Fri, Oct 24, 2014 at 1:13 PM, Jim Nasby wrote: > It's a valid concern, but I think the way to handle it if needed is to limit > the number of connections a user can open. Or perhaps another option would > be to change the permissions on the related functions (do we check ACLs for > internal fun

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-24 Thread Jim Nasby
On 10/22/14, 7:49 PM, Joshua D. Drake wrote: I lean toward a push into core because: +1 3. I am not sure I buy into the super-user argument. Just because the functionality is there, doesn't mean it has to be used. It's a valid concern, but I think the way to handle it if needed is to limi

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-22 Thread Joshua D. Drake
What about moving it the opposite direction, from contrib all the way into core? The main advantage of that in my book is that it might make it easier to reduce the code duplication between pg_background and exec_simple_query; the main disadvantage is that it reduces the super-user's control,

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-22 Thread Robert Haas
On Wed, Oct 22, 2014 at 7:12 PM, Joshua D. Drake wrote: >> I don't think pg_background is merely a test program: I think it's a >> quite useful piece of functionality. It can be used for running >> VACUUM from a procedure, which is something people have asked for more >> than once, or for simulat

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-22 Thread Joshua D. Drake
On 10/22/2014 04:03 PM, Robert Haas wrote: On Wed, Oct 8, 2014 at 6:32 PM, Andres Freund wrote: I got to ask: Why is it helpful that we have this in contrib? I have a good share of blame to bear for that, but I think we need to stop dilluting contrib evermore with test programs. These have a p

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-22 Thread Robert Haas
On Wed, Oct 8, 2014 at 6:32 PM, Andres Freund wrote: > I got to ask: Why is it helpful that we have this in contrib? I have a > good share of blame to bear for that, but I think we need to stop > dilluting contrib evermore with test programs. These have a place, but I > don't think it should be co

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-14 Thread Petr Jelinek
On 09/10/14 00:32, Andres Freund wrote: From c835a06f20792556d35a0eee4c2fa21f5f23e8a3 Mon Sep 17 00:00:00 2001 From: Robert Haas Date: Fri, 11 Jul 2014 09:53:40 -0400 Subject: [PATCH 6/6] pg_background: Run commands in a background worker, and get the results. The currently-active GUC values

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-14 Thread Robert Haas
On Wed, Oct 8, 2014 at 6:32 PM, Andres Freund wrote: >> /* >> + * Arrange to remove a dynamic shared memory mapping at cleanup time. >> + * >> + * dsm_keep_mapping() can be used to preserve a mapping for the entire >> + * lifetime of a process; this function reverses that decision, making >> + *

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Amit Kapila
On Thu, Oct 9, 2014 at 4:02 AM, Andres Freund wrote: > > > /* > > + * Arrange to remove a dynamic shared memory mapping at cleanup time. > > + * > > + * dsm_keep_mapping() can be used to preserve a mapping for the entire > > + * lifetime of a process; this function reverses that decision, making

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-10-08 15:03:01 -0400, Robert Haas wrote: > #1 - Just committed, per discussion last week. > #2 - No comments, possibly because it's pretty boring. fine with me. > #3 - Most people seem happy with v2, modulo the above renaming request. If you do it without the error handling stuff, I'm go

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
> /* > + * Arrange to remove a dynamic shared memory mapping at cleanup time. > + * > + * dsm_keep_mapping() can be used to preserve a mapping for the entire > + * lifetime of a process; this function reverses that decision, making > + * the segment owned by the current resource owner. This may b

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Petr Jelinek
On 08/10/14 21:03, Robert Haas wrote: On Wed, Oct 8, 2014 at 10:09 AM, Andres Freund wrote: WRT my complaint in the other subthread, I think PQsendMethods should just be renamed to PQcommMethods or similar. That'd, in combination with a /* at some point we might want to add methods for receivi

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Robert Haas
On Wed, Oct 8, 2014 at 10:09 AM, Andres Freund wrote: > On 2014-09-10 16:53:12 -0400, Robert Haas wrote: >> diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h >> index 5da9d8d..0b8db42 100644 >> --- a/src/include/libpq/libpq.h >> +++ b/src/include/libpq/libpq.h >> @@ -37,6 +37,31 @

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-09-10 16:53:12 -0400, Robert Haas wrote: > diff --git a/src/include/libpq/libpq.h b/src/include/libpq/libpq.h > index 5da9d8d..0b8db42 100644 > --- a/src/include/libpq/libpq.h > +++ b/src/include/libpq/libpq.h > @@ -37,6 +37,31 @@ typedef struct > } u; > } PQArg

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-08 Thread Andres Freund
On 2014-09-29 13:38:37 -0400, Robert Haas wrote: > On Mon, Sep 29, 2014 at 12:05 PM, Stephen Frost wrote: > > Lastly, I will say that I feel it'd be good to support bi-directional > > communication as I think it'll be needed eventually, but I'm not sure > > that has to happen now. > > I agree we

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-02 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Oct 1, 2014 at 4:56 PM, Stephen Frost wrote: > > Was just thinking that we might be able to work out what needs to be > > done without having to actually do it on a per-character basis. That > > said, I'm not sure it's really worth t

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-02 Thread Robert Haas
On Wed, Oct 1, 2014 at 4:56 PM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> On Mon, Sep 29, 2014 at 12:05 PM, Stephen Frost wrote: >> > Perhaps I'm just being a bit over the top, but all this per-character >> > work feels a bit ridiculous.. When we're using MAXIMUM_ALI

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-10-01 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Sep 29, 2014 at 12:05 PM, Stephen Frost wrote: > > Perhaps I'm just being a bit over the top, but all this per-character > > work feels a bit ridiculous.. When we're using MAXIMUM_ALIGNOF, I > > suppose it's not so bad, but is there no hope t

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-29 Thread Robert Haas
On Mon, Sep 29, 2014 at 12:05 PM, Stephen Frost wrote: > Perhaps I'm just being a bit over the top, but all this per-character > work feels a bit ridiculous.. When we're using MAXIMUM_ALIGNOF, I > suppose it's not so bad, but is there no hope to increase that and make > this whole process more ef

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-29 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > Attached is a contrib module that lets you launch arbitrary command in > a background worker, and supporting infrastructure patches for core. Very cool! Started looking into this while waiting on a few CLOBBER_CACHE_ALWAYS runs to finish (ug

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-26 Thread Robert Haas
On Sat, Sep 20, 2014 at 3:03 AM, Amit Kapila wrote: > Okay, but as there is no predictability (it can be either same as what > launching process has at the when it has launched background worker > or it could be some different value if got changed later due to sighup) > which GUC value will be use

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-20 Thread Amit Kapila
On Fri, Sep 12, 2014 at 12:07 AM, Robert Haas wrote: > On Thu, Sep 11, 2014 at 7:34 AM, Amit Kapila wrote: > > Don't we need some way to prohibit changing GUC by launching process, > > once it has shared the existing GUC? > > Nope. I mean, eventually, for true parallelism ... we absolutely will

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-11 Thread Andres Freund
On 2014-09-11 21:27:15 +0200, Petr Jelinek wrote: > On 11/09/14 20:37, Robert Haas wrote: > >OK. I still think we should go back and PGDLLIMPORT-ize all the GUC > >variables. > > +1 Same here. I think Tom was the only one against it, but pretty much everyone else was for it. We should fix all t

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-11 Thread Petr Jelinek
On 11/09/14 20:37, Robert Haas wrote: 1. This patch generates warning on windows 1>pg_background.obj : error LNK2001: unresolved external symbol StatementTimeout You need to add PGDLLIMPORT for StatementTimeout OK. I still think we should go back and PGDLLIMPORT-ize all the GUC variables. +

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-11 Thread Robert Haas
On Thu, Sep 11, 2014 at 7:34 AM, Amit Kapila wrote: > Don't we need some way to prohibit changing GUC by launching process, > once it has shared the existing GUC? Nope. I mean, eventually, for true parallelism ... we absolutely will need that. But pg_background itself doesn't need that; it's pe

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-11 Thread Petr Jelinek
On 10/09/14 22:53, Robert Haas wrote: Here's what the other approach looks like. I can't really see doing this way and then only providing hooks for those two functions, so this is with hooks for all the send-side stuff. Original version: 9 files changed, 295 insertions(+), 3 deletions(-) This

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-11 Thread Amit Kapila
On Fri, Jul 25, 2014 at 11:41 PM, Robert Haas wrote: > > Patch 4 adds infrastructure that allows one session to save all of its > non-default GUC values and another session to reload those values. > This was written by Amit Khandekar and Noah Misch. It allows > pg_background to start up the backg

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-10 Thread Robert Haas
On Wed, Sep 10, 2014 at 4:01 PM, Robert Haas wrote: >> Yes, although my issue with the hooks was not that you only provided them >> for 2 functions, but the fact that it had no structure and the >> implementation was "if hook set do this, else do that" which I don't see >> like a good way of doing

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-10 Thread Robert Haas
On Tue, Sep 9, 2014 at 6:03 PM, Petr Jelinek wrote: >> - The remaining functions are pq_init(), pq_comm_reset(), pq_flush(), >> pq_flush_if_writable(), pq_is_send_pending(), pq_putmessage(), >> pq_putmessage_noblock(), pq_startcopyout(), and pq_endcopyout(). >> These are the ones that I think are

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Amit Kapila
On Wed, Sep 10, 2014 at 2:18 AM, Robert Haas wrote: > > On Tue, Sep 9, 2014 at 1:18 PM, Petr Jelinek wrote: > >> I think that's completely wrong. As the patch series demonstrates, > >> it's not limited to propagating ErrorResponse and NoticeResponse. It > >> can also propagate NotifyResponse an

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Petr Jelinek
On 09/09/14 22:48, Robert Haas wrote: On Tue, Sep 9, 2014 at 1:18 PM, Petr Jelinek wrote: I think that's completely wrong. As the patch series demonstrates, it's not limited to propagating ErrorResponse and NoticeResponse. It can also propagate NotifyResponse and RowDescription and DataRow an

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Robert Haas
On Tue, Sep 9, 2014 at 1:18 PM, Petr Jelinek wrote: >> I think that's completely wrong. As the patch series demonstrates, >> it's not limited to propagating ErrorResponse and NoticeResponse. It >> can also propagate NotifyResponse and RowDescription and DataRow and >> anything else that comes al

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Petr Jelinek
On 09/09/14 18:49, Robert Haas wrote: On Tue, Sep 9, 2014 at 12:33 PM, Petr Jelinek wrote: I am also not sure that I like the pq_redirect_to_shm_mq being directly exposed for use in bgworker, what I would like is to have elog interface to which you tell that you want errors sent to shm_mq handl

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Robert Haas
On Tue, Sep 9, 2014 at 12:33 PM, Petr Jelinek wrote: > I am not sure if what Andres proposed is the right solution, but I do agree > here that the hook definitely isn't. Well, that doesn't get me very far. Andres never responded to my previous comments about why I did it that way, and you're not

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Petr Jelinek
On 29/07/14 18:51, Robert Haas wrote: On Mon, Jul 28, 2014 at 1:50 PM, Andres Freund wrote: What I'm thinking of is providing an actual API for the writes instead of hooking into the socket API in a couple places. I.e. have something like typedef struct DestIO DestIO; struct DestIO { voi

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-09 Thread Amit Kapila
On Tue, Sep 9, 2014 at 2:32 AM, Robert Haas wrote: > On Mon, Sep 8, 2014 at 2:05 AM, Amit Kapila wrote: > >> Another point about error handling is that to execute the sql in > >> function pg_background_worker_main(), it starts the transaction > >> which I think doesn't get aborted if error occurs

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-08 Thread Robert Haas
On Mon, Sep 8, 2014 at 2:05 AM, Amit Kapila wrote: >> Another point about error handling is that to execute the sql in >> function pg_background_worker_main(), it starts the transaction >> which I think doesn't get aborted if error occurs > > For this I was just referring error handling code of >

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-08 Thread Robert Haas
On Mon, Sep 8, 2014 at 1:09 AM, Amit Kapila wrote: >> > Don't you need a PG_TRY block here to reset pq_mq_busy? >> >> No. If shm_mq_sendv is interrupted, we can't use the shm_mq any more. >> But since that should only happen if an interrupt arrives while the >> queue is full, I think that's OK. >

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-07 Thread Amit Kapila
On Mon, Sep 8, 2014 at 10:39 AM, Amit Kapila wrote: > > On Sat, Jul 26, 2014 at 9:32 PM, Robert Haas wrote: > > On Fri, Jul 25, 2014 at 4:16 PM, Alvaro Herrera > > wrote: > > > On Fri, Jul 25, 2014 at 02:11:32PM -0400, Robert Haas wrote: > > >> + pq_mq_busy = true; > > >> + > > >> + iov[

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-09-07 Thread Amit Kapila
On Sat, Jul 26, 2014 at 9:32 PM, Robert Haas wrote: > On Fri, Jul 25, 2014 at 4:16 PM, Alvaro Herrera > wrote: > > On Fri, Jul 25, 2014 at 02:11:32PM -0400, Robert Haas wrote: > >> + pq_mq_busy = true; > >> + > >> + iov[0].data = &msgtype; > >> + iov[0].len = 1; > >> + iov[1].data

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-29 Thread Robert Haas
On Mon, Jul 28, 2014 at 1:50 PM, Andres Freund wrote: > Don't get me wrong, I don't object to anything in here. It's just that > the bigger picture can help giving sensible feedback. Right. I did not get you wrong. :-) The reason I'm making a point of it is that, if somebody wants to object to

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-28 Thread Andres Freund
On 2014-07-26 12:20:34 -0400, Robert Haas wrote: > On Sat, Jul 26, 2014 at 4:37 AM, Andres Freund wrote: > > On 2014-07-25 14:11:32 -0400, Robert Haas wrote: > >> Attached is a contrib module that lets you launch arbitrary command in > >> a background worker, and supporting infrastructure patches

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-26 Thread Robert Haas
On Sat, Jul 26, 2014 at 4:37 AM, Andres Freund wrote: > On 2014-07-25 14:11:32 -0400, Robert Haas wrote: >> Attached is a contrib module that lets you launch arbitrary command in >> a background worker, and supporting infrastructure patches for core. > > Cool. > > I assume this 'fell out' of the w

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-26 Thread Robert Haas
On Fri, Jul 25, 2014 at 4:16 PM, Alvaro Herrera wrote: > On Fri, Jul 25, 2014 at 02:11:32PM -0400, Robert Haas wrote: >> + pq_mq_busy = true; >> + >> + iov[0].data = &msgtype; >> + iov[0].len = 1; >> + iov[1].data = s; >> + iov[1].len = len; >> + >> + Assert(pq_mq_handle !=

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-26 Thread Andres Freund
Hi, On 2014-07-25 14:11:32 -0400, Robert Haas wrote: > Attached is a contrib module that lets you launch arbitrary command in > a background worker, and supporting infrastructure patches for core. Cool. I assume this 'fell out' of the work towards parallelism? Do you think all of the patches (ex

[HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-25 Thread Robert Haas
Attached is a contrib module that lets you launch arbitrary command in a background worker, and supporting infrastructure patches for core. You can launch queries and fetch the results back, much as you could do with a dblink connection back to the local database but without the hassles of dealing

Re: [HACKERS] pg_background (and more parallelism infrastructure patches)

2014-07-25 Thread Alvaro Herrera
On Fri, Jul 25, 2014 at 02:11:32PM -0400, Robert Haas wrote: > + pq_mq_busy = true; > + > + iov[0].data = &msgtype; > + iov[0].len = 1; > + iov[1].data = s; > + iov[1].len = len; > + > + Assert(pq_mq_handle != NULL); > + result = shm_mq_sendv(pq_mq_handle, iov, 2, false