On Thu, May 7, 2015 at 3:09 AM, Noah Misch wrote:
>> Each individual queue has only a single reader and a single writer.
>> In your example, there would be three queues, not just one. Of
>> course, one could design a new shared memory data structure
>> representing a collection of related queues,
On Tue, May 05, 2015 at 01:05:38PM -0400, Robert Haas wrote:
> On Sat, May 2, 2015 at 12:35 PM, Noah Misch wrote:
> > Perhaps, rather than model it as master M waiting on worker list W1|W2|W3,
> > model it with queue-nonempty and queue-nonfull events, one pair per queue.
That comment of mine was
On Sat, May 2, 2015 at 12:35 PM, Noah Misch wrote:
> Perhaps, rather than model it as master M waiting on worker list W1|W2|W3,
> model it with queue-nonempty and queue-nonfull events, one pair per queue.
Each individual queue has only a single reader and a single writer.
In your example, there w
On Tue, Apr 21, 2015 at 02:08:00PM -0400, Robert Haas wrote:
> On Tue, Mar 24, 2015 at 3:26 PM, Andres Freund wrote:
[proposal: teach deadlock detector about parallel master waiting on workers]
> > deadlock.c is far from simple, and at least I don't find the control
> > flow to be particularly c
On Wed, Apr 29, 2015 at 12:23 PM, Robert Haas wrote:
> So, I think it makes sense to split up this patch in two. There's no
> real debate, AFAICS, about anything in the patch other than the
> heavyweight locking stuff. So I'd like to go ahead and commit the
> rest. That's attached here as paral
On Sun, Apr 26, 2015 at 9:57 PM, Amit Langote
wrote:
> On 2015-04-22 AM 04:14, Robert Haas wrote:
>> We should check IsParallelWorker() for operations that are allowed in
>> the master during parallel mode, but not allowed in the workers - e.g.
>> the master can scan its own temporary relations, b
On Sun, Apr 26, 2015 at 2:03 PM, Euler Taveira wrote:
> On 19-03-2015 15:13, Robert Haas wrote:
>> On Wed, Mar 18, 2015 at 5:36 PM, Andres Freund
>> wrote:
>>> Reading the README first, the rest later. So you can comment on my
>>> comments, while I actually look at the code. Parallelism, yay!
>>
On 2015-04-22 AM 04:14, Robert Haas wrote:
>
> We should check IsParallelWorker() for operations that are allowed in
> the master during parallel mode, but not allowed in the workers - e.g.
> the master can scan its own temporary relations, but its workers
> can't. We should check IsInParallelMod
On 19-03-2015 15:13, Robert Haas wrote:
> On Wed, Mar 18, 2015 at 5:36 PM, Andres Freund wrote:
>> Reading the README first, the rest later. So you can comment on my
>> comments, while I actually look at the code. Parallelism, yay!
>
I'm also looking at this piece of code and found some low-hangi
On Tue, Apr 21, 2015 at 11:38 PM, Robert Haas wrote:
>
>
> After thinking about it a bit more, I realized that even if we settle
> on some solution to that problem, there's another issues: the
> wait-edges created by this system don't really have the same semantics
> as regular lock waits. Suppos
On Mon, Apr 20, 2015 at 6:49 PM, Peter Geoghegan wrote:
> I see that you're using git format-patch to generate this. But the
> patch is only patch 1/4. Is that intentional? Where are the other
> pieces?
>
> I think that the parallel seqscan patch, and the assessing parallel
> safety patch are inte
On Tue, Mar 24, 2015 at 11:56 PM, Alvaro Herrera
wrote:
>> Well, it's not actually the same message. They're all a bit
>> different. Or mostly all of them. And the variable part is not a
>> command name, as in the PreventTransactionChain() case, so it would
>> affect translatability if we did t
On Tue, Mar 24, 2015 at 3:26 PM, Andres Freund wrote:
>> You'd need some kind of
>> API that says "pretend I'm waiting for this lock, but don't really
>> wait for it", and you'd need to be darn sure that you removed yourself
>> from the wait queue again before doing any other heavyweight lock
>> m
On Thu, Mar 19, 2015 at 11:13 AM, Robert Haas wrote:
> Here is yet another version of this patch. In addition to the fixes
> mentioned above, this version includes some minor rebasing around
> recent commits, and also better handling of the case where we discover
> that we cannot launch workers a
Robert Haas wrote:
> On Wed, Mar 18, 2015 at 7:10 PM, Andres Freund wrote:
> >> + /*
> >> + * For now, parallel operations are required to be strictly
> >> read-only.
> >> + * Unlike heap_update() and heap_delete(), an insert should never
> >> create
> >> + * a combo CID, so
On Tue, Mar 24, 2015 at 3:26 PM, Andres Freund wrote:
>> >> + - The currently active user ID and security context. Note that this is
>> >> +the fourth user ID we restore: the initial step of binding to the
>> >> correct
>> >> +database also involves restoring the authenticated user ID.
Andres Freund writes:
> Also: Man, trying to understand the guts of deadlock.c only made me
> understand how *friggin* expensive deadlock checking is. I'm really
> rather surprised that it only infrequently causes problems.
The reason for that is that we only run deadlock checking if something's
On 2015-03-19 14:13:59 -0400, Robert Haas wrote:
> On Wed, Mar 18, 2015 at 5:36 PM, Andres Freund wrote:
> > Reading the README first, the rest later. So you can comment on my
> > comments, while I actually look at the code. Parallelism, yay!
>
> Sorry, we don't support parallelism yet. :-)
And
On Wed, Mar 18, 2015 at 7:10 PM, Andres Freund wrote:
> On 2015-03-18 12:02:07 -0400, Robert Haas wrote:
>> diff --git a/src/backend/access/heap/heapam.c
>> b/src/backend/access/heap/heapam.c
>> index cb6f8a3..173f0ba 100644
>> --- a/src/backend/access/heap/heapam.c
>> +++ b/src/backend/access/he
On 2015-03-18 12:02:07 -0400, Robert Haas wrote:
> diff --git a/src/backend/access/heap/heapam.c
> b/src/backend/access/heap/heapam.c
> index cb6f8a3..173f0ba 100644
> --- a/src/backend/access/heap/heapam.c
> +++ b/src/backend/access/heap/heapam.c
> @@ -2234,6 +2234,17 @@ static HeapTuple
> heap_
Hi,
Reading the README first, the rest later. So you can comment on my
comments, while I actually look at the code. Parallelism, yay!
On 2015-03-18 12:02:07 -0400, Robert Haas wrote:
> +Instead, we take a more pragmatic approach: we try to make as many of the
> +operations that are safe outside o
On Fri, Mar 6, 2015 at 7:01 AM, Amit Kapila wrote:
> Today, while testing parallel_seqscan patch, I encountered one
> intermittent issue (it hangs in below function) and I have question
> related to below function.
>
> +void
> +WaitForParallelWorkersToFinish(ParallelContext *pcxt)
> +{
> ..
> + fo
On Sun, Feb 15, 2015 at 11:48 AM, Robert Haas wrote:
>
> On Fri, Feb 13, 2015 at 2:22 AM, Michael Paquier
> wrote:
> > On Thu, Feb 12, 2015 at 3:59 AM, Robert Haas
wrote:
> >>
> >> We're not seeing eye to eye here yet, since I don't accept your
> >> example scenario and you don't accept mine. L
On 2015-02-11 13:59:04 -0500, Robert Haas wrote:
> On Tue, Feb 10, 2015 at 8:45 PM, Andres Freund wrote:
> > The only reason I'd like it to be active is because that'd *prohibit*
> > doing the crazier stuff. There seems little reason to not da it under
> > the additional protection against crazy t
On Thu, Feb 12, 2015 at 3:59 AM, Robert Haas wrote:
> We're not seeing eye to eye here yet, since I don't accept your
> example scenario and you don't accept mine. Let's keep discussing.
>
> Meanwhile, here's an updated patch.
>
A lot of cool activity is showing up here, so moved the patch to C
On Wed, Feb 11, 2015 at 1:59 PM, Robert Haas wrote:
> I'm not sure what you mean by the "severity" of the lock. How about
> marking the locks that the worker inherited from the parallel master
> and throwing an error if it tries to lock one of those objects in a
> mode that it does not already ho
On 2015-02-10 11:49:58 -0500, Robert Haas wrote:
> On Sat, Feb 7, 2015 at 7:20 PM, Andres Freund wrote:
> > * Don't like CreateParallelContextForExtension much as a function name -
> > I don't think we normally equate the fact that code is located in a
> > loadable library with the extension m
On Sat, Feb 7, 2015 at 7:20 PM, Andres Freund wrote:
> Observations:
> * Some tailing whitespace in the readme. Very nice otherwise.
Fixed. Thanks.
> * Don't like CreateParallelContextForExtension much as a function name -
> I don't think we normally equate the fact that code is located in a
Hi,
On 2015-02-06 22:43:21 -0500, Robert Haas wrote:
> Here's v4, with that fixed and a few more tweaks.
If you attached files generated with 'git format-patch' I could directly
apply then with the commit message and such. All at once if it's
mutliple patches, as individual commits. On nontrivia
On Wed, Feb 4, 2015 at 9:40 PM, Robert Haas wrote:
>
> On Fri, Jan 30, 2015 at 12:08 PM, Robert Haas
wrote:
> > Here's a new version. Andres mentioned previously that he thought it
> > would be a good idea to commit the addition of
> > BackgroundWorkerInitializeConnectionByOid() separately, as h
On Fri, Jan 30, 2015 at 6:38 PM, Jim Nasby wrote:
> On 1/30/15 11:08 AM, Robert Haas wrote:
>> The final patch attached her (parallel-dummy-v2.patch) has been
>> updated slightly to incorporate some prefetching logic. It's still
>> just demo code and is not intended for commit. I'm not sure whet
On 1/30/15 11:08 AM, Robert Haas wrote:
The final patch attached her (parallel-dummy-v2.patch) has been
updated slightly to incorporate some prefetching logic. It's still
just demo code and is not intended for commit. I'm not sure whether
the prefetching logic can actually be made to improve pe
On Wed, Jan 21, 2015 at 6:35 PM, Robert Haas wrote:
>
> On Wed, Jan 21, 2015 at 2:11 AM, Amit Kapila
wrote:
> > On Tue, Jan 20, 2015 at 9:52 PM, Robert Haas
wrote:
> >> On Tue, Jan 20, 2015 at 9:41 AM, Amit Kapila
> >> wrote:
> >> > It seems [WaitForBackgroundWorkerShutdown] has possibility to
On Wed, Jan 21, 2015 at 2:11 AM, Amit Kapila wrote:
> On Tue, Jan 20, 2015 at 9:52 PM, Robert Haas wrote:
>> On Tue, Jan 20, 2015 at 9:41 AM, Amit Kapila
>> wrote:
>> > It seems [WaitForBackgroundWorkerShutdown] has possibility to wait
>> > forever.
>> > Assume one of the worker is not able to s
On Tue, Jan 20, 2015 at 9:52 PM, Robert Haas wrote:
>
> On Tue, Jan 20, 2015 at 9:41 AM, Amit Kapila
wrote:
> > It seems [WaitForBackgroundWorkerShutdown] has possibility to wait
forever.
> > Assume one of the worker is not able to start (not able to attach
> > to shared memory or some other reas
On Tue, Jan 20, 2015 at 9:41 AM, Amit Kapila wrote:
> It seems [WaitForBackgroundWorkerShutdown] has possibility to wait forever.
> Assume one of the worker is not able to start (not able to attach
> to shared memory or some other reason), then status returned by
> GetBackgroundWorkerPid() will be
On Sat, Jan 17, 2015 at 3:40 AM, Robert Haas wrote:
>
> New patch attached. I'm going to take the risk of calling this v1
> (previous versions have been 0.x), since I've now done something about
> the heavyweight locking issue, as well as fixed the message-looping
> bug Amit pointed out. It doub
On 2015-01-05 11:27:57 -0500, Robert Haas wrote:
> On Sat, Jan 3, 2015 at 7:31 PM, Andres Freund wrote:
> > * I wonder if parallel contexts shouldn't be tracked via resowners
>
> That is a good question. I confess that I'm somewhat fuzzy about
> which things should be tracked via the resowner me
On Thu, Jan 15, 2015 at 9:09 AM, Amit Kapila wrote:
> On Thu, Jan 15, 2015 at 6:52 PM, Robert Haas wrote:
>> On Thu, Jan 15, 2015 at 7:00 AM, Amit Kapila
>> wrote:
>> > +HandleParallelMessages(void)
>> > +{
>> > ..
>> > ..
>> > + for (i = 0; i < pcxt->nworkers; ++i)
>> > + {
>> > + /*
>> > + * R
On Thu, Jan 15, 2015 at 6:52 PM, Robert Haas wrote:
>
> On Thu, Jan 15, 2015 at 7:00 AM, Amit Kapila
wrote:
> > +HandleParallelMessages(void)
> > +{
> > ..
> > ..
> > + for (i = 0; i < pcxt->nworkers; ++i)
> > + {
> > + /*
> > + * Read messages for as long as we have an error queue; if we
> > + *
On Thu, Jan 15, 2015 at 7:00 AM, Amit Kapila wrote:
> On Tue, Jan 13, 2015 at 1:33 AM, Robert Haas wrote:
>> On Thu, Jan 8, 2015 at 6:52 AM, Amit Kapila
>> wrote:
>> > + seg = dsm_attach(DatumGetInt32(main_arg));
>> >
>> > Here, I think DatumGetUInt32() needs to be used instead of
>> > DatumGetI
On Tue, Jan 13, 2015 at 1:33 AM, Robert Haas wrote:
>
> On Thu, Jan 8, 2015 at 6:52 AM, Amit Kapila
wrote:
> > + seg = dsm_attach(DatumGetInt32(main_arg));
> >
> > Here, I think DatumGetUInt32() needs to be used instead of
> > DatumGetInt32() as the segment handle is uint32.
>
> OK, I'll change t
On Thu, Jan 8, 2015 at 6:52 AM, Amit Kapila wrote:
> + seg = dsm_attach(DatumGetInt32(main_arg));
>
> Here, I think DatumGetUInt32() needs to be used instead of
> DatumGetInt32() as the segment handle is uint32.
OK, I'll change that in the next version.
--
Robert Haas
EnterpriseDB: http://www.e
On Wed, Jan 7, 2015 at 3:54 PM, Jim Nasby wrote:
> Agreed, I was more concerned with calls to nextval(), which don't seem to be
> prevented in parallel mode?
It looks prevented:
/*
* Forbid this during parallel operation because, to make it work,
* the cooperating backends would ne
On Wed, Jan 7, 2015 at 11:03 PM, Robert Haas wrote:
>
> I have little doubt that this version is still afflicted with various
> bugs, and the heavyweight locking issue remains to be dealt with, but
> on the whole I think this is headed in the right direction.
>
+ParallelMain(Datum main_arg)
{
..
On 1/7/15, 9:39 AM, Robert Haas wrote:
sequence.c: Is it safe to read a sequence value in a parallel worker if the
>cache_value is > 1?
No, because the sequence cache isn't synchronized between the workers.
Maybe it would be safe if cache_value == 1, but there's not much
use-case: how often are
On Tue, Jan 6, 2015 at 9:37 PM, Jim Nasby wrote:
> CreateParallelContext(): Does it actually make sense to have nworkers=0?
> ISTM that's a bogus case.
I'm not sure whether we'll ever use the zero-worker case in
production, but I've certainly found it useful for
performance-testing.
> Also, sinc
On 7 January 2015 at 13:11, Robert Haas wrote:
> On Tue, Jan 6, 2015 at 4:37 PM, Simon Riggs wrote:
>> So when you say "Only the top frame of the transaction state stack is
>> copied" you don't mean the top, you mean the bottom (the latest
>> subxact)? Which then becomes the top in the parallel w
On Tue, Jan 6, 2015 at 4:37 PM, Simon Riggs wrote:
> So when you say "Only the top frame of the transaction state stack is
> copied" you don't mean the top, you mean the bottom (the latest
> subxact)? Which then becomes the top in the parallel worker? OK...
The item most recently added to the sta
On 6 January 2015 at 21:37, Simon Riggs wrote:
> I get it now and agree
Yes, very much.
Should we copy both the top-level frame and the current subxid?
Hot Standby links subxids directly to the top-level, so this would be similar.
If we copied both, we wouldn't need to special case the Get fun
On 1/6/15, 10:33 AM, Robert Haas wrote:
>Entrypints?
Already noted by Andres; fixed in the attached version.
Perhaps we only parallelize while drinking beer... ;)
CreateParallelContext(): Does it actually make sense to have nworkers=0? ISTM
that's a bogus case. Also, since the number of work
On 6 January 2015 at 21:01, Robert Haas wrote:
> On Tue, Jan 6, 2015 at 3:04 PM, Simon Riggs wrote:
>> If you can explain it in more detail in comments and README, I may
>> agree. At present, I don't get it and it makes me nervous.
>>
>> The comment says
>> "Only the top frame of the transaction
On Tue, Jan 6, 2015 at 3:04 PM, Simon Riggs wrote:
> If you can explain it in more detail in comments and README, I may
> agree. At present, I don't get it and it makes me nervous.
>
> The comment says
> "Only the top frame of the transaction state stack is copied to a parallel
> worker"
> but I'm
On 6 January 2015 at 16:33, Robert Haas wrote:
>> These comments don’t have any explanation or justification
>
> OK, I rewrote them. Hopefully it's better now.
Thanks for new version and answers.
>> * Transaction stuff strikes me as overcomplicated and error prone.
>> Given there is no explana
On Mon, Jan 5, 2015 at 9:57 PM, Robert Haas wrote:
>
> On Sat, Jan 3, 2015 at 7:31 PM, Andres Freund
wrote:
>
> > * Doesn't the restriction in GetSerializableTransactionSnapshotInt()
> > apply for repeatable read just the same?
>
> Yeah. I'm not sure whether we really need that check at all, b
On 22 December 2014 at 19:14, Robert Haas wrote:
> Here is another new version, with lots of bugs fixed.
An initial blind review, independent of other comments already made on thread.
OK src/backend/access/heap/heapam.c
heapam.c prohibitions on update and delete look fine
OK src/backend/access
Robert Haas writes:
> On Sat, Jan 3, 2015 at 7:31 PM, Andres Freund wrote:
>> * I wonder if parallel contexts shouldn't be tracked via resowners
> That is a good question. I confess that I'm somewhat fuzzy about
> which things should be tracked via the resowner mechanism vs. which
> things shou
On Sat, Jan 3, 2015 at 7:31 PM, Andres Freund wrote:
> A couple remarks:
> * Shouldn't this provide more infrastructure to deal with the fact that
> we might get less parallel workers than we had planned for?
Maybe, but I'm not really sure what that should look like. My working
theory is that
On Fri, Jan 2, 2015 at 9:04 AM, Amit Kapila wrote:
> While working on parallel seq-scan patch to adapt this framework, I
> noticed few things and have questions regrading the same.
>
> 1.
> Currently parallel worker just attaches to error queue, for tuple queue
> do you expect it to be done in the
On 2014-12-22 14:14:31 -0500, Robert Haas wrote:
> On Fri, Dec 19, 2014 at 8:59 AM, Robert Haas wrote:
> > And here is a new version.
>
> Here is another new version, with lots of bugs fixed.
A couple remarks:
* Shouldn't this provide more infrastructure to deal with the fact that
we might get
On Thu, Dec 18, 2014 at 1:23 AM, Robert Haas wrote:
>
>
> In the meantime, I had a good chat with Heikki on IM yesterday which
> gave me some new ideas on how to fix up the transaction handling in
> here, which I am working on implementing. So hopefully I will have
> that by then.
>
> I am also h
On Thu, Dec 18, 2014 at 4:53 AM, Robert Haas wrote:
> On Wed, Dec 17, 2014 at 9:16 AM, Simon Riggs wrote:
>> On 12 December 2014 at 22:52, Robert Haas wrote:
>>
>>> I would be remiss if I failed to mention that this patch includes work
>>> by my colleagues Amit Kapila, Rushabh Lathia, and Jeevan
On Wed, Dec 17, 2014 at 2:53 PM, Robert Haas wrote:
> In the meantime, I had a good chat with Heikki on IM yesterday which
> gave me some new ideas on how to fix up the transaction handling in
> here, which I am working on implementing. So hopefully I will have
> that by then.
And here is a new
On Wed, Dec 17, 2014 at 9:16 AM, Simon Riggs wrote:
> On 12 December 2014 at 22:52, Robert Haas wrote:
>
>> I would be remiss if I failed to mention that this patch includes work
>> by my colleagues Amit Kapila, Rushabh Lathia, and Jeevan Chalke, as
>> well as my former colleague Noah Misch; and
On 12 December 2014 at 22:52, Robert Haas wrote:
> I would be remiss if I failed to mention that this patch includes work
> by my colleagues Amit Kapila, Rushabh Lathia, and Jeevan Chalke, as
> well as my former colleague Noah Misch; and that it would not have
> been possible without the patient
Attached is a patch that adds two new concepts: parallel mode, and
parallel contexts. The idea of this code is to provide a framework
for specific parallel things that you want to do, such as parallel
sequential scan or parallel sort. When you're in parallel mode,
certain operations - like DDL, a
66 matches
Mail list logo
