On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> - When replaying b-tree deletions, we currently wait out/cancel all
> running (read-only) transactions. We take the ultra-conservative
> stance because we don't know how recent the tuples being deleted are.
> If we could store a better
Simon Riggs wrote:
> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>
>> - When switching from standby mode to normal operation, we momentarily
>> hold all AccessExclusiveLocks held by prepared xacts twice, needing
>> twice the lock space. You can run out of lock space at that point,
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> - When switching from standby mode to normal operation, we momentarily
> hold all AccessExclusiveLocks held by prepared xacts twice, needing
> twice the lock space. You can run out of lock space at that point,
> causing failover to fai
Simon Riggs wrote:
> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>
>> - The assumption that b-tree vacuum records don't need conflict
>> resolution because we did that with the additional cleanup-info record
>> works ATM, but it hinges on the fact that we don't delete any tuples
>
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> - The assumption that b-tree vacuum records don't need conflict
> resolution because we did that with the additional cleanup-info record
> works ATM, but it hinges on the fact that we don't delete any tuples
> marked as killed while we
On Thu, 2009-11-19 at 10:13 +0200, Heikki Linnakangas wrote:
> At backend start, we normally take
> RowExclusiveLock on the database in postinit.c, but you had to modify
> to acquire AccessShareLock instead in standby mode.
The consensus from earlier discussion was that allowing users to grab
Row
Greg Stark wrote:
> From discussions in the bar it sounds like this was actually a false
> start however as the RecentGlobalXmin in the backend doing the split
> could be less aggressive than the RecentGlobalXmin used by some other
> backend to hit the hint bits leading to inconsistent results :(
On Fri, 2009-11-20 at 16:40 +0900, Josh Berkus wrote:
> Yes. I'm realizing that because of the highly techincal nature of the
> discussion and the language used few people other than you and Heikki
> are aware of the major issues which still need work. It would be
> helpful if someone could post
On Sun, 2009-11-15 at 17:17 -0500, Tom Lane wrote:
> Josh Berkus writes:
> > So I'm in favor of committing part of the HS code even if there are
> > known failure conditions, as long as those conditions are well-defined.
>
> If we're thinking of committing something that is known broken, I would
On Fri, Nov 20, 2009 at 7:31 AM, Heikki Linnakangas
wrote:
> Simon Riggs wrote:
>> On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
>>> I missed the original discussion of this problem, do you happen to
>>> remember the subject or url for the details?
>>
>> December 2008; hackers; you, me and
Simon,
> This post isn't really very helpful. You aren't providing the
> second part of the discussion, nor even requesting that this issue be
> fixed. I can see such comments being taken up by people with a clear
> interest in dissing HS.
OK, I'm requesting that the issue be fixed. I'm not sur
Simon Riggs wrote:
> On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
>> I missed the original discussion of this problem, do you happen to
>> remember the subject or url for the details?
>
> December 2008; hackers; you, me and Heikki.
Yep:
http://archives.postgresql.org/message-id/494b5ffe.4
On Fri, 2009-11-20 at 06:47 +, Greg Stark wrote:
> On Fri, Nov 20, 2009 at 2:58 AM, Andrew Dunstan wrote:
> > Right. The major use I was hoping for from HS was exactly to be able to run
> > long-running queries. In once case I am thinking of we have moved the
> > business intelligence uses off
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
> On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
> > - When replaying b-tree deletions, we currently wait out/cancel all
> > running (read-only) transactions. We take the ultra-conservative stance
> > because we don't know how recent the tuples
On Fri, Nov 20, 2009 at 2:58 AM, Andrew Dunstan wrote:
> Right. The major use I was hoping for from HS was exactly to be able to run
> long-running queries. In once case I am thinking of we have moved the
> business intelligence uses off the OLTP server onto a londiste replica, and
> I was really
On Fri, Nov 20, 2009 at 11:14 AM, Josh Berkus wrote:
> On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
>> - When replaying b-tree deletions, we currently wait out/cancel all
>> running (read-only) transactions. We take the ultra-conservative stance
>> because we don't know how recent the tuples be
Joshua D. Drake wrote:
> On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
>> On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
>>> - When replaying b-tree deletions, we currently wait out/cancel all
>>> running (read-only) transactions. We take the ultra-conservative stance
>>> because we don't
Simon Riggs wrote:
On Thu, 2009-11-19 at 17:15 +0900, Tatsuo Ishii wrote:
Simon Riggs wrote:
Recovery does *not* take the same locks as the original statements on
the master took. For example, the WAL record for an INSERT just makes
its changes without acquiring locks. This is OK as long as we
On Thu, 2009-11-19 at 17:15 +0900, Tatsuo Ishii wrote:
> > Simon Riggs wrote:
> > > Recovery does *not* take the same locks as the original statements on
> > > the master took. For example, the WAL record for an INSERT just makes
> > > its changes without acquiring locks. This is OK as long as we o
On Thu, 2009-11-19 at 10:13 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > Recovery does *not* take the same locks as the original statements on
> > the master took. For example, the WAL record for an INSERT just makes
> > its changes without acquiring locks. This is OK as long as we onl
Joshua D. Drake wrote:
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
- When replaying b-tree deletions, we currently wait out/cancel all
running (read-only) transactions. We take the ultra-conservative stance
because we don't kno
On Fri, 2009-11-20 at 11:14 +0900, Josh Berkus wrote:
> On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
> > - When replaying b-tree deletions, we currently wait out/cancel all
> > running (read-only) transactions. We take the ultra-conservative stance
> > because we don't know how recent the tuples
On 11/15/09 11:07 PM, Heikki Linnakangas wrote:
> - When replaying b-tree deletions, we currently wait out/cancel all
> running (read-only) transactions. We take the ultra-conservative stance
> because we don't know how recent the tuples being deleted are. If we
> could store a better estimate for
Tom Lane wrote:
> Heikki Linnakangas writes:
>> Tatsuo Ishii wrote:
>>> Sorry for confusion. My example is under normal PostgreSQL, not under
>>> HS enabled.
>
>> You get the same result in standby:
>
> AFAICT Tatsuo's example just shows that we might wish to add a check
> for read-only transact
Heikki Linnakangas writes:
> Tatsuo Ishii wrote:
>> Sorry for confusion. My example is under normal PostgreSQL, not under
>> HS enabled.
> You get the same result in standby:
AFAICT Tatsuo's example just shows that we might wish to add a check
for read-only transaction mode before parsing an INS
Tatsuo Ishii wrote:
> Sorry for confusion. My example is under normal PostgreSQL, not under
> HS enabled.
You get the same result in standby:
postgres=# begin;
BEGIN
postgres=# prepare a(int) as insert into foo values($1);
PREPARE
postgres=# SELECT * FROM pg_locks;
locktype │ database │ relati
> Simon Riggs wrote:
> > Recovery does *not* take the same locks as the original statements on
> > the master took. For example, the WAL record for an INSERT just makes
> > its changes without acquiring locks. This is OK as long as we only allow
> > read-only users to acquire AccessShareLocks. If w
Simon Riggs wrote:
> Recovery does *not* take the same locks as the original statements on
> the master took. For example, the WAL record for an INSERT just makes
> its changes without acquiring locks. This is OK as long as we only allow
> read-only users to acquire AccessShareLocks. If we allowed
On Wed, 2009-11-18 at 14:51 +0200, Heikki Linnakangas wrote:
> Tatsuo Ishii wrote:
> >>> Please correct me if I'm wrong. Parse will result in obtaining
> >>> RowExclusiveLock on the target table if it is parsing
> >>> INSERT/UPDATE/DELETE. If so, is this ok in the standby?
> >> Any attempt to take
Tatsuo Ishii wrote:
>>> Please correct me if I'm wrong. Parse will result in obtaining
>>> RowExclusiveLock on the target table if it is parsing
>>> INSERT/UPDATE/DELETE. If so, is this ok in the standby?
>> Any attempt to take RowExclusiveLock will fail.
>>
>> Any attempt to execute INSERT/UPDATE/
Robert Haas wrote:
> On Mon, Nov 16, 2009 at 11:07 AM, Kevin Grittner
> wrote:
>> Tom Lane wrote:
>>
>>> I agree with Heikki that it would be better not to commit as long
>>> as any clear showstoppers remain unresolved.
>>
>> I agree that it would be better not to commit as long as any of the
>>
> > Please correct me if I'm wrong. Parse will result in obtaining
> > RowExclusiveLock on the target table if it is parsing
> > INSERT/UPDATE/DELETE. If so, is this ok in the standby?
>
> Any attempt to take RowExclusiveLock will fail.
>
> Any attempt to execute INSERT/UPDATE/DELETE will fail.
>
On Mon, 2009-11-16 at 19:06 +0900, Tatsuo Ishii wrote:
> > > - Does Hot Standby allow to use prepared query (not prepared
> > > transaction) in standby? I mean: Parse message from frontend can be
> > > accepted by standby?
> >
> > Yes, no problem with any of those kind of facilities
>
> Pleas
On Mon, Nov 16, 2009 at 11:07 AM, Kevin Grittner
wrote:
> Tom Lane wrote:
>
>> I agree with Heikki that it would be better not to commit as long as
>> any clear showstoppers remain unresolved.
>
> I agree that it would be better not to commit as long as any of the
> following are true:
>
> (1) T
Tom Lane wrote:
> I agree with Heikki that it would be better not to commit as long as
> any clear showstoppers remain unresolved.
I agree that it would be better not to commit as long as any of the
following are true:
(1) There are any known issues which would break things for clusters
> > - Does Hot Standby allow to use prepared query (not prepared
> > transaction) in standby? I mean: Parse message from frontend can be
> > accepted by standby?
>
> Yes, no problem with any of those kind of facilities
Please correct me if I'm wrong. Parse will result in obtaining
RowExclusiv
On Mon, 2009-11-16 at 13:23 +0900, Tatsuo Ishii wrote:
> Just a question:
>
> - Does Hot Standby allow to use prepared query (not prepared
> transaction) in standby? I mean: Parse message from frontend can be
> accepted by standby?
Yes, no problem with any of those kind of facilities
> - Can
Tatsuo Ishii wrote:
> In my understanding, Parse will aquire locks on the target table. Is
> this harmless?
That's ok, you can take AccessShareLocks in a standby. All queries lock
the target table (in AccessShare mode).
--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com
--
Se
> Tatsuo Ishii wrote:
> > Just a question:
> >
> > - Does Hot Standby allow to use prepared query (not prepared
> > transaction) in standby? I mean: Parse message from frontend can be
> > accepted by standby?
>
> Yes.
In my understanding, Parse will aquire locks on the target table. Is
this
Tatsuo Ishii wrote:
> Just a question:
>
> - Does Hot Standby allow to use prepared query (not prepared
> transaction) in standby? I mean: Parse message from frontend can be
> accepted by standby?
Yes.
> - Can we create tempory tables in standby?
No, because creating a temporary table needs
Just a question:
- Does Hot Standby allow to use prepared query (not prepared
transaction) in standby? I mean: Parse message from frontend can be
accepted by standby?
- Can we create tempory tables in standby?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
> After some time thinking about the best way
"David E. Wheeler" writes:
> On Nov 15, 2009, at 2:17 PM, Tom Lane wrote:
>> I agree with Heikki that it would be better not to commit as long as
>> any clear showstoppers remain unresolved.
> If ever there were an argument for topic branches, *this is it*.
How so? They've got a perfectly good
On Nov 15, 2009, at 2:17 PM, Tom Lane wrote:
>> So I'm in favor of committing part of the HS code even if there are
>> known failure conditions, as long as those conditions are well-defined.
>
> If we're thinking of committing something that is known broken, I would
> want to have a clearly defin
On Nov 15, 2009, at 4:19 PM, Simon Riggs wrote:
On Sun, 2009-11-15 at 23:14 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
Simon Riggs wrote:
Right now, I don't know which you
consider to be the must-fix issues, hence the thr
Josh Berkus writes:
> So I'm in favor of committing part of the HS code even if there are
> known failure conditions, as long as those conditions are well-defined.
If we're thinking of committing something that is known broken, I would
want to have a clearly defined and trust-inspiring escape str
On Sun, 2009-11-15 at 23:14 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
> >> Simon Riggs wrote:
> >>> Right now, I don't know which you
> >>> consider to be the must-fix issues, hence the thread.
> >> Ok, could you tackle th
Simon Riggs wrote:
> On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> Right now, I don't know which you
>>> consider to be the must-fix issues, hence the thread.
>> Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
>> index pages after the
On 11/15/09 12:58 PM, Simon Riggs wrote:
> On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> Right now, I don't know which you
>>> consider to be the must-fix issues, hence the thread.
>> Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
>>
On Sun, 2009-11-15 at 22:45 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > Right now, I don't know which you
> > consider to be the must-fix issues, hence the thread.
>
> Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
> index pages after the last b-tree vacuum r
Simon Riggs wrote:
> Right now, I don't know which you
> consider to be the must-fix issues, hence the thread.
Ok, could you tackle the b-tree vacuum bug, where we neglect to pin the
index pages after the last b-tree vacuum record? Thanks.
--
Heikki Linnakangas
EnterpriseDB http://www.ente
On Sun, 2009-11-15 at 21:56 +0200, Heikki Linnakangas wrote:
> If you actually want to help, can you please focus on fixing the
> must-fix bugs we know about? We can then discuss which of the
> remaining known issues we're willing to live with.
I intend to work on all of the issues, so not sure w
Simon Riggs wrote:
> On Sun, 2009-11-15 at 21:20 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
>>>
The LSN doesn't help there, because when an itemid is marked as dead,
the LSN is not updated.
>>> I was thinking we
Simon Riggs wrote:
> You agree there should be two phases?
I'm hesitant to say 'yes', because then you will harass me with "but you
said that you would be OK with fixing X, Y, Z later! Why don't you
commit already!".
Of course there should be several phases! We've *already* punted a lot
of stuff
On Sun, 2009-11-15 at 21:20 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
> >
> >> The LSN doesn't help there, because when an itemid is marked as dead,
> >> the LSN is not updated.
> >
> > I was thinking we could update the
On Sun, Nov 15, 2009 at 7:29 PM, Simon Riggs wrote:
> You agree there should be two phases?
>
I don't understand this repeated suggestion of "phases". Nobody's
every suggested that we would refuse to add new features to HS after
the initial commit or the 8.5 release. Of course there should be lat
On Sun, 2009-11-15 at 20:37 +0200, Heikki Linnakangas wrote:
> Robert Haas wrote:
> > But a
> > query getting canceled because it touches a lot of tables sounds more
> > like a limitation than an outright bug,
>
> It's not that the query might get canceled. It will abort WAL recovery,
> kill all
Simon Riggs wrote:
> On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
>
>> The LSN doesn't help there, because when an itemid is marked as dead,
>> the LSN is not updated.
>
> I was thinking we could update the index block LSN without writing WAL
> using the LSN of the heap block that
On Sun, 2009-11-15 at 20:30 +0200, Heikki Linnakangas wrote:
> The LSN doesn't help there, because when an itemid is marked as dead,
> the LSN is not updated.
I was thinking we could update the index block LSN without writing WAL
using the LSN of the heap block that leads to the killed tuple.
Pre
On Sun, 2009-11-15 at 13:15 -0500, Robert Haas wrote:
> I know Simon has said that he feels that the effort
> to finish the HS and SR patches for 9/15 was somewhat of an artificial
> deadline, but ISTM that with only 3 months remaining until the close
> of the final CommitFest for this release, and
Robert Haas wrote:
> But a
> query getting canceled because it touches a lot of tables sounds more
> like a limitation than an outright bug,
It's not that the query might get canceled. It will abort WAL recovery,
kill all backends, and bring the whole standby down.
--
Heikki Linnakangas
Ent
Simon Riggs wrote:
> On Sun, 2009-11-15 at 19:36 +0200, Heikki Linnakangas wrote:
>> Simon Riggs wrote:
>>> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>>>
The assumption that b-tree vacuum records don't need conflict
resolution because we did that with the additional cle
On Sun, Nov 15, 2009 at 11:49 AM, Heikki Linnakangas
wrote:
> Agreed. Believe me, I'd like to have this committed as much as everyone
> else. But once I do that, I'm also committing myself to fix all the
> remaining issues before the release. The criteria for committing is: is
> it good enough tha
On Sun, 2009-11-15 at 19:36 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> >
> >> The assumption that b-tree vacuum records don't need conflict
> >> resolution because we did that with the additional cleanup-info record
> >>
Simon Riggs wrote:
> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>
>> The assumption that b-tree vacuum records don't need conflict
>> resolution because we did that with the additional cleanup-info record
>> works ATM, but it hinges on the fact that we don't delete any tuples
>>
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> The assumption that b-tree vacuum records don't need conflict
> resolution because we did that with the additional cleanup-info record
> works ATM, but it hinges on the fact that we don't delete any tuples
> marked as killed while we d
Robert Haas wrote:
> OK, but... there will always be things that will bring down the
> stand-by, just as there will always be things that can bring down the
> primary. A bucket of ice-water will probably do it, for example. I
> mean, it would be great to make it better, but is it so bad that we
>
On Sun, Nov 15, 2009 at 9:50 AM, Heikki Linnakangas
wrote:
> Simon Riggs wrote:
>> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>>> - If WAL recovery runs out of lock space while acquiring an
>>> AccessExclusiveLock on behalf of a transaction that ran in the master,
>>> it will FAT
On Sun, 2009-11-15 at 16:50 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> >> - If WAL recovery runs out of lock space while acquiring an
> >> AccessExclusiveLock on behalf of a transaction that ran in the master,
> >> it wil
On Sun, 2009-11-15 at 14:47 +, Greg Stark wrote:
> On Sun, Nov 15, 2009 at 2:32 PM, Simon Riggs wrote:
> >> - The "standby delay" is measured as current timestamp - timestamp of
> >> last replayed commit record. If there's little activity in the master,
> >> that can lead to surprising results
Simon Riggs wrote:
> On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
>> - If WAL recovery runs out of lock space while acquiring an
>> AccessExclusiveLock on behalf of a transaction that ran in the master,
>> it will FATAL and abort recovery, bringing down the standby. Seems like
>> it
On Sun, Nov 15, 2009 at 2:32 PM, Simon Riggs wrote:
>> - The "standby delay" is measured as current timestamp - timestamp of
>> last replayed commit record. If there's little activity in the master,
>> that can lead to surprising results. For example, imagine that
>> max_standby_delay is set to 8
On Sun, 2009-11-15 at 16:07 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > There are two remaining areas of significant thought/effort:
>
> Here's a list of other TODO items I've collected so far. Some of them
> are just improvements or nice-to-have stuff, but some are more serious:
>
Simon Riggs wrote:
> There are two remaining areas of significant thought/effort:
Here's a list of other TODO items I've collected so far. Some of them
are just improvements or nice-to-have stuff, but some are more serious:
- If WAL recovery runs out of lock space while acquiring an
AccessExclusi
On Sun, Nov 15, 2009 at 3:06 AM, Simon Riggs wrote:
> Please can we agree a way forwards?
I don't have a strong position on the technical issues, but I am very
much in favor of getting something committed, even something with
limitations, as soon as we practically can. Getting this feature into
Simon Riggs wrote:
> * Issues relating to handling of prepared transactions
> There are some delicate issues surrounding what happens at the end of
> recovery if there is a prepared transaction still holding an access
> exclusive lock.
Can you describe in more detail what problem this is again? We
Simon Riggs wrote:
> We need to wait for all current transactions to complete. (i.e. any
> backend that has (or could) take an xid or an AccessExclusiveLock before
> it commits.). Similar-ish to the wait for a CREATE INDEX CONCURRENTLY.
>
> The standby already performs this wait in the case where
On Sunday, November 15, 2009, Simon Riggs wrote:
> On Sun, 2009-11-15 at 10:00 +0100, Magnus Hagander wrote:
>
>> What does the time depend on?
>
> We need to wait for all current transactions to complete. (i.e. any
> backend that has (or could) take an xid or an AccessExclusiveLock before
> it co
On Sun, 2009-11-15 at 10:00 +0100, Magnus Hagander wrote:
> What does the time depend on?
We need to wait for all current transactions to complete. (i.e. any
backend that has (or could) take an xid or an AccessExclusiveLock before
it commits.). Similar-ish to the wait for a CREATE INDEX CONCURREN
On Sun, Nov 15, 2009 at 09:06, Simon Riggs wrote:
> * Issues relating to handling of prepared transactions
> There are some delicate issues surrounding what happens at the end of
> recovery if there is a prepared transaction still holding an access
> exclusive lock. It is straightforward to say, a
After some time thinking about the best way forward for Hot Standby, I
have some observations and proposals.
First, the project is very large. We have agreed ways to trim the patch,
yet it remains large. Trying to do everything in one lump is almost
always a bad plan, so we need to phase things.
80 matches
Mail list logo
