On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
There's one snag I currently can see, namely that we actually need to
prevent that a formerly dropped relfilenode is getting reused. Not
entirely sure what the best way for that is.
I'm not sure in detail, but it seems to me that this all
On Tue, Oct 29, 2013 at 10:47 AM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
There's one snag I currently can see, namely that we actually need to
prevent that a formerly dropped relfilenode is getting reused. Not
entirely sure what the best
On 2013-10-29 11:28:44 -0400, Robert Haas wrote:
On Tue, Oct 29, 2013 at 10:47 AM, Andres Freund and...@2ndquadrant.com
wrote:
On 2013-10-28 11:54:31 -0400, Robert Haas wrote:
There's one snag I currently can see, namely that we actually need to
prevent that a formerly dropped
On Tue, Oct 29, 2013 at 11:43 AM, Andres Freund and...@2ndquadrant.com wrote:
I think modifying GetNewRelFileNode() is attacking the problem from
the wrong end. The point is that when a table is dropped, that fact
can be communicated to the same machine machinery that's been tracking
the
On Fri, Oct 25, 2013 at 7:57 AM, Andres Freund and...@2ndquadrant.com wrote:
However, I'm leery about the idea of using a relation fork for this.
I'm not sure whether that's what you had it mind, but it gives me the
willies. First, it adds distributed overhead to the system, as
previously
On Fri, Oct 25, 2013 at 8:14 AM, Andres Freund and...@2ndquadrant.com wrote:
So, I thought about this for some more and I think I've a partial
solution to the problem.
The worst thing about deadlocks that occur in the above is that they
could be the VACUUM FULL waiting for the restart LSN[1]
On 2013-10-28 12:04:01 -0400, Robert Haas wrote:
On Fri, Oct 25, 2013 at 8:14 AM, Andres Freund and...@2ndquadrant.com wrote:
I wonder if this is isn't maybe sufficient. Yes, it can deadlock, but
that's already the case for VACUUM FULLs of system tables, although less
likely. And it will
On Mon, Oct 28, 2013 at 12:17 PM, Andres Freund and...@2ndquadrant.com wrote:
In general, I don't think waiting on an XID is sufficient because a
process can acquire a heavyweight lock without having an XID. Perhaps
use the VXID instead?
But decoding doesn't care about transactions that
On 2013-10-24 10:59:21 -0400, Robert Haas wrote:
On Tue, Oct 22, 2013 at 2:13 PM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund and...@2ndquadrant.com
wrote:
That strikes me as a flaw in the
On 2013-10-21 16:15:58 +0200, Andres Freund wrote:
I don't think I understand exactly what you have in mind for (2); can
you elaborate? I have always thought that having a
WaitForDecodingToCatchUp() primitive was a good way of handling
changes that were otherwise too difficult to track
On Tue, Oct 22, 2013 at 2:13 PM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund and...@2ndquadrant.com
wrote:
That strikes me as a flaw in the implementation rather than the idea.
You're
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund and...@2ndquadrant.com wrote:
So. As it turns out that solution isn't sufficient in the face of VACUUM
FULL and mixed DML/DDL transaction that have not yet been decoded.
To reiterate, as published it works like:
For every modification of catalog
On 2013-10-22 10:52:48 -0400, Robert Haas wrote:
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund and...@2ndquadrant.com wrote:
So. As it turns out that solution isn't sufficient in the face of VACUUM
FULL and mixed DML/DDL transaction that have not yet been decoded.
To reiterate, as
On Tue, Oct 22, 2013 at 11:02 AM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-22 10:52:48 -0400, Robert Haas wrote:
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund and...@2ndquadrant.com
wrote:
So. As it turns out that solution isn't sufficient in the face of VACUUM
FULL and
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of delivering the basis of
what you need to do parallel sort in a transaction that previously has
performed DDL.
Currently you cannot do anything in
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of delivering the basis of
what you need to do parallel sort in a transaction that previously has
performed
On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of delivering the basis of
what you need to do
On 22.10.2013 19:23, Andres Freund wrote:
On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
BTW: That would have the nice side-effect of
On Tue, Oct 22, 2013 at 12:25 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
Or just hand over a copy of the combocid map to the worker, along with the
snapshot. Seems a lot simpler than this wide cids business..
Yes, that's what Noah and I talked about doing. Or possibly even
making
On 2013-10-22 19:25:31 +0300, Heikki Linnakangas wrote:
On 22.10.2013 19:23, Andres Freund wrote:
On 2013-10-22 19:19:19 +0300, Heikki Linnakangas wrote:
On 22.10.2013 19:12, Andres Freund wrote:
On 2013-10-18 20:26:16 +0200, Andres Freund wrote:
4) Store both (cmin, cmax) for catalog tuples.
On 2013-10-22 11:59:35 -0400, Robert Haas wrote:
So I have a new idea for handling this problem, which seems obvious in
retrospect. What if we make the VACUUM FULL or CLUSTER log the old
CTID - new CTID mappings? This would only need to be done for
catalog tables, and maybe could be
On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund and...@2ndquadrant.com wrote:
It seems to me that you have to think of the CTID map as tied to a
relfilenode; if you try to use one relfilenode's map with a different
relfilenode, it's obviously not going to work. So don't do that.
It has to be
On 2013-10-22 13:57:53 -0400, Robert Haas wrote:
On Tue, Oct 22, 2013 at 1:08 PM, Andres Freund and...@2ndquadrant.com wrote:
That strikes me as a flaw in the implementation rather than the idea.
You're presupposing a patch where the necessary information is
available in WAL yet you don't
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund and...@2ndquadrant.com wrote:
I know of the following solutions:
1) Don't allow VACUUM FULL on catalog tables if wal_level = logical.
2) Make VACUUM FULL prevent DDL and then wait till all changestreams
have decoded up to the current point.
3)
On 2013-10-21 09:32:12 -0400, Robert Haas wrote:
On Fri, Oct 18, 2013 at 2:26 PM, Andres Freund and...@2ndquadrant.com wrote:
I know of the following solutions:
1) Don't allow VACUUM FULL on catalog tables if wal_level = logical.
2) Make VACUUM FULL prevent DDL and then wait till all
On Mon, Oct 21, 2013 at 1:52 PM, Andres Freund and...@2ndquadrant.com wrote:
In my opinion, (4) is too ugly to consider. I think that if we start
playing games like this, we're opening up the doors to lots of subtle
bugs and future architectural pain that will be with us for many, many
On 2013-10-21 14:22:17 -0400, Robert Haas wrote:
On Mon, Oct 21, 2013 at 1:52 PM, Andres Freund and...@2ndquadrant.com wrote:
In my opinion, (4) is too ugly to consider. I think that if we start
playing games like this, we're opening up the doors to lots of subtle
bugs and future
On Mon, Oct 21, 2013 at 2:27 PM, Andres Freund and...@2ndquadrant.com wrote:
I think it's a complete no-go. Consider, e.g., the comment for
MaxTupleAttributeNumber, which you've blithely falsified. Even if you
update the comment and the value, I'm not inspired by the idea of
subtracting 32
On 2013-10-21 14:50:54 -0400, Robert Haas wrote:
On Mon, Oct 21, 2013 at 2:27 PM, Andres Freund and...@2ndquadrant.com wrote:
I think it's a complete no-go. Consider, e.g., the comment for
MaxTupleAttributeNumber, which you've blithely falsified. Even if you
update the comment and the
Hi,
On 2013-10-21 21:36:02 +0200, Andres Freund wrote:
I think this approach would have lower maintenance overhead in
comparison to the previous solution of Handling CommandIds because it's
actually much simpler.
Btw, I think the new approach would allow for *easier* modifications
about
On 2013-10-14 09:36:03 -0400, Robert Haas wrote:
I thought and implemented that in the beginning. Unfortunately it's not
enough :(. That's probably the issue that took me longest to understand
in this patchseries...
Combocids can only fix the case where a transaction actually has create
On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund and...@2ndquadrant.com wrote:
Well, I just think relying on specific symbol names in the .so file is
kind of unfortunate. It means that, for example, you can't have
multiple output plugins provided by a single .so. And in general I
think it's
On Mon, Oct 14, 2013 at 5:07 PM, Andres Freund and...@2ndquadrant.com wrote:
So, see the attatched benchmark skript. I've always done using a disk
bound and a memory bound (using eatmydata, preventing fsyncs) run.
* unpatched run, wal_level = hot_standby, eatmydata
* unpatched run, wal_level
On 2013-10-15 08:42:20 -0400, Robert Haas wrote:
On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund and...@2ndquadrant.com wrote:
Well, I just think relying on specific symbol names in the .so file is
kind of unfortunate. It means that, for example, you can't have
multiple output plugins
On 2013-10-15 08:49:26 -0400, Robert Haas wrote:
On Mon, Oct 14, 2013 at 5:07 PM, Andres Freund and...@2ndquadrant.com wrote:
So, see the attatched benchmark skript. I've always done using a disk
bound and a memory bound (using eatmydata, preventing fsyncs) run.
* unpatched run, wal_level
On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
If we go for CSV I think we should put the entire primary key as one
column (containing all the columns) and the entire row another.
What about columns like:
* action B|I|U|D|C
* xid
* timestamp
* tablename
* key name
* key column names
*
On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund and...@2ndquadrant.com wrote:
It allows you to use the shared libary both as a normal extension loaded
via shared_preload_library or adhoc and as an output plugin which seems
like a sensible goal.
We could have a single _PG_init_output_plugin()
On 10/15/2013 01:42 PM, Robert Haas wrote:
On Mon, Oct 14, 2013 at 9:51 AM, Andres Freund and...@2ndquadrant.com wrote:
Well, I just think relying on specific symbol names in the .so file is
kind of unfortunate. It means that, for example, you can't have
multiple output plugins provided by a
On 10/15/2013 02:47 PM, Andres Freund wrote:
On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
If we go for CSV I think we should put the entire primary key as one
column (containing all the columns) and the entire row another.
just use JSON :)
What about columns like:
* action B|I|U|D|C
*
On Tue, Oct 15, 2013 at 9:47 AM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
If we go for CSV I think we should put the entire primary key as one
column (containing all the columns) and the entire row another.
What about columns like:
*
On Tue, Oct 15, 2013 at 10:09 AM, Hannu Krosing ha...@krosing.net wrote:
I don't see how you can fail to know what all is.
We instinctively know what all is - as in the famous case of buddhist
ordering a
hamburger - Make me All wit Everything :) - but the requirements of
different
On 2013-10-15 10:20:55 -0400, Robert Haas wrote:
For multi-master / conflict resolution you may also want all old
values to make sure that they have not changed on target.
The patch as proposed doesn't make that information available. If you
want that to be an option, now would be the
On 2013-10-15 10:09:05 -0400, Robert Haas wrote:
On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund and...@2ndquadrant.com wrote:
It allows you to use the shared libary both as a normal extension loaded
via shared_preload_library or adhoc and as an output plugin which seems
like a sensible
On Tue, Oct 15, 2013 at 10:27 AM, Andres Freund and...@2ndquadrant.com wrote:
I think part of the problem may be that you're using the library name
to identify the output plugin. I'm not excited about that design.
For functions, you give the function a name and that is a pointer to
where to
On 2013-10-15 10:15:14 -0400, Robert Haas wrote:
On Tue, Oct 15, 2013 at 9:47 AM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-15 15:17:58 +0200, Andres Freund wrote:
If we go for CSV I think we should put the entire primary key as one
column (containing all the columns) and the
On 2013-10-15 10:34:53 -0400, Robert Haas wrote:
On Tue, Oct 15, 2013 at 10:27 AM, Andres Freund and...@2ndquadrant.com
wrote:
I think part of the problem may be that you're using the library name
to identify the output plugin. I'm not excited about that design.
For functions, you give
On Tue, Oct 15, 2013 at 10:56 AM, Andres Freund and...@2ndquadrant.com wrote:
That means you allow trivial remote code execution since you could try
to load system() or something else that's available in every shared
object. Now you can argue that that's OK since we have special checks
for
On Tue, Oct 15, 2013 at 11:02:39AM -0400, Robert Haas wrote:
goals may be in conflict; we'll have to pick something.
Note that parsing COPYs is a major PITA from most languages...
Perhaps we should make the default output json instead? With every
action terminated by a nullbyte?
On 2013-10-15 11:02:39 -0400, Robert Haas wrote:
If the plugin interface isn't rich enough to provide a convenient way
to avoid that, then it needs to be fixed so that it is, because it
will be a common requirement.
Oh, it surely is possibly to avoid repeating it. The output plugin
On Tue, Oct 15, 2013 at 10:48 AM, Andres Freund and...@2ndquadrant.com wrote:
What about columns like:
* action B|I|U|D|C
BEGIN and COMMIT?
That's B and C, yes. You'd rather not have them? When would you replay
the commit without an explicit message telling you to?
No, BEGIN and COMMIT
On 10/15/2013 07:56 AM, Andres Freund wrote:
Well, just providing the C API + an example in a first step didn't work
out too badly for FDWs. I am pretty sure that once released there will
soon be extensions for it on PGXN or whatever for special usecases.
I suspect so, too. But I also
On Tue, Oct 15, 2013 at 10:09:05AM -0400, Robert Haas wrote:
On Tue, Oct 15, 2013 at 9:17 AM, Andres Freund and...@2ndquadrant.com wrote:
User: So, what's new in PostgreSQL 9.4?
Hacker: Well, now we have logical replication!
User: Why is that cool?
Hacker: Well, streaming replication is
On Tue, Oct 15, 2013 at 7:09 AM, Robert Haas robertmh...@gmail.com wrote:
Let's try that again.
User: Wow, that sounds great. How do I use it?
Hacker: Well, currently, the output gets dumped as a series of text
files that are designed to be parsed using a scripting language. We
have sample
On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund and...@2ndquadrant.com wrote:
I don't see any need for SQL syntax. I was just thinking that the
_PG_init function could fill in a structure and then call
RegisterLogicalReplicationOutputPlugin(mystruct).
Hm. We can do that, but what'd be the
On 2013-10-14 09:36:03 -0400, Robert Haas wrote:
On Fri, Oct 11, 2013 at 12:57 PM, Andres Freund and...@2ndquadrant.com
wrote:
I don't see any need for SQL syntax. I was just thinking that the
_PG_init function could fill in a structure and then call
On 2013-10-14 15:51:14 +0200, Andres Freund wrote:
It'd probably not hurt to redo those benchmarks to make sure...
Yes, I think it would be good to characterize it more precisely than
a bit, so people know what to expect.
A bit was below the 3% range for loops of adding columns.
On Thu, Oct 10, 2013 at 7:11 PM, Andres Freund and...@2ndquadrant.com wrote:
Hi Robert,
On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
I spent some time looking at the sample plugin (patch 9/12). Here are
some review comments:
- I think that the decoding plugin interface should work more
Hi,
On 2013-10-11 09:08:43 -0400, Robert Haas wrote:
On Thu, Oct 10, 2013 at 7:11 PM, Andres Freund and...@2ndquadrant.com wrote:
On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
I spent some time looking at the sample plugin (patch 9/12). Here are
some review comments:
- I think that
Hi Robert,
On 2013-10-09 14:49:46 -0400, Robert Haas wrote:
I spent some time looking at the sample plugin (patch 9/12). Here are
some review comments:
- I think that the decoding plugin interface should work more like the
foreign data wrapper interface. Instead of using pg_dlsym to look
On Mon, Sep 30, 2013 at 6:44 PM, Andres Freund and...@2ndquadrant.com wrote:
The series from friday was a bit too buggy - obviously I was too
tired. So here's a new one:
* fix pg_recvlogical makefile (Thanks Steve)
* fix two commits not compiling properly without later changes (Thanks Kevin)
On 10/03/2013 04:00 PM, Andres Freund wrote:
Ok, there were a couple of bugs because I thought mxacts wouldn't need
to be supported. So far your testcase doesn't crash the database
anymore - it spews some internal errors though, so I am not sure if
it's entirely fixed for you. Thanks for
On 2013-10-07 09:56:11 -0400, Steve Singer wrote:
On 10/03/2013 04:00 PM, Andres Freund wrote:
Ok, there were a couple of bugs because I thought mxacts wouldn't need to
be supported. So far your testcase doesn't crash the database anymore - it
spews some internal errors though, so I am not
On 2013-10-01 16:11:47 -0400, Steve Singer wrote:
On 09/30/2013 06:44 PM, Andres Freund wrote:
Hi,
The series from friday was a bit too buggy - obviously I was too
tired. So here's a new one:
With this series I've also noticed
#2 0x007741a7 in ExceptionalCondition (
On 10/03/2013 12:38 PM, Andres Freund wrote:
Does your code use SELECT FOR UPDATE/SHARE on system or
treat_as_catalog tables? Greetings, Andres Freund
Yes.
It declares sl_table and sl_sequence and sl_set as catalog.
It does a
SELECT ..
from @NAMESPACE@.sl_table T, @NAMESPACE@.sl_set
On 2013-10-03 13:03:07 -0400, Steve Singer wrote:
On 10/03/2013 12:38 PM, Andres Freund wrote:
Does your code use SELECT FOR UPDATE/SHARE on system or treat_as_catalog
tables? Greetings, Andres Freund
Yes.
It declares sl_table and sl_sequence and sl_set as catalog.
It does a
SELECT
On 09/30/2013 06:44 PM, Andres Freund wrote:
Hi,
The series from friday was a bit too buggy - obviously I was too
tired. So here's a new one:
With this series I've also noticed
#2 0x007741a7 in ExceptionalCondition (
conditionName=conditionName@entry=0x7c2908 !(!(tuple-t_infomask
66 matches
Mail list logo
