Re: [HACKERS] postgres-r

Hi, [ CC'ing to the postgres-r mailing list ] Mark Mielke wrote: > On 08/12/2009 12:04 PM, Suno Ano wrote: >> can anybody tell me, is there a roadmap with regards to >> http://www.postgres-r.org ? I'm glad you're asking. >> I would love to see it become production-ready asap. Yes, me too. Do y

Re: [HACKERS] postgres-r

On 08/12/2009 12:04 PM, Suno Ano wrote: can anybody tell me, is there a roadmap with regards to http://www.postgres-r.org ? I would love to see it become production-ready asap. Even a breakdown of what is left to do might be useful in case any of us want to pick at it. :-) Cheers, mark

[HACKERS] postgres-r

Hello, can anybody tell me, is there a roadmap with regards to http://www.postgres-r.org ? I would love to see it become production-ready asap. pgpy65IJnozC6.pgp Description: PGP signature

[HACKERS] Postgres-R (8) Architecture published

Dear hackers, during the last three months, I've compiled my thoughts and ideas around Postgtres-R into a rounded concept covering lots of aspects. It integrates findings from up-to-date research papers and it's meant to lighten up the future direction of development for Postgres-R. See www.postgr

Re: [HACKERS] Postgres-R pacth

Hi, Imre Geczy wrote: > What kind of form or method must be used to patch that it can work correctly? I finally got around writing some installation instructions: http://www.postgres-r.org/documentation/installation Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@

[HACKERS] Postgres-R pacth

Hi All, I would like to ask a help to Postgres-R... because have tried to use it with any version from CVS and normal Postgres source code but could not do patch it. What kind of form or method must be used to patch that it can work correctly? Thanks. Imre

Re: [HACKERS] postgres-R

Hi, Marcelo Martins wrote: Anyone knows a link that has some docs about how to get that setup ? Besides the README and other documentation in the source, there's admittedly not much. Check the archive of this mailing list. Also is it stable enough for production ? No. I though getting p

Re: [HACKERS] postgres-R

Marcelo Martins wrote: Anyone knows a link that has some docs about how to get that setup ? Also is it stable enough for production ? I though getting postgreSQL from CVS and compiling was not such a good idea since the CVSROOT is probably not stable, is that wrong ? since I could not find in

[HACKERS] postgres-R

Anyone knows a link that has some docs about how to get that setup ? Also is it stable enough for production ? I though getting postgreSQL from CVS and compiling was not such a good idea since the CVSROOT is probably not stable, is that wrong ? since I could not find info out there this is wh

Re: [HACKERS] Postgres-R

Hi, K, Niranjan (NSN - IN/Bangalore) wrote: > Thanks for the information. > For Step5 (starting ensemble daemon).- > I set the multicast address to both nodes (Node 1& Node 2 eth0: 224.0.0.9/4) > before starting the ensemble. And started the server application mtalk in > node 1 & node 2 and th

Re: [HACKERS] Postgres-R

Hi, leiyonghua wrote: ./configure --enable-replication make & make install You certainly also want --enable-debug and --enable-cassert, maybe also additional flags for the C compiler, like -DRMGR_DEBUG, please check the source code for these. 4. install the GCS ensemble, according the docu

Re: [HACKERS] Postgres-R

Unfortunately, I'am getting the error as below when I start the gossip. I had followed the same steps as you mentioned. REFLECT:I'm not in the list of gossip hosts, exiting (the hosts are [cluster_1|cluster_2]) cluster_1 & cluster_2 are node names & are the in /etc/hosts. Did you face this?

Re: [HACKERS] Postgres-R

hi, Assume that we have two node node 0 , 192.168.0.2 node 1 , 192.168.0.3 1. add a host entry in /etc/hosts for hostname resolving. 2. add the host list in configuration 'ensemble.conf' for gossip service: ENS_GOSSIP_HOSTS=node0:node1 3. set the envrionment variable ENS_CONFIG_FILE export ENS_CONF

Re: [HACKERS] Postgres-R

Thanks for the information. For Step5 (starting ensemble daemon).- I set the multicast address to both nodes (Node 1& Node 2 eth0: 224.0.0.9/4) before starting the ensemble. And started the server application mtalk in node 1 & node 2 and then client application in node 1 & node 2. But the count

Re: [HACKERS] Postgres-R

[EMAIL PROTECTED] 写道: I wish to set up the Postgres-R environment, could you please let me know the steps for setting it up. Thanks. yeah, actually, i have not been successful to set up this, but let me give some information for you. 1. download the postgresql snapshot source code from he

Re: [HACKERS] Postgres-R: internal messaging

Hi, That's now changed in today's snapshot of Postgres-R: the postmaster no longer uses imessages (and thus shared memory) to communicate with the replication manager. Instead the manager signals the postmaster using a newish PMSIGNAL for requesting new helper backends. It now only requests o

Re: [HACKERS] Postgres-R: internal messaging

Hi, Tom Lane wrote: I hope you're not expecting the contents of shared memory to still be trustworthy after a backend crash. Hm.. that's a good point. So I either need to bullet-proof the imessages with checksums or some such. I'm not sure that's doable reliably. Not to speak about performan

Re: [HACKERS] Postgres-R: internal messaging

Markus Wanner <[EMAIL PROTECTED]> writes: > ... crashes are more difficult. IMO the replication > manager needs to stay alive during this reinitialization, to keep the > GCS connection. However, it can easily detach from shared memory > temporarily (the imessages stuff is the only shmem place it

Re: [HACKERS] Postgres-R: internal messaging

Hi, what follows are some comments after trying to understand how the autovacuum launcher works and thoughts on how to apply this to the replication manager in Postgres-R. The initial comments in autovacuum.c say: If the fork() call fails in the postmaster, it sets a flag in the shared memo

Re: [HACKERS] Postgres-R: internal messaging

Hi, Tom Lane wrote: You should also look at the current code for communication between autovac launcher and autovac workers. That seems to be largely a similar problem, and it's been solved in a way that seems to be safe enough with respect to the postmaster vs shared memory issue. Oh yeah, t

Re: [HACKERS] Postgres-R: internal messaging

Alexey Klyukin <[EMAIL PROTECTED]> writes: > Markus Wanner wrote: >> I'm currently doing this with imessages as well, >> which violates the rule that the postmaster may not to touch shared >> memory. I didn't look into ripping that out, yet. I'm not sure it can be >> done with the existing si

Re: [HACKERS] Postgres-R: internal messaging

Hi Alexey, thanks for your feedback, these are interesting points. Alexey Klyukin wrote: In Replicator we avoided the need for postmaster to read/write backend's shmem data by using it as a signal forwarder. When a backend wants to inform a special process (i.e. queue monitor) about replication

Re: [HACKERS] Postgres-R: internal messaging

Markus Wanner wrote: > Besides the communication between the replication manager and the > backends, which is currently done by using these imessages, the > replication manager also needs to communicate with the postmaster: it > needs to be able to request new helper backends and it wants to be > n

[HACKERS] Postgres-R: internal messaging

Hi, As you certainly know by now, Postgres-R introduces an additional manager process. That one is forked from the postmaster, so are all backends, no matter if they are processing local or remote transactions. That led to a communication problem, which has originally (i.e. around Postgres-R for

Re: [HACKERS] Postgres-R: tuple serialization

Hi, Decibel! wrote: Currently, londiste triggers are per-row, not deferred. IIRC, londiste is the same. ISTM it'd be much better if we had per-statement triggers that could see what data had changed; that'd likely be a lot more efficient than doing stuff per-row. Well, now that I think about

Re: [HACKERS] Postgres-R: tuple serialization

On Jul 22, 2008, at 4:43 PM, Markus Wanner wrote: Decibel! wrote: ISTM that both londiste and Slony would be able to make use of these improvements as well. A modular replication system should be able to use a variety of methods for logging data changes and then applying them on a subscribe

Re: [HACKERS] Postgres-R: tuple serialization

Hi, Decibel! wrote: ISTM that both londiste and Slony would be able to make use of these improvements as well. A modular replication system should be able to use a variety of methods for logging data changes and then applying them on a subscriber, so long as some kind of common transport can b

Re: [HACKERS] Postgres-R: tuple serialization

On Jul 22, 2008, at 3:04 AM, Markus Wanner wrote: yesterday, I promised to outline the requirements of Postgres-R for tuple serialization, which we have been talking about before. There are basically three types of how to serialize tuple changes, depending on whether they originate from an I

Re: [HACKERS] Postgres-R: primary key patches

Dimitri Fontaine <[EMAIL PROTECTED]> writes: > Note that while slony doesn't require a dump/restore to get activated, it > seems to me (as a non user of it) that it still plays with catalog, > preventing "normal" usage of pg_dump... As of 8.3 there are some new trigger features in core that were

Re: [HACKERS] Postgres-R: primary key patches

Hi, Dimitri Fontaine wrote: This part of Markus's mail makes me think the need may change if Postgres-R is ever integrated into -core: Yes, in that case, you'd have replication already compiled in and distributed with standard Postgres. However, ATM that's pipe dreaming and I'm pretty sure n

Re: [HACKERS] Postgres-R: primary key patches

Le mardi 22 juillet 2008, Christopher Browne a écrit : > A most pointed case where that will cause heartburn of the "I refuse > to use this" sort is if that disruption needs to take place when > recovering from the failure of a node. That sort of disruption is > certainly counterproductive to the

Re: [HACKERS] Postgres-R: primary key patches

Markus Wanner wrote: >> Ideally, we'd take an outage and add the primary key. But suppose we >> can't afford to do so? > > You are assuming that one doesn't need to take an outage to start > replication in the first place. As Postgres-R comes with system catalog > changes, that's not the case

Re: [HACKERS] Postgres-R: primary key patches

Hi, Christopher Browne wrote: Markus Wanner <[EMAIL PROTECTED]> writes: Thinking about index creation time doesn't make sense, as long as we still need a dump/restore cycle to setup replication. And even then, that operational issue has n

Re: [HACKERS] Postgres-R: primary key patches

Markus Wanner <[EMAIL PROTECTED]> writes: > Thinking about index creation time doesn't make sense, as long as we > still need a dump/restore cycle to setup replication. And even then, > that operational issue has nothing to do with the question

Re: [HACKERS] Postgres-R: primary key patches

Hi, chris wrote: I'll describe a scenario to suggest where it might happen. - A system is implemented, using the database, and, for some reason, no PRIMARY KEY is defined for a table. Someone forgot; it got misconfigured; a mistake was probably made. - The system then goes into production

Re: [HACKERS] Postgres-R: primary key patches

[EMAIL PROTECTED] (Markus Wanner) writes: > chris wrote: >> I agree with you that tables are *supposed* to have primary keys; >> that's proper design, and if tables are missing them, then something >> is definitely broken. > > Ah, I see, so you are not concerned about tables with a PRIMARY KEY > fo

[HACKERS] Postgres-R: tuple serialization

Hi, yesterday, I promised to outline the requirements of Postgres-R for tuple serialization, which we have been talking about before. There are basically three types of how to serialize tuple changes, depending on whether they originate from an INSERT, UPDATE or DELETE. For updates and delete

Re: [HACKERS] Postgres-R: primary key patches

Hi, Alvaro Herrera wrote: Markus Wanner wrote: (Although, I'm still less than thrilled about the internal storage format of these tuple collections. That can certainly be improved and simplified.) Care to expand more on what it is? Well, what I really dislike is the overhead in code to f

Re: [HACKERS] Postgres-R: primary key patches

Markus Wanner wrote: > (Although, I'm still less than thrilled about the internal storage > format of these tuple collections. That can certainly be improved and > simplified.) Care to expand more on what it is? On Replicator we're using the binary send/recv routines to transmit tuples. (Ob

Re: [HACKERS] Postgres-R: primary key patches

Hi, chris wrote: You may want to have a chat with Jan; he's got some thoughts on a more general purpose mechanism that would be good for this as well as for (we think) extremely efficient bulk data loading. Jan, mind to share your thoughts? What use cases for such a general purpose mechanism

Re: [HACKERS] Postgres-R: primary key patches

Hi, chris wrote: I agree with you that tables are *supposed* to have primary keys; that's proper design, and if tables are missing them, then something is definitely broken. Ah, I see, so you are not concerned about tables with a PRIMARY KEY for which one wants another REPLICATION KEY, but on

Re: [HACKERS] Postgres-R: primary key patches

[EMAIL PROTECTED] (Markus Wanner) writes: > Hello Chris, > > chris wrote: >> Slony-I does the same, with the "variation" that it permits the option >> of using a "candidate primary key," namely an index that is unique+NOT >> NULL. >> >> If it is possible to support that broader notion, that might m

Re: [HACKERS] Postgres-R: primary key patches

Hi, Alvaro Herrera wrote: I think the point here is that you need to distinguish which tuple you need to update. For this, our Replicator uses the primary key only; there's no way to use another candidate key (unique not null). It would certainly be possible to use a different candidate key,

Re: [HACKERS] Postgres-R: primary key patches

Markus Wanner wrote: > Gregory Stark wrote: >> It would be nice if there was a way for Slony to express to the server that >> really, it only needs any UNIQUE NOT NULL combination of columns to match. >> Once the server has any such combination which matches it can skip checking >> the rest. I ca

Re: [HACKERS] Postgres-R: primary key patches

Hi, I realize that you are talk about Slony, let me answer for the Postgres-R case, anyway. Gregory Stark wrote: Hm, it occurs to me that really Slony should be saying WHERE (col1,col2,...) = ('x','y','z',...) Hm.. that would mean increasing the amount of work for the remote backend, whic

Re: [HACKERS] Postgres-R: primary key patches

Hi, Tom Lane wrote: It's the default foreign key reference column(s) for the table That's why I think it makes for a pretty good replication key as well. Regards Markus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.pos

Re: [HACKERS] Postgres-R: primary key patches

Hi, sorry, some strange key-combination made my mail client send too early... I myself wrote: I was trying to say that Postgres-R internally relies only on a unique index with not null constraint. It doesn't care if you name it PRIMARY KEY or REPLICATION KEY or whatever. So, it's just a ques

Re: [HACKERS] Postgres-R: primary key patches

Hi, David Fetter wrote: While I'm a "chicken" rather than a "pig" on this project , I believe that covering the more general case right from the start would be a much better plan. I was trying to say that Postgres-R internally relies only o

Re: [HACKERS] Postgres-R: primary key patches

"David Fetter" <[EMAIL PROTECTED]> writes: > On Fri, Jul 18, 2008 at 03:04:08PM +0200, Markus Schiltknecht wrote: >> Hello Chris, >> >> chris wrote: >>> Slony-I does the same, with the "variation" that it permits the >>> option of using a "candidate primary key," namely an index that is >>> unique

Re: [HACKERS] Postgres-R: primary key patches

David Fetter <[EMAIL PROTECTED]> writes: > On Fri, Jul 18, 2008 at 03:04:08PM +0200, Markus Schiltknecht wrote: >> But what do we have primary keys for, in the first place? > We have them because people are used to thinking in terms of a > "PRIMARY KEY," not because that concept is actually distin

Re: [HACKERS] Postgres-R: primary key patches

On Fri, Jul 18, 2008 at 03:04:08PM +0200, Markus Schiltknecht wrote: > Hello Chris, > > chris wrote: >> Slony-I does the same, with the "variation" that it permits the >> option of using a "candidate primary key," namely an index that is >> unique+NOT NULL. >> >> If it is possible to support that b

Re: [HACKERS] Postgres-R: primary key patches

Hello Chris, chris wrote: Slony-I does the same, with the "variation" that it permits the option of using a "candidate primary key," namely an index that is unique+NOT NULL. If it is possible to support that broader notion, that might make addition of these sorts of logic more widely useful.

Re: [HACKERS] Postgres-R: primary key patches

[EMAIL PROTECTED] (Markus Wanner) writes: > as you might know, Postgres-R relies on primary keys to address tuples > of a table. It cannot replicate tables without a primary key. Slony-I does the same, with the "variation" that it permits the option of using a "candidate primary key," namely an in

Re: [HACKERS] Postgres-R source code release

On Wed, Jul 16, 2008 at 09:35:28PM +0200, Markus Schiltknecht wrote: > Hi, > > David Fetter wrote: >> Would you mind if I were to make a git branch for it on >> ? > > I've set up a git-daemon with the Postgres-R patch here: > > git://postgres-r.org/repo > > Since it's a

Re: [HACKERS] Postgres-R: current state of development

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, First, thanks a lot for opening Postgres-R, I hope -core will find in your code as many good ideas and code as possible :) Le 15 juil. 08 à 18:48, Markus Wanner a écrit : A pretty general framework for helper processes is provided. I think

[HACKERS] Postgres-R: primary key patches

Hi, as you might know, Postgres-R relies on primary keys to address tuples of a table. It cannot replicate tables without a primary key. Primary keys currently aren't really used within the executor, so I had to extended and modify Postgres here and there, to get the required information. To

Re: [HACKERS] Postgres-R source code release

Hi, David Fetter wrote: Would you mind if I were to make a git branch for it on ? I've set up a git-daemon with the Postgres-R patch here: git://postgres-r.org/repo Since it's a distributed VCS, you should be able to mirror that to git.postgtresql.org somehow (i

Re: [HACKERS] Postgres-R source code release

Hi, Alvaro Herrera wrote: I think the way to go here is to have Markus open up his Monotone repo, or convince him to migrate it to Git, but I really doubt that's ever going to happen. He he... good guess ;-) However, as much as I personally like monotone and as much as I dislike git for being

[HACKERS] Postgres-R: current state of development

Hi, After having published the source code, I'd like to add some words about the current state of the project. Postgres-R is currently capable of replicating tuples via binary change sets, does proper conflict detection and prevention. It offers three different timing methods: sync, eager an

Re: [HACKERS] Postgres-R source code release

On 15/07/2008, David E. Wheeler <[EMAIL PROTECTED]> wrote: > > > Of course, I am personally happy with SVN but hey :P > > > > You can't have tried a merge in SVN if that's so :P :P > > Those of us who have been doing it for years, in CVS and in SVN, aren't too > worried about it. > Follow the san

Re: [HACKERS] Postgres-R source code release

On Jul 14, 2008, at 16:54, David Fetter wrote: Of course, I am personally happy with SVN but hey :P You can't have tried a merge in SVN if that's so :P :P Those of us who have been doing it for years, in CVS and in SVN, aren't too worried about it. Best, David -- Sent via pgsql-hackers

Re: [HACKERS] Postgres-R source code release

On Mon, Jul 14, 2008 at 04:22:47PM -0700, Joshua D. Drake wrote: > On Mon, 2008-07-14 at 14:41 -0700, David Fetter wrote: > > On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote: > > > David Fetter wrote: > > > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > > > > > >

Re: [HACKERS] Postgres-R source code release

On Mon, 2008-07-14 at 14:41 -0700, David Fetter wrote: > On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote: > > David Fetter wrote: > > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > > > > > Would you mind if I were to make a git branch for it on > > >

Re: [HACKERS] Postgres-R source code release

David Fetter wrote: > On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote: > > David Fetter wrote: > > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > > > > > Would you mind if I were to make a git branch for it on > > > ? > > > > That's

Re: [HACKERS] Postgres-R source code release

On Mon, Jul 14, 2008 at 05:35:28PM -0400, Alvaro Herrera wrote: > David Fetter wrote: > > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > > > Would you mind if I were to make a git branch for it on > > ? > > That's very likely wasted effort, since obv

Re: [HACKERS] Postgres-R source code release

David Fetter wrote: > On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > Would you mind if I were to make a git branch for it on > ? That's very likely wasted effort, since obviously Markus has got a Monotone tree somewhere ... -- Alvaro Herrera

Re: [HACKERS] Postgres-R source code release

On Mon, Jul 14, 2008 at 05:42:21PM +0200, Markus Wanner wrote: > Dear Hackers, > > it has been two years, since I've presented my work on Postgres-R for > the first time to a broader audience in Toronto. I continued maintaining > that code in my spare time, but to be honest, I didn't have much

Re: [HACKERS] Postgres-R source code release

On Mon, Jul 14, 2008 at 11:42 AM, Markus Wanner <[EMAIL PROTECTED]> wrote: > As a first step towards a community project, I've cleaned up the code and > tried to comment and document it as good as I can. You are welcome to > download the latest patch from http://www.postgres-r.org/downloads/. It >

Re: [HACKERS] Postgres-R source code release

On Mon, 2008-07-14 at 17:42 +0200, Markus Wanner wrote: > As much as I'd like to change that, I now think the best thing for the > project itself is to open it up and release the source code. I'd very > much like to get others aboard and turn Postgres-R into a real > community project again. Of c

[HACKERS] Postgres-R source code release

Dear Hackers, it has been two years, since I've presented my work on Postgres-R for the first time to a broader audience in Toronto. I continued maintaining that code in my spare time, but to be honest, I didn't have much time for it. As much as I'd like to change that, I now think the best