Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread roger peppe
On 4 July 2014 02:01, Tim Penhey tim.pen...@canonical.com wrote: Hi folks, Very shortly we are going to start on the work to be able to store multiple environments within a single mongo database. Most of our current entities are stored in the database with their name or id fields serialized

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread William Reade
My expectation is that: 1) We certainly need the environment UUID as a separate field for the shard key. 2) We *also* need the environment UUID as an _id prefix to keep our watchers sane. 2a) If we had separate collections per environment, we wouldn't; but AIUI, scaling mongo by adding

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Meinel
I would think that if we have to put environ-uuid into the _id field, then we wouldn't need yet-another field to shard on (at least if we put it at the beginning of the field). John =:- On Fri, Jul 4, 2014 at 2:24 PM, William Reade william.re...@canonical.com wrote: My expectation is that:

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread roger peppe
On 4 July 2014 11:24, William Reade william.re...@canonical.com wrote: My expectation is that: 1) We certainly need the environment UUID as a separate field for the shard key. 2) We *also* need the environment UUID as an _id prefix to keep our watchers sane. 2a) If we had separate

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Meinel
... It can be a great advantage when scaling to be able to partition the transactions across different parts of the database. If we want this to be able to scale, I think we *have* to make it work without requiring transactions across environments. There is no way that we can scale as far

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Weldon
On Fri, Jul 4, 2014 at 6:56 AM, Gustavo Niemeyer gust...@niemeyer.net wrote: 1. change the _id field to be a composed field where it is the concatenation of the environment id and the existing id or name field. If we do take this approach, I strongly recommend having the fields that make

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread Gustavo Niemeyer
On Fri, Jul 4, 2014 at 6:01 AM, roger peppe roger.pe...@canonical.com wrote: There is another possiblity: we could just use a different collection name prefix for each environment. There is no hard limit on the number of collections in mongo (see

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread Gustavo Niemeyer
On Fri, Jul 4, 2014 at 10:32 AM, roger peppe roger.pe...@canonical.com wrote: It won't be possible to shard the transaction log. Why not? The thing I'm trying to get across is: until we know one way or another, I believe it would be better to choose the (much) simpler option and use the