Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread roger peppe
On 4 July 2014 15:17, Gustavo Niemeyer gust...@niemeyer.net wrote: On Fri, Jul 4, 2014 at 10:32 AM, roger peppe roger.pe...@canonical.com wrote: It won't be possible to shard the transaction log. Why not? I had assumed that because every client needs to see every transaction there would

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread Gustavo Niemeyer
On Mon, Jul 7, 2014 at 10:09 AM, roger peppe roger.pe...@canonical.com wrote: I had assumed that because every client needs to see every transaction there would likely be no benefit to sharding the log, although technically you could shard on transaction id. I'd be Clients don't need to see

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread roger peppe
On 7 July 2014 14:27, Gustavo Niemeyer gust...@niemeyer.net wrote: On Mon, Jul 7, 2014 at 10:09 AM, roger peppe roger.pe...@canonical.com wrote: I had assumed that because every client needs to see every transaction there would likely be no benefit to sharding the log, although technically

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread roger peppe
On 7 July 2014 16:59, Gustavo Niemeyer gust...@niemeyer.net wrote: On Mon, Jul 7, 2014 at 12:26 PM, roger peppe roger.pe...@canonical.com wrote: On 7 July 2014 14:27, Gustavo Niemeyer gust...@niemeyer.net wrote: On Mon, Jul 7, 2014 at 10:09 AM, roger peppe roger.pe...@canonical.com wrote:

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread Gustavo Niemeyer
On Mon, Jul 7, 2014 at 2:03 PM, roger peppe roger.pe...@canonical.com wrote: The latter might turn out to be quite awkward, though there's probably a nice solution I don't see. Suppose we've got three environments, A, B and C. We have transactions that span {A, B}, {B, C} and {C, A}. How

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-07 Thread Ian Booth
On 04/07/14 23:56, Gustavo Niemeyer wrote: On Thu, Jul 3, 2014 at 10:01 PM, Tim Penhey tim.pen...@canonical.com wrote: As far as I know (and I may be wrong), if you are adding a document to the mongo collection, and you do not specify an _id field, mongo will create a unique value for you.

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread roger peppe
On 4 July 2014 02:01, Tim Penhey tim.pen...@canonical.com wrote: Hi folks, Very shortly we are going to start on the work to be able to store multiple environments within a single mongo database. Most of our current entities are stored in the database with their name or id fields serialized

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread William Reade
My expectation is that: 1) We certainly need the environment UUID as a separate field for the shard key. 2) We *also* need the environment UUID as an _id prefix to keep our watchers sane. 2a) If we had separate collections per environment, we wouldn't; but AIUI, scaling mongo by adding

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Meinel
I would think that if we have to put environ-uuid into the _id field, then we wouldn't need yet-another field to shard on (at least if we put it at the beginning of the field). John =:- On Fri, Jul 4, 2014 at 2:24 PM, William Reade william.re...@canonical.com wrote: My expectation is that:

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread roger peppe
On 4 July 2014 11:24, William Reade william.re...@canonical.com wrote: My expectation is that: 1) We certainly need the environment UUID as a separate field for the shard key. 2) We *also* need the environment UUID as an _id prefix to keep our watchers sane. 2a) If we had separate

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Meinel
... It can be a great advantage when scaling to be able to partition the transactions across different parts of the database. If we want this to be able to scale, I think we *have* to make it work without requiring transactions across environments. There is no way that we can scale as far

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread John Weldon
On Fri, Jul 4, 2014 at 6:56 AM, Gustavo Niemeyer gust...@niemeyer.net wrote: 1. change the _id field to be a composed field where it is the concatenation of the environment id and the existing id or name field. If we do take this approach, I strongly recommend having the fields that make

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread Gustavo Niemeyer
On Fri, Jul 4, 2014 at 6:01 AM, roger peppe roger.pe...@canonical.com wrote: There is another possiblity: we could just use a different collection name prefix for each environment. There is no hard limit on the number of collections in mongo (see

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-04 Thread Gustavo Niemeyer
On Fri, Jul 4, 2014 at 10:32 AM, roger peppe roger.pe...@canonical.com wrote: It won't be possible to shard the transaction log. Why not? The thing I'm trying to get across is: until we know one way or another, I believe it would be better to choose the (much) simpler option and use the

RFC: mongo _id fields in the multi-environment juju server world

2014-07-03 Thread Tim Penhey
Hi folks, Very shortly we are going to start on the work to be able to store multiple environments within a single mongo database. Most of our current entities are stored in the database with their name or id fields serialized to bson as the _id field. As far as I know (and I may be wrong), if

Re: RFC: mongo _id fields in the multi-environment juju server world

2014-07-03 Thread John Meinel
According to the mongo docs: http://docs.mongodb.org/manual/core/document/#record-documents The field name _id is reserved for use as a primary key; its value must be unique in the collection, is immutable, and may be of any type other than an array. That makes it sound like we *could* use an