Re: [Openstack] [metering] resources metadata
On Wed, May 16 2012, Loic Dachary wrote: It makes sense and I updated the wiki accordingly: http://wiki.openstack.org/EfficientMetering?action=diffrev2=81rev1=80 What do you think ? I think we can remove the payload field, since it's stored in resource_metadata. -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
On Mon, May 14 2012, Loic Dachary wrote: Each set of metering data will need to be associated with the appropriate metadata from the resource at the time the metering information was collected. The rate of change of metadata and metering events are different, though, so the timestamps of the metadata records are unlikely to match exactly with the values in the metering records. Depending on the clock resolution, it would be possible to have metadata changes and meter data with the same timestamp, resulting in an incorrect association. Indeed, good point. We can work around that by maintaining proper foreign key references using the metadata version field as you describe in the schema above (so the resource id and metadata version value point to the correct metadata record). It will make recording the metering data less efficient because we will need to determine the current version for the resource metadata, but we can optimize that eventually through indexes and caching. Aggregation will also need to take the metadata version into account, so everywhere in the list of queries we say by resource_id we need to change that to by resource_id and version. I added the idea of a format version for when the payload format changes and tried to write down a description of the metadata storage matching this thread in the wiki. http://wiki.openstack.org/EfficientMetering?action=diffrev2=80rev1=78 What do you think ? I'm jumping in a bit late in the discussion, but there may be a point I miss in the current definition because, I think it's getting too complicated. We now have 2 payload fields: one for meter and one for metadata. For example, if you look at the c1 counter (instance) you need to store the type as payload of the meter. This is a metadata of the instance, but it's not currently defined as being stored in metadata, but in the payload field of the meter. Moreover, I'm rather sure there will soon be a counter with the need of 2 different payload information, and we'll have a problem since we can only store one in the current meter schema, so we'll store the second one as a metadata or something. So clearly the initial payload solution is not enough. OTOH I find the metadata proposal in another table too much complicated. Why not storing what metadata in the meter.payload field in the same table (e.g. as a JSON string)? I miss the point of the introduction of a dedicated metadata table with version string. It sounds to me like early optimization, which is the root of all evil. :) But I might miss something. -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
On 05/15/2012 12:05 PM, Julien Danjou wrote: OTOH I find the metadata proposal in another table too much complicated. Why not storing what metadata in the meter.payload field in the same table (e.g. as a JSON string)? I would be much simpler to store the metadata in the resource_id field which could be renamed into resource field. Instead of resource_id=134123 we could have resource={ 'id': 134123, 'name': 'foobar', 'flavor': 'm1.small' etc.. } There would be no need for versioning, format, separate table, etc. etc. The only convention would be that it's a hash with at least one field : the id of the resource. The rest is metadata. It will use a lot of disk space with highly redundant information. Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
On Tue, May 15 2012, Loic Dachary wrote: On 05/15/2012 12:05 PM, Julien Danjou wrote: OTOH I find the metadata proposal in another table too much complicated. Why not storing what metadata in the meter.payload field in the same table (e.g. as a JSON string)? I would be much simpler to store the metadata in the resource_id field which could be renamed into resource field. That'd be even more radical. Instead of resource_id=134123 we could have resource={ 'id': 134123, 'name': 'foobar', 'flavor': 'm1.small' etc.. } There would be no need for versioning, format, separate table, etc. etc. The only convention would be that it's a hash with at least one field : the id of the resource. The rest is metadata. It will use a lot of disk space with highly redundant information. Ok, so the current proposal is just early optimization, as I understood. If you want to optimize the storage, why not use resource_id as a foreign key to the metatable table which would contains unique records of metadata? That would allow to store identical metadata once (and therefore optimize space) and will be much simpler. There would not be any need of version, timestamp, or whatever on metadata. -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
Copying the list... On Tue, May 15, 2012 at 10:26 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Tue, May 15, 2012 at 8:21 AM, Julien Danjou julien.dan...@enovance.com wrote: On Tue, May 15 2012, Loic Dachary wrote: On 05/15/2012 12:05 PM, Julien Danjou wrote: OTOH I find the metadata proposal in another table too much complicated. Why not storing what metadata in the meter.payload field in the same table (e.g. as a JSON string)? I would be much simpler to store the metadata in the resource_id field which could be renamed into resource field. That'd be even more radical. I like it because it would simplify the messaging. We can leave the storage optimization question to the daemon that stores the data. Instead of resource_id=134123 we could have resource={ 'id': 134123, 'name': 'foobar', 'flavor': 'm1.small' etc.. } There would be no need for versioning, format, separate table, etc. etc. The only convention would be that it's a hash with at least one field : the id of the resource. The rest is metadata. It will use a lot of disk space with highly redundant information. Ok, so the current proposal is just early optimization, as I understood. If you want to optimize the storage, why not use resource_id as a foreign key to the metatable table which would contains unique records of metadata? That would allow to store identical metadata once (and therefore optimize space) and will be much simpler. There would not be any need of version, timestamp, or whatever on metadata. -- Julien Danjou // eNovance http://enovance.com // ✉ julien.dan...@enovance.com ☎ +33 1 49 70 99 81 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata (was: public API design)
On Fri, May 11, 2012 at 3:55 PM, Loic Dachary l...@enovance.com wrote: - The interesting metadata for a resource may depend on the type of resource. Do we need separate tables for that or can we normalize somehow? - How do we map a resource to the correct version of its metadata at any given time? Timestamps seem brittle. - Do we need to reflect the metadata in the aggregation API? Hi, I started a new thread for the metadata topic. I suspect it deserves it. Although I was reluctant to acknowledge that the metadate should be stored by the metering, yesterday's meeting made me realize that it was mandatory. The compelling reason ( for me ;-) is that it would make it much more difficult to implement a billing system if the metering does not provide a simple way to extract metadata and display it in a human readable way (or meaningfull to accountants ?) . I see two separate questions : a) how to store and query metadata ? b) what is the semantic of metadata for a given resource ? My hunch is that there will never be a definitive answer to b) and that the best we can do is to provide a format and leave the semantic to the documentation of the metering system, explaining the metadata of a resource. Regarding the storage of the metadata, the metering could listen / poll events creating / updating / deleting a given resource and store a history log indexed by the resource id. Something like: { meter_type: TTT, resource_id: RRR, metadata: [{ version: , timestamp: TIME1, payload: PAYLOAD1 }, { version: , timestamp: TIME3, payload: PAYLOAD2 }] } With PPP being the resource dependant metadata that depends on the type of the resource. And the metadata array being an ordered list of the successive states of the resource over time. The VVV version accounting for changes in the format of the payload. The query would be : GET /resource/meter_type/resource_id/TIME2 and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[ I'm not sure why you think timestamp is brittle. Maybe I'm missing something. Each set of metering data will need to be associated with the appropriate metadata from the resource at the time the metering information was collected. The rate of change of metadata and metering events are different, though, so the timestamps of the metadata records are unlikely to match exactly with the values in the metering records. Depending on the clock resolution, it would be possible to have metadata changes and meter data with the same timestamp, resulting in an incorrect association. We can work around that by maintaining proper foreign key references using the metadata version field as you describe in the schema above (so the resource id and metadata version value point to the correct metadata record). It will make recording the metering data less efficient because we will need to determine the current version for the resource metadata, but we can optimize that eventually through indexes and caching. Aggregation will also need to take the metadata version into account, so everywhere in the list of queries we say by resource_id we need to change that to by resource_id and version. Doug Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
On 05/14/2012 04:15 PM, Doug Hellmann wrote: On Fri, May 11, 2012 at 3:55 PM, Loic Dachary l...@enovance.com mailto:l...@enovance.com wrote: - The interesting metadata for a resource may depend on the type of resource. Do we need separate tables for that or can we normalize somehow? - How do we map a resource to the correct version of its metadata at any given time? Timestamps seem brittle. - Do we need to reflect the metadata in the aggregation API? Hi, I started a new thread for the metadata topic. I suspect it deserves it. Although I was reluctant to acknowledge that the metadate should be stored by the metering, yesterday's meeting made me realize that it was mandatory. The compelling reason ( for me ;-) is that it would make it much more difficult to implement a billing system if the metering does not provide a simple way to extract metadata and display it in a human readable way (or meaningfull to accountants ?) . I see two separate questions : a) how to store and query metadata ? b) what is the semantic of metadata for a given resource ? My hunch is that there will never be a definitive answer to b) and that the best we can do is to provide a format and leave the semantic to the documentation of the metering system, explaining the metadata of a resource. Regarding the storage of the metadata, the metering could listen / poll events creating / updating / deleting a given resource and store a history log indexed by the resource id. Something like: { meter_type: TTT, resource_id: RRR, metadata: [{ version: , timestamp: TIME1, payload: PAYLOAD1 }, { version: , timestamp: TIME3, payload: PAYLOAD2 }] } With PPP being the resource dependant metadata that depends on the type of the resource. And the metadata array being an ordered list of the successive states of the resource over time. The VVV version accounting for changes in the format of the payload. The query would be : GET /resource/meter_type/resource_id/TIME2 and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[ I'm not sure why you think timestamp is brittle. Maybe I'm missing something. Each set of metering data will need to be associated with the appropriate metadata from the resource at the time the metering information was collected. The rate of change of metadata and metering events are different, though, so the timestamps of the metadata records are unlikely to match exactly with the values in the metering records. Depending on the clock resolution, it would be possible to have metadata changes and meter data with the same timestamp, resulting in an incorrect association. Indeed, good point. We can work around that by maintaining proper foreign key references using the metadata version field as you describe in the schema above (so the resource id and metadata version value point to the correct metadata record). It will make recording the metering data less efficient because we will need to determine the current version for the resource metadata, but we can optimize that eventually through indexes and caching. Aggregation will also need to take the metadata version into account, so everywhere in the list of queries we say by resource_id we need to change that to by resource_id and version. I added the idea of a format version for when the payload format changes and tried to write down a description of the metadata storage matching this thread in the wiki. http://wiki.openstack.org/EfficientMetering?action=diffrev2=80rev1=78 What do you think ? -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata
On Mon, May 14, 2012 at 1:04 PM, Loic Dachary l...@enovance.com wrote: On 05/14/2012 04:15 PM, Doug Hellmann wrote: On Fri, May 11, 2012 at 3:55 PM, Loic Dachary l...@enovance.com wrote: - The interesting metadata for a resource may depend on the type of resource. Do we need separate tables for that or can we normalize somehow? - How do we map a resource to the correct version of its metadata at any given time? Timestamps seem brittle. - Do we need to reflect the metadata in the aggregation API? Hi, I started a new thread for the metadata topic. I suspect it deserves it. Although I was reluctant to acknowledge that the metadate should be stored by the metering, yesterday's meeting made me realize that it was mandatory. The compelling reason ( for me ;-) is that it would make it much more difficult to implement a billing system if the metering does not provide a simple way to extract metadata and display it in a human readable way (or meaningfull to accountants ?) . I see two separate questions : a) how to store and query metadata ? b) what is the semantic of metadata for a given resource ? My hunch is that there will never be a definitive answer to b) and that the best we can do is to provide a format and leave the semantic to the documentation of the metering system, explaining the metadata of a resource. Regarding the storage of the metadata, the metering could listen / poll events creating / updating / deleting a given resource and store a history log indexed by the resource id. Something like: { meter_type: TTT, resource_id: RRR, metadata: [{ version: , timestamp: TIME1, payload: PAYLOAD1 }, { version: , timestamp: TIME3, payload: PAYLOAD2 }] } With PPP being the resource dependant metadata that depends on the type of the resource. And the metadata array being an ordered list of the successive states of the resource over time. The VVV version accounting for changes in the format of the payload. The query would be : GET /resource/meter_type/resource_id/TIME2 and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[ I'm not sure why you think timestamp is brittle. Maybe I'm missing something. Each set of metering data will need to be associated with the appropriate metadata from the resource at the time the metering information was collected. The rate of change of metadata and metering events are different, though, so the timestamps of the metadata records are unlikely to match exactly with the values in the metering records. Depending on the clock resolution, it would be possible to have metadata changes and meter data with the same timestamp, resulting in an incorrect association. Indeed, good point. Although it turns out the case I was actually worried about, resizing instances, may be supported by only some hypervisors. As a result, this is less of a concern and I could afford to have us postpone handling changing metadata until a later version of ceilometer. We still need to collect the initial data, in case the resource is deleted, but that is far less complicated and there is no sense making extra trouble for ourselves if other users of 1.0 will not need the feature, either. Does anyone else in the group have feedback on how important it is? We can work around that by maintaining proper foreign key references using the metadata version field as you describe in the schema above (so the resource id and metadata version value point to the correct metadata record). It will make recording the metering data less efficient because we will need to determine the current version for the resource metadata, but we can optimize that eventually through indexes and caching. Aggregation will also need to take the metadata version into account, so everywhere in the list of queries we say by resource_id we need to change that to by resource_id and version. I added the idea of a format version for when the payload format changes and tried to write down a description of the metadata storage matching this thread in the wiki. http://wiki.openstack.org/EfficientMetering?action=diffrev2=80rev1=78 What do you think ? That looks good. I am looking forward to getting Julien's code merged in so I can start working with it. -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] [metering] resources metadata (was: public API design)
- The interesting metadata for a resource may depend on the type of resource. Do we need separate tables for that or can we normalize somehow? - How do we map a resource to the correct version of its metadata at any given time? Timestamps seem brittle. - Do we need to reflect the metadata in the aggregation API? Hi, I started a new thread for the metadata topic. I suspect it deserves it. Although I was reluctant to acknowledge that the metadate should be stored by the metering, yesterday's meeting made me realize that it was mandatory. The compelling reason ( for me ;-) is that it would make it much more difficult to implement a billing system if the metering does not provide a simple way to extract metadata and display it in a human readable way (or meaningfull to accountants ?) . I see two separate questions : a) how to store and query metadata ? b) what is the semantic of metadata for a given resource ? My hunch is that there will never be a definitive answer to b) and that the best we can do is to provide a format and leave the semantic to the documentation of the metering system, explaining the metadata of a resource. Regarding the storage of the metadata, the metering could listen / poll events creating / updating / deleting a given resource and store a history log indexed by the resource id. Something like: { meter_type: TTT, resource_id: RRR, metadata: [{ version: , timestamp: TIME1, payload: PAYLOAD1 }, { version: , timestamp: TIME3, payload: PAYLOAD2 }] } With PPP being the resource dependant metadata that depends on the type of the resource. And the metadata array being an ordered list of the successive states of the resource over time. The VVV version accounting for changes in the format of the payload. The query would be : GET /resource/meter_type/resource_id/TIME2 and it would return PAYLOAD1 if TIME2 is in the range [TIME1,TIME3[ I'm not sure why you think timestamp is brittle. Maybe I'm missing something. Cheers -- Loïc Dachary Chief Research Officer // eNovance labs http://labs.enovance.com // ✉ l...@enovance.com ☎ +33 1 49 70 99 82 ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp