[openstack-dev] [gnocchi] Redis for storage backend
Hi all, I see the latest Gnocchi support using Redis as a storage backend, I am testing performance of Redis and Ceph, it seems using Redis as storage backend we can achieve more realtime metric data, gnocchi status shows there is always few metric to process. Is Redis a recommend storage backend ? -- Tang Yaguang __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
On Fri, Oct 13 2017, Yaguang Tang wrote: > I see the latest Gnocchi support using Redis as a storage backend, I am > testing performance of Redis and Ceph, it seems using Redis as storage > backend we can achieve more realtime metric > data, gnocchi status shows there is always few metric to process. > > Is Redis a recommend storage backend ? Redis is recommended as an incoming measures backend, not really as a storage – though it really depends on the size of your installation. Up until 4.0 version, Gnocchi process metrics every $metricd.metric_processing_delay seconds. With version 4.1 (to be released), the Redis incoming driver has a more realtime processing delay which avoids having to poll for incoming data. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
On 2017-10-13 03:37 AM, Julien Danjou wrote: > On Fri, Oct 13 2017, Yaguang Tang wrote: > >> I see the latest Gnocchi support using Redis as a storage backend, I am >> testing performance of Redis and Ceph, it seems using Redis as storage >> backend we can achieve more realtime metric >> data, gnocchi status shows there is always few metric to process. >> >> Is Redis a recommend storage backend ? > > Redis is recommended as an incoming measures backend, not really as a > storage – though it really depends on the size of your installation. > > Up until 4.0 version, Gnocchi process metrics every > $metricd.metric_processing_delay seconds. With version 4.1 (to be > released), the Redis incoming driver has a more realtime processing > delay which avoids having to poll for incoming data. > what Julien said :) redis as a storage driver really depends on how you setup persistence[1] and how much you trust it[2]. would love to see your redis vs ceph numbers compared to mine[3] :) [1] https://redis.io/topics/persistence [2] https://aphyr.com/posts/283-jepsen-redis [3] https://medium.com/@gord.chung/gnocchi-4-introspective-a83055e99776 (tested part of 4.1 optimisations) cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
Hi Gordon, We launched 300vms and each vm has about 10 metrics, OpenStack cluster have 3 controllers and 2 compute nodes(ceph replication is set to 2). what we want to archive is to make all metric measures data get processed as soon as possible, metric processing delay is set to 10s, and ceilometer polling interval is 30s. when the backend of incoming and storeage is set to ceph, the average of "gnocchi status" shows that there are around 7000 measures waiting to be process, but when changing incoming and storage backend to Redis, the result of gnocchi status shows unprocessed measures is around 200. we try to add more metricd process on every controller nodes, to improve the performance of calculate and writing speed to storage backend but have little effect. On Fri, Oct 13, 2017 at 9:03 PM, gordon chung wrote: > > > On 2017-10-13 03:37 AM, Julien Danjou wrote: > > On Fri, Oct 13 2017, Yaguang Tang wrote: > > > >> I see the latest Gnocchi support using Redis as a storage backend, I am > >> testing performance of Redis and Ceph, it seems using Redis as storage > >> backend we can achieve more realtime metric > >> data, gnocchi status shows there is always few metric to process. > >> > >> Is Redis a recommend storage backend ? > > > > Redis is recommended as an incoming measures backend, not really as a > > storage – though it really depends on the size of your installation. > > > > Up until 4.0 version, Gnocchi process metrics every > > $metricd.metric_processing_delay seconds. With version 4.1 (to be > > released), the Redis incoming driver has a more realtime processing > > delay which avoids having to poll for incoming data. > > > > what Julien said :) > > redis as a storage driver really depends on how you setup persistence[1] > and how much you trust it[2]. > > would love to see your redis vs ceph numbers compared to mine[3] :) > > [1] https://redis.io/topics/persistence > [2] https://aphyr.com/posts/283-jepsen-redis > [3] https://medium.com/@gord.chung/gnocchi-4-introspective-a83055e99776 > (tested part of 4.1 optimisations) > > cheers, > > -- > gord > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Tang Yaguang __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
On 2017-10-18 12:15 PM, Yaguang Tang wrote: > > We launched 300vms and each vm has about 10 metrics, OpenStack cluster > have 3 controllers and 2 compute nodes(ceph replication is set to 2). seems smaller than my test, i have 20K metrics in my test > > what we want to archive is to make all metric measures data get > processed as soon as possible, metric processing delay is set to 10s, > and ceilometer polling interval is 30s. are you batching the data you push to gnocchi? in gnocchi4.1, the redis driver will (attempt to) process immediately, rather than cyclically using the metric processing delay. > > when the backend of incoming and storeage is set to ceph, the average of > "gnocchi status" > shows that there are around 7000 measures waiting to be process, but > when changing incoming and storage backend to Redis, the result of > gnocchi status shows unprocessed measures is around 200. i should clarify, having a high gnocchi status is not a bad thing, ie, if you just send a large spike of measures, it's expected to jump. it's bad if never shrinks. that said, how many workers do you have? i have 18 workers for 20K metrics and it takes 2minutes i believe? do you have debug enable? how long does it take to process metric? when i tested gnocchi+ceph vs gnocchi+redis, i didn't see a very large difference in performance (redis was definitely better though). maybe it's your ceph environment? > > we try to add more metricd process on every controller nodes, to improve > the performance of > calculate and writing speed to storage backend but have little effect. performance should increase (relatively) proportionally. ie. if you 2x metricd, you should perform (almost) 2x quicker. if you add 5% more metricd, you should perform (almost) 5% quicker. cheers, -- gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
Hi Gordon， Thanks for your test results， we investigate more on our env, finally it turns out that our ceph cluster isn't work as expected. which made gnocchi performance decrease a lot. On Thu, Oct 19, 2017 at 1:09 AM, gordon chung wrote: > > > On 2017-10-18 12:15 PM, Yaguang Tang wrote: > > > > We launched 300vms and each vm has about 10 metrics, OpenStack cluster > > have 3 controllers and 2 compute nodes(ceph replication is set to 2). > > seems smaller than my test, i have 20K metrics in my test > > > > > what we want to archive is to make all metric measures data get > > processed as soon as possible, metric processing delay is set to 10s, > > and ceilometer polling interval is 30s. > > are you batching the data you push to gnocchi? in gnocchi4.1, the redis > driver will (attempt to) process immediately, rather than cyclically > using the metric processing delay. > > > > > when the backend of incoming and storeage is set to ceph, the average of > > "gnocchi status" > > shows that there are around 7000 measures waiting to be process, but > > when changing incoming and storage backend to Redis, the result of > > gnocchi status shows unprocessed measures is around 200. > > i should clarify, having a high gnocchi status is not a bad thing, ie, > if you just send a large spike of measures, it's expected to jump. it's > bad if never shrinks. > > that said, how many workers do you have? i have 18 workers for 20K > metrics and it takes 2minutes i believe? do you have debug enable? how > long does it take to process metric? > > when i tested gnocchi+ceph vs gnocchi+redis, i didn't see a very large > difference in performance (redis was definitely better though). maybe > it's your ceph environment? > > > > > we try to add more metricd process on every controller nodes, to improve > > the performance of > > calculate and writing speed to storage backend but have little effect. > > performance should increase (relatively) proportionally. ie. if you 2x > metricd, you should perform (almost) 2x quicker. if you add 5% more > metricd, you should perform (almost) 5% quicker. > > cheers, > > -- > gord > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Tang Yaguang __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
