Re: [openstack-dev] Gnocchi sizing on production

[gordon chung]

On 28/12/16 05:49 AM, Sam Huracan wrote:
> Thanks,
>
>
> How can I increase processing delay? I think it could increase the
> number of measures in queue.
>

you can also force the measures in backlog/queue to be processed on 
request by passing in refresh=True. in theory, you could turn off all 
metricd workers doing this. of course, this will just push all 
calculations to request time which may result in longer response time 
and probably higher memory usage.

cheers,

-- 
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Julien Danjou]

On Wed, Dec 28 2016, Sam Huracan wrote:

> How can I increase processing delay? I think it could increase the number
> of measures in queue.

It would, but that'll allow gnocchi-metricd to process them in batch, so
it'll use less CPU overall. Obviously, the computing of metrics will be
less real-time-y.

You can increase `metric_processing_delay' to make it less aggressive.
The default is 60 seconds.

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Sam Huracan]

Thanks,


How can I increase processing delay? I think it could increase the number
of measures in queue.

2016-12-28 16:29 GMT+07:00 Julien Danjou :

> On Wed, Dec 28 2016, Sam Huracan wrote:
>
> > I recently use VM on lab for Gnocchi Server, with 4 vCPU, 8192 MB RAM,
> and
> > after configuring 10 workers, I realize the number or gnocchi measures
> > decrease significantly (gnocchi status), but it drain much more CPU and
> > RAM, and VM freezed after 30 minutes running. :)
>
> Gnocchi precomputes everything, that's why it uses CPU. If you want to
> have it use less CPU, you should increase the processing delay or modify
> the default archive policies to compute less. :)
>
> --
> Julien Danjou
> ;; Free Software hacker
> ;; https://julien.danjou.info
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Julien Danjou]

On Wed, Dec 28 2016, Sam Huracan wrote:

> I recently use VM on lab for Gnocchi Server, with 4 vCPU, 8192 MB RAM, and
> after configuring 10 workers, I realize the number or gnocchi measures
> decrease significantly (gnocchi status), but it drain much more CPU and
> RAM, and VM freezed after 30 minutes running. :)

Gnocchi precomputes everything, that's why it uses CPU. If you want to
have it use less CPU, you should increase the processing delay or modify
the default archive policies to compute less. :)

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Sam Huracan]

Hi guys,

I'm using gnocchi version 3.0:
https://github.com/openstack/gnocchi/tree/stable/3.0 + OpenStack Mitaka,
storing gnocchi metrics on Ceph Storage
All metrics I need collect are VM resources: CPU, Memory, Disk, Network. We
're going to monitor thoses metrics every minutes for alarming when
reaching threshold, and keep them 1 month for billing. (medium archive
policy)

I recently use VM on lab for Gnocchi Server, with 4 vCPU, 8192 MB RAM, and
after configuring 10 workers, I realize the number or gnocchi measures
decrease significantly (gnocchi status), but it drain much more CPU and
RAM, and VM freezed after 30 minutes running. :)

Anyone has ever deployed gnocchi on production, Could you share your
experiences?

Thanks and regards



2016-12-28 5:22 GMT+07:00 Mike Perez :

> On 16:07 Dec 27, Sam Huracan wrote:
> > Hi,
> >
> > I'm testing gnocchi on lab and realizing it drains CPU and RAM too much.
> >
> > Do you have a recommendtion for sizing Gnocchi Server configuration?
> >
> > Thanks and regards,
>
> Hi Sam,
>
> I recommend asking the OpenStack operators mailing list [1] for
> configuration
> help. It's likely someone on their has knowledge of running Gnocchi in
> production, as this list is mostly about development discussions.  Thanks!
>
> [1] - http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-operators
>
> --
> Mike Perez
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Mike Perez]

On 16:07 Dec 27, Sam Huracan wrote:
> Hi,
> 
> I'm testing gnocchi on lab and realizing it drains CPU and RAM too much.
> 
> Do you have a recommendtion for sizing Gnocchi Server configuration?
> 
> Thanks and regards,

Hi Sam,

I recommend asking the OpenStack operators mailing list [1] for configuration
help. It's likely someone on their has knowledge of running Gnocchi in
production, as this list is mostly about development discussions.  Thanks!

[1] - http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Alex Krzos]

Hi Sam,

I have also been testing Gnocchi and found high resource utilization
depending on the gnocchi configuration and archival-policy.  Since
hardware can vary and your ability to process metrics will be cpu
heavy and your ability to store metrics will be limited by your disk
io it will be highly dependent upon your hardware and worker counts.
Could you elaborate on the configuration and version you are using
along with the resource count you have metrics collected on?  The more
data points we can share with the community the better defaults and
scale/capacity guidelines we can provide.  Also if you could elboarate
on what aggregations you truely need, we can take that into account
for the default policies.

I can make the following recommendations of moving Gnocchi API (hosted
as a WSGI app in httpd) and Gnocchi metricd to separate hardware (if
you are not doing this already.)  This will prevent resource
contention between other OpenStack Services and the Telemetry
Services.

Alex Krzos | Performance Engineering
Red Hat
Desk: 919-754-4280
Mobile: 919-909-6266

On Tue, Dec 27, 2016 at 4:07 AM, Sam Huracan  wrote:
> Hi,
>
> I'm testing gnocchi on lab and realizing it drains CPU and RAM too much.
>
> Do you have a recommendtion for sizing Gnocchi Server configuration?
>
> Thanks and regards,
>
> Sam
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Gnocchi sizing on production

[Julien Danjou]

On Tue, Dec 27 2016, Sam Huracan wrote:

Hi Sam,

> I'm testing gnocchi on lab and realizing it drains CPU and RAM too much.
>
> Do you have a recommendtion for sizing Gnocchi Server configuration?

Which version are you testing? What's the RAM consumption?
You can reduce the default archive policies and increase processing
latency to decrease CPU usage.

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev