[openstack-dev] [ceilometer][libvirt] Libvirt error during instance disk allocation metering

[Ilya Tyaptin]

Hi, folks!

In our deployed envs we met with a libvirt error *"missing storage backend
for network files using rbd protocol"* in *virDomainGetBlockInfo* call [1]

.
This exception is raised when Ceilometer are trying to get info about VM
disk usage and allocation.
It only affects getting measures for a some disk pollsters which added in
this CR [2]

 with specified libvirt call [3]

 .
These pollsters have been added in the Kilo cycle and successful work in
Kilo deployments, but it doesn't work now.

Also, we have a bug in the upstream launchpad [4]

 but it have not been fixed yet.

I would glad to see any ideas about root cause of this issue or ways to
fixing it.

Thank you in advance!

References:
[1] Traceback 


./ceilometer-polling.log.0:4192:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk Traceback (most recent call last):
./ceilometer-polling.log.0:4193:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk   File
"/usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/disk.py",
line 703, in get_samples
./ceilometer-polling.log.0:4194:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk instance,
./ceilometer-polling.log.0:4195:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk   File
"/usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/disk.py",
line 672, in _populate_cache
./ceilometer-polling.log.0:4196:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk for disk, info in disk_info:
./ceilometer-polling.log.0:4197:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk   File
"/usr/lib/python2.7/dist-packages/ceilometer/compute/virt/libvirt/inspector.py",
line 215, in inspect_disk_info
./ceilometer-polling.log.0:4198:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk block_info = domain.blockInfo(device)
./ceilometer-polling.log.0:4199:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk   File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 658, in blockInfo
./ceilometer-polling.log.0:4200:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk if ret is None: raise libvirtError
('virDomainGetBlockInfo() failed', dom=self)
./ceilometer-polling.log.0:4201:2015-11-17 16:20:54.807 14107 ERROR
ceilometer.compute.pollsters.disk libvirtError: internal error: missing
storage backend for network files using rbd protocol

[2] CR with this commit:
https://review.openstack.org/#/c/145819/23/ceilometer/compute/virt/libvirt/inspector.py,cm

[3] Code entry:
https://github.com/openstack/ceilometer/blob/stable/liberty/ceilometer/compute/virt/libvirt/inspector.py#L215
[4] Upstream bug: https://bugs.launchpad.net/ceilometer/+bug/1457440


Best regards,

Tyaptin Il​y​a,

Ceilometer developer,

Mirantis Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Gnocchi] Added Ilya Tyaptin as core reviewer

[Ilya Tyaptin]

​Thank you very much!

Best regards,

Tyaptin Ilia




On Fri, Aug 28, 2015 at 10:32 PM, gord chung  wrote:

>
> On 28/08/15 02:48 PM, Julien Danjou wrote:
>
> Hi fellows,
>
> Ilya did a few good contributions to Gnocchi, especially around the
> InfluxDB driver, so I'm glad to add him to the list of core reviewers.
>
> Welcome aboard.
>
> Cheers,
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> well deserved Ilya!
>
> --
> gord
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Gnocchi][Ceilometer] Gnocchi performance tests

[Ilya Tyaptin]

Hi, folks!

We decided to make a performance gnocchi testing as a part of summit talk
preparation.

Folks who are working at Gnocchi these days, may you add to etherpad test
scenarios which are most helpful and interesting in your opinion?


Etherpad - https://etherpad.openstack.org/p/gnocchi_performance_tests

-- 

Best regards,

Tyaptin Ilia,

Junior Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ceilometer] Performance degradation (docs) for sampling rate less than 10 mins

[Ilya Tyaptin]

Hi Patanjali!

We have inspected this question and got a document with results of this
testing.
Tests are running on physical lab and includes 2000 VMs ran by Nova and 60
second polling interval.

Expanded information in doc
https://docs.google.com/a/mirantis.com/document/d/1jvqIy6fWQBvTZEfdnk37FvF5xtA3l7Cx09VNoe6bBRU

If you have any question I'll be happy to answer.

On Thu, Sep 18, 2014 at 7:56 PM, Srikanta Patanjali 
wrote:

> Hi Team,
>
> I was considering to change the default sampling rate of the Ceilometer
> from 10 mins to less than that. I foresee an adverse impact on its
> performance due to increase in the data inside the MondoDB.
>
> I was wondering if the QA team (or anyone else) has done any of the load
> tests with these parameters ? It would be helpful to have access to these
> results (if any).
>
> If not, I would like to know the views on increasing the sample rate value
> (in the pipleline.yaml file)
>
> Cheers,
> Srikanta
> InIT ¦ ICCLab
> ZHAW
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ceilometer] Work at bug "Ceilometer HBase driver cannot handle non-ascii characters"

[Ilya Tyaptin]

Hi, folks!

I want to engage this bug
 [1].

But fixing of this bug has raised question.
Issue is that happybase does not support unicode symbols and approaches to
processing unicode symbols is needed to discuss.

I suggest next variants:
1. Anyway to check containing unicode symbols in string, if contains then
screen theys.
2. Transform unicode symbols like a html encoding (/xa0 -> %2Fxa0, for
example)

But in these approaches we can't be sure that we haven't false positives at
getting data from HBase.

Thanks for all, wait for your response!

[1] https://bugs.launchpad.net/ceilometer/+bug/1350826

-- 
Best regards,

Tyaptin Ilia,

Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Performance tests of ceilometer-collector and ceilometer-api with different backends

[Ilya Tyaptin]

>I am not sure if I read the 'Testing api' section correctly. Is that table in 
>seconds?  If so a REST API that takes over two minutes (sample-list for Hbase, 
>meter-list in Mongo) doesn't sound very good.

Tim, Joe, in api tests values are in seconds. It's known issues and we
will log and meter step by step api work: backend, ceilometer-api,
cli.
Test results may help find weak link.

>++, and also need the SQL driver tested as well.
Jay, test results for mysql will be ASAP.


Best regards,

Tyaptin Ilia,

Intern Software Engineer.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Performance tests of ceilometer-collector and ceilometer-api with different backends

[Ilya Tyaptin]

Hi, Swann!

Thanks for your feedback)



On Wed, Apr 23, 2014 at 2:33 PM, Swann Croiset  wrote:
> Hi Ilya,
>
> Interresting, thanks for sharing.
> So the quick conclusion to your numbers seems indicated that mongodb is more
> efficient for both reading and writing,
> except for 2 cases for retrieving data (meters and resouces listing) ..
>

It's not so indisputable fact. Performance fall may happen due to
cluster VMs base.
For future tests we've already added standalone mysql and standalone
hbase backend.
Also we will deploy mongo cluster on vms in the nearest future

> However for the reading operations,
> it's should be confirmed (or precised) where the time is really spent, would
> be interresting to compute the distribution of times spent by each layer :
> backend -> api -> cli  .. similarly to what you did for collector by custom
> logging (or by instrumentation..)
>
> To add additional use cases (and to be more relevant) it will be good to use
> queries executed by billing systems or the alarm evaluator aka filtering a
> limited subsets of samples (by resource and/or user and/or tenant) .. to see
> the numbers without retrieving ten of thousands of samples.

They are good ideas. I'll add it to tests and show results as soon as possible.

> btw, others indicators should help to give a good picture, I see for now:
> errors rate, queue lenght (rabbit), returned samples|meters|resources by API
> calls, missing samples (after the populating)
> and some system metrics also.

In present time we are calculating the time which messages are waiting
for in rabbitmq queue. This metric has the same meaning as queue
length. Also we logs backend errors but not so many errors as we might
expect happens in tests.


> what was the caracteristics of serveurs used for these load test?

Controller with 16Gb RAM, 8 procs and 3 VMs with 8 GB RAM and 8 procs
(for Hbase).



Best regards,

Tyaptin Ilia,

Intern Software Engineer.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Ceilometer] Performance tests of ceilometer-collector and ceilometer-api with different backends

[Ilya Tyaptin]

Hi team!

In light of discussions about ceilometer backends, we decided to test
performance of different
storage backends with collector and api services because these services
depend on backends availability.

For the collector testing we are using not completely real data, we are
generating looking like real samples with variable rate, sending them to
the collector and metering the time of these messages processing. Testing
result is the time between message receiving for recording message to db.

For the api testing we are only comparing the time of requests to api with
different backends.

We have prepared a document with more detailed description of test plan and
first results.
This url: 
*https://docs.google.com/document/d/1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing
*

Please, add you cases and proposals for perfomance testing of collector and
api to the document comments.

Also you may use etherpad if it is more convenient.
https://etherpad.openstack.org/p/performance_test_for_ceilometer_collector_and_api

---

Best regards,

Tyaptin Ilia,

Intern Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer]

[Ilya Tyaptin]

Hi!

In this method "q" is query argument and used for filtering response.

In code you may set this parameter with list of dicts, like this:

q=[{"field": "resource_id", "value": "a", "op": "eq"}]

More examples of queries at
http://docs.openstack.org/developer/ceilometer/webapi/v2.html?highlight=query#api-and-cli-query-examples


On Tue, Apr 8, 2014 at 1:13 AM, Hachem Chraiti  wrote:

> hi erveryone,thats a python code:
>
> from ceilometerclient.v2 import client
>
> ceilometer =client.Client(endpoint='http://controller:8777/v2/resources',
> token='e8e70342225d64d1d20a')
>
> print  ceilometer.resources.list(q)
>
>
> whats this "q" parameter??
>
> Sincerly ,
> Chraiti Hachem,software engineer
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Best regards,

Tyaptin Ilia,

Intern Software Engineer.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev