Re: [Openstack-operators] Scaling Ceilometer compute agent?
Hello Bill Possible to share how many instance and how many meter per instance you collecting and getting this error ? I guess for scaling purpose , you are talking about this , right http://docs.openstack.org/ha-guide/controller-ha-telemetry.html Regards VM From: Bill Jones Date: Tuesday 14 June 2016 at 18:03 To: "openstack-oper." Subject: [Openstack-operators] Scaling Ceilometer compute agent? Has anyone had any experience with scaling ceilometer compute agents? We're starting to see messages like this in logs for some of our compute agents: > WARNING ceilometer.openstack.common.loopingcall [-] task interval_task at 0x2092cf8> run outlasted interval by 293.25 sec > This is an indication that the compute agent failed to execute its pipeline processing within the allotted interval (in our case 10 min). The result of this is that less instance samples are generated per hour than expected, and this causes billing issues for us due to the way we calculate usage. It looks like we have three options for addressing this: make the pipeline run faster, increase the interval time, or scale the compute agents. I'm investigating the latter. I think I read in the ceilometer architecture docs that the agents are designed to scale, but I don't see anything in the docs on how to facilitate that. Any pointers would be appreciated. Thanks, Bill ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Scaling Ceilometer compute agent?
Thanks for info Bill Regards VM From: Bill Jones Date: Wednesday 15 June 2016 at 00:26 To: Vahric Muhtaryan Cc: "openstack-oper." Subject: Re: [Openstack-operators] Scaling Ceilometer compute agent? Thanks for all the pointers. Vahric, we're running into this in our lab on a compute host with 135 instances and 12 meters, 3 of which we developed. /Bill On Tue, Jun 14, 2016 at 2:54 PM, Vahric Muhtaryan wrote: > Hello Bill > > Possible to share how many instance and how many meter per instance you > collecting and getting this error ? > > I guess for scaling purpose , you are talking about this , right > http://docs.openstack.org/ha-guide/controller-ha-telemetry.html > > Regards > VM > > From: Bill Jones > Date: Tuesday 14 June 2016 at 18:03 > To: "openstack-oper." > Subject: [Openstack-operators] Scaling Ceilometer compute agent? > > Has anyone had any experience with scaling ceilometer compute agents? > > We're starting to see messages like this in logs for some of our compute > agents: > >> WARNING ceilometer.openstack.common.loopingcall [-] task > interval_task at 0x2092cf8> run outlasted interval by 293.25 sec >> > This is an indication that the compute agent failed to execute its pipeline > processing within the allotted interval (in our case 10 min). The result of > this is that less instance samples are generated per hour than expected, and > this causes billing issues for us due to the way we calculate usage. > > It looks like we have three options for addressing this: make the pipeline run > faster, increase the interval time, or scale the compute agents. I'm > investigating the latter. > > I think I read in the ceilometer architecture docs that the agents are > designed to scale, but I don't see anything in the docs on how to facilitate > that. Any pointers would be appreciated. > > Thanks, > Bill > ___ OpenStack-operators mailing > list > OpenStack-operators@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mail > man/listinfo/openstack-operators ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] VXLAN / Tenant Network Issue
Hello Grant ,
Possible to share ml2_conf.ini , dhcp_agent.ini and l3_agent.ini files ?
Regards
VM
From: Grant Morley
Date: Thursday 8 September 2016 at 15:12
To: OpenStack Operators
Cc:
Subject: [Openstack-operators] VXLAN / Tenant Network Issue
Hi All,
We are working off the OSA deployment for a new cloud system we are building
and everything seems to be working apart from the tenant VXLAN network. We
have tried various troubleshooting but the initial DHCP request, is not
making it out of the linux bridge on the compute node. We have checked all
physical networking and switch setup and they appear to be fine.
Below is an output of related networking components that we have configured.
(Sorry for the long post but wanted to get as much info on here) Can anyone
see what might be causing the issue or where we have gone wrong?
Neutron subnet and router:
neutron) net-list
+--+
++
| id | name
| subnets|
+--+
++
| b1da0a4f-2d06-46af-92aa-962c7a7c36f9 | ext-net
| 405f439c-51bb-40b6-820a-9048c2ee69fe |
| |
| 185.136.232.0/22 |
| a256ccb2-273a-4738-97ab-bd8bfbc2a2cc | HA network tenant
7b5aad6af3ee450ea60e06aaaba2da50 | 6d98faac-2e3b-43c8-bcd6-f9a6f5dcc45e
|
| |
| 169.254.192.0/18 |
| f88ceab1-a392-4281-8c60-f57d171a8029 | vxlan-172
| 367e88eb-b09f-4ce5-bfff-5d9e0b0e14b0
| 172.16.0.0/24
+--+
++
(neutron) net-show f88ceab1-a392-4281-8c60-f57d171a8029
+---+--+
| Field | Value|
+---+--+
| admin_state_up| True |
| id| f88ceab1-a392-4281-8c60-f57d171a8029 |
| mtu | 0|
| name | vxlan-172|
| port_security_enabled | True |
| provider:network_type | vxlan|
| provider:physical_network | |
| provider:segmentation_id | 21 |
| router:external | False|
| shared| False|
| status| ACTIVE |
| subnets | 367e88eb-b09f-4ce5-bfff-5d9e0b0e14b0 |
| tenant_id | 7b5aad6af3ee450ea60e06aaaba2da50 |
+---+--+
(neutron) router-show f31ed1fb-1b90-46e3-b869-d9374e3d08b1
+---+---
-+
| Field | Value
|
+---+---
-+
| admin_state_up| True
|
| distributed | False
|
| external_gateway_info | {"network_id":
"b1da0a4f-2d06-46af-92aa-962c7a7c36f9", "enable_snat": true,
"external_fixed_ips": [{"subnet_id": |
| | "405f439c-51bb-40b6-820a-9048c2ee69fe",
"ip_address": "185.136.232.55"}]}
|
| ha| True
|
| id| f31ed1fb-1b90-46e3-b869-d9374e3d08b1
|
| name | ext-router
|
| routes |
|
| status| ACTIVE
|
| tenant_id | 7b5aad6af3ee450ea60e06aaaba2da50
|
+---+---
-+
(neutron) router-port-list f31ed1fb-1b90-46e3-b869-d9374e3d08b1
+--+
+---+--+
| id | name
| mac_address | fixed_ips|
+--+
+---+--+
| 443d8a0e-833e-4dd2-9320-c2a361e97bf0 | HA port tenant
| fa:16
[Openstack-operators] Windows 2016 with Openstack 2016
Hello All , Anybody success to run Windows 2016 on KVM ? I checked the web site looks like not supported yet http://www.linux-kvm.org/page/Guest_Support_Status I saw only Nutanix acropolis success http://vmwaremine.com/2016/04/18/install-windows-server-2016-tp4-nutanix-acr opolis-hypervisor/#sthash.mr5zvd4n.dpbs HPE Helion also looks like not supporting https://docs.hpcloud.com/hos-4.x/helion/planning/hw_support_matrix.html#min_ hardware__guestos-kvm Some known issues https://bugzilla.redhat.com/show_bug.cgi?id=1252134 https://bugzilla.redhat.com/show_bug.cgi?id=1346153 Redhat talking about limit CPU support https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/htm l/6.8_Release_Notes/known_issues_virtualization.html but I don¹t think so I hanged there Anyone succeed to install Windows 2016 with kvm on Openstack ? Regards Vahric Muhtaryan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] Any serious stability and performance issues on RBD as ephemeral storage ?
Hello All, I would like to use ephemeral disks with ceph instead of on nova compute node. I saw that there is an option to configure it but find many different bugs and reports for its not working , not stable , no success at the instance creation time. Anybody In this list use ceph as an ephemeral storage without any problem ? Could you pls share your experiences pls ? Regards VM ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Any serious stability and performance issues on RBD as ephemeral storage ?
Thanks Mike , Thanks for sharing VM From: Mike Smith Date: Saturday, 10 December 2016 at 08:22 To: Vahric Muhtaryan Cc: David Medberry , "openstack-operators@lists.openstack.org" , "m...@mattjarvis.org.uk" Subject: Re: [Openstack-operators] Any serious stability and performance issues on RBD as ephemeral storage ? Vahric - We use Ceph for nova ephemeral as well on thousands of VMs and we love it. Fast provisioning of VMs, solid, reliable and flexible. Works great with live-migration. The only time we ever have a problem is an OSD is allowed to get too full. Never let that happen! Mike Smith Lead Cloud Systems Architect Overstock.com <http://overstock.com> > On Dec 8, 2016, at 10:36 AM, Matt Jarvis wrote: > > I'd say using Ceph for ephemeral disks is the most common deployment pattern, > at DataCentred we've been using it for years. It's rock solid and has been for > several releases. There were some edge case issues around resizing and > snapshotting, but I think that's all been fixed in the last couple of > iterations. > > On Thu, Dec 8, 2016 at 4:58 PM, David Medberry wrote: >> We've been using it and recommending it for years. It solves many many >> problems with a running cloud and there have been very few issues. Pay close >> attention when upgrading versions of CEPH and do things in the right order >> and you will be fine! >> >> On Wed, Dec 7, 2016 at 7:51 AM, Vahric Muhtaryan wrote: >>> Hello All, >>> >>> I would like to use ephemeral disks with ceph instead of on nova compute >>> node. I saw that there is an option to configure it but find many different >>> bugs and reports for its not working , not stable , no success at the >>> instance creation time. >>> Anybody In this list use ceph as an ephemeral storage without any problem ? >>> Could you pls share your experiences pls ? >>> >>> Regards >>> VM >>> >>> ___ >>> OpenStack-operators mailing list >>> OpenStack-operators@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >> >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you. ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] Openstack Ceph Backend and Performance Information Sharing
Hello All , For a long time we are testing Ceph from Firefly to Kraken , tried to optimise many things which are very very common I guess like test tcmalloc version 2.1 , 2,4 , jemalloc , setting debugs 0/0 , op_tracker and such others and I believe with out hardware we almost reach to end of the road. Some vendor tests mixed us a lot like samsung http://www.samsung.com/semiconductor/support/tools-utilities/All-Flash-Array -Reference-Design/downloads/Samsung_NVMe_SSDs_and_Red_Hat_Ceph_Storage_CS_20 160712.pdf , DELL Dell PowerEdge R730xd Performance and Sizing Guide for Red Hat <https://www.google.com.tr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&; uact=8&ved=0ahUKEwiA4Z28_pTSAhXCJZoKHSYVD0AQFggeMAA&url=http%3A%2F%2Fen.comm unity.dell.com%2Ftechcenter%2Fcloud%2Fm%2Fdell_cloud_resources%2F20442913%2F download&usg=AFQjCNGGADYZkbABD_GZ8YMct4E19KSAXA&sig2=YZCEHMq7tnXSpVydMDacIg> and from intel http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/201508 13_S303E_Zhang.pdf At the end using 3 replica (Actually most of vendors are testing with 2 but I believe that its very very wrong way to do because when some of failure happen you should wait 300 sec which is configurable but from blogs we understaood that sometimes OSDs can be down and up again because of that I believe very important to set related number but we do not want instances freeze ) with config below with 4K , random and fully write only . I red a lot about OSD and OSD process eating huge CPU , yes it is and we are very well know that we couldn¹t get total of iOPS capacity of each raw SSD drives. My question is , can you pls share almost same or closer config or any config test or production results ? Key is write, not %70 of read % 30 write or full read things Hardware : 6 x Node Each Node Have : 2 Socker CPU 1.8 GHZ each and total 16 core 3 SSD + 12 HDD (SSDs are in journal mode 4 HDD to each SSD) Raid Cards Configured Raid 0 We did not see any performance different with JBOD mode of raid card because of that continued with raid 0 Also raid card write back cache is used because its adding extra IOPS too ! Achieved IOPS : 35 K (Single Client) We tested up to 10 Clients which ceph fairly share this usage like almost 4K for each Test Command : fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k iodepth=256 --size=1G --numjobs=8 --readwrite=randwrite group_reporting Regards Vahric Muhtaryan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] ceph vs gluster for block
Hello All , For a long time we are testing Ceph and today we also want to test GlusterFS Interesting thing is maybe with single client we can not get IOPS what we get from ceph cluster . (From ceph getting max 35 K IOPS for % 100 random write and gluster gave us 15-17K ) But interesting thing when add additional client to test its get same IOPS with first client means overall performance is doubled , couldn¹t test more client but also interesting things is glusterfs do not use/eat CPU like Ceph , a few percent of CPU is used. I would like to ask with Openstack , anybody use GlusterFS for instance workload ? Anybody used both of them in production and can compare ? Or share experience ? Regards Vahric Muhtaryan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
