Re: [Openstack-operators] How to apply the bugfix for keystone pyldap (#1736241)

[Jesse Pretorius]

> From: David Young 
> mailto:dav...@funkypenguin.co.nz>>
> Date: Friday, December 8, 2017 at 10:34 PM
> To: 
> "openstack-operators@lists.openstack.org"
>  
> mailto:openstack-operators@lists.openstack.org>>
> Subject: [Openstack-operators] How to apply the bugfix for keystone pyldap 
> (#1736241)


> This bug was found in the wild, and is currently affecting my OSA playbook, 
> which needs to build a repo container:
> https://bugs.launchpad.net/openstack-ansible/+bug/1736241

> I see it's in process of being fixed, I'm curious re how I'll apply it to my 
> environment.

> My playbooks etc are 15.1.8 - do I need to do a minor release update to 
> receive the fix, or will it "just work" the next time I build a repo 
> container?

Hi David,

You’ll need to perform a minor upgrade to 15.1.13 in order to get it. See [1] 
for more details regarding how to do that.

[1] 
https://docs.openstack.org/openstack-ansible/ocata/upgrade-guide/minor-upgrade.html#execute-a-minor-version-upgrade



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] (openstack-ansible) Mitaka 13.3.12 failing to report correct resource usage

[Jesse Pretorius]

From: "Craft, Mike" 
Date: Thursday, March 2, 2017 at 7:50 PM
To: "openstack-operators@lists.openstack.org" 

Subject: [Openstack-operators] (openstack-ansible) Mitaka 13.3.12 failing to 
report correct resource usage

Hello,

Did an upgrade from liberty to mitaka and we are experiencing an issue where 
the audit of locally available compute resource usage is not seeing instances 
that were deployed prior to the upgrade. Has anyone experienced this before? We 
hard power cycled all the vms (and hypervisor) to make sure they are running on 
the latest nova version. This is impacting scheduling of resources/quota usage 
as you could imagine ☺

If you look in the nova database, can you see those instances there? From the 
look of things, your hypervisor has no instances allocated to it in its 
database. I hope that you have a backup! ☺



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OSA Newton Release Deployment in test environment

[Jesse Pretorius]

LXC will setup eth0 in the container which will NAT through the host to access 
the internet.

Clearly there’s something wrong with that traffic path, so step through it like 
you would with any environment by validating that it is able to reach its 
gateway (host), then next hop, etc.

From: Amit Kumar 
Date: Monday, February 6, 2017 at 8:17 AM
To: gustavo panizzo 
Cc: "openstack-operators@lists.openstack.org" 

Subject: Re: [Openstack-operators] OSA Newton Release Deployment in test 
environment

Error is because LXC repo container on infra node is not having connectivity to 
internet and is unable to download these packages. Thing is that why containers 
are not having connectivity to internet whereas infra node is having 
connectivity to internet.

Any idea if this is a general behavior that LXC containers are not able to 
reach internet?

On Mon, Feb 6, 2017 at 1:39 PM, gustavo panizzo 
mailto:g...@zumbi.com.ar>> wrote:
On Sat, Feb 04, 2017 at 11:52:10PM +0530, Amit Kumar wrote:
> Hi All,
>
> used.\nWARNING: The following packages cannot be authenticated!\n

dpkg/apt are telling you what the problem is, go to the server and fix
apt


> python-pkg-resources python-setuptools\n", "stdout_lines": ["Reading
> package lists...", "Building dependency tree...", "Reading state
> information...", "The following extra packages will be installed:", "
> python-pkg-resources", "Suggested packages:", " python-openssl-doc
> python-openssl-dbg python-distribute", " python-distribute-doc doc-base",
> "The following NEW packages will be installed:", " python-openssl
> python-pkg-resources python-pyasn1 python-setuptools", "0 upgraded, 4 newly
> installed, 0 to remove and 4 not upgraded.", "Need to get 418 kB of
> archives.", "After this operation, 1801 kB of additional disk space will be

> used.", "WARNING: The following packages cannot be authenticated!", "
> python-pkg-resources python-setuptools"]}
same error again


--
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333

keybase: https://keybase.io/gfa



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

I *think* you’d have to set the following in your 
/etc/openstack_deploy/user_vriables.yml file:

openstack_service_publicuri_proto: http
openstack_external_ssl: false
haproxy_ssl: false

I might be missing more, but that should get you started.

From: Achi Hamza 
Date: Friday, November 18, 2016 at 1:56 PM
To: Jesse Pretorius , 
"OpenStack-operators@lists.openstack.org" 

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse.
So should i set the parameter haproxy_ssl to false in the default folder of the 
haproxy_server role or somewhere else ?

On 18 November 2016 at 13:43, Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>> wrote:
Ah, then that’s the cause. You can’t have both external and internal addresses 
be the same unless you disable SSL for public endpoints.

From: Achi Hamza mailto:h16m...@gmail.com>>
Date: Friday, November 18, 2016 at 7:02 AM
To: Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>>, 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
mailto:openstack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

No Jesse, you got me wrong. My external_lv_vip_address and 
internal_vip_lb_address are the same (172.16.1.2), which is also the IP address 
of node01 on which haproxy is running.

This is how it looks on my openstack_user_config.yml file:
global_overrides:
  internal_lb_vip_address: 172.16.1.2
  external_lb_vip_address: 172.16.1.2
  management_bridge: "br-mgmt"


On 17 November 2016 at 20:24, Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>> wrote:
Hmm, that’s odd – if you configured your external_lv_vip_address and 
internal_vip_lb_address to be different then that should not be happening, 
because SSL is implemented on the external VIP. We do not support the use of 
both SSL and non-SSL on the same IP address as that is not something that can 
be done (share the same port, but use HTTPS internally and HTTP externally).

Are you sure that both addresses in your configuration are different?

From: Achi Hamza mailto:h16m...@gmail.com>>
Date: Thursday, November 17, 2016 at 5:58 PM
To: Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>>, 
"klindg...@godaddy.com<mailto:klindg...@godaddy.com>" 
mailto:klindg...@godaddy.com>>, 
"OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>"
 
mailto:OpenStack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse and Lindgren,

Thank you both for the responses. think i figured out the root cause of the 
problem, which is SSL. But first the answers of your questions:
Are you deploying haproxy onto the same host as the repo container, or a 
different host?
Yes, it is on the same host.

Have you bound the VIP address manually?
No, through openstack-ansible playbooks.

Is the VIP address shared in some way – ie is it used for the host and haproxy?
Yes, it is used for the host and haproxy (keepalived is disabled).

After tcpdumping ( tcpdump   -i any port 8181 -n) i found that the packets from 
galera conatiner are not forwarded to repo container through the VIP, and the 
haproxy file is full with this error message:
root@node01:~# tail /var/log/haproxy.log
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38050<http://172.16.1.94:38050> [17/Nov/2016:18:02:16.786] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38052<http://172.16.1.94:38052> [17/Nov/2016:18:02:16.791] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38054<http://172.16.1.94:38054> [17/Nov/2016:18:02:16.795] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38056<http://172.16.1.94:38056> [17/Nov/2016:18:02:16.800] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38058<http://172.16.1.94:38058> [17/Nov/2016:18:02:16.805] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38060<http://172.16.1.94:38060> [17/Nov/2016:18:02:16.809] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38062<http://172.16.1.94:38062> [17/Nov/2016:18:02:16.814] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38064<http://172.16.1.94:38064> [17/Nov/2016:18:02:16.819] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38066<http://172.16.1.94:38066> [17/Nov/2016:18:02:16.823] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:

Re: [Openstack-operators] feedback on pymysql

[Jesse Pretorius]

 From: Matt Fischer 

As a part of our upgrades to Newton we are transitioning our services to use 
pymysql rather than the deprecated MySQL-Python [1]. I believe pymsql has been 
the default in devstack and the gate for sometime now and that MySQL-Python is 
essentially untested and not updated, hence our desire to switch.

devstack is one thing, but I'm curious if anyone has experience operating in 
production with this, especially if there are issues. I've not seen anything in 
testing but if anyone else has I'd love to know positive or negative.

[1] https://wiki.openstack.org/wiki/PyMySQL_evaluation#MySQL-Connector-Python

As a data point, OpenStack-Ansible switched to using pymysql by default for 
Liberty. We’ve had no negative feedback from any deployers using OSA as far as 
I know.


Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

From: Achi Hamza 
Date: Thursday, November 17, 2016 at 1:57 PM
To: Jesse Pretorius , 
"OpenStack-operators@lists.openstack.org" 

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse, but these iptables rules are just applied on the deployment 
node not the host nodes. do i have to omit these rules even on the deployment 
node ?

Thank you

Ah, then that’s a red herring. As long as your hosts can reach the internet 
through it, then you’re good on that front.

Let’s go back to verifying access to the repo – try checking access from the 
repo server to itself:

ansible repo_all -m uri -a "url=http://localhost:8181/os-releases/";

or

ansible repo_all –m shell –a "curl http://localhost:8181/os-releases/";



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

Please continue to make use of the list instead of going off list. The list 
archives are a reference point for others who may find themselves in the same 
situation.

Right, so the service is up. The load balancer is up, so we’re down to 
networking issues or perhaps something a little more obscure.

Can you confirm whether you’re able to reach the repo contents from each of the 
following points: repo container, repo container host, deployment host

ie

ansible repo_all -m shell -a "curl 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt";
ansible hosts -m shell -a "curl 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt";
curl http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt

J

From: Achi Hamza 
Date: Thursday, November 17, 2016 at 10:59 AM
To: Jesse Pretorius 
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse.

here are the output of the 2 commands:

nginx server is UP:
root@maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a 
"service nginx status"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e 
@/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: 
enabled)
   Active: active (running) since Wed 2016-11-16 13:12:57 UTC; 21h ago
 Main PID: 2729 (nginx)
Tasks: 3
   Memory: 956.0K
  CPU: 5.832s
   CGroup: /system.slice/nginx.service
   ├─2729 nginx: master process /usr/sbin/nginx -g daemon on; 
master_process on
   ├─2730 nginx: worker process
   └─2731 nginx: worker process

Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Starting A high 
performance web server and a reverse proxy server...
Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Started A high 
performance web server and a reverse proxy server.

the repo-all* is also UP:
root@node01:~# hatop -s /var/run/haproxy.stat
>>> rabbitmq_mgmt-back
..ner-2148a5b81 DOWN   L4CON 1  000 
0 0 0 0
BACKEND   0 DOWN 0  000 
0 0   410 0

>>> repo_all-front-1
FRONTEND  0 OPEN 0  000 
0 1  4096 0

>>> repo_all-back
..ner-82b4e1f61 UP L7OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

>>> repo_cache-front-1
FRONTEND  0 OPEN 0  000 
0 1  4096 0

>>> repo_cache-back
..ner-82b4e1f61 UP L7OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

>>> repo_git-front-1
FRONTEND  0 OPEN 0  000 
0 0  4096 0

>>> repo_git-back
..ner-82b4e1f61 UP L4OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

 1-STATUS  2-TRAFFIC  3-HTTP  4-ERRORS  5-CLI       
  [#30/#0]

What else please ?

On 17 November 2016 at 11:47, Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>> wrote:
OK, so it’s clearly there in terms of data.

Can you now verify whether the web server is up on the repo server?

ansible repo_all -m shell -a "service nginx status"

Also check haproxy by accessing the haproxy host and running:

hatop –s /var/run/haproxy.stat
Look specifically for the repo_all-* entries and check whether they’re up/down.
There should be at least one front end and one back end.

Thanks,

Jesse

From: Achi Hamza mailto:h16m...@gmail.com>>
Date: Thursday, November 17, 2016 at 10:39 AM
To: Jesse Pretorius 
mailto:jesse.pretor...@rackspace.co.uk>>
Cc: 
"OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>"
 
mailto:OpenStack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse,

Yes, i can:

root@maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "find 
/var/www/repo/os-releases -type f"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e 
@/e

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

Hi Achi,

Can you list the files in /var/www/repo/os-release?

ansible repo_all -m shell -a "find /var/www/repo/os-releases -type f"

Thanks,

Jesse
IRC: odyssey4me

From: Achi Hamza 
Date: Thursday, November 17, 2016 at 10:01 AM
To: "OpenStack-operators@lists.openstack.org" 
, "openstack-...@lists.openstack.org" 

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse and Jean-Philippe,

Thank you both for your responses.

@Jesse
I am just having one node for the infra and one repo, for now:

 root@maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "du 
-sh /var/www/repo/*"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e 
@/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
900K
/var/www/repo/links
656M
/var/www/repo/openstackgit
964K
/var/www/repo/os-releases
4.0K
/var/www/repo/pkg-cache
105M
/var/www/repo/pools
273M
/var/www/repo/venvs

@Jean-Philippe
Yes, i am using haproxy. I cannot wget the 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt file 
neither form the bare metal node nor from within the container:

root@node01-galera-container-368f269a:~# wget -c 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
--2016-11-17 09:53:19--  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:20--  (try: 2)  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:22--  (try: 3)  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

I attached my haproxy.conf file, can you please check it ? If the problem 
persists i will pop into the irc channel.

Thank you,
Hamza






On 17 November 2016 at 10:10, Jean-Philippe Evrard 
mailto:jean-philippe.evr...@rackspace.co.uk>>
 wrote:
Hello,

Is this a greenfield newton?
Could you wget your 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  file 
reliably?
Are you using haproxy?  Are all the backends OK there?

Don’t hesitate to come by our irc channel, it’s probably easier to have a 
conversation there.

Best regards,
Jean-Philippe

From: Achi Hamza mailto:h16m...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-...@lists.openstack.org>>
Date: Thursday, 17 November 2016 at 08:56
To: 
"openstack-...@lists.openstack.org" 
mailto:openstack-...@lists.openstack.org>>
Subject: [openstack-dev] [openstack-ansible] pip issues


Hi all,

Context:
openstact-ansible newton, tag: 14.0.1
OS: ubuntu 16.04 LTS

the pip command is unable to download packages form the repo container. I can 
reach the repo container from within galera container through the port 8181:

root@node01:~# lxc-attach -n node01_galera_container-368f269a
root@node01-galera-container-368f269a:~# nc -nzv 172.16.1.2 8181
Connection to 172.16.1.2 8181 port [tcp/*] succeeded!

However, when i run galera-install.yml playbook it throws the following error 
(but the root cause is pip_install role)  :
root@maas:/opt/openstack-ansible/playbooks# openstack-ansible galera-install.yml
………..
TASK [pip_install : Install pip packages (fall back mode)] *
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (5 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (4 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (3 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (2 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (1 
retries left).fatal: [node01_galera_container-368f269a]: FAILED! => {"changed": 
false, "cmd": "/usr/local/bin/pip install -U --isolated --constraint 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  
ndg-httpsclient requests", "failed": true, "msg": "\n:stderr: Retrying 
(Retry(total=4, connect=None, read=None, redirect=None)) after connection 
broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': 
/os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=3, 
connect=None, read=None, redirect=None)) after connection broken by 
'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': 
/os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=2, 
connect=None, read=None, redirect=None)) after connection broken by 
'ProtocolError('Connection aborted.', BadStatusLine(\

Re: [Openstack-operators] openstack-ansible - recovery of failed infrastructure host

[Jesse Pretorius]

Hi Ben,

If you stand up a new infrastructure host with the same IP address and name in 
openstack_user_config then execute the setup-hosts, setup-infrastructure and 
setup-openstack plays it should put everything back in place for you without 
further intervention.

Jesse
IRC: odyssey4me

On 10/24/16, 9:58 AM, "Ben Lagunilla"  wrote:

Hi All,

How do we go about restoring a failed infrastructure host? Setup is 
openstack mitaka with high availability configured on 3 infrastructure hosts.

Thanks!



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances failing to launch when rbd backed (ansible Liberty setup)

[Jesse Pretorius]

Hi Grant/Chris,

Can you verify whether there is a symlink from inside the venv to the location 
of the RBD python libs in the system packages? If there isn’t one then you 
should be able to re-execute the os-nova-install.yml playbook as it gets 
executed here: 
https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/os_nova/tasks/nova_compute_kvm_install.yml#L103-L123

If you can file a bug with some information regarding what actions led you to 
get to this point it’d be great for us to try and figure out what went wrong so 
that we can improve things for the future.

Thanks,

Jesse
IRC: odyssey4me

From: Grant Morley 
Date: Friday, October 21, 2016 at 3:23 PM
To: Chris Sarginson , OpenStack Operators 

Cc: "ian.ba...@serverchoice.com" 
Subject: Re: [Openstack-operators] Instances failing to launch when rbd backed 
(ansible Liberty setup)


Hi Chris,

That bug suggests there has been a fix in 12.0.8. Hopefully 12.0.16 should have 
that fixed still with a bit of luck , unless the fix hasn't been carried over.

I can see that the bug report says there is also a fix 12.0.9 and 12.0.11.

Do you know if there is a workaround we can try for this? As ideally I don't 
want to be downgrading the openstack_ansible version.

Regards,

On 21/10/16 14:09, Chris Sarginson wrote:
It seems like it may be an occurrence of this bug, as you look to be using 
python venvs:

https://bugs.launchpad.net/openstack-ansible/+bug/1509837

2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py",
 line 117, in __init__

Chris

On Fri, 21 Oct 2016 at 13:19 Grant Morley 
mailto:gr...@absolutedevops.io>> wrote:

Hi all,

We have a openstack-ansible setup and have ceph installed for the backend. 
However whenever we try and launch a new instance it fails to launch and we get 
the following error:

2016-10-21 12:08:06.241 70661 INFO nova.virt.libvirt.driver 
[req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 
324844c815084205995aff10b03a85e1 - - -] [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Creating image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager 
[req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 
324844c815084205995aff10b03a85e1 - - -] [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Instance failed to spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Traceback (most recent call last):
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py",
 line 2156, in _build_resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] yield resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py",
 line 2009, in _build_and_run_instance
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] block_device_info=block_device_info)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2527, in spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] admin_pass=admin_password)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2939, in _create_image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] backend = image('disk')
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2884, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] fname + suffix, image_type)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py",
 line 967, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] return backend(instance=instance, 
disk_name=disk_name)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/py

Re: [Openstack-operators] Moving from distro packages to containers (or virtualenvs...)

[Jesse Pretorius]

On 13 May 2016 at 19:59, Joshua Harlow  wrote:

>
> So I guess its like the following (correct me if I am wrong):
>
> openstack-ansible
> -
>
> 1. Sets up LXC containers from common base on deployment hosts (ansible
> here to do this)
> 2. Installs things into those containers (virtualenvs, packages, git
> repos, other ... more ansible)
> 3. Connects all the things together (more more ansible).
> 4. Decommissions existing container (if it exists) and replaces with new
> container (more more more ansible).
> 5. <>
>

Almost.

As OpenStack-Ansible treats the LXC containers like hosts (this is why OSA
supports deploying to LXC machine containers and to VM's or normal hosts)
we don't replace containers - we simply deploy the new venv into a new
folder, reconfigure, then restart the service to use the new venv.

To speed things up in large environments we pre-build the venvs on a repo
server, then all hosts or containers grab them from the repo server

The mechanisms we use allow deployers to customise the packages built into
the venvs (you might need an extra driver in the neutron/cinder venvs, for
instance) and allow the OpenStack services to build directly from any git
source (this means you can maintain your own fork with all the fixes you
need, if you want to).

With OpenStack-Ansible you're also not forced to commit to the integrated
build. Each service role is broken out into its own repository, so you're
able to write your own Ansible playbooks to consume the roles which setup
the services in any way that pleases you.

The advantage in our case of using the LXC containers is that if something
ends up broken somehow in the binary packages (this hasn't happened yet in
my experience) you're able to simply blow away the container and rebuild it.

I hope this helps. Feel free to ping me any more questions.

---
Jesse
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [OpenStack-Ansible] Liberty to Mitaka upgrade?

[Jesse Pretorius]

On 20 April 2016 at 04:27, Dale Baley  wrote:

> Is there a document or guide for upgrading from Liberty to Mitaka yet?
>

Hi Dale,

The active work to test and implement any plays to assist with upgrades has
not yet been implemented for Liberty->Mitaka. We hope to do this work by
Newton Milestone-1. If you wish to participate in the process for getting
this done then please join us in #openstack-ansible to compare notes.

The starting point for this work is always to begin by using the same
process as is used for minor upgrades [1] in a test environment and to see
whether anything breaks, then to prepare patches to resolve that. As our
roles and plays already automatically detect and deal with things like DB
migrations and the software version changes this may actually work
perfectly well - it's just that we've not had the chance to really get down
to testing the upgrades with focal attention yet.

[1]
http://docs.openstack.org/developer/openstack-ansible/install-guide/app-minorupgrade.html

Best regards,

Jesse
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [OpenStack-Ansible] Liberty and Ubuntu 16.04

[Jesse Pretorius]

On 6 April 2016 at 13:50, Andreas Vallin  wrote:

> Hi
> I am wondering if there will be possible to run Ubuntu 16.04 LTS with
> Liberty and the openstack-ansible project? Are you planning to "support"
> that?
>

Hi Andreas,

Support for Ubuntu 16.04 LTS is work that will be done in the Newton cycle
and released with Newton.

Backports may be considered for Mitaka if the work is completed by Newton
Milestone-1, but it is very unlikely that it will be backported to Liberty
as that is a stable codebase which already has several production
deployments and we don't want to risk reducing their stability.

We're having a work session at the Summit for the Ubuntu 16.04 work, so if
you can make it we'd be happy to see you join us to plan out the work and
get started on some of the initial patches.

Best regards,

Jesse
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [OpenStack-Ansible] Mitaka release

[Jesse Pretorius]

The OpenStack-Ansible project is pleased to announce the availability of
it's Mitaka release, v13.0.0, and with it comes the following features:

Increased modularity:
The Ansible roles we provide deploy OpenStack services directly from a git
source into python venvs, and deploy their key infrastructure dependencies.
For Mitaka we have broken the roles out into their own repositories in
order to allow deployers to make use of them with their own Ansible
playbooks. This further increases the options available to deployers,
giving even more choice for how to use the tools we provide to suit the
needs of the target environment.

Improved Usability:
We have made great strides in improving documentation for both developers
and deployers in order to improve the usability of OpenStack-Ansible
overall.

Additional services:
OpenStack-Ansible can now deploy Neutron LBaaSv2.
OpenStack-Ansible can now deploy Neutron FWaaS.
OpenStack-Ansible has a new experimental roles for the deployment of
Ironic, Designate, Zaqar, Magnum, Barbican. Each of these roles are still
in their early stages of development and are in varying states of
functional completion. Anyone interested in joining the development process
is welcome to make contact with us through the ML or on IRC in
#openstack-ansible.

Increased test coverage:
While we still have full integration testing on every commit to ensure that
the deployment of OpenStack by OpenStack-Ansible's playbooks really works,
we increased test coverage for the dynamic inventory and individual roles
in order to increase the test coverage, improve code quality, reduce
regressions and to cover more difficult test cases (eg: the major version
upgrade of MariaDB).

Some of the work intended for inclusion in the Mitaka release unfortunately
missed the deadline, so we expect that it will be completed and backported
to Mitaka early in the Newton cycle. This work includes:
 - The inclusion of Ironic in the integrated build.
 - Support for Nuage as a networking provider for Neutron.
 - Support for OVS as an ML2 provider for Neutron.

Generally speaking, it has been exciting to see how our community has grown
in the Mitaka cycle. The activity in the IRC channel has shown that we now
have even more organisations making use of OSA to deploy both private and
public OpenStack clouds.

Looking forward into the Newton cycle we'll be continuing work on Multi-OS
enablement, adding support for Ubuntu 16.04 LTS, taking advantage of
Ansible 2.x, revisiting the dynamic inventory with a view on how
environments are deployed across regions and with Cells v2, and of course
adding support for Liberty->Mitaka upgrades.

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-ansible] Mid Cycle Sprint

[Jesse Pretorius]

Hi everyone,

Thank you all for responding and giving your feedback.

Given that the Ops Mid Cycle [1] is in the UK (15-16 Feb) and AnsibleFest
[2] is also in the UK (18 Feb), and also that everyone seems ok with the UK
as a location for the mid cycle then we shall have it in the UK!

The Ops Mid Cycle organisers have also graciously offered for us to be a
part of the Ops Mid Cycle agenda. I would like as many of us as possible to
play a part in the Ops Mid Cycle - listening, learning and contributing
where possible. As such, I'd like us to limit the number of sessions we
have within the Ops Mid Cycle. Let's have a few open design-style sessions.

Thereafter, as discussed in today's community meeting [3], we can break
down into work sessions where we can work on specific goals. This will
happen on the Wednesday 17 Feb and we will be hosted at the Rackspace
offices in Hayes (West London).

I've setup an Etherpad [4] to gather proposals for Fishbowl Sessions (at
the Ops Mid Cycle) and Work Sessions (for 17 Feb). Please add any sessions
you'd like to facilitate/moderate. If there are sessions there, then please
feel free to add your +1 to show that you'd like to see it.

As discussed in the meeting today [3] I'm on holiday until the New Year.
Please all have a wonderful time over the year's end and I'll see you all
bright-eyed and bushy tailed next year!

Best regards,

Jesse
IRC: odyssey4me

[1] https://etherpad.openstack.org/p/MAN-ops-meetup
[2] http://www.ansible.com/ansiblefest
[3]
http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2015/openstack_ansible_meeting.2015-12-17-16.01.log.html
[4] https://etherpad.openstack.org/p/openstack-ansible-mitaka-midcycle
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mid Cycle Sprint

[Jesse Pretorius]

Hi everyone,

At the Mitaka design summit in Tokyo we had some corridor discussions about
doing a mid-cycle meetup for the purpose of continuing some design
discussions and doing some specific sprint work.

***
I'd like indications of who would like to attend and what
locations/dates/topics/sprints would be of interest to you.
***

For guidance/background I've put some notes together below:

Location

We have contributors, deployers and downstream consumers across the globe
so picking a venue is difficult. Rackspace have facilities in the UK
(Hayes, West London) and in the US (San Antonio) and are happy for us to
make use of them.

Dates
-
Most of the mid-cycles for upstream OpenStack projects are being held in
January. The Operators mid-cycle is on February 15-16.

As I feel that it's important that we're all as involved as possible in
these events, I would suggest that we schedule ours after the Operators
mid-cycle.

It strikes me that it may be useful to do our mid-cycle immediately after
the Ops mid-cycle, and do it in the UK. This may help to optimise travel
for many of us.

Format
--
The format of the summit is really for us to choose, but typically they're
formatted along the lines of something like this:

Day 1: Big group discussions similar in format to sessions at the design
summit.

Day 2: Collaborative code reviews, usually performed on a projector, where
the goal is to merge things that day (if a review needs more than a single
iteration, we skip it. If a review needs small revisions, we do them on the
spot).

Day 3: Small group / pair programming.

Topics
--
Some topics/sprints that come to mind that we could explore/do are:
 - Install Guide Documentation Improvement [1]
 - Development Documentation Improvement (best practises, testing, how to
develop a new role, etc)
 - Upgrade Framework [2]
 - Multi-OS Support [3]

[1] https://etherpad.openstack.org/p/oa-install-docs
[2] https://etherpad.openstack.org/p/openstack-ansible-upgrade-framework
[3] https://etherpad.openstack.org/p/openstack-ansible-multi-os-support

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Keystone audit logs with haproxy

[Jesse Pretorius]

On 25 November 2015 at 05:40, Ajay Kalambur (akalambu) 
wrote:

> Hi
> Have a deployment where keystone sits behind a ha proxy node. Now
> authentication requests are made to a vip. Problem is when there is an
> authentication failure we cannot track the remote ip that failed login as
> all authentication failures show the VIP ip since ha proxy fwds the request
> to a backend keystone server
>
> How do we use a load balancer like ha proxy and also track the remote
> failed ip for authentication failures
> We get all authentication failures showing up with remote ip as vip ip
>

It's probably best to enable the forwardfor option [1] and ensure that your
Keystone logs record that information. This is relatively trivial if
Keystone is using Apache/wsgi, but I can't recall whether the eventlet
server logs the info.

[1]
https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20forwardfor
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ansible-OpenStack:cannot ssh into the containers

[Jesse Pretorius]

On 24 November 2015 at 17:02, Achi Hamza  wrote:

> Hi,
>
> I am trying to deploy openstack using openstack-ansible project. I am
> using Liberty release (tag *12.0.0*).
> The problem i have is that Ansible is not able to SSH into the containers:
>
> *TASK: [Wait for ssh to be available]
> ** *
> failed: [infra01_neutron_server_container-4d09aa86 -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.216:22
> failed: [infra01_utility_container-4da899df -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.237.186:22
> failed: [infra01_neutron_agents_container-bd7aded5 -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.237.90:22
> failed: [infra01_memcached_container-12186fcf -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.177:22
> failed: [infra01_rabbit_mq_container-7103e2f2 -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.178:22
> failed: [infra01_galera_container-b691e13b -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.60:22
> failed: [infra01_nova_conductor_container-98abf1df -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.8:22
> failed: [infra01_horizon_container-5756dc20 -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.47:22
> failed: [infra01_nova_cert_container-17a8689a -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> .
> .
> .
>
>
> I can ping the containers:
>
>
> *root@infra01:~# ping infra01_galera_container-b691e13bPING
> infra01_galera_container-b691e13b (172.29.236.60) 56(84) bytes of data.*
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=1 ttl=64 time=0.103 ms
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=2 ttl=64 time=0.086 ms
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=3 ttl=64 time=0.110 ms
>

Hi Amza,

My apologies for only getting back to you right now.

We've been seeing similar SSH connectivity issues in gate checks and have
what we believe is a solution in a review in flight [1]. The patch is
easily implemented in your environment.

Failing that I have a few suggestions:

1 - Implement using at least the tag 12.0.1 as we fixed some pretty crucial
release bugs between 12.0.0 and 12.0.1. You can also implement from the
head of the Liberty branch.
2 - If your host(s) are a little slow you may wish to implement an
increased delay in /etc/openstack_deploy/user_variables.yml using the
variable 'ssh_delay'.
3 - If possible, try and catch us in the Freenode IRC channel
#openstack-ansible - there're often people there who can help you
troubleshoot.

[1] https://review.openstack.org/249942
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [keystone] Request for Feedback: Online database migrations

[Jesse Pretorius]

Hi everyone,

As we all know, upgrades are hard. Part of the problem relates to down time
due to offline database migrations.

The Keystone team has put together a spec [1] and is seeking feedback for
implementing online schema migrations.

Please weigh in, as operators, on the spec presented.

[1] https://review.openstack.org/245186

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Spec Review Time :)

[Jesse Pretorius]

On 20 November 2015 at 03:35, Tom Fifield  wrote:

>
>
> https://review.openstack.org/#/q/status:open+AND+project:openstack/nova-specs,n,z
>
>
 FYI, it's possible to use some regex in your query - for example this will
give you all currently open specs in all specs repositories:

https://review.openstack.org/#/q/project:%255Eopenstack/.*-specs+status:open,n,z
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Documentation update planning

[Jesse Pretorius]

Hi everyone,

In the community meeting [1] this week we'll be having a facilitated
discussion with regards to documentation improvements for OpenStack-Ansible.

If you have any feedback you'd like considered, would like to be part of
the conversation, or are keen to contribute then please add to the etherpad
[2], respond to this email and ideally be in the meeting. :)

[1]
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting
[2] https://etherpad.openstack.org/p/oa-install-docs

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Fedora/CentOS/other Support

[Jesse Pretorius]

Hi everyone,

There has been interest expressed from various corners to support more than
just Ubuntu in OpenStack-Ansible. While the project is very happy to work
on this, no-one has yet stepped up to take a lead role in the execution of
this work.

The current community has done some research into appropriate patterns to
use and has a general idea of how to do it - but in order to actually
execute there need to be enough people who commit to actually maintaining
the work once it's done. We don't want to carry the extra code if we don't
also pick up extra contributors to maintain the code.

If you're interested in seeing this become a reality and are prepared to
commit some time to making it happen, the project will be happy to work
with you to achieve it.

In this week's community meeting [1] we'll initiate a discussion on the
topic with the purpose of determining who's on board with the work.

In the following week's community meeting we will have a work breakdown
discussion where we'll work on filling out an etherpad [2] with what we
need to get done and who will be doing it. Naturally the etherpad can be
populated already by anyone who's interested. If you're actually happy to
volunteer to assist with the work, please feel free to add a note to that
effect on the etherpad.

[1]
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting
[2] https://etherpad.openstack.org/p/openstack-ansible-multi-os-support

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit Summary

[Jesse Pretorius]

Hi everyone,

I've put together some thoughts based on the Mitaka Summit from the
OpenStack-Ansible point of view:
http://odyssey4me.github.io/openstack/ansible/mitaka/summit/2015/11/17/mitaka-summit.html

Please feel free to ping me with any feedback!

Thanks,

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit Schedule

[Jesse Pretorius]

Hi everyone,

I've added the final details for the summit sessions to the etherpad:
https://etherpad.openstack.org/p/openstack-ansible-mitaka-summit

Our sessions are open to anyone with an interest in deploying OpenStack
with Ansible.

Note that our two primary topics for discussion are:

Image-based Deployments:
We ask the questions:
 - What benefits are there to using image-based deployments?
 - How are people doing it today?

Production-ready Upgrades
We ask the questions:
 - What does it take to do upgrades in a production OpenStack environment?
 - How can we orchestrate it successfully?

We are very interested in gathering feedback from a broad group of
OpenStack operators based on the above topics and would love to have your
feedback!

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit sessions

[Jesse Pretorius]

Hi everyone,

As the Mitaka summit draws nearer, I'd like a broad view of what people
would like to discuss at the summit. This can include anyone's input!
Obviously our space and time will be limited, so any sessions that we don't
get to formally do at the summit we'll either try to do informally during
the summit (in a workgroup session or something like that), or we'll try to
engage at an alternative date (like a midcycle).

Some sessions have already been proposed - please feel free to add to the
list. Try to use the same format so that we know whether you're
facilitating or whether you want someone else to, what preparation
attendees would need, etc.

https://etherpad.openstack.org/p/openstack-ansible-mitaka-summit

It's probably a good idea to review the currently registered blueprints [0]
and specs [1] before adding any proposed sessions - just to make sure that
you're not covering something that's already on the go. For any current
blueprints/specs we're certainly open to more discussion, but it'd be great
to see that discussion happen in the spec reviews. :)

[0] https://blueprints.launchpad.net/openstack-ansible
[1]
https://review.openstack.org/#/q/project:openstack/openstack-ansible-specs+status:open,n,z

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Release: Kilo 11.1.0

[Jesse Pretorius]

Hi everyone,

The openstack-ansible community is pleased to announce our release of the
Kilo 11.1.0 milestone.

This release fixes 57 bugs and implements 4 major blueprints, including:

* Keystone as a Federated Identity Provider
* Keystone as a Federated Service Provider
* Horizon Federation Single Sign-On
* Multi-Region Swift Clustering
* Ceilometer

For the full details please see launchpad [1].

For 11.2.0 [2] we'll be optionally incorporating Ceph as a back-end for
Glance, Cinder and Nova. Between now and then we expect to release at least
two hotfix releases in our usual cadence of once every two weeks.

If you're looking to kick the tyres to see how we do things or just want to
try OpenStack out and have a test system up and running within around 60
minutes, then try out our all-in-one deployment [3] which is used for
development/testing.

We welcome everyone to give it a try, participate in reviews [4] and get
involved! [5] If you have any questions, contact us in #openstack-ansible.

Best regards,

Jesse
IRC: odyssey4me

[1] https://launchpad.net/openstack-ansible/+milestone/11.1.0
[2] https://launchpad.net/openstack-ansible/+milestone/11.2.0
[3]
https://github.com/stackforge/os-ansible-deployment/blob/kilo/development-stack.rst
[4]
https://review.openstack.org/#/q/project:stackforge/os-ansible-deployment,n,z
[5]
https://github.com/stackforge/os-ansible-deployment/blob/master/CONTRIBUTING.rst
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [os-ansible-deployment][openstack-ansible] Release review/bug list for tomorrow's meeting

[Jesse Pretorius]

On 29 July 2015 at 22:04, Jesse Pretorius  wrote:

> The following reviews are in-flight and are important for the upcoming
> releases, and therefore there is a need for more reviews and in some cases
> backports once the master patches have landed:
> https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z
>
> The upcoming releases (this weekend) are:
>
> Kilo: https://launchpad.net/openstack-ansible/+milestone/11.1.0
>
Unfortunately we have had to postpone the release for 11.1.0 until Monday
as we have missed our deadline for several key patches merging.

The entire available core team agreed that this would be best in the
#openstack-ansible channel. It'd be better for us all to have a bit more
time to review the final list of patches, especially considering several of
them are quite substantial.

We'll revisit this on Monday - please continue to review if you have a
moment to do so before then.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] [os-ansible-deployment] Kilo -> Liberty Upgrade Problems

[Jesse Pretorius]

I'm adding openstack-operators too as this is a discussion that I think it
would be useful to have their input for.

On 30 July 2015 at 19:18, Ian Cordasco  wrote:

> Hey all,
>
> As you may have seen elsewhere on openstack-dev, OpenStack is changing the
> versioning for the service projects. This means our previous upgrade
> solution will not continue to work. For context, one of our project's
> goals is to have in-place upgrades be a reality. Previously, using our
> repository (package) mirror doing:
>
> # pip install -U {{servicename}}
>
> Was perfectly fine. The problem will now occur that 2015.1.0 (kilo) is
> more recent than any of the new service version numbers (as far as pip is
> concerned). This would be resolved if the release management team for
> OpenStack properly used an epoch to indicate that the versioning scheme is
> fundamentally different and something like Glance 8.0.0 should sort after
> Glance 2015.1.0, but they won't (for reasons that you can read in earlier
> threads on this list).
>

Yes. This is going to cause quite a few headaches I'm sure.


> So, in order to satisfy the goal of in-place upgrades, we need a way
> around this. Currently, we use a tool called 'yaprt' to build wheels and
> repository indices for our project. This tool can (and currently does)
> create reports of the built files and exports those reports as JSON. We
> can use this to know the version generated for a service and then instead
> do:
>
> # pip install {{servicename}}=={{yaprt_generated_version}}
>
> This will force pip to ignore the fact that the existing (kilo)
> installation is actually supposed to sort as more recent because you're
> telling it to install a very specific version. This will likely need to be
> our upgrade path going forward unless we also require operators to clear
> out their existing repository mirror of packages with Kilo versions (and
> then we can go back to relying on pip's version sorting semantics to do
> pip install -U {{servicename}}).
>

So what would the resulting version be? Would the python wheel be 2016.x.x
or would the file simply be named that so that we're sitting with this
workaround for only one cycle and future cycles can revert to the previous
process?


> This is, at the moment, the seemingly simplest way to work around the
> brokenness that is the upstream versioning change.
>
> If you can think of a different way of approaching this, we'd love to hear
> it. If not, Kevin or myself will probably start working on this approach
> in a week or two so it's ready for when Liberty is actually released and
> we can start testing upgrades from the kilo branch to master (which is
> currently tracking liberty).
>

This is not a fun problem to have to solve, but it seems a reasonable
solution. Whatever we do I'd prefer to see it as a solution that we only
have to carry for one cycle so that all versioning matches upstream from
then on. If that's not possible then some sort of epoch-style workaround
like this may just be something we have to live with.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [os-ansible-deployment][openstack-ansible] Release review/bug list for tomorrow's meeting

[Jesse Pretorius]

Hi everyone,


We need some support for reviews and bug updates, ideally before the next
meeting at 16:00 UTC tomorrow as per
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting


The following reviews are in-flight and are important for the upcoming
releases, and therefore there is a need for more reviews and in some cases
backports once the master patches have landed:
https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z


The upcoming releases (this weekend) are:

Kilo: https://launchpad.net/openstack-ansible/+milestone/11.1.0

Juno: https://launchpad.net/openstack-ansible/+milestone/10.1.11


I’d appreciate it if everyone could take a look at the reviews – some of
which need to be rebased/changed - and the bugs that are not yet ‘In
Progress’ and decide whether the solutions for the bugs will make it or not
in time for the release.

We’ll discuss everything in the list in the meeting and decide on the best
course of action.

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OSAD for RHEL

[Jesse Pretorius]

On 9 July 2015 at 05:54, John Dewey  wrote:

> IMO - registering the systems with subscription manager or pointing to in
> house yum repos should be included as part of system bootstrapping, and not
> a part of OSAD.  OSAD should simply install the specific packages for the
> alternate distro.
>

Agreed, trying to cater for all things that everyone wants in their
bootstrapping is a rabbit hole best not ventured into as it'll bloat the
project considerably.


> Might also be a good time to abstract the system packaging module into a
> higher level one which handles `yum` or `apt` behind the scenes.We can
> then manage the list of packages per distro[1].  Throwing this out as an
> idea vs copy-paste every apt with a yum section.
>

Ansible appears to be building this abstraction already for v2 [1], but has
a means to do this in an alternative way [2].

[1]
https://github.com/ansible/ansible-modules-core/blob/devel/packaging/os/package.py
[2] http://serverfault.com/a/697736
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [os-ansible-deployment] Feedback on Keystone Federation Spec

[Jesse Pretorius]

Hi everyone,

There was quite a bit of fanfare around the new federation features in
OpenStack Kilo.

In the os-ansible-deployment/openstack-ansible project we've been putting
together a view on how to implement federation with as little complexity as
possible.

We've been working on some prototype code which can be seen by looking at
the patches on the blueprint whiteboard [1] and have also prepared a spec
for the implementation [2].

We'd like to get some feedback from the broader community - from deployers
interested in using the feature and from developers/deployers who've worked
with federation. The feedback we'd like to see is both in terms of the spec
and the prototype code (which is changing quite frequently as we figure out
the bits and pieces).

The follow-on to this work will be to specifically add the capability to
make use of an ADFS IdP for a Keystone SP. This work will be linked to
another blueprint [3] which is still a work in progress.

I look forward to the review feedback!

[1]
https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-federation
[2] https://review.openstack.org/194147
[3]
https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-sp-adfs-idp

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Collaborative Logstash filter development

[Jesse Pretorius]

Hi everyone,

During the summit at Vancouver and at time both prior and after there has
been expressed interest in more collaborative development of Logstash
filters in an OpenStack environment.

Current examples which I'm aware of include:
 - https://github.com/godaddy/openstack-logstash
 - https://github.com/osops/tools-logging
 - http://docs.openstack.org/infra/system-config/logstash.html
 -
https://github.com/rcbops/rpc-extras/tree/master/rpcd/playbooks/roles/logstash/templates

There was some discussion in the Operators track at the design summit about
setting up a collaborative location, but to my knowledge there has been no
firm decision about it.

It is plausible that we could put together a repository which is purely
focused on the filters (not the input/output sections) and which could be
consumed by downstream users (eg: openstack-infra). To do this, however,
will require a group of interested parties to agree on how this will be
done. Considering that it will likely be accepted into the Big Tent it will
likely also require a PTL and core reviewing team.

We (Rackspace Private Cloud) are happy to seed initial configurations and
work with a team of other parties to put together a standardised set of
filters which are usable and useful. When agreed that the filters are ready
we will also work with openstack-infra to implement the filters into
logstash.openstack.org.

How would the Operator community like to proceed with this? Shall we
arrange some sort of IRC discussion to figure out how best to organise this
and to identify next steps?

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] 100% CPU and hangs if syslog is restarted

[Jesse Pretorius]

On 28 May 2015 at 22:45, Nick Jones  wrote:
>
> On 28 May 2015, at 19:10, Tim Bell  wrote:
>
> Using UDP is a great workaround but it does not feel like a fix... can't
> the daemons realise that the syslog socket is not alive and reconnect.
> Given it affects most of the OpenStack projects, a fix inside one of the
> oslo logging libraries (if the error can be detected there) would be great.
>
> We too have been bitten hard by this issue in the past - way before Juno -
> when using rsyslog logging to a remote target (i.e Logstash).  We
> eventually went down the route of using log-courier [1] on x86 and beaver
> [2] on ARM (due to the lack of support for Go).
>
> Both have worked out well for us - if you’re using Logstash it might be
> worth looking into either of these as a solution instead of switching to
> UDP and hoping that you don’t lose any messages that you might care about.
>
> [1] https://github.com/driskell/log-courier
> [2] https://github.com/josegonzalez/python-beaver
>
>
We also tried using direct syslog logging from OpenStack services and hit
the same issues. We've opted to rather let the services log natively, then
have python-beaver forward the logs. This scales well, provides a
consistent log forwarding method and also allows us to do multi-line event
consolidation at the source (necessary because logstash doesn't scale well
if you try to do it there). You can find our work, use it, derive from it
and contribute feedback here: https://github.com/rcbops/rpc-extras/pull/123
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenSource ELK configurations

[Jesse Pretorius]

On 27 May 2015 at 20:58, Joe Gordon  wrote:

>
> On Wed, May 27, 2015 at 12:27 PM, Joseph Bajin 
> wrote:
>
>> I think those from the Infra Project are a great start, but I do think
>> they are missing a lot.
>>
>
> I am sure they are, and we would love contributions to make them better.
> The filters from the infra project are used every day by developers use to
> hunt down issues. And the more similar the developer and operator
> experience is here the easier it will be for the two groups to talk about
> the same problems.
>

As part of development for Rackspace Private Cloud we've done a ton of work
in this space and are happy to contribute the work upstream. The current
work is exposed in https://github.com/rcbops/rpc-extras/pull/123 and is
available for anyone to review/comment/consume.


> Instead of breaking down the message into parts, it just breaks down the
>> type (INFO, DEBUG, etc) and then makes the rest of the message greedy.
>> That doesn't help searching or graphing or anything like that (or at least
>> makes it more difficult over time). At least that is what I have seen.
>>
>
> One of the reasons for this is because most of the log messages are very
> unstructured. But I have a hunch we can do better then what we have today.
>

We've done quite a bit of this. We extract transaction ID's for each
project we deploy (including swift), response times to API queries, byte
sizes, etc.


> We are using a little bit tweaked version of the godaddy's scripts.  That
>> seems to give us a good amount of detail that we can search and filter on.
>> I'll see if I can post my versions of them to the repo.
>>
>
We derived some stuff from https://github.com/godaddy/openstack-logstash
and other stuff from https://github.com/osops/tools-logging - I think it's
time that we find a common place to consolidate and standardise.
openstack-infra would appear to be a good place to do this.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Specifying multiple tenants for aggregate_multitenancy_isolation_filter

[Jesse Pretorius]

On 28 January 2015 at 06:19, Tim Bell  wrote:

>  +1 for commas. Configuration files with JSON is OK but commas for CLIs.
>

+1 for commas from an operator perspective
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Migrating instances according to load on host

[Jesse Pretorius]

The closest thing I've seen to that has been the IBM Resource Scheduler
(closed source) and this:

https://github.com/BMDan/OpenStack-Hypervisor-Balance

I've not tried it myself.

There have been discussions around doing this, but I've not seen code
submissions yet.
On Tue, 16 Dec 2014 at 20:19 Sławek Kapłoński  wrote:

> Hello,
>
> Thanks for show me some direction what I should look for :)
> I search blueprints of nova with 'drs' and I found somethink like this:
> https://blueprints.launchpad.net/nova/+spec/resource-optimization-service
> It would be nice to have it in nova :)
>
> ---
> Best regards
> Slawek Kaplonski
> sla...@kaplonski.pl
>
> Dnia wtorek, 16 grudnia 2014 12:05:41 Abel Lopez pisze:
> > Sounds like you're describing VMWare DRS.
> > It's a pretty neat feature that aims to reduce outages during maintenance
> > and also reduce the impact of noisy neighbor.
> > To my knowledge, we don't have that today, but I haven't been following
> the
> > wishlist for nova
> > > On Dec 16, 2014, at 12:01 PM, Sławek Kapłoński 
> > > wrote:
> > >
> > > Hello,
> > >
> > > I'm wondering is ther some solution (maybe separate project or some
> > > plugin)
> > > which allow to automatically live-migrate (or migrate when instances
> are
> > > stopped) instances between hosts according to load which instance
> generate
> > > on host. For example if there are 10 instances on host and all of then
> > > use lot cpu or ram some of them should be migrated to different (free)
> > > host. When there are instances which are do not heavy load - then
> should
> > > be migrated to one host where will be for example 15 such instances. Do
> > > you know about something like that for openstack?
> > >
> > > ---
> > > Best regards
> > > Slawek Kaplonski
> > > slawek@kaplonski.pl___
> > > OpenStack-operators mailing list
> > > OpenStack-operators@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack-operators___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Storage Backend: Sheepdog, Ceph, or GlusterFS

[Jesse Pretorius]

On 6 November 2014 13:01, Hossein Zabolzadeh  wrote:

> Thanks for your opinion. But I am looking for the real difference
> between them...
> - Which one is better support in openstack?
> - Which one provides better unified storage backend for all openstack
> storage controllers(cinder, swift and glance)?
>

I don't think that Gluster or Sheepdog provide a storage back-end capable
of providing block storage (cinder) and object storage (swift) back-ends.
Only Ceph provides a properly unified back-end for both. Ceph has also been
supported for cinder for over two years - it's very mature. The Rados
Gateway (Object Storage/Swift Interface) is much more recent, but I have
yet to see problems with it - as long as you do acknowledge that it is not
Swift.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Architecture Opinions

[Jesse Pretorius]

I'd like to clarify a few things, specifically related to Ceph usage, in
less of a rushed response. :)

Note - my production experience has only been with Ceph Dumpling. Plenty of
great patches which resolve many of the issues I've experienced have
landed, so YMMV.

On 30 September 2014 15:06, Jesse Pretorius 
wrote:

> I would recommend ensuring that:
>
> 1) ceph-mon's and ceph-osd's are not hosted on the same server - they both
> demand plenty of cpu cycles
>

The ceph-mon will generally not use much CPU. If a whole chassis is lost,
you'll see it spike heavily, but it'll drop off again after the rebuild is
complete. I would still recommend keeping at least one ceph-mon on a host
that isn't hosting OSD's. The mons are where all clients get the data
location details from, so at least one really needs to be available no
matter what happens.

And, FYI, I would definitely recommend implementing separate networks for
client access and the storage back-end. This can allow you to ensure that
your storage replication traffic is separated and you can tune the QoS for
each differently.


> 5) instance storage on ceph doesn't work very well if you're trying to use
> the kernel module or cephfs - make sure you're using ceph volumes as the
> underlying storage (I believe this has been patched in for Juno)
>

cephfs, certainly in Dumpling, is not production ready - our experiment
with using it in production was quickly rolled back when one of the client
servers lost connection to the ceph-mds for some reason and the storage on
it became inaccessible. The client connection to the mds in Dumpling isn't
as resilient as the client connection for the block device.

By 'use the kernel module' I mean create an image and mounting it to the
server through the ceph block device kernel module, then building a file
system on it and using it like you would any network-based storage.
We found that when using one image as shared storage between servers,
updates from one server wasn't always visible quickly enough (within a
minute) on the other server. If you choose to use a single image per
server, then only mount server2's image on server1 in a disaster recovery
situation then it should be just fine.
We did find that mounting a file system using the kernel module would tend
to cause a kernel panic when trying to disconnect the storage. Note that
there have been several improvements in the revisions after Dumpling,
including some bug fixes for issues that look similar to what we
experienced.

By "make sure you're using ceph volumes as the underlying storage" I meant
that each instance root disk should be stored as its own Ceph Image in a
storage pool. This can be facilitated directly from nova by using
'images_type=rbd' in nova.conf which became available in OpenStack Havana.
Support for using RBD for Ephemeral disks as well finally landed in Juno
(see https://bugs.launchpad.net/nova/+bug/1226351), as did support for
copy-on-write cloning (see
https://blueprints.launchpad.net/nova/+spec/rbd-clone-image-handler) which
rounds out the feature set for using an RBD back-end quite nicely. :)
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Architecture Opinions

[Jesse Pretorius]

On 29 September 2014 23:28, Erik McCormick 
wrote:

> The architecture I intended to deploy was this:
>
> 2 HAProxy nodes to load balance active / active APIs and Horizon
> 2 HAProxy nodes to load balance a Galera Mysql cluster
> 2 Control nodes with all API services
> 3 Galera / MySQL nodes
> 3 MongoDB nodes running replica sets for Ceilometer
> 2 Neutron Nodes running Active / Passive L3/DHCP/LBaaS agents (hopefully
> active active L3 in Juno)
> 3 Ceph-Mon nodes
> 3 Ceph-OSD nodes (hosting Cinder, Glance, and potentially instance storage)
> X number of compute nodes depending on the requirement
>
> The reference architectures I'm seeing out of Redhat and Mirantis among
> others seem to like putting all of the above eggs except the Storage into 3
> identical baskets. This just feels bad and painful to me and like it would
> lead to very badly performing everything. Am I totally just stuck in the
> past with how I'm thinking of setting all this up?
>

Depending on the size of your deployment, it's safe enough to combine
almost everything. It does also depend on the hardware you have available.

I would recommend ensuring that:

1) ceph-mon's and ceph-osd's are not hosted on the same server - they both
demand plenty of cpu cycles
2) ceilometer's storage into mongodb is demanding, so it's best to ensure
that mongodb is on different storage to galera
3) neutron's l3 agent active/passive configuration works just fine - it's
probably best to keep them on their own servers in order to handle the
appropriate throughput required
4) ceph-osd's should not run on your compute or OpenStack controller nodes
- kvm/osd contention on the cpu will cause all sort of odd issues
5) instance storage on ceph doesn't work very well if you're trying to use
the kernel module or cephfs - make sure you're using ceph volumes as the
underlying storage (I believe this has been patched in for Juno)
6) neutron has the ability to include proper ha for dhcp agents - we
currently (in a Grizzly environment) have a script that creates a DHCP
service on every network, but I believe that beyond Grizzly the HA story
has been sorted out

HTH,

Jesse
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators