Re: [Openstack-operators] How to apply the bugfix for keystone pyldap (#1736241)

[Jesse Pretorius]

> From: David Young 
> >
> Date: Friday, December 8, 2017 at 10:34 PM
> To: 
> "openstack-operators@lists.openstack.org"
>  
> >
> Subject: [Openstack-operators] How to apply the bugfix for keystone pyldap 
> (#1736241)


> This bug was found in the wild, and is currently affecting my OSA playbook, 
> which needs to build a repo container:
> https://bugs.launchpad.net/openstack-ansible/+bug/1736241

> I see it's in process of being fixed, I'm curious re how I'll apply it to my 
> environment.

> My playbooks etc are 15.1.8 - do I need to do a minor release update to 
> receive the fix, or will it "just work" the next time I build a repo 
> container?

Hi David,

You’ll need to perform a minor upgrade to 15.1.13 in order to get it. See [1] 
for more details regarding how to do that.

[1] 
https://docs.openstack.org/openstack-ansible/ocata/upgrade-guide/minor-upgrade.html#execute-a-minor-version-upgrade



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] (openstack-ansible) Mitaka 13.3.12 failing to report correct resource usage

[Jesse Pretorius]

From: "Craft, Mike" 
Date: Thursday, March 2, 2017 at 7:50 PM
To: "openstack-operators@lists.openstack.org" 

Subject: [Openstack-operators] (openstack-ansible) Mitaka 13.3.12 failing to 
report correct resource usage

Hello,

Did an upgrade from liberty to mitaka and we are experiencing an issue where 
the audit of locally available compute resource usage is not seeing instances 
that were deployed prior to the upgrade. Has anyone experienced this before? We 
hard power cycled all the vms (and hypervisor) to make sure they are running on 
the latest nova version. This is impacting scheduling of resources/quota usage 
as you could imagine ☺

If you look in the nova database, can you see those instances there? From the 
look of things, your hypervisor has no instances allocated to it in its 
database. I hope that you have a backup! ☺



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OSA Newton Release Deployment in test environment

[Jesse Pretorius]

LXC will setup eth0 in the container which will NAT through the host to access 
the internet.

Clearly there’s something wrong with that traffic path, so step through it like 
you would with any environment by validating that it is able to reach its 
gateway (host), then next hop, etc.

From: Amit Kumar 
Date: Monday, February 6, 2017 at 8:17 AM
To: gustavo panizzo 
Cc: "openstack-operators@lists.openstack.org" 

Subject: Re: [Openstack-operators] OSA Newton Release Deployment in test 
environment

Error is because LXC repo container on infra node is not having connectivity to 
internet and is unable to download these packages. Thing is that why containers 
are not having connectivity to internet whereas infra node is having 
connectivity to internet.

Any idea if this is a general behavior that LXC containers are not able to 
reach internet?

On Mon, Feb 6, 2017 at 1:39 PM, gustavo panizzo 
> wrote:
On Sat, Feb 04, 2017 at 11:52:10PM +0530, Amit Kumar wrote:
> Hi All,
>
> used.\nWARNING: The following packages cannot be authenticated!\n

dpkg/apt are telling you what the problem is, go to the server and fix
apt


> python-pkg-resources python-setuptools\n", "stdout_lines": ["Reading
> package lists...", "Building dependency tree...", "Reading state
> information...", "The following extra packages will be installed:", "
> python-pkg-resources", "Suggested packages:", " python-openssl-doc
> python-openssl-dbg python-distribute", " python-distribute-doc doc-base",
> "The following NEW packages will be installed:", " python-openssl
> python-pkg-resources python-pyasn1 python-setuptools", "0 upgraded, 4 newly
> installed, 0 to remove and 4 not upgraded.", "Need to get 418 kB of
> archives.", "After this operation, 1801 kB of additional disk space will be

> used.", "WARNING: The following packages cannot be authenticated!", "
> python-pkg-resources python-setuptools"]}
same error again


--
1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333

keybase: https://keybase.io/gfa



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

I *think* you’d have to set the following in your 
/etc/openstack_deploy/user_vriables.yml file:

openstack_service_publicuri_proto: http
openstack_external_ssl: false
haproxy_ssl: false

I might be missing more, but that should get you started.

From: Achi Hamza <h16m...@gmail.com>
Date: Friday, November 18, 2016 at 1:56 PM
To: Jesse Pretorius <jesse.pretor...@rackspace.co.uk>, 
"OpenStack-operators@lists.openstack.org" 
<OpenStack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse.
So should i set the parameter haproxy_ssl to false in the default folder of the 
haproxy_server role or somewhere else ?

On 18 November 2016 at 13:43, Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>> wrote:
Ah, then that’s the cause. You can’t have both external and internal addresses 
be the same unless you disable SSL for public endpoints.

From: Achi Hamza <h16m...@gmail.com<mailto:h16m...@gmail.com>>
Date: Friday, November 18, 2016 at 7:02 AM
To: Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>>, 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
<openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

No Jesse, you got me wrong. My external_lv_vip_address and 
internal_vip_lb_address are the same (172.16.1.2), which is also the IP address 
of node01 on which haproxy is running.

This is how it looks on my openstack_user_config.yml file:
global_overrides:
  internal_lb_vip_address: 172.16.1.2
  external_lb_vip_address: 172.16.1.2
  management_bridge: "br-mgmt"


On 17 November 2016 at 20:24, Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>> wrote:
Hmm, that’s odd – if you configured your external_lv_vip_address and 
internal_vip_lb_address to be different then that should not be happening, 
because SSL is implemented on the external VIP. We do not support the use of 
both SSL and non-SSL on the same IP address as that is not something that can 
be done (share the same port, but use HTTPS internally and HTTP externally).

Are you sure that both addresses in your configuration are different?

From: Achi Hamza <h16m...@gmail.com<mailto:h16m...@gmail.com>>
Date: Thursday, November 17, 2016 at 5:58 PM
To: Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>>, 
"klindg...@godaddy.com<mailto:klindg...@godaddy.com>" 
<klindg...@godaddy.com<mailto:klindg...@godaddy.com>>, 
"OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>"
 
<OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse and Lindgren,

Thank you both for the responses. think i figured out the root cause of the 
problem, which is SSL. But first the answers of your questions:
Are you deploying haproxy onto the same host as the repo container, or a 
different host?
Yes, it is on the same host.

Have you bound the VIP address manually?
No, through openstack-ansible playbooks.

Is the VIP address shared in some way – ie is it used for the host and haproxy?
Yes, it is used for the host and haproxy (keepalived is disabled).

After tcpdumping ( tcpdump   -i any port 8181 -n) i found that the packets from 
galera conatiner are not forwarded to repo container through the VIP, and the 
haproxy file is full with this error message:
root@node01:~# tail /var/log/haproxy.log
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38050<http://172.16.1.94:38050> [17/Nov/2016:18:02:16.786] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38052<http://172.16.1.94:38052> [17/Nov/2016:18:02:16.791] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38054<http://172.16.1.94:38054> [17/Nov/2016:18:02:16.795] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38056<http://172.16.1.94:38056> [17/Nov/2016:18:02:16.800] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38058<http://172.16.1.94:38058> [17/Nov/2016:18:02:16.805] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38060<http://172.16.1.94:38060> [17/Nov/2016:18:02:16.809] 
repo_all-front-1/1: SSL handshake failure
Nov 17 18:02:16 localhost haproxy[30180]: 
172.16.1.94:38062<http://172.16.1.94:38062> [17/No

Re: [Openstack-operators] feedback on pymysql

[Jesse Pretorius]

 From: Matt Fischer 

As a part of our upgrades to Newton we are transitioning our services to use 
pymysql rather than the deprecated MySQL-Python [1]. I believe pymsql has been 
the default in devstack and the gate for sometime now and that MySQL-Python is 
essentially untested and not updated, hence our desire to switch.

devstack is one thing, but I'm curious if anyone has experience operating in 
production with this, especially if there are issues. I've not seen anything in 
testing but if anyone else has I'd love to know positive or negative.

[1] https://wiki.openstack.org/wiki/PyMySQL_evaluation#MySQL-Connector-Python

As a data point, OpenStack-Ansible switched to using pymysql by default for 
Liberty. We’ve had no negative feedback from any deployers using OSA as far as 
I know.


Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

From: Achi Hamza <h16m...@gmail.com>
Date: Thursday, November 17, 2016 at 1:57 PM
To: Jesse Pretorius <jesse.pretor...@rackspace.co.uk>, 
"OpenStack-operators@lists.openstack.org" 
<OpenStack-operators@lists.openstack.org>
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse, but these iptables rules are just applied on the deployment 
node not the host nodes. do i have to omit these rules even on the deployment 
node ?

Thank you

Ah, then that’s a red herring. As long as your hosts can reach the internet 
through it, then you’re good on that front.

Let’s go back to verifying access to the repo – try checking access from the 
repo server to itself:

ansible repo_all -m uri -a "url=http://localhost:8181/os-releases/;

or

ansible repo_all –m shell –a "curl http://localhost:8181/os-releases/;



Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

Please continue to make use of the list instead of going off list. The list 
archives are a reference point for others who may find themselves in the same 
situation.

Right, so the service is up. The load balancer is up, so we’re down to 
networking issues or perhaps something a little more obscure.

Can you confirm whether you’re able to reach the repo contents from each of the 
following points: repo container, repo container host, deployment host

ie

ansible repo_all -m shell -a "curl 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt;
ansible hosts -m shell -a "curl 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt;
curl http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt

J

From: Achi Hamza <h16m...@gmail.com>
Date: Thursday, November 17, 2016 at 10:59 AM
To: Jesse Pretorius <jesse.pretor...@rackspace.co.uk>
Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Thank you Jesse.

here are the output of the 2 commands:

nginx server is UP:
root@maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a 
"service nginx status"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e 
@/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: 
enabled)
   Active: active (running) since Wed 2016-11-16 13:12:57 UTC; 21h ago
 Main PID: 2729 (nginx)
Tasks: 3
   Memory: 956.0K
  CPU: 5.832s
   CGroup: /system.slice/nginx.service
   ├─2729 nginx: master process /usr/sbin/nginx -g daemon on; 
master_process on
   ├─2730 nginx: worker process
   └─2731 nginx: worker process

Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Starting A high 
performance web server and a reverse proxy server...
Nov 16 13:12:57 node01-repo-container-82b4e1f6 systemd[1]: Started A high 
performance web server and a reverse proxy server.

the repo-all* is also UP:
root@node01:~# hatop -s /var/run/haproxy.stat
>>> rabbitmq_mgmt-back
..ner-2148a5b81 DOWN   L4CON 1  000 
0 0 0 0
BACKEND   0 DOWN 0  000 
0 0   410 0

>>> repo_all-front-1
FRONTEND  0 OPEN 0  000 
0 1  4096 0

>>> repo_all-back
..ner-82b4e1f61 UP L7OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

>>> repo_cache-front-1
FRONTEND  0 OPEN 0  000 
0 1  4096 0

>>> repo_cache-back
..ner-82b4e1f61 UP L7OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

>>> repo_git-front-1
FRONTEND  0 OPEN 0  000 
0 0  4096 0

>>> repo_git-back
..ner-82b4e1f61 UP L4OK  1  000 
0 0 0 0
BACKEND   1 UP   1  000 
0 0   410 0

 1-STATUS  2-TRAFFIC  3-HTTP  4-ERRORS  5-CLI       
  [#30/#0]

What else please ?

On 17 November 2016 at 11:47, Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>> wrote:
OK, so it’s clearly there in terms of data.

Can you now verify whether the web server is up on the repo server?

ansible repo_all -m shell -a "service nginx status"

Also check haproxy by accessing the haproxy host and running:

hatop –s /var/run/haproxy.stat
Look specifically for the repo_all-* entries and check whether they’re up/down.
There should be at least one front end and one back end.

Thanks,

Jesse

From: Achi Hamza <h16m...@gmail.com<mailto:h16m...@gmail.com>>
Date: Thursday, November 17, 2016 at 10:39 AM
To: Jesse Pretorius 
<jesse.pretor...@rackspace.co.uk<mailto:jesse.pretor...@rackspace.co.uk>>
Cc: 
"OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>"
 
<OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>>

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse,

Yes, 

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip issues

[Jesse Pretorius]

Hi Achi,

Can you list the files in /var/www/repo/os-release?

ansible repo_all -m shell -a "find /var/www/repo/os-releases -type f"

Thanks,

Jesse
IRC: odyssey4me

From: Achi Hamza 
Date: Thursday, November 17, 2016 at 10:01 AM
To: "OpenStack-operators@lists.openstack.org" 
, "openstack-...@lists.openstack.org" 

Subject: Re: [Openstack-operators] [openstack-dev] [openstack-ansible] pip 
issues

Hi Jesse and Jean-Philippe,

Thank you both for your responses.

@Jesse
I am just having one node for the infra and one repo, for now:

 root@maas:/opt/openstack-ansible/playbooks# ansible repo_all -m shell -a "du 
-sh /var/www/repo/*"
Variable files: "-e @/etc/openstack_deploy/user_secrets.yml -e 
@/etc/openstack_deploy/user_variables.yml "
node01_repo_container-82b4e1f6 | SUCCESS | rc=0 >>
900K
/var/www/repo/links
656M
/var/www/repo/openstackgit
964K
/var/www/repo/os-releases
4.0K
/var/www/repo/pkg-cache
105M
/var/www/repo/pools
273M
/var/www/repo/venvs

@Jean-Philippe
Yes, i am using haproxy. I cannot wget the 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt file 
neither form the bare metal node nor from within the container:

root@node01-galera-container-368f269a:~# wget -c 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
--2016-11-17 09:53:19--  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:20--  (try: 2)  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.
--2016-11-17 09:53:22--  (try: 3)  
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt
Connecting to 172.16.1.2:8181... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

I attached my haproxy.conf file, can you please check it ? If the problem 
persists i will pop into the irc channel.

Thank you,
Hamza






On 17 November 2016 at 10:10, Jean-Philippe Evrard 
>
 wrote:
Hello,

Is this a greenfield newton?
Could you wget your 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  file 
reliably?
Are you using haproxy?  Are all the backends OK there?

Don’t hesitate to come by our irc channel, it’s probably easier to have a 
conversation there.

Best regards,
Jean-Philippe

From: Achi Hamza >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Thursday, 17 November 2016 at 08:56
To: 
"openstack-...@lists.openstack.org" 
>
Subject: [openstack-dev] [openstack-ansible] pip issues


Hi all,

Context:
openstact-ansible newton, tag: 14.0.1
OS: ubuntu 16.04 LTS

the pip command is unable to download packages form the repo container. I can 
reach the repo container from within galera container through the port 8181:

root@node01:~# lxc-attach -n node01_galera_container-368f269a
root@node01-galera-container-368f269a:~# nc -nzv 172.16.1.2 8181
Connection to 172.16.1.2 8181 port [tcp/*] succeeded!

However, when i run galera-install.yml playbook it throws the following error 
(but the root cause is pip_install role)  :
root@maas:/opt/openstack-ansible/playbooks# openstack-ansible galera-install.yml
………..
TASK [pip_install : Install pip packages (fall back mode)] *
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (5 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (4 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (3 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (2 
retries left).
FAILED - RETRYING: TASK: pip_install : Install pip packages (fall back mode) (1 
retries left).fatal: [node01_galera_container-368f269a]: FAILED! => {"changed": 
false, "cmd": "/usr/local/bin/pip install -U --isolated --constraint 
http://172.16.1.2:8181/os-releases/14.0.1/requirements_constraints.txt  
ndg-httpsclient requests", "failed": true, "msg": "\n:stderr: Retrying 
(Retry(total=4, connect=None, read=None, redirect=None)) after connection 
broken by 'ProtocolError('Connection aborted.', BadStatusLine(\"''\",))': 
/os-releases/14.0.1/requirements_constraints.txt\nRetrying (Retry(total=3, 
connect=None, read=None, redirect=None)) after connection broken by 
'ProtocolError('Connection aborted.', 

Re: [Openstack-operators] openstack-ansible - recovery of failed infrastructure host

[Jesse Pretorius]

Hi Ben,

If you stand up a new infrastructure host with the same IP address and name in 
openstack_user_config then execute the setup-hosts, setup-infrastructure and 
setup-openstack plays it should put everything back in place for you without 
further intervention.

Jesse
IRC: odyssey4me

On 10/24/16, 9:58 AM, "Ben Lagunilla"  wrote:

Hi All,

How do we go about restoring a failed infrastructure host? Setup is 
openstack mitaka with high availability configured on 3 infrastructure hosts.

Thanks!



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Instances failing to launch when rbd backed (ansible Liberty setup)

[Jesse Pretorius]

Hi Grant/Chris,

Can you verify whether there is a symlink from inside the venv to the location 
of the RBD python libs in the system packages? If there isn’t one then you 
should be able to re-execute the os-nova-install.yml playbook as it gets 
executed here: 
https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/os_nova/tasks/nova_compute_kvm_install.yml#L103-L123

If you can file a bug with some information regarding what actions led you to 
get to this point it’d be great for us to try and figure out what went wrong so 
that we can improve things for the future.

Thanks,

Jesse
IRC: odyssey4me

From: Grant Morley 
Date: Friday, October 21, 2016 at 3:23 PM
To: Chris Sarginson , OpenStack Operators 

Cc: "ian.ba...@serverchoice.com" 
Subject: Re: [Openstack-operators] Instances failing to launch when rbd backed 
(ansible Liberty setup)


Hi Chris,

That bug suggests there has been a fix in 12.0.8. Hopefully 12.0.16 should have 
that fixed still with a bit of luck , unless the fix hasn't been carried over.

I can see that the bug report says there is also a fix 12.0.9 and 12.0.11.

Do you know if there is a workaround we can try for this? As ideally I don't 
want to be downgrading the openstack_ansible version.

Regards,

On 21/10/16 14:09, Chris Sarginson wrote:
It seems like it may be an occurrence of this bug, as you look to be using 
python venvs:

https://bugs.launchpad.net/openstack-ansible/+bug/1509837

2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py",
 line 117, in __init__

Chris

On Fri, 21 Oct 2016 at 13:19 Grant Morley 
> wrote:

Hi all,

We have a openstack-ansible setup and have ceph installed for the backend. 
However whenever we try and launch a new instance it fails to launch and we get 
the following error:

2016-10-21 12:08:06.241 70661 INFO nova.virt.libvirt.driver 
[req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 
324844c815084205995aff10b03a85e1 - - -] [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Creating image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager 
[req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 
324844c815084205995aff10b03a85e1 - - -] [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Instance failed to spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] Traceback (most recent call last):
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py",
 line 2156, in _build_resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] yield resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py",
 line 2009, in _build_and_run_instance
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] block_device_info=block_device_info)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2527, in spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] admin_pass=admin_password)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2939, in _create_image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] backend = image('disk')
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
 line 2884, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] fname + suffix, image_type)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f]   File 
"/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py",
 line 967, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 
5633d98e-5f79-4c13-8d45-7544069f0e6f] return backend(instance=instance, 
disk_name=disk_name)
2016-10-21 

Re: [Openstack-operators] Moving from distro packages to containers (or virtualenvs...)

[Jesse Pretorius]

On 13 May 2016 at 19:59, Joshua Harlow  wrote:

>
> So I guess its like the following (correct me if I am wrong):
>
> openstack-ansible
> -
>
> 1. Sets up LXC containers from common base on deployment hosts (ansible
> here to do this)
> 2. Installs things into those containers (virtualenvs, packages, git
> repos, other ... more ansible)
> 3. Connects all the things together (more more ansible).
> 4. Decommissions existing container (if it exists) and replaces with new
> container (more more more ansible).
> 5. <>
>

Almost.

As OpenStack-Ansible treats the LXC containers like hosts (this is why OSA
supports deploying to LXC machine containers and to VM's or normal hosts)
we don't replace containers - we simply deploy the new venv into a new
folder, reconfigure, then restart the service to use the new venv.

To speed things up in large environments we pre-build the venvs on a repo
server, then all hosts or containers grab them from the repo server

The mechanisms we use allow deployers to customise the packages built into
the venvs (you might need an extra driver in the neutron/cinder venvs, for
instance) and allow the OpenStack services to build directly from any git
source (this means you can maintain your own fork with all the fixes you
need, if you want to).

With OpenStack-Ansible you're also not forced to commit to the integrated
build. Each service role is broken out into its own repository, so you're
able to write your own Ansible playbooks to consume the roles which setup
the services in any way that pleases you.

The advantage in our case of using the LXC containers is that if something
ends up broken somehow in the binary packages (this hasn't happened yet in
my experience) you're able to simply blow away the container and rebuild it.

I hope this helps. Feel free to ping me any more questions.

---
Jesse
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [OpenStack-Ansible] Liberty to Mitaka upgrade?

[Jesse Pretorius]

On 20 April 2016 at 04:27, Dale Baley  wrote:

> Is there a document or guide for upgrading from Liberty to Mitaka yet?
>

Hi Dale,

The active work to test and implement any plays to assist with upgrades has
not yet been implemented for Liberty->Mitaka. We hope to do this work by
Newton Milestone-1. If you wish to participate in the process for getting
this done then please join us in #openstack-ansible to compare notes.

The starting point for this work is always to begin by using the same
process as is used for minor upgrades [1] in a test environment and to see
whether anything breaks, then to prepare patches to resolve that. As our
roles and plays already automatically detect and deal with things like DB
migrations and the software version changes this may actually work
perfectly well - it's just that we've not had the chance to really get down
to testing the upgrades with focal attention yet.

[1]
http://docs.openstack.org/developer/openstack-ansible/install-guide/app-minorupgrade.html

Best regards,

Jesse
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-ansible] Mid Cycle Sprint

[Jesse Pretorius]

Hi everyone,

Thank you all for responding and giving your feedback.

Given that the Ops Mid Cycle [1] is in the UK (15-16 Feb) and AnsibleFest
[2] is also in the UK (18 Feb), and also that everyone seems ok with the UK
as a location for the mid cycle then we shall have it in the UK!

The Ops Mid Cycle organisers have also graciously offered for us to be a
part of the Ops Mid Cycle agenda. I would like as many of us as possible to
play a part in the Ops Mid Cycle - listening, learning and contributing
where possible. As such, I'd like us to limit the number of sessions we
have within the Ops Mid Cycle. Let's have a few open design-style sessions.

Thereafter, as discussed in today's community meeting [3], we can break
down into work sessions where we can work on specific goals. This will
happen on the Wednesday 17 Feb and we will be hosted at the Rackspace
offices in Hayes (West London).

I've setup an Etherpad [4] to gather proposals for Fishbowl Sessions (at
the Ops Mid Cycle) and Work Sessions (for 17 Feb). Please add any sessions
you'd like to facilitate/moderate. If there are sessions there, then please
feel free to add your +1 to show that you'd like to see it.

As discussed in the meeting today [3] I'm on holiday until the New Year.
Please all have a wonderful time over the year's end and I'll see you all
bright-eyed and bushy tailed next year!

Best regards,

Jesse
IRC: odyssey4me

[1] https://etherpad.openstack.org/p/MAN-ops-meetup
[2] http://www.ansible.com/ansiblefest
[3]
http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2015/openstack_ansible_meeting.2015-12-17-16.01.log.html
[4] https://etherpad.openstack.org/p/openstack-ansible-mitaka-midcycle
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mid Cycle Sprint

[Jesse Pretorius]

Hi everyone,

At the Mitaka design summit in Tokyo we had some corridor discussions about
doing a mid-cycle meetup for the purpose of continuing some design
discussions and doing some specific sprint work.

***
I'd like indications of who would like to attend and what
locations/dates/topics/sprints would be of interest to you.
***

For guidance/background I've put some notes together below:

Location

We have contributors, deployers and downstream consumers across the globe
so picking a venue is difficult. Rackspace have facilities in the UK
(Hayes, West London) and in the US (San Antonio) and are happy for us to
make use of them.

Dates
-
Most of the mid-cycles for upstream OpenStack projects are being held in
January. The Operators mid-cycle is on February 15-16.

As I feel that it's important that we're all as involved as possible in
these events, I would suggest that we schedule ours after the Operators
mid-cycle.

It strikes me that it may be useful to do our mid-cycle immediately after
the Ops mid-cycle, and do it in the UK. This may help to optimise travel
for many of us.

Format
--
The format of the summit is really for us to choose, but typically they're
formatted along the lines of something like this:

Day 1: Big group discussions similar in format to sessions at the design
summit.

Day 2: Collaborative code reviews, usually performed on a projector, where
the goal is to merge things that day (if a review needs more than a single
iteration, we skip it. If a review needs small revisions, we do them on the
spot).

Day 3: Small group / pair programming.

Topics
--
Some topics/sprints that come to mind that we could explore/do are:
 - Install Guide Documentation Improvement [1]
 - Development Documentation Improvement (best practises, testing, how to
develop a new role, etc)
 - Upgrade Framework [2]
 - Multi-OS Support [3]

[1] https://etherpad.openstack.org/p/oa-install-docs
[2] https://etherpad.openstack.org/p/openstack-ansible-upgrade-framework
[3] https://etherpad.openstack.org/p/openstack-ansible-multi-os-support

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [keystone] Request for Feedback: Online database migrations

[Jesse Pretorius]

Hi everyone,

As we all know, upgrades are hard. Part of the problem relates to down time
due to offline database migrations.

The Keystone team has put together a spec [1] and is seeking feedback for
implementing online schema migrations.

Please weigh in, as operators, on the spec presented.

[1] https://review.openstack.org/245186

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ansible-OpenStack:cannot ssh into the containers

[Jesse Pretorius]

On 24 November 2015 at 17:02, Achi Hamza  wrote:

> Hi,
>
> I am trying to deploy openstack using openstack-ansible project. I am
> using Liberty release (tag *12.0.0*).
> The problem i have is that Ansible is not able to SSH into the containers:
>
> *TASK: [Wait for ssh to be available]
> ** *
> failed: [infra01_neutron_server_container-4d09aa86 -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.216:22
> failed: [infra01_utility_container-4da899df -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.237.186:22
> failed: [infra01_neutron_agents_container-bd7aded5 -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.237.90:22
> failed: [infra01_memcached_container-12186fcf -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.177:22
> failed: [infra01_rabbit_mq_container-7103e2f2 -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.178:22
> failed: [infra01_galera_container-b691e13b -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.60:22
> failed: [infra01_nova_conductor_container-98abf1df -> 127.0.0.1] =>
> {"elapsed": 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.238.8:22
> failed: [infra01_horizon_container-5756dc20 -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> msg: Timeout when waiting for search string OpenSSH in 172.29.236.47:22
> failed: [infra01_nova_cert_container-17a8689a -> 127.0.0.1] => {"elapsed":
> 301, "failed": true}
> .
> .
> .
>
>
> I can ping the containers:
>
>
> *root@infra01:~# ping infra01_galera_container-b691e13bPING
> infra01_galera_container-b691e13b (172.29.236.60) 56(84) bytes of data.*
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=1 ttl=64 time=0.103 ms
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=2 ttl=64 time=0.086 ms
> 64 bytes from infra01_galera_container-b691e13b (172.29.236.60):
> icmp_seq=3 ttl=64 time=0.110 ms
>

Hi Amza,

My apologies for only getting back to you right now.

We've been seeing similar SSH connectivity issues in gate checks and have
what we believe is a solution in a review in flight [1]. The patch is
easily implemented in your environment.

Failing that I have a few suggestions:

1 - Implement using at least the tag 12.0.1 as we fixed some pretty crucial
release bugs between 12.0.0 and 12.0.1. You can also implement from the
head of the Liberty branch.
2 - If your host(s) are a little slow you may wish to implement an
increased delay in /etc/openstack_deploy/user_variables.yml using the
variable 'ssh_delay'.
3 - If possible, try and catch us in the Freenode IRC channel
#openstack-ansible - there're often people there who can help you
troubleshoot.

[1] https://review.openstack.org/249942
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Keystone audit logs with haproxy

[Jesse Pretorius]

On 25 November 2015 at 05:40, Ajay Kalambur (akalambu) 
wrote:

> Hi
> Have a deployment where keystone sits behind a ha proxy node. Now
> authentication requests are made to a vip. Problem is when there is an
> authentication failure we cannot track the remote ip that failed login as
> all authentication failures show the VIP ip since ha proxy fwds the request
> to a backend keystone server
>
> How do we use a load balancer like ha proxy and also track the remote
> failed ip for authentication failures
> We get all authentication failures showing up with remote ip as vip ip
>

It's probably best to enable the forwardfor option [1] and ensure that your
Keystone logs record that information. This is relatively trivial if
Keystone is using Apache/wsgi, but I can't recall whether the eventlet
server logs the info.

[1]
https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20forwardfor
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Spec Review Time :)

[Jesse Pretorius]

On 20 November 2015 at 03:35, Tom Fifield  wrote:

>
>
> https://review.openstack.org/#/q/status:open+AND+project:openstack/nova-specs,n,z
>
>
 FYI, it's possible to use some regex in your query - for example this will
give you all currently open specs in all specs repositories:

https://review.openstack.org/#/q/project:%255Eopenstack/.*-specs+status:open,n,z
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Documentation update planning

[Jesse Pretorius]

Hi everyone,

In the community meeting [1] this week we'll be having a facilitated
discussion with regards to documentation improvements for OpenStack-Ansible.

If you have any feedback you'd like considered, would like to be part of
the conversation, or are keen to contribute then please add to the etherpad
[2], respond to this email and ideally be in the meeting. :)

[1]
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting
[2] https://etherpad.openstack.org/p/oa-install-docs

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Fedora/CentOS/other Support

[Jesse Pretorius]

Hi everyone,

There has been interest expressed from various corners to support more than
just Ubuntu in OpenStack-Ansible. While the project is very happy to work
on this, no-one has yet stepped up to take a lead role in the execution of
this work.

The current community has done some research into appropriate patterns to
use and has a general idea of how to do it - but in order to actually
execute there need to be enough people who commit to actually maintaining
the work once it's done. We don't want to carry the extra code if we don't
also pick up extra contributors to maintain the code.

If you're interested in seeing this become a reality and are prepared to
commit some time to making it happen, the project will be happy to work
with you to achieve it.

In this week's community meeting [1] we'll initiate a discussion on the
topic with the purpose of determining who's on board with the work.

In the following week's community meeting we will have a work breakdown
discussion where we'll work on filling out an etherpad [2] with what we
need to get done and who will be doing it. Naturally the etherpad can be
populated already by anyone who's interested. If you're actually happy to
volunteer to assist with the work, please feel free to add a note to that
effect on the etherpad.

[1]
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting
[2] https://etherpad.openstack.org/p/openstack-ansible-multi-os-support

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit Summary

[Jesse Pretorius]

Hi everyone,

I've put together some thoughts based on the Mitaka Summit from the
OpenStack-Ansible point of view:
http://odyssey4me.github.io/openstack/ansible/mitaka/summit/2015/11/17/mitaka-summit.html

Please feel free to ping me with any feedback!

Thanks,

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit Schedule

[Jesse Pretorius]

Hi everyone,

I've added the final details for the summit sessions to the etherpad:
https://etherpad.openstack.org/p/openstack-ansible-mitaka-summit

Our sessions are open to anyone with an interest in deploying OpenStack
with Ansible.

Note that our two primary topics for discussion are:

Image-based Deployments:
We ask the questions:
 - What benefits are there to using image-based deployments?
 - How are people doing it today?

Production-ready Upgrades
We ask the questions:
 - What does it take to do upgrades in a production OpenStack environment?
 - How can we orchestrate it successfully?

We are very interested in gathering feedback from a broad group of
OpenStack operators based on the above topics and would love to have your
feedback!

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Mitaka Summit sessions

[Jesse Pretorius]

Hi everyone,

As the Mitaka summit draws nearer, I'd like a broad view of what people
would like to discuss at the summit. This can include anyone's input!
Obviously our space and time will be limited, so any sessions that we don't
get to formally do at the summit we'll either try to do informally during
the summit (in a workgroup session or something like that), or we'll try to
engage at an alternative date (like a midcycle).

Some sessions have already been proposed - please feel free to add to the
list. Try to use the same format so that we know whether you're
facilitating or whether you want someone else to, what preparation
attendees would need, etc.

https://etherpad.openstack.org/p/openstack-ansible-mitaka-summit

It's probably a good idea to review the currently registered blueprints [0]
and specs [1] before adding any proposed sessions - just to make sure that
you're not covering something that's already on the go. For any current
blueprints/specs we're certainly open to more discussion, but it'd be great
to see that discussion happen in the spec reviews. :)

[0] https://blueprints.launchpad.net/openstack-ansible
[1]
https://review.openstack.org/#/q/project:openstack/openstack-ansible-specs+status:open,n,z

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] Release: Kilo 11.1.0

[Jesse Pretorius]

Hi everyone,

The openstack-ansible community is pleased to announce our release of the
Kilo 11.1.0 milestone.

This release fixes 57 bugs and implements 4 major blueprints, including:

* Keystone as a Federated Identity Provider
* Keystone as a Federated Service Provider
* Horizon Federation Single Sign-On
* Multi-Region Swift Clustering
* Ceilometer

For the full details please see launchpad [1].

For 11.2.0 [2] we'll be optionally incorporating Ceph as a back-end for
Glance, Cinder and Nova. Between now and then we expect to release at least
two hotfix releases in our usual cadence of once every two weeks.

If you're looking to kick the tyres to see how we do things or just want to
try OpenStack out and have a test system up and running within around 60
minutes, then try out our all-in-one deployment [3] which is used for
development/testing.

We welcome everyone to give it a try, participate in reviews [4] and get
involved! [5] If you have any questions, contact us in #openstack-ansible.

Best regards,

Jesse
IRC: odyssey4me

[1] https://launchpad.net/openstack-ansible/+milestone/11.1.0
[2] https://launchpad.net/openstack-ansible/+milestone/11.2.0
[3]
https://github.com/stackforge/os-ansible-deployment/blob/kilo/development-stack.rst
[4]
https://review.openstack.org/#/q/project:stackforge/os-ansible-deployment,n,z
[5]
https://github.com/stackforge/os-ansible-deployment/blob/master/CONTRIBUTING.rst
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [openstack-ansible] [os-ansible-deployment] Kilo - Liberty Upgrade Problems

[Jesse Pretorius]

I'm adding openstack-operators too as this is a discussion that I think it
would be useful to have their input for.

On 30 July 2015 at 19:18, Ian Cordasco ian.corda...@rackspace.com wrote:

 Hey all,

 As you may have seen elsewhere on openstack-dev, OpenStack is changing the
 versioning for the service projects. This means our previous upgrade
 solution will not continue to work. For context, one of our project's
 goals is to have in-place upgrades be a reality. Previously, using our
 repository (package) mirror doing:

 # pip install -U {{servicename}}

 Was perfectly fine. The problem will now occur that 2015.1.0 (kilo) is
 more recent than any of the new service version numbers (as far as pip is
 concerned). This would be resolved if the release management team for
 OpenStack properly used an epoch to indicate that the versioning scheme is
 fundamentally different and something like Glance 8.0.0 should sort after
 Glance 2015.1.0, but they won't (for reasons that you can read in earlier
 threads on this list).


Yes. This is going to cause quite a few headaches I'm sure.


 So, in order to satisfy the goal of in-place upgrades, we need a way
 around this. Currently, we use a tool called 'yaprt' to build wheels and
 repository indices for our project. This tool can (and currently does)
 create reports of the built files and exports those reports as JSON. We
 can use this to know the version generated for a service and then instead
 do:

 # pip install {{servicename}}=={{yaprt_generated_version}}

 This will force pip to ignore the fact that the existing (kilo)
 installation is actually supposed to sort as more recent because you're
 telling it to install a very specific version. This will likely need to be
 our upgrade path going forward unless we also require operators to clear
 out their existing repository mirror of packages with Kilo versions (and
 then we can go back to relying on pip's version sorting semantics to do
 pip install -U {{servicename}}).


So what would the resulting version be? Would the python wheel be 2016.x.x
or would the file simply be named that so that we're sitting with this
workaround for only one cycle and future cycles can revert to the previous
process?


 This is, at the moment, the seemingly simplest way to work around the
 brokenness that is the upstream versioning change.

 If you can think of a different way of approaching this, we'd love to hear
 it. If not, Kevin or myself will probably start working on this approach
 in a week or two so it's ready for when Liberty is actually released and
 we can start testing upgrades from the kilo branch to master (which is
 currently tracking liberty).


This is not a fun problem to have to solve, but it seems a reasonable
solution. Whatever we do I'd prefer to see it as a solution that we only
have to carry for one cycle so that all versioning matches upstream from
then on. If that's not possible then some sort of epoch-style workaround
like this may just be something we have to live with.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [os-ansible-deployment][openstack-ansible] Release review/bug list for tomorrow's meeting

[Jesse Pretorius]

On 29 July 2015 at 22:04, Jesse Pretorius jesse.pretor...@gmail.com wrote:

 The following reviews are in-flight and are important for the upcoming
 releases, and therefore there is a need for more reviews and in some cases
 backports once the master patches have landed:
 https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z

 The upcoming releases (this weekend) are:

 Kilo: https://launchpad.net/openstack-ansible/+milestone/11.1.0

Unfortunately we have had to postpone the release for 11.1.0 until Monday
as we have missed our deadline for several key patches merging.

The entire available core team agreed that this would be best in the
#openstack-ansible channel. It'd be better for us all to have a bit more
time to review the final list of patches, especially considering several of
them are quite substantial.

We'll revisit this on Monday - please continue to review if you have a
moment to do so before then.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [os-ansible-deployment][openstack-ansible] Release review/bug list for tomorrow's meeting

[Jesse Pretorius]

Hi everyone,


We need some support for reviews and bug updates, ideally before the next
meeting at 16:00 UTC tomorrow as per
https://wiki.openstack.org/wiki/Meetings/openstack-ansible#Agenda_for_next_meeting


The following reviews are in-flight and are important for the upcoming
releases, and therefore there is a need for more reviews and in some cases
backports once the master patches have landed:
https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z


The upcoming releases (this weekend) are:

Kilo: https://launchpad.net/openstack-ansible/+milestone/11.1.0

Juno: https://launchpad.net/openstack-ansible/+milestone/10.1.11


I’d appreciate it if everyone could take a look at the reviews – some of
which need to be rebased/changed - and the bugs that are not yet ‘In
Progress’ and decide whether the solutions for the bugs will make it or not
in time for the release.

We’ll discuss everything in the list in the meeting and decide on the best
course of action.

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OSAD for RHEL

[Jesse Pretorius]

On 9 July 2015 at 05:54, John Dewey j...@dewey.ws wrote:

 IMO - registering the systems with subscription manager or pointing to in
 house yum repos should be included as part of system bootstrapping, and not
 a part of OSAD.  OSAD should simply install the specific packages for the
 alternate distro.


Agreed, trying to cater for all things that everyone wants in their
bootstrapping is a rabbit hole best not ventured into as it'll bloat the
project considerably.


 Might also be a good time to abstract the system packaging module into a
 higher level one which handles `yum` or `apt` behind the scenes.We can
 then manage the list of packages per distro[1].  Throwing this out as an
 idea vs copy-paste every apt with a yum section.


Ansible appears to be building this abstraction already for v2 [1], but has
a means to do this in an alternative way [2].

[1]
https://github.com/ansible/ansible-modules-core/blob/devel/packaging/os/package.py
[2] http://serverfault.com/a/697736
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [os-ansible-deployment] Feedback on Keystone Federation Spec

[Jesse Pretorius]

Hi everyone,

There was quite a bit of fanfare around the new federation features in
OpenStack Kilo.

In the os-ansible-deployment/openstack-ansible project we've been putting
together a view on how to implement federation with as little complexity as
possible.

We've been working on some prototype code which can be seen by looking at
the patches on the blueprint whiteboard [1] and have also prepared a spec
for the implementation [2].

We'd like to get some feedback from the broader community - from deployers
interested in using the feature and from developers/deployers who've worked
with federation. The feedback we'd like to see is both in terms of the spec
and the prototype code (which is changing quite frequently as we figure out
the bits and pieces).

The follow-on to this work will be to specifically add the capability to
make use of an ADFS IdP for a Keystone SP. This work will be linked to
another blueprint [3] which is still a work in progress.

I look forward to the review feedback!

[1]
https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-federation
[2] https://review.openstack.org/194147
[3]
https://blueprints.launchpad.net/openstack-ansible/+spec/keystone-sp-adfs-idp

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Collaborative Logstash filter development

[Jesse Pretorius]

Hi everyone,

During the summit at Vancouver and at time both prior and after there has
been expressed interest in more collaborative development of Logstash
filters in an OpenStack environment.

Current examples which I'm aware of include:
 - https://github.com/godaddy/openstack-logstash
 - https://github.com/osops/tools-logging
 - http://docs.openstack.org/infra/system-config/logstash.html
 -
https://github.com/rcbops/rpc-extras/tree/master/rpcd/playbooks/roles/logstash/templates

There was some discussion in the Operators track at the design summit about
setting up a collaborative location, but to my knowledge there has been no
firm decision about it.

It is plausible that we could put together a repository which is purely
focused on the filters (not the input/output sections) and which could be
consumed by downstream users (eg: openstack-infra). To do this, however,
will require a group of interested parties to agree on how this will be
done. Considering that it will likely be accepted into the Big Tent it will
likely also require a PTL and core reviewing team.

We (Rackspace Private Cloud) are happy to seed initial configurations and
work with a team of other parties to put together a standardised set of
filters which are usable and useful. When agreed that the filters are ready
we will also work with openstack-infra to implement the filters into
logstash.openstack.org.

How would the Operator community like to proceed with this? Shall we
arrange some sort of IRC discussion to figure out how best to organise this
and to identify next steps?

-- 
Jesse Pretorius
IRC: odyssey4me
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenSource ELK configurations

[Jesse Pretorius]

On 27 May 2015 at 20:58, Joe Gordon joe.gord...@gmail.com wrote:


 On Wed, May 27, 2015 at 12:27 PM, Joseph Bajin josephba...@gmail.com
 wrote:

 I think those from the Infra Project are a great start, but I do think
 they are missing a lot.


 I am sure they are, and we would love contributions to make them better.
 The filters from the infra project are used every day by developers use to
 hunt down issues. And the more similar the developer and operator
 experience is here the easier it will be for the two groups to talk about
 the same problems.


As part of development for Rackspace Private Cloud we've done a ton of work
in this space and are happy to contribute the work upstream. The current
work is exposed in https://github.com/rcbops/rpc-extras/pull/123 and is
available for anyone to review/comment/consume.


 Instead of breaking down the message into parts, it just breaks down the
 type (INFO, DEBUG, etc) and then makes the rest of the message greedy.
 That doesn't help searching or graphing or anything like that (or at least
 makes it more difficult over time). At least that is what I have seen.


 One of the reasons for this is because most of the log messages are very
 unstructured. But I have a hunch we can do better then what we have today.


We've done quite a bit of this. We extract transaction ID's for each
project we deploy (including swift), response times to API queries, byte
sizes, etc.


 We are using a little bit tweaked version of the godaddy's scripts.  That
 seems to give us a good amount of detail that we can search and filter on.
 I'll see if I can post my versions of them to the repo.


We derived some stuff from https://github.com/godaddy/openstack-logstash
and other stuff from https://github.com/osops/tools-logging - I think it's
time that we find a common place to consolidate and standardise.
openstack-infra would appear to be a good place to do this.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Specifying multiple tenants for aggregate_multitenancy_isolation_filter

[Jesse Pretorius]

On 28 January 2015 at 06:19, Tim Bell tim.b...@cern.ch wrote:

  +1 for commas. Configuration files with JSON is OK but commas for CLIs.


+1 for commas from an operator perspective
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Storage Backend: Sheepdog, Ceph, or GlusterFS

[Jesse Pretorius]

On 6 November 2014 13:01, Hossein Zabolzadeh zabolza...@gmail.com wrote:

 Thanks for your opinion. But I am looking for the real difference
 between them...
 - Which one is better support in openstack?
 - Which one provides better unified storage backend for all openstack
 storage controllers(cinder, swift and glance)?


I don't think that Gluster or Sheepdog provide a storage back-end capable
of providing block storage (cinder) and object storage (swift) back-ends.
Only Ceph provides a properly unified back-end for both. Ceph has also been
supported for cinder for over two years - it's very mature. The Rados
Gateway (Object Storage/Swift Interface) is much more recent, but I have
yet to see problems with it - as long as you do acknowledge that it is not
Swift.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators