Re: [openstack-dev] [infra][qa][requirements] Pip 10 is on the way

[Arun SAG]

Hello,

On Thu, Apr 5, 2018 at 5:39 PM, Paul Belanger  wrote:

> Yah, I agree your approach is the better, i just wanted to toggle what was
> supported by default. However, it is pretty broken today.  I can't imagine
> anybody actually using it, if so they must be carrying downstream patches.
>
> If we think USE_VENV is valid use case, for per project VENV, I suggest we
> continue to fix it and update neutron to support it.  Otherwise, we maybe 
> should
> rip and replace it.

I work for Yahoo (Oath). We use USE_VENV a lot in our CI. We use VENVs
to deploy software to
production as well. we have some downstream patches to devstack to fix
some issues with
USE_VENV feature, i would be happy to upstream them. Please do not rip
this out. Thanks.


-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Race in FixedIP.associate_pool

[Arun SAG]

Hello,


On Tue, Dec 12, 2017 at 12:22 PM, Arun SAG <saga...@gmail.com> wrote:
> Hello,
>
> We are running nova-network in ocata. We use mysql in a master-slave
> configuration, The master is read/write, and all reads go to the slave
> (slave_connection is set). When we tried to boot multiple VMs in
> parallel (lets say 15), we see a race in allocate_for_instance's
> FixedIP.associate_pool. We see FixedIP.associate_pool associates an
> IP, but later in the code we try to read the allocated FixedIP using
> objects.FixedIPList.get_by_instance_uuid and it throws
> FixedIPNotFoundException. We also checked the slave replication status
> and Seconds_Behind_Master: 0
>
[snip]
>
> This kind of how the logs look like
> 2017-12-08 22:33:37,124 DEBUG
> [yahoo.contrib.ocata_openstack_yahoo_plugins.nova.network.manager]
> /opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py:get_instance_nw_info:894
> Fixed IP NOT found for instance
> 2017-12-08 22:33:37,125 DEBUG
> [yahoo.contrib.ocata_openstack_yahoo_plugins.nova.network.manager]
> /opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py:get_instance_nw_info:965
> Built network info: |[]|
> 2017-12-08 22:33:37,126 INFO [nova.network.manager]
> /opt/openstack/venv/nova/lib/python2.7/site-packages/nova/network/manager.py:allocate_for_instance:428
> Allocated network: '[]' for instance
> 2017-12-08 22:33:37,126 ERROR [oslo_messaging.rpc.server]
> /opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:_process_incoming:164
> Exception during message handling
> Traceback (most recent call last):
>   File 
> "/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
> line 155, in _process_incoming
> res = self.dispatcher.dispatch(message)
>   File 
> "/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 222, in dispatch
> return self._do_dispatch(endpoint, method, ctxt, args)
>   File 
> "/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 192, in _do_dispatch
> result = func(ctxt, **new_args)
>   File 
> "/opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py",
> line 347, in allocate_for_instance
> vif = nw_info[0]
> IndexError: list index out of range
>
>
> This problem goes way when we get rid of the slave_connection setting
> and just use single master. Has any one else seen this? Any
> recommendation to fix this issue?
>
> This issue is kind of  similar to https://bugs.launchpad.net/nova/+bug/1249065
>

If anyone is running into db race while running database in
master-slave mode with async replication, The bug has been identified
and getting fixed  here
https://bugs.launchpad.net/oslo.db/+bug/1746116

-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Race in FixedIP.associate_pool

[Arun SAG]

Hi Jay,

On Fri, Dec 15, 2017 at 1:56 PM, Jay Pipes  wrote:

> Can you point us to the code that is generating the above? It seems that
> get_instance_nw_info() in the Yahoo! manager.py contrib module line 965 is
> trying to build network information for an empty list of vNICs... where is
> that list of vNICs coming from?


The vNICs are empty because objects.FixedIPList.get_by_instance_uuid
is empty 
https://github.com/openstack/nova/blob/master/nova/network/manager.py#L527
The Yahoo! manager.py's get_by_instance_uuid is essentially same as
the upstream code except we change the VIF_TYPE in
get_instance_nw_info

@messaging.expected_exceptions(exception.InstanceNotFound)
def get_instance_nw_info(self, context, instance_id, rxtx_factor,
 host, instance_uuid=None, **kwargs):
"""Creates network info list for instance.

called by allocate_for_instance and network_api
context needs to be elevated
:returns: network info list [(network,info),(network,info)...]
where network = dict containing pertinent data from a network db object
and info = dict containing pertinent networking data
"""
if not uuidutils.is_uuid_like(instance_id):
instance_id = instance_uuid
instance_uuid = instance_id
LOG.debug('Get instance network info', instance_uuid=instance_uuid)

try:
fixed_ips = objects.FixedIPList.get_by_instance_uuid(
context, instance_uuid)
except exception.FixedIpNotFoundForInstance:
fixed_ips = []

LOG.debug('Found %d fixed IPs associated to the instance in the '
  'database.',
  len(fixed_ips), instance_uuid=instance_uuid)

nw_info = network_model.NetworkInfo()
# (saga): The default VIF_TYPE is bridge. We need to use OVS
# This is the only reason we copied this method from the base class
if not CONF.network_driver or CONF.network_driver ==
'nova.network.linux_net':
if CONF.linuxnet_interface_driver ==
'nova.network.linux_net.LinuxOVSInterfaceDriver':
vif_type = network_model.VIF_TYPE_OVS


Here are the sequence of actions happen in nova-network

1. allocate_for_instance calls -> allocate_fixed_ips
2. FixedIPs are successfully associated (we can see this in the log)
3. allocate_for_instance calls get_instance_nw_info, which in turn
gets the fixedip's associated in step 2 using
objects.FixedIPList.get_by_instance_uuid, This raises FixedIPNotFound
exception

We remove the slave and just ran with just single master, the errors
went away. We also switched to using semi-synchronous replication
between master
and slave,  the errors went away too. All of this points to a race
between write and read to the DB.

Does openstack expects synchronous replication to read-only slaves?


-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Race in FixedIP.associate_pool

[Arun SAG]

Hello,

We are running nova-network in ocata. We use mysql in a master-slave
configuration, The master is read/write, and all reads go to the slave
(slave_connection is set). When we tried to boot multiple VMs in
parallel (lets say 15), we see a race in allocate_for_instance's
FixedIP.associate_pool. We see FixedIP.associate_pool associates an
IP, but later in the code we try to read the allocated FixedIP using
objects.FixedIPList.get_by_instance_uuid and it throws
FixedIPNotFoundException. We also checked the slave replication status
and Seconds_Behind_Master: 0

mysql> show slave status \G
*** 1. row ***
   Slave_IO_State: Waiting for master to send event
  Master_Host: master.hostname
  Master_User: repl
  Master_Port: 3306
Connect_Retry: 60
  Master_Log_File: mysqld-bin.03
  Read_Master_Log_Pos: 142605121
   Relay_Log_File: mysqld-relay-bin.06
Relay_Log_Pos: 142602813
Relay_Master_Log_File: mysqld-bin.03
 Slave_IO_Running: Yes
Slave_SQL_Running: Yes
  Replicate_Do_DB:
  Replicate_Ignore_DB: mysql,mysql
   Replicate_Do_Table:
   Replicate_Ignore_Table:
  Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
   Last_Errno: 0
   Last_Error:
 Skip_Counter: 0
  Exec_Master_Log_Pos: 142605121
  Relay_Log_Space: 142603151
  Until_Condition: None
   Until_Log_File:
Until_Log_Pos: 0
   Master_SSL_Allowed: No
   Master_SSL_CA_File:
   Master_SSL_CA_Path:
  Master_SSL_Cert:
Master_SSL_Cipher:
   Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
   Last_SQL_Errno: 0
   Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
 Master_Server_Id: 172687130
  Master_UUID: 8112eeef-de1e-11e7-8098-ac162da46cbc
 Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
  SQL_Remaining_Delay: NULL
  Slave_SQL_Running_State: Slave has read all relay log; waiting
for the slave I/O thread to update it
   Master_Retry_Count: 86400
  Master_Bind:
  Last_IO_Error_Timestamp:
 Last_SQL_Error_Timestamp:
   Master_SSL_Crl:
   Master_SSL_Crlpath:
   Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0

This kind of how the logs look like
2017-12-08 22:33:37,124 DEBUG
[yahoo.contrib.ocata_openstack_yahoo_plugins.nova.network.manager]
/opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py:get_instance_nw_info:894
Fixed IP NOT found for instance
2017-12-08 22:33:37,125 DEBUG
[yahoo.contrib.ocata_openstack_yahoo_plugins.nova.network.manager]
/opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py:get_instance_nw_info:965
Built network info: |[]|
2017-12-08 22:33:37,126 INFO [nova.network.manager]
/opt/openstack/venv/nova/lib/python2.7/site-packages/nova/network/manager.py:allocate_for_instance:428
Allocated network: '[]' for instance
2017-12-08 22:33:37,126 ERROR [oslo_messaging.rpc.server]
/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py:_process_incoming:164
Exception during message handling
Traceback (most recent call last):
  File 
"/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
line 155, in _process_incoming
res = self.dispatcher.dispatch(message)
  File 
"/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 222, in dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
  File 
"/opt/openstack/venv/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
line 192, in _do_dispatch
result = func(ctxt, **new_args)
  File 
"/opt/openstack/venv/nova/lib/python2.7/site-packages/yahoo/contrib/ocata_openstack_yahoo_plugins/nova/network/manager.py",
line 347, in allocate_for_instance
vif = nw_info[0]
IndexError: list index out of range


This problem goes way when we get rid of the slave_connection setting
and just use single master. Has any one else seen this? Any
recommendation to fix this issue?

This issue is kind of  similar to https://bugs.launchpad.net/nova/+bug/1249065

-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [distributions][devstack][requirements] Upper constraints broken in stable/ocata on CentOS 7/RHEL 7

[Arun SAG]

Hi,

RHEL/CentOS 7 released new libvirt packages (upgraded from 2.x to 3.x)
which broke bootstrapping devstack on stable/ocata branch. I have a review
up here to fix this in upper-requirements.txt
https://review.openstack.org/#/c/491032/

On the review i was asked to check if upgrading to libvirt-python 3.5.0
will break any distributions, Is there a way to reach out to people running
various distributions officially? If there are distro/vendor liaisons in
this list, can you help me by reaching out to your internal teams to check
whether it is okay to upgrade libvirt-python to 3.5.x in stable/ocata?

Thanks
-- 
Arun S A G
http://zer0c00l.in/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] Partition images in ironic

[Arun SAG]

Hi All,

I am looking for operators who have deployed OS using partition images
in ironic. I have
following questions

1. The partition image deployment by default seems to install grub2. How
did that work for distro's like RHEL6 which has legacy version of grub?

2. Any one has created and supported LVM/soft raid with partition
images? How did you
do that? How hard was it?


-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][ironic][cinder][nova] 'tar' as an image disk_format

[Arun SAG]

Hi Flavio,

On Wed, Jan 27, 2016 at 4:50 AM, Flavio Percoco  wrote:
> [snip]
> However, as a community, I think we should send a clear message and protect 
> our users and, in this case, the best way
> is to avoid adding this format as supported.
>

To address some of the concerns i have added a security impact
statement on the spec

1. Ironic doesn't unpack the OS tarball, it will be unpacked on the
target node in a ramdisk using tar utility. (tar -avxf)
2. The moment you allow an un-trusted  OS image to be deployed, the
expected security is None. An advisory
doesn't need to manipulate the extraction of the tarball to gain
access in that case.
3. In docker the vulnerability is high because a vulnerable container
can infect the host system.
4. I understand the concerns with the conversion API's , and they are
valid. Please feel free to not support tar as a conversion target.




-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance][ironic][cinder][nova] 'tar' as an image disk_format

[Arun SAG]

Hi Duncan,

On Sat, Jan 23, 2016 at 2:00 PM, Duncan Thomas  wrote:
> I guess my wisdom would be 'why'? What does this enable you to do that you
> couldn't do with similar ease with the formats we have and are people trying
> to do that frequently.
>

As far as i know the ironic only supports creating gpt partitions in UEFI mode.
GPT  partitions are needed if the disk size is more than 2 TB. The whole disk
image generated by DIB only supports MSDOS disk labels. However with tar
payload we are supporting both MSDOS and GPT disk labels. GPT is a must in
a data center environment that bios boot's the server and have >2TB HDD's.
This is one use case i think of. Please correct me if i am wrong here.

Tar based OS images are simple, They are created by installing packages in a
chroot and compressing them as tar. This is simpler than building disk images
where file systems and loop devices have to be created.

Tar based images are faster to transfer over the network than whole disk images.

Tar based OS images are getting popular, especially redhat installers
like anaconda are
already supporting them.


> We've seen in cinder that image formats have a definite security surface to
> them, and with glance adding arbitrary conversion pipelines, that surface is
> going to increase with every format we add. This should mean we tend towards
> being increasingly conservative I think.
>

I understand that this creates more headaches in cinder.

> We've heard a possible feature, but zero use case that I can see. Why is
> this better than converting your days to a supported format?

The feature is called 'tar-payload'  converting to any other format
doesn't make sense.
Do you have a suggestion as to what format we should consider
converting to instead of
adding a disk_format named tar?

-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic]Unable to locate configuration file in second reboot

[Arun SAG]

Hi,

On Thu, Dec 10, 2015 at 4:34 AM, Zhi Chang  wrote:
>
> And my tftp.conf like this http://paste.openstack.org/show/481472/
>
> I upload a movie in https://youtu.be/jktPIjEmMV8, at 04:30 there is a
> error happens.
>
> Could someone give me some idea?

There is not enough information here,

Did the installation finish? How does your 'ironic node-list' and
'ironic node-show ' look like after the machine reboots? It looks like
tftp config is missing, How does your ironic tftp config template look
like? Once the installation is done ironic conductor is supposed to
switch the tftp config so that the machine can boot from harddisk,
looks like it is not happening here.

-- 
Arun S A G
http://zer0c00l.in/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev