Re: [Openstack-operators] leaving Openstack mailing lists

[Blair Bethwaite]

Good luck with whatever you are doing next Saverio, you've been a great
asset to the community and will be missed!

On Thu, 6 Sep 2018 at 23:43, Saverio Proto  wrote:

> Hello,
>
> I will be leaving this mailing list in a few days.
>
> I am going to a new job and I will not be involved with Openstack at
> least in the short term future.
> Still, it was great working with the Openstack community in the past few
> years.
>
> If you need to reach me about any bug/patch/review that I submitted in
> the past, just write directly to my email. I will try to give answers.
>
> Cheers
>
> Saverio
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>


-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] large high-performance ephemeral storage

[Blair Bethwaite]

Hey Joe,

Thanks! So shall we settle on fio as a standard IO micro benchmarking tool?
Seems to me the minimum we want is throughput and IOPs oriented tests for
both the guest OS workload profile and the some sort of large working set
application workload. For the latter it is probably best to ignore multiple
files and focus solely on queue depth for parallelism, some sort of mixed
block size profile/s, and some sort of r/w mix (where write <=50% to
acknowledge this is ephemeral storage so hopefully something is using it
soon after storing). Thoughts?

Cheers,
Blair

On Thu., 14 Jun. 2018, 00:24 Joe Topjian,  wrote:

> Yes, you can! The kernel documentation for read/write limits actually uses
> /dev/null in the examples :)
>
> But more seriously: while we have not architected specifically for high
> performance, for the past few years, we have used a zpool of cheap spindle
> disks and 1-2 SSD disks for caching. We have ZFS configured for
> deduplication which helps for the base images but not so much for ephemeral.
>
> If you have a standard benchmark command in mind to run, I'd be happy to
> post the results. Maybe others could do the same to create some type of
> matrix?
>
> On Wed, Jun 13, 2018 at 8:18 AM, Blair Bethwaite <
> blair.bethwa...@gmail.com> wrote:
>
>> Hi Jay,
>>
>> Ha, I'm sure there's some wisdom hidden behind the trolling here?
>>
>> Believe me, I have tried to push these sorts of use-cases toward volume
>> or share storage, but in the research/science domain there is often more
>> accessible funding available to throw at infrastructure stop-gaps than
>> software engineering (parallelism is hard). PS: when I say ephemeral I
>> don't necessarily mean they aren't doing backups and otherwise caring that
>> they have 100+TB of data on a stand alone host.
>>
>> PS: I imagine you can set QoS limits on /dev/null these days via CPU
>> cgroups...
>>
>> Cheers,
>>
>>
>> On Thu., 14 Jun. 2018, 00:03 Jay Pipes,  wrote:
>>
>>> On 06/13/2018 09:58 AM, Blair Bethwaite wrote:
>>> > Hi all,
>>> >
>>> > Wondering if anyone can share experience with architecting Nova KVM
>>> > boxes for large capacity high-performance storage? We have some
>>> > particular use-cases that want both high-IOPs and large capacity local
>>> > storage.
>>> >
>>> > In the past we have used bcache with an SSD based RAID0 write-through
>>> > caching for a hardware (PERC) backed RAID volume. This seemed to work
>>> > ok, but we never really gave it a hard time. I guess if we followed a
>>> > similar pattern today we would use lvmcache (or are people still using
>>> > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with
>>> > write cache.
>>> >
>>> > Is the collective wisdom to use LVM based instances for these
>>> use-cases?
>>> > Putting a host filesystem with qcow2 based disk images on it can't
>>> help
>>> > performance-wise... Though we have not used LVM based instance storage
>>> > before, are there any significant gotchas? And furthermore, is it
>>> > possible to use set IO QoS limits on these?
>>>
>>> I've found /dev/null to be the fastest ephemeral storage system, bar
>>> none.
>>>
>>> Not sure if you can set QoS limits on it though.
>>>
>>> Best,
>>> -jay
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] large high-performance ephemeral storage

[Blair Bethwaite]

Lol! Ok, forgive me, I wasn't sure if I had regular or existential Jay on
the line :-).

On Thu., 14 Jun. 2018, 00:24 Jay Pipes,  wrote:

> On 06/13/2018 10:18 AM, Blair Bethwaite wrote:
> > Hi Jay,
> >
> > Ha, I'm sure there's some wisdom hidden behind the trolling here?
>
> I wasn't trolling at all. I was trying to be funny. Attempt failed I
> guess :)
>
> Best,
> -jay
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] large high-performance ephemeral storage

[Blair Bethwaite]

Hi Jay,

Ha, I'm sure there's some wisdom hidden behind the trolling here?

Believe me, I have tried to push these sorts of use-cases toward volume or
share storage, but in the research/science domain there is often more
accessible funding available to throw at infrastructure stop-gaps than
software engineering (parallelism is hard). PS: when I say ephemeral I
don't necessarily mean they aren't doing backups and otherwise caring that
they have 100+TB of data on a stand alone host.

PS: I imagine you can set QoS limits on /dev/null these days via CPU
cgroups...

Cheers,

On Thu., 14 Jun. 2018, 00:03 Jay Pipes,  wrote:

> On 06/13/2018 09:58 AM, Blair Bethwaite wrote:
> > Hi all,
> >
> > Wondering if anyone can share experience with architecting Nova KVM
> > boxes for large capacity high-performance storage? We have some
> > particular use-cases that want both high-IOPs and large capacity local
> > storage.
> >
> > In the past we have used bcache with an SSD based RAID0 write-through
> > caching for a hardware (PERC) backed RAID volume. This seemed to work
> > ok, but we never really gave it a hard time. I guess if we followed a
> > similar pattern today we would use lvmcache (or are people still using
> > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with
> > write cache.
> >
> > Is the collective wisdom to use LVM based instances for these use-cases?
> > Putting a host filesystem with qcow2 based disk images on it can't help
> > performance-wise... Though we have not used LVM based instance storage
> > before, are there any significant gotchas? And furthermore, is it
> > possible to use set IO QoS limits on these?
>
> I've found /dev/null to be the fastest ephemeral storage system, bar none.
>
> Not sure if you can set QoS limits on it though.
>
> Best,
> -jay
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] large high-performance ephemeral storage

[Blair Bethwaite]

Hi all,

Wondering if anyone can share experience with architecting Nova KVM boxes
for large capacity high-performance storage? We have some particular
use-cases that want both high-IOPs and large capacity local storage.

In the past we have used bcache with an SSD based RAID0 write-through
caching for a hardware (PERC) backed RAID volume. This seemed to work ok,
but we never really gave it a hard time. I guess if we followed a similar
pattern today we would use lvmcache (or are people still using bcache with
confidence?) with a few TB of NVMe and a NL-SAS array with write cache.

Is the collective wisdom to use LVM based instances for these use-cases?
Putting a host filesystem with qcow2 based disk images on it can't help
performance-wise... Though we have not used LVM based instance storage
before, are there any significant gotchas? And furthermore, is it possible
to use set IO QoS limits on these?

-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] pci passthrough & numa affinity

[Blair Bethwaite]

Hi Jon,

Following up to the question you asked during the HPC on OpenStack
panel at the summit yesterday...

You might have already seen Daniel Berrange's blog on this topic:
https://www.berrange.com/posts/2017/02/16/setting-up-a-nested-kvm-guest-for-developing-testing-pci-device-assignment-with-numa/
? He essentially describes how you can get around the issue of the
naive flat pci bus topology in the guest - exposing numa affinity of
the PCIe root ports requires newish qemu and libvirt.

However, best I can tell there is no way to do this with Nova today.
Are you interested in working together on a spec for this?

The other related feature of interest here (newer though - no libvirt
support yet I think) is gpu cliques
(https://github.com/qemu/qemu/commit/dfbee78db8fdf7bc8c151c3d29504bb47438480b),
would be really nice to have a way to set these up through Nova once
libvirt supports it.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] IRC meeting 2100UTC

[Blair Bethwaite]

Hi all,

Reminder there's a Scientific SIG meeting coming up in about 6.5
hours. All comers welcome.

(https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_March_20th_2018)
 IRC Meeting March 20th 2018 
2018-03-20 2100 UTC in channel #openstack-meeting

# Forum brainstorming
(https://etherpad.openstack.org/p/YVR18-scientific-sig-brainstorming)
# Discussion regarding state of GPU passthrough and interest in
collaborating to resolve security issues
# AOB


-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] outstanding issues with GPU passthrough

[Blair Bethwaite]

I forgot to specifically address one of the questions that Belmiro raised,
which is regarding device clean-up. I guess this would be relevant to
Ironic bare-metal clouds too.

If the hypervisor has blocked access to problematic areas of the PCI config
space then this probably isn't necessary, but as I mentioned below, this
isn't happening with QEMU/KVM today.

I asked an NVIDIAn about forcing a firmware flash on the device as a
possible means to ensure the firmware is correct (something that could be
done between device allocations by some management layer like Cyborg). He
told this would definitely not be recommended for the risk of bricking the
device, besides I couldn't find any tools that do this. Apparently the
firmware is signed. However there doesn't seem to be any publicly available
technical detail on the signing process, so I don't know whether it enables
the device to verify the source of a firmware write, or if it's just
something that NVIDIA's own drivers check by reading the firmware ROM.

On Tue., 20 Mar. 2018, 17:47 Blair Bethwaite, <blair.bethwa...@monash.edu>
wrote:

> Hi all,
>
> This has turned into a bit of a screed I'm afraid...
>
> Last week I had pings from both the folks at CERN and Catalyst Cloud about
> GPU-accelerated OpenStack instances with PCI passthrough, specifically
> asking about the security issues I've mentioned in community forums
> previously, and any other gotchas. So I figured I'd attempt to answer what
> I can on-list and hopefully others will share too. I'll also take an action
> to propose a Forum session something along the lines of "GPUs - state of
> the art & practice" where we can come together in Vancouver to discuss
> further. Just writing this has prompted me to pick-up where I last left off
> on this - currently looking for some upstream QEMU expertise...
>
> Firstly, there is this wiki page: https://wiki.openstack.org/wiki/GPUs,
> which should be relevant but could be expanded based on this discussion.
>
> **Issues / Gotchas**
>
> These days for the most part things just seem to work if you follow the
> basic docs and then ensure the usual things required with vfio-pci, like
> making sure no other host drivers are bound to the device/s. These are the
> basics which have been covered in a number of Summit presentations, most
> recently see
> https://www.openstack.org/videos/sydney-2017/not-only-for-miners-gpu-integration-in-nova-environment
> for a good overview. This blog http://www.vscaler.com/gpu-passthrough/,
> although a little dated, is still relevant. Perhaps it would be worth
> adding these things to docs or wiki.
>
> One issue that we've hit a couple of times now, most recently only last
> week, is with apparmor on Ubuntu being too restrictive when the passthrough
> device needs to be reattached post-snapshot. This has been discussed
> on-list in the past - see "[Openstack-operators] PCI Passthrough issues"
> for a good post-mortem from Jon@MIT. So far I'm not sure if this most
> recent incarnation is due to a new bug with newer cloudarchive hypervisor
> stack, or because we have stale templates in our Puppet that are
> overwriting something we should be getting from the package-shipped
> apparmor rules - if it turns out to be a new bug we'll report upstream...
>
> Perhaps more concerning for new deployers today (assuming deep-learning is
> a major motivator for adopting this capability) is that GPU P2P doesn't
> work inside a typical guest instance with multiple GPUs passed through.
> That's because the emulated flat PCI (not even PCIe) topology inside the
> guest will make the device drivers think this isn't possible. However, GPU
> clique support was added to QEMU 2.11 in
> https://github.com/qemu/qemu/commit/dfbee78db8fdf7bc8c151c3d29504bb47438480b#diff-38093e21794c7a4987feb71e498dbdc6.
> There's no Libvirt support for this yet, so I'd expect it to be at least
> another couple of cycles before we might see this hitting Nova. In any
> case, we are about to start kicking the tires on it and will report back.
>
> **Security**
>
> The big issue that's inherent with PCI passthrough is that you have to
> give up the whole device (and arguably whole server if you are really
> concerned about security). There is also potential complexity with switched
> PCIe topologies, which you're likely to encounter on any host with more
> than a couple of GPUs - if you have "wrong" PCIe chipset then you may not
> be able to properly isolate the devices from each other. I believe the
> hyperscalers may use PCIe switching fabrics with external device housing,
> as opposed to GPUs directly inside the host. They have gone to some effort
> to ensure things like PCIe Address Translation Services (ATS) get turned
> off - ATS is basically an IOMM

[Openstack-operators] outstanding issues with GPU passthrough

[Blair Bethwaite]

ve analysed these issues and given technical guidance
on risk mitigations to partners.

We haven't solved any of this at Monash, but we're also not running a
public cloud and only have a limited set of internal tenants/projects that
have access to our GPU flavors, so it's not a big risk to us at the moment.
As far as trying to lock some of this down goes, the good news is that QEMU
appears to have an existing mechanism in place to block/intercept accesses
to these control registers/windows (see hw/vfio/pci-quirks.c). So it's a
matter of getting a reference that can be used to add the appropriate
quirks...

**Future**

If the GPU clique support works for P2P that will be great. But at least
from NVIDIA's side it seems that the Linux mdev based vGPU is the way
forward (you can still have a 1:1 vGPU:pGPU allocation for heavy
workloads). Last I heard, we could expect a Linux host-side driver for this
within a month or so. There is at least one interesting architectural
complication inherent in the vGPU licensing model though, which is that the
guest vGPU drivers will need to be able to the vGPU license server/s, which
necessarily requires some link between tenant and provider networks.
Haven't played with any of this first-hand yet so not sure how problematic
(or not) it might be.

Anyway, hopefully all this is useful in some way. Perhaps if we get enough
customers pressuring NVIDIA SAs to disclose the PCIe security info, it
might get us somewhere on the road to securing passthrough.

Cheers,

-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800 <+61%203%209903%202800>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] about rebuild instance booted from volume

[Blair Bethwaite]

Please do not default to deleting it, otherwise someone will eventually be
back here asking why an irate user has just lost data. The better scenario
is that the rebuild will fail (early - before impact to the running
instance) with a quota error.

Cheers,

On Thu., 15 Mar. 2018, 00:46 Matt Riedemann,  wrote:

> On 3/14/2018 3:42 AM, 李杰 wrote:
> >
> >  This is the spec about  rebuild a instance booted from
> > volume.In the spec,there is a
> >question about if we should delete the old root_volume.Anyone who
> > is interested in
> >booted from volume can help to review this. Any suggestion is
> > welcome.Thank you!
> >The link is here.
> >Re:the rebuild spec:https://review.openstack.org/#/c/532407/
>
> Copying the operators list and giving some more context.
>
> This spec is proposing to add support for rebuild with a new image for
> volume-backed servers, which today is just a 400 failure in the API
> since the compute doesn't support that scenario.
>
> With the proposed solution, the backing root volume would be deleted and
> a new volume would be created from the new image, similar to how boot
> from volume works.
>
> The question raised in the spec is whether or not nova should delete the
> root volume even if its delete_on_termination flag is set to False. The
> semantics get a bit weird here since that flag was not meant for this
> scenario, it's meant to be used when deleting the server to which the
> volume is attached. Rebuilding a server is not deleting it, but we would
> need to replace the root volume, so what do we do with the volume we're
> replacing?
>
> Do we say that delete_on_termination only applies to deleting a server
> and not rebuild and therefore nova can delete the root volume during a
> rebuild?
>
> If we don't delete the volume during rebuild, we could end up leaving a
> lot of volumes lying around that the user then has to clean up,
> otherwise they'll eventually go over quota.
>
> We need user (and operator) feedback on this issue and what they would
> expect to happen.
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] big windows guests on skylake server cpus with libvirt+kvm

[Blair Bethwaite]

Hi all,

Has anyone else tried this combination?

We've set up some new computes with dual Xeon Gold 6150s (18c/36t), so
72 logical cores with hyperthreading. We're trying to launch a Windows
Server 2012 R2 guest (hyperv enlightenments enabled via image
properties os_type=windows, and virtio 141 drivers, built with
CloudBase's scripts (thanks guys!)) with 72 vCPUs (host-passthrough)
and 340GB RAM. On qemu 2.5 from cloudarchive Newton the guest sits in
a boot loop, with qemu 2.10 from Pike we hit the "Windows failed to
start" screen (status: 0xc001).

The interesting thing is that smaller numbers of vCPUs, e.g. splitting
the host in half 2x 36 vCPU guests, seems to be working fine. We've
also set the guest CPU topology so that Windoze is seeing the host
topology, but still no joy. Looking for some secret sauce we can poor
on this?

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [neutron] [os-vif] VF overcommitting and performance in SR-IOV

[Blair Bethwaite]

This is starting to veer into magic territory for my level of
understanding so beware... but I believe there are (or could be
depending on your exact hardware) PCI config space considerations.
IIUC each SRIOV VF will have its own PCI BAR. Depending on the window
size required (which may be determined by other hardware features such
as flow-steering), you can potentially hit compatibility issues with
your server BIOS not supporting mapping of addresses which surpass
4GB. This can then result in the device hanging on initialisation (at
server boot) and effectively bricking the box until the device is
removed.

We have seen this first hand on a Dell R730 with Mellanox ConnectX-4
card (there are several other Dell 13G platforms with the same BIOS
chipsets). We were explicitly increasing the PCI BAR size for the
device (not upping the number of VFs) in relation to a memory
exhaustion issue when running MPI collective communications on hosts
with 28+ cores, we only had 16 (or maybe 32, I forget) VFs configured
in the firmware.

At the end of that support case (which resulted in a replacement NIC),
the support engineer's summary included:
"""
-When a BIOS limits the BAR to be contained in the 4GB address space -
it is a BIOS limitation.
Unfortunately, there is no way to tell - Some BIOS implementations use
proprietary heuristics to decide when to map a specific BAR below 4GB.

-When SR-IOV is enabled, and num-vfs is high, the corresponding VF BAR
can be huge.
In this case, the BIOS may exhaust the ~2GB address space that it has
available below 4GB.
In this case, the BIOS may hang – and the server won’t boot.
"""

At the very least you should ask your hardware vendors some very
specific questions before doing anything that might change your PCI
BAR sizes.

Cheers,

On 23 January 2018 at 11:44, Pedro Sousa  wrote:
> Hi,
>
> I have sr-iov in production in some customers with maximum number of VFs and
> didn't notice any performance issues.
>
> My understanding is that of course you will have performance penalty if you
> consume all those vfs, because you're dividing the bandwidth across them,
> but other than if they're are there doing nothing you won't notice anything.
>
> But I'm just talking from my experience :)
>
> Regards,
> Pedro Sousa
>
> On Mon, Jan 22, 2018 at 11:47 PM, Maciej Kucia  wrote:
>>
>> Thank you for the reply. I am interested in SR-IOV and pci whitelisting is
>> certainly involved.
>> I suspect that OpenStack itself can handle those numbers of devices,
>> especially in telco applications where not much scheduling is being done.
>> The feedback I am getting is from sysadmins who work on network
>> virtualization but I think this is just a rumor without any proof.
>>
>> The question is if performance penalty from SR-IOV drivers or PCI itself
>> is negligible. Should cloud admin configure maximum number of VFs for
>> flexibility or should it be manually managed and balanced depending on
>> application?
>>
>> Regards,
>> Maciej
>>
>>>
>>>
>>> 2018-01-22 18:38 GMT+01:00 Jay Pipes :

 On 01/22/2018 11:36 AM, Maciej Kucia wrote:
>
> Hi!
>
> Is there any noticeable performance penalty when using multiple virtual
> functions?
>
> For simplicity I am enabling all available virtual functions in my
> NICs.


 I presume by the above you are referring to setting your
 pci_passthrough_whitelist on your compute nodes to whitelist all VFs on a
 particular PF's PCI address domain/bus?

> Sometimes application is using only few of them. I am using Intel and
> Mellanox.
>
> I do not see any performance drop but I am getting feedback that this
> might not be the best approach.


 Who is giving you this feedback?

 The only issue with enabling (potentially 254 or more) VFs for each PF
 is that each VF will end up as a record in the pci_devices table in the 
 Nova
 cell database. Multiply 254 or more times the number of PFs times the 
 number
 of compute nodes in your deployment and you can get a large number of
 records that need to be stored. That said, the pci_devices table is well
 indexed and even if you had 1M or more records in the table, the access of 
 a
 few hundred of those records when the resource tracker does a
 PciDeviceList.get_by_compute_node() [1] will still be quite fast.

 Best,
 -jay

 [1]
 https://github.com/openstack/nova/blob/stable/pike/nova/compute/resource_tracker.py#L572
 and then

 https://github.com/openstack/nova/blob/stable/pike/nova/pci/manager.py#L71

> Any recommendations?
>
> Thanks,
> Maciej
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

Re: [Openstack-operators] [openstack-dev] Mixed service version CI testing

[Blair Bethwaite]

+1!

It may also be worth testing a step where Nova & Neutron remain at N-1.

On 20 December 2017 at 04:58, Matt Riedemann  wrote:
> During discussion in the TC channel today [1], we got talking about how
> there is a perception that you must upgrade all of the services together for
> anything to work, at least the 'core' services like
> keystone/nova/cinder/neutron/glance - although maybe that's really just
> nova/cinder/neutron?
>
> Anyway, I posit that the services are not as tightly coupled as some people
> assume they are, at least not since kilo era when microversions started
> happening in nova.
>
> However, with the way we do CI testing, and release everything together, the
> perception is there that all things must go together to work.
>
> In our current upgrade job, we upgrade everything to N except the
> nova-compute service, that remains at N-1 to test rolling upgrades of your
> computes and to make sure guests are unaffected by the upgrade of the
> control plane.
>
> I asked if it would be valuable to our users (mostly ops for this right?) if
> we had an upgrade job where everything *except* nova were upgraded. If
> that's how the majority of people are doing upgrades anyway it seems we
> should make sure that works.
>
> I figure leaving nova at N-1 makes more sense because nova depends on the
> other services (keystone/glance/cinder/neutron) and is likely the harder /
> slower upgrade if you're going to do rolling upgrades of your compute nodes.
>
> This type of job would not run on nova changes on the master branch, since
> those changes would not be exercised in this type of environment. So we'd
> run this on master branch changes to
> keystone/cinder/glance/neutron/trove/designate/etc.
>
> Does that make sense? Would this be valuable at all? Or should the opposite
> be tested where we upgrade nova to N and leave all of the dependent services
> at N-1?
>
> Really looking for operator community feedback here.
>
> [1]
> http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2017-12-19.log.html#t2017-12-19T15:14:15
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Blair Bethwaite]

Hi all - please note this conversation has been split variously across
-dev and -operators.

One small observation from the discussion so far is that it seems as
though there are two issues being discussed under the one banner:
1) maintain old releases for longer
2) do stable releases less frequently

It would be interesting to understand if the people who want longer
maintenance windows would be helped by #2.

On 14 November 2017 at 09:25, Doug Hellmann  wrote:
> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>> >> The concept, in general, is to create a new set of cores from these
>> >> groups, and use 3rd party CI to validate patches. There are lots of
>> >> details to be worked out yet, but our amazing UC (User Committee) will
>> >> be begin working out the details.
>> >
>> > What is the most worrying is the exact "take over" process. Does it mean 
>> > that
>> > the teams will give away the +2 power to a different team? Or will our 
>> > (small)
>> > stable teams still be responsible for landing changes? If so, will they 
>> > have to
>> > learn how to debug 3rd party CI jobs?
>> >
>> > Generally, I'm scared of both overloading the teams and losing the control 
>> > over
>> > quality at the same time :) Probably the final proposal will clarify it..
>>
>> The quality of backported fixes is expected to be a direct (and only?)
>> interest of those new teams of new cores, coming from users and
>> operators and vendors. The more parties to establish their 3rd party
>
> We have an unhealthy focus on "3rd party" jobs in this discussion. We
> should not assume that they are needed or will be present. They may be,
> but we shouldn't build policy around the assumption that they will. Why
> would we have third-party jobs on an old branch that we don't have on
> master, for instance?
>
>> checking jobs, the better proposed changes communicated, which directly
>> affects the quality in the end. I also suppose, contributors from ops
>> world will likely be only struggling to see things getting fixed, and
>> not new features adopted by legacy deployments they're used to maintain.
>> So in theory, this works and as a mainstream developer and maintainer,
>> you need no to fear of losing control over LTS code :)
>>
>> Another question is how to not block all on each over, and not push
>> contributors away when things are getting awry, jobs failing and merging
>> is blocked for a long time, or there is no consensus reached in a code
>> review. I propose the LTS policy to enforce CI jobs be non-voting, as a
>> first step on that way, and giving every LTS team member a core rights
>> maybe? Not sure if that works though.
>
> I'm not sure what change you're proposing for CI jobs and their voting
> status. Do you mean we should make the jobs non-voting as soon as the
> branch passes out of the stable support period?
>
> Regarding the review team, anyone on the review team for a branch
> that goes out of stable support will need to have +2 rights in that
> branch. Otherwise there's no point in saying that they're maintaining
> the branch.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Blair Bethwaite]

I missed this session but the discussion strikes a chord as this is
something I've been saying on my user survey every 6 months.

On 11 November 2017 at 09:51, John Dickinson  wrote:
> What I heard from ops in the room is that they want (to start) one release a
> year who's branch isn't deleted after a year. What if that's exactly what we
> did? I propose that OpenStack only do one release a year instead of two. We
> still keep N-2 stable releases around. We still do backports to all open
> stable branches. We still do all the things we're doing now, we just do it
> once a year instead of twice.

+1

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Sydney lightning talks session

[Blair Bethwaite]

Hi again all,

There's still room for one or two more lightning talks in this session
tomorrow. And as has become tradition there will be a prize for the
best talk thanks to Arkady Kanevsky from Dell!

Please sign up and share your stories - we don't bite.

On 18 October 2017 at 08:27, Blair Bethwaite <blair.bethwa...@gmail.com> wrote:
> Hi all,
>
> Once again the Scientific SIG (nee WG) has a dedicated lightning talk
> session happening at the Sydney Summit. If have any interesting
> OpenStack + Science and/or HPC stories then please through you hat in
> the ring at:
> https://etherpad.openstack.org/p/sydney-scientific-sig-lightning-talks
>
> --
> Cheers,
> ~Blairo



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] today's meeting at 2100UTC (now!) cancelled

[Blair Bethwaite]

Hi all,

Today's meeting is cancelled as the usual chairs are en-route to Sydney!

Apologies for the short notice - timezones meant I only just confirmed this.

-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ceph @ OpenStack Sydney Summit

[Blair Bethwaite]

Hi all,

(Apologies for the shotgun mail.)

Following this up for anyone heading to Sydney in a week. I did end up
getting a Ceph BoF on the program:
https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20490

If you have stuff you'd like to talk about, or especially share,
please shout out and/or add to
https://etherpad.openstack.org/p/SYD-forum-Ceph-OpenStack-BoF.

Also, hope to see some of the core team there!

Cheers,

On 7 July 2017 at 13:47, Blair Bethwaite <blair.bethwa...@gmail.com> wrote:
> Hi all,
>
> Are there any "official" plans to have Ceph events co-hosted with OpenStack
> Summit Sydney, like in Boston?
>
> The call for presentations closes in a week. The Forum will be organised
> throughout September and (I think) that is the most likely place to have
> e.g. Ceph ops sessions like we have in the past. Some of my local colleagues
> have also expressed interest in having a CephFS BoF.
>
> --
> Cheers,
> ~Blairo



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] IRC meeting today at 1100UTC (in 1 hour) in #openstack-meeting

[Blair Bethwaite]

Hi all,

We have an IRC meeting today at 1100 UTC in channel #openstack-meeting.

A light agenda today, mainly looking for input into the SC17 OpenStack
in HPC BOF 
(http://sc17.supercomputing.org/presentation/?id=bof208=sess389).

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Best practice against DDoS on openstack

[Blair Bethwaite]

Similarly, if you have the capability in your compute gear you could do
SR-IOV and push the problem entirely into the instance (but then you miss
out on Neutron secgroups and have to rely entirely on in-instance
firewalls).

Cheers,

On 25 October 2017 at 01:41, Jeremy Stanley  wrote:

> On 2017-10-24 20:18:30 +0900 (+0900), Jean-Philippe Méthot wrote:
> > We’ve just recently been hit on by a low-level DDoS on one of our
> > compute nodes. The attack was fulling our conntrack table while
> > having no noticeable impact on our server load, which is why it
> > took us a while to detect it. Is there any recommended practice
> > regarding server configuration to reduce the impact of a DDoS on
> > the whole compute node and thus, prevent it from going down? I
> > understand that increasing the size of the conntrack table is one,
> > but outside of that?
>
> You might want to look into using iptables -j REJECT -m connlimit
> --connlimit-above some threshold with matches for the individual
> ports' addresses... I'm not a heavy on this end of operations but
> others here probably know how to add hooks for something like that.
> Of course this only moves the denial of service down to the
> individual instance being targeted or used rather than knocking the
> entire compute node offline (hopefully anyway), and is no substitute
> for actual attack mitigation devices/services inline on the network.
> --
> Jeremy Stanley
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>


-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Guest crash and KVM unhandled rdmsr

[Blair Bethwaite]

Hi Saverio,

On 13 October 2017 at 09:05, Saverio Proto  wrote:
> I found this link in my browser history:
> https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/1583819

Thanks. Yes, have seen that one too.

> Is it the same messages that you are seeing in Xenial ?

There are a handful of different MSRs mentioned, and these are not
always the same across each "burst" of unhandled rdmsr log messages.

After a bit of further head-scratching we think the crashes are
actually occurring due to kernel panics following SLUB memory
allocation failures related to heavy Lustre workloads. We've now
started patching Lustre clients for that and seeing the guest crash
stop. At this point I don't have any idea why this seems to correspond
with rdmsr attempts though (and actually it looks like the rdmsr
happens just prior to the SLUB failures).

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] Sydney lightning talks session

[Blair Bethwaite]

Hi all,

Once again the Scientific SIG (nee WG) has a dedicated lightning talk
session happening at the Sydney Summit. If have any interesting
OpenStack + Science and/or HPC stories then please through you hat in
the ring at:
https://etherpad.openstack.org/p/sydney-scientific-sig-lightning-talks

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Guest crash and KVM unhandled rdmsr

[Blair Bethwaite]

Hi all,

Has anyone seen guest crashes/freezes associated with KVM unhandled rdmsr
messages in dmesg on the hypervisor?

We have seen these messages before but never with a strong correlation to
guest problems. However over the past couple of weeks this is happening
almost daily with consistent correlation for a set of hosts dedicated to a
particular HPC workload. So far as I know the workload has not changed, but
we have just recently moved the hypervisors to Ubuntu Xenial (though they
were already on the Xenial kernel previously) and done minor guest
(CentOS7) updates. CPU mode is host-passthrough. Currently trying to figure
out if the CPU flags in the guest have changed since the host upgrade...

Cheers,
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] IRC meeting today at 1100UTC in #openstack-meeting

[Blair Bethwaite]

Hi all,

We have an IRC meeting today at 1100 UTC in channel #openstack-meeting

We are short a couple of chairs today but would like to start planning
out our picks from the Summit schedule and confirming interest in
presentation slots for our Scientific Lightening Talk session. Plus I
have a couple of recent bits of work at Monash I can talk about if
people are interested.

Agenda and meeting details are here:
https://wiki.openstack.org/wiki/Scientific_working_group#IRC_Meeting_October_11th_2017
# Sydney Summit session prep
# Monash OpenStack HPC expansion plans
# Monash Mellanox SR-IOV / RoCE tuning shenanigans

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Blair Bethwaite]

Also CC-ing os-ops as someone else may have encountered this before
and have further/better advice...

On 27 September 2017 at 18:40, Blair Bethwaite
<blair.bethwa...@gmail.com> wrote:
> On 27 September 2017 at 18:14, Stephen Finucane <sfinu...@redhat.com> wrote:
>> What you're probably looking for is the 'reserved_host_memory_mb' option. 
>> This
>> defaults to 512 (at least in the latest master) so if you up this to 4192 or
>> similar you should resolve the issue.
>
> I don't see how this would help given the problem description -
> reserved_host_memory_mb would only help avoid causing OOM when
> launching the last guest that would otherwise fit on a host based on
> Nova's simplified notion of memory capacity. It sounds like both CPU
> and NUMA pinning are in play here, otherwise the host would have no
> problem allocating RAM on a different NUMA node and OOM would be
> avoided.
>
> Jakub, your numbers sound reasonable to me, i.e., use 60 out of 64GB
> when only considering QEMU overhead - however I would expect that
> might  be a problem on NUMA node0 where there will be extra reserved
> memory regions for kernel and devices. In such a configuration where
> you are wanting to pin multiple guests into each of multiple NUMA
> nodes I think you may end up needing different flavor/instance-type
> configs (using less RAM) for node0 versus other NUMA nodes. Suggest
> freshly booting one of your hypervisors and then with no guests
> running take a look at e.g. /proc/buddyinfo/ and /proc/zoneinfo to see
> what memory is used/available and where.
>
> --
> Cheers,
> ~Blairo



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] s/WG/SIG/g

[Blair Bethwaite]

Hi all,

If you happen to have been following along with recent discussions
about introducing OpenStack SIGs then this won't come as a surprise.
PS: the openstack-sig mailing list has been minted - get on it!

The meta-SIG is now looking for existing WGs who wish to convert to
SIGs, see 
http://lists.openstack.org/pipermail/openstack-sigs/2017-July/22.html.
The Scientific-WG is a good candidate for this because, at our core
(as I see it), we've never really been about bounded task-oriented
goals, but more of an open community of OpenStack
operators/architects/users. At any point we may have groups working on
particular goals, e.g., the OpenStack HPC book, performance
benchmarking/troubleshooting, integration architectures, and so on -
these groups could in future be spun out to their own WGs if
warranted.

What does this mean, practically? Essentially we just do some renaming
here and there and move our mailing list discussions to
openstack-s...@lists.openstack.org.

We've already discussed this in the past couple of meetings and so far
had no objections, so we're planning to move ahead with it soon. The
intention of this thread is to canvas broader input.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [scientific] WG IRC meeting: 2100 UTC - opportunistic capacity etc.

[Blair Bethwaite]

Hi Stig,

It occurs to me we have not yet had any discussion on the recent WG->SIG
proposal, which includes the [scientific] posse, so adding that to the
agenda too.

Cheers,

On 5 September 2017 at 19:29, Stig Telfer  wrote:

> Hello all -
>
> We have a Scientific WG IRC meeting today at 2100 UTC in channel
> #openstack-meeting.  Everyone is welcome.
>
> The agenda is here: https://wiki.openstack.org/
> wiki/Scientific_working_group#IRC_Meeting_September_5th_2017
>
> Today we’d like to discuss people’s solutions for making use of
> opportunistic capacity in OpenStack, to achieve maximum utilisation from
> private cloud resources.  We’ve also got updates on the new edition of the
> Scientific OpenStack book, and plenty of upcoming events to plan for.
>
> Cheers,
> Stig
>
>
> ___
> User-committee mailing list
> user-commit...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>
>


-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] GPU passthrough success and failure records

[Blair Bethwaite]

Hi folks,

Related to this, I wonder if anyone has ever seen something like a pci
bus error on a GPU node...? We have a fleet of Dell R730s with dual
K80s and we are periodically seeing the host reset with the hardware
log recording a message like:
"A fatal error was detected on a component at bus 4 device 8 function 0."

Which in this case refers to:
$ lspci -t -d 10b5:8747
-+-[:82]---00.0-[83-85]--+-08.0-[84]--
  |   \-10.0-[85]--
 +-[:03]---00.0-[04-06]--+-08.0-[05]--
  |   \-10.0-[06]--

One of the downstream(?) PCIe endpoint facing ports, i.e., the GPU
side of the PCIe switch.

This error causes the host to unceremoniously reset. No error to be
found anywhere host side, just the hardware log. These are currently
Ubuntu Trusty hosts with 4.4 kernel. GPU burn testing does not seem to
trigger it and the host can go back into production and never (so far)
see the issue again. But we've now seen this about 10 times over the
last 12-18 months across a fleet of ~30 of these hosts (sometimes
twice on the same host months apart, but several distinct hosts
overall).

Cheers,

On 7 May 2017 at 07:55, Blair Bethwaite <blair.bethwa...@gmail.com> wrote:
> Hi all,
>
> I've been (very slowly) working on some docs detailing how to setup an
> OpenStack Nova Libvirt+QEMU-KVM deployment to provide GPU-accelerated
> instances. In Boston I hope to chat to some of the docs team and
> figure out an appropriate upstream guide to fit that into. One of the
> things I'd like to provide is a community record (better than ML
> archives) of what works and doesn't. I've started a first attempt at
> collating some basics here:
> https://etherpad.openstack.org/p/GPU-passthrough-model-success-failure
>
> I know there are at a least a few lurkers out there doing this too so
> please share your own experience. Once there is a bit more data there
> it probably makes sense to convert to a tabular format of some kind
> (but wasn't immediately obvious to me how that should look given there
> are several long list fields)
>
> --
> Cheers,
> ~Blairo



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [OpenStack-docs] [doc] Operations Guide removal

[Blair Bethwaite]

Hi Alex,

I just managed to take a half hour to look at this and have a few
questions/comments towards making a plan for how to proceed with
moving the Ops Guide content to the wiki...

1) Need to define wiki location and structure. Curiously at the moment
there is already meta content at
https://wiki.openstack.org/wiki/Documentation/OpsGuide, Maybe the
content could live at https://wiki.openstack.org/wiki/OpsGuide? I
think it makes sense to follow the existing structure with possible
exception of culling wrong / very-out-of-date content (but perhaps
anything like that should be done as a later step and keep it simple
aiming for a "like-for-like" migration to start with)...?

2) Getting the content into the wiki. Looks like there is no obvious
up-to-date RST import functionality for MediaWiki. Pandoc seems as
though it might support some useful conversions but I didn't try this
yet and don't have any experience with it - can anyone say with
authority whether it is worth pursuing?

3) Future management - obvious can of worms given this is much better
addressed by all the tooling and scaffolding the docs team already
provides around the repos... but nonetheless some expectations may
need to be set upfront to avoid future pain.

Cheers,

On 15 July 2017 at 01:48, Alexandra Settle  wrote:
> Hi operators,
>
> Please be aware that I have proposed the following patch:
> https://review.openstack.org/#/c/483937/
>
>
>
> This removes the Operations Guide from the openstack-manuals repo and will
> no longer be accessible via docs.openstack.org after the patch merges.
>
> The documentation team does not have the resources to move the ops guide to
> the wiki ourselves. If you are able to work on the migration, please check
> out the ‘before-migration’ tag in the repo to access the deleted content.
> [0]
>
> Once the ops guide is migrated to the wiki, we will help you add a redirect
> so that the old link on docs.openstack.org will allow users to find the
> content in the new location.
>
> Thanks,
>
> Alex
>
>
>
> [0] https://github.com/openstack/openstack-manuals/tree/before-migration
>
>
> ___
> OpenStack-docs mailing list
> openstack-d...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
>



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [scientific] 0900 UTC meeting time change?

[Blair Bethwaite]

Hi all,

Review up for this: https://review.openstack.org/478317

Cheers,

On 21 June 2017 at 18:54, Stig Telfer <stig.openst...@telfer.org> wrote:

> Doing future meetings at 1100UTC would also be fine by me.  I’ll put it on
> the agenda for today’s meeting (in ~10 minutes)
>
> Stig
>
>
> On 21 Jun 2017, at 09:13, Blair Bethwaite <blair.bethwa...@monash.edu>
> wrote:
>
> Thanks Pierre. That's also my preference.
>
> Just to be clear, today's 0900 UTC meeting (45 mins from now) is going
> ahead at the usual time.
>
> On 21 Jun. 2017 5:21 pm, "Pierre Riteau" <prit...@uchicago.edu> wrote:
>
> Hi Blair,
>
> I strongly prefer 1100 UTC.
>
> Pierre
>
> > On 21 Jun 2017, at 06:54, Blair Bethwaite <blair.bethwa...@monash.edu>
> wrote:
> >
> > Hi all,
> >
> > The Scientific-WG's 0900 UTC meeting time (it's the non-US friendly
> time) is increasingly difficult for me to make. A couple of meetings back
> we discussed changing it and had general agreement. The purpose here is to
> get a straw poll of preferences for -2 or +2 to the current time, i.e., do
> you prefer 0700 or 1100 UTC instead (subject to meeting channel
> availability)?
> >
> > Cheers,
> > b1airo
> > ___
> > User-committee mailing list
> > user-commit...@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>


-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [openstack-dev] [all][tc] Turning TC/UC workgroups into OpenStack SIGs

[Blair Bethwaite]

There is a not insignificant degree of irony in the fact that this
conversation has splintered so that anyone only reading openstack-operators
and/or user-committee is missing 90% of the picture Maybe I just need a
new ML management strategy.

I'd like to add a +1 to Sean's suggestion about WG/SIG/team/whatever tags
on reviews etc. This is something I've also suggested in the past:
http://lists.openstack.org/pipermail/user-committee/2016-October/001328.html.
My thinking at the time was that it would provide a tractable basis for
chairs to build standing discussion items around and help get more user &
ops eyes on blueprints/reviews/etc.

On 27 June 2017 at 10:25, Melvin Hillsman  wrote:

>
>
> On Wed, Jun 21, 2017 at 11:55 AM, Matt Riedemann 
> wrote:
>
>> On 6/21/2017 11:17 AM, Shamail Tahir wrote:
>>
>>>
>>>
>>> On Wed, Jun 21, 2017 at 12:02 PM, Thierry Carrez >> > wrote:
>>>
>>> Shamail Tahir wrote:
>>> > In the past, governance has helped (on the UC WG side) to reduce
>>> > overlaps/duplication in WGs chartered for similar objectives. I
>>> would
>>> > like to understand how we will handle this (if at all) with the
>>> new SIG
>>> > proposa?
>>>
>>> I tend to think that any overlap/duplication would get solved
>>> naturally,
>>> without having to force everyone through an application process that
>>> may
>>> discourage natural emergence of such groups. I feel like an
>>> application
>>> process would be premature optimization. We can always encourage
>>> groups
>>> to merge (or clean them up) after the fact. How much
>>> overlaps/duplicative groups did you end up having ?
>>>
>>>
>>> Fair point, it wasn't many. The reason I recalled this effort was
>>> because we had to go through the exercise after the fact and that made the
>>> volume of WGs to review much larger than had we asked the purpose whenever
>>> they were created. As long as we check back periodically and not let the
>>> work for validation/clean up pile up then this is probably a non-issue.
>>>
>>>
>>> > Also, do we have to replace WGs as a concept or could SIG
>>> > augment them? One suggestion I have would be to keep projects on
>>> the TC
>>> > side and WGs on the UC side and then allow for spin-up/spin-down
>>> of SIGs
>>> > as needed for accomplishing specific goals/tasks (picture of a
>>> diagram
>>> > I created at the Forum[1]).
>>>
>>> I feel like most groups should be inclusive of all community, so I'd
>>> rather see the SIGs being the default, and ops-specific or
>>> dev-specific
>>> groups the exception. To come back to my Public Cloud WG example, you
>>> need to have devs and ops in the same group in the first place before
>>> you would spin-up a "address scalability" SIG. Why not just have a
>>> Public Cloud SIG in the first place?
>>>
>>>
>>> +1, I interpreted originally that each use-case would be a SIG versus
>>> the SIG being able to be segment oriented (in which multiple use-cases
>>> could be pursued)
>>>
>>>
>>>  > [...]
>>> > Finally, how will this change impact the ATC/AUC status of the SIG
>>> > members for voting rights in the TC/UC elections?
>>>
>>> There are various options. Currently you give UC WG leads the AUC
>>> status. We could give any SIG lead both statuses. Or only give the
>>> AUC
>>> status to a subset of SIGs that the UC deems appropriate. It's
>>> really an
>>> implementation detail imho. (Also I would expect any SIG lead to
>>> already
>>> be both AUC and ATC somehow anyway, so that may be a non-issue).
>>>
>>>
>>> We can discuss this later because it really is an implementation detail.
>>> Thanks for the answers.
>>>
>>>
>>> --
>>> Thierry Carrez (ttx)
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> >> subscribe>
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Shamail Tahir
>>> t: @ShamailXD
>>> tz: Eastern Time
>>>
>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> I think a key point you're going to want to convey and repeat ad nauseum
>> with this SIG idea is that each SIG is focused on a specific use case and
>> they can be spun up and spun down. Assuming that's what you want them to be.
>>
>> One 

Re: [Openstack-operators] [dev] [doc] Operations Guide future

[Blair Bethwaite]

Hi Alex,

On 2 June 2017 at 23:13, Alexandra Settle  wrote:
> O I like your thinking – I’m a pandoc fan, so, I’d be interested in
> moving this along using any tools to make it easier.

I can't realistically offer much time on this but I would be happy to
help (ad-hoc) review/catalog/clean-up issues with export.

> I think my only proviso (now I’m thinking about it more) is that we still
> have a link on docs.o.o, but it goes to the wiki page for the Ops Guide.

Agreed, need to maintain discoverability.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [scientific] 0900 UTC meeting time change?

[Blair Bethwaite]

Thanks Pierre. That's also my preference.

Just to be clear, today's 0900 UTC meeting (45 mins from now) is going
ahead at the usual time.

On 21 Jun. 2017 5:21 pm, "Pierre Riteau" <prit...@uchicago.edu> wrote:

Hi Blair,

I strongly prefer 1100 UTC.

Pierre

> On 21 Jun 2017, at 06:54, Blair Bethwaite <blair.bethwa...@monash.edu>
wrote:
>
> Hi all,
>
> The Scientific-WG's 0900 UTC meeting time (it's the non-US friendly time)
is increasingly difficult for me to make. A couple of meetings back we
discussed changing it and had general agreement. The purpose here is to get
a straw poll of preferences for -2 or +2 to the current time, i.e., do you
prefer 0700 or 1100 UTC instead (subject to meeting channel availability)?
>
> Cheers,
> b1airo
> ___
> User-committee mailing list
> user-commit...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] 0900 UTC meeting time change?

[Blair Bethwaite]

Hi all,

The Scientific-WG's 0900 UTC meeting time (it's the non-US friendly time)
is increasingly difficult for me to make. A couple of meetings back we
discussed changing it and had general agreement. The purpose here is to get
a straw poll of preferences for -2 or +2 to the current time, i.e., do you
prefer 0700 or 1100 UTC instead (subject to meeting channel availability)?

Cheers,
b1airo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [dev] [doc] Operations Guide future

[Blair Bethwaite]

Hi Alex,

Likewise for option 3. If I recall correctly from the summit session
that was also the main preference in the room?

On 2 June 2017 at 11:15, George Mihaiescu  wrote:
> +1 for option 3
>
>
>
> On Jun 1, 2017, at 11:06, Alexandra Settle  wrote:
>
> Hi everyone,
>
>
>
> I haven’t had any feedback regarding moving the Operations Guide to the
> OpenStack wiki. I’m not taking silence as compliance. I would really like to
> hear people’s opinions on this matter.
>
>
>
> To recap:
>
>
>
> Option one: Kill the Operations Guide completely and move the Administration
> Guide to project repos.
> Option two: Combine the Operations and Administration Guides (and then this
> will be moved into the project-specific repos)
> Option three: Move Operations Guide to OpenStack wiki (for ease of
> operator-specific maintainability) and move the Administration Guide to
> project repos.
>
>
>
> Personally, I think that option 3 is more realistic. The idea for the last
> option is that operators are maintaining operator-specific documentation and
> updating it as they go along and we’re not losing anything by combining or
> deleting. I don’t want to lose what we have by going with option 1, and I
> think option 2 is just a workaround without fixing the problem – we are not
> getting contributions to the project.
>
>
>
> Thoughts?
>
>
>
> Alex
>
>
>
> From: Alexandra Settle 
> Date: Friday, May 19, 2017 at 1:38 PM
> To: Melvin Hillsman , OpenStack Operators
> 
> Subject: Re: [Openstack-operators] Fwd: [openstack-dev] [openstack-doc]
> [dev] What's up doc? Summit recap edition
>
>
>
> Hi everyone,
>
>
>
> Adding to this, I would like to draw your attention to the last dot point of
> my email:
>
>
>
> “One of the key takeaways from the summit was the session that I joint
> moderated with Melvin Hillsman regarding the Operations and Administration
> Guides. You can find the etherpad with notes here:
> https://etherpad.openstack.org/p/admin-ops-guides  The session was really
> helpful – we were able to discuss with the operators present the current
> situation of the documentation team, and how they could help us maintain the
> two guides, aimed at the same audience. The operator’s present at the
> session agreed that the Administration Guide was important, and could be
> maintained upstream. However, they voted and agreed that the best course of
> action for the Operations Guide was for it to be pulled down and put into a
> wiki that the operators could manage themselves. We will be looking at
> actioning this item as soon as possible.”
>
>
>
> I would like to go ahead with this, but I would appreciate feedback from
> operators who were not able to attend the summit. In the etherpad you will
> see the three options that the operators in the room recommended as being
> viable, and the voted option being moving the Operations Guide out of
> docs.openstack.org into a wiki. The aim of this was to empower the
> operations community to take more control of the updates in an environment
> they are more familiar with (and available to others).
>
>
>
> What does everyone think of the proposed options? Questions? Other thoughts?
>
>
>
> Alex
>
>
>
> From: Melvin Hillsman 
> Date: Friday, May 19, 2017 at 1:30 PM
> To: OpenStack Operators 
> Subject: [Openstack-operators] Fwd: [openstack-dev] [openstack-doc] [dev]
> What's up doc? Summit recap edition
>
>
>
>
>
> -- Forwarded message --
> From: Alexandra Settle 
> Date: Fri, May 19, 2017 at 6:12 AM
> Subject: [openstack-dev] [openstack-doc] [dev] What's up doc? Summit recap
> edition
> To: "openstack-d...@lists.openstack.org"
> 
> Cc: "OpenStack Development Mailing List (not for usage questions)"
> 
>
>
> Hi everyone,
>
>
> The OpenStack manuals project had a really productive week at the OpenStack
> summit in Boston. You can find a list of all the etherpads and attendees
> here: https://etherpad.openstack.org/p/docs-summit
>
>
>
> As we all know, we are rapidly losing key contributors and core reviewers.
> We are not alone, this is happening across the board. It is making things
> harder, but not impossible. Since our inception in 2010, we’ve been climbing
> higher and higher trying to achieve the best documentation we could, and
> uphold our high standards. This is something to be incredibly proud of.
> However, we now need to take a step back and realise that the amount of work
> we are attempting to maintain is now out of reach for the team size that we
> have. At the moment we have 13 cores, of which none are full time
> contributors or reviewers. This includes myself.
>
>
>
> That being said! I have spent the last week at the summit talking to some of
> our leaders, including Doug 

Re: [Openstack-operators] [nova][ironic][scheduler][placement] IMPORTANT: NOT Getting rid of the automated reschedule functionality

[Blair Bethwaite]

Thanks Jay,

I wonder whether there is an easy-ish way to collect stats about the
sorts of errors deployers see in that catchall, so that when this
comes back around in a release or two there might be some less
anecdotal data available...?

Cheers,

On 24 May 2017 at 06:43, Jay Pipes  wrote:
> Hello Dear Operators,
>
> OK, we've heard you loud and (mostly) clear. We won't remove the automated
> rescheduling behavior from Nova. While we will be removing the primary cause
> of reschedules (resource overconsumption races), we cannot yet eliminate the
> catchall exception handling on the compute node that triggers a retry of the
> instance launch. Nor will we be able to fully ameliorate the
> affinity/anti-affinity last-minute violation problems for at least another
> release.
>
> So, we'll continue to support basic retries within the originally-selected
> cell.
>
> Best,
> -jay
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][ironic][scheduler][placement] IMPORTANT: Getting rid of the automated reschedule functionality

[Blair Bethwaite]

On 23 May 2017 at 05:33, Dan Smith  wrote:
> Sure, the diaper exception is rescheduled currently. That should
> basically be things like misconfiguration type things. Rescheduling
> papers over those issues, which I don't like, but in the room it surely
> seemed like operators thought that they still needed to be handled.

Operators don't want retries to mask configuration issues (appropriate
errors should still be captured in places where operators can process
them on a regular basis) but what they want even less is any further
complexity or "soft" failures exposed to end-users.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Fwd: [AOSUG] OpenStack Australia Day Melbourne - 1 month to go!

[Blair Bethwaite]

Aussie Stackers, get amongst it!

-- Forwarded message --
From: Jessica Field <aosug-annou...@meetup.com>
Date: 3 May 2017 at 16:25
Subject: [AOSUG] OpenStack Australia Day Melbourne - 1 month to go!
To: aosug-annou...@meetup.com


Hi OzStackers,

There’s now less than one month to go until OpenStack Australia Day comes
to Melbourne!

We’ve added a few speakers to the website to give you an idea of what to
expect on the day. So far, the agenda features local speakers from the
University
of Melbourne <http://www.unimelb.edu.au/>, Monash University
<https://www.monash.edu/> and the National eResearch Collaboration Tools
and Resources project (Nectar <https://nectar.org.au/>), as well as
international speakers from the OpenStack Foundation <http://openstack.org/>
, Cloudify by Gigaspaces <https://www.gigaspaces.com/>, EasyStack
<http://www.easystack.cn/en/> and more.

Speaker submissions are open until the 8th of May, so there’s still time to
submit your presentation. To submit a talk, please visit:
australiaday.openstack.org.au

OpenStack Australia Day Melbourne is supported by Aptira
<http://aptira.com/>, Veritas <http://veritas.com/>, Cumulus Networks
<https://www.cumulusnetworks.com/>, Rackspace
<https://www.rackspace.com/en-au>, RedHat <https://www.redhat.com/en>, SUSE
<https://www.suse.com/> and The Register <https://www.theregister.co.uk/>.
If you’d like to be involved, please let us know as soon as possible. We’ve
added a range of unique add-on packages including food carts and cocktail
packages to help draw more attention to your sponsor area on the day. Oh –
did we mention there’s a donut wall?! For more information or to request a
sponsorship pack, please visit australiaday.openstack.org.au.

Tickets are still on sale, but spaces are limited. Register today
<http://australiaday.openstack.org.au/> to avoid disappointment. I hope to
see you all there!

Jess




--
This message was sent by Meetup on behalf of Jessica Field
<https://www.meetup.com/Australian-OpenStack-User-Group/members/42165752/>
from Australian OpenStack User Group
<https://www.meetup.com/Australian-OpenStack-User-Group/>.
To report this message, please click here
<https://www.meetup.com/report_abuse/mailing_list/80005514/>
To block the sender of this message, please click here
<https://www.meetup.com/members/42165752/>
To unsubscribe from special announcements from your Organizer(s), click here
<https://www.meetup.com/Australian-OpenStack-User-Group/optout/?submit=true&_ms_unsub=true=orgBdcst>

Meetup, POB 4668 #37895 NY NY USA 10163 <#m_1026249296055576842_> |
supp...@meetup.com



-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova][blazar][scientific] advanced instance scheduling: reservations and preeemption - Forum session

[Blair Bethwaite]

On 2 May 2017 at 05:50, Jay Pipes  wrote:
> Masahito Muroi is currently marked as the moderator, but I will indeed be
> there and happy to assist Masahito in moderating, no problem.

The more the merrier :-).

There is a rather unfortunate clash here with the Scientific-WG BoF
session. Location permitting I will head to that first and then run to
catch-up on the end of this.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [nova][blazar][scientific] advanced instance scheduling: reservations and preeemption - Forum session

[Blair Bethwaite]

Hi all,

Following up to the recent thread "[Openstack-operators] [scientific]
Resource reservation requirements (Blazar) - Forum session" and adding
openstack-dev.

This is now a confirmed forum session
(https://www.openstack.org/summit/boston-2017/summit-schedule/events/18781/advanced-instance-scheduling-reservations-and-preemption)
to cover any advanced scheduling use-cases people want to talk about,
but in particular focusing on reservations and preemption as they are
big priorities particularly for scientific deployers.

Etherpad draft is
https://etherpad.openstack.org/p/BOS-forum-advanced-instance-scheduling,
please attend and contribute! In particular I'd appreciate background
spec and review links added to the etherpad.

Jay, would you be able and interested to moderate this from the Nova side?

Cheers,

On 12 April 2017 at 05:22, Jay Pipes <jaypi...@gmail.com> wrote:
> On 04/11/2017 02:08 PM, Pierre Riteau wrote:
>>>
>>> On 4 Apr 2017, at 22:23, Jay Pipes <jaypi...@gmail.com
>>> <mailto:jaypi...@gmail.com>> wrote:
>>>
>>> On 04/04/2017 02:48 PM, Tim Bell wrote:
>>>>
>>>> Some combination of spot/OPIE
>>>
>>>
>>> What is OPIE?
>>
>>
>> Maybe I missed a message: I didn’t see any reply to Jay’s question about
>> OPIE.
>
>
> Thanks!
>
>> OPIE is the OpenStack Preemptible Instances
>> Extension: https://github.com/indigo-dc/opie
>> I am sure other on this list can provide more information.
>
>
> Got it.
>
>> I think running OPIE instances inside Blazar reservations would be
>> doable without many changes to the implementation.
>> We’ve talked about this idea several times, this forum session would be
>> an ideal place to draw up an implementation plan.
>
>
> I just looked through the OPIE source code. One thing I'm wondering is why
> the code for killing off pre-emptible instances is being done in the
> filter_scheduler module?
>
> Why not have a separate service that merely responds to the raising of a
> NoValidHost exception being raised from the scheduler with a call to go and
> terminate one or more instances that would have allowed the original request
> to land on a host?
>
> Right here is where OPIE goes and terminates pre-emptible instances:
>
> https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L92-L100
>
> However, that code should actually be run when line 90 raises NoValidHost:
>
> https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L90
>
> There would be no need at all for "detecting overcommit" here:
>
> https://github.com/indigo-dc/opie/blob/master/opie/scheduler/filter_scheduler.py#L96
>
> Simply detect a NoValidHost being returned to the conductor from the
> scheduler, examine if there are pre-emptible instances currently running
> that could be terminated and terminate them, and re-run the original call to
> select_destinations() (the scheduler call) just like a Retry operation
> normally does.
>
> There's be no need whatsoever to involve any changes to the scheduler at
> all.
>
>>>> and Blazar would seem doable as long as the resource provider
>>>> reserves capacity appropriately (i.e. spot resources>>blazar
>>>> committed along with no non-spot requests for the same aggregate).
>>>> Is this feasible?
>
>
> No. :)
>
> As mentioned in previous emails and on the etherpad here:
>
> https://etherpad.openstack.org/p/new-instance-reservation
>
> I am firmly against having the resource tracker or the placement API
> represent inventory or allocations with a temporal aspect to them (i.e.
> allocations in the future).
>
> A separate system (hopefully Blazar) is needed to manage the time-based
> associations to inventories of resources over a period in the future.
>
> Best,
> -jay
>
>>> I'm not sure how the above is different from the constraints I mention
>>> below about having separate sets of resource providers for preemptible
>>> instances than for non-preemptible instances?
>>>
>>> Best,
>>> -jay
>>>
>>>> Tim
>>>>
>>>> On 04.04.17, 19:21, "Jay Pipes" <jaypi...@gmail.com
>>>> <mailto:jaypi...@gmail.com>> wrote:
>>>>
>>>>On 04/03/2017 06:07 PM, Blair Bethwaite wrote:
>>>>> Hi Jay,
>>>>>
>>>>> On 4 April 2017 at 00:20, Jay Pipes <jaypi...@gmail.com
>>>> <mailto:jaypi...@gmail.com>> wrote:
>>>>>> However, implementing the above in any useful fash

Re: [Openstack-operators] [openstack-dev] [scientific][nova][cyborg] Special Hardware Forum session

[Blair Bethwaite]

Thanks Rochelle. I encourage everyone to dump thoughts into the
etherpad (https://etherpad.openstack.org/p/BOS-forum-special-hardware
- feel free to garden it as you go!) so we can have some chance of
organising a coherent session. In particular it would be useful to
know what is going to be most useful for the Nova and Cyborg devs so
that we can give that priority before we start the show-and-tell /
knowledge-share that is often a large part of these sessions. I'd also
be very happy to have a co-moderator if any wants to volunteer.

On 26 April 2017 at 03:11, Rochelle Grober  wrote:
>
> I know that some cyborg folks and nova folks are planning to be there. Now
> we need to drive some ops folks.
>
>
> Sent from HUAWEI AnyOffice
> From:Blair Bethwaite
> To:openstack-...@lists.openstack.org,openstack-oper.
> Date:2017-04-25 08:24:34
> Subject:[openstack-dev] [scientific][nova][cyborg] Special Hardware Forum
> session
>
> Hi all,
>
> A quick FYI that this Forum session exists:
> https://www.openstack.org/summit/boston-2017/summit-schedule/events/18803/special-hardware
> (https://etherpad.openstack.org/p/BOS-forum-special-hardware) is a
> thing this Forum.
>
> It would be great to see a good representation from both the Nova and
> Cyborg dev teams, and also ops ready to share their experience and
> use-cases.
>
> --
> Cheers,
> ~Blairo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova][glance] Who needs multiple api_servers?

[Blair Bethwaite]

On 29 April 2017 at 01:46, Mike Dorman  wrote:
> I don’t disagree with you that the client side choose-a-server-at-random is 
> not a great load balancer.  (But isn’t this roughly the same thing that 
> oslo-messaging does when we give it a list of RMQ servers?)  For us it’s more 
> about the failure handling if one is down than it is about actually equally 
> distributing the load.

Maybe not great, but still better than making operators deploy (often
complex) full-featured external LBs when they really just want
*enough* redundancy. In many cases this seems to just create pets in
the control plane. I think it'd be useful if all OpenStack APIs and
their clients actively handled this poor-man's HA without having to
resort to haproxy etc, or e.g., assuming operators own the DNS.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova][glance] Who needs multiple api_servers?

[Blair Bethwaite]

On 28 April 2017 at 21:17, Sean Dague <s...@dague.net> wrote:
> On 04/28/2017 12:50 AM, Blair Bethwaite wrote:
>> We at Nectar are in the same boat as Mike. Our use-case is a little
>> bit more about geo-distributed operations though - our Cells are in
>> different States around the country, so the local glance-apis are
>> particularly important for caching popular images close to the
>> nova-computes. We consider these glance-apis as part of the underlying
>> cloud infra rather than user-facing, so I think we'd prefer not to see
>> them in the service-catalog returned to users either... is there going
>> to be a (standard) way to hide them?
>
> In a situation like this, where Cells are geographically bounded, is
> there also a Region for that Cell/Glance?

Hi Sean. Nope, just the one global region and set of user-facing APIs.
Those other glance-apis are internal architectural details and should
be hidden from the public catalog so as not to confuse users and/or
over-expose information.

Cheers,

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [nova][glance] Who needs multiple api_servers?

[Blair Bethwaite]

We at Nectar are in the same boat as Mike. Our use-case is a little
bit more about geo-distributed operations though - our Cells are in
different States around the country, so the local glance-apis are
particularly important for caching popular images close to the
nova-computes. We consider these glance-apis as part of the underlying
cloud infra rather than user-facing, so I think we'd prefer not to see
them in the service-catalog returned to users either... is there going
to be a (standard) way to hide them?

On 28 April 2017 at 09:15, Mike Dorman  wrote:
> We make extensive use of the [glance]/api_servers list.  We configure that on 
> hypervisors to direct them to Glance servers which are more “local” 
> network-wise (in order to reduce network traffic across security 
> zones/firewalls/etc.)  This way nova-compute can fail over in case one of the 
> Glance servers in the list is down, without putting them behind a load 
> balancer.  We also don’t run https for these “internal” Glance calls, to save 
> the overhead when transferring images.
>
> End-user calls to Glance DO go through a real load balancer and then are 
> distributed out to the Glance servers on the backend.  From the end-user’s 
> perspective, I totally agree there should be one, and only one URL.
>
> However, we would be disappointed to see the change you’re suggesting 
> implemented.  We would lose the redundancy we get now by providing a list.  
> Or we would have to shunt all the calls through the user-facing endpoint, 
> which would generate a lot of extra traffic (in places where we don’t want 
> it) for image transfers.
>
> Thanks,
> Mike
>
>
>
> On 4/27/17, 4:02 PM, "Matt Riedemann"  wrote:
>
> On 4/27/2017 4:52 PM, Eric Fried wrote:
> > Y'all-
> >
> >   TL;DR: Does glance ever really need/use multiple endpoint URLs?
> >
> >   I'm working on bp use-service-catalog-for-endpoints[1], which intends
> > to deprecate disparate conf options in various groups, and centralize
> > acquisition of service endpoint URLs.  The idea is to introduce
> > nova.utils.get_service_url(group) -- note singular 'url'.
> >
> >   One affected conf option is [glance]api_servers[2], which currently
> > accepts a *list* of endpoint URLs.  The new API will only ever return 
> *one*.
> >
> >   Thus, as planned, this blueprint will have the side effect of
> > deprecating support for multiple glance endpoint URLs in Pike, and
> > removing said support in Queens.
> >
> >   Some have asserted that there should only ever be one endpoint URL for
> > a given service_type/interface combo[3].  I'm fine with that - it
> > simplifies things quite a bit for the bp impl - but wanted to make sure
> > there were no loudly-dissenting opinions before we get too far down this
> > path.
> >
> > [1]
> > 
> https://blueprints.launchpad.net/nova/+spec/use-service-catalog-for-endpoints
> > [2]
> > 
> https://github.com/openstack/nova/blob/7e7bdb198ed6412273e22dea72e37a6371fce8bd/nova/conf/glance.py#L27-L37
> > [3]
> > 
> http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-04-27.log.html#t2017-04-27T20:38:29
> >
> > Thanks,
> > Eric Fried (efried)
> > .
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> +openstack-operators
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific][nova][cyborg] Special Hardware Forum session

[Blair Bethwaite]

Hi all,

A quick FYI that this Forum session exists:
https://www.openstack.org/summit/boston-2017/summit-schedule/events/18803/special-hardware
(https://etherpad.openstack.org/p/BOS-forum-special-hardware) is a
thing this Forum.

It would be great to see a good representation from both the Nova and
Cyborg dev teams, and also ops ready to share their experience and
use-cases.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

[Blair Bethwaite]

Hi Tim,

It does seem feasible, but imagine the aggregate juggling... it's
something of an indictment that from where we are today this seems
like a step forward. I'm not a fan of pushing that load onto operators
when it seems like what we actually need is fully-fledged workload
scheduling in Nova.

Cheers,

On 5 April 2017 at 04:48, Tim Bell <tim.b...@cern.ch> wrote:
> Some combination of spot/OPIE and Blazar would seem doable as long as the 
> resource provider reserves capacity appropriately (i.e. spot 
> resources>>blazar committed along with no non-spot requests for the same 
> aggregate).
>
> Is this feasible?
>
> Tim
>
> On 04.04.17, 19:21, "Jay Pipes" <jaypi...@gmail.com> wrote:
>
> On 04/03/2017 06:07 PM, Blair Bethwaite wrote:
> > Hi Jay,
> >
> > On 4 April 2017 at 00:20, Jay Pipes <jaypi...@gmail.com> wrote:
> >> However, implementing the above in any useful fashion requires that 
> Blazar
> >> be placed *above* Nova and essentially that the cloud operator turns 
> off
> >> access to Nova's  POST /servers API call for regular users. Because if 
> not,
> >> the information that Blazar acts upon can be simply circumvented by 
> any user
> >> at any time.
> >
> > That's something of an oversimplification. A reservation system
> > outside of Nova could manipulate Nova host-aggregates to "cordon off"
> > infrastructure from on-demand access (I believe Blazar already uses
> > this approach), and it's not much of a jump to imagine operators being
> > able to twiddle the available reserved capacity in a finite cloud so
> > that reserved capacity can be offered to the subset of users/projects
> > that need (or perhaps have paid for) it.
>
> Sure, I'm following you up until here.
>
> > Such a reservation system would even be able to backfill capacity
> > between reservations. At the end of the reservation the system
> > cleans-up any remaining instances and preps for the next
> > reservation.
>
> By "backfill capacity between reservations", do you mean consume
> resources on the compute hosts that are "reserved" by this paying
> customer at some date in the future? i.e. Spot instances that can be
> killed off as necessary by the reservation system to free resources to
> meet its reservation schedule?
>
> > The are a couple of problems with putting this outside of Nova though.
> > The main issue is that pre-emptible/spot type instances can't be
> > accommodated within the on-demand cloud capacity.
>
> Correct. The reservation system needs complete control over a subset of
> resource providers to be used for these spot instances. It would be like
> a hotel reservation system being used for a motel where cars could
> simply pull up to a room with a vacant sign outside the door. The
> reservation system would never be able to work on accurate data unless
> some part of the motel's rooms were carved out for reservation system to
> use and cars to not pull up and take.
>
>  >  You could have the
> > reservation system implementing this feature, but that would then put
> > other scheduling constraints on the cloud in order to be effective
> > (e.g., there would need to be automation changing the size of the
> > on-demand capacity so that the maximum pre-emptible capacity was
> > always available). The other issue (admittedly minor, but still a
> > consideration) is that it's another service - personally I'd love to
> > see Nova support these advanced use-cases directly.
>
> Welcome to the world of microservices. :)
>
> -jay
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

[Blair Bethwaite]

Hi Jay,

On 5 April 2017 at 03:21, Jay Pipes <jaypi...@gmail.com> wrote:
> On 04/03/2017 06:07 PM, Blair Bethwaite wrote:
>> That's something of an oversimplification. A reservation system
>> outside of Nova could manipulate Nova host-aggregates to "cordon off"
>> infrastructure from on-demand access (I believe Blazar already uses
>> this approach), and it's not much of a jump to imagine operators being
>> able to twiddle the available reserved capacity in a finite cloud so
>> that reserved capacity can be offered to the subset of users/projects
>> that need (or perhaps have paid for) it.
>
>
> Sure, I'm following you up until here.
>
>> Such a reservation system would even be able to backfill capacity
>> between reservations. At the end of the reservation the system
>> cleans-up any remaining instances and preps for the next
>> reservation.
>
>
> By "backfill capacity between reservations", do you mean consume resources
> on the compute hosts that are "reserved" by this paying customer at some
> date in the future? i.e. Spot instances that can be killed off as necessary
> by the reservation system to free resources to meet its reservation
> schedule?

That is one possible use-case, but it could also backfill with other
reservations that do not overlap. This is a common feature of HPC job
schedulers that have to deal with the competing needs of large
parallel jobs (single users with temporal workload constraints) and
many small jobs (many users with throughput needs).

>> The are a couple of problems with putting this outside of Nova though.
>> The main issue is that pre-emptible/spot type instances can't be
>> accommodated within the on-demand cloud capacity.
>
>
> Correct. The reservation system needs complete control over a subset of
> resource providers to be used for these spot instances. It would be like a
> hotel reservation system being used for a motel where cars could simply pull
> up to a room with a vacant sign outside the door. The reservation system
> would never be able to work on accurate data unless some part of the motel's
> rooms were carved out for reservation system to use and cars to not pull up
> and take.

In order to make reservations, yes. However, preemptible instances are
a valid use-case without also assuming reservations (they just happen
to complement each other). If we want the system to be really useful
and flexible we should be considering leases and queuing, e.g.:

- Leases requiring a single VM or groups of VMs that must run in parallel.
- Best-effort leases, which will wait in a queue until resources
become available.
- Advance reservation leases, which must start at a specific time.
- Immediate leases, which must start right now, or not at all.

The above bullets are pulled from
http://haizea.cs.uchicago.edu/whatis.html (Haizea is a scheduling
framework that can plug into OpenNebula), and I believe these fit very
well with the scheduling needs of the majority of private & hybrid
clouds. It also has other notable features such as preemptible leases.

I remain perplexed by the fact that OpenStack, as the preeminent open
private cloud framework, still only deals in on-demand access as
though most cloud-deployments are infinite. Yet today users have to
keep polling the boot API until they get something: "not now... not
now... not now..." - no queuing, no fair-share, nothing. Users should
only ever see NoValidHost if they requested "an instance now or not at
all".

I do not mean to ignore the existence of Blazar here, but development
on that has only recently started up again and part of the challenge
for Blazar is that resource leases, even simple whole compute nodes,
don't seem to have ever been well supported in Nova.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

[Blair Bethwaite]

Hi Jay,

On 4 April 2017 at 00:20, Jay Pipes  wrote:
> However, implementing the above in any useful fashion requires that Blazar
> be placed *above* Nova and essentially that the cloud operator turns off
> access to Nova's  POST /servers API call for regular users. Because if not,
> the information that Blazar acts upon can be simply circumvented by any user
> at any time.

That's something of an oversimplification. A reservation system
outside of Nova could manipulate Nova host-aggregates to "cordon off"
infrastructure from on-demand access (I believe Blazar already uses
this approach), and it's not much of a jump to imagine operators being
able to twiddle the available reserved capacity in a finite cloud so
that reserved capacity can be offered to the subset of users/projects
that need (or perhaps have paid for) it. Such a reservation system
would even be able to backfill capacity between reservations. At the
end of the reservation the system cleans-up any remaining instances
and preps for the next reservation.

The are a couple of problems with putting this outside of Nova though.
The main issue is that pre-emptible/spot type instances can't be
accommodated within the on-demand cloud capacity. You could have the
reservation system implementing this feature, but that would then put
other scheduling constraints on the cloud in order to be effective
(e.g., there would need to be automation changing the size of the
on-demand capacity so that the maximum pre-emptible capacity was
always available). The other issue (admittedly minor, but still a
consideration) is that it's another service - personally I'd love to
see Nova support these advanced use-cases directly.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session

[Blair Bethwaite]

Hi all,

The below was suggested for a Forum session but we don't yet have a
submission or name to chair/moderate. I, for one, would certainly be
interested in providing input. Do we have any owners out there?

Resource reservation requirements:
==
The Blazar project [https://wiki.openstack.org/wiki/Blazar] has been
revived following Barcelona and will soon release a new version. Now
is a good time to get involved and share requirements with the
community. Our development priorities are described through Blueprints
on Launchpad: https://blueprints.launchpad.net/blazar

In particular, support for pre-emptible instances could be combined
with resource reservation to maximize utilization on unreserved
resources.+1

Is Blazar the right project to discuss reservations of finite
consumable resources like software licenses?

  Blazar would like to ultimately support many different kinds of
resources (volumes, floating IPs, etc.). Software licenses can be
another type.
==
(https://etherpad.openstack.org/p/BOS-UC-brainstorming-scientific-wg)

Cheers,

-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] backup to object store - tool recommendations

[Blair Bethwaite]

Hi Saverio. Looks like we will have that issue within a week or two once we
finally upgrade our rgw (lagging behind the rest of our cluster). I will
try logging​ this as a priority with Red Hat support.

On 29 Mar. 2017 18:39, "Saverio Proto" <ziopr...@gmail.com> wrote:

> Hello all,
>
> we use rclone a lot, and we are happy with it.
>
> the real problem I would say is that a lot of these tools use the
> latest AWS4 signature.
>
> AFAIK the radosgw with Ceph Jewel and Openstack keystone integration
> supports only AWS2 signature because of this bug:
> http://tracker.ceph.com/issues/19056
>
> is anyone else hitting this ?
>
> Saverio
>
> 2017-03-27 22:11 GMT+02:00 John Dickinson <m...@not.mn>:
> >
> >
> > On 27 Mar 2017, at 4:39, Blair Bethwaite wrote:
> >
> >> Hi all,
> >>
> >> Does anyone have any recommendations for good tools to perform
> >> file-system/tree backups and restores to/from a (Ceph RGW-based)
> >> object store (Swift or S3 APIs)? Happy to hear about both FOSS and
> >> commercial options please.
> >>
> >> I'm interested in:
> >> 1) tools known to work or not work at all for a basic file-based data
> backup
> >
> > There's a bunch of backup tools that will work with the Swift API and/or
> the S3 API.
> >
> > Veritas, Commvault, Trilio, and CloudBerry all work. There's other
> companies too that can back specific stuff up to Swift (e.g. Percona with
> MySQL).
> >
> > (The above list taken from https://www.swiftstack.com/solutions/backup
> [my employer] because it's the first linkable place I knew of to answer
> your question.)
> >
> >
> > --John
> >
> >
> >
> >
> >>
> >> Plus these extras:
> >> 2) preserves/restores correct file metadata (e.g. owner, group, acls
> etc)
> >> 3) preserves/restores xattrs
> >> 4) backs up empty directories and files
> >> 5) supports some sort of snapshot/versioning/differential
> >> functionality, i.e., will keep a copy or diff or last N versions of a
> >> file or whole backup set, e.g., so that one can restore yesterday's
> >> file/s or last week's but not have to keep two full copies to achieve
> >> it
> >> 6) is readily able to restore individual files
> >> 7) can encrypt/decrypt client side
> >> 8) anything else I should be considering?
> >>
> >> --
> >> Cheers,
> >> ~Blairo
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] Boston Forum - Formal Submission Now Open!

[Blair Bethwaite]

Hi Melvin,

Just to confirm - the Forum will run Monday through Thursday, and
presumably the session scheduling will be flexible to meet the needs of the
leads/facilitators?

Cheers,
b1airo

On 21 Mar. 2017 6:56 am, "Melvin Hillsman"  wrote:

> Hey everyone!
>
> We have made it to the next stage of the topic selection process for the
> Forum in Boston.
>
> Starting today, our submission tool is open for you to submit abstracts
> for the most popular sessions that came out of your brainstorming. *Please
> note that the etherpads are not being pulled into the submission tool and
> discussion around which sessions to submit are encouraged.*
>
> We are asking all session leaders to submit their abstracts at:
>
> http://forumtopics.openstack.org/
>
> *before 11:59PM UTC on Sunday April 2nd!*
>
> We are looking for a good mix of project-specific, cross-project or
> strategic/whole-of-community discussions, and *sessions that emphasize
> collaboration between users and developers are most welcome!*
>
> We assume that anything submitted to the system has achieved a good amount
> of discussion and consensus that it is a worthwhile topic. After
> submissions close, a team of representatives from the User Committee, the
> Technical Committee, and Foundation staff will take the sessions proposed
> by the community and fill out the schedule.
>
> You can expect the draft schedule to be released on April 10th.
>
> Further details about the Forum can be found at: https://wiki.openstack.
> org/wiki/Forum
>
> Regards,
>
> OpenStack User Committee
>
>
> ___
> User-committee mailing list
> user-commit...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific] Reminder: IRC meeting Wednesday 0900

[Blair Bethwaite]

Hi all -

We have a Scientific WG IRC meeting coming up in a few hours (Wednesday at
0900 UTC) in channel #openstack-meeting.  All welcome.

The agenda has one simple goal:
Follow-up on and finalise Boston Forum proposals and assign leaders to
submit.

Cheers,

-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] backup to object store - tool recommendations

[Blair Bethwaite]

Hi all,

Does anyone have any recommendations for good tools to perform
file-system/tree backups and restores to/from a (Ceph RGW-based)
object store (Swift or S3 APIs)? Happy to hear about both FOSS and
commercial options please.

I'm interested in:
1) tools known to work or not work at all for a basic file-based data backup

Plus these extras:
2) preserves/restores correct file metadata (e.g. owner, group, acls etc)
3) preserves/restores xattrs
4) backs up empty directories and files
5) supports some sort of snapshot/versioning/differential
functionality, i.e., will keep a copy or diff or last N versions of a
file or whole backup set, e.g., so that one can restore yesterday's
file/s or last week's but not have to keep two full copies to achieve
it
6) is readily able to restore individual files
7) can encrypt/decrypt client side
8) anything else I should be considering?

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Dealing with ITAR in OpenStack private clouds

[Blair Bethwaite]

Could just avoid Glance snapshots and indeed Nova ephemeral storage
altogether by exclusively booting from volume with your ITAR volume type or
AZ. I don't know what other ITAR regulations there might be, but if it's
just what JM mentioned earlier then doing so would let you have ITAR and
non-ITAR VMs hosted on the same compute nodes as there would be no local
HDD storage involved.

On 23 Mar. 2017 2:28 am, "Jonathan D. Proulx"  wrote:

On Tue, Mar 21, 2017 at 09:03:36PM -0400, Davanum Srinivas wrote:
:Oops, Hit send before i finished
:
:https://info.massopencloud.org/wp-content/uploads/2016/
03/Workshop-Resource-Federation-in-a-Multi-Landlord-Cloud.pdf
:https://git.openstack.org/cgit/openstack/mixmatch
:
:Essentially you can do a single cinder proxy that can work with
:multiple cinder backends (one use case)

The mixmatch suff is interesting but it's designed ofr sharing rather
than exclusion, is very young and adds complexity that's likely not
wanted here. It is a good read though!

For Block Storage you can have 'volume types' with different back ends and
you can set quotas per project for each instance type.  I've used this
to deprecate old storage by setting quota on 'old' type to zero.
Presumably you you have an ITAR type that ITAR projects had quota on
and a nonITAR type for other projects and never the twains should
meet.

For VMS I use host aggregates and instance metadata to seperate
'special' hardware.  Again instance access can be per project so
having ITAR and notITAR aggregates and matiching instance types with
appopriate access lists can likely solve that.

I've not tried to do anything similar with Image Storage, so  not sure
if there's a way to restrict projects to specific glance stores.  IF
all images were nonITAR and only provisioned with restricted
info after launch *maybe* you could get away with that, though I
suppose you'd need to disallow snapshots for ITAR projects
at least...perhaps someone has a better answer here.

-Jon

:
:Thanks,
:Dims
:
:On Tue, Mar 21, 2017 at 8:59 PM, Davanum Srinivas 
wrote:
:> Jonathan,
:>
:> The folks from Boston University have done some work around this idea:
:>
:> https://github.com/openstack/mixmatch/blob/master/doc/
source/architecture.rst
:>
:>
:> On Tue, Mar 21, 2017 at 7:33 PM, Jonathan Mills 
wrote:
:>> Friends,
:>>
:>> I’m reaching out for assistance from anyone who may have confronted the
:>> issue of dealing with ITAR data in an OpenStack cloud being used in some
:>> department of the Federal Gov.
:>>
:>> ITAR (https://www.pmddtc.state.gov/regulations_laws/itar.html) is a less
:>> restrictive level of security than classified data, but it has some
thorny
:>> aspects to it, particularly where media is concerned:
:>>
:>> * you cannot co-mingle ITAR and non-ITAR data on the same physical hard
:>> drives, and any drive, once it has been “tainted” with any ITAR data,
is now
:>> an ITAR drive
:>>
:>> * when ITAR data is destroyed, a DBAN is insufficient — instead, you
:>> physically shred the drive.  No need to elaborate on how destructive
this
:>> can get if you accidentally mingle ITAR with non-ITAR
:>>
:>> Certainly the multi-tenant model of OpenStack holds great promise in
Federal
:>> agencies for supporting both ITAR and non-ITAR worlds, but great care
must
:>> be taken that *somehow* things like Glance and Cinder don’t get mixed
up.
:>> One must ensure that the ITAR tenants can only access Glance/Cinder in
ways
:>> such that their backend storage is physically separate from any non-ITAR
:>> tenants.  Certainly I understand that Glance/Cinder can support multiple
:>> storage backend types, such as File & Ceph, and maybe that is an avenue
to
:>> explore to achieving the physical separation.  But what if you want to
have
:>> multiple different File backends?
:>>
:>> Do the ACLs exist to ensure that non-ITAR tenants can’t access ITAR
:>> Glance/Cinder backends, and vice versa?
:>>
:>> Or…is it simpler to just build two OpenStack clouds….?
:>>
:>> Your thoughts will be most appreciated,
:>>
:>>
:>> Jonathan Mills
:>>
:>> NASA Goddard Space Flight Center
:>>
:>>
:>> ___
:>> OpenStack-operators mailing list
:>> OpenStack-operators@lists.openstack.org
:>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>>
:>
:>
:>
:> --
:> Davanum Srinivas :: https://twitter.com/dims
:
:
:
:--
:Davanum Srinivas :: https://twitter.com/dims
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list

Re: [Openstack-operators] Dealing with ITAR in OpenStack private clouds

[Blair Bethwaite]

On 22 March 2017 at 13:33, Jonathan Mills  wrote:
>
> To what extent is it possible to “lock” a tenant to an availability zone,
> to guarantee that nova scheduler doesn’t land an ITAR VM (and possibly the
> wrong glance/cinder) into a non-ITAR space (and vice versa)…
>

Yes, definitely a few different ways to skin that cat with Nova aggregates
and scheduler filters. The answer ultimately depends on what you want UX to
be like, i.e., for both default non-ITAR projects and ITAR specific
projects...?

For just that concern, Mike Lowe was chatting with me off list about using
> Regions….but I should probably let Mike speak for himself if he wants.
> Having never used anything other than the default “RegionOne” I can’t speak
> to the capabilities.
>

Could do certainly, but sounds like a whole lot of extra operational
effort/overhead versus logical separation. The answer probably depends on
both scale and process maturity.

-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Dealing with ITAR in OpenStack private clouds

[Blair Bethwaite]

Dims, it might be overkill to introduce multi-Keystone + federation (I just
quickly skimmed the PDF so apologies if I have the wrong end of it)?

Jon, you could just have multiple cinder-volume services and backends. We
do this in the Nectar cloud - each site has cinder AZs matching nova AZs.
By default the API won't let you attach a volume to a host in a
non-matching AZ, maybe that's enough for you(?), but you could probably
take it further with other cinder scheduler filters.

On 22 March 2017 at 12:03, Davanum Srinivas  wrote:

> Oops, Hit send before i finished
>
> https://info.massopencloud.org/wp-content/uploads/2016/
> 03/Workshop-Resource-Federation-in-a-Multi-Landlord-Cloud.pdf
> https://git.openstack.org/cgit/openstack/mixmatch
>
> Essentially you can do a single cinder proxy that can work with
> multiple cinder backends (one use case)
>
> Thanks,
> Dims
>
> On Tue, Mar 21, 2017 at 8:59 PM, Davanum Srinivas 
> wrote:
> > Jonathan,
> >
> > The folks from Boston University have done some work around this idea:
> >
> > https://github.com/openstack/mixmatch/blob/master/doc/
> source/architecture.rst
> >
> >
> > On Tue, Mar 21, 2017 at 7:33 PM, Jonathan Mills 
> wrote:
> >> Friends,
> >>
> >> I’m reaching out for assistance from anyone who may have confronted the
> >> issue of dealing with ITAR data in an OpenStack cloud being used in some
> >> department of the Federal Gov.
> >>
> >> ITAR (https://www.pmddtc.state.gov/regulations_laws/itar.html) is a
> less
> >> restrictive level of security than classified data, but it has some
> thorny
> >> aspects to it, particularly where media is concerned:
> >>
> >> * you cannot co-mingle ITAR and non-ITAR data on the same physical hard
> >> drives, and any drive, once it has been “tainted” with any ITAR data,
> is now
> >> an ITAR drive
> >>
> >> * when ITAR data is destroyed, a DBAN is insufficient — instead, you
> >> physically shred the drive.  No need to elaborate on how destructive
> this
> >> can get if you accidentally mingle ITAR with non-ITAR
> >>
> >> Certainly the multi-tenant model of OpenStack holds great promise in
> Federal
> >> agencies for supporting both ITAR and non-ITAR worlds, but great care
> must
> >> be taken that *somehow* things like Glance and Cinder don’t get mixed
> up.
> >> One must ensure that the ITAR tenants can only access Glance/Cinder in
> ways
> >> such that their backend storage is physically separate from any non-ITAR
> >> tenants.  Certainly I understand that Glance/Cinder can support multiple
> >> storage backend types, such as File & Ceph, and maybe that is an avenue
> to
> >> explore to achieving the physical separation.  But what if you want to
> have
> >> multiple different File backends?
> >>
> >> Do the ACLs exist to ensure that non-ITAR tenants can’t access ITAR
> >> Glance/Cinder backends, and vice versa?
> >>
> >> Or…is it simpler to just build two OpenStack clouds….?
> >>
> >> Your thoughts will be most appreciated,
> >>
> >>
> >> Jonathan Mills
> >>
> >> NASA Goddard Space Flight Center
> >>
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>
> >
> >
> >
> > --
> > Davanum Srinivas :: https://twitter.com/dims
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Flavors

[Blair Bethwaite]

Hi Chris,

On 17 Mar. 2017 15:24, "Chris Friesen" <chris.frie...@windriver.com> wrote:

On 03/16/2017 07:06 PM, Blair Bethwaite wrote:

Statement: breaks bin packing / have to match flavor dimensions to hardware
> dimensions.
> Comment: neither of these ring true to me given that most operators tend to
> agree that memory is there first constraining resource dimension and it is
> difficult to achieve high CPU utilisation before memory is exhausted. Plus
> virtualisation is inherently about resource sharing and over-provisioning,
>

Ah whoops, that was meant to be: "resource sharing and (optionally)
over-provisioning", where I was broadly including security and convenience
under resource sharing. There are of course many other factors.

unless you have very detailed knowledge of your workloads a priori (or some
> cycle-stealing/back-filling mechanism) you will always have
> under-utilisation
> (possibly quite high on average) in some resource dimensions.
>

I think this would be highly dependent on the workload.  A virtual router
is going to run out of CPU/network bandwidth far before memory is exhausted.


Absolutely, which is why I hinted at today's general IaaS workload and
said, "...unless you have very detailed knowledge of your workloads a
priori". NFV focused clouds would clearly look quite different, and I
suppose with the rise of OpenStack at telcos there would be quite a few
such deployments floating around now.

For similar reasons I'd disagree that virtualization is inherently about
over-provisioning and suggest that (in some cases at least) it's more about
flexibility over time.  Our customers generally care about maximizing
performance and so nothing is over-provisioned...disk, NICs, CPUs, RAM are
generally all exclusively allocated.


Sure, in our three Nova Cells we have a big mix of workload. One Cell is
HPC oriented and so has no over-provisioning. Another is performance
oriented (fast cores, fast network) but still moderately over-provisioned
(cgroups to manage resource share between flavors). The other is general
purpose.

For me an interesting question to know the answer to here would be at what
point you have to stop resource sharing to guarantee your performance
promises/SLAs (disregarding memory over-provisioning). My gut says that
unless you are also doing all the other high-end performance tuning (CPU &
memory pinning​, NUMA topology, hugepages, optimised networking such as
macvtap or SRIOV, plus all the regular host-side system/BIOS and power
settings) you'll see very little benefit, i.e., under-provisioning on its
own is not a performance win.

Cheers,
Blair
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Flavors

[Blair Bethwaite]

There have been previous proposals (and if memory serves, even some
blueprints) for API extensions to allow this but they have apparently
stagnated. On the face of it I think OpenStack should support this (more
choice = win!) - doesn't mean that every cloud needs to use the feature. Is
it worth trying to resurrect some feature development around this? Sounds
like a potential forum session? We have already seen responses here from a
number of active and prominent operators, some seem to be quite emotive,
which could indicate this hits on some sore-points/bugbears.

Some comments about points raised already in this thread...

Statement: makes pricing hard because users can monopolise a subset of
infrastructure resource dimensions (e.g. memory, disk IOPs) leading to some
dimensions (e.g. CPU) being underutilised.
Comment: it may exacerbate the problem if users can request resource
dimensions with no limits or dependencies imposed, but that problem
typically already exists to some extent in any general-purpose deployment -
as others have mentioned they mostly find themselves constrained by memory.
That does not seem like a hard problem to workaround, e.g., dimensions
could have both absolute lower and upper bounds plus dynamic bounds based
on the setting of other dimensions.

Statement: breaks bin packing / have to match flavor dimensions to hardware
dimensions.
Comment: neither of these ring true to me given that most operators tend to
agree that memory is there first constraining resource dimension and it is
difficult to achieve high CPU utilisation before memory is exhausted. Plus
virtualisation is inherently about resource sharing and over-provisioning,
unless you have very detailed knowledge of your workloads a priori (or some
cycle-stealing/back-filling mechanism) you will always have
under-utilisation (possibly quite high on average) in some resource
dimensions.

Other thoughts...

A feature such as this also opens up the interesting possibility of
soft/fuzzy resource requests, which could be very useful in a private (i.e.
constrained) cloud environment, e.g., "give me an instance with 2-4 cores
and 8-16GB RAM and at least 500 IOPs".

Some of the statements in this thread also lend credence to the need for a
way to provide preemptible instances which would provide one way to
back-fill compute/cpu based resources.

Cheers,

On 17 March 2017 at 04:21, Tomáš Vondra  wrote:

> We at Homeatcloud.com do exactly this in our VPS service. The user can
> configure the VPS with any combination of CPU, RAM, and disk. However a)
> the configurations are all about 10% the size of the physical machines and
> b) the disks are in a SAN array, provisioned as volumes. So I give the
> users some flexibility and can better see what configurations they actually
> want and build new hypervisors with that in mind. They mostly want up to 4
> GB RAM anyway, si it’s not a big deal.
>
> Tomas Vondra
>
>
>
> *From:* Adam Lawson [mailto:alaw...@aqorn.com]
> *Sent:* Thursday, March 16, 2017 5:57 PM
> *To:* Jonathan D. Proulx
> *Cc:* OpenStack Operators
> *Subject:* Re: [Openstack-operators] Flavors
>
>
>
> One way I know some providers work around this when using OpenStack is by
> fronting the VM request with some code in the web server that checks if the
> requested spec has an existing flavor. if so, use the flavor, if not, use
> an admin account that creates a new flavor and assign use it for that user
> request then remove if when the build is complete. This naturally impacts
> your control over hardware efficiency but it makes your scenario possible
> (for better or for worse). I also hate being forced to do what someone else
> decided was going to be best for me. That's my decision and thanksully with
> openStack, this kind of thing is rather easy to do.
>
>
>
> //adam
>
>
>
> *Adam Lawson*
>
>
>
> Principal Architect, CEO
>
> Office: +1-916-794-5706 <(916)%20794-5706>
>
>
>
> On Thu, Mar 16, 2017 at 7:52 AM, Jonathan D. Proulx 
> wrote:
>
>
> I have always hated flavors and so do many of my users.
>
> On Wed, Mar 15, 2017 at 03:22:48PM -0700, James Downs wrote:
> :On Wed, Mar 15, 2017 at 10:10:00PM +, Fox, Kevin M wrote:
> :> I think the really short answer is something like: It greatly
> simplifies scheduling and billing.
> :
> :The real answer is that once you buy hardware, it's in a fixed radio of
> CPU/Ram/Disk/IOPS, etc.
>
> This while apparently reasonable is BS (at least in private cloud
> space).  What users request and what they actualy use are wildly
> divergent.
>
> *IF* usage of claimed resorces were at or near optimal then this might
> be true .  But if people are claiming 32G of ram because that how much
> you assigned to a 16 vCPU instance type but really just need 16
> threads with 2G or 4G then you packing still sucks.
>
> I'm mostly bound on memory so I mostly have my users select on that
> basis and over provide and over provision CPU since that can be
> 

Re: [Openstack-operators] [User-committee] [scientific][scientific-wg] Reminder: IRC meeting Wednesday 0900

[Blair Bethwaite]

Sub-etherpad for Scientific-specific brainstorming:
https://etherpad.openstack.org/p/BOS-UC-brainstorming-scientific-wg

On 1 March 2017 at 05:02, Stig Telfer <stig.openst...@telfer.org> wrote:

> Greetings all -
>
> We have a Scientific WG IRC meeting on Wednesday at 0900 UTC in channel
> #openstack-meeting.  Everyone is welcome.
>
> The agenda[1] is a simple one - brainstorming for Scientific WG input to
> the Forum at Boston.
>
> Best wishes,
> Stig
>
> [1] https://wiki.openstack.org/wiki/Scientific_working_group#
> IRC_Meeting_March_1st_2017
> [2] http://eavesdrop.openstack.org/#Scientific_Working_Group
>
> ___
> User-committee mailing list
> user-commit...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>



-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [User-committee] [scientific][scientific-wg] Scientific-WG IRC meeting Weds 0900 UTC

[Blair Bethwaite]

Hi all -

We have a meeting coming up in about 12 hours:
2017-02-15 0900 UTC in channel #openstack-meeting

This is substantively a repeat of last week's agenda for alternate timezones

- Boston Declaration update from Martial
- Hypervisor tuning update from Blair
- Blair's experiences with RoCE over layer-3 ECMP fabrics
- Save the date: Boston Cloud Declaration (confirmed for Thursday, May
11th and Friday May 12th)
- Repo for WG documentation
(http://git.openstack.org/cgit/openstack/scientific-wg/)
- Scientific OpenStack Workshop(s) at SC2017
(http://sc17.supercomputing.org/submitters/workshops/,
https://etherpad.openstack.org/p/SC17WorkshopWorksheet)

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] What would you like in Pike?

[Blair Bethwaite]

Hi Tim,

We did wonder in last week's meeting whether quota management and nested
project support (particularly which flows are most important) would be a
good session for the Boston Forum...? Would you be willing to lead such a
discussion?

Cheers,

On 19 January 2017 at 19:59, Tim Bell  wrote:

>
> On 18 Jan 2017, at 23:20, Matt Jarvis  wrote:
>
> I think one of the problems we're seeing now is that a lot of operators
> have actually already scratched some of these missing functionality itches
> like quota management and project nesting by handling those scenarios in
> external management systems. I know we certainly did at DataCentred. That
> probably means these things don't surface enough to upstream as
> requirements, whereas for new users who aren't necessarily in the loop with
> community communication they may well be creating friction to adoption.
>
>
> For the quota management, I think the first discussions were in the Hong
> Kong summit around the Boson project and this has moved backwards and
> forwards between services, libraries and improving the code. While the user
> need is relatively simple to state, these are not simple problems to solve
> so it is often difficult for the items to get to the priority lists.
>
> One of the difficulties we have found is that we could get staff for a
> project such as quota management for a short period (e.g. 1 year). However,
> from the initial specification to code acceptance is often an extended
> period so these sort of changes can get stalled but the people contributing
> need to show results for their work (such as a thesis).
>
> From the scientific working group discussions, the quota and nesting
> discussions have come up regularly so the requirements are still there.
>
> Tim
>
> On Wed, Jan 18, 2017 at 10:06 PM, Sam Morrison  wrote:
>
>> I would love it if all the projects policy.json was actually usable. Too
>> many times the policy.json isn’t the only place where authN happens with
>> lots of hard coded is_admin etc.
>>
>> Just the ability to to have a certain role to a certain thing would be
>> amazing. It makes it really hard to have read only users to generate
>> reports with that we can show our funders how much people use our openstack
>> cloud.
>>
>> Cheers,
>> Sam
>> (non-enterprise)
>>
>>
>>
>> On 18 Jan 2017, at 6:10 am, Melvin Hillsman  wrote:
>>
>> Well said, as a consequence of this thread being on the mailing list, I
>> hope that we can get *all* operators, end-users, and app-developers to
>> respond. If you are aware of folks who do not fall under the "enterprise"
>> label please encourage them directly to respond; I would encourage everyone
>> to do the same.
>>
>> On Tue, Jan 17, 2017 at 11:52 AM, Silence Dogood 
>> wrote:
>>
>>> I can see a huge problem with your contributing operators... all of them
>>> are enterprise.
>>>
>>> enterprise needs are radically different from small to medium deployers
>>> who openstack has traditionally failed to work well for.
>>>
>>> On Tue, Jan 17, 2017 at 12:47 PM, Piet Kruithof 
>>> wrote:
>>>
 Sorry for the late reply, but wanted to add a few things.

 OpenStack UX did suggest to the foundation that the community needs a
 second survey that focuses exclusively on operators.  The rationale was
 that the user survey is primarily focused on marketing data and there isn't
 really a ton of space for additional questions that focuses exclusively on
 operators. We also recommended a second survey called a MaxDiff study that
 enabled operators to identify areas of improvement and also rate them in
 order of importance including distance.

 There is also an etherpad that asked operators three priorities for
 OpenStack:

 https://etherpad.openstack.org/p/mitaka-openstackux-enterprise-goals

 It was distributed about a year ago, so I'm not sure how much of it was
 relevant.  The list does include responses from folks at TWC, Walmart,
 Pacific Northwest Labs, BestBuy, Comcast, NTTi3 and the US government. It
 might be a good place for the group to add their own improvements as well
 as "+" other peoples suggestions.

 There is also a list of studies that have been conducted with operators
 on behalf of the community. The study included quotas, deployment and
 information needs. Note that the information needs study extended beyond
 docs to things like the need to easily google solutions and the need for
 SMEs.

 Hope this is helpful.

 Piet

 ___
 OPENSTACK USER EXPERIENCE STATUS
 The goal of this presentation is to provide an overview of research
 that was conducted on behalf of the OpenStack community.  All of the
 studies conducted on behalf of the OpenStack community were included in
 this presentation.

 Why this 

Re: [Openstack-operators] [Glance] [Nova] Multiple backends / Qcow Derived Images

[Blair Bethwaite]

On 5 January 2017 at 19:47, Rui Chen  wrote:
> Ah, Adam, got your point, I found two related Nova blueprints that were
> similar with your idea,
> but there are not any activities about them from 2014, I hadn't dive deep
> into these comments,
> you might get some background information at there.
>
> https://blueprints.launchpad.net/nova/+spec/image-precacher

Whilst it is a valid use-case, I don't see a huge amount of utility in
this as compared to running a local nova-compute facing glance-api +
glance-cache. You can have that as a control plane service, and with
LAN-typical bandwidths available you're probably better off saving on
the compute side capacity requirements and/or complexity of management
as opposed to the extra few seconds of download time incurred on the
occasional boot of a not frequently used image.

If absolute minimal instance boot times are required then some kind of
CoW capable shared storage might be a better option.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Glance] [Nova] Multiple backends / Qcow Derived Images

[Blair Bethwaite]

Hi Adam,

On 5 January 2017 at 08:48, Adam Lawson  wrote:
> Just a friendly bump. To clarify, the ideas being tossed around are to host
> QCOW images on each Compute node so the provisioning is faster (i.e. less
> dependency on network connectivity to a shared back-end). I need to know if
> this is possible or not. So far, I've seen nothing that suggests that it is
> but i want to confirm that.

If you are using qcow based ephemeral disks then you will have
something like a /var/lib/nova/instances/_base/ dir on each
nova-compute. That directory holds the backing files for each
"derived" disk file in /var/lib/nova/instances//. It also acts
as a local image cache - their are corresponding nova-compute config
options controlling whether and how often unused base files get
removed. See e.g.
http://www.pixelbeat.org/docs/openstack_libvirt_images/ for a great
dive into the possibilities.

> Also, derived images is a QCOW thing[1], I'm wondering if creating these
> dynamically is supported by Nova and/or Glance.

The default (typical?) configuration does use qcow layering, but you
can also change your nova-compute settings to e.g. force all instances
to run off raw disk files.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Analogs of EC2 dedicated instances & dedicated hosts?

[Blair Bethwaite]

Hi Conrad,

On 20 December 2016 at 09:24, Kimball, Conrad  wrote:
> · Dedicated instances:  an OpenStack tenant can deploy VM instances
> that are guaranteed to not share a compute host with any other tenant (for
> example, as the tenant I want physical segregation of my compute).

You can certainly do this, however you will need to configure either
scheduler and/or host aggregates on a per case/tenant basis for
projects that have this isolation requirement - depending on how
dynamic this is in your environment you may want to automate that. In
any case, the AggregateMultiTenancyIsolation scheduler filter is what
you want I think. Alternatively, if the requirement can be met using
VM images then the IsolatedHostsFilter may also be an option (e.g. the
VM image is kept private and only tenant/s allowed to use that image
on that host will have Glance member access to the image).

> · Dedicated hosts: goes beyond dedicated instances, allowing an
> OpenStack tenant to explicitly place only specific VM instances onto the
> same compute host (for example, as the tenant I want to place VMs foo and
> bar onto the same compute host to share a software license that is licensed
> per host).

As Kris said, ServerGroup filters are probably the way to go for this
one, but the IsolatedHostsFilter may also work if the licensing
requirements can be expressed at the Glance image level.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific-wg] SC16 get together - P.F. Chang's @ 8.30pm tonight

[Blair Bethwaite]

Hi all -

In the fashion of JITS (just in time scheduling), we OpenStack+HPC folk are
planning to converge at P.F. Chang's (https://goo.gl/maps/YN8v26CBctp
- West 300 South) at 8.30pm following on from the technical programme
reception.

Hope to see you there!

Cheers,
Blair
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific-wg] OpenStack at SC16

[Blair Bethwaite]

Hi folks,

There's a superuser blog live now detailing OpenStack-related
goings-on at SC this week:
http://superuser.openstack.org/articles/openstack-supercomputing-2016/

Cheers,

-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [MassivelyDistributed] bi-weekly meeting

[Blair Bethwaite]

Devil's advocate - what is "full enough"? Surely another channel is
essentially free and having flexibility in available timing is of utmost
importance?

On 8 Nov 2016 5:37 PM, "Tony Breeds"  wrote:

> On Mon, Nov 07, 2016 at 05:52:43PM +0100, lebre.adr...@free.fr wrote:
> > Dear all,
> >
> > We are still looking for an irc channel for our meeting:
> https://review.openstack.org/#/c/393899
> > There is no available channels for the slot we selected during our
> face-to-face meeting in Barcelona.
> >
> > If I'm correct, we have two possibilities:
> > - Determine another slot: the first available slot on Wednesday is at
> 17:00 UTC.
>
> Or you could look at 1400 on Wednesday
>
> > - Ask for the creation of a new IRC channel dedicated to our WG:
> something line #openstack-massively-distributed
>
> This is an option but you can't hold meetings in project specific channels
> as
> the meeting bot only works correctly in the #openstack-meeting rooms.
>
> We from time to time get asked to cerate a 5th meeting room.  Looking at
> [1] I
> don't feel like we're full enough to persue that.
>
> Yours Tony.
> [1] https://docs.google.com/spreadsheets/d/1lQHKCQa4wQmnWpTMB3DLltY81kICh
> ZumHLHzoMNF07c/edit?usp=sharing
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Disable console for an instance

[Blair Bethwaite]

Lol! I don't mind - Microsoft do support and produce some pretty good
research, I just wish they'd fix licensing!

On 27 October 2016 at 16:11, Jonathan D. Proulx <j...@csail.mit.edu> wrote:
> On Thu, Oct 27, 2016 at 04:08:26PM +0200, Blair Bethwaite wrote:
> :On 27 October 2016 at 16:02, Jonathan D. Proulx <j...@csail.mit.edu> wrote:
> :> don't put a getty on the TTY :)
> :
> :Do you know how to do that with Windows? ...you can see the desire for
> :sandboxing now :-).
>
> Sigh yes I see, http://goodbye-microsoft.com/ has a good solution IMHO
>
> :--
> :Cheers,
> :~Blairo



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Disable console for an instance

[Blair Bethwaite]

Hi George,

On 27 October 2016 at 16:15, George Mihaiescu  wrote:
> Did you try playing with Nova's policy file and limit the scope for
> "compute_extension:console_output": "" ?

No, interesting idea though... I suspect it's actually the
get_*_console policies we'd need to tweak, I think console_output
probably refers to the console log? Anyway, not quite sure how we'd
craft policy that would enable us to disable these on a per instance
basis though - is it possible to reference image metadata in the
context of the policy rule?

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Disable console for an instance

[Blair Bethwaite]

On 27 October 2016 at 16:02, Jonathan D. Proulx  wrote:
> don't put a getty on the TTY :)

Do you know how to do that with Windows? ...you can see the desire for
sandboxing now :-).

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Disable console for an instance

[Blair Bethwaite]

Looks like this is not currently possible. Does anyone else have an
interest in such a feature?

I'm thinking about it from the perspective of a public cloud user who wants
to build highly secure / sandboxed instances. Having a virtual terminal
straight into a guest login prompt, especially one that allows reset of the
guest, is not desirable.

On 13 October 2016 at 04:37, Blair Bethwaite <blair.bethwa...@gmail.com>
wrote:

> Hi all,
>
> Does anyone know whether there is a way to disable the novnc console on a
> per instance basis?
>
> Cheers,
> Blair
>



-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Barcelona Scientific BoF: Call for lightning talks

[Blair Bethwaite]

Hi all,

A reminder, we are still looking for people interested in giving
lightning talks in the Scientific WG BoF session on Wednesday @2:15pm
[1]. This is a great opportunity to quickly share highlights of what
you have been or are working on with the community.

[1] 
https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/16779/scientific-working-group-bof-and-poster-session

Cheers,
Blair

On 11 October 2016 at 23:36, Stig Telfer <stig.openst...@telfer.org> wrote:
> Hello all -
>
> We have our schedule confirmed and will be having a BoF for Scientific 
> OpenStack users at 2:15pm on the summit Wednesday: 
> https://www.openstack.org/summit/barcelona-2016/summit-schedule/events/16779/scientific-working-group-bof-and-poster-session
>
> We are planning to run some lightning talks in this session, typically up to 
> 5 minutes long.  If you or your institution have been implementing some 
> bright ideas that take OpenStack into new territory for research computing 
> use cases, lets hear it!
>
> Please follow up to me and Blair (Scientific WG co-chairs) if you’re 
> interested in speaking and would like to bag a slot.
>
> Best wishes,
> Stig
>



-- 
Blair Bethwaite
Senior HPC Consultant

Monash eResearch Centre
Monash University
Room G26, 15 Innovation Walk, Clayton Campus
Clayton VIC 3800
Australia
Mobile: 0439-545-002
Office: +61 3-9903-2800

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-operators][ceph][nova] How do you handle Nova on Ceph?

[Blair Bethwaite]

Hi Adam,

I agree somewhat, capacity management and growth at scale is something
of a pain. Ceph gives you a hugely powerful and flexible way to manage
data-placement through crush but there is very little quality info
about, or examples of, non-naive crushmap configurations.

I think I understand what you are getting at in regards to
failure-domain, e.g., a large cluster of 1000+ drives may require a
single storage pool (e.g., for nova) across most/all of that storage.
The chances of overlapping drive failures (overlapping meaning before
recovery has completed) in multiple nodes is higher the more drives
there are in the pool unless you design your crushmap to limit the
size of any replica-domain (i.e., the leaf crush bucket that a single
copy of an object may end up in). And in the rbd use case, if you are
unlucky and even lose just a tiny fraction of objects, due to random
placement there is a good chance you have lost a handful of objects
from most/all rbd volumes in the cluster, which could make for many
unhappy users with potentially unrecoverable filesystems in those
rbds.

The guys at UnitedStack did a nice presentation that touched on this a
while back 
(http://www.slideshare.net/kioecn/build-an-highperformance-and-highdurable-block-storage-service-based-on-ceph)
but I'm not sure I follow their durability model just from these
slides, and if you're going to play with this you really do want a
tool to calculate/simulate the impact the changes.

Interesting discussion - maybe loop in ceph-users?

Cheers,

On 14 October 2016 at 19:53, Adam Kijak  wrote:
>> 
>> From: Clint Byrum 
>> Sent: Wednesday, October 12, 2016 10:46 PM
>> To: openstack-operators
>> Subject: Re: [Openstack-operators] [openstack-operators][ceph][nova] How do  
>>you handle Nova on Ceph?
>>
>> Excerpts from Adam Kijak's message of 2016-10-12 12:23:41 +:
>> > > 
>> > > From: Xav Paice 
>> > > Sent: Monday, October 10, 2016 8:41 PM
>> > > To: openstack-operators@lists.openstack.org
>> > > Subject: Re: [Openstack-operators] [openstack-operators][ceph][nova] How 
>> > > do you handle Nova on Ceph?
>> > >
>> > > I'm really keen to hear more about those limitations.
>> >
>> > Basically it's all related to the failure domain ("blast radius") and risk 
>> > management.
>> > Bigger Ceph cluster means more users.
>>
>> Are these risks well documented? Since Ceph is specifically designed
>> _not_ to have the kind of large blast radius that one might see with
>> say, a centralized SAN, I'm curious to hear what events trigger
>> cluster-wide blasts.
>
> In theory yes, Ceph is desgined to be fault tolerant,
> but from our experience it's not always like that.
> I think it's not well documented, but I know this case:
> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg32804.html
>
>> > Growing the Ceph cluster temporary slows it down, so many users will be 
>> > affected.
>> One might say that a Ceph cluster that can't be grown without the users
>> noticing is an over-subscribed Ceph cluster. My understanding is that
>> one is always advised to provision a certain amount of cluster capacity
>> for growing and replicating to replaced drives.
>
> I agree that provisioning a fixed size Cluster would solve some problems but 
> planning the capacity is not always easy.
> Predicting the size and making it cost effective (empty big Ceph cluster 
> costs a lot on the beginning) is quite difficult.
> Also adding a new Ceph cluster will be always more transparent to users than 
> manipulating existing one especially when growing pool PGs)
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Disable console for an instance

[Blair Bethwaite]

Hi all,

Does anyone know whether there is a way to disable the novnc console on a
per instance basis?

Cheers,
Blair
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-hpc] What's the state of openstack-hpc now?

[Blair Bethwaite]

EX 8747 48-Lane, 5-Port PCI
> Express Gen 3 (8.0 GT/s) Switch (rev ca)
> 87:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI
> Express Gen 3 (8.0 GT/s) Switch (rev ca)
> 88:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
> 89:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
> [root@r-001 ~]# nvidia-smi topo --matrix
> GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 mlx4_0 CPU Affinity
> GPU0 X PIX PHB PHB SOC SOC SOC SOC SOC 0-11,24-35
> GPU1 PIX X PHB PHB SOC SOC SOC SOC SOC 0-11,24-35
> GPU2 PHB PHB X PIX SOC SOC SOC SOC SOC 0-11,24-35
> GPU3 PHB PHB PIX X SOC SOC SOC SOC SOC 0-11,24-35
> GPU4 SOC SOC SOC SOC X PIX PHB PHB PHB 12-23,36-47
> GPU5 SOC SOC SOC SOC PIX X PHB PHB PHB 12-23,36-47
> GPU6 SOC SOC SOC SOC PHB PHB X PIX PHB 12-23,36-47
> GPU7 SOC SOC SOC SOC PHB PHB PIX X PHB 12-23,36-47
> mlx4_0 SOC SOC SOC SOC PHB PHB PHB PHB X
>
> Legend:
>
>   X   = Self
>   SOC = Path traverses a socket-level link (e.g. QPI)
>   PHB = Path traverses a PCIe host bridge
>   PXB = Path traverses multiple PCIe internal switches
>   PIX = Path traverses a PCIe internal switch
>
>
> Cheers,
> Andrew
>
>
> Andrew J. Younge
> School of Informatics & Computing
> Indiana University/Bloomington, IN USA
> ajyou...@indiana.edu/http://ajyounge.com
>
>
> On Tue, Sep 27, 2016 at 4:37 AM, Blair Bethwaite
> <blair.bethwa...@gmail.com> wrote:
> > Hi Andrew, hi John -
> >
> > I've just started trying to get CUDA P2P working in our virtualized
> > HPC environment. I figure this must be something you solved already in
> > order to produce the aforementioned paper, but having read it a couple
> > of times I don't think it provides enough detail about the guest
> > config, hoping you can shed some light...
> >
> > The issue I'm grappling with is that despite using a qemu-kvm machine
> > type (q35) with an emulated PCIe bus and seeing that indeed the P2P
> > capable GPUs (NVIDIA K80s) are attached to that bus, and nvidia-smi
> > sees them as sharing a PHB, the simpleP2P CUDA sample fails when
> > checking their ability to communicate with each other. Is there some
> > magic config I might be missing, did you need to make any PCI-ACS
> > changes?
> >
> > Best regards,
> > Blair
> >
> >
> > On 16 March 2016 at 07:57, Blair Bethwaite <blair.bethwa...@gmail.com>
> wrote:
> >>
> >> Hi Andrew,
> >>
> >> On 16 March 2016 at 05:28, Andrew J Younge <ajyou...@indiana.edu>
> wrote:
> >> > point to a recent publication of ours at VEE15 titled "Supporting High
> >> > Performance Molecular Dynamics in Virtualized Clusters using IOMMU,
> >> > SR-IOV, and GPUDirect."  In the paper we show that using Nvidia GPUs
> >> ...
> >> > http://dl.acm.org/citation.cfm?id=2731194
> >>
> >> Oooh interesting - GPUDirect too. That's something I've been wanting
> >> to try out in our environment. Will take a look a your paper...
> >>
> >> --
> >> Cheers,
> >> ~Blairo
> >
> >
> > --
> > Cheers,
> > ~Blairo
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Barcelona Ops Meetup Planning

[Blair Bethwaite]

Hi David,

Good point, hadn't noticed the docs ones. I'll keep an eye on how the
schedule pans out versus where I can be and slot something into
relevant etherpads closer to the time.

Cheers,
Blair

On 5 October 2016 at 22:23, David Medberry <medbe...@gmail.com> wrote:
> There is a docs session or two. Also I think it could be discussed in the
> Nova section. As a stretch, we could cover on lightning talks. There is also
> the Friday work sessions. So I think plenty of options. We also have at
> least three placeholder sessions.
>
>
> On Oct 4, 2016 11:28 PM, "Blair Bethwaite" <blair.bethwa...@gmail.com>
> wrote:
>>
>> Hi all,
>>
>> I've just had a look at this with a view to adding 10 minutes
>> somewhere on what to do with the hypervisor tuning guide, but I see
>> the free-form notes on the etherpad have been marked as "Old", so
>> figure it's better to discuss here first... Could maybe fit under
>> ops-nova or ops-hardware?
>>
>> Cheers,
>> Blair
>>
>> On 23 September 2016 at 14:53, Tom Fifield <t...@openstack.org> wrote:
>> > Bump! Please add your +1s or session suggestions. We're also looking for
>> > lightning talks.
>> >
>> > https://etherpad.openstack.org/p/BCN-ops-meetup
>> >
>> >
>> > On 20/09/16 21:50, Matt Van Winkle wrote:
>> >>
>> >> Hello all,
>> >>
>> >> The etherpad to gather ideas from you all for sessions during the
>> >> Barcelona Ops Meetup.  Please take some time to review and add your
>> >> comments here:
>> >>
>> >>
>> >>
>> >> https://etherpad.openstack.org/p/BCN-ops-meetup
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> The meetup planning team will be meeting weekly between now and the
>> >> summit to plan all this.  If You are interested in helping with that
>> >> process, you are welcome to join us in #openstack-operators at 14:00
>> >> UTC
>> >> on Tuesdays.  We are always ready to welcome new help.
>> >>
>> >>
>> >>
>> >> Thanks!
>> >>
>> >> VW
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ___
>> >> OpenStack-operators mailing list
>> >> OpenStack-operators@lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >>
>> >
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>>
>> --
>> Cheers,
>> ~Blairo
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Barcelona Ops Meetup Planning

[Blair Bethwaite]

Hi all,

I've just had a look at this with a view to adding 10 minutes
somewhere on what to do with the hypervisor tuning guide, but I see
the free-form notes on the etherpad have been marked as "Old", so
figure it's better to discuss here first... Could maybe fit under
ops-nova or ops-hardware?

Cheers,
Blair

On 23 September 2016 at 14:53, Tom Fifield  wrote:
> Bump! Please add your +1s or session suggestions. We're also looking for
> lightning talks.
>
> https://etherpad.openstack.org/p/BCN-ops-meetup
>
>
> On 20/09/16 21:50, Matt Van Winkle wrote:
>>
>> Hello all,
>>
>> The etherpad to gather ideas from you all for sessions during the
>> Barcelona Ops Meetup.  Please take some time to review and add your
>> comments here:
>>
>>
>>
>> https://etherpad.openstack.org/p/BCN-ops-meetup
>>
>>
>>
>>
>>
>> The meetup planning team will be meeting weekly between now and the
>> summit to plan all this.  If You are interested in helping with that
>> process, you are welcome to join us in #openstack-operators at 14:00 UTC
>> on Tuesdays.  We are always ready to welcome new help.
>>
>>
>>
>> Thanks!
>>
>> VW
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Rados Gateway to Swift migration

[Blair Bethwaite]

Nice! But I'm curious, why the need to migrate?

On 5 October 2016 at 13:29, Xav Paice  wrote:
> On Wed, 2016-10-05 at 13:28 +1300, Xav Paice wrote:
>> On Tue, 2016-10-04 at 17:48 -0600, Curtis wrote:
>> > Maybe you have someone on staff who loves writing lua (for haproxy)? :)
>> >
>>
>> Well, maybe not that far, but yeah we're now thinking down that route.
>> If we get there, I'll quickly write something up about it.  Many thanks
>> for the suggestion :)
>>
>
> OK, that wasn't as hard as I thought it would be.  Thanks Curtis!
>
> Haproxy config snippet, in case anyone else has the same need (note, not
> in production, only rudimentary testing, and slightly sanitized for a
> mailing list):
>
> frontend objectstore
>   bind :::8843 ssl crt host.crt.pem ca-file ca.pem no-sslv3
>   mode http
>   acl rgw_customer path_beg -m sub -f /some/list/of/rgw-folks
>   use_backend rgw00 if rgw_customer
>   default_backend swift-pxy00
>
> backend rgw00
>   balance roundrobin
>   mode http
>   http-request set-path /%[path,regsub(/v1/AUTH_.{32},swift/v1)]
>   server rgw1 10.0.0.111:8443 check ca-file ca.pem crt host.crt.pem ssl
>   server rgw2 10.0.0.230:8443 check ca-file ca.pem crt host.crt.pem ssl
>
> backend swift-pxy00
>   balance roundrobin
>   mode http
>   option httpchk HEAD /healthcheck HTTP/1.0
>   option forwardfor
>   option http-server-close
>   timeout http-keep-alive 500
>   server opxy1 10.0.0.123:8843 check ca-file ca.pem crt host.crt.pem ssl
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-hpc] What's the state of openstack-hpc now?

[Blair Bethwaite]

Hi Andrew, hi John -

I've just started trying to get CUDA P2P working in our virtualized
HPC environment. I figure this must be something you solved already in
order to produce the aforementioned paper, but having read it a couple
of times I don't think it provides enough detail about the guest
config, hoping you can shed some light...

The issue I'm grappling with is that despite using a qemu-kvm machine
type (q35) with an emulated PCIe bus and seeing that indeed the P2P
capable GPUs (NVIDIA K80s) are attached to that bus, and nvidia-smi
sees them as sharing a PHB, the simpleP2P CUDA sample fails when
checking their ability to communicate with each other. Is there some
magic config I might be missing, did you need to make any PCI-ACS
changes?

Best regards,
Blair


On 16 March 2016 at 07:57, Blair Bethwaite <blair.bethwa...@gmail.com> wrote:
>
> Hi Andrew,
>
> On 16 March 2016 at 05:28, Andrew J Younge <ajyou...@indiana.edu> wrote:
> > point to a recent publication of ours at VEE15 titled "Supporting High
> > Performance Molecular Dynamics in Virtualized Clusters using IOMMU,
> > SR-IOV, and GPUDirect."  In the paper we show that using Nvidia GPUs
> ...
> > http://dl.acm.org/citation.cfm?id=2731194
>
> Oooh interesting - GPUDirect too. That's something I've been wanting
> to try out in our environment. Will take a look a your paper...
>
> --
> Cheers,
> ~Blairo


-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Public cloud operators group in Barcelona

[Blair Bethwaite]

Hi Matt,

I think your dot points make sense. And yes, I was thinking about Science
Cloud overlap. I see Science Clouds as potentially sharing most or all of
these attributes (with the notable exception being charging in terms of end
users seeing a $ figure, showback and/or instance/cpu hour quotas might be
used instead). However, Science Clouds are probably not accessible to the
general public but require membership in, or sponsorship from, some
community in order to get an account.

The specific problems you mention, rather than attributes of the deployment
models, are probably the best way to define this in a way that let's folks
determine their interest. The biggest difference seems to be that
commercial providers have no major interest in the lifecycle management of
customer infrastructure, whereas for Science Clouds turning things off and
cleaning up data is a policy issue that does not appear to have resulted in
any common tooling yet.

Cheers,
Blair

On 22 Sep 2016 6:35 PM, "Matt Jarvis" <matt.jar...@datacentred.co.uk> wrote:

> Hey Blair
>
> Now you've done it ! OK, I'll bite and have a go at categorising that a
> bit :
>
> 1. Multi-tenant - tenants need clear separation
> 2. Self service sign up - customers on board themselves
> 3. Some kind of charging model in place which requires resource accounting
> 4. API endpoints and possibly management applications presented outside of
> your own network
>
> Not sure that entirely works, but best I can come up with off the top of
> my head. For commercial public clouds we have some very specific problem
> spaces in common - handling credit cards and personal information, fraud
> prevention, commercial modelling for capacity planning etc. Is where you're
> going with this that some of the science clouds share some of the
> attributes above ?
>
> Matt
>
> On 22 September 2016 at 00:40, Blair Bethwaite <blair.bethwa...@gmail.com>
> wrote:
>
>> Hi Matt,
>>
>> At considerable risk of heading down a rabbit hole... how are you
>> defining "public" cloud for these purposes?
>>
>> Cheers,
>> Blair
>>
>> On 21 September 2016 at 18:14, Matt Jarvis <matt.jar...@datacentred.co.uk
>> > wrote:
>>
>>> Given there are quite a few public cloud operators in Europe now, is
>>> there any interest in a public cloud group meeting as part of the ops
>>> meetup in Barcelona ? I already know many of you, but I think it could be
>>> very useful to share our experiences with a wider group.
>>>
>>> DataCentred Limited registered in England and Wales no. 05611763
>>> ___
>>> OpenStack-operators mailing list
>>> OpenStack-operators@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>>
>>
>>
>> --
>> Cheers,
>> ~Blairo
>>
>
>
> DataCentred Limited registered in England and Wales no. 05611763
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Public cloud operators group in Barcelona

[Blair Bethwaite]

Hi Matt,

At considerable risk of heading down a rabbit hole... how are you defining
"public" cloud for these purposes?

Cheers,
Blair

On 21 September 2016 at 18:14, Matt Jarvis 
wrote:

> Given there are quite a few public cloud operators in Europe now, is there
> any interest in a public cloud group meeting as part of the ops meetup in
> Barcelona ? I already know many of you, but I think it could be very useful
> to share our experiences with a wider group.
>
> DataCentred Limited registered in England and Wales no. 05611763
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>


-- 
Cheers,
~Blairo
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Converged infrastructure

[Blair Bethwaite]

Following on from Edmund's issues... People talking about doing this
typically seem to cite cgroups as the way to avoid CPU and memory
related contention - has anyone been successful in e.g. setting up
cgroups on a nova qemu+kvm hypervisor to limit how much of the machine
nova uses?

On 1 September 2016 at 04:15, Edmund Rhudy (BLOOMBERG/ 120 PARK)
 wrote:
> We currently run converged at Bloomberg with Ceph (all SSD) and I strongly
> dislike it. OSDs and VMs battle for CPU time and memory, VMs steal memory
> that would go to the HV pagecache, and it puts a real dent in any plans to
> be able to deploy hypervisors (mostly) statelessly. Ceph on our largest
> compute cluster spews an endless litany of deep-scrub-related HEALTH_WARNs
> because of memory steal from the VMs depleting available pagecache memory.
> We're going to increase the OS memory reservation in nova.conf to try to
> alleviate some of the worst of the memory steal, but it's been one hack
> after another to keep it going. I hope to be able to re-architect our design
> at some point to de-converge Ceph from the compute nodes so that the two
> sides can evolve separately once more.
>
> From: matt.jar...@datacentred.co.uk
> Subject: Re:[Openstack-operators] Converged infrastructure
>
> Time once again to dredge this topic up and see what the wider operators
> community thinks this time :) There were a fair amount of summit submissions
> for Barcelona talking about converged and hyper-converged infrastructure, it
> seems to be the topic de jour from vendors at the minute despite feeling
> like we've been round this before with Nebula, Piston Cloud etc.
>
> Like a lot of others we run Ceph, and we absolutely don't converge our
> storage and compute nodes for a variety of performance and management
> related reasons. In our experience, the hardware and tuning characteristics
> of both types of nodes are pretty different, in any kind of recovery
> scenarios Ceph eats memory, and it feels like creating a SPOF.
>
> Having said that, with pure SSD clusters becoming more common, some of those
> issues may well be mitigated, so is anyone doing this in production now ? If
> so, what does your hardware platform look like, and are there issues with
> these kinds of architectures ?
>
> Matt
>
> DataCentred Limited registered in England and Wales no. 05611763
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [scientific] Seeking users of Infiniband

[Blair Bethwaite]

Hi Stig,

When you say IB are you specifically talking about link-layer, or more the
RDMA capability and IB semantics supported by the drivers and APIs (so both
native IB and RoCE)?

Cheers,

On 17 Aug 2016 2:28 AM, "Stig Telfer"  wrote:

> Hi All -
>
> I’m looking for some new data points on people’s experience of using
> Infiniband within a virtualised OpenStack configuration.
>
> Is there anyone on the list who is doing this, and how well does it work?
>
> Many thanks,
> Stig
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Blazar? (Reservations and/or scheduled termination)

[Blair Bethwaite]

We discussed Blazar fairly extensively in a couple of recent
scientific-wg meetings. I'm having trouble searching out right the irc
log to support this but IIRC the problem with Blazar as is for the
typical virtualised cloud (non-Ironic) use-case is that it uses an
old/deprecated Nova API extension in order to reserve capacity for a
lease (I think it was something to do with shelve-offload, but I could
have my wires crossed). Whereas with Ironic it manipulates
host-aggregates.

I like the idea of the positive, "yes I'm still using this",
confirmation, but it needs a tie into the dashboard. For those of us
using identity federations to bootstrap users this would also help
with account cleanup, as users who no longer have access (e.g., left
the sector) to the dashboard would not be able to extend resource
usage (without talking to support etc).

Cheers,

On 4 August 2016 at 03:23, Tim Bell  wrote:
>
> When I last looked, Blazar allows you to reserve instances for a given time. 
> An example would be
>
> - We are organizing a user training session for 100 physicists from Monday to 
> Friday
> - We need that each user is able to create 2 VMs within a single shared 
> project (as the images etc. are set up before)
> - OpenStack should ensure that these resources would be available (or reject 
> the request) and schedule other Blazar requests around it
>
> This does need other functionality though which is currently not there
>
> - Spot instances, i.e. give me the resources if they are available but kill 
> when you have a reservation. Alvaro is proposing a talk for some work done in 
> Indigo Datacloud project for this at the summit (votes welcome).
> - Impending shutdown notification .. save what you want quickly because 
> you’re going to be deleted
>
> We have a use case where we offer each user a quota of 2 VMs for ‘Personal’ 
> use. This gives a good onboarding experience but the problem is that our 
> users forget they asked for resources. Something where we can define a 
> default lifetime for a VM (ideally, a project) and users need to positively 
> confirm they want to extend the lifetime would help identify these forgotten 
> VMs.
>
> I think a good first step would be
>
> - An agreed metadata structure for a VM with an expiry date
> - An agreed project metadata giving the default length of the VM
> - An OSops script which finds those VMs exceeding the agreed period and goes 
> through a ‘suspend’ for N days followed by a delete M days afterwards (to 
> catch the accidents)
>
> I think this can be done in Nova as is if this is not felt to be a ‘standard’ 
> function but we should agree on the names/concepts.
>
> Some of the needs are covered in 
> https://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html
>
> Tim
>
>
> On 03/08/16 18:47, "Jonathan D. Proulx"  wrote:
>
> Hi All,
>
> As a private cloud operatior who doesn't charge internal users, I'd
> really like a way to force users to set an exiration time on their
> instances so if they forget about them they go away.
>
> I'd though Blazar was the thing to look at and Chameleoncloud.org
> seems to be using it (any of you around here?) but it also doesn't
> look like it's seen substantive work in a long time.
>
> Anyone have operational exprience with blazar to share or other
> solutions?
>
> -Jon
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Blazar? (Reservations and/or scheduled termination)

[Blair Bethwaite]

On 4 August 2016 at 12:48, Sam Morrison  wrote:
>
>> On 4 Aug 2016, at 3:12 AM, Kris G. Lindgren  wrote:
>>
>> We do something similar.  We give everyone in the company an account on the 
>> internal cloud.  By default they have a user- project.  We have a 
>> Jenkins job that adds metadata to all vm’s that are in user- projects.  We 
>> then have additional jobs that read that metadata and determine when the VM 
>> has been alive for x period of time.  At 45 days we send an email saying 
>> that we will remove the vm in 15 days, and they can request a 30 day 
>> extension (which really just resets some metadata information on the vm).  
>> On day 60 the vm is shut down and removed.  For non user- projects, people 
>> are allowed to have their vm’s created as long as they want.
>
> What stops a user modifying the metadata? Do you have novas policy.json set 
> up so they can’t?

System-metadata?

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] nova resize on shared storage

[Blair Bethwaite]

On 1 August 2016 at 13:30, Marcus Furlong  wrote:
> Looks like there is a bug open which suggests that it should be using
> RPC calls, rather than commands executed over ssh:
>
> https://bugs.launchpad.net/nova/+bug/1459782

I agree, no operator in their right mind wants to turn this on for a
production cloud, but it's a capability that a lot of useful higher
level tooling wants to exploit (e.g. right-sizing solutions). IIRC
this was discussed some time ago and I thought there was something in
the dev pipeline to address it. Looking at the bug it does mention the
related live-migration cleanup work that happened ~Havana or so, I
guess the cold-migrate/resize pathway was missed or did it get stuck
in review?

On this point in the bug report:
==
There's a complication though. In virt.libvirt.utils.copy_image() we
also rely on passwordless authentication to do either "rsync" or "scp"
to copy the image file over when doing cold migration with local
storage. So for the case of local storage we'd still need to set up
passwordless ssh between compute nodes to handle cold migration.
==

Passwordless ssh for services need not be so scary, it just needs to
be managed right... Fortunately OpenSSH has a rather cool feature
(that lots of people seem not to know about) - it supports auth by
certificate, by which I mean an appropriately configured sshd can
authenticate a client's cert based on the fact that it was signed by a
trusted SSH CA without any need to have a record of the client's
public key. Signed certs are valid for a limited time, so you can
imagine building some automation that created a short-lived cert on
demand that was valid just long enough to establish the scp connection
needed to complete a cold-migration or resize. See "man ssh-keygen" ->
CERTIFICATES.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [kolla] question of default users for operators

[Blair Bethwaite]

Sounds like a recipe for confusion?

On 1 August 2016 at 10:23, Steven Dake (stdake)  wrote:
>
>
> On 7/31/16, 7:13 AM, "Jay Pipes"  wrote:
>
>>On 07/29/2016 11:35 PM, Steven Dake (stdake) wrote:
>>> Hey folks,
>>>
>>> In Kolla we have a significant bug in that Horizon can't be used because
>>> it requires a member user.  We have a few approaches to fixing this
>>> problem in mind, but want to understand what Operators want.  Devstack
>>> itself has switched between member and _member_ although Devstack isn't
>>> our reference: Operators are.
>>>
>>> Which of those do operators prefer (if either).
>>>
>>> Another question was whether we should create a default user named
>>> "user" and use this in place of member.
>>
>>"member" is a role, not a user. The default unprivileged user is named
>>"demo" and has the "member" role in a project called "demo".
>
> Yup, thanks for the correction.
>
>>
>>Are you asking what the name of the unprivileged *role* should be by
>>default in Kolla installations? Or are you asking what, if any, a
>>default unprivileged *user* should be called?
>
> The idea of adding a user called "user" with a role called "user" was
> proposed.  Curious if operators care to want that.
>
> Regards
> -steve
>
>>
>>Best,
>>-jay
>>
>>___
>>OpenStack-operators mailing list
>>OpenStack-operators@lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] PCI Passthrough issues

[Blair Bethwaite]

Hi Joe, Jon -

We seem to be good now on both qemu 2.3 and 2.5 with kernel 3.19
(lowest we've tried). Also thanks to Jon we had an easy fix for the
snapshot issues!

Next question - has anyone figured out how to make GPU P2P work? We
haven't tried very hard yet, but with our current setup we're telling
Nova to pass through the GK210GL "3D controller" and that results in
the guest seeing individual GPUs attached to a virtualised PCI bus,
even when e.g. passing through two K80s on the same board. Next
obvious step is to try passing through the on-board PLX PCI bridge,
but wondering whether anyone else has been down this path yet?

Cheers,

On 20 July 2016 at 12:57, Blair Bethwaite <blair.bethwa...@gmail.com> wrote:
> Thanks for the confirmation Joe!
>
> On 20 July 2016 at 12:19, Joe Topjian <j...@topjian.net> wrote:
>> Hi Blair,
>>
>> We only updated qemu. We're running the version of libvirt from the Kilo
>> cloudarchive.
>>
>> We've been in production with our K80s for around two weeks now and have had
>> several users report success.
>>
>> Thanks,
>> Joe
>>
>> On Tue, Jul 19, 2016 at 5:06 PM, Blair Bethwaite <blair.bethwa...@gmail.com>
>> wrote:
>>>
>>> Hilariously (or not!) we finally hit the same issue last week once
>>> folks actually started trying to do something (other than build and
>>> load drivers) with the K80s we're passing through. This
>>>
>>> https://devtalk.nvidia.com/default/topic/850833/pci-passthrough-kvm-for-cuda-usage/
>>> is the best discussion of the issue I've found so far, haven't tracked
>>> down an actual bug yet though. I wonder whether it has something to do
>>> with the memory size of the device, as we've been happy for a long
>>> time with other NVIDIA GPUs (GRID K1, K2, M2070, ...).
>>>
>>> Jon, when you grabbed Mitaka Qemu, did you also update libvirt? We're
>>> just working through this and have tried upgrading both but are
>>> hitting some issues with Nova and Neutron on the compute nodes,
>>> thinking it may libvirt related but debug isn't helping much yet.
>>>
>>> Cheers,
>>>
>>> On 8 July 2016 at 00:54, Jonathan Proulx <j...@csail.mit.edu> wrote:
>>> > On Thu, Jul 07, 2016 at 11:13:29AM +1000, Blair Bethwaite wrote:
>>> > :Jon,
>>> > :
>>> > :Awesome, thanks for sharing. We've just run into an issue with SRIOV
>>> > :VF passthrough that sounds like it might be the same problem (device
>>> > :disappearing after a reboot), but haven't yet investigated deeply -
>>> > :this will help with somewhere to start!
>>> >
>>> > :By the way, the nouveau mention was because we had missed it on some
>>> > :K80 hypervisors recently and seen passthrough apparently work, but
>>> > :then the NVIDIA drivers would not build in the guest as they claimed
>>> > :they could not find a supported device (despite the GPU being visible
>>> > :on the PCI bus).
>>> >
>>> > Definitely sage advice!
>>> >
>>> > :I have also heard passing mention of requiring qemu
>>> > :2.3+ but don't have any specific details of the related issue.
>>> >
>>> > I didn't do a bisection but with qemu 2.2 (from ubuntu cloudarchive
>>> > kilo) I was sad and with 2.5 (from ubuntu cloudarchive mitaka but
>>> > installed on a kilo hypervisor) I am working.
>>> >
>>> > Thanks,
>>> > -Jon
>>> >
>>> >
>>> > :Cheers,
>>> > :
>>> > :On 7 July 2016 at 08:13, Jonathan Proulx <j...@csail.mit.edu> wrote:
>>> > :> On Wed, Jul 06, 2016 at 12:32:26PM -0400, Jonathan D. Proulx wrote:
>>> > :> :
>>> > :> :I do have an odd remaining issue where I can run cuda jobs in the vm
>>> > :> :but snapshots fail and after pause (for snapshotting) the pci device
>>> > :> :can't be reattached (which is where i think it deletes the snapshot
>>> > :> :it took).  Got same issue with 3.16 and 4.4 kernels.
>>> > :> :
>>> > :> :Not very well categorized yet, but I'm hoping it's because the VM I
>>> > :> :was hacking on had it's libvirt.xml written out with the older qemu
>>> > :> :maybe?  It had been through a couple reboots of the physical system
>>> > :> :though.
>>> > :> :
>>> > :> :Currently building a fresh instance and bashing more keys...
>>> > :>
>>> > :> After an ugly bout of b

Re: [Openstack-operators] [swift] data distribution imbalance - what are the .data.XXXXX files?

[Blair Bethwaite]

Romain,

Thanks a lot for that pointer, it was surprisingly difficult to find
anything about this via usual web search. After a little scripting and
a bit of coordinated effort we're looking much healthier.

We noticed the worst offender directories appeared to have multiple
"nested" rsync temp files left behind, i.e., the replicator on a peer
host was trying to copy existing rsync partial files (e.g.
./objects/27913/91c/1b4263d0e93452b461572bbf57f9591c/.1467136978.92866.data.1RdQku.bCyhfi.6JlGol)
and thus quickly compounding the problem!

Cheers,

On 20 July 2016 at 20:25, Romain LE DISEZ <romain.openst...@ledisez.net> wrote:
> Hi Blair,
>
> they are temporary files from rsync, when the replicator tried to replicate a 
> partition and failed for some reason.
>
> You can safely delete them as long as there mtime is a bit old (do not delete 
> a file currently being replicated). Since 2.7, Swift take care of that:
> https://github.com/openstack/swift/blob/master/CHANGELOG#L226
>
>
>
> Le Mercredi 20 Juillet 2016 10:17 CEST, Blair Bethwaite 
> <blair.bethwa...@gmail.com> a écrit:
>
>> Hi all,
>>
>> As per the subject, wondering where these files come from, e.g.,:
>>
>> root@stor010:/srv/node/sdc1/objects# ls -la
>> ./109794/359/6b389b24749b7046344ffd2a42aab359
>> total 1195784
>> drwxr-xr-x 2 swift swift  4096 Jun  8 04:11 .
>> drwxr-xr-x 3 swift swift53 May 22 05:05 ..
>> -rw--- 1 swift swift 20480 Jun  8 04:11 1463857426.65100.data
>> -rw--- 1 swift swift   6225920 Jun  3 00:42 .1463857426.65100.data.aCtGLk
>> -rw--- 1 swift swift 197754880 Jun  2 11:49 .1463857426.65100.data.AMQhPo
>> -rw--- 1 swift swift  33980416 Jun  3 00:41 .1463857426.65100.data.CkpDSv
>> -rw--- 1 swift swift   7634944 Jun  3 04:02
>> .1463857426.65100.data.CkpDSv.CtrQws
>> -rw--- 1 swift swift 189399040 Jun  1 18:42 .1463857426.65100.data.CRFb2k
>> -rw--- 1 swift swift  47644672 Jun  2 11:51 .1463857426.65100.data.dKsUZI
>> -rw--- 1 swift swift 157122560 Jun  3 13:57 .1463857426.65100.data.GpmbOK
>> -rw--- 1 swift swift 174489600 Jun  2 11:50 .1463857426.65100.data.MAoI3y
>> -rw--- 1 swift swift 174358528 Jun  3 00:42 .1463857426.65100.data.Pbsk7S
>> -rw--- 1 swift swift  31064064 Jun  1 18:42 .1463857426.65100.data.xlmmie
>>
>> We have a geo-replicated cluster that is currently suffering from
>> major outliers in disk usage, i.e.:
>>
>> [2016-07-20 18:08:33] Checking disk usage now
>> Distribution Graph:
>>   0%2 *
>>  32%1
>>  34%   19 **
>>  35%   49 **
>>  36%  127 
>> *
>>  37%  111 
>>  38%   40 *
>>  39%   12 **
>>  40%6 ***
>>  41%6 ***
>>  42%2 *
>>  43%4 **
>>  44%3 *
>>  45%1
>>  46%3 *
>>  47%4 **
>>  48%3 *
>>  50%1
>>  51%2 *
>>  52%1
>>  53%1
>>  54%1
>>  56%3 *
>>  58%1
>>  62%2 *
>>  63%1
>>  71%1
>>  73%1
>>  75%1
>>  76%1
>>  78%2 *
>>  88%1
>>  92%2 *
>>  95%1
>>  96%1
>>  100%3 *
>> Disk usage: space used: 395001580875776 of 995614295568384
>> Disk usage: space free: 600612714692608 of 995614295568384
>> Disk usage: lowest: 0.0%, highest: 100.0%, avg: 39.6741572147%
>>
>> It looks like this is attributable to a handful of object directories
>> with lots of data.XX files in them, whereas >99% of object dirs
>> just have a single .data file, e.g. this from one of the disks at
>> ~60%:
>>
>> root@stor010:/srv/node/sdc1/objects# find . -mindepth 4 -type f
>> -printf "%h\n" | sort | uniq -c | sort -rnk 1 | head -20
>>
>> 733 ./151107/3b5/9390f9c2ceee07f059a0d1f651e423b5
>>  11 ./109794/359/6b389b24749b7046344ffd2a42aab359
>>   9 ./248385/60c/f2907cb0b290def6f614bf46a715a60c
>>   5 ./222791/888/d991c8db1e2f1e724c1a4f52914f7888
>>   4 ./257772/140/fbbb1c017a841e6e821ed707025fe140
>>   4 ./231068/ca6/e1a706f50dd99f97fafeba6bd1f47ca6
>>   4 ./215734/80c/d2adbf087b09ca24cc546497d265180c
>>   3 ./248166/8b9/f259b3b0113b522ec5ef5753588438b9
>>   3 ./221060/383/d7e101fd86eeef65f8c89f3f99ce4383
>>   2 ./38609/203/25b46d47af8ee700e87ceab33748b203
>>   2 ./27961/a78/1b4e5665c16e1da8a70cc093a2bbba78
>>   2 ./158466/d43

[Openstack-operators] [swift] data distribution imbalance - what are the .data.XXXXX files?

[Blair Bethwaite]

Hi all,

As per the subject, wondering where these files come from, e.g.,:

root@stor010:/srv/node/sdc1/objects# ls -la
./109794/359/6b389b24749b7046344ffd2a42aab359
total 1195784
drwxr-xr-x 2 swift swift  4096 Jun  8 04:11 .
drwxr-xr-x 3 swift swift53 May 22 05:05 ..
-rw--- 1 swift swift 20480 Jun  8 04:11 1463857426.65100.data
-rw--- 1 swift swift   6225920 Jun  3 00:42 .1463857426.65100.data.aCtGLk
-rw--- 1 swift swift 197754880 Jun  2 11:49 .1463857426.65100.data.AMQhPo
-rw--- 1 swift swift  33980416 Jun  3 00:41 .1463857426.65100.data.CkpDSv
-rw--- 1 swift swift   7634944 Jun  3 04:02
.1463857426.65100.data.CkpDSv.CtrQws
-rw--- 1 swift swift 189399040 Jun  1 18:42 .1463857426.65100.data.CRFb2k
-rw--- 1 swift swift  47644672 Jun  2 11:51 .1463857426.65100.data.dKsUZI
-rw--- 1 swift swift 157122560 Jun  3 13:57 .1463857426.65100.data.GpmbOK
-rw--- 1 swift swift 174489600 Jun  2 11:50 .1463857426.65100.data.MAoI3y
-rw--- 1 swift swift 174358528 Jun  3 00:42 .1463857426.65100.data.Pbsk7S
-rw--- 1 swift swift  31064064 Jun  1 18:42 .1463857426.65100.data.xlmmie

We have a geo-replicated cluster that is currently suffering from
major outliers in disk usage, i.e.:

[2016-07-20 18:08:33] Checking disk usage now
Distribution Graph:
  0%2 *
 32%1
 34%   19 **
 35%   49 **
 36%  127 *
 37%  111 
 38%   40 *
 39%   12 **
 40%6 ***
 41%6 ***
 42%2 *
 43%4 **
 44%3 *
 45%1
 46%3 *
 47%4 **
 48%3 *
 50%1
 51%2 *
 52%1
 53%1
 54%1
 56%3 *
 58%1
 62%2 *
 63%1
 71%1
 73%1
 75%1
 76%1
 78%2 *
 88%1
 92%2 *
 95%1
 96%1
 100%3 *
Disk usage: space used: 395001580875776 of 995614295568384
Disk usage: space free: 600612714692608 of 995614295568384
Disk usage: lowest: 0.0%, highest: 100.0%, avg: 39.6741572147%

It looks like this is attributable to a handful of object directories
with lots of data.XX files in them, whereas >99% of object dirs
just have a single .data file, e.g. this from one of the disks at
~60%:

root@stor010:/srv/node/sdc1/objects# find . -mindepth 4 -type f
-printf "%h\n" | sort | uniq -c | sort -rnk 1 | head -20

733 ./151107/3b5/9390f9c2ceee07f059a0d1f651e423b5
 11 ./109794/359/6b389b24749b7046344ffd2a42aab359
  9 ./248385/60c/f2907cb0b290def6f614bf46a715a60c
  5 ./222791/888/d991c8db1e2f1e724c1a4f52914f7888
  4 ./257772/140/fbbb1c017a841e6e821ed707025fe140
  4 ./231068/ca6/e1a706f50dd99f97fafeba6bd1f47ca6
  4 ./215734/80c/d2adbf087b09ca24cc546497d265180c
  3 ./248166/8b9/f259b3b0113b522ec5ef5753588438b9
  3 ./221060/383/d7e101fd86eeef65f8c89f3f99ce4383
  2 ./38609/203/25b46d47af8ee700e87ceab33748b203
  2 ./27961/a78/1b4e5665c16e1da8a70cc093a2bbba78
  2 ./158466/d43/9ac0b8bd4e592fe6f3360a731bac7d43
  2 ./141275/588/89f6efb554e964aeebffa9dad9e17588
  1 ./99980/fad/61a3214454426f7b30fc62773eb3bfad
  1 ./99980/fa4/61a311d12248a2e4ca8d2f61a3adafa4
  1 ./99980/ed9/61a32b429174f6099b0b35b0103e4ed9
  1 ./99980/e76/61a3129de66d7541ce127f07a7737e76
  1 ./99980/e3d/61a329332713fdacedb10834901f6e3d
  1 ./99980/e1f/61a306b87307636cc569bd0bc047ee1f
  1 ./99980/db8/61a30a00a9b7edd3318be25bbed47db8

root@stor010:/srv/node/sdc1/objects# du -sh
./151107/3b5/9390f9c2ceee07f059a0d1f651e423b5
957G./151107/3b5/9390f9c2ceee07f059a0d1f651e423b5

We're on version 2.5.0.7.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] PCI Passthrough issues

[Blair Bethwaite]

Thanks for the confirmation Joe!

On 20 July 2016 at 12:19, Joe Topjian <j...@topjian.net> wrote:
> Hi Blair,
>
> We only updated qemu. We're running the version of libvirt from the Kilo
> cloudarchive.
>
> We've been in production with our K80s for around two weeks now and have had
> several users report success.
>
> Thanks,
> Joe
>
> On Tue, Jul 19, 2016 at 5:06 PM, Blair Bethwaite <blair.bethwa...@gmail.com>
> wrote:
>>
>> Hilariously (or not!) we finally hit the same issue last week once
>> folks actually started trying to do something (other than build and
>> load drivers) with the K80s we're passing through. This
>>
>> https://devtalk.nvidia.com/default/topic/850833/pci-passthrough-kvm-for-cuda-usage/
>> is the best discussion of the issue I've found so far, haven't tracked
>> down an actual bug yet though. I wonder whether it has something to do
>> with the memory size of the device, as we've been happy for a long
>> time with other NVIDIA GPUs (GRID K1, K2, M2070, ...).
>>
>> Jon, when you grabbed Mitaka Qemu, did you also update libvirt? We're
>> just working through this and have tried upgrading both but are
>> hitting some issues with Nova and Neutron on the compute nodes,
>> thinking it may libvirt related but debug isn't helping much yet.
>>
>> Cheers,
>>
>> On 8 July 2016 at 00:54, Jonathan Proulx <j...@csail.mit.edu> wrote:
>> > On Thu, Jul 07, 2016 at 11:13:29AM +1000, Blair Bethwaite wrote:
>> > :Jon,
>> > :
>> > :Awesome, thanks for sharing. We've just run into an issue with SRIOV
>> > :VF passthrough that sounds like it might be the same problem (device
>> > :disappearing after a reboot), but haven't yet investigated deeply -
>> > :this will help with somewhere to start!
>> >
>> > :By the way, the nouveau mention was because we had missed it on some
>> > :K80 hypervisors recently and seen passthrough apparently work, but
>> > :then the NVIDIA drivers would not build in the guest as they claimed
>> > :they could not find a supported device (despite the GPU being visible
>> > :on the PCI bus).
>> >
>> > Definitely sage advice!
>> >
>> > :I have also heard passing mention of requiring qemu
>> > :2.3+ but don't have any specific details of the related issue.
>> >
>> > I didn't do a bisection but with qemu 2.2 (from ubuntu cloudarchive
>> > kilo) I was sad and with 2.5 (from ubuntu cloudarchive mitaka but
>> > installed on a kilo hypervisor) I am working.
>> >
>> > Thanks,
>> > -Jon
>> >
>> >
>> > :Cheers,
>> > :
>> > :On 7 July 2016 at 08:13, Jonathan Proulx <j...@csail.mit.edu> wrote:
>> > :> On Wed, Jul 06, 2016 at 12:32:26PM -0400, Jonathan D. Proulx wrote:
>> > :> :
>> > :> :I do have an odd remaining issue where I can run cuda jobs in the vm
>> > :> :but snapshots fail and after pause (for snapshotting) the pci device
>> > :> :can't be reattached (which is where i think it deletes the snapshot
>> > :> :it took).  Got same issue with 3.16 and 4.4 kernels.
>> > :> :
>> > :> :Not very well categorized yet, but I'm hoping it's because the VM I
>> > :> :was hacking on had it's libvirt.xml written out with the older qemu
>> > :> :maybe?  It had been through a couple reboots of the physical system
>> > :> :though.
>> > :> :
>> > :> :Currently building a fresh instance and bashing more keys...
>> > :>
>> > :> After an ugly bout of bashing I've solve my failing snapshot issue
>> > :> which I'll post here in hopes of saving someonelse
>> > :>
>> > :> Short version:
>> > :>
>> > :> add "/dev/vfio/vfio rw," to
>> > /etc/apparmor.d/abstractions/libvirt-qemu
>> > :> add "ulimit -l unlimited" to /etc/init/libvirt-bin.conf
>> > :>
>> > :> Longer version:
>> > :>
>> > :> What was happening.
>> > :>
>> > :> * send snapshot request
>> > :> * instance pauses while snapshot is pending
>> > :> * instance attempt to resume
>> > :> * fails to reattach pci device
>> > :>   * nova-compute.log
>> > :> Exception during message handling: internal error: unable to
>> > execute QEMU command 'device_add': Device initialization failedcompute.log
>> > :>
>> > :>   * qemu/.log
>> > :> vfio: failed to open /dev/vfio/vfio: 

Re: [Openstack-operators] PCI Passthrough issues

[Blair Bethwaite]

Hilariously (or not!) we finally hit the same issue last week once
folks actually started trying to do something (other than build and
load drivers) with the K80s we're passing through. This
https://devtalk.nvidia.com/default/topic/850833/pci-passthrough-kvm-for-cuda-usage/
is the best discussion of the issue I've found so far, haven't tracked
down an actual bug yet though. I wonder whether it has something to do
with the memory size of the device, as we've been happy for a long
time with other NVIDIA GPUs (GRID K1, K2, M2070, ...).

Jon, when you grabbed Mitaka Qemu, did you also update libvirt? We're
just working through this and have tried upgrading both but are
hitting some issues with Nova and Neutron on the compute nodes,
thinking it may libvirt related but debug isn't helping much yet.

Cheers,

On 8 July 2016 at 00:54, Jonathan Proulx <j...@csail.mit.edu> wrote:
> On Thu, Jul 07, 2016 at 11:13:29AM +1000, Blair Bethwaite wrote:
> :Jon,
> :
> :Awesome, thanks for sharing. We've just run into an issue with SRIOV
> :VF passthrough that sounds like it might be the same problem (device
> :disappearing after a reboot), but haven't yet investigated deeply -
> :this will help with somewhere to start!
>
> :By the way, the nouveau mention was because we had missed it on some
> :K80 hypervisors recently and seen passthrough apparently work, but
> :then the NVIDIA drivers would not build in the guest as they claimed
> :they could not find a supported device (despite the GPU being visible
> :on the PCI bus).
>
> Definitely sage advice!
>
> :I have also heard passing mention of requiring qemu
> :2.3+ but don't have any specific details of the related issue.
>
> I didn't do a bisection but with qemu 2.2 (from ubuntu cloudarchive
> kilo) I was sad and with 2.5 (from ubuntu cloudarchive mitaka but
> installed on a kilo hypervisor) I am working.
>
> Thanks,
> -Jon
>
>
> :Cheers,
> :
> :On 7 July 2016 at 08:13, Jonathan Proulx <j...@csail.mit.edu> wrote:
> :> On Wed, Jul 06, 2016 at 12:32:26PM -0400, Jonathan D. Proulx wrote:
> :> :
> :> :I do have an odd remaining issue where I can run cuda jobs in the vm
> :> :but snapshots fail and after pause (for snapshotting) the pci device
> :> :can't be reattached (which is where i think it deletes the snapshot
> :> :it took).  Got same issue with 3.16 and 4.4 kernels.
> :> :
> :> :Not very well categorized yet, but I'm hoping it's because the VM I
> :> :was hacking on had it's libvirt.xml written out with the older qemu
> :> :maybe?  It had been through a couple reboots of the physical system
> :> :though.
> :> :
> :> :Currently building a fresh instance and bashing more keys...
> :>
> :> After an ugly bout of bashing I've solve my failing snapshot issue
> :> which I'll post here in hopes of saving someonelse
> :>
> :> Short version:
> :>
> :> add "/dev/vfio/vfio rw," to  /etc/apparmor.d/abstractions/libvirt-qemu
> :> add "ulimit -l unlimited" to /etc/init/libvirt-bin.conf
> :>
> :> Longer version:
> :>
> :> What was happening.
> :>
> :> * send snapshot request
> :> * instance pauses while snapshot is pending
> :> * instance attempt to resume
> :> * fails to reattach pci device
> :>   * nova-compute.log
> :> Exception during message handling: internal error: unable to execute 
> QEMU command 'device_add': Device initialization failedcompute.log
> :>
> :>   * qemu/.log
> :> vfio: failed to open /dev/vfio/vfio: Permission denied
> :> vfio: failed to setup container for group 48
> :> vfio: failed to get group 48
> :> * snapshot disappears
> :> * instance resumes but without passed through device (hard reboot
> :> reattaches)
> :>
> :> seeing permsission denied I though would be an easy fix but:
> :>
> :> # ls -l /dev/vfio/vfio
> :> crw-rw-rw- 1 root root 10, 196 Jul  6 14:05 /dev/vfio/vfio
> :>
> :> so I'm guessing I'm in apparmor hell, I try adding "/dev/vfio/vfio
> :> rw," to  /etc/apparmor.d/abstractions/libvirt-qemu rebooting the
> :> hypervisor and trying again which gets me a different libvirt error
> :> set:
> :>
> :> VFIO_MAP_DMA: -12
> :> vfio_dma_map(0x5633a5fa69b0, 0x0, 0xa, 0x7f4e7be0) = -12 (Cannot 
> allocate memory)
> :>
> :> kern.log (and thus dmesg) showing:
> :> vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
> :>
> :> Getting rid of this one required inserting 'ulimit -l unlimited' into
> :> /etc/init/libvirt-bin.conf in the 'script' section:
> :>
> :> 
> :> script
> :> [ -r /etc/default/libvirt-bin ] && . /

Re: [Openstack-operators] Reaching VXLAN tenant networks from outside (without floating IPs)

[Blair Bethwaite]

On 30 June 2016 at 05:17, Gustavo Randich  wrote:
>
> - other?

FWIW, the other approach that might be suitable (depending on your
project/tenant isolation requirements) is simply using a flat provider
network (or networks, i.e., VLAN per project) within your existing
managed private address space, then you have no requirement for a
Neutron router. This model seems a lot easier to visualise when
starting out with Neutron and can side-step a lot of integration
problems.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific-wg] Meeting reminder: in ~7 hours @ 2100 UTC in #openstack-meeting

[Blair Bethwaite]

Hi all,

Scientific-WG regular meeting is on soon, draft agenda below and at
https://wiki.openstack.org/wiki/Scientific_working_group.


2016-07-12 2100 UTC in channel #openstack-meeting

# Review of Activity Areas and opportunities for progress
## Bare metal
### Networking requirements/considerations
## Parallel filesystems
## Accounting and scheduling
# OpenStack & HPC white paper
# Other business
## Contributions sought to the Hypervisor Tuning Guide
## HPC / Research speaker track at Barcelona Summit

Regards,
Blair & Stig

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific-wg] high-performance/parallel file-systems panel at Barcelona

[Blair Bethwaite]

Hi all,

Just pondering summit talk submissions and wondering if anyone else
out there is interested in participating in a HPFS panel session...?

Assuming we have at least one person already who can cover direct
mounting of Lustre into OpenStack guests then it'd be nice to find
folks who have experience integrating with other filesystems and/or
approaches, e.g., GPFS, Manila, Object Storage translation gateways.

-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] PCI Passthrough issues

[Blair Bethwaite]

Jon,

Awesome, thanks for sharing. We've just run into an issue with SRIOV
VF passthrough that sounds like it might be the same problem (device
disappearing after a reboot), but haven't yet investigated deeply -
this will help with somewhere to start!

By the way, the nouveau mention was because we had missed it on some
K80 hypervisors recently and seen passthrough apparently work, but
then the NVIDIA drivers would not build in the guest as they claimed
they could not find a supported device (despite the GPU being visible
on the PCI bus). I have also heard passing mention of requiring qemu
2.3+ but don't have any specific details of the related issue.

Cheers,

On 7 July 2016 at 08:13, Jonathan Proulx  wrote:
> On Wed, Jul 06, 2016 at 12:32:26PM -0400, Jonathan D. Proulx wrote:
> :
> :I do have an odd remaining issue where I can run cuda jobs in the vm
> :but snapshots fail and after pause (for snapshotting) the pci device
> :can't be reattached (which is where i think it deletes the snapshot
> :it took).  Got same issue with 3.16 and 4.4 kernels.
> :
> :Not very well categorized yet, but I'm hoping it's because the VM I
> :was hacking on had it's libvirt.xml written out with the older qemu
> :maybe?  It had been through a couple reboots of the physical system
> :though.
> :
> :Currently building a fresh instance and bashing more keys...
>
> After an ugly bout of bashing I've solve my failing snapshot issue
> which I'll post here in hopes of saving someonelse
>
> Short version:
>
> add "/dev/vfio/vfio rw," to  /etc/apparmor.d/abstractions/libvirt-qemu
> add "ulimit -l unlimited" to /etc/init/libvirt-bin.conf
>
> Longer version:
>
> What was happening.
>
> * send snapshot request
> * instance pauses while snapshot is pending
> * instance attempt to resume
> * fails to reattach pci device
>   * nova-compute.log
> Exception during message handling: internal error: unable to execute QEMU 
> command 'device_add': Device initialization failedcompute.log
>
>   * qemu/.log
> vfio: failed to open /dev/vfio/vfio: Permission denied
> vfio: failed to setup container for group 48
> vfio: failed to get group 48
> * snapshot disappears
> * instance resumes but without passed through device (hard reboot
> reattaches)
>
> seeing permsission denied I though would be an easy fix but:
>
> # ls -l /dev/vfio/vfio
> crw-rw-rw- 1 root root 10, 196 Jul  6 14:05 /dev/vfio/vfio
>
> so I'm guessing I'm in apparmor hell, I try adding "/dev/vfio/vfio
> rw," to  /etc/apparmor.d/abstractions/libvirt-qemu rebooting the
> hypervisor and trying again which gets me a different libvirt error
> set:
>
> VFIO_MAP_DMA: -12
> vfio_dma_map(0x5633a5fa69b0, 0x0, 0xa, 0x7f4e7be0) = -12 (Cannot 
> allocate memory)
>
> kern.log (and thus dmesg) showing:
> vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
>
> Getting rid of this one required inserting 'ulimit -l unlimited' into
> /etc/init/libvirt-bin.conf in the 'script' section:
>
> 
> script
> [ -r /etc/default/libvirt-bin ] && . /etc/default/libvirt-bin
> ulimit -l unlimited
> exec /usr/sbin/libvirtd $libvirtd_opts
> end script
>
>
> -Jon
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] PCI Passthrough issues

[Blair Bethwaite]

Hi Jon,

Do you have the nouveau driver/module loaded in the host by any
chance? If so, blacklist, reboot, repeat.

Whilst we're talking about this. Has anyone had any luck doing this
with hosts having a PCI-e switch across multiple GPUs?

Cheers,

On 6 July 2016 at 23:27, Jonathan D. Proulx  wrote:
> Hi All,
>
> Trying to spass through some Nvidia K80 GPUs to soem instance and have
> gotten to the place where Nova seems to be doing the right thing gpu
> instances scheduled on the 1 gpu hypervisor I have and for inside the
> VM I see:
>
> root@gpu-x1:~# lspci | grep -i k80
> 00:06.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
>
> And I can install nvdia-361 driver and get
>
> # ls /dev/nvidia*
> /dev/nvidia0  /dev/nvidiactl  /dev/nvidia-uvm  /dev/nvidia-uvm-tools
>
> Once I load up cuda-7.5 and build the exmaples none fo the run
> claiming there's no cuda device.
>
> # ./matrixMul
> [Matrix Multiply Using CUDA] - Starting...
> cudaGetDevice returned error no CUDA-capable device is detected (code 38), 
> line(396)
> cudaGetDeviceProperties returned error no CUDA-capable device is detected 
> (code 38), line(409)
> MatrixA(160,160), MatrixB(320,160)
> cudaMalloc d_A returned error no CUDA-capable device is detected (code 38), 
> line(164)
>
> I'm not familiar with cuda really but I did get some example code
> running on the physical system for burn in over the weekend (sicne
> reinstaleld so no nvidia driver on hypervisor).
>
> Following various online examples  for setting up pass through I set
> the kernel boot line on the hypervisor to:
>
> # cat /proc/cmdline
> BOOT_IMAGE=/boot/vmlinuz-3.13.0-87-generic 
> root=UUID=d9bc9159-fedf-475b-b379-f65490c71860 ro console=tty0 
> console=ttyS1,115200 intel_iommu=on iommu=pt rd.modules-load=vfio-pci 
> nosplash nomodeset intel_iommu=on iommu=pt rd.modules-load=vfio-pci 
> nomdmonddf nomdmonisw
>
> Puzzled that I apparently have the device but it is apparently
> nonfunctional, where do I even look from here?
>
> -Jon
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [scientific-wg] Lustre war stories

[Blair Bethwaite]

Hi Álvaro, hi David -

NB: adding os-ops.

David, we have some real-time Lustre war stories we can share and
hopefully provide some positive conclusions to come Barcelona. I've
given an overview of what we're doing below. Are there any specifics
you were interested in when you raised Lustre in the meeting?

Our present approach leans on SRIOV and has worked with both
nova-network and now Neutron, and actually though this would work with
the Mellanox Neutron ML2 driver we are not using OpenStack Networking
to orchestrate this yet. It achieves the plumbing required to get a
parallel filesystem integrated into a typical virtualised OpenStack
deployment, but it does not "cloudify" the parallel filesystem in
anyway (for this you really need the filesystem to have some concept
of multi-tenancy and/or strong client isolation) and has quite limited
applicability to one or a small number of trusted users/projects.

Currently we have a single high-performance data network per cluster,
that is a high-bandwidth and RDMA capable Ethernet fabric. Our
Mellanox NICs (not sure if other vendors have similar features?) allow
us to restrict SRIOV virtual functions (VFs) to specific VLANs, so we
tie them to the data VLAN and then use PCI passthrough (care of
private Nova instance-types) to give guests a PCI VF plugged straight
into that network. Guests need to load appropriate drivers and
configure their own L3. Lustre servers are traditional bare-metal
affairs sitting at the bottom of that subnet.

We have one deployment like this which has been running Lustre over
TCP for about 12 months. That seems to work pretty well except that we
are in the midst of investigating high rx_errors on the servers and
discards on the switches, which seem like they might be
causing/related to Lustre write checksum errors that we see a lot of -
these don't seem to be fatal or data corrupting but rather Lustre
transport level errors which might cause write errors to propagate to
clients, but we're unsure... That particular problem does not seem to
be inherently related to our host configs or use of SRIOV though, more
likely a fabric config issue.

We have a second slightly larger deployment with a similar
configuration, the most notable difference for that one is that it is
using the o2iblnd (o2ib Lustre Network Driver), i.e., Lustre is
configured as for IB but is really running on RoCE. We plan to extract
some performance comparisons from this over coming weeks (there I
would like to compare with both TCP over SRIOV and TCP over
linux-bridge). Probably the main issue with this setup so far is the
need to build Lustre modules against both kernel and Mellanox OFED -
normally compute nodes like this stay very static, but now that they
are cloud instances there is a natural push towards more frequent
updating, and there is not a great deal of clarity about which
combinations are currently supported.

Cheers,

On 6 July 2016 at 21:31, Álvaro López García  wrote:
> On 06 Jul 2016 (11:58), Stig Telfer wrote:
>> Hi Dave -
>
> Hi all,
>
>> I’d like to introduce Wojciech and Matt from across town at the
>> University.  Wojciech and Matt work on managing and developing the
>> Lustre storage here at Cambridge University.  Right now we are just
>> getting started on integrating Lustre into OpenStack but Blair (also
>> copied) has a similar setup up and running already at Monash
>> University in Melbourne.
>>
>> Parallel filesystems in general are an activity area for the
>> Scientific Working Group, so we try to keep people in touch about what
>> works and what is possible.
>
> Yes, it would be awesome to get some other user stories about parallel
> filesystems and share our approaches, concerns and hacks.
>
>> I’m aware that Lustre is also used in this way in CSC in Finland and
>> (from today’s discussion) Álvaro has a similar configuration using
>> GPFS at his university in Spain.
>
> A bit of context. We are CSIC are operating two separate computing
> infrastructures, one HPC node part of the Spanish supercompting network,
> and one HTC cluster, plugged to the European Grid Infrastructure. Access
> policies for both systems are completely different, and they  are being
> used by a variety of disciplines (high energy physics, astrophysics,
> cosmology, engineering, bio, etc.). Both systems rely on GPFS: the HPC
> node leverages Infiniband for the storage network, whereas the HTC one
> uses 10GbE or 1GbE.
>
> In the IRC meeting I said that these filesystems were not shared, but I
> was wrong. All the filesystems are shared across both infrastructures,
> with the particularity that the HPC filesystems are only shared as read
> only outside the HPC node (they are shared over Ethernet)
>
> The interesting part is that we are running a complete SGE cluster (HTC)
> on top of OpenStack, with access to GPFS. The HTC cluster is subject to
> periodic updates due to middleware upgrades, therefore we were running a
> completely virtualized 

Re: [Openstack-operators] [User-committee] [scientific-wg] possible time change for non-US fortnightly meeting

[Blair Bethwaite]

Thanks for your replies all.

Four yeas and no nays is good enough for me! Meeting infra time
change: https://review.openstack.org/336914

So, next Scientific WG meeting will be this Wednesday @ 0900 UTC in
openstack-meeting.

Cheers,

On 30 June 2016 at 18:55, Peter Love <p.l...@lancaster.ac.uk> wrote:
> Much better!  (say all the .uk people)
>
> On 30 June 2016 at 09:48, Stig Telfer <stig.openst...@telfer.org> wrote:
>> Sounds good to me too!
>>
>> Best wishes,
>> Stig
>>
>>
>>> On 29 Jun 2016, at 07:49, Dario Vianello <da...@ebi.ac.uk> wrote:
>>>
>>> Same here :-)
>>>
>>> Thanks!
>>>
>>>
>>> Dario Vianello
>>>
>>> Cloud Bioinformatics Application Architect
>>> European Bioinformatics Institute (EMBL-EBI)
>>> Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, UK
>>> Email: da...@ebi.ac.uk
>>>
>>>> On 28 Jun 2016, at 13:42, <alexander.di...@stfc.ac.uk> 
>>>> <alexander.di...@stfc.ac.uk> wrote:
>>>>
>>>> 0900 would work better for me J
>>>>
>>>> Thanks
>>>>
>>>> Alexander
>>>>
>>>> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
>>>> Sent: 28 June 2016 12:37
>>>> To: user-committee <user-commit...@lists.openstack.org>; openstack-oper. 
>>>> <openstack-operators@lists.openstack.org>
>>>> Subject: [User-committee] [scientific-wg] possible time change for non-US 
>>>> fortnightly meeting
>>>>
>>>> Hi all,
>>>>
>>>> Currently the scientific-wg is meeting weekly on irc with alternating 
>>>> times week to week - 2100 UTC Tues this week and 0700 UTC Weds next week. 
>>>> The basic idea being to have both US and non-US friendly times.
>>>>
>>>> The former time is pretty well attended but the latter is somewhat hit and 
>>>> miss so we're considering whether it should be adjusted. Would it help you 
>>>> attend if we pushed the 0700 UTC to 0900 or later?
>>>>
>>>> Cheers,
>>>> Blairo & Stig
>>>>
>>>> ___
>>>> User-committee mailing list
>>>> user-commit...@lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>>>
>>> ___
>>> User-committee mailing list
>>> user-commit...@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>>
>>
>> ___
>> User-committee mailing list
>> user-commit...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee
>
> ___
> User-committee mailing list
> user-commit...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Data-Migration Juno -> Mitaka

[Blair Bethwaite]

Hi Roland -

GUTS looks cool! But I took Michael's question to be more about
control plane data than end-user instances etc...?

Michael - If that's the case then you probably want to start with
dumping your present Juno DBs, importing into your Mitaka test DB and
then attempting the migrations to get to Mitaka, if they work then you
might be able to bring up a "clone cloud" (of course there is probably
a whole lot of network specific config in there that won't work unless
you are doing this in a separate/isolated name-and-address space,
there's also all the config files...). Also, as others have noted on
this list recently, live upgrades are only supported/tested(?) between
successive versions.

Cheers,

On 29 June 2016 at 09:54, Roland Chan  wrote:
> Hi Michael
>
> We built a tool called GUTS to migrate various assets between OpenStack
> deployment (and other things as well). You can check it out at
> https://github.com/aptira/guts. It can migrate Instances, Volumes, Networks,
> Tenants, Users and Security Groups from one OpenStack to another.
>
> It's a work in progress, but we're always happy to accept input.
>
> Hope this helps, feel free to contact me if you need anything.
>
> Roland
>
>
>
> On 28 June 2016 at 16:07, Michael Stang 
> wrote:
>>
>> Hello all,
>>
>>
>>
>> we setup a small test environment of Mitaka to learn about the
>> installation and the new features. Before we try the Upgrade of out Juno
>> production environment we want to migrate all it’s data to the Mitaka
>> installation as a backup and also to make tests.
>>
>>
>>
>> Is there an easy way to migrate the data from the Juno environment to the
>> mitaka environment or has this to be done manually piece by piece? I found
>> already a tool named CloudFerry but the instructions to use it are not much
>> and also there seems to be no support for mitaka by now, is there any other
>> software/tool to help for migrating data?
>>
>>
>>
>> Thanks and kind regards,
>>
>> Michael
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



-- 
Cheers,
~Blairo

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators