Re: [Openstack-operators] large high-performance ephemeral storage
On 6/13/2018 10:54 AM, Chris Friesen wrote: Also, migration and resize are not supported for LVM-backed instances. I proposed a patch to support them (https://review.openstack.org/#/c/337334/) but hit issues and never got around to fixing them up. Yup, I guess I should have read the entire thread first. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
On 6/13/2018 8:58 AM, Blair Bethwaite wrote: Though we have not used LVM based instance storage before, are there any significant gotchas? I know you can't resize/cold migrate lvm-backed ephemeral root disk instances: https://github.com/openstack/nova/blob/343c2bee234568855fd9e6ba075a05c2e70f3388/nova/virt/libvirt/driver.py#L8136 However, StarlingX has a patch for that (pretty sure anyway, I know WindRiver had one): https://review.openstack.org/#/c/337334/ -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
On 06/13/2018 07:58 AM, Blair Bethwaite wrote: Is the collective wisdom to use LVM based instances for these use-cases? Putting a host filesystem with qcow2 based disk images on it can't help performance-wise... Though we have not used LVM based instance storage before, are there any significant gotchas? And furthermore, is it possible to use set IO QoS limits on these? LVM has the drawback that deleting instances results in significant disk traffic while the volume is scrubbed with zeros. If you don't care about security you can set a config option to turn this off. Also, while this is happening I think your disk resource tracking will be wrong because nova assumes the space is available. (At least it used to be this way, I haven't checked that code recently.) Also, migration and resize are not supported for LVM-backed instances. I proposed a patch to support them (https://review.openstack.org/#/c/337334/) but hit issues and never got around to fixing them up. Chris ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
fio is fine with me. I'll lazily defer to your expertise on the right fio commands to run for each case. :) If we're going to test within the guest, that's going to introduce a new set of variables, right? Should we settle on a standard flavor (maybe two if we wanted to include both virtio and virtio-scsi) or should the results make note of what local configuration was used? On Wed, Jun 13, 2018 at 8:45 AM, Blair Bethwaite wrote: > Hey Joe, > > Thanks! So shall we settle on fio as a standard IO micro benchmarking > tool? Seems to me the minimum we want is throughput and IOPs oriented tests > for both the guest OS workload profile and the some sort of large working > set application workload. For the latter it is probably best to ignore > multiple files and focus solely on queue depth for parallelism, some sort > of mixed block size profile/s, and some sort of r/w mix (where write <=50% > to acknowledge this is ephemeral storage so hopefully something is using it > soon after storing). Thoughts? > > Cheers, > Blair > > On Thu., 14 Jun. 2018, 00:24 Joe Topjian, wrote: > >> Yes, you can! The kernel documentation for read/write limits actually >> uses /dev/null in the examples :) >> >> But more seriously: while we have not architected specifically for high >> performance, for the past few years, we have used a zpool of cheap spindle >> disks and 1-2 SSD disks for caching. We have ZFS configured for >> deduplication which helps for the base images but not so much for ephemeral. >> >> If you have a standard benchmark command in mind to run, I'd be happy to >> post the results. Maybe others could do the same to create some type of >> matrix? >> >> On Wed, Jun 13, 2018 at 8:18 AM, Blair Bethwaite < >> blair.bethwa...@gmail.com> wrote: >> >>> Hi Jay, >>> >>> Ha, I'm sure there's some wisdom hidden behind the trolling here? >>> >>> Believe me, I have tried to push these sorts of use-cases toward volume >>> or share storage, but in the research/science domain there is often more >>> accessible funding available to throw at infrastructure stop-gaps than >>> software engineering (parallelism is hard). PS: when I say ephemeral I >>> don't necessarily mean they aren't doing backups and otherwise caring that >>> they have 100+TB of data on a stand alone host. >>> >>> PS: I imagine you can set QoS limits on /dev/null these days via CPU >>> cgroups... >>> >>> Cheers, >>> >>> >>> On Thu., 14 Jun. 2018, 00:03 Jay Pipes, wrote: >>> On 06/13/2018 09:58 AM, Blair Bethwaite wrote: > Hi all, > > Wondering if anyone can share experience with architecting Nova KVM > boxes for large capacity high-performance storage? We have some > particular use-cases that want both high-IOPs and large capacity local > storage. > > In the past we have used bcache with an SSD based RAID0 write-through > caching for a hardware (PERC) backed RAID volume. This seemed to work > ok, but we never really gave it a hard time. I guess if we followed a > similar pattern today we would use lvmcache (or are people still using > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with > write cache. > > Is the collective wisdom to use LVM based instances for these use-cases? > Putting a host filesystem with qcow2 based disk images on it can't help > performance-wise... Though we have not used LVM based instance storage > before, are there any significant gotchas? And furthermore, is it > possible to use set IO QoS limits on these? I've found /dev/null to be the fastest ephemeral storage system, bar none. Not sure if you can set QoS limits on it though. Best, -jay ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >>> ___ >>> OpenStack-operators mailing list >>> OpenStack-operators@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >>> >> ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
Hey Joe, Thanks! So shall we settle on fio as a standard IO micro benchmarking tool? Seems to me the minimum we want is throughput and IOPs oriented tests for both the guest OS workload profile and the some sort of large working set application workload. For the latter it is probably best to ignore multiple files and focus solely on queue depth for parallelism, some sort of mixed block size profile/s, and some sort of r/w mix (where write <=50% to acknowledge this is ephemeral storage so hopefully something is using it soon after storing). Thoughts? Cheers, Blair On Thu., 14 Jun. 2018, 00:24 Joe Topjian, wrote: > Yes, you can! The kernel documentation for read/write limits actually uses > /dev/null in the examples :) > > But more seriously: while we have not architected specifically for high > performance, for the past few years, we have used a zpool of cheap spindle > disks and 1-2 SSD disks for caching. We have ZFS configured for > deduplication which helps for the base images but not so much for ephemeral. > > If you have a standard benchmark command in mind to run, I'd be happy to > post the results. Maybe others could do the same to create some type of > matrix? > > On Wed, Jun 13, 2018 at 8:18 AM, Blair Bethwaite < > blair.bethwa...@gmail.com> wrote: > >> Hi Jay, >> >> Ha, I'm sure there's some wisdom hidden behind the trolling here? >> >> Believe me, I have tried to push these sorts of use-cases toward volume >> or share storage, but in the research/science domain there is often more >> accessible funding available to throw at infrastructure stop-gaps than >> software engineering (parallelism is hard). PS: when I say ephemeral I >> don't necessarily mean they aren't doing backups and otherwise caring that >> they have 100+TB of data on a stand alone host. >> >> PS: I imagine you can set QoS limits on /dev/null these days via CPU >> cgroups... >> >> Cheers, >> >> >> On Thu., 14 Jun. 2018, 00:03 Jay Pipes, wrote: >> >>> On 06/13/2018 09:58 AM, Blair Bethwaite wrote: >>> > Hi all, >>> > >>> > Wondering if anyone can share experience with architecting Nova KVM >>> > boxes for large capacity high-performance storage? We have some >>> > particular use-cases that want both high-IOPs and large capacity local >>> > storage. >>> > >>> > In the past we have used bcache with an SSD based RAID0 write-through >>> > caching for a hardware (PERC) backed RAID volume. This seemed to work >>> > ok, but we never really gave it a hard time. I guess if we followed a >>> > similar pattern today we would use lvmcache (or are people still using >>> > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with >>> > write cache. >>> > >>> > Is the collective wisdom to use LVM based instances for these >>> use-cases? >>> > Putting a host filesystem with qcow2 based disk images on it can't >>> help >>> > performance-wise... Though we have not used LVM based instance storage >>> > before, are there any significant gotchas? And furthermore, is it >>> > possible to use set IO QoS limits on these? >>> >>> I've found /dev/null to be the fastest ephemeral storage system, bar >>> none. >>> >>> Not sure if you can set QoS limits on it though. >>> >>> Best, >>> -jay >>> >>> ___ >>> OpenStack-operators mailing list >>> OpenStack-operators@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>> >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> >> > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
Lol! Ok, forgive me, I wasn't sure if I had regular or existential Jay on the line :-). On Thu., 14 Jun. 2018, 00:24 Jay Pipes, wrote: > On 06/13/2018 10:18 AM, Blair Bethwaite wrote: > > Hi Jay, > > > > Ha, I'm sure there's some wisdom hidden behind the trolling here? > > I wasn't trolling at all. I was trying to be funny. Attempt failed I > guess :) > > Best, > -jay > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
Yes, you can! The kernel documentation for read/write limits actually uses /dev/null in the examples :) But more seriously: while we have not architected specifically for high performance, for the past few years, we have used a zpool of cheap spindle disks and 1-2 SSD disks for caching. We have ZFS configured for deduplication which helps for the base images but not so much for ephemeral. If you have a standard benchmark command in mind to run, I'd be happy to post the results. Maybe others could do the same to create some type of matrix? On Wed, Jun 13, 2018 at 8:18 AM, Blair Bethwaite wrote: > Hi Jay, > > Ha, I'm sure there's some wisdom hidden behind the trolling here? > > Believe me, I have tried to push these sorts of use-cases toward volume or > share storage, but in the research/science domain there is often more > accessible funding available to throw at infrastructure stop-gaps than > software engineering (parallelism is hard). PS: when I say ephemeral I > don't necessarily mean they aren't doing backups and otherwise caring that > they have 100+TB of data on a stand alone host. > > PS: I imagine you can set QoS limits on /dev/null these days via CPU > cgroups... > > Cheers, > > > On Thu., 14 Jun. 2018, 00:03 Jay Pipes, wrote: > >> On 06/13/2018 09:58 AM, Blair Bethwaite wrote: >> > Hi all, >> > >> > Wondering if anyone can share experience with architecting Nova KVM >> > boxes for large capacity high-performance storage? We have some >> > particular use-cases that want both high-IOPs and large capacity local >> > storage. >> > >> > In the past we have used bcache with an SSD based RAID0 write-through >> > caching for a hardware (PERC) backed RAID volume. This seemed to work >> > ok, but we never really gave it a hard time. I guess if we followed a >> > similar pattern today we would use lvmcache (or are people still using >> > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with >> > write cache. >> > >> > Is the collective wisdom to use LVM based instances for these >> use-cases? >> > Putting a host filesystem with qcow2 based disk images on it can't help >> > performance-wise... Though we have not used LVM based instance storage >> > before, are there any significant gotchas? And furthermore, is it >> > possible to use set IO QoS limits on these? >> >> I've found /dev/null to be the fastest ephemeral storage system, bar none. >> >> Not sure if you can set QoS limits on it though. >> >> Best, >> -jay >> >> ___ >> OpenStack-operators mailing list >> OpenStack-operators@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
On 06/13/2018 10:18 AM, Blair Bethwaite wrote: Hi Jay, Ha, I'm sure there's some wisdom hidden behind the trolling here? I wasn't trolling at all. I was trying to be funny. Attempt failed I guess :) Best, -jay ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
Hi Jay, Ha, I'm sure there's some wisdom hidden behind the trolling here? Believe me, I have tried to push these sorts of use-cases toward volume or share storage, but in the research/science domain there is often more accessible funding available to throw at infrastructure stop-gaps than software engineering (parallelism is hard). PS: when I say ephemeral I don't necessarily mean they aren't doing backups and otherwise caring that they have 100+TB of data on a stand alone host. PS: I imagine you can set QoS limits on /dev/null these days via CPU cgroups... Cheers, On Thu., 14 Jun. 2018, 00:03 Jay Pipes, wrote: > On 06/13/2018 09:58 AM, Blair Bethwaite wrote: > > Hi all, > > > > Wondering if anyone can share experience with architecting Nova KVM > > boxes for large capacity high-performance storage? We have some > > particular use-cases that want both high-IOPs and large capacity local > > storage. > > > > In the past we have used bcache with an SSD based RAID0 write-through > > caching for a hardware (PERC) backed RAID volume. This seemed to work > > ok, but we never really gave it a hard time. I guess if we followed a > > similar pattern today we would use lvmcache (or are people still using > > bcache with confidence?) with a few TB of NVMe and a NL-SAS array with > > write cache. > > > > Is the collective wisdom to use LVM based instances for these use-cases? > > Putting a host filesystem with qcow2 based disk images on it can't help > > performance-wise... Though we have not used LVM based instance storage > > before, are there any significant gotchas? And furthermore, is it > > possible to use set IO QoS limits on these? > > I've found /dev/null to be the fastest ephemeral storage system, bar none. > > Not sure if you can set QoS limits on it though. > > Best, > -jay > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] large high-performance ephemeral storage
On 06/13/2018 09:58 AM, Blair Bethwaite wrote: Hi all, Wondering if anyone can share experience with architecting Nova KVM boxes for large capacity high-performance storage? We have some particular use-cases that want both high-IOPs and large capacity local storage. In the past we have used bcache with an SSD based RAID0 write-through caching for a hardware (PERC) backed RAID volume. This seemed to work ok, but we never really gave it a hard time. I guess if we followed a similar pattern today we would use lvmcache (or are people still using bcache with confidence?) with a few TB of NVMe and a NL-SAS array with write cache. Is the collective wisdom to use LVM based instances for these use-cases? Putting a host filesystem with qcow2 based disk images on it can't help performance-wise... Though we have not used LVM based instance storage before, are there any significant gotchas? And furthermore, is it possible to use set IO QoS limits on these? I've found /dev/null to be the fastest ephemeral storage system, bar none. Not sure if you can set QoS limits on it though. Best, -jay ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
[Openstack-operators] large high-performance ephemeral storage
Hi all, Wondering if anyone can share experience with architecting Nova KVM boxes for large capacity high-performance storage? We have some particular use-cases that want both high-IOPs and large capacity local storage. In the past we have used bcache with an SSD based RAID0 write-through caching for a hardware (PERC) backed RAID volume. This seemed to work ok, but we never really gave it a hard time. I guess if we followed a similar pattern today we would use lvmcache (or are people still using bcache with confidence?) with a few TB of NVMe and a NL-SAS array with write cache. Is the collective wisdom to use LVM based instances for these use-cases? Putting a host filesystem with qcow2 based disk images on it can't help performance-wise... Though we have not used LVM based instance storage before, are there any significant gotchas? And furthermore, is it possible to use set IO QoS limits on these? -- Cheers, ~Blairo ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
