Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Matt, i'll take a stab at it thanks, dims On Tue, Dec 16, 2014 at 5:18 PM, Matt Riedemann wrote: > > > On 12/30/2013 7:30 PM, Clint Byrum wrote: >> >> Excerpts from Day, Phil's message of 2013-12-30 11:05:17 -0800: >>> >>> Hi, so it seems we were saying the same thing - new vms get a shared >>> "blank" (empty) file system, not blank disc. How big a problem it is that >>> in many cases this will be the already created ext3 disk and not ext4 >>> depends I guess on how important consistency is to you (to me its pretty >>> important). Either way the change as it stands wont give all new vms an >>> ext4 fs as intended, so its flawed in that regard. >>> >>> Like you I was thinking that we may have to move away from "default" >>> being in the file name to fix this. >>> >> >> Indeed, "default"'s meaning is mutable and thus it is flawed as a >> cache key. >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > jogo brought this bug up in IRC today [1]. The bug report says that we > should put in the Icehouse release notes that ext3 is going to be changed to > ext4 in Juno but that never happened. So question is, now that we're well > into Kilo, what can be done about this now? The thread here talks about > doing more than just changing the default value like in the original change > [2], but is someone willing to work on that? > > [1] https://bugs.launchpad.net/nova/+bug/1266262 > [2] https://review.openstack.org/#/c/63209/ > > -- > > Thanks, > > Matt Riedemann > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
On 12/30/2013 7:30 PM, Clint Byrum wrote: Excerpts from Day, Phil's message of 2013-12-30 11:05:17 -0800: Hi, so it seems we were saying the same thing - new vms get a shared "blank" (empty) file system, not blank disc. How big a problem it is that in many cases this will be the already created ext3 disk and not ext4 depends I guess on how important consistency is to you (to me its pretty important). Either way the change as it stands wont give all new vms an ext4 fs as intended, so its flawed in that regard. Like you I was thinking that we may have to move away from "default" being in the file name to fix this. Indeed, "default"'s meaning is mutable and thus it is flawed as a cache key. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev jogo brought this bug up in IRC today [1]. The bug report says that we should put in the Icehouse release notes that ext3 is going to be changed to ext4 in Juno but that never happened. So question is, now that we're well into Kilo, what can be done about this now? The thread here talks about doing more than just changing the default value like in the original change [2], but is someone willing to work on that? [1] https://bugs.launchpad.net/nova/+bug/1266262 [2] https://review.openstack.org/#/c/63209/ -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
As a follow-up, I've filed this bug: https://bugs.launchpad.net/nova/+bug/1266262 We'll be working around this in TripleO by setting default_ephemeral_format for our baremetal clouds, but I feel we should document in Icehouse's release notes that the default will be changed to ext4 in Juno, so that deployers can set it to ext3 in their Icehouse deployments. The question I'd have is should we print a warning if users leave it unset in their configs in Icehouse so they know what to expect in Juno, or is the documentation enough? Excerpts from Day, Phil's message of 2013-12-29 00:12:50 -0800: > Hi Folks, > > As highlighted in the thread "minimal review period for functional changes" > I'd like to propose that change is https://review.openstack.org/#/c/63209/ is > reverted because: > > > - It causes inconsistent behaviour in the system, as any existing > "default" backing files will have ext3 in them, so a VM will now get either > ext3 or 3ext4 depending on whether the host they get created on already has a > backing file of the required size or not. I don't think the existing design > ever considered the default FS changing - maybe we shouldn't have files that > include "default" as the file system type if it can change over time, and the > name should always reflect the FS type. > > > > - It introduces a new a new requirement for GuestOS's to have to > support ext4 in order to work with the default configuration. I think > that's a significant enough change that it needs to be flagged, discussed, > and planned. > > > > > > I'm about to go off line for a few days and won't have anything other than > patchy e-mail access, otherwise I'd submit the change myself ;-) > > > Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Change to revert the default to ext3: https://review.openstack.org/#/c/64666/ > -Original Message- > From: Clint Byrum [mailto:cl...@fewbar.com] > Sent: 31 December 2013 01:31 > To: openstack-dev > Subject: Re: [openstack-dev] [nova] - Revert change of default ephemeral fs > to ext4 > > Excerpts from Day, Phil's message of 2013-12-30 11:05:17 -0800: > > Hi, so it seems we were saying the same thing - new vms get a shared > "blank" (empty) file system, not blank disc. How big a problem it is that in > many cases this will be the already created ext3 disk and not ext4 depends I > guess on how important consistency is to you (to me its pretty important). > Either way the change as it stands wont give all new vms an ext4 fs as > intended, so its flawed in that regard. > > > > Like you I was thinking that we may have to move away from "default" > being in the file name to fix this. > > > > Indeed, "default"'s meaning is mutable and thus it is flawed as a cache key. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Excerpts from Day, Phil's message of 2013-12-30 11:05:17 -0800: > Hi, so it seems we were saying the same thing - new vms get a shared "blank" > (empty) file system, not blank disc. How big a problem it is that in many > cases this will be the already created ext3 disk and not ext4 depends I guess > on how important consistency is to you (to me its pretty important). Either > way the change as it stands wont give all new vms an ext4 fs as intended, so > its flawed in that regard. > > Like you I was thinking that we may have to move away from "default" being in > the file name to fix this. > Indeed, "default"'s meaning is mutable and thus it is flawed as a cache key. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Hi, so it seems we were saying the same thing - new vms get a shared "blank" (empty) file system, not blank disc. How big a problem it is that in many cases this will be the already created ext3 disk and not ext4 depends I guess on how important consistency is to you (to me its pretty important). Either way the change as it stands wont give all new vms an ext4 fs as intended, so its flawed in that regard. Like you I was thinking that we may have to move away from "default" being in the file name to fix this. I don't think the cache clean up code ever removes the ephemeral backing files though at the moment. Phil Sent from Samsung Mobile Original message From: Pádraig Brady Date: To: "OpenStack Development Mailing List (not for usage questions)" ,"Day, Phil" Subject: Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4 On 12/30/2013 12:39 AM, Pádraig Brady wrote: > On 12/29/2013 08:12 AM, Day, Phil wrote: >> Hi Folks, >> >> As highlighted in the thread “minimal review period for functional changes” >> I’d like to propose that change is https://review.openstack.org/#/c/63209/ >> is reverted because: >> >> - It causes inconsistent behaviour in the system, as any existing >> "default" backing files will have ext3 in them, so a VM will now get either >> ext3 or 3ext4 depending on whether the host they get created on already has >> a backing file of the required size or not. I don't think the existing >> design ever considered the default FS changing - maybe we shouldn’t have >> files that include "default" as the file system type if it can change over >> time, and the name should always reflect the FS type. > > I'm not sure this is a practical issue since ephemeral storage is built up > from blank by each instance Phil> Maybe it varies by hypervisor; my understanding is that at least in libvirt the ephemeral disks are cow layers on a shared common backing file that has the file system in it. Phil> The naming convention includes the size and fs type or "default" and is only created once per size-fs combination. Phil> So this is a real issue - I think that maybe the eventual change to ext4 need to be combined withoving away from "default" in the file name. Right, what I meant by each instance building ephemeral up from blank each time, is that each instance will go from either a blank ext3 or blank ext4 each time, so if they support ext4 then there should be no practical issue. Now agreed there is a consistency issue, which could have performance consistency issues for example, so it's not ideal. To be complete, for the libvirt case we don't even use these persistent backing files for ephemeral disks if use_cow_images=False or with LVM backing. To be most general and avoid this consistency issue, I suppose we could change the name format for these cached CoW base images from ephemeral_10_default, ephemeral_20_linux etc. to 'ephemeral_20_' + md5(mkfs_command)[:7] That would impose a performance hit at first boot with this new logic, and we'd have to double check that the cache cleaning logic would handle removing unused older format images. Alternatively we might just document this issue and put the onus on users to clean these cached ephemeral disk on upgrade (the commit is already tagged as DocImpact). thanks, Pádraig. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
There is a big difference between DocImpact (i.e. needing some improvements to the doc) and putting the migration effort onto the users without automation to help... as I have said previously, while this (and others like it) may be a worthwhile change in isolation, making the upgrade process more difficult needs to be balanced with the technical benefits of the change. These questions need to be part of the review approval. Tim > -Original Message- > From: Pádraig Brady [mailto:p...@draigbrady.com] > Sent: 30 December 2013 14:00 > To: OpenStack Development Mailing List (not for usage questions); Day, Phil > Subject: Re: [openstack-dev] [nova] - Revert change of default ephemeral fs > to ext4 > ... > To be most general and avoid this consistency issue, I suppose we could > change the name format for these cached CoW base images from > ephemeral_10_default, ephemeral_20_linux etc. > to 'ephemeral_20_' + md5(mkfs_command)[:7] That would impose a performance > hit at first boot with this new logic, and we'd have to > double check that the cache cleaning logic would handle removing unused older > format images. > Alternatively we might just document this issue and put the onus on users to > clean these cached ephemeral disk on upgrade (the commit > is already tagged as DocImpact). > > thanks, > Pádraig. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
On 12/30/2013 12:39 AM, Pádraig Brady wrote: > On 12/29/2013 08:12 AM, Day, Phil wrote: >> Hi Folks, >> >> As highlighted in the thread “minimal review period for functional changes” >> I’d like to propose that change is https://review.openstack.org/#/c/63209/ >> is reverted because: >> >> - It causes inconsistent behaviour in the system, as any existing >> "default" backing files will have ext3 in them, so a VM will now get either >> ext3 or 3ext4 depending on whether the host they get created on already has >> a backing file of the required size or not. I don't think the existing >> design ever considered the default FS changing - maybe we shouldn’t have >> files that include "default" as the file system type if it can change over >> time, and the name should always reflect the FS type. > > I'm not sure this is a practical issue since ephemeral storage is built up > from blank by each instance Phil> Maybe it varies by hypervisor; my understanding is that at least in libvirt the ephemeral disks are cow layers on a shared common backing file that has the file system in it. Phil> The naming convention includes the size and fs type or "default" and is only created once per size-fs combination. Phil> So this is a real issue - I think that maybe the eventual change to ext4 need to be combined withoving away from "default" in the file name. Right, what I meant by each instance building ephemeral up from blank each time, is that each instance will go from either a blank ext3 or blank ext4 each time, so if they support ext4 then there should be no practical issue. Now agreed there is a consistency issue, which could have performance consistency issues for example, so it's not ideal. To be complete, for the libvirt case we don't even use these persistent backing files for ephemeral disks if use_cow_images=False or with LVM backing. To be most general and avoid this consistency issue, I suppose we could change the name format for these cached CoW base images from ephemeral_10_default, ephemeral_20_linux etc. to 'ephemeral_20_' + md5(mkfs_command)[:7] That would impose a performance hit at first boot with this new logic, and we'd have to double check that the cache cleaning logic would handle removing unused older format images. Alternatively we might just document this issue and put the onus on users to clean these cached ephemeral disk on upgrade (the commit is already tagged as DocImpact). thanks, Pádraig. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Sent from Samsung Mobile Original message From: Pádraig Brady Date: To: "OpenStack Development Mailing List (not for usage questions)" Cc: "Day, Phil" Subject: Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4 > - It causes inconsistent behaviour in the system, as any existing > "default" backing files will have ext3 in them, so a VM will now get either > ext3 or 3ext4 depending on whether the host they get created on already has a > backing file of the required size or not. I don't think the existing design > ever considered the default FS changing - maybe we shouldn’t have files that > include "default" as the file system type if it can change over time, and the > name should always reflect the FS type. > I'm not sure this is a practical issue since ephemeral storage is built up > from blank by each instance Maybe it varies by hypervisor; my understanding is that at least in libvirt the ephemeral disks are cow layers on a shared common backing file that has the file system in it. The naming convention includes the size and fs type or "default" and is only created once per size-fs combination. So this is a real issue - I think that maybe the eventual change to ext4 need to be combined withoving away from "default" in the file name. Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
On 12/29/2013 08:12 AM, Day, Phil wrote: > Hi Folks, > > > > As highlighted in the thread “minimal review period for functional changes” > I’d like to propose that change is https://review.openstack.org/#/c/63209/ is > reverted because: > > > > - It causes inconsistent behaviour in the system, as any existing > "default" backing files will have ext3 in them, so a VM will now get either > ext3 or 3ext4 depending on whether the host they get created on already has a > backing file of the required size or not. I don't think the existing design > ever considered the default FS changing - maybe we shouldn’t have files that > include "default" as the file system type if it can change over time, and the > name should always reflect the FS type. I'm not sure this is a practical issue since ephemeral storage is built up from blank by each instance > - It introduces a new a new requirement for GuestOS's to have to > support ext4 in order to work with the default configuration. I think > that's a significant enough change that it needs to be flagged, discussed, > and planned. This is a genuine concern, though given how long ext4 has been available, ext4 does seem to be the right default. I.E. the significant performance benefits of ext4 creation over ext3, the onus should be on systems requiring support for ext3 only guests, to config appropriately through the existing config vars. This info would be appropriate for Icehouse release notes. In any case I agree that this should have been discussed more, and it would be useful for people to mention on this thread, any guests that would have issues with ext4 support. Note I'm assuming that mke2fs creates an ext4 file system compat with the earliest ext4 implementations, though have not confirmed that? BTW, if you disable ext4 options so that the FS is mountable by ext3, you essentially are creating an ext3 file system and so lose the benefits. $ truncate -s 1T t.fs $ time mke2fs -F -t ext3 t.fs real1m6.223s $ du -hs t.fs 17G t.fs $ time mke2fs -F -t ext4 -O ^extent,^flex_bg,^huge_file,^uninit_bg,^dir_nlink,^extra_isize t.fs real1m16.809s $ du -hs t.fs 17G t.fs $ time mke2fs -F -t ext4 t.fs real0m1.925s $ du -hs t.fs 139M.fs ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] - Revert change of default ephemeral fs to ext4
Hi Folks, As highlighted in the thread "minimal review period for functional changes" I'd like to propose that change is https://review.openstack.org/#/c/63209/ is reverted because: - It causes inconsistent behaviour in the system, as any existing "default" backing files will have ext3 in them, so a VM will now get either ext3 or 3ext4 depending on whether the host they get created on already has a backing file of the required size or not. I don't think the existing design ever considered the default FS changing - maybe we shouldn't have files that include "default" as the file system type if it can change over time, and the name should always reflect the FS type. - It introduces a new a new requirement for GuestOS's to have to support ext4 in order to work with the default configuration. I think that's a significant enough change that it needs to be flagged, discussed, and planned. I'm about to go off line for a few days and won't have anything other than patchy e-mail access, otherwise I'd submit the change myself ;-) Phil ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
