Excerpts from Dustin Kirkland's message of 2016-01-19 20:27:51 -0800: > On Sat, Jan 16, 2016 at 7:49 PM, Clint Byrum <cl...@ubuntu.com> wrote: > > Excerpts from Dustin Kirkland's message of 2016-01-16 04:25:58 -0800: > >> On Fri, Jan 15, 2016 at 2:25 AM, Seth Arnold <seth.arn...@canonical.com> > >> wrote: > >> > On Thu, Jan 14, 2016 at 12:27:58PM +0200, Dustin Kirkland wrote: > >> >> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles > >> >> are created/deleted as needed. If your root disk is lvm-encrypted, > >> >> then obviously such swap files are encrypted, too. > >> > > >> > I've been severely skeptical of the swapspace package: > >> > > >> > - Swap is used when the system is already under pressure; a few hundred > >> > megs is great and probably for the best but if the system is actively > >> > pushing beyond that then it's being pushed too hard. > >> > > >> > - If the swap space is going to be allocated on the fly, that means the > >> > disk blocks have to zeroed on the fly, when the system is under > >> > pressure, rather than at some leisurely time beforehand. > >> > > >> > - If the swap space is allocated on a filesystem, it's probably being > >> > allocated from a fragmented filesystem that's 90% full rather than a > >> > nice contiguous block of space as it would with a swap partition. > >> > > >> > - Accessing further into a file may involve loading multiple indirect > >> > blocks from disk into unswappable kernel memory. A swap partition does > >> > not require indirection blocks. > >> > > >> > - If the swap space allocated from a filesystem pushes the filesystem to > >> > 95% full (or whatever is left after accounting for reserved blocks), > >> > programs will error and almost nothing handles "disk full" errors > >> > gracefully. Swap partitions do not cause surprise gigabyte losses in > >> > free space. > >> > > >> > - Swap files can't be allocated from btrfs filesystems and probably > >> > shouldn't be allocated from zfs filesystems either. (Swap on zvols, > >> > maybe.) > >> > > >> > Perhaps the swapspace package uses some tasteful tunables to mitigate > >> > against my concerns but the end result is that it contributes extra load, > >> > extra IO pressure, and extra uncertainty at a time when the system is > >> > already experiencing too much load, too much IO pressure, and too much > >> > uncertainty. > >> > > >> > The risks and downsides of swapspace feel like a lot compared to the > >> > slight hassle of having the installer make a swap partition. > >> > >> I count 4 "if's", 3 "probably's", 2 "should/would's", and 1 "maybe" in > >> that reply :-) > >> > >> Perhaps try it out? > >> > >> I've been running it and /tmp on tmpfs for several years (since before > >> ~precise) on my desktop on an encrypted LVM partition. My machine has > >> a lot of memory (16GB), though I do push it hard), and have never > >> noticed a swapspace-related problem. I've also used this combination > >> on hundreds of servers, and several production systems. > >> > > > > The 'if' and 'probably' are missing in your anecdotal evidence though. > > If you use the servers the way you have, it will probably work fine. > > Also we're talking about cloud instances, not "servers", which have > > quite different use and performance profiles. > > > > I'd like to see even some rudimentary experiments done with realistic > > workloads before saying this is a better idea than leaving things as > > they are. We've all speculated and provided anecdotal evidence enough to > > warrant such an investigation for those who speculate it will be a > > worthwhile change. > > Sure, done! You can find a detailed statistical analysis, as well as > the raw data for your download and treatment at: > > http://blog.dustinkirkland.com/2016/01/data-driven-analysis-tmp-on-tmpfs.html > > Based on a statistical analysis of 502 physical and virtual servers > running production and test services at Canonical (including > databases, websites, OpenStack, ubuntu.com, launchpad.net, et al.), > 96.6% of them could fit all of the data they currently have in /tmp, > entirely in half of the free memory available in the system. That > ratio goes up to 99.2% of the systems surveyed (i.e., all but 4) when > one takes into account both free available memory and available swap. > The remaining 4 systems are are currently using [101 GB, 42 GB, 13 GB, > and 10 GB] of swap, respectively, and are themselves somewhat special > cases. >
This is a very cool window into actual running servers, and I appreciate the effort that has gone into this. This shows that most of the server workloads will work fine with /tmp on tmpfs. I actually would love to have a program which ran through a set of disk and memory metrics from something like influxdb and basically told me "these should all be using tmpfs you dummy!". However, I do want to point out that I find the sample size very small. A snapshot of a single point in time from every server means that we have no qualification given for what time this was, or what the box was doing at the time. While the standing usage on healthy systems at a given point in time is a good initial indicator that this experiment should continue, I'm not sure I would conclude on this basis alone that it means a change would provide more benefits than challenges for users. What it means to me is "it's worth digging just a little deeper". So basically, if I thought the benefits were worth the work, I'd want a similar collection to be done hourly over a prolonged period that includes peak usage times for whatever these servers do. I'd probably also try to simulate common workloads and operations, maybe run all of the autopkgtests on typical cloud VMs with and without the change to see what effects could be observed. But that's all nice-to-haves. The real reason I'm still skeptical is simply that I don't see /tmp usage being a major factor on cloud instances. But, don't take my skepticism as a rejection. I just don't see it as a priority. I've got zero data that would refute any of this. > Moreover, Ubuntu is hardly the first Linux/UNIX distribution that has > considered putting /tmp on tmpfs by default. Solaris has used a tmpfs > since 1994. Fedora moved to /tmp on tmpfs in 2012, as did ArchLinux. > Things seem to be working okay there... > For Solaris, they have quite a lot of influence on the programs they ship, and I'm certain they'd have made sure anything using large temp files would use /var/tmp. I think that Debian and Ubuntu should actually do that too, and it shouldn't even be that hard. For the other Linux distros, this is a great point. Has anybody asked the leaders of Fedora and Arch's communities if their users are experiencing any undue burden? If not, then that's certainly a good reason to throw on the pile. > As a recap, the benefits of /tmp on tmpfs are: > - Performance: reads, writes, and seeks are insanely fast in a tmpfs; > as fast as accessing RAM (I tested 1.4GB/s writes and 1.1GB/s reads > to/from tmpfs) This sounds great on the surface, however, what real thing does this benefit? Giant git merges? Sort file usage? If somebody is developing an app that uses /tmp, they're using /tmp because of these reasons: a) They are processing data in an unpredictable way where using RAM directly may be too costly. or b) They're using a tool that only operates on files on disk. or (Let's not go to the place where they're just using /tmp because they don't know better). Having tmpfs as /tmp for (a) could provide a performance benefit in many cases, but opens the system up to swap space exhaustion in extreme cases. (b) is safer than (a), and also on a very busy system probably benefits mostly from the lack of inode writes. None of the data presented actually helps us with the performance speculation unfortunately. It doesn't detract from the allure of a magic pill to make most things faster of course. :) Count me as "+/- 0" at this point. -- Ubuntu-cloud mailing list Ubuntu-cloud@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud
