Forming drives together into a cache hierarchy
Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. Is there some magic in LVM that can do this? If there is nothing at the block device level can it be done at the filesystem level? -- Paul Richards -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
2009/9/2 Paul Richards paul.richa...@gmail.com: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. Is there some magic in LVM that can do this? If there is nothing at the block device level can it be done at the filesystem level? This looks close: http://www.cis.fiu.edu/~zhaom/dmcache But unfortunately this isn't available in a standard Debian installation. Perhaps I simply need to wait a while longer. -- Paul Richards -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
On 2009-09-02 05:30, Paul Richards wrote: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. I'd think about selling the SSD and either buy more RAM or a SCSI controller and a 10K RPM drive. Is there some magic in LVM that can do this? If there is nothing at the block device level can it be done at the filesystem level? -- Brawndo's got what plants crave. It's got electrolytes! -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
Paul Richards wrote: 2009/9/2 Paul Richards paul.richa...@gmail.com: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. Is there some magic in LVM that can do this? If there is nothing at the block device level can it be done at the filesystem level? This looks close: http://www.cis.fiu.edu/~zhaom/dmcache But unfortunately this isn't available in a standard Debian installation. Perhaps I simply need to wait a while longer. Hybrid drives were experimented with, and found not to be economically feasible. (Windows Vista has the ReadyBoost feature.) Soon enough, SSDs will be large enough, reliable enough, and cheap enough to be mainstream. Until then, if speed is of the essence, buy more RAM and fast SAS hard drives. Mark Allums -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
Ron Johnson: On 2009-09-02 05:30, Paul Richards wrote: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? -- snip I'd think about selling the SSD and either buy more RAM or a SCSI controller and a 10K RPM drive. While I would never recommend to sell an SSD (only to buy a larger one :)), I think buying more RAM is probably the way to go. Nothing beats the filesystem cache, not even an SSD. My recommendation: buy more RAM, install the OS to an SSD and copy the data most often used there as well. Put the rest on a regular disk and write a boot script to force the data you are interested in into the filesystem cache. If you cannot tell which data you are interested in in advance or if this subset of your data changes often, caching on an SSD is a hard problem to solve anyway. J. -- I want to look younger than my friends so I will fight ageing as long as I can. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
Re: Forming drives together into a cache hierarchy
Paul Richards wrote at 2009-09-02 05:30 -0500: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. If there is nothing at the block device level can it be done at the filesystem level? This reminds me of the FSCACHE kernel option, of which I know nothing but the config description: This option enables a generic filesystem caching manager that can be used by various network and other filesystems to cache data locally. Different sorts of caches can be plugged in, depending on the resources available. See Documentation/filesystems/caching/fscache.txt for more information. signature.asc Description: Digital signature
Re: Forming drives together into a cache hierarchy
2009/9/2 Jochen Schulz m...@well-adjusted.de: Ron Johnson: On 2009-09-02 05:30, Paul Richards wrote: Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? -- snip I'd think about selling the SSD and either buy more RAM or a SCSI controller and a 10K RPM drive. While I would never recommend to sell an SSD (only to buy a larger one :)), I think buying more RAM is probably the way to go. Nothing beats the filesystem cache, not even an SSD. My recommendation: buy more RAM, install the OS to an SSD and copy the data most often used there as well. Put the rest on a regular disk and write a boot script to force the data you are interested in into the filesystem cache. If you cannot tell which data you are interested in in advance or if this subset of your data changes often, caching on an SSD is a hard problem to solve anyway. I have to admit I'm slightly surprised by your response. Caching has massive benefit for main memory, so much so that we have several layers of cache. What is different about hard disks? I don't see why RAM (filesystem cache) - SSD - Rotating disk is such a bad idea.. An SSD cache has a number of benefits over a filesystem cache in RAM. Price per GB will be much lower, and also it will persist over a reboot. -- Paul Richards -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
Paul Richards: 2009/9/2 Jochen Schulz m...@well-adjusted.de: My recommendation: buy more RAM, install the OS to an SSD and copy the data most often used there as well. Put the rest on a regular disk and write a boot script to force the data you are interested in into the filesystem cache. If you cannot tell which data you are interested in in advance or if this subset of your data changes often, caching on an SSD is a hard problem to solve anyway. I have to admit I'm slightly surprised by your response. Caching has massive benefit for main memory, so much so that we have several layers of cache. What is different about hard disks? I don't see why RAM (filesystem cache) - SSD - Rotating disk is such a bad idea.. It was not my intention to suggest that your solution would not work or was a bad idea. I just wanted to suggest an easier method which would require almost no setup and no maintenance at all. Adding another layer of complexity on top of, say, LVM and crypto devices will make managing your system more complex while the speed gain my be smaller than by simply using more RAM (depends on how large you cache needs to be, obviously). An SSD cache has a number of benefits over a filesystem cache in RAM. Price per GB will be much lower, and also it will persist over a reboot. True. My reply was based on the impression that there are no mature, widely deployed solutions for your idea. Hence, I suggested an alternative. J. -- In the west we kill people like chickens. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
RE: Forming drives together into a cache hierarchy
From: Paul Richards [mailto:paul.richa...@gmail.com] Sent: Wednesday, September 02, 2009 3:30 AM Is there a way to combine a large slow drive with a fast small drive in such a way that the faster drive simply becomes a cache for the larger drive? I imagine a computer where I'd like to have a small fast SSD and a large but slow spinning disk. I'd like to put the two block devices together so that I'm presented with a single logical block device, where the operating system is using the SSD as a cache (either inclusive or exclusive, write-though or otherwise) of the spinning disk. Is there some magic in LVM that can do this? If there is nothing at the block device level can it be done at the filesystem level? Using an SSD as a cache for a hard drive isn't a good idea, since SSD's are orders of magnitude slower in writing than hard drives are. Put your OS on an SSD, and put /home (and probably /var and /tmp) on a fast hard drive, and call it a day. Here's a great article giving you more info than you probably ever wanted to know about the various flavors of SSDs: http://www.anandtech.com/storage/showdoc.aspx?i=3531 Hope this helps! -- Kevin -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
On 2009-09-02 11:38, Paul Richards wrote: [snip] An SSD cache has a number of benefits over a filesystem cache in RAM. Price per GB will be much lower, and also it will persist over a An SSD drive cheaper than RAM sticks?? -- Brawndo's got what plants crave. It's got electrolytes! -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Forming drives together into a cache hierarchy
Ron Johnson: On 2009-09-02 11:38, Paul Richards wrote: [snip] An SSD cache has a number of benefits over a filesystem cache in RAM. Price per GB will be much lower, and also it will persist over a An SSD drive cheaper than RAM sticks?? Sure. An 80GB ntel X25m still costs at least 200,- EUR, 2,50 EUR per GB. You cannot get RAM sticks cheaper than that. J. -- After the millenium I will shoot to kill. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
Re: Forming drives together into a cache hierarchy
Kevin Ross: Using an SSD as a cache for a hard drive isn't a good idea, since SSD's are orders of magnitude slower in writing than hard drives are. That strongly depends on the model. Intel SSDs are significantly faster than traditional hard disks when doing random writes. Other SSDs are better at sequential writes and outperform hard disks in that area. Here's a great article giving you more info than you probably ever wanted to know about the various flavors of SSDs: http://www.anandtech.com/storage/showdoc.aspx?i=3531 There's also a recent follow-up: http://www.anandtech.com/storage/showdoc.aspx?i=3631 J. -- I think the environment will be okay. [Agree] [Disagree] http://www.slowlydownward.com/NODATA/data_enter2.html signature.asc Description: Digital signature
