Re: [ilugd] LVM and ext3 size relationship
On Wed, Nov 17, 2010 at 1:09 PM, Sirtaj Singh Kang sir...@sirtaj.netwrote: On 11/17/2010 10:57 AM, Rakesh Kumar wrote: [snip] Great to know this because i have never seen this ever. But i want to know one thing that when i studied this, found that we keep this size bigger because of metadata. And if it is correct metadata will increase with the size of PV and VG then how can a specific formula be derived. The PV and VG are irrelevant here, the goal is to find out how much logical ext3fs capacity we get from a raw block device of a certain size. When creating an LV we get a block device, ext3fs doesn't care whether it's an LV or a raw disk partition. Note that lvextend increases only the _unformatted_ capacity. So if you extend by 2G, the useful capacity you get for new files on the ext3 filesystem on that volume is approximately 1.58% less than 2G due to ext3fs overhead. OK that's why you derived this 1*1.0158. Thanks for posting such an useful formula, -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
2010/11/13 Raj Mathur (राज माथुर) r...@linux-delhi.org On Saturday 13 Nov 2010, Rakesh Kumar wrote: I think nobody is here to teach you, but as far as it is concerned to solve any problem, i have already answered you. With all due respect, you haven't. Taj asked a very specific question: If I want to grow an ext3 fs by size X in mb, how do I calculate the size Y by which I have to first resize the LVM volume on which it resides? Most sources I've read online say something like make it a little bigger just to be safe but I'm hoping there is a better and more accurate formula than that. I haven't seen an answer on this list so far. And if Taj, with his gazillions of years of experience, can't come up with an answer, I doubt if there are too many people on this list with the necessary technical qualifications who can. I know I certainly can't. I was hoping one of the deep-dive and/or broad implementation experience types like Ashish or Karanbir would have an idea, but given their lack of response, I assume they're as clueless as the rest of us on this matter. I don't know about the gazillions of years of experience for anyone but my work that i have done on Linux. And there are some very experienced person on this list but this doesn't mean that a concept will be changed. I think if someone is not able to answer, he/she shouldn't post unnecessary replies IMHO. Please note that read up on OS theory and read up on LVM fundamentals, while useful in themselves, are no use in closing the issue that was raised. He wanted a formula. He has not got a formula. End of story. I have only suggested him to read the concepts from the book(if available) and was not intended to finish this. This is a very useful book and an internationally accepted book. Regards, -- Raju -- Raj Mathurr...@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F PsyTrance Chill: http://schizoid.in/ || It is the mind that moves ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd Sorry if i couldn't explain the topic but i was only intended help him to find the exact formula. Because i as much i have experienced, i never found any error in LVM even i have never used any formula. -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Sat, Nov 13, 2010 at 6:02 PM, Sirtaj Singh Kang sir...@sirtaj.netwrote: On 13-Nov-10, at 12:39 PM, Raj Mathur (राज माथुर) wrote: [snip] He wanted a formula. He has not got a formula. Here's a formula: y = x * 1.0158 Explanation: I decided to check the ext3 overhead empirically, since I not keen to delve into the resize2fs sources. I've attached a simple python script that I used on a bunch of machines to measure the difference between block device size (reported by /proc/partitions) and fs size (as reported by statfs). It currently reports on all mounted ext2 and ext3 filesystems. Great to know this because i have never seen this ever. But i want to know one thing that when i studied this, found that we keep this size bigger because of metadata. And if it is correct metadata will increase with the size of PV and VG then how can a specific formula be derived. I might be doing something really dumb, but it looks like there is 1.56-58% overhead for almost all ext3 filesystems over 1G in size created with default mkfs flags. It gets higher for smaller filesystems. Note that the journal overhead may not be included in this value. If anyone gets different values or finds any errors in the attached script I'd appreciate a note. End of story. It's kind of a boring story. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd Sorry if i wrote rudely anywhere but my intention was not to disrespect anyone. Everybody is doing his/her work at their respected places and is here to learn from other and contribute for others. So their shouldn't be any matter of unnecessary dispute type of discussions. -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Wed, Nov 17, 2010 at 10:57 AM, Rakesh Kumar kumar3...@gmail.com wrote: On Sat, Nov 13, 2010 at 6:02 PM, Sirtaj Singh Kang sir...@sirtaj.netwrote: On 13-Nov-10, at 12:39 PM, Raj Mathur (राज माथुर) wrote: [snip] He wanted a formula. He has not got a formula. Here's a formula: y = x * 1.0158 Explanation: I decided to check the ext3 overhead empirically, since I not keen to delve into the resize2fs sources. I've attached a simple python script that I used on a bunch of machines to measure the difference between block device size (reported by /proc/partitions) and fs size (as reported by statfs). It currently reports on all mounted ext2 and ext3 filesystems. Great to know this because i have never seen this ever. But i want to know one thing that when i studied this, found that we keep this size bigger because of metadata. And if it is correct metadata will increase with the size of PV and VG then how can a specific formula be derived. I might be doing something really dumb, but it looks like there is 1.56-58% overhead for almost all ext3 filesystems over 1G in size created with default mkfs flags. It gets higher for smaller filesystems. Note that the journal overhead may not be included in this value. If anyone gets different values or finds any errors in the attached script I'd appreciate a note. End of story. It's kind of a boring story. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd Sorry if i wrote rudely anywhere but my intention was not to disrespect anyone. Everybody is doing his/her work at their respected places and is here to learn from other and contribute for others. So their shouldn't be any matter of unnecessary dispute type of discussions. -- Regards RAKESH Allow Your Own Inner Light to Guide You And why do we keep the size greater than the desired. I mean if i want to extend the size by 2G, doesn't it work? # lvextend -L+2G /dev/Volume_group/lvm # resize_reiserfs -f /dev/Volume_group/lvm -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On 11/17/2010 10:57 AM, Rakesh Kumar wrote: [snip] Great to know this because i have never seen this ever. But i want to know one thing that when i studied this, found that we keep this size bigger because of metadata. And if it is correct metadata will increase with the size of PV and VG then how can a specific formula be derived. The PV and VG are irrelevant here, the goal is to find out how much logical ext3fs capacity we get from a raw block device of a certain size. When creating an LV we get a block device, ext3fs doesn't care whether it's an LV or a raw disk partition. Note that lvextend increases only the _unformatted_ capacity. So if you extend by 2G, the useful capacity you get for new files on the ext3 filesystem on that volume is approximately 1.58% less than 2G due to ext3fs overhead. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Mon, Nov 8, 2010 at 4:16 PM, Sirtaj Singh Kang wrote: Hi all, I'd like some assistance in understanding the relationship between the size of an LVM2 logical volume and ext3. If I want to grow an ext3 fs by size X in mb, how do I calculate the size Y by which I have to first resize the LVM volume on which it resides? Most sources I've read online say something like make it a little bigger just to be safe but I'm hoping there is a better and more accurate formula than that. does -L option of lvextend not serve the purpose? i use it to resize [increase ;-)] the size of LVs before resizing/inflating the filesystem (xfs). or did i just misread your question? hth -- manish ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On 13-Nov-10, at 12:39 PM, Raj Mathur (राज माथुर) wrote: [snip] He wanted a formula. He has not got a formula. Here's a formula: y = x * 1.0158 Explanation: I decided to check the ext3 overhead empirically, since I not keen to delve into the resize2fs sources. I've attached a simple python script that I used on a bunch of machines to measure the difference between block device size (reported by /proc/partitions) and fs size (as reported by statfs). It currently reports on all mounted ext2 and ext3 filesystems. I might be doing something really dumb, but it looks like there is 1.56-58% overhead for almost all ext3 filesystems over 1G in size created with default mkfs flags. It gets higher for smaller filesystems. Note that the journal overhead may not be included in this value. If anyone gets different values or finds any errors in the attached script I'd appreciate a note. End of story. It's kind of a boring story. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
I keep forgetting that attachments are scrubbed. Script is available at: http://sirtaj.net/projects/fscomp.py -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Saturday 13 Nov 2010, Sirtaj Singh Kang wrote: I decided to check the ext3 overhead empirically, since I not keen to delve into the resize2fs sources. I've attached a simple python script that I used on a bunch of machines to measure the difference between block device size (reported by /proc/partitions) and fs size (as reported by statfs). It currently reports on all mounted ext2 and ext3 filesystems. Afraid the script didn't make it to the list -- Mailman scrubs all attachments by default. Can you upload somewhere and post a link? -- Raju -- Raj Mathurr...@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F PsyTrance Chill: http://schizoid.in/ || It is the mind that moves ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On 08-Nov-10, at 4:16 PM, Sirtaj Singh Kang wrote: [snip] I'd like some assistance in understanding the relationship between the size of an LVM2 logical volume and ext3. While nobody appears to have an answer to my question (if a good answer exists), I did some more hunting. For those who are interested, this blog post is worth a look: http://www.lisnichenko.com/articles/ext3-file-system-overhead-disclosed-part-2.html Unfortunately it doesn't address my specific situation, ie it doesn't explain what happens during resize, but it's still a useful summary. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Sat, Nov 13, 2010 at 2:07 AM, Sirtaj Singh Kang sir...@sirtaj.netwrote: On 08-Nov-10, at 4:16 PM, Sirtaj Singh Kang wrote: [snip] I'd like some assistance in understanding the relationship between the size of an LVM2 logical volume and ext3. While nobody appears to have an answer to my question (if a good answer exists) I think nobody is here to teach you, but as far as it is concerned to solve any problem, i have already answered you. , I did some more hunting. For those who are interested, this blog post is worth a look: http://www.lisnichenko.com/articles/ext3-file-system-overhead-disclosed-part-2.html Very good article. Again i would say there is nothing special in this article. If you know little bit mathematics basics of file system you can perform it. For more detail on this topic i would prefer you to take a look at book Operating System Concept by Galvin. You would definitely like it and probably you would find the remedy. Unfortunately it doesn't address my specific situation, ie it doesn't explain what happens during resize, but it's still a useful summary. Look there is nothing special in this. Operating Systems has a very special algorithm to handle resizing of file systems which take care of files written in the it. Think of it once that how do we access any file on the disk? Every file system has a link associated with the file to access it, more likely the info about the files in a file system is stored in a table. Nowadays dynamic file systems are also in use(/proc directory in Unix and Linux), but we should talk on general scenarios here. A file system is devided in smaller unit called block, particularly ext3 has the block size of 4kb. When we perform resizing OS first look in the file system table and then perform the resizing operation. It does not gurantees for the protection of data, however. It was a brief of file system resizing, but it has a lot more to think and discuss upon. I would suggest you to read that book. And yes don't get confuse between LVM and et3 these are two different things and are controlled by each other. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Saturday 13 Nov 2010, Rakesh Kumar wrote: I think nobody is here to teach you, but as far as it is concerned to solve any problem, i have already answered you. With all due respect, you haven't. Taj asked a very specific question: If I want to grow an ext3 fs by size X in mb, how do I calculate the size Y by which I have to first resize the LVM volume on which it resides? Most sources I've read online say something like make it a little bigger just to be safe but I'm hoping there is a better and more accurate formula than that. I haven't seen an answer on this list so far. And if Taj, with his gazillions of years of experience, can't come up with an answer, I doubt if there are too many people on this list with the necessary technical qualifications who can. I know I certainly can't. I was hoping one of the deep-dive and/or broad implementation experience types like Ashish or Karanbir would have an idea, but given their lack of response, I assume they're as clueless as the rest of us on this matter. Please note that read up on OS theory and read up on LVM fundamentals, while useful in themselves, are no use in closing the issue that was raised. He wanted a formula. He has not got a formula. End of story. Regards, -- Raju -- Raj Mathurr...@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F PsyTrance Chill: http://schizoid.in/ || It is the mind that moves ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On 09-Nov-10, at 12:57 AM, Rakesh Kumar wrote: [snip] First of all i would suggest you to go through some good tutorial on it. Thereafter i would add that LVM is basically something which is very helpful when you are running out of disk capacity. And it allows you to expand the disk size without loosing any data. ext3 is the format of file system. Thanks for the tip, I guess I was a little unclear. The volgroup has plenty of unallocated space, I just need to increase the size of one of the ext3 partitions that resides in a logical volume on the volgroup. This concludes that if you have to increase the size of ext3 partition by x, you should increase the size of LV by just slightly greater than your need, to get the optimum results. Yes that is what I gathered from elsewhere too. What I am hoping for is a precise definition of slightly. Is it a fixed amount indepedent of fs size? Simply a multiple of fs block size? A recurring amount to accommodate new backups of the superblock? I'm able to guess these, but I'd really like something resembling hard numbers. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
Re: [ilugd] LVM and ext3 size relationship
On Tue, Nov 9, 2010 at 1:14 AM, Sirtaj Singh Kang sir...@sirtaj.net wrote: On 09-Nov-10, at 12:57 AM, Rakesh Kumar wrote: [snip] First of all i would suggest you to go through some good tutorial on it. Thereafter i would add that LVM is basically something which is very helpful when you are running out of disk capacity. And it allows you to expand the disk size without loosing any data. ext3 is the format of file system. Thanks for the tip, I guess I was a little unclear. The volgroup has plenty of unallocated space, I just need to increase the size of one of the ext3 partitions that resides in a logical volume on the volgroup. This concludes that if you have to increase the size of ext3 partition by x, you should increase the size of LV by just slightly greater than your need, to get the optimum results. Yes that is what I gathered from elsewhere too. What I am hoping for is a precise definition of slightly. Is it a fixed amount indepedent of fs size? Simply a multiple of fs block size? A recurring amount to accommodate new backups of the superblock? I'm able to guess these, but I'd really like something resembling hard numbers. Slighly has not a very big significant here. but you can increase your size equal to the size of LV, but may be because the fs is divided into small blocks so you might have take the size of fs slightly greater or equal to the size of LV. -Taj. ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd -- Regards RAKESH Allow Your Own Inner Light to Guide You ___ Ilugd mailing list Ilugd@lists.linux-delhi.org http://frodo.hserus.net/mailman/listinfo/ilugd
