Re: [gentoo-user] [OT]: Optimal formatting a SDcard (64GB) with partions of diffent sizes and filesystems?
Fernando Rodriguez frodriguez.develo...@outlook.com [15-08-08 05:42]: On Friday, August 07, 2015 7:01:29 PM Mick wrote: On Friday 07 Aug 2015 04:27:15 Fernando Rodriguez wrote: On Thursday, August 06, 2015 6:18:59 PM meino.cra...@gmx.de wrote: Hi, for my tablet PC I used a used 32GB FAT32 formatted SDcard. The formatting was already done by the manufacturer. Then I screwed it up and had to do the partioning and formatting myself again. No big deal, I thought -- and was wrong. Yes, the thing I got could be read and written. But it was DAMN slow in comparison to the original formatting. I googled and found a description, which described exactly, what I wanted: An optimal formatting for one big FAT32 partion. I did it again ;) and: TADA! The speed was back. LINK:http://zero1-st.blogspot.de/2012/05/formatting-fat32-volumes-larger- than.html Now I need the something identical but explained in a way that it can be successfully applied to any partion layout and any SDcard size. Currently the new SDcard has 64GB (yes, the tablet eats that size well :) and needs at least two partions: One FAT32 and one ext4. May be that I need a different layout later. To what aspect and logic do I have to keep my eyes on, when it comes partioning/formatting any SDcard size with any partion layout and any filesystem? Thank you very much in advance for any help! Best regards, Meino I wrote a long reply to this and it appears to have been swallowed by /dev/null. SD cards don't have 128K blocks. Except for the very early ones (standard capacity), they are divided in allocation units (AU) that are 1MB to 4MB for SDHC and even larger for SDXC. The only way to get that value is by reading a register in the card (so you can't do it in usermode on linux). The AUs are divided into Recording Units (RUs). The size of these can be deduced from the card speed class (that's the number inside the C on the label), and the card capacity. For class 2 and 4 if the card is less than 1GB it's 16KB, otherwise it's 32KB. For class 6 it is 64KB, and for class 10 it's 512KB. After an AU is erased you can write to any of the free RUs in any order in blocks of 512 bytes sequentially (the block size is configurable by the driver but 512 is the most common). But if you write to a nonfree RU then all non- free RU get copied to a new AU. So the performance hit depends on how many non-free RUs are in the AU when this happens. So to get the best performance you need to align the first FAT cluster on an AU boundary and that the RUs used by the reserved sectors after the FAT are free. This is not so easy from usermode because you can't get the AU size and you can't erase the AU to make sure reserved sectors are free. The Windows 7 and later format utility will do it if you don't partition the card. The next best thing is to align it to an RU which should be pretty easy. You could guess the AU size by writting blocks of RU size from the start of the card and timing it. Every time you hit the AU boundary there will be a longer delay. For more details see the SD specification (chapter 4.13). https://www.sdcard.org/downloads/pls/ They also have formatter tools for Windows and OSX. I tried the Windows version years ago but had problems with it (can't remember what). Excellent information Fernando, thank you! So there is no tool for me to use to read the AU/RU on the chip? The RU can be calculated from the card size and speed class, that's the number printed on the card label inside the C. I don't know of any tools to get the AU and it looks like it's not exported to userspace so any such tool would have to guess it. However, if you want to hack your kernel all it takes is one line on /usr/src/linux/drivers/mmc/core/debugfs.c. Add the following towards the end of mmc_ios_show before the return statement: seq_printf(s, au (sectors):\t%u\n, host-card-ssr.au); then you can do: cat /sys/kernel/debug/mmc0/ios Multiply the value by 512 to get the size in bytes. It can be up to 64MB. -- Fernando Rodriguez Hi Fernando, I looked into the kernel ... but ... h ... for me it seems this is only for onboard mmc flash... which is often found on embedded linux boards. Or am I wrong? I am using a micro sdcard in a sdcard-reader plugged into an USB port of my Linux PC. (Later it will be inserted into the sdcard slot of my tablet PC...) I am asking just to prevent to make things screwed up. Have a nice sunday! :) Best regards, Meino
Re: [gentoo-user] [OT]: Optimal formatting a SDcard (64GB) with partions of diffent sizes and filesystems?
On Sunday, August 09, 2015 9:31:37 AM meino.cra...@gmx.de wrote: Fernando Rodriguez frodriguez.develo...@outlook.com [15-08-08 05:42]: On Friday, August 07, 2015 7:01:29 PM Mick wrote: On Friday 07 Aug 2015 04:27:15 Fernando Rodriguez wrote: On Thursday, August 06, 2015 6:18:59 PM meino.cra...@gmx.de wrote: Hi, for my tablet PC I used a used 32GB FAT32 formatted SDcard. The formatting was already done by the manufacturer. Then I screwed it up and had to do the partioning and formatting myself again. No big deal, I thought -- and was wrong. Yes, the thing I got could be read and written. But it was DAMN slow in comparison to the original formatting. I googled and found a description, which described exactly, what I wanted: An optimal formatting for one big FAT32 partion. I did it again ;) and: TADA! The speed was back. LINK:http://zero1-st.blogspot.de/2012/05/formatting-fat32-volumes-larger- than.html Now I need the something identical but explained in a way that it can be successfully applied to any partion layout and any SDcard size. Currently the new SDcard has 64GB (yes, the tablet eats that size well :) and needs at least two partions: One FAT32 and one ext4. May be that I need a different layout later. To what aspect and logic do I have to keep my eyes on, when it comes partioning/formatting any SDcard size with any partion layout and any filesystem? Thank you very much in advance for any help! Best regards, Meino I wrote a long reply to this and it appears to have been swallowed by /dev/null. SD cards don't have 128K blocks. Except for the very early ones (standard capacity), they are divided in allocation units (AU) that are 1MB to 4MB for SDHC and even larger for SDXC. The only way to get that value is by reading a register in the card (so you can't do it in usermode on linux). The AUs are divided into Recording Units (RUs). The size of these can be deduced from the card speed class (that's the number inside the C on the label), and the card capacity. For class 2 and 4 if the card is less than 1GB it's 16KB, otherwise it's 32KB. For class 6 it is 64KB, and for class 10 it's 512KB. After an AU is erased you can write to any of the free RUs in any order in blocks of 512 bytes sequentially (the block size is configurable by the driver but 512 is the most common). But if you write to a nonfree RU then all non- free RU get copied to a new AU. So the performance hit depends on how many non-free RUs are in the AU when this happens. So to get the best performance you need to align the first FAT cluster on an AU boundary and that the RUs used by the reserved sectors after the FAT are free. This is not so easy from usermode because you can't get the AU size and you can't erase the AU to make sure reserved sectors are free. The Windows 7 and later format utility will do it if you don't partition the card. The next best thing is to align it to an RU which should be pretty easy. You could guess the AU size by writting blocks of RU size from the start of the card and timing it. Every time you hit the AU boundary there will be a longer delay. For more details see the SD specification (chapter 4.13). https://www.sdcard.org/downloads/pls/ They also have formatter tools for Windows and OSX. I tried the Windows version years ago but had problems with it (can't remember what). Excellent information Fernando, thank you! So there is no tool for me to use to read the AU/RU on the chip? The RU can be calculated from the card size and speed class, that's the number printed on the card label inside the C. I don't know of any tools to get the AU and it looks like it's not exported to userspace so any such tool would have to guess it. However, if you want to hack your kernel all it takes is one line on /usr/src/linux/drivers/mmc/core/debugfs.c. Add the following towards the end of mmc_ios_show before the return statement: seq_printf(s, au (sectors):\t%u\n, host-card-ssr.au); then you can do: cat /sys/kernel/debug/mmc0/ios Multiply the value by 512 to get the size in bytes. It can be up to 64MB. Hi Fernando, I looked into the kernel ... but ... h ... for me it seems this is only for onboard mmc flash... which is often found on embedded linux boards. Or am I wrong? I am using a micro sdcard in a sdcard-reader plugged into an USB port of my Linux PC. (Later it will be inserted into the sdcard slot of my tablet PC...) I am asking just to prevent to make things screwed up. Have a nice sunday! :) Best regards, Meino They use the same driver. The first SD cards where just MMC with a
[gentoo-user] Re: Hubris?
Paul Tobias tobias.pal at gmail.com writes: I was wondering if *anyone* has actually seen this work. I'm referring to booting a raid1 btrfs volume without performing a user-space device scan, using only the kernel `rootflags=device` setting. I have been struggling with this in various settings and am slowly starting to believe that this scenario is simply broken. It works, but a patched kernel is needed. Take a look at https://forums.gentoo.org/viewtopic-p-7275724.html The patch there was still working on the latest kernel a while ago. I used it on 2 of my systems, but I moved on and now using dracut everywhere. I not sure but the OpenSuse btrfs (non-raid1) standard install uses only btrfs partition types, including /boot and all other partitions. I do not think it uses dracut or others, but I'm not that use to opensuse to make that call. I've installed other linux distros on top of now so I no longer have thank opensuse-13.2. install to look at that install. I do know know what if any kernel patches were used either. Other linux installs such as the gentoo-derivative (calculate) linux might be worth experimentation to see if they work as you expect. Also, did you look at Rich's notes: https://docs.google.com/document/ d/1VJlJyYLTZScta9a81xgKOIBjYsG3_VfxxmUSxG23Uxg/edit?pli=1 hth, James
Re: [gentoo-user] Hubris?
On 9 Aug 2015 17:15, Jeremi Piotrowski jeremi.piotrow...@gmail.com wrote: On Wed, Jul 29, 2015 at 12:53 PM, Bruce Schultz brul...@gmail.com wrote: On 29 July 2015 6:18:43 AM AEST, Neil Bothwick n...@digimed.co.uk wrote: On Wed, 29 Jul 2015 05:29:18 +1000, Bruce Schultz wrote: But I think you do if your btrfs is raid 1. The kernel can't mount multidisk btrfs until it done a btrfs device scan in userspace, run from initramfs. According to the btrfs wiki you can pass device=/dev/sda1,device=/dev/sdb1 on the kernel boot line. I'd forgotten that option. Btrfs wiki also says this though: Using device is not recommended, as it is sensitive to device names changing. You should really be using a initramfs. Most modern distributions will do this for you automatically if you install their own btrfs-progs package. I was wondering if *anyone* has actually seen this work. I'm referring to booting a raid1 btrfs volume without performing a user-space device scan, using only the kernel `rootflags=device` setting. I have been struggling with this in various settings and am slowly starting to believe that this scenario is simply broken. It works, but a patched kernel is needed. Take a look at https://forums.gentoo.org/viewtopic-p-7275724.html The patch there was still working on the latest kernel a while ago. I used it on 2 of my systems, but I moved on and now using dracut everywhere.
[gentoo-user] Missing digest for *** Tree looks messed up.
Howdy, I'm sure I'm not alone on monitoring -dev to see upcoming changes. I noticed they switched to git or something and it seems to have caused a bit of trouble. At least I think it may have. I did my usual 'eix-sync emerge -uvaDN world'. The sync took MUCH longer than usual. I'm talking a WHOLE LOT longer than usual. My first thought, one time thing because of the changes, maybe. Then I got a screen full of this sort of stuff. * Missing digest for '/var/cache/portage/tree/media-libs/fontconfig/fontconfig-2.11.1-r2.ebuild' * Missing digest for '/var/cache/portage/tree/media-plugins/frei0r-plugins/frei0r-plugins-1.3.ebuild' * Missing digest for '/var/cache/portage/tree/sys-fs/mdadm/mdadm-3.3.1-r2.ebuild' * Missing digest for '/var/cache/portage/tree/kde-base/kdm/kdm-4.11.19.ebuild' * Missing digest for '/var/cache/portage/tree/media-plugins/gst-plugins-resindvd/gst-plugins-resindvd-0.10.23-r1.ebuild' * Missing digest for '/var/cache/portage/tree/sys-fs/fuse/fuse-2.9.4.ebuild' * Missing digest for '/var/cache/portage/tree/net-dialup/ppp/ppp-2.4.7.ebuild' - * Missing digest for '/var/cache/portage/tree/net-misc/wget/wget-1.16.ebuild' | * Missing digest for '/var/cache/portage/tree/sys-apps/dbus/dbus-1.8.16.ebuild' * Missing digest for '/var/cache/portage/tree/media-video/transcode/transcode-1.1.7-r3.ebuild' - * Missing digest for '/var/cache/portage/tree/kde-base/kwin/kwin-4.11.19.ebuild' * Missing digest for '/var/cache/portage/tree/media-fonts/efont-unicode/efont-unicode-0.4.2-r1.ebuild' | * Missing digest for '/var/cache/portage/tree/kde-apps/kolourpaint/kolourpaint-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/kde-apps/kbreakout/kbreakout-4.14.3.ebuild' - * Missing digest for '/var/cache/portage/tree/dev-cpp/eigen/eigen-3.1.3.ebuild' * Missing digest for '/var/cache/portage/tree/x11-proto/xcb-proto/xcb-proto-1.11.ebuild' | * Missing digest for '/var/cache/portage/tree/app-portage/flaggie/flaggie-0.2.1.ebuild' * Missing digest for '/var/cache/portage/tree/kde-apps/plasma-apps/plasma-apps-4.14.3.ebuild' / * Missing digest for '/var/cache/portage/tree/x11-libs/libXi/libXi-1.7.4.ebuild' * Missing digest for '/var/cache/portage/tree/kde-misc/krusader/krusader-2.4.0_beta3-r1.ebuild' \ * Manifest not found for '/var/cache/portage/tree/virtual/acl/acl-0-r1.ebuild' - * Missing digest for '/var/cache/portage/tree/dev-libs/libcdio/libcdio-0.92.ebuild' * Missing digest for '/var/cache/portage/tree/media-libs/libass/libass-0.11.2.ebuild' | * Missing digest for '/var/cache/portage/tree/kde-apps/pairs/pairs-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/x11-misc/obconf-qt/obconf-qt-0.1.0.ebuild' / * Missing digest for '/var/cache/portage/tree/kde-base/kstartupconfig/kstartupconfig-4.11.19.ebuild' * Missing digest for '/var/cache/portage/tree/dev-ruby/rdoc/rdoc-4.1.2-r1.ebuild' / * Missing digest for '/var/cache/portage/tree/kde-apps/kalgebra/kalgebra-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/app-admin/webmin/webmin-1.730.ebuild' \ * Missing digest for '/var/cache/portage/tree/x11-libs/libXau/libXau-1.0.8.ebuild' * Missing digest for '/var/cache/portage/tree/dev-python/pillow/pillow-2.8.1.ebuild' / * Missing digest for '/var/cache/portage/tree/kde-base/libkgreeter/libkgreeter-4.11.19.ebuild' * Missing digest for '/var/cache/portage/tree/kde-apps/ksnapshot/ksnapshot-4.14.3.ebuild' | * Missing digest for '/var/cache/portage/tree/net-im/pidgin/pidgin-2.10.11.ebuild' * Missing digest for '/var/cache/portage/tree/media-sound/gsm/gsm-1.0.13-r1.ebuild' - * Manifest not found for '/var/cache/portage/tree/virtual/ffmpeg/ffmpeg-9-r2.ebuild' * Missing digest for '/var/cache/portage/tree/sys-libs/cracklib/cracklib-2.9.1-r1.ebuild' | * Missing digest for '/var/cache/portage/tree/kde-apps/kiten/kiten-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/dev-perl/MailTools/MailTools-2.120.0-r1.ebuild' - * Missing digest for '/var/cache/portage/tree/net-misc/netifrc/netifrc-0.2.2.ebuild' * Missing digest for '/var/cache/portage/tree/x11-apps/iceauth/iceauth-1.0.7.ebuild' \ * Missing digest for '/var/cache/portage/tree/kde-apps/kalzium/kalzium-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/dev-libs/liblinear/liblinear-196-r1.ebuild' / * Missing digest for '/var/cache/portage/tree/app-misc/tmux/tmux-1.9a.ebuild' * Missing digest for '/var/cache/portage/tree/media-libs/libraw/libraw-0.16.0-r1.ebuild' \ * Missing digest for '/var/cache/portage/tree/kde-apps/kfmclient/kfmclient-4.14.3.ebuild' * Missing digest for '/var/cache/portage/tree/app-admin/hddtemp/hddtemp-0.3_beta15-r7.ebuild' / * Missing digest for '/var/cache/portage/tree/app-misc/screen/screen-4.0.3-r6.ebuild' * Missing digest for '/var/cache/portage/tree/media-libs/raptor/raptor-2.0.9.ebuild' - * Missing digest for '/var/cache/portage/tree/app-text/yelp-tools/yelp-tools-3.14.1.ebuild' * Missing digest for
Re: [gentoo-user] Missing digest for *** Tree looks messed up.
On Sun, Aug 9, 2015 at 8:15 PM, Dale rdalek1...@gmail.com wrote: I'm sure I'm not alone on monitoring -dev to see upcoming changes. I noticed they switched to git or something and it seems to have caused a bit of trouble. There are a few issues they're working through, but nothing really serious. There were bound to be a few bumps with the change, and most of the -dev posts are around trying to re-establish conventions, especially around things like commit comments. The desire is to make the history easy to read, parse, etc, but it isn't really stopping work from getting done and nobody is going to care if their history looks a little different. At least I think it may have. I did my usual 'eix-sync emerge -uvaDN world'. The sync took MUCH longer than usual. I'm talking a WHOLE LOT longer than usual. My first thought, one time thing because of the changes, maybe. Then I got a screen full of this sort of stuff. Interesting. As I understand it rsync generation should be turned off, but maybe some changes did make their way out. I don't believe commits are getting to the rsync servers right now, so you might see a delay in package updates for a day or so. Whether you've already seen it or will see it in the future, I would expect the first rsync to be much longer. The git migration probably touched virtually every file in the tree in some way, which means that rsync is going to be modifying just about everything. I think it actually remains to be seen whether rsync is still the fastest way to sync the new tree. If you sync from anonymous git (which is supported by repos.conf) that would probably be pretty efficient, since by design git keeps track of what did and didn't change and it doesn't need to read every inode in /usr/portage to figure it out, unlike rsync. I made a validator that finds what changed between git revisions and you can do it VERY efficiently by comparing the tree one directory level at a time (you can tell whether anything in app-backup changed in a commit without having to read any of the files, and if something changed you can repeat that one level at a time until you read the actual files that changed - it is a O(log base dir-size) tree search times the number of commits (O(n))). Now, for enough time passing (many months most likely) rsync might be more efficient since at some point you end up reading almost all the files anyway and rsync doesn't care if a file changed three times or if it changed once. TL;DR - don't worry about it too much, but don't be surprised if emerge --sync doesn't give you anything new for a day or two. You can of course clone any of the URLs under https://gitweb.gentoo.org/repo/gentoo.git/ to get the latest changes now straight from git. We're up to 210 commits in git already today. I'm sure the transition could have been less bumpy, but I'm glad we're finally seeing it happen. This has been in the works for a very long time and sometimes you just have to pull the trigger. If rsync is down for a day it isn't the end of the world. Oh, and a historical repo is posted at: https://github.com/gentoo/gentoo-gitmig-20150809-draft (that isn't official - I'm sure the official one will be gentoo-hosted, but robbat has his hands plenty full already) -- Rich
Re: [gentoo-user] Missing digest for *** Tree looks messed up.
2015-08-09 19:05 GMT-06:00 Rich Freeman ri...@gentoo.org: TL;DR - don't worry about it too much, but don't be surprised if emerge --sync doesn't give you anything new for a day or two . The rsync git mirror on github is up arlready I have been using this, for some months, and has been working great. I have this: /etc/portage/repos.conf/gentoo.conf: [gentoo] location = /var/lib/portage/repos/gentoo sync-type = git sync-uri = https://github.com/gentoo/gentoo-portage-rsync-mirror auto-sync = true Will this be preferred for users or should we users consider change to the same repo devs will commit to? I know I care about history, but not always, and not in every computer I have that uses gentoo, I'm fine with reading it from the copy of the new git repo I already have in my directory dedicated to messing around with overlays. You can of course clone any of the URLs under https://gitweb.gentoo.org/repo/gentoo.git/ to get the latest changes now straight from git. We're up to 210 commits in git already today. I'm sure the transition could have been less bumpy, but I'm glad we're finally seeing it happen. This has been in the works for a very long time and sometimes you just have to pull the trigger. If rsync is down for a day it isn't the end of the world. Is good to see gentoo modernizing it's development infrastructure, certainly, contribuiting will be much easier now. have you already decided the way you will handle user constributions? (I have read you will avoid merge commits, will devs be in charge of rebasing pull request from users?)
Re: [gentoo-user] Missing digest for *** Tree looks messed up.
2015-08-09 22:53 GMT-06:00 Jc García jyo.gar...@gmail.com: Is good to see gentoo modernizing it's development infrastructure, certainly, contribuiting will be much easier now. have you already decided the way you will handle user constributions? (I have read you will avoid merge commits, will devs be in charge of rebasing pull request from users?) ... BTW; gitg doesn't handle well the new git repo, trying to load the first commit after I clicked on it took all my RAM and swap(8G total). And vim + gitv coudln't handle the first commit either.
Re: [gentoo-user] Hubris?
On Wed, Jul 29, 2015 at 12:53 PM, Bruce Schultz brul...@gmail.com wrote: On 29 July 2015 6:18:43 AM AEST, Neil Bothwick n...@digimed.co.uk wrote: On Wed, 29 Jul 2015 05:29:18 +1000, Bruce Schultz wrote: But I think you do if your btrfs is raid 1. The kernel can't mount multidisk btrfs until it done a btrfs device scan in userspace, run from initramfs. According to the btrfs wiki you can pass device=/dev/sda1,device=/dev/sdb1 on the kernel boot line. I'd forgotten that option. Btrfs wiki also says this though: Using device is not recommended, as it is sensitive to device names changing. You should really be using a initramfs. Most modern distributions will do this for you automatically if you install their own btrfs-progs package. I was wondering if *anyone* has actually seen this work. I'm referring to booting a raid1 btrfs volume without performing a user-space device scan, using only the kernel `rootflags=device` setting. I have been struggling with this in various settings and am slowly starting to believe that this scenario is simply broken.
