Re: ro first branch
Hi Ian & list,
On Mon, Nov 01, 2010 at 09:23:19AM -0400, Ian Stakenvicius, Aerobiology
Research wrote:
>
>On 30/10/10 10:50 PM, nomad Bellcam wrote:
>
> > Date: Sat, 30 Oct 2010 22:35:19 -0400
> > From: [1]...@aerobiology.ca
> > To: [2]damonswir...@hotmail.com
> > Subject: Re: ro first branch
> >
> > On 30/10/10 09:26 PM, nomad Bellcam wrote:
> > > hello,
> > >
> > > it is unclear to me from the docs and man page whether is is possible
> to
> > > have a rw aufs file system with a ro "first (upper-most) branch. the
> man
> > > page says "The default branch permission for the first branch is
> ârwâ,
> and
> > > the rest is âro.â" and "When you mount or remount your union
> without
> âo ro
> > > common mount option and without writable branch, aufs will warn you
> that the
> > > first branch should be writable." but i cannot find any more info
> regarding
> > > this.
> > >
> > > what i want to do is this:
> > >
> > > mount -t aufs -o br:one=ro:two=rw none base
> > >
> > > this works, but when i try to write to base i cannot (ro file
> system).
> i am
> > > not sure why aufs will not simply write to "two"
> > >
> > > if this is not possible, what is the best solution for keeping "one"
> at the
> > > top of the stack? i thought of making a dummy empty directory an
> mounting
> > > it like this:
> > >
> > > mount -t aufs -o br:dummy=rw:one=ro:two=rw none base
> > >
> > > is this the recommended way?
> > >
> > > sincerely,
> > >
> > > proctor
> >
> > What's the end goal of this, exactly? I mean, why can't you mount -o
> > br:two=rw:one=ro ?? No matter what, the read-write branch -must- stand
> > on top of the reference (read-only) branches, as that's the only way
> for
> > it to actually work like a regular fs whenever a change is made.
>
> > What's the end goal of this, exactly? I mean, why can't you mount -o
> > br:two=rw:one=ro ??
>
>what i am trying to accomplish is mounting user-defined changes to a file
>system from a live-cd (knoppix).
>
>i want to be able to make changes to the base knoppix system (using a rw
>aufs overlay on e.g. usb stick).
>
>then i want to burn the resulting "changes file" to the dvd and mount them
>from there. since
>
>they no longer reside on a writeable media i must use "ro" (i believe).
> but
>i still require them to
>
>be the "highest level changes" made to the system at that point.
>
>with "one" being my custom changes and "two" being the base knoppix system,
>
>i can do
>
>br:two=rw:one=r
>
>but (i believe) any files that are in both "one" and "two" would only be
>visible in "two" (the highest level overlay)
>
>in this case. is this assumption/understanding correct?
>
>OK so to reiterate:
>- You have the livecd (i'll call it 'x')
>- You have your user-defined changes to the livecd (i'll call it 'y')
>So firstly what you're doing is you mount -o br:y=rw,x=ro to make the image
>for 'y', right? Then you burn it to DVD, so that it becomes read-only.
>So then what you need, if I'm understanding this properly, is a -third- fs
>(say, a tmpfs) 'z' that serves as your rw branch when you're using the dvd.
>Then you can mount -o br:z=rw,y=ro,x=ro ... it will be up to you whether or
>not any changes in 'z' get written back to 'y' or not.
I believe that what you are attempting to do, is already implemented in
Knoppix. Please have a look at minirt.gz's /init:
mountunion(){
local dir tree=""
for dir in /KNOPPIX /KNOPPIX[0-9]; do
[ -d "$dir" ] && tree="$dir=ro${tree:+:$tree}"
done
mount -t aufs -o "br:$1=rw:$tree,noplink" unionfs /UNIONFS
return $?
}
mountunion() supports incremental overlays, i.e. the filesystem mounted
as /KNOPPIX is the base, /KNOPPIX[0-9] are changes vs. the base and vs
each other in ascending order, and the only argument to mountunion() is
the ONLY writable branch, either ramdisk or a read-write on-disk image
that's mounted somewhere.
Don't worry about "unionfs" here, it's just naming because of historical
reasons.
Now, if you want to record your changes from, let's say /ramdisk, to
another read-only image, you would do a
mkisofs -R /ramdisk | create_compressed_fs - 65536 > /tmp/KNOPPIX2, and
put the resulting file on your new DVD master directory into the KNOPPIX
folder. On next boot, KNOPPIX/KNOPPIX2 will be read-only mounted as
/KNOPPIX2, and mountunion() will stack your changes there on top of the
base image and put the read-write branch on top of that.
If you would like to do this without rebooting, you have to call the
full mount -t aufs command with all branches and with /UNIONFS as target
mountpoint, which will shadow the previous mount. In theory, a "-o
remount" with the addit
aufs as lifetime enhancer?! [Re: Squeeze on USB]
my own experience, I was not able to kill an usual USB flash disk
or SD card by "excessive" writes yet. I tried hard, really, even with
the mount -o sync option. The german c't magazine has also run write
tests over a year with millions of writes, and was not able to kill a
single block.
In an embedded device without an intelligent flash controller, however,
I'm using JFFS2 or _only_ controlled writes, i.e. I regularly mount the
device read-only, put every changed file on a tmpfs ramdisk (symlinked
configuration files to the tmpfs), and only write changed data back to
flash (or create a configuration tar archive) during a user-initiated
"save configuration" command.
For devices with controllers, poweroff during a write cycle can kill the
flash managements internal structure, and the flash drive can become
inaccessible beyond any possibility of repair. This can happen, for
example, if you remove an SD card while writing to it. This way, I
turned several 8GB flash cards read-only 4MB (!) devices that are not
even partitionable anymore. But this is totally independent from
problems with excessive writes.
A quote from Wikipedia:
"In practice, flash file systems are only used for memory technology
devices (MTDs), which are embedded flash memories that do not have a
controller. Removable flash memory cards and USB flash drives have
built-in controllers to perform wear leveling and error correction so
use of a specific flash file system does not add any benefit."
Summary: using aufs just for saving write cycles on flash, is probably
the wrong approach, yet it may be more comfortable ("lazy"?) than using
a device-specific method of wear-levelling or controlled writes.
> To reach this I have to use aufs, to unify some directories and files from
> /var of the system. But I need to keep some important, not to recently
> written configuration files, such as alsamixer settings, or crontab. With
> your valuable work and help I'm close :D
> (Some days ago I have been tested the wrong configuration some days there
> was about 30 writes, usual USB can handle about 1. I think it's good
> for me.)
If you are able to kill an usual USB flash drive by writing "too many
times" to it, maybe using a method that I did not know yet, please tell
me where I can buy such a one for tests. An usual flash drive that you
buy in a regular shop will handle several million file writes, and you
won't even notice defective blocks because they are also mapped out by
the internal controller and replaced by blocks on a "spare block" list.
Regards
-Klaus Knopper
PS: Now this got somewhat lengthy, sorry. I may reuse this explanation for
an article later.
--
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
Re: aufs as lifetime enhancer?! [Re: Squeeze on USB]
Hi Junjiro, On Wed, Nov 30, 2011 at 11:52:03AM +0900, sf...@users.sourceforge.net wrote: > > Hello Kalus, > > Klaus Knopper: > > From my own experience, I was not able to kill an usual USB flash disk > > or SD card by "excessive" writes yet. I tried hard, really, even with > > the mount -o sync option. The german c't magazine has also run write > > tests over a year with millions of writes, and was not able to kill a > > single block. > > Just out of curiosity, how did you make your "writes"? Various approaches, from just creating and deleting files over the file system layer with the partition mounted with sync flag in an endless loop, to dd on the raw device. Writing in sync mode and without buffers is extremely slow, so no matter which kind of test, file-based or device-based, I write only a few blocks at "the same location" (on OS side) to quickly reach the specified number of erase cycles. > It may be good to use direct-io for such tests, which makes buffer > cached disabled. Like in dd if=/dev/urandom of=/dev/sdb oflag=direct,sync bs=131072 ? You can add count=1 in order to overwrite only the first blocks, and count the repeats in a loop, so the test does not take forever until you reach the alleged write limit. A simple bash example, trying to destroy the first block of /dev/sdb containing the partition table and MBR by excessive writes: count=0 while true; do dd if=/dev/urandom of=/dev/sdb oflag=direct,sync bs=131072 count=1 2>/dev/null echo -n -e "\r $((count++))" done If partitioning /dev/sdb again would fail, after a sufficient number of writes to block zero, the destructive test was successful. Though I'm confident that the integrated controller will balance the writes amongst all physical blocks of the device on each write cycle, and block zero of /dev/sdb is not really the physical block zero on the NAND chips, I'd recommend not trying this with just your most expensive SSD. ;-) Of course I'm using aufs on Knoppix for keeping the file system layout handy without adding a lot of symlinks between read-only and read-write space, so installations and updates are possible where the base system is read-only. Just on embedded devices, I think that using aufs only for saving write cycles on the physical storage device is not the perfect way to go, especially when RAM and CPU power are tight. Regards -Klaus -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d
Re: aufs as lifetime enhancer?! [Re: Squeeze on USB]
Hello Junjiro & list, On Wed, Nov 30, 2011 at 08:28:39PM +0900, sf...@users.sourceforge.net wrote: > > Klaus Knopper: > > Various approaches, from just creating and deleting files over the file > ::: > > OK. > > > > Like in > > dd if=/dev/urandom of=/dev/sdb oflag=direct,sync bs=131072 > > ? > > Yes, oflag=direct is what I meant. > Even if you applied direct-io, you could not see the write limit on your > flash, right? I never got a single defective sector yet after several ten thousands of writes, right, so the assumption stated on wikipedia that, at least with common USB sticks and SD cards, an internal controller will manage the physical writes to blocks on these devices, seems to be correct. Here is a reference to tests that Heise Magazine did, which yields the same result (they were not able to kill an USB stick even when rewriting the same logical address 16 million times): http://www.heise.de/ct/hotline/Flash-Haltbarkeit-296140.html However, power failure or removal of media during a write, regularly caused random defects (mostly complete loss of capacity), so this seems to be the drawback of "intelligent" write block handling vs. software-side-only wear leveling and defective block management. Also, flash media ages. Manufacturers only guarantee cells to hold the information up to 10 years, so, flash is probably not good for long term storage. Well... Similar problem for magnetical or optical storage, which make it up to 50 years only. Printed paper lasts much longer. But that is a different topic. ;-) > > ... I think that using aufs only for > > saving write cycles on the physical storage device is not the perfect > > way to go, especially when RAM and CPU power are tight. > > Agreed. > And I have suggested "tovis" NOT to use aufs for such purpose. aufs is good for a lot of other things. :-) Regards -Klaus -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d
Re: auFS 3.0 Problems
Hi, Just a short note about aufs-utils/tools in Knoppix: On Sat, Dec 10, 2011 at 10:45:48AM +0900, sf...@users.sourceforge.net wrote: > Hi, > > "Michael S. Zick": > > Several problems here with auFS on a 3.0.4 kernel. > > > > All I can say about the first two is they aren't > > expected based on the on-line man.html. > ::: > > The first one is a simple case that the xino section of the manual > > does not mention that the external xino file must not exist at > > mount time: > ::: > > [10481.277111] aufs au_xino_create:693:mount[4720]: open > > /auspace/auxino(-17) > > (from the aufs manual) > -- > .B xino=filename > ::: > The files are created per an aufs and per a branch filesystem, and > unlinked. So you > cannot find this file, but it exists and is read/written frequently by > aufs. > -- > > I think "The files are created" is enough since the number 17 in the > message "open ... (-17)" is "17:EEXIST:File exists". > > > > [10620.585441] aufs au_opts_verify:1368:mount[4727]: first branch should be > > rw > ::: > > Which looks like it would work but it does not. > > The message is just a warning and the mount operations succeeded. > I am afraid you misunderstood the warning and the error. > > > auplink:plink.c:223: AUFS_CTL_PLINK_MAINT: Inappropriate ioctl for device > > It means the versions of the aufs kernel module and aufs-util are > unmatched. You should upgrade aufs-util. If your distribution (Knoppix) > doesn't provide the matched version of aufs-util to the aufs kernel > module, then I'd suggest you to ask the distribution maintainer first > (still Klaus Knopper?) Yes. Sorry, I was unaware of aufs-utils having a kernel version dependency, but considering it uses ioctls, it's understandable. I have not upgraded aufs-utils in a while, frankly, because I never used it. Knoppix uses aufs as compiled-in file system via mount, with the noplink option, so I did not run into problems with synchronizing on umount/shutdown. > If you cannot wait for them to provide the > package, then you should get aufs-util.git and install it by yourself. You could compile the aufs-utils from the aufs git matching the kernel version, or get the corresponding aufs-tools package from the debian archives. I'll take a note to upgrade aufs-tools on Knoppix for the next version. > I guess you would notice that what you did (renaming aufs tools under > /sbin) is almost equivalent to uninstall aufs-util actually. :-) > > -- > > > The open call for the xino file needs to have "truncate if exists" rather > > than error exit. > > No, I don't agree. > Let's consider the case when user specifies some important file to > "xino=" accidentally, and aufs truncate it. The user never be able to > recover the file, it is totally gone. Comparing the simple case where he > runs "rm /important/file" accidentally, in the "rm" case he can blame it > on himself with shouting "Idiot, idiot. Idiot Me!" because it is obvious > that "rm" removes the file. in the "xino=/important/file" and if aufs > trucates it, he will never blame to himself. Instead he will say "Aufs > should never remove/truncate the "xino=" file! Damn it! Just make it an > error!!" because new users don't know what xino is and never expect the > file is gone. I'd say you are over-careful here. A user that specifies an important file as target to the xino= option is unlikely to blame you for killing a file that he actually just specified by himself as target for destruction, the same way as he would not blame "rm" for not asking "Do you REALLY want to remove this file?" without the -i option. Anyways, it's your design decision, you are the author, and I think you have documented the behaviour well enough. :-) Regards -Klaus -- Learn Windows Azure Live! Tuesday, Dec 13, 2011 Microsoft is holding a special Learn Windows Azure training event for developers. It will provide a great way to learn Windows Azure and what it provides. You can attend the event by watching it streamed LIVE online. Learn more at http://p.sf.net/sfu/ms-windowsazure
Re: auFS/chroot example(s)
Hello Mike, On Fri, Dec 16, 2011 at 05:42:29AM -0600, Michael S. Zick wrote: > On Thu December 15 2011, sf...@users.sourceforge.net wrote: > > > > "Michael S. Zick": > > > @JRO - I embedded the css into the html file, so all you have > > > to do (if you want to) is save the "raw" html file, then push > > > it to your auFS website with a link to it from somewhere. > > > > > > The *.txt file is probably the better choice for your utils/samples > > > repository directory. > > > > Instead, I will add this for README and http://aufs.sf.net. > > -- > > Contributions > > - Michael S. Zick wrote good examples (currently one example), and made > > them public at > > http://hg.minimodding.com/repos/aufs/auroot.hg/ > > -- > > > > That will work for the au-chroot example(s). > > On the chance that I publish something other than > the au-chroot example(s), one directory higher: > http://hg.minimodding.com/repos/aufs/ > > Gives a list of anything related to aufs (currently one). > > - - - - > > I updated one of my desktop installs yesterday to > Debian/Wheezy (kernel-3.1.0) so will be able to > learn if the distributed auFS-utils still has the > same problems I saw in the Knoppix LiveDVD. You mean, the too old version of aufs-utils which does not match the new aufs+kernel version? I'll probably create a package for this. Btw, I'm currently working with Kernel 3.1.5 and the latest git aufs (noplink mount flag). Regards -Klaus -- Learn Windows Azure Live! Tuesday, Dec 13, 2011 Microsoft is holding a special Learn Windows Azure training event for developers. It will provide a great way to learn Windows Azure and what it provides. You can attend the event by watching it streamed LIVE online. Learn more at http://p.sf.net/sfu/ms-windowsazure
Re: aufs3 GIT release
Hello, It seems that the 3.9 branch of aufs3-standalone.git (not aufs3-linux.git) is not available yet on sourceforge or github, or did I miss something? Regards -Klaus On Mon, May 06, 2013 at 10:36:02AM +0900, sf...@users.sourceforge.net wrote: > o news > - a new branch aufs3.9 is created, and aufs3.x-rcN is for linux-3.10. > - aufs3-linux.git on SourceForge is unstable and unreliable. Until it > calm down, try git://github.com/sfjro/aufs3-linux.git instead. > > o enhancement > - the simple linked list for the aufs pseudo-link is restructured, and > it becomes a very simple hashed list. > > J. R. Okajima > > -- > - aufs3-linux.git#aufs3.2..aufs3.8 branch > aufs: plink hlist 1/5, tiny, split au_plink_half_refresh() > aufs: plink hlist 2/5, tiny, implement au_sphl for hlist > aufs: plink hlist 3/5, convert the plink list into hlist > aufs: plink hlist 4/5, make it a hashed array > aufs: plink hlist 5/5, implement /debug/aufs/si_/plink > aufs: update the donator list > > - aufs3-linux.git#aufs3.9 branch > Addtion to above, > aufs: for 3.9, remove the pos param from hlist func > > - aufs3-linux.git#aufs3.x-rcN branch > ditto > > - aufs3-standalone.git > ditto > > - aufs-util.git > none -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
Re: aufs3 GIT release
On Tue, May 07, 2013 at 11:57:58AM +0900, sf...@users.sourceforge.net wrote: > > Klaus Knopper: > > It seems that the 3.9 branch of aufs3-standalone.git (not > > aufs3-linux.git) is not available yet on sourceforge or github, or did I > > miss something? > > The aufs3-standalone.git exists on SourceForge only, and it surely > contains the aufs3.9 branch. > What made you it doesn't? $ rm -f aufs3-standalone.git $ git clone git://aufs.git.sourceforge.net/gitroot/aufs/aufs3-standalone.git aufs3-standalone.git $ cd aufs3-standalone.git $ git branch -a * master remotes/origin/HEAD -> origin/master remotes/origin/aufs3.0 remotes/origin/aufs3.1 remotes/origin/aufs3.2 remotes/origin/aufs3.3 remotes/origin/aufs3.4 remotes/origin/aufs3.5 remotes/origin/aufs3.6 remotes/origin/aufs3.7 remotes/origin/aufs3.8 remotes/origin/aufs3.x-rcN remotes/origin/master But I remember you wrote earlier something about having to check out new branches separately, just can't find your mail right now. Regards -Klaus -- Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may
Kernel 3.9 noplink problem
Hi, On Mon, May 06, 2013 at 10:36:02AM +0900, sf...@users.sourceforge.net wrote: > o enhancement > - the simple linked list for the aufs pseudo-link is restructured, and > it becomes a very simple hashed list. Could it be that this simplification broke the "noplink" mount option in aufs3.9 for Kernel 3.9? I get I/O errors, and dmesg tells me that "au_cpup_single: hi exists on b0 but plink is disabled". The message is wrong in every way, since hapens when I copy a new file to the aufs mountpoint, which definitely does NOT already exist on the writable branch 0. My aufs config: CONFIG_AUFS_FS=y CONFIG_AUFS_BRANCH_MAX_127=y # CONFIG_AUFS_BRANCH_MAX_511 is not set # CONFIG_AUFS_BRANCH_MAX_1023 is not set # CONFIG_AUFS_BRANCH_MAX_32767 is not set CONFIG_AUFS_SBILIST=y # CONFIG_AUFS_HNOTIFY is not set CONFIG_AUFS_EXPORT=y CONFIG_AUFS_INO_T_64=y # CONFIG_AUFS_RDU is not set CONFIG_AUFS_PROC_MAP=y # CONFIG_AUFS_SP_IATTR is not set # CONFIG_AUFS_SHWH is not set CONFIG_AUFS_BR_RAMFS=y CONFIG_AUFS_BR_FUSE=y CONFIG_AUFS_POLL=y CONFIG_AUFS_BR_HFSPLUS=y CONFIG_AUFS_BDEV_LOOP=y My mount tree: mount -t aufs -o br:/ramdisk=rw:/KNOPPIX=ro,noplink unionfs /UNIONFS Date: Sat May 4 04:15:22 2013 +0900 aufs3.9 20130506 Kernel 3.9.2. Regards -Klaus -- AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
Re: Kernel 3.9 noplink problem [SOLVED]
Hello Junjiro,
Thanks fpr pointing this out. I was not aware that I had hit the same
problem with a (seemingly) different scenario. This is new to me, since
until now, I had always used the sticky bit on the overlay branch and
never occured permission problems that resulted in I/O errors. Also, I
always use noplink because I did not need any.
On Tue, May 14, 2013 at 10:57:51AM +0900, sf...@users.sourceforge.net wrote:
>
> Hi Klaus,
>
> Klaus Knopper:
> > Could it be that this simplification broke the "noplink" mount option in
> > aufs3.9 for Kernel 3.9? I get I/O errors, and dmesg tells me that
> > "au_cpup_single: hi exists on b0 but plink is disabled".
>
> If the message is preceeded by "I/O Error, failed removing broken
> entry(-1, -1)" and the sticky bit is set to your upper RW layer, then
> I'd suggest you to try "chmod -t /ramdisk" or "mount -t tmpfs -o
> mode=0755 tmpfs /ramdisk" after reading
> http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04202.html
> and its thread.
I have not seen that message "I/O Error, failed removing broken
entry(-1, -1)" in dmesg, but mode=0755 in my initrd (where I set up the
aufs stack) actually fixes the problem.
As for your question in the recent thread reply
http://www.mail-archive.com/aufs-users@lists.sourceforge.net/msg04203.html
("Hmm... what should I do..."), I have no perfect answer. If it is
documented that sticky bits on the writable branch only (which is
default for tmpfs unfortunately) are bad, it's fine with me, but I think
that at least the dmesg error message is somewhat misleading and should
rather say something related to permissions, than "I/O error ... plink
...". It confused me, and will probably confuse other aufs users, too,
since the mount point permission change is not even related to plinks,
at least it's not apparent to a simple aufs user like me. ;-)
Regards & thanks for saving me time for searching
-Klaus
--
AlienVault Unified Security Management (USM) platform delivers complete
security visibility with the essential security capabilities. Easily and
efficiently configure, manage, and operate all of your security controls
from a single console and one unified framework. Download a free trial.
http://p.sf.net/sfu/alienvault_d2d
