Re: Kernel oops on aufs remount

[Yair Yarom]

   OK, I've compiled with the new release (and 5.10.105) and it works.
   Thanks,

   On Sun, 13 Mar 2022 at 14:03, J. R. Okajima <[1]hooanon...@gmail.com>
   wrote:

 Hello Yair,
 > Our kernel is vanilla 5.10.104 with the aufs merged (commit
 9e76bce469e6 (I
 > guess aufs version 5.10.82)).
 > We compile our kernel from source with aufs remote merged. When
 compiling
 > without the commits from this year (i.e. merging 919d03004ab4 from
 > 2021-12-27) this doesn't happen (I didn't have time for a full
 bisect).
 >
 > The exact command that fails in our case is:
 > mount -oremount aufs /var
 This bug may be same as the one PB (on github) reported recently.
 The fix will be released tomorrow, while I am not sure whether it
 solves
 your case or not.A  Anyway here I attach it.A  Apply the patch and
 test
 it.
 J. R. Okajima

   --

  /|   |
  \/   | Yair Yarom | System Group (DevOps)
  []   | The Rachel and Selim Benin School
  [] /\| of Computer Science and Engineering
  []//\\/  | The Hebrew University of Jerusalem
  [//  \\  | T +972-2-5494522 | F +972-2-5494522
  //\  | [2]ir...@cs.huji.ac.il
 //|

References

   1. mailto:hooanon...@gmail.com
   2. mailto:ir...@cs.huji.ac.il

Kernel oops on aufs remount

[Yair Yarom]

   Hi,
   I was trying to useA  5.10.104 with aufs, but calling mount -oremount
   causes a kernel oops.
   Our kernel is vanilla 5.10.104 with the aufs merged (commit
   9e76bce469e6 (I guess aufs version 5.10.82)).
   We compile our kernel from source with aufs remote merged. When
   compiling without the commits from this year (i.e. merging 919d03004ab4
   from 2021-12-27) this doesn't happen (I didn't have time for a full
   bisect).
   The exact command that fails in our case is:
   mount -oremount aufs /var
   Attached is additional debug data.
   Please let me know if there's anything else I can do.
   Thanks,
   --
  /|   |
  \/   | Yair Yarom | System Group (DevOps)
  []   | The Rachel and Selim Benin School
  [] /\| of Computer Science and Engineering
  []//\\/  | The Hebrew University of Jerusalem
  [//  \\  | T +972-2-5494522 | F +972-2-5494522
  //\  | [1]ir...@cs.huji.ac.il
 //|

References

   1. mailto:ir...@cs.huji.ac.il


debug.tgz
Description: application/compressed-tar

Re: aufs 4.9 removexattr issue

[Yair Yarom]

Thanks, the patch works for me (we're usually not using acl, and that's
what I've checked).

Yair.

On Sun, Jan 22 2017, sf...@users.sourceforge.net wrote:

> Hello Yair,
>
> Yair Yarom:
>> We're also seeing this issue, where removexattr returns EINVAL. Our ro
>> branch is nfs (without acl), and rw branch is tmpfs (without
>> xattr). Mounting with noacl or with +icex doesn't help. On kernel 4.8 it
>> worked fine.
>>
>> Simply mv'ing a directory from ext4 to the aufs will complain:
>> $ mv /tmp/aaa /usr/
>> mv: preserving permissions for =E2=80=98/usr/aaa=E2=80=99: Invalid argument
>>
>> Our kernel is 4.9.1 with the aufs patch (commit d75e77a on 4.9 branch)
>
> Thanx for the report.
> I think I could see the scenario. Please test this patch.
> You will need "acl" mount option for aufs to gain the full support of
> posix-acl.
>
> By the way, Justin's original post to this ML was not delivered to me. I
> just got one line "cc @sfjro" msg from him, which was posted to
> https://github.com/docker/docker/issues/30245. And I replied to github only.
> Although I could read the same post on github, it was fortunate for me
> that you replied to his post. Otherwise I could find the post much later.
>
>
> J. R. Okajima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

Re: aufs 4.9 removexattr issue

[Yair Yarom]

We're also seeing this issue, where removexattr returns EINVAL. Our ro
branch is nfs (without acl), and rw branch is tmpfs (without
xattr). Mounting with noacl or with +icex doesn't help. On kernel 4.8 it
worked fine.

Simply mv'ing a directory from ext4 to the aufs will complain:
$ mv /tmp/aaa /usr/
mv: preserving permissions for ‘/usr/aaa’: Invalid argument

Our kernel is 4.9.1 with the aufs patch (commit d75e77a on 4.9 branch)

Regards,
Yair.



a.tgz
Description: proc and sys


trace
Description: strace


On Fri, Jan 20 2017, Justin Cormack  wrote:

> removexattr seems to behave differently on aufs on kernel 4.9. Guessing
> that this was missed in the xattr changes in the port.
>
> doing a cp -rp on 4.4 kernel I got:
>
> fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tt"..., 1024) = 462
> read(3, "", 1024)   = 0
> close(3)= 0
> geteuid()   = 0
> stat("/bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> lstat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> lstat("/bar/foo", 0x7ffdad662030)   = -1 ENOENT (No such file or
> directory)
> mkdir("/bar/foo", 0700) = 0
> lstat("/bar/foo", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
> open("/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> getdents(3, /* 2 entries */, 32768) = 48
> getdents(3, /* 0 entries */, 32768) = 0
> close(3)= 0
> utimensat(AT_FDCWD, "/bar/foo", [{1484927717, 203576439}, {1484927716, 0}],
> 0) = 0
> lchown("/bar/foo", 0, 0)= 0
> getxattr("/foo", "system.posix_acl_access", 0x7ffdad661c30, 132) = -1
> ENODATA (No data available)
> stat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> getxattr("/foo", "system.posix_acl_default", 0x7ffdad661c30, 132) = -1
> ENODATA (No data available)
> stat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> setxattr("/bar/foo", "system.posix_acl_access",
> "\2\0\0\0\1\0\7\0\377\377\377\377\4\0\5\0\377\377\377\377
> \0\5\0\377\377\377\377", 28, 0) = 0
> removexattr("/bar/foo", "system.posix_acl_default") = 0
>
> but on 4.9 I get:
>
>
> fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tt"..., 1024) = 476
> read(3, "", 1024)   = 0
> close(3)= 0
> geteuid()   = 0
> stat("/bar", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> lstat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> lstat("/bar/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> open("/foo", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
> fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> getdents(3, /* 2 entries */, 32768) = 48
> getdents(3, /* 0 entries */, 32768) = 0
> close(3)= 0
> utimensat(AT_FDCWD, "/bar/foo", [{1484927717, 203576439}, {1484927716, 0}],
> 0) = 0
> getxattr("/foo", "system.posix_acl_access", 0x7fff6a3446e0, 132) = -1
> ENODATA (No data available)
> stat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> getxattr("/foo", "system.posix_acl_default", 0x7fff6a3446e0, 132) = -1
> ENODATA (No data available)
> stat("/foo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
> setxattr("/bar/foo", "system.posix_acl_access",
> "\2\0\0\0\1\0\7\0\377\377\377\377\4\0\5\0\377\377\377\377
> \0\5\0\377\377\377\377", 28, 0) = 0
> removexattr("/bar/foo", "system.posix_acl_default") = -1 EINVAL (Invalid
> argument)
> open("/usr/lib/x86_64-linux-gnu/charset.alias", O_RDONLY|O_NOFOLLOW) = -1
> ENOENT (No such file or directory)
> write(2, "cp: ", 4cp: ) = 4
> write(2, "preserving permissions for '/bar"..., 37preserving permissions
> for '/bar/foo') = 37
> write(2, ": Invalid argument", 18: Invalid argument)  = 18
> write(2, "\n", 1
>
> This makes the call fail, as removexattr is now returning EINVAL in 4.9
>
> Both systems are running 20161219 aufs standalone
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot

Re: Permissions for root user

[Yair Yarom]

On Sat, Apr 04 2015, sf...@users.sourceforge.net wrote:

 sf...@users.sourceforge.net:
 Your case looks different a little bit from Christoph Pleger's, but it
 must be same essentially. I've found linux NFS client always sets
 MS_POSIXACL (which is a flag in VFS layer) even if it was mounted with
 'noacl.' I don't think it illegal or violation something, but it surely
 makes aufs (and me) confused. I may need some dirty workaround, but I'll
 consider more and more.
 When I found a solution, I'll send you a patch and ask a test.

 Here is my current solution, a dirty work-around.
 When you have time, pleast test.

The patch indeed fixes my problem.

Thank you very much,
Yair.

--
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15utm_medium=emailutm_campaign=VA_SF

Re: Permissions for root user

[Yair Yarom]

On Wed, Apr 01 2015, sf...@users.sourceforge.net wrote:

 How was the configuration?
 If CONFIG_AUFS_XATTR is enabled, try this patch and specify 'verbose'
 option when mounting aufs. It will print the xattr name. If XATTR (ACL)
 causes the problem, then this patch and 'verbose' will tell us.

 # mount -t aufs -o br:$tmp/rw=rw:$tmp/nfs=ro,verbose aufs $tmp/union

CONFIG_AUFS_XATTR is enabled. Adding verbose (and/or -v) doesn't print
anything that I could find, do I need another option for verbose? or
where should it be printed?


 If CONFIG_AUFS_XATTR is disabled, then XATTR/ACL is not the problem. I'd
 ask you to enable CONFIG_AUFS_DEBUG and run

 # echo 1  /sys/module/aufs/parameter/debug
 mv -v $tmp/union/a/{b,c}
 # echo 0  /sys/module/aufs/parameter/debug

I've enabled debug (for the mount as well). For good measure, I'm
attaching all the relevant data (including dmesg output). The kernel is
3.19.3 with the patch you sent, and the only aufs mounted is the one
after running the script (when not unmounting it).

In the coming week, I probably won't have much time to check this
further.

Yair.



aufs.data.tgz
Description: application/gtar-compressed
--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

Re: Permissions for root user

[Yair Yarom]

On Tue, Mar 31 2015, sf...@users.sourceforge.net wrote:

 On my test system where NFS server is linux, the first mv back failed.
   :::
 exporting localhost:/tmp/irush/export
 + mv -v /tmp/irush/nfs/a/b /tmp/irush/nfs/a/c
 `/tmp/irush/nfs/a/b' - `/tmp/irush/nfs/a/c'
 mv: cannot move `/tmp/irush/nfs/a/b' to `/tmp/irush/nfs/a/c': Permission 
 denied
   :::

This is weird, assuming no_root_squash - it should work..

 Anyway  ACL is a good hint actually.
 While I cannot reproduce your problem on my side, I'd suggest you to try
 a branch attribute icexsys for the writable branch such as
 # mount -t aufs -o br:$tmp/rw=rw+icexsys:$tmp/nfs=ro aufs $tmp/union

This didn't help. I still see Operation not supported in the aufs
mv. This time I tried on a freshly complied 3.19.3.


Yair.

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

Re: Permissions for root user

[Yair Yarom]

On Mon, Mar 30 2015, Yair Yarom ir...@cs.huji.ac.il wrote:

 On Sat, Mar 28 2015, sf...@users.sourceforge.net wrote:

 Are you using user-space NFS server too?
 If so, does this script surely reproduce the problem?
 This script prints 0 at the end when everything is fine.

 Our nfs servers are NetAPP and FreeBSD machines (not user space...).

 The script itself prints 0, however I currently don't have time to
 thoroughly check it. In a couple of days when I have time I'll try to
 see if I can make a script that reproduces the problem.


It seems the script doesn't mount the nfs, so I'm a bit puzzled about
what it was suppose to check.

In any case, though I don't think it was my original problem (but I
might be wrong), it appears that currently if the nfs is mounted with
noacl, it might fail with Operation not supported even if the nfs
mount is rw and can be changed directly on the mount.

The attached script reproduces the Operation not supported when trying
to move a file on the aufs (though moving it on the nfs works). It's not
as nice as yours, it prints some stuff, and assumes nfs-kernel-server is
already running.

Yair.



c.sh.bz2
Description: BZip2 compressed data
--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

Re: Permissions for root user

[Yair Yarom]

On Mon, Mar 30 2015, sf...@users.sourceforge.net wrote:

 Yair Yarom:
 Our nfs servers are NetAPP and FreeBSD machines (not user space...).

 Ok.
 And your NFS client is debian 3.16.7-ckt7-1 as Christoph's?

Sorry, I've probably should have mentioned it, we compile our own
kernels (currently only with aufs patch, but sometimes with other
patches). I've seen this in 3.16.2, 3.17.8 and I think 3.18.1. When I
have time I will check exactly which kernels.

Yair.

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

Re: Permissions for root user

[Yair Yarom]

On Sat, Mar 28 2015, sf...@users.sourceforge.net wrote:

 Yair Yarom:
 I think we encountered a similar issue, in our case with the /var/log/*
 directories (and sometimes /var/cache/man). I don't think it's a real
 permission issue as we get Operation not supported and not Permission
 denied.

 Are you using user-space NFS server too?
 If so, does this script surely reproduce the problem?
 This script prints 0 at the end when everything is fine.

Our nfs servers are NetAPP and FreeBSD machines (not user space...).

The script itself prints 0, however I currently don't have time to
thoroughly check it. In a couple of days when I have time I'll try to
see if I can make a script that reproduces the problem.

Yair.

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

Re: Permissions for root user

[Yair Yarom]

Hello,

I think we encountered a similar issue, in our case with the /var/log/*
directories (and sometimes /var/cache/man). I don't think it's a real
permission issue as we get Operation not supported and not Permission
denied.

I noticed that it happens if the rw branch doesn't contain the
directory. So in our case we solved it by creating all the directories
on the rw branch, something like:

# root is nfs ro mounted on / and on /boot/root
# /var_rw is local disk or tmpfs
# /var is mounted /var_rw=rw,/var=ro

cd /boot/root/var
for d in log `find log -mindepth 1`; do
dst=/var_rw/$d
if [[ ! -e $dst ]]; then
if [[ -d $d ]]; then
echo creating $dst
mkdir $dst
chown --reference=$d $dst
chmod --reference=$d $dst
fi
fi
done


Regards,
Yair.


On Wed, Mar 25 2015, Christoph Pleger christoph.ple...@cs.tu-dortmund.de 
wrote:

 Hello,

 Basically aufs respects all permissions on branch fs and follows its
 behaviour. Are you using aufs on NFS client with the writable NFS
 branch, or export aufs on NFS server?

 I am exporting as readonly from the NFS server and I am using aufs on the
 client to make files and directories writable there. To achieve this, I
 changed/added some things in the initrd nfs script: Instead of mounting
 the NFS filesystem to /root, the script mounts it to /roroot, then creates
 a writable aufs filesystem by

 ln -s /proc/mounts /etc/mtab
 mount -t tmpfs tmpfs /tmpfs
 mount -t aufs -o dirs=/tmp=rw:/roroot=ro union1 /root

 - /proc/mounts (instead of the output of mount(8))
 - /sys/module/aufs/*
 - /sys/fs/aufs/* (if you have them)
 - kernel configuration or /proc/config.gz (if you have it)

 This information is attached.

 - linux kernel version
   if your kernel is not plain, for example modified by distributor,
   the url where i can download its source is necessary too.

 Debian kernel version 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1
 (2015-03-01) x86_64 GNU/Linux, downloadable from
 http://ftp.debian.org/debian/pool/main/l/linux/

 - aufs version which was printed at loading the module or booting the
   system, instead of the date you downloaded.

 aufs 3.16-20140908

 - configuration (define/undefine CONFIG_AUFS_xxx)

 This is part of the kernel configuration.

 Regards
   Christoph




 --
 Dive into the World of Parallel Programming The Go Parallel Website, sponsored
 by Intel and developed in partnership with Slashdot Media, is your hub for all
 things parallel software development, from weekly thought leadership blogs to
 news, videos, case studies, tutorials and more. Take a look and join the 
 conversation now. http://goparallel.sourceforge.net/

--
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/

aufsed /etc + sungrid = Oops

[Yair Yarom]

Hi,

We have here diskless machines with aufsed /etc (rw tmpfs /etc_rw + ro
root /etc on /etc). When trying to run sungrid's qmaster daemon, we get
an oops. Without the aufsed /etc there's no oops (even when there are
other aufsed directories).

The kernel is 2.6.33.3 with aufs standalone from git (origin/aufs2, as
there doesn't seem to be origin/aufs2-33). I'm unfamiliar with git, so I
haven't managed to get the current git version. On boot it prints: aufs
2-standalone.tree-20100301

/sys/module/aufs/parameters/brs has 1
no /debug/

Attached is a tarball with the /proc/mounts, /proc/config.gz,
si_586... from /sys/fs/aufs/, and the oops (the oops sometime has a
different backtrace but it all seems to be related to mmap).

Any help before we start compiling sungrid manually will be very
appreciated,

Yair.




aufs-oops.tgz
Description: oops data
--

Re: aufs1 combined with aufs2

[Yair Yarom]

Hi,

Thanks, I was hoping to avoid this since it tends to be one of the things you
forget to remove or forget they are there during an update (but my other
solution was to mount /sbin using aufs :)).

Thanks,
Yair.


sf...@users.sourceforge.net writes:

 Hi,

 David:
 Why don't you just write a wrapper to the binaries that decides which aufs
 binary version to run according the kernel version?

 It seems to be the best way for Yair.

 The utilities in aufs1 and aufs2 have some incompatibilities, though the
 basic feature is equivalent.

 For instance, you should flush aufs pseudo-links at remount/umount time.
 In aufs1, the script named /sbin/auplink handles it with using find and
 aulchown binary.
 In aufs2, auplink is re-written by C language and aulchown is
 unused. The major difference is that aufs2 has an ioctl interface to
 maintain the pseudo-links, and the auplink scirpt in aufs1 doesn't know.
 If you use aufs1 auplink script for aufs2, you may break pseudo-link.
 If you use aufs2 auplink binary for aufs1, it will exit in error.


 J. R. Okajima

--