Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Wed, Apr 10, 2019 at 12:00 AM Alexander Larsson wrote: > > On Tue, Apr 9, 2019 at 4:57 PM Alexander Larsson > wrote: > > > > On Thu, Apr 4, 2019 at 1:09 PM Akira TAGOH wrote: > > > > > > Yes. done. hope that works well. > > > > Yes, this now seems to work well. I think this is good to go! > > Oh, and once this lands, any chance we could have a minimal backport > of the changes to fontconfig 2.13.1 for the LTS flatpak runtimes? Thanks for testing. for minimal backport, let me figure it out and apply for Fedora updates. you can pull it in from there. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Apr 9, 2019 at 4:57 PM Alexander Larsson wrote: > > On Thu, Apr 4, 2019 at 1:09 PM Akira TAGOH wrote: > > > > Yes. done. hope that works well. > > Yes, this now seems to work well. I think this is good to go! Oh, and once this lands, any chance we could have a minimal backport of the changes to fontconfig 2.13.1 for the LTS flatpak runtimes?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Apr 4, 2019 at 1:09 PM Akira TAGOH wrote: > > On Thu, Apr 4, 2019 at 5:46 PM Alexander Larsson > wrote: > > There are still some issues though: > > * Only flatpak 1.2.0 and later generates the dir-remapping fontconfig > > file, so earlier versions of flatpak will regenerate caches for > > /run/host/fonts. > > * On current distros, with the uuid file generated the host fc-cache > > will only generate the uuid style cache files, and these will not be > > used by the fontconfig in the sandbox. > > > > The first I think is fine, just update your flatpak to the latest > > stable release and it will be fast. > > > > However, the second is a problem. It means that the second we update > > the flatpak runtime to the latest fontconfig all flatpak apps will get > > a slow first startup on all current distros. > > > > Is it possible that we could make the new fontconfig not generate uuid > > cache files, but look for them as fallback? I.e. if the (possibly > > salted) regular path checksum doesn't exist, look for a .uuid file in > > that dir and use that for lookup? > > Yes. done. hope that works well. Yes, this now seems to work well. I think this is good to go!
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Apr 4, 2019 at 5:46 PM Alexander Larsson wrote: > There are still some issues though: > * Only flatpak 1.2.0 and later generates the dir-remapping fontconfig > file, so earlier versions of flatpak will regenerate caches for > /run/host/fonts. > * On current distros, with the uuid file generated the host fc-cache > will only generate the uuid style cache files, and these will not be > used by the fontconfig in the sandbox. > > The first I think is fine, just update your flatpak to the latest > stable release and it will be fast. > > However, the second is a problem. It means that the second we update > the flatpak runtime to the latest fontconfig all flatpak apps will get > a slow first startup on all current distros. > > Is it possible that we could make the new fontconfig not generate uuid > cache files, but look for them as fallback? I.e. if the (possibly > salted) regular path checksum doesn't exist, look for a .uuid file in > that dir and use that for lookup? Yes. done. hope that works well. > > On Thu, Apr 4, 2019 at 8:50 AM Akira TAGOH wrote: > > > > Ah, thank you for catching this up. I fixed similar case before but > > missed the case for sub directories. fixed. > > > > On Wed, Apr 3, 2019 at 10:01 PM Alexander Larsson > > wrote: > > > > > > On Tue, Apr 2, 2019 at 2:35 PM Akira TAGOH wrote: > > > > > > > > Thanks for testing, Alex. > > > > > > Doing some more testing, and I think I have found an issue. > > > > > > On the host i regenerate caches with your branch, getting FC_DEBUG=16 > > > output (among other output): > > > > > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > > /usr/share/fonts/cantarell, salt: (null)) > > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > > /usr/share/fonts/cantarell, salt: (null)) > > > charsets 5 -> 1 leaves 75 -> 15 > > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > > /usr/share/fonts/cantarell, salt: (null)) > > > > > > In the flatpak-generated config in the sandbox have: > > > /run/host/fonts > > > /run/host/fonts > > > > > > Then, running fc-list in the sandbox prints: > > > > > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > > salt: (null)) > > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > > salt: (null)) > > > charsets 5 -> 1 leaves 75 -> 15 > > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > > salt: (null)) > > > FcDirCacheWriteDir dir "/run/host/fonts/cantarell" file > > > "/home/alex/.var/app/org.gnome.eog/cache/fontconfig//85bba0c73358da0b93a259c9d2b16b14-le64.cache-7" > > > > > > Note how these get different checksums, meaning that the host cache is > > > not used. I think this is due to the double-slash in "mapped to > > > /usr/share/fonts//cantarell". > > > > > > > > -- > > Akira TAGOH -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Ok, this seems to work well with the latest version of the branch and my test case: https://gist.github.com/alexlarsson/8912637dd6173591a8dec7c043cfa01d However, I had to fix an issue in flatpak-builder: https://github.com/flatpak/flatpak-builder/pull/277 Because fc-cache was picking up the SOURCE_DATE_EPOCH of the build and injecting it into the cache builds, thus causing it to create invalid caches (were not matching the on-disk dir mtime so was never used). My fix of this is is to reset the directory mtimes to 0 during the build (as they will be when deploying), as this makes fc-cache ignore the SOURCE_DATE_EPOCH. So, if we get a new fontconfig with this landed, we could update the existing runtimes to that version, and then we will be prepared for when distros update to fontconfig without the uuid support. There are still some issues though: * Only flatpak 1.2.0 and later generates the dir-remapping fontconfig file, so earlier versions of flatpak will regenerate caches for /run/host/fonts. * On current distros, with the uuid file generated the host fc-cache will only generate the uuid style cache files, and these will not be used by the fontconfig in the sandbox. The first I think is fine, just update your flatpak to the latest stable release and it will be fast. However, the second is a problem. It means that the second we update the flatpak runtime to the latest fontconfig all flatpak apps will get a slow first startup on all current distros. Is it possible that we could make the new fontconfig not generate uuid cache files, but look for them as fallback? I.e. if the (possibly salted) regular path checksum doesn't exist, look for a .uuid file in that dir and use that for lookup? On Thu, Apr 4, 2019 at 8:50 AM Akira TAGOH wrote: > > Ah, thank you for catching this up. I fixed similar case before but > missed the case for sub directories. fixed. > > On Wed, Apr 3, 2019 at 10:01 PM Alexander Larsson > wrote: > > > > On Tue, Apr 2, 2019 at 2:35 PM Akira TAGOH wrote: > > > > > > Thanks for testing, Alex. > > > > Doing some more testing, and I think I have found an issue. > > > > On the host i regenerate caches with your branch, getting FC_DEBUG=16 > > output (among other output): > > > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > /usr/share/fonts/cantarell, salt: (null)) > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > /usr/share/fonts/cantarell, salt: (null)) > > charsets 5 -> 1 leaves 75 -> 15 > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > > /usr/share/fonts/cantarell, salt: (null)) > > > > In the flatpak-generated config in the sandbox have: > > /run/host/fonts > > /run/host/fonts > > > > Then, running fc-list in the sandbox prints: > > > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > salt: (null)) > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > salt: (null)) > > charsets 5 -> 1 leaves 75 -> 15 > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > > salt: (null)) > > FcDirCacheWriteDir dir "/run/host/fonts/cantarell" file > > "/home/alex/.var/app/org.gnome.eog/cache/fontconfig//85bba0c73358da0b93a259c9d2b16b14-le64.cache-7" > > > > Note how these get different checksums, meaning that the host cache is > > not used. I think this is due to the double-slash in "mapped to > > /usr/share/fonts//cantarell". > > > > -- > Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Ah, thank you for catching this up. I fixed similar case before but missed the case for sub directories. fixed. On Wed, Apr 3, 2019 at 10:01 PM Alexander Larsson wrote: > > On Tue, Apr 2, 2019 at 2:35 PM Akira TAGOH wrote: > > > > Thanks for testing, Alex. > > Doing some more testing, and I think I have found an issue. > > On the host i regenerate caches with your branch, getting FC_DEBUG=16 > output (among other output): > > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > /usr/share/fonts/cantarell, salt: (null)) > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > /usr/share/fonts/cantarell, salt: (null)) > charsets 5 -> 1 leaves 75 -> 15 > cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: > /usr/share/fonts/cantarell, salt: (null)) > > In the flatpak-generated config in the sandbox have: > /run/host/fonts > /run/host/fonts > > Then, running fc-list in the sandbox prints: > > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > salt: (null)) > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > salt: (null)) > charsets 5 -> 1 leaves 75 -> 15 > cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: > /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), > salt: (null)) > FcDirCacheWriteDir dir "/run/host/fonts/cantarell" file > "/home/alex/.var/app/org.gnome.eog/cache/fontconfig//85bba0c73358da0b93a259c9d2b16b14-le64.cache-7" > > Note how these get different checksums, meaning that the host cache is > not used. I think this is due to the double-slash in "mapped to > /usr/share/fonts//cantarell". -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Fixed. thanks. On Wed, Apr 3, 2019 at 9:38 PM Alexander Larsson wrote: > > I noticed this in the FC_DEBUG=16 output: >/run/host/fonts -> /usr/share/fonts (salt: (null)) > > Printf NULL like this is fine in glibc, but can crash in other OSes, > so might be nice to fix. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Apr 2, 2019 at 2:35 PM Akira TAGOH wrote: > > Thanks for testing, Alex. Doing some more testing, and I think I have found an issue. On the host i regenerate caches with your branch, getting FC_DEBUG=16 output (among other output): cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: /usr/share/fonts/cantarell, salt: (null)) cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: /usr/share/fonts/cantarell, salt: (null)) charsets 5 -> 1 leaves 75 -> 15 cache: /6ba42aef58711b5caaf10d690066-le64.cache-7 (dir: /usr/share/fonts/cantarell, salt: (null)) In the flatpak-generated config in the sandbox have: /run/host/fonts /run/host/fonts Then, running fc-list in the sandbox prints: cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), salt: (null)) cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), salt: (null)) charsets 5 -> 1 leaves 75 -> 15 cache: /85bba0c73358da0b93a259c9d2b16b14-le64.cache-7 (dir: /run/host/fonts/cantarell (mapped to /usr/share/fonts//cantarell), salt: (null)) FcDirCacheWriteDir dir "/run/host/fonts/cantarell" file "/home/alex/.var/app/org.gnome.eog/cache/fontconfig//85bba0c73358da0b93a259c9d2b16b14-le64.cache-7" Note how these get different checksums, meaning that the host cache is not used. I think this is due to the double-slash in "mapped to /usr/share/fonts//cantarell".
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
I noticed this in the FC_DEBUG=16 output: /run/host/fonts -> /usr/share/fonts (salt: (null)) Printf NULL like this is fine in glibc, but can crash in other OSes, so might be nice to fix.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Apr 2, 2019 at 2:35 PM Akira TAGOH wrote: > > Thanks for testing, Alex. > > On Mon, Apr 1, 2019 at 11:35 PM Alexander Larsson > wrote: > > I don't understand why its generating cache files with the same > > filenames, even though the salt for /usr/share/fonts is different. > > I've added some debugging output (try FC_DEBUG=16 if you want to see > more) to see how fontconfig determines cache filename with given > config. and I found etc/fonts/conf.d/50-flatpak.conf has: > > > /usr/share/fonts > > and reading config files in ASCII order. so when comparing the name of > "50-flatpak-salted.conf" and "50-flatpak.conf", 50-flatpak-salted.conf > is read first. then it was reset and overwritten (newly added in fact) > by the non unique string in 50-flatpak.conf. > That's the reason why you got same cache names in two runs. > So if you don't need to keep the above two lines in 50-flatpak.conf, > you can drop them from there and add to > 50-flatpak-salted.conf. Thanks! This was super confusing. I was assuming "50-flatpak" would sort before "50-flatpak-salted", and i guess it does, but what gets sorted is "50-flatpak.conf"... I'm gonna rename it to 49-flatpak-salted to make it less confusing.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
There are another problem that salt can't be overwritten by another dir elements coming later. which looks intuitive to me. so I'll fix it. On Tue, Apr 2, 2019 at 9:35 PM Akira TAGOH wrote: > > Thanks for testing, Alex. > > On Mon, Apr 1, 2019 at 11:35 PM Alexander Larsson > wrote: > > I don't understand why its generating cache files with the same > > filenames, even though the salt for /usr/share/fonts is different. > > I've added some debugging output (try FC_DEBUG=16 if you want to see > more) to see how fontconfig determines cache filename with given > config. and I found etc/fonts/conf.d/50-flatpak.conf has: > > > /usr/share/fonts > > and reading config files in ASCII order. so when comparing the name of > "50-flatpak-salted.conf" and "50-flatpak.conf", 50-flatpak-salted.conf > is read first. then it was reset and overwritten (newly added in fact) > by the non unique string in 50-flatpak.conf. > That's the reason why you got same cache names in two runs. > So if you don't need to keep the above two lines in 50-flatpak.conf, > you can drop them from there and add to > 50-flatpak-salted.conf. > > > And why does it think they are invalid after having just generated > > them? (The script that generates the 50-flatpak-salted.conf removes > > all caches, so the claimed to be invalid files are definitely created > > by fc-cache...) > > From the debugging message for consideration: > /usr/cache/fontconfig: cleaning cache directory > FcCacheTimeValid dir "/usr/share/fonts/liberation-fonts" cache > checksum 1554199639.0 dir checksum 155420102 > 3.810342069 > /usr/cache/fontconfig: invalid cache file: > e9b75eee6795385ae3f53a200406b963-le64.cache-7 > > Those sum is coming from mtime of cache and dir. fontconfig detects > that dir has been updated but cache wasn't. so considered it is > outdated... > > > Also, i worry about the fact that it seems to list all the fonts in > > /usr/share/fonts twice, the second time complaining about looping. > > Maybe the reset-dirs is not working in fc-cache, so its picking up the > > first instance of /usr/share/fonts? > > No. that is because scanning paths recursively twice when parsing dir > element in config and generating caches. so that is irrelevant to this > though. > Hmm, maybe good to shutting that message up or stop scanning recursively > either. > > > Also, why is it warning about > > "/usr/share/fontconfig/conf.avail/05-reset-dirs-sample.conf"? Why is > > it even trying to parse this file? It should not be used? > > This isn't actually new. when path can't be converted with > prefix="xdg" (for instance, unable to determine relevant XDG env vars > and even hone dir) and so on, that makes a path empty. but that > message seems accidentally shown. I've reverted this behavior to the > past one. > > -- > Akira TAGOH -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Thanks for testing, Alex. On Mon, Apr 1, 2019 at 11:35 PM Alexander Larsson wrote: > I don't understand why its generating cache files with the same > filenames, even though the salt for /usr/share/fonts is different. I've added some debugging output (try FC_DEBUG=16 if you want to see more) to see how fontconfig determines cache filename with given config. and I found etc/fonts/conf.d/50-flatpak.conf has: /usr/share/fonts and reading config files in ASCII order. so when comparing the name of "50-flatpak-salted.conf" and "50-flatpak.conf", 50-flatpak-salted.conf is read first. then it was reset and overwritten (newly added in fact) by the non unique string in 50-flatpak.conf. That's the reason why you got same cache names in two runs. So if you don't need to keep the above two lines in 50-flatpak.conf, you can drop them from there and add to 50-flatpak-salted.conf. > And why does it think they are invalid after having just generated > them? (The script that generates the 50-flatpak-salted.conf removes > all caches, so the claimed to be invalid files are definitely created > by fc-cache...) >From the debugging message for consideration: /usr/cache/fontconfig: cleaning cache directory FcCacheTimeValid dir "/usr/share/fonts/liberation-fonts" cache checksum 1554199639.0 dir checksum 155420102 3.810342069 /usr/cache/fontconfig: invalid cache file: e9b75eee6795385ae3f53a200406b963-le64.cache-7 Those sum is coming from mtime of cache and dir. fontconfig detects that dir has been updated but cache wasn't. so considered it is outdated... > Also, i worry about the fact that it seems to list all the fonts in > /usr/share/fonts twice, the second time complaining about looping. > Maybe the reset-dirs is not working in fc-cache, so its picking up the > first instance of /usr/share/fonts? No. that is because scanning paths recursively twice when parsing dir element in config and generating caches. so that is irrelevant to this though. Hmm, maybe good to shutting that message up or stop scanning recursively either. > Also, why is it warning about > "/usr/share/fontconfig/conf.avail/05-reset-dirs-sample.conf"? Why is > it even trying to parse this file? It should not be used? This isn't actually new. when path can't be converted with prefix="xdg" (for instance, unable to determine relevant XDG env vars and even hone dir) and so on, that makes a path empty. but that message seems accidentally shown. I've reverted this behavior to the past one. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Mar 26, 2019 at 9:09 AM Alexander Larsson wrote: > > On Tue, Mar 26, 2019 at 7:15 AM Akira TAGOH wrote: > > > > Hi Alex, > > > > Have you tried new implementation yet? I believe it should be > > reflected our discussion though, I want to see some comment from you > > because flatpak is only customer for salt thing so far. if it works, I > > can commit it to master and make a release for them. > > I have not, but I will put it on the list, will have a look later this week. So, i'm trying to test this, but I'm running into a weird issue I don't understand. I started by creating a new flatpak runtime, deriving from gnome 3.30, adding just the fontconfig from your branch. All the files requred are here: https://gist.github.com/alexlarsson/8912637dd6173591a8dec7c043cfa01d Including a build.sh which will build the entire thing. The weird thing is that i'm failing to create the cache in one of two cases. What happens is that flatpak-builder first builds the SDK, and after building that it runs the "cleanup-commands", which (see the gist for the commands) produces this output: CLEANUP SDK created /etc/fonts/conf.d/50-flatpak-salted.conf: /usr/share/fonts regenerating system caches for sdk Fontconfig warning: "/usr/etc/fonts/fonts.conf", line 33: empty font directory name ignored Fontconfig warning: "/usr/share/fontconfig/conf.avail/05-reset-dirs-sample.conf", line 7: empty font directory name ignored /usr/share/fonts: caching, new cache contents: 0 fonts, 7 dirs /usr/share/fonts/cantarell: caching, new cache contents: 5 fonts, 0 dirs /usr/share/fonts/dejavu: caching, new cache contents: 22 fonts, 0 dirs /usr/share/fonts/eosrei-emojione: caching, new cache contents: 1 fonts, 0 dirs /usr/share/fonts/gnu-free: caching, new cache contents: 12 fonts, 0 dirs /usr/share/fonts/google-crosextra-caladea: caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/google-crosextra-carlito: caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/liberation-fonts: caching, new cache contents: 12 fonts, 0 dirs /run/host/fonts: skipping, no such directory /run/host/user-fonts: skipping, no such directory /usr/share/fonts/cantarell: skipping, looped directory detected /usr/share/fonts/dejavu: skipping, looped directory detected /usr/share/fonts/eosrei-emojione: skipping, looped directory detected /usr/share/fonts/gnu-free: skipping, looped directory detected /usr/share/fonts/google-crosextra-caladea: skipping, looped directory detected /usr/share/fonts/google-crosextra-carlito: skipping, looped directory detected /usr/share/fonts/liberation-fonts: skipping, looped directory detected /usr/cache/fontconfig: cleaning cache directory /app/cache/fontconfig: not cleaning non-existent cache directory /run/host/fonts-cache: not cleaning non-existent cache directory /run/host/user-fonts-cache: not cleaning non-existent cache directory /usr/var/cache/fontconfig: cleaning cache directory fc-cache: succeeded resulting /usr/cache/fontconfig: total 184 -rw-rw-r-- 1 1000 1000 2536 Apr 1 14:25 240592bdd7bdf1b16d89e7edeb4d2486-le64.cache-7 -rw-rw-r-- 1 1000 1000 77096 Apr 1 14:25 38abaa55af3aedbc13e8d17a987f8026-le64.cache-7 -rw-rw-r-- 1 1000 1000 47104 Apr 1 14:25 46df990863fe9cfbed3fe21e89696ec6-le64.cache-7 -rw-rw-r-- 1 1000 1000 6936 Apr 1 14:25 612dd7798ef7fe64f776a6e49f128560-le64.cache-7 -rw-rw-r-- 1 1000 1000 8376 Apr 1 14:25 b599ef9b1adc87dbcc1c41de14b1d733-le64.cache-7 -rw-rw-r-- 1 1000 1000 6312 Apr 1 14:25 b5aefa49b91159320ae79a3bc3a8ec09-le64.cache-7 -rw-r--r-- 1 1000 1000 200 Apr 1 14:25 CACHEDIR.TAG -rw-rw-r-- 1 1000 1000 424 Apr 1 14:25 e9903abff2d89c54a7d6bfcd0dde9b25-le64.cache-7 -rw-rw-r-- 1 1000 1000 19928 Apr 1 14:25 e9b75eee6795385ae3f53a200406b963-le64.cache-7 Then it starts over with the platform (non-devel runtime) and does something similar to it. This produces the following output: CLEANUP PLATFORM created /etc/fonts/conf.d/50-flatpak-salted.conf: /usr/share/fonts regenerating system caches for platform Fontconfig warning: "/usr/etc/fonts/fonts.conf", line 33: empty font directory name ignored Fontconfig warning: "/usr/share/fontconfig/conf.avail/05-reset-dirs-sample.conf", line 7: empty font directory name ignored /usr/share/fonts: caching, new cache contents: 0 fonts, 7 dirs /usr/share/fonts/cantarell: caching, new cache contents: 5 fonts, 0 dirs /usr/share/fonts/dejavu: caching, new cache contents: 22 fonts, 0 dirs /usr/share/fonts/eosrei-emojione: caching, new cache contents: 1 fonts, 0 dirs /usr/share/fonts/gnu-free: caching, new cache contents: 12 fonts, 0 dirs /usr/share/fonts/google-crosextra-caladea: caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/google-crosextra-carlito: caching, new cache contents: 4 fonts, 0 dirs /usr/share/fonts/liberation-fonts: caching, new cache contents: 12 fonts, 0 dirs /run/host/fonts: skipping, no such directory /run/host/user-fonts: skipping, no such directory /usr/share/fonts/cantarell: skipping, loop
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Mar 26, 2019 at 7:15 AM Akira TAGOH wrote: > > Hi Alex, > > Have you tried new implementation yet? I believe it should be > reflected our discussion though, I want to see some comment from you > because flatpak is only customer for salt thing so far. if it works, I > can commit it to master and make a release for them. I have not, but I will put it on the list, will have a look later this week.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Hi Alex, Have you tried new implementation yet? I believe it should be reflected our discussion though, I want to see some comment from you because flatpak is only customer for salt thing so far. if it works, I can commit it to master and make a release for them. TIA, On Fri, Feb 8, 2019 at 2:34 PM Akira TAGOH wrote: > > No comments on this (yet) so just opened a merge request here: > https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/30 > > If anyone can review, that would be appreciated. > > On Thu, Jan 31, 2019 at 7:36 PM Akira TAGOH wrote: > > > > I've done implementation for salt: > > https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework > > > > That should works as expected. > > For summary to support this on flatpak side, All of flatpaks needs to have: > > > >/usr/share/fonts > > > > and more if there are any fonts directories on sandbox where has same > > directory names on host. > > or may be better having a salt regardless of it to make them > > differenciated to host perhaps. I'll leave it to you, Alex. > > > > No salt should be needed for remap-dir so far. > > > > Ah, and if you want to manage fonts directories in a single file and > > don't want to modify fonts.conf, you can put prior to > > re-define own fonts directories with dir elements. > > > > Please test it. > > > > On Thu, Jan 31, 2019 at 5:56 PM Alexander Larsson > > wrote: > > > > > > On Thu, Jan 31, 2019 at 9:54 AM Akira TAGOH wrote: > > > > > > > > Also if host dirs are available as is on sandbox like Alex concerned, > > > > we could simply have: > > > > > > > > /opt/fonts > > > > > > Yeah, assuming there is no global default salt that messes this up. > > > > > > > > -- > > Akira TAGOH > > > > -- > Akira TAGOH -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
No comments on this (yet) so just opened a merge request here: https://gitlab.freedesktop.org/fontconfig/fontconfig/merge_requests/30 If anyone can review, that would be appreciated. On Thu, Jan 31, 2019 at 7:36 PM Akira TAGOH wrote: > > I've done implementation for salt: > https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework > > That should works as expected. > For summary to support this on flatpak side, All of flatpaks needs to have: > >/usr/share/fonts > > and more if there are any fonts directories on sandbox where has same > directory names on host. > or may be better having a salt regardless of it to make them > differenciated to host perhaps. I'll leave it to you, Alex. > > No salt should be needed for remap-dir so far. > > Ah, and if you want to manage fonts directories in a single file and > don't want to modify fonts.conf, you can put prior to > re-define own fonts directories with dir elements. > > Please test it. > > On Thu, Jan 31, 2019 at 5:56 PM Alexander Larsson > wrote: > > > > On Thu, Jan 31, 2019 at 9:54 AM Akira TAGOH wrote: > > > > > > Also if host dirs are available as is on sandbox like Alex concerned, > > > we could simply have: > > > > > > /opt/fonts > > > > Yeah, assuming there is no global default salt that messes this up. > > > > -- > Akira TAGOH -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: > Yeah, assuming there is no global default salt that messes this up. It sounds like having 'salt' values in dir and remap-dir elements is what we want then -- no need for separate salt elements. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
I've done implementation for salt: https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework That should works as expected. For summary to support this on flatpak side, All of flatpaks needs to have: /usr/share/fonts and more if there are any fonts directories on sandbox where has same directory names on host. or may be better having a salt regardless of it to make them differenciated to host perhaps. I'll leave it to you, Alex. No salt should be needed for remap-dir so far. Ah, and if you want to manage fonts directories in a single file and don't want to modify fonts.conf, you can put prior to re-define own fonts directories with dir elements. Please test it. On Thu, Jan 31, 2019 at 5:56 PM Alexander Larsson wrote: > > On Thu, Jan 31, 2019 at 9:54 AM Akira TAGOH wrote: > > > > Also if host dirs are available as is on sandbox like Alex concerned, > > we could simply have: > > > > /opt/fonts > > Yeah, assuming there is no global default salt that messes this up. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 31, 2019 at 9:54 AM Akira TAGOH wrote: > > Also if host dirs are available as is on sandbox like Alex concerned, > we could simply have: > > /opt/fonts Yeah, assuming there is no global default salt that messes this up.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 31, 2019 at 9:40 AM Keith Packard wrote: > > Alexander Larsson writes: > > > We don't want a global salt for everything in the container. > > I guess I wonder why not? Salt + dir inside the container will always be > unique. The place where you want to have different salt is for > directories mapped from the host; I think those will always be in > remap-dir clauses, if we have salt there, that should work? It will be unique per runtime, as the default salt comes from the runtime generated file in the sandbox. But, if multiple apps use the same runtime they will always use the same salt for its /app/share/fonts, even though it could differ, which seems risky. But worse, if the runtime changes the default salt, then the caches built into the app will be for the wrong salt until the app is rebuilt. > > In > > reality things are more complicated than that. For example, an app may > > bundle fonts, which will be in like /app/share/fonts, in addition to > > the runtime fonts in /usr/share/fonts. These come from different > > places and may individually be different in a different (or updated) > > app, so the directories need to have different salts. > > If building the flatpak generates the font caches, then per-flatpak salt > would make those correct. Only as long as the runtime salt doesn't change. But its also a robustness issue, if for whatever reason (cache version, etc) a cache is generated for the app dir I want that to map to a uniqe cache id rather than one shared between every app using that runtime. > > Also, it is quite possible that some host font directory is *not* > > remapped, but still visible to the app. For example /opt/fonts for an > > app that has filesystem access. If for whatever reason fontconfig > > looks at this directory it should not apply any salt for it. > > Oh, so some host directories may be visible unmapped and unknown to the > flatpak? In that case, we'll need to enumerate all flatpak visible font > directories separately. What happens today is that if you host has fonts somewhere, say /opt/fonts, and the app has --filesystem=host access, then /opt will be visible as-is in the sandbox (along with essentially everything but /usr and /app). These directories are not (currently) added to the fontconfig setup in the sandbox, so it will not be automatically picked up. But that doesn't mean fontconfig in the sandbox can *never* see it. For example, the app could call FcConfigAppFontAddDir() if it has a per-app font dir option or something. We *could* make flatpak enumerate *all* configured host font directories and set up snippets for them. However, currently we don't, and I don't see a huge need for this.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 31, 2019 at 5:35 PM Keith Packard wrote: > > Alexander Larsson writes: > > > As I said in an earlier email, it needs to be in the individual dir > > elements, because a global salt is not right. > > Do you want it in the elements directly? That would be more > straightforward in many ways and could avoid troubles with separate salt > declarations that take effect more broadly than one directory. > > So, one file (generated at flatpak creation time) with > > /usr/share/fonts > /run/host/fonts > > and another (generated at runtime) with > > /run/host/fonts > > Presumably you will mask all host configured font paths somehow? Maybe > you need to be able to inherit the 'salt' value from the host (if set)? > If so, we could have: > > /run/host/fonts Yeah, I agree with it. Having a salt in dir and remap-dir would flexibly works I think. Though, given that there are no salt in host, we just need to have a salt for dirs inside sandbox only. so salt shouldn't be needed for remap-dir in this case. Also if host dirs are available as is on sandbox like Alex concerned, we could simply have: /opt/fonts > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: > We don't want a global salt for everything in the container. I guess I wonder why not? Salt + dir inside the container will always be unique. The place where you want to have different salt is for directories mapped from the host; I think those will always be in remap-dir clauses, if we have salt there, that should work? > In > reality things are more complicated than that. For example, an app may > bundle fonts, which will be in like /app/share/fonts, in addition to > the runtime fonts in /usr/share/fonts. These come from different > places and may individually be different in a different (or updated) > app, so the directories need to have different salts. If building the flatpak generates the font caches, then per-flatpak salt would make those correct. > Also, it is quite possible that some host font directory is *not* > remapped, but still visible to the app. For example /opt/fonts for an > app that has filesystem access. If for whatever reason fontconfig > looks at this directory it should not apply any salt for it. Oh, so some host directories may be visible unmapped and unknown to the flatpak? In that case, we'll need to enumerate all flatpak visible font directories separately. I think we need a complete enumeration of the cases; I keep seeing more options... -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: > As I said in an earlier email, it needs to be in the individual dir > elements, because a global salt is not right. Do you want it in the elements directly? That would be more straightforward in many ways and could avoid troubles with separate salt declarations that take effect more broadly than one directory. So, one file (generated at flatpak creation time) with /usr/share/fonts /run/host/fonts and another (generated at runtime) with /run/host/fonts Presumably you will mask all host configured font paths somehow? Maybe you need to be able to inherit the 'salt' value from the host (if set)? If so, we could have: /run/host/fonts -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 31, 2019 at 7:14 AM Akira TAGOH wrote: > > On Thu, Jan 31, 2019 at 8:07 AM Keith Packard wrote: > > > > This would avoid collision between one and origins. and assuming that > > > flatpaks can load config from host too, we could have: > > > > > > 10-salt.conf (from host): > > > > > > > I'd leave this out and not have salt in the host. > > Sure. though implicit thing may breaks something easily like own salt > affects 'remap-dir' unexpectedly. that should be documented carefully. > or should we have salt attribute in remap-dir and dir elements > instead? that would be obvious. As I said in an earlier email, it needs to be in the individual dir elements, because a global salt is not right.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Wed, Jan 30, 2019 at 10:33 AM Akira TAGOH wrote: > > On Wed, Jan 30, 2019 at 5:02 AM Keith Packard wrote: > > Yes, that was the plan. Alexander suggested the following syntax: > > > > /usr/share/fonts > > > > I think this will work, although it seems a bit fragile. In particular, > > if the host has salt for some directories, those will be defined > > relative to the host paths, not the flatpak paths. > > Right. though this syntax looks primitive. it might manages to be done > if one writes everything for what they want. but if we can have easier > way - which can be done with minimal effort and inheritance for others > from host - that may be better. > > > Do we need to process the 'salt' elements and 'remap-dir' elements in > > order and remap old salt elements as remap-dir elements get loaded? That > > also seems fragile to me. > > Hm, to deal with more complicated cases, I guess we may need to have > one global salt to affect everything and a path-specific salt for > remapped path. for flatpak case, they want to have a global salt to > change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set > salt from host in 'remap-dir' to build cache filenames on host (for > /run/host/fonts and so on). > This would avoid collision between one and origins. and assuming that > flatpaks can load config from host too, we could have: We don't want a global salt for everything in the container. In reality things are more complicated than that. For example, an app may bundle fonts, which will be in like /app/share/fonts, in addition to the runtime fonts in /usr/share/fonts. These come from different places and may individually be different in a different (or updated) app, so the directories need to have different salts. Also, it is quite possible that some host font directory is *not* remapped, but still visible to the app. For example /opt/fonts for an app that has filesystem access. If for whatever reason fontconfig looks at this directory it should not apply any salt for it.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Tue, Jan 29, 2019 at 9:02 PM Keith Packard wrote: > > Akira TAGOH writes: > Do we need to process the 'salt' elements and 'remap-dir' elements in > order and remap old salt elements as remap-dir elements get loaded? That > also seems fragile to me. > > Perhaps some command that the flatpak could run to generate host salt > values so that it could remap them into new salt elements using the > mapped paths? > > Alternatively, we could just assume that only flatpak will use the salt > mechanism and leave this for a future enhancement? I think in practice salt on the host-side of something like this will only be there if we're using multiple levels of containers, which all pass down the outermost fonts. I don't think this is going to be very common at all, so maybe just ignoring it for now is fine?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Yep, that looks good to me. I don't time this week to hack on the code, and it looks like you're doing great anyways; let me know if you need help in any way with the implementation work. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 31, 2019 at 8:07 AM Keith Packard wrote: > We may want a command line tool that extracts data from the config > for use by flatpak in building the dynamic configuration, including > things like salt values per directory. Yeah, that might be made to > work with flatpak essentially manually overriding the salt configuration > so that it uses the flatpak-relative names. Aha. yeah, that's a good idea. > > This would avoid collision between one and origins. and assuming that > > flatpaks can load config from host too, we could have: > > > > 10-salt.conf (from host): > > > > I'd leave this out and not have salt in the host. Sure. though implicit thing may breaks something easily like own salt affects 'remap-dir' unexpectedly. that should be documented carefully. or should we have salt attribute in remap-dir and dir elements instead? that would be obvious. > The salt here would need to have CDATA for the target directories, and I > think flatpack wants to split the dynamic from static config bits. > > Dynamic (built at runtime): > > /run/host/fonts > > Static (built in the flatpak): > > /usr/share/fonts Yep, that looks good to me. > I think a dir-reset makes a lot of sense so that the flatpak can control > the set of font paths used. Building a command-line tool that flatpak > can use to discover the relevant fontconfig information seems like a > useful improvement; as I recall, flatpak is currently assuming > /usr/share/fonts and ~/.fonts are used on the host. > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Hm, to deal with more complicated cases, I guess we may need to have > one global salt to affect everything and a path-specific salt for > remapped path. Or, more likely, no salt at all for the outermost layer as it doesn't really add anything here. > for flatpak case, they want to have a global salt to > change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set > salt from host in 'remap-dir' to build cache filenames on host (for > /run/host/fonts and so on). We may want a command line tool that extracts data from the config for use by flatpak in building the dynamic configuration, including things like salt values per directory. Yeah, that might be made to work with flatpak essentially manually overriding the salt configuration so that it uses the flatpak-relative names. > This would avoid collision between one and origins. and assuming that > flatpaks can load config from host too, we could have: > > 10-salt.conf (from host): > I'd leave this out and not have salt in the host. > 50-flatpak.conf (sandbox specific): > /run/host/fonts > > /usr/share/fonts The salt here would need to have CDATA for the target directories, and I think flatpack wants to split the dynamic from static config bits. Dynamic (built at runtime): /run/host/fonts Static (built in the flatpak): /usr/share/fonts > > First salt element affects to 'remap-dir' and second one overrides it > for paths and change a salt in sandbox. I think we can put that into the remap-dir element as both of those are built at runtime? > To make things easier, we may also want to export all of dir elements > from fonts.conf to the separate file. flatpak can replace it with > 50-flatpak.conf in this case. or the file operation isn't desirable, > let's implement dir-reset element or something like that. I think a dir-reset makes a lot of sense so that the flatpak can control the set of font paths used. Building a command-line tool that flatpak can use to discover the relevant fontconfig information seems like a useful improvement; as I recall, flatpak is currently assuming /usr/share/fonts and ~/.fonts are used on the host. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Le 2019-01-30 11:23, Nicolas Mailhot a écrit : Le 2019-01-30 10:32, Akira TAGOH a écrit : This would avoid collision between one and origins. and assuming that flatpaks can load config from host too, we could have: 10-salt.conf (from host): Note that the "id" attribute has special meaning and constrains in xml. I'm not saying it should not be used, but it will restrict future evolutions more than another choice. One of the constrains being it must be unique. So if you use id for salt, you effectively forbid the use of the same salt for two different elements. (of course one can cheat with non-conformant xml parsers, that restricts interoperability though). Regards, -- Nicolas Mailhot
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Le 2019-01-30 10:32, Akira TAGOH a écrit : This would avoid collision between one and origins. and assuming that flatpaks can load config from host too, we could have: 10-salt.conf (from host): Note that the "id" attribute has special meaning and constrains in xml. I'm not saying it should not be used, but it will restrict future evolutions more than another choice. -- Nicolas Mailhot
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Wed, Jan 30, 2019 at 5:02 AM Keith Packard wrote: > Yes, that was the plan. Alexander suggested the following syntax: > > /usr/share/fonts > > I think this will work, although it seems a bit fragile. In particular, > if the host has salt for some directories, those will be defined > relative to the host paths, not the flatpak paths. Right. though this syntax looks primitive. it might manages to be done if one writes everything for what they want. but if we can have easier way - which can be done with minimal effort and inheritance for others from host - that may be better. > Do we need to process the 'salt' elements and 'remap-dir' elements in > order and remap old salt elements as remap-dir elements get loaded? That > also seems fragile to me. Hm, to deal with more complicated cases, I guess we may need to have one global salt to affect everything and a path-specific salt for remapped path. for flatpak case, they want to have a global salt to change a salt in sandbox (for /usr/share/fonts in sandbox etc) and set salt from host in 'remap-dir' to build cache filenames on host (for /run/host/fonts and so on). This would avoid collision between one and origins. and assuming that flatpaks can load config from host too, we could have: 10-salt.conf (from host): 50-flatpak.conf (sandbox specific): /run/host/fonts /usr/share/fonts First salt element affects to 'remap-dir' and second one overrides it for paths and change a salt in sandbox. To make things easier, we may also want to export all of dir elements from fonts.conf to the separate file. flatpak can replace it with 50-flatpak.conf in this case. or the file operation isn't desirable, let's implement dir-reset element or something like that. > > Perhaps some command that the flatpak could run to generate host salt > values so that it could remap them into new salt elements using the > mapped paths? > > Alternatively, we could just assume that only flatpak will use the salt > mechanism and leave this for a future enhancement? > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Hi, > > We are still missing a piece of a salt to deal with a directory name > separately where possibly have different fonts in sandbox etc. my tree > based on Keith's previous implementation works and passed test cases > except this salt thing: > > https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework Awesome, thanks for getting this going. I'm digging out from being away from the office for a week... > So have we got a consensus on letting flatpak provide a separate > config file contained a salt? Yes, that was the plan. Alexander suggested the following syntax: /usr/share/fonts I think this will work, although it seems a bit fragile. In particular, if the host has salt for some directories, those will be defined relative to the host paths, not the flatpak paths. Do we need to process the 'salt' elements and 'remap-dir' elements in order and remap old salt elements as remap-dir elements get loaded? That also seems fragile to me. Perhaps some command that the flatpak could run to generate host salt values so that it could remap them into new salt elements using the mapped paths? Alternatively, we could just assume that only flatpak will use the salt mechanism and leave this for a future enhancement? -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Hi, We are still missing a piece of a salt to deal with a directory name separately where possibly have different fonts in sandbox etc. my tree based on Keith's previous implementation works and passed test cases except this salt thing: https://gitlab.freedesktop.org/tagoh/fontconfig/commits/flatpak-rework So have we got a consensus on letting flatpak provide a separate config file contained a salt? On Fri, Jan 25, 2019 at 9:39 PM Keith Packard wrote: > > Akira TAGOH writes: > > > Sure. I started to implement it based on Keith's branch. > > Awesome. I'll be back home "tomorrow" and able to spend some time on > this next week. > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Sure. I started to implement it based on Keith's branch. Awesome. I'll be back home "tomorrow" and able to spend some time on this next week. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Fri, Jan 25, 2019 at 4:35 PM Alexander Larsson wrote: > > On Fri, Jan 25, 2019 at 5:07 AM Akira TAGOH wrote: > > > > > > > > > > as-path=... ? > > > > That sounds good to me. > > This sounds good to me to. I updated the PR: > https://github.com/flatpak/flatpak/pull/2635#issuecomment-457482239 > > Can I get an ack on this format? Sure. I started to implement it based on Keith's branch. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Fri, Jan 25, 2019 at 5:07 AM Akira TAGOH wrote: > > > > > > as-path=... ? > > That sounds good to me. This sounds good to me to. I updated the PR: https://github.com/flatpak/flatpak/pull/2635#issuecomment-457482239 Can I get an ack on this format?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 9:43 PM Alexander Larsson wrote: > > On Thu, Jan 24, 2019 at 1:18 PM Akira TAGOH wrote: > > > > On Thu, Jan 24, 2019 at 8:55 PM Alexander Larsson > > wrote: > > > I agree that cache path is wrong as we have something else by that name. > > > > > > host-path kinda hardcodes the sandbox case for rewriting though. Maybe > > > "remapped-path"? > > > > "remap" looks redundant to the element name. simply "to" or "as" as > > you proposed that earlier? > > as-path=... ? That sounds good to me. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 1:18 PM Akira TAGOH wrote: > > On Thu, Jan 24, 2019 at 8:55 PM Alexander Larsson > wrote: > > I agree that cache path is wrong as we have something else by that name. > > > > host-path kinda hardcodes the sandbox case for rewriting though. Maybe > > "remapped-path"? > > "remap" looks redundant to the element name. simply "to" or "as" as > you proposed that earlier? as-path=... ?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 8:55 PM Alexander Larsson wrote: > I agree that cache path is wrong as we have something else by that name. > > host-path kinda hardcodes the sandbox case for rewriting though. Maybe > "remapped-path"? "remap" looks redundant to the element name. simply "to" or "as" as you proposed that earlier? -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 12:20 PM Akira TAGOH wrote: > > On Thu, Jan 24, 2019 at 7:54 PM Keith Packard wrote: > > I think we might bike-shed on the names here a bit -- 'real-path' is > > pretty ambiguous as both paths are 'real', one is just the file system > > path and the other is the cache path. I like using the file system path > > as the contents of the element and the cache path as the property, but > > perhaps the property name could be 'cache-path' instead? > > Hmm, that looks not intuitive to me. in fact both are also a font > path. "cache path" are the sort of /var/cache/fontconfig or so. how > about "host-path"? although one can see it is remapping a path to > somewhere but still missing how this actually works in fontconfig. > i.e. to determine a cache filename to read. I agree that cache path is wrong as we have something else by that name. host-path kinda hardcodes the sandbox case for rewriting though. Maybe "remapped-path"?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 8:20 PM Akira TAGOH wrote: > Hmm, that looks not intuitive to me. in fact both are also a font > path. "cache path" are the sort of /var/cache/fontconfig or so. how > about "host-path"? although one can see it is remapping a path to > somewhere but still missing how this actually works in fontconfig. > i.e. to determine a cache filename to read. Actually there are no way to guess how cache filenames are determined without looking at source code. dir element is to add a font path. and "remap-" dir is to remap a font path to somewhere. that is consistent enough at this point. I revoke the last sentence where is starting from "although". -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 7:54 PM Keith Packard wrote: > I think we might bike-shed on the names here a bit -- 'real-path' is > pretty ambiguous as both paths are 'real', one is just the file system > path and the other is the cache path. I like using the file system path > as the contents of the element and the cache path as the property, but > perhaps the property name could be 'cache-path' instead? Hmm, that looks not intuitive to me. in fact both are also a font path. "cache path" are the sort of /var/cache/fontconfig or so. how about "host-path"? although one can see it is remapping a path to somewhere but still missing how this actually works in fontconfig. i.e. to determine a cache filename to read. > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 12:02 PM Alexander Larsson wrote: > > On Thu, Jan 24, 2019 at 11:59 AM Alexander Larsson > wrote: > > > > On Thu, Jan 24, 2019 at 11:54 AM Keith Packard wrote: > > > > > > Alexander Larsson writes: > > > > > > $ cat /run/host/font-dirs.xml > > > > > > > > > > > > > > > > /run/host/fonts > > > > > > > real-path="/home/alex/.fonts">/run/host/user-fonts > > > > > > > > > > > > Is this format acceptable? Its mostly about naming the nodes and the > > > > attributes, so its basically trivial. If you want i can rename things > > > > or change orders, but I'd really just like an Ack on something. > > > > > > Format looks OK. > > > > > > I think we might bike-shed on the names here a bit -- 'real-path' is > > > pretty ambiguous as both paths are 'real', one is just the file system > > > path and the other is the cache path. I like using the file system path > > > as the contents of the element and the cache path as the property, but > > > perhaps the property name could be 'cache-path' instead? > > > > Or maybe 'cache-as' ? > > Actually, i think cache-path fits better with the "remap-dir" node > name. I'll change the PR. https://github.com/flatpak/flatpak/pull/2635#issuecomment-457158368
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 11:59 AM Alexander Larsson wrote: > > On Thu, Jan 24, 2019 at 11:54 AM Keith Packard wrote: > > > > Alexander Larsson writes: > > > > $ cat /run/host/font-dirs.xml > > > > > > > > > > > > /run/host/fonts > > > > > real-path="/home/alex/.fonts">/run/host/user-fonts > > > > > > > > > Is this format acceptable? Its mostly about naming the nodes and the > > > attributes, so its basically trivial. If you want i can rename things > > > or change orders, but I'd really just like an Ack on something. > > > > Format looks OK. > > > > I think we might bike-shed on the names here a bit -- 'real-path' is > > pretty ambiguous as both paths are 'real', one is just the file system > > path and the other is the cache path. I like using the file system path > > as the contents of the element and the cache path as the property, but > > perhaps the property name could be 'cache-path' instead? > > Or maybe 'cache-as' ? Actually, i think cache-path fits better with the "remap-dir" node name. I'll change the PR.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 24, 2019 at 11:54 AM Keith Packard wrote: > > Alexander Larsson writes: > > $ cat /run/host/font-dirs.xml > > > > > > > > /run/host/fonts > > > real-path="/home/alex/.fonts">/run/host/user-fonts > > > > > > Is this format acceptable? Its mostly about naming the nodes and the > > attributes, so its basically trivial. If you want i can rename things > > or change orders, but I'd really just like an Ack on something. > > Format looks OK. > > I think we might bike-shed on the names here a bit -- 'real-path' is > pretty ambiguous as both paths are 'real', one is just the file system > path and the other is the cache path. I like using the file system path > as the contents of the element and the cache path as the property, but > perhaps the property name could be 'cache-path' instead? Or maybe 'cache-as' ?
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: $ cat /run/host/font-dirs.xml > > > > /run/host/fonts > /run/host/user-fonts > > > Is this format acceptable? Its mostly about naming the nodes and the > attributes, so its basically trivial. If you want i can rename things > or change orders, but I'd really just like an Ack on something. Format looks OK. I think we might bike-shed on the names here a bit -- 'real-path' is pretty ambiguous as both paths are 'real', one is just the file system path and the other is the cache path. I like using the file system path as the contents of the element and the cache path as the property, but perhaps the property name could be 'cache-path' instead? -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Wed, Jan 23, 2019 at 9:49 PM Keith Packard wrote: > > Akira TAGOH writes: > > > Keith, > > > > I'm fine with it. do you have a time to work on it? or already started > > working? > > If not, please let me know. > > I've been at a conference all week, but hope to have time next > week. Would be happy to see others get this started, or collaborate in > any way. If I do get started, I'll be pushing stuff frequently and > posting mail so I don't block others. So, what i really want to happen immediately (ideally this week) is to decide on a format for the directory renaming configuration, because I want to get that into the next stable flatpak release. To expedite this I just made up some shit and made a flatpak PR for generating that format: https://github.com/flatpak/flatpak/pull/2635 As an example, this generates on my system: $ ./flatpak run --command=sh org.gnome.gedit [📦 org.gnome.gedit flatpak]$ cat /run/host/font-dirs.xml /run/host/fonts /run/host/user-fonts Is this format acceptable? Its mostly about naming the nodes and the attributes, so its basically trivial. If you want i can rename things or change orders, but I'd really just like an Ack on something.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Keith, > > I'm fine with it. do you have a time to work on it? or already started > working? > If not, please let me know. I've been at a conference all week, but hope to have time next week. Would be happy to see others get this started, or collaborate in any way. If I do get started, I'll be pushing stuff frequently and posting mail so I don't block others. Would be awesome to get this done in the next couple of weeks... -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Keith, I'm fine with it. do you have a time to work on it? or already started working? If not, please let me know. On Tue, Jan 22, 2019 at 12:33 AM Alexander Larsson wrote: > > On Sun, Jan 13, 2019 at 7:21 PM Keith Packard wrote: > > > > Alexander Larsson writes: > > > > Yeah, there will be two files. One static in the runtime > > > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak > > > (/run/host/fontconf.xml). > > > > Sounds like we've got a plan for this part -- fix my mapping code to use > > config bits separate from the elements, then add a 'salt' > > mechanism in the config bits that stirs in some random data when > > generating the cache keys for specific directory trees. > > Whats the status of this. Can we come up with an agreed upon format > for doing the dir remapping (outside of the dir nodes) soon? I'm about > to do a flatpak 1.2 release and would like to preempt things by > generating the dir mapping dynamic snippet so we can use it when we > update fontconfig in the runtime. (Note: We need not yet finalize the > salt stuff, because that will not be in the dynamic flatpak-generated > part of this.) -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Sun, Jan 13, 2019 at 7:21 PM Keith Packard wrote: > > Alexander Larsson writes: > > Yeah, there will be two files. One static in the runtime > > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak > > (/run/host/fontconf.xml). > > Sounds like we've got a plan for this part -- fix my mapping code to use > config bits separate from the elements, then add a 'salt' > mechanism in the config bits that stirs in some random data when > generating the cache keys for specific directory trees. Whats the status of this. Can we come up with an agreed upon format for doing the dir remapping (outside of the dir nodes) soon? I'm about to do a flatpak 1.2 release and would like to preempt things by generating the dir mapping dynamic snippet so we can use it when we update fontconfig in the runtime. (Note: We need not yet finalize the salt stuff, because that will not be in the dynamic flatpak-generated part of this.)
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Sun, Jan 13, 2019 at 8:53 PM Alexander Larsson wrote: > Yes:ish. It should not *normally* happen. But you may run into it in > uncommon situations like e.g. chrome using a statically linked version > of fontconfig that has a different fontconfig cache format. > > > > However, flatpak will never parse the entire xml fontconfig file > > > format (which isn't even really stable over time), so such a config > > > would have to be external in a simpler config format. > > > > The fontconfig xml format is quite stable and is designed to be > > manipulated by tools that do not understand the full contents, hence > > using XML and providing a suitable DTD. > > We recently ran into issues with fontconfig xml parsing errors in chrome > when using config files from a newer host fontconfig that were not > parsable by the statically linked chrome copy, so it is not perfect. Right. that is really a pain in the neck. not adding new syntax is hard to improve and grow. that said, just ignoring unknown syntax would makes harder to find an error. we may need to think about measures for that like checking a version of config and library say, but anyway. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Sun, Jan 13, 2019 at 7:21 PM Keith Packard wrote: > > Yes:ish. It should not *normally* happen. But you may run into it in > > uncommon situations like e.g. chrome using a statically linked version > > of fontconfig that has a different fontconfig cache format. > > Hrm. I don't get the sense that we've got a solution to this problem > yet. Given that the contents of the flatpak cannot change on the fly, > would it be sufficient to generate a new 'salt' value for the flatpak > each time it is changed? It might be sufficient to use the name of the > flatpak including a version as this salt (that has the advantage of > making the flatpak reproducible, which you probably want to encourage). The plan was to generate a new random salt each build. We *do* in general want to make flatpak builds as reproducible as possible, because the magic of ostree means identical files are deduplicated on disk and not downloaded on updates. However, if fixed set of small files change each build that is not a huge problem. So, in such a setup we're ok as long as the statically linked fontconfig is new enough to respect the salt config. However, we could run into issues if e.g. chrome statically links to an older fontconfig. Such fonconfig versions would ignore the salt, and miss the runtime-shipped caches, instead picking up the host caches. It seems in that case the host cache would have a mismatching mtime though, and we'd generate a new one, plus we'd regenerate the host cache for the remapped dir. So this should work, at some cost in performance. > > Yeah, there will be two files. One static in the runtime > > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak > > (/run/host/fontconf.xml). > > Sounds like we've got a plan for this part -- fix my mapping code to use > config bits separate from the elements, then add a 'salt' > mechanism in the config bits that stirs in some random data when > generating the cache keys for specific directory trees. > > Let's figure out how we should handle the stale flatpak fonts cache > issue. Once we've settled that, we can go implement the whole mess and > get a new fontconfig release made in time for debian freeze. Sounds good to me. Actually, i'm planning a flatpak stable (1.2) release in the next few weeks, and I would like to add support for the flatpak-side of this to the release. So, if we can just decide the syntax for the dir remapping bits now I can add support for that. Then we can do the runtime part when there is a fontconfig release that supports it.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: > Ugh. Sorry about that! Thanks. Bugs happen, fortunately I was able to track this one down and fix it (we all love free software!) > Yes:ish. It should not *normally* happen. But you may run into it in > uncommon situations like e.g. chrome using a statically linked version > of fontconfig that has a different fontconfig cache format. Hrm. I don't get the sense that we've got a solution to this problem yet. Given that the contents of the flatpak cannot change on the fly, would it be sufficient to generate a new 'salt' value for the flatpak each time it is changed? It might be sufficient to use the name of the flatpak including a version as this salt (that has the advantage of making the flatpak reproducible, which you probably want to encourage). > I agree with this, but the point was that we can't just modify the > fontconfig to be dynamic always. We need to design it such that > whatever flatpak generates is optional for the runtime to pick up when > it is able to handle it. Sounds like being able to handle an arbitrary host fonts directory mounted as /run/host/fonts will really help avoid future issues. And that needs to be in the fontconfig used inside the flatpak, which means we can't wait for a system which has fonts in a different place and plan on fixing it in the host. > Yeah, there will be two files. One static in the runtime > (/etc/fonts/conf.d/50-flatpak.xml), and one generated by flatpak > (/run/host/fontconf.xml). Sounds like we've got a plan for this part -- fix my mapping code to use config bits separate from the elements, then add a 'salt' mechanism in the config bits that stirs in some random data when generating the cache keys for specific directory trees. Let's figure out how we should handle the stale flatpak fonts cache issue. Once we've settled that, we can go implement the whole mess and get a new fontconfig release made in time for debian freeze. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Fri, Jan 11, 2019 at 11:07 PM Keith Packard wrote: > > Alexander Larsson writes: > > A .uuid file was added and removed to every directory in the font tree > which contained no fonts or sub-directories; (a) the directory mtime was > changed, causing the system to re-scan the fonts and then > re-create/re-delete the .uuid file (goto a). Somehow this would > eventually stablize (I'm not entirely sure how). It was a 'surprising' > problem that happened only once in a while and it took several days of > searching to locate as the only symptom was that firefox would hang for > 'a while' and then start working. Once I ran strace on the process while > hung, it was pretty easy to track down, but having fontconfig affecting > directory mtimes was not what I expected... Ugh. Sorry about that! > > However, the second problem is that it puts demands on the *host*, as > > it now has to match the layout of the runtimes so the pathnames can > > match identically. > > Well, I was thinking that the runtime would dynamically adapt to the > host environment and change the configuration inside to match the > host. > > Is it that flatpak treats fonts very specially, or are there other bits > of host data injected into the flatpak runtime in a similar fashion? The point of any kind of container system is that you can have a container filesystem that is guaranteed (to some 99% extent anyway) to be the same wherever you run it. This way any kind of testing and hardcoding you do on your system will be valid on the users system, elimiting a whole form of bugs. There are also technical issues with doing this. First of all, the runtime (which is basically a directory that gets mounted on /usr have no real requirements, its up to whoever create it to decide how and what goes in there. So, say the host stores fonts in /usr/data/fonts or whatever here might not be a corresponding directory in the runtime to bind-mount on top of. And if there is there is no guarantee that its not used for something else. Basically, flatpak lets the runtime have full control of /usr, and anything special (i.e. not direct mappings of directories the sandbox is supposed to have access too) that we inject in the sandbox is in /run. There are a bunch of stuff we inject in the sandbox other than fonts, but not a lot. First of all there are various unix domain sockets for things like dbus, X11, etc. Then fonts, icon themes, timezone config, resolv config, and the host /usr (as /run/host/usr) if the sandbox has full filesystem access. > Essentially, I was imagining the flatpak runtime system would discover > the set of fontconfig directories exposed by the host and inject those > into the runtime using matching paths. Right now, you inject two fixed > paths (/usr/share/fonts and ~/.fonts), but if we really want to deal > with systems that put fonts in other places, then presumably flatpak > will need to adapt anyways. They are fixed, but at flatpak build time, so a "weird" system could configure flatpak to pick them up elsewhere, while exposing them in a standardized location in the sandbox. True, this only allows two directories, but I don't think this is really a huge issue in practice. > > Maybe "weird" setups like nix will run into issues. You might know > > this better. > > Good point; as new distros start experimenting with different filesystem > layouts, we will need to be more cautious about assuming fixed paths > of any kind. Same with experimenting with the layout of the runtime itself. > > Yes, but if you update the runtime and /usr/share/fonts-minimal > > changed in the new version (but has same mtime), then the stale cache > > file in the users homedir will still be used. > > Ah. Thanks for explaining this. It seems like the only way this could > happen is if the cache file within the flatpak was stale and a > replacement generated and written to the user's homedir (as the only > writable location available). Yes:ish. It should not *normally* happen. But you may run into it in uncommon situations like e.g. chrome using a statically linked version of fontconfig that has a different fontconfig cache format. > > However, flatpak will never parse the entire xml fontconfig file > > format (which isn't even really stable over time), so such a config > > would have to be external in a simpler config format. > > The fontconfig xml format is quite stable and is designed to be > manipulated by tools that do not understand the full contents, hence > using XML and providing a suitable DTD. We recently ran into issues with fontconfig xml parsing errors in chrome when using config files from a newer host fontconfig that were not parsable by the statically linked chrome copy, so it is not perfect. Additionally there is a question of forward compat. If flatpak was to generate a "new" xml option that older fontconfigs were using, then we would not support existing runtimes with older fontconfigs in it. So, what we need is to be able to create sn
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: >> I also fought with fontconfig for about a week when the release >> including them was installed on my machine as firefox would spin >> whenever it found a directory with no fonts. At the time, I felt injured >> by this change. > > Hmm, why was it doing that though? Doesn't seem like it would have to. A .uuid file was added and removed to every directory in the font tree which contained no fonts or sub-directories; (a) the directory mtime was changed, causing the system to re-scan the fonts and then re-create/re-delete the .uuid file (goto a). Somehow this would eventually stablize (I'm not entirely sure how). It was a 'surprising' problem that happened only once in a while and it took several days of searching to locate as the only symptom was that firefox would hang for 'a while' and then start working. Once I ran strace on the process while hung, it was pretty easy to track down, but having fontconfig affecting directory mtimes was not what I expected... > You can't trust directory mtimes in this way. A file in the directory > can be updated without modifying the directory mtime. That is only > modified when you create or remove files. Fontconfig already assumes that all 'interesting' changes result in directory mtime changes -- it doesn't expect files to be changed in place. If you have a system doing this, fontconfig will fail. > However, the second problem is that it puts demands on the *host*, as > it now has to match the layout of the runtimes so the pathnames can > match identically. Well, I was thinking that the runtime would dynamically adapt to the host environment and change the configuration inside to match the host. Is it that flatpak treats fonts very specially, or are there other bits of host data injected into the flatpak runtime in a similar fashion? Essentially, I was imagining the flatpak runtime system would discover the set of fontconfig directories exposed by the host and inject those into the runtime using matching paths. Right now, you inject two fixed paths (/usr/share/fonts and ~/.fonts), but if we really want to deal with systems that put fonts in other places, then presumably flatpak will need to adapt anyways. Once you've discovered where fonts are being stored in the host, having those get mounted in a matching path in the runtime doesn't seem like a huge step to me. It's a fairly simple matter of changing the mount target path and injecting an xml fragment file into /etc/fonts/conf.d > Maybe "weird" setups like nix will run into issues. You might know > this better. Good point; as new distros start experimenting with different filesystem layouts, we will need to be more cautious about assuming fixed paths of any kind. > Yes, but if you update the runtime and /usr/share/fonts-minimal > changed in the new version (but has same mtime), then the stale cache > file in the users homedir will still be used. Ah. Thanks for explaining this. It seems like the only way this could happen is if the cache file within the flatpak was stale and a replacement generated and written to the user's homedir (as the only writable location available). I think that's just a bug in the flatpak generation -- the cache file within the flatpak should always be up-to-date as (I assume) the fontconfig library provided in the flatpak would have been used to generate that cache file. And, a future flatpak shipped without that bug would have a correct cache file, which would presumably be used in preference to the one stored in the home directory? > Initially I imagined uuid files would work somewhat like this. I.e. > you put a .uuid file in the top /usr/share/fonts directory, and all > the subdirectories would hash based on uuid + relative path. That sounds like the same as my 'salt' idea, except you're storing it in the font directories rather than the font configuration. > However, flatpak will never parse the entire xml fontconfig file > format (which isn't even really stable over time), so such a config > would have to be external in a simpler config format. The fontconfig xml format is quite stable and is designed to be manipulated by tools that do not understand the full contents, hence using XML and providing a suitable DTD. However, in practice it's become far easier to use XML snippets instead, so let's figure out how to do that instead. > For example, we could have a /etc/fonts/uuids file which is a simple list > like: > > /usr/share/fonts b81b806a-fb12-4a31-b458-181b1be0ec23 > > And then flatpak could read this and generate one for the sandbox that said: > > /run/host/fonts b81b806a-fb12-4a31-b458-181b1be0ec23 Yeah, moving the 'salt' out of the font directory and into the font configuration seems like a good direction here. And placing it in a separate file (although probably in XML format) seems like a better idea than merging it into the existing elements. > However, the /etc/fonts/uuids file would still not be reproducible, so > i'm not sur
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Fri, Jan 11, 2019 at 7:06 PM Alexander Larsson wrote: > We would have to turn that into a dynamic snippet. But that would be a > problem for pre-existing flatpak binaries which doesn't do this. > Could we instead have a way to modify a previously added dir element? > Or maybe it could handle duplicate tags such that they keep the > original order, but update the mapping? Then we can have both the > static snippet an a dynamic one for later flatpak versions. There was an idea in the previous discussion to have a new element to reset and split up into a file. that may helps you to have an own desired font directories list instead of modifying entire fonts.conf dynamically. this can be put into /etc/fonts/conf.d. so it can be activated no matter how apps has own modified fonts.conf as long as it contains a line to read config files from conf.d. This would makes easier to have a map. > > > Here's an alternative proposal: > > > > Add a per 'dir' element 'salt' value, which is stirred into the > > path name when generating the cache key. You'd generate this > > randomly when the flatpak was created so that all cache keys > > would not collide with entries using a different (or absent) > > salt value. > > > > With this, and my path->key mapping series, we would be able to access > > the existing cache files for external fonts (via the mapping mechanism), as > > well as avoid collisions between internal and external font paths within > > the cache. And we wouldn't have .uuid files (see above). > > I like this in theory. But i don't want to rewrite the entire runtime > font config xml file (its just to fragile). I much prefer if such a > salt could be added by just dropping a file somewhere. I.e. a > fonts.conf snippet that tweaked the salt of a previously defined dir > element. > > So, counter proposal: > > Flatpak generates at startup a file like this in /run/host/fontconf.xml > > /run/host/fonts> > /run/host/user-fonts> > > In the runtime we create at build-time a /etc/fonts/conf.d/ file: > > /usr/share/fonts > # Duplicate this with static versions for old flatpak versions > /run/host/fonts> > # This will (if it exists) override the above with the live values > /run/host/fontconf.xml > > We are basically in full control of existing flatpak runtimes, so this > could easily be added to all of them an propagated out. However, the > new flatpak version that creates the fontconf.xml file will not reach > distros in a while, so the above duplicates the "most likely to be > right" value for /usr/share/fonts for older flatpak versions to pick > up. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Fri, Jan 11, 2019 at 12:17 PM Akira TAGOH wrote: > > On Thu, Jan 10, 2019 at 7:54 PM Alexander Larsson > wrote: > > Here is my proposal: > > > > Make the uuid *generation* optional and manual. Then, when we create > > the flatpak runtime we run fc-cache --make-uuid (or something) to > > generate the uuid files. Then fontconfig would never confuse the > > sandboxed /usr/share/fonts with any other, and since we would get a > > new uuid each time we regenerated the runtime it would correctly pick > > up stale caches when we update the runtime (even with no mtime > > change). > > This would requires the root privilege to create uuid file on > directories where root own though, are you going to have setuid to > flatpak? > or can this be done without it? This would be done when creating the runtime, and at that time we have write rights to /usr. Its not actually owned by root by that time though, we build the image as some other user.
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 10, 2019 at 7:54 PM Alexander Larsson wrote: > Here is my proposal: > > Make the uuid *generation* optional and manual. Then, when we create > the flatpak runtime we run fc-cache --make-uuid (or something) to > generate the uuid files. Then fontconfig would never confuse the > sandboxed /usr/share/fonts with any other, and since we would get a > new uuid each time we regenerated the runtime it would correctly pick > up stale caches when we update the runtime (even with no mtime > change). This would requires the root privilege to create uuid file on directories where root own though, are you going to have setuid to flatpak? or can this be done without it? > > This would make the default installation of fontconfig reproducible, > and it would solve the first problem (don't mix up sandboxed and host > font dirs). It would also let you opt-in to the uuid in other cases > where it makes sense. For instance, you could have a uuid file on a > NFS share or USB drive font dir, so that any caches for it will always > be the same no matter how it happens to be mounted. > > We still wouldn't have a way to reuse host caches which were mounted > in a different way, but if we assume all conflicting directories use > uuids (like they would in the flatpak case), then we could solve this > in a pretty simple way by a config file saying "treat all instances of > /run/host/fonts as /usr/share/fonts", and I could make flatpak > generate such a file. -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 10, 2019 at 9:41 PM Keith Packard wrote: > > Alexander Larsson writes: > > > I'd like to repeat that this is not really flatpak specific as such. > > The issue can happen in multiple cases like nfs mounts, multi-boot > > systems, docker containers, etc. > > Sure, any place where path names are not the same would cause the same > issue. I think your other examples are unlikely to exhibit this in > practice though -- when NFS is used for system-level sharing, the normal > configuration tooling goes to extreme measures to ensure that pathnames > are the same across all systems. Yeah, practice seems to indicate that this is not presently a problem. But as the world moves more and more towards containerization in many places and other tools like snappy also target the desktop it is imho important to at least consider it. > > Flatpak has an additional weakness here, which is that we don't store > > mtimes (to maximize content sharing abilities the mtime is not part of > > the content addressing). This means the mtime can't be used to detect > > a stale cache, so we use the uuid to detect such changes. > > I don't understand this -- the only way UUID could be used to trigger a > stale cache is by being missing. Otherwise, I thought mtimes were still > the only method used to know if the cache file was out of date? Lemme describe the expected mode of operation for fontconfig in a flatpak: First of all, /usr comes from the runtime, and it is immutable. We only ever modify the runtime by creating a new version of it in a separate place and pointing new sandbox instances at the new directory, and /usr is mounted read-only in the sandbox. Since all the files in the runtime are maintained by ostree they all have a mtime of 0. The runtime contains /usr/share/fonts with uuid files. These uuid files are re-generated each time the runtime is rebuilt, so an updated runtime will get a new uuid. It also contains a pre-generated font caches for the fonts in the runtime based on the uuid. In the sandbox $XDG_CACHE_HOME is set to ~/.var/app/$APPID/cache, and this is the first writable cache directory in the fontconfig config, so all caches will be per-app. However, if you update the app (say from 1.0 to 2.0) the same cache dir will be used, just not between two completely different apps). In the normal case we will never need to generate any caches for the runtime fonts. However, in weird cases (maybe someone pulled in a fontconfig that has a different font cache format or whatever) the cache generated will be based on the uuid, so if you update the runtime you will get a new uuid and the old cache will not be used (if we did we could not detect it was stale using the mtime). Host fonts are mounted in /run/host/[user-]fonts, and host caches in /run/host/[user-]fonts-cache, and the runtime has a config snippet to add these. If the host has uuid for the dirs, then we can pick up the host caches from the host cache dirs. If not, then the app will on the first run re-generate the caches for /run/host/fonts and store it in the per-app cache dir. These directories are just bind-mounts of the host stuff, so normally they have proper mtimes and we can detect stale caches fine, its just slow the first time you start the app. However, on an ostree-using base system (such as fedora silverblue) the mtime issue may actually cause us to use stale caches here. > > I realize this is not what the reproducible builds project wants, but > > it is what the UUID was added for. > > The UUID files also violate one of the principle design goals I adopted > for fontconfig when cache files were moved to a separate directory -- > never touch the font directories. I think an optional approach fits with this. I.e. when it makes sense, allow the font dirs to carry some identity, but don't touch them by default. > I also fought with fontconfig for about a week when the release > including them was installed on my machine as firefox would spin > whenever it found a directory with no fonts. At the time, I felt injured > by this change. Hmm, why was it doing that though? Doesn't seem like it would have to. > Hrm. Could some combination of mtime checking and content addressing > work? Consider a .uuid file generated from a hash of the directory > contents. If the .uuid file is older than the directory, you could > regenerate it reasonably quickly by hashing the directory contents. That > would be faster than re-scanning all of the fonts in the directory at > least. You can't trust directory mtimes in this way. A file in the directory can be updated without modifying the directory mtime. That is only modified when you create or remove files. > > I'm willing to make *some* changes to flatpak, but I'm not sure this > > is the right approach. First of all it just looks at a tiny subset of > > the problem (only flatpak, and only one directory). > > Changing flatpak to *always* mount host directories at the same place in > the container sh
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Alexander Larsson writes: > I'd like to repeat that this is not really flatpak specific as such. > The issue can happen in multiple cases like nfs mounts, multi-boot > systems, docker containers, etc. Sure, any place where path names are not the same would cause the same issue. I think your other examples are unlikely to exhibit this in practice though -- when NFS is used for system-level sharing, the normal configuration tooling goes to extreme measures to ensure that pathnames are the same across all systems. Given that no-one has ever complained about this issue until flatpak came along, I'd like to suggest that flatpak is somewhat unique in its requirements here. Which means that we should feel free to find solutions involving changes within flatpak, instead of attempting to fix this solely within fontconfig. I'm not yet certain that we couldn't come up with a fix that didn't affect flatpak, but we've got some strong indications that this isn't possible. > Flatpak has an additional weakness here, which is that we don't store > mtimes (to maximize content sharing abilities the mtime is not part of > the content addressing). This means the mtime can't be used to detect > a stale cache, so we use the uuid to detect such changes. I don't understand this -- the only way UUID could be used to trigger a stale cache is by being missing. Otherwise, I thought mtimes were still the only method used to know if the cache file was out of date? > I think this misrepresents what the UUID is for. The UUID represents > a uniqe identifier for the *location*, not the contents, and the goal > is to make it independent on how you found the directory. If you add > fonts to that directory, then you're supposed to keep the UUID because > you want to regenerate the same cache for the new content. I may have mis-represented the UUID idea in my email, but please accept that I do understand that they are designed to be a path-independent identifier for each directory. The problem is not the UUID file itself, but in how the key it contains is generated. To generate a reproducible UUID file, you have to have a deterministic algorithm using only data in the source files and toolchain along with the font directory path name. But that means that /usr/share/fonts will necessarily have the same key on all systems. So we end with conflicting requirements -- reproducibility requires that the font cache key for '/usr/share/fonts' be the same everywhere, but flatpak requires that they differ so that the host /usr/share/fonts have a different cache key than the flatpak /usr/share/fonts. > I realize this is not what the reproducible builds project wants, but > it is what the UUID was added for. The UUID files also violate one of the principle design goals I adopted for fontconfig when cache files were moved to a separate directory -- never touch the font directories. I also fought with fontconfig for about a week when the release including them was installed on my machine as firefox would spin whenever it found a directory with no fonts. At the time, I felt injured by this change. I add this here just to let you know that I am biased against any design including UUID files and would prefer a solution which eliminates them. > If the UUID really *was* content addressed, then it would change each > time some font was added, and old font caches would become stale (and > reaped via some other way like mtimes). In this case the fact that > caches between the sandbox and the hosts collide is not even a > problem, since the cached data is identical and could be shared. The > problem is rather that the font directory is mutable, and if it > changes without immediately updating the uuid you run into issues. Hrm. Could some combination of mtime checking and content addressing work? Consider a .uuid file generated from a hash of the directory contents. If the .uuid file is older than the directory, you could regenerate it reasonably quickly by hashing the directory contents. That would be faster than re-scanning all of the fonts in the directory at least. I think this would assume that font file names were globally unique, which may not be reasonable. It also ignores the question of reaping stale cache files; presumably fc-cache could do that somehow... > I'm willing to make *some* changes to flatpak, but I'm not sure this > is the right approach. First of all it just looks at a tiny subset of > the problem (only flatpak, and only one directory). Changing flatpak to *always* mount host directories at the same place in the container should solve the problem for all directories. > Secondly, it is likely to run into issues having non-standard > paths. For example, the fedora flatpak runtime is created from the > standard fedora rpms, so it will have to be tweaked post install, and > its possible that some code hard-coded > /usr/share/fonts/some-specific-font which is in the app but not the > host.. This is not an entirely theoretical i
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > Indeed, that would be able to accomplish both with the minimal efforts > for us at least. though they might came up with this but they didn't > do it that way. so there might be some reason why they didn't do so. > packaging issue perhaps? flatpak appears to change many pathnames used to access host files. I don't know if there are other subsystems affected by these changes, perhaps we'll see more in the future though. > I can't figure out completely but, fontconfig may needs to deal with > different namespaces in a cache filename to avoid a collision between > host and sandboxes. dunno if we may see different state in the future > but it might be represented as a depth in a filename to make it > different like 0:-le64.cache- for host and > 1:-le64.cache- for a sandbox. > > we could increase a depth > for a child in sandbox as needed, anyway. > We can mix up caches that is located at the same place then. the last > missing piece would be to map them to the right place. flatpaks should > knows where they mounted directories to. they can create a map table > with proper parent depth and current depth I think. That would require customizing the contents of a flatpak on install, or perhaps this could be done when the flatpak was run? > I may be missing something so this might not work though... This seems to extend the change I proposed; which provides an indirection between the actual filename and the font config cache database key. Instead of just mapping sandbox names to external names (which results in collisions), we also add some 'salt' to the sandbox names to perturb the generated key for internal paths. Let's look at some examples: sandbox prefix cache prefix -- -- /run/host/fonts /usr/share/fonts /run/host/user-fonts/home/keithp/.fonts / sandbox-depth-1/ 'sandbox-depth-1' is the "salt" added to the cache keys for paths within the sandbox to ensure they do not collide with cache keys for paths outside of the sandbox. You'd add all mounted file systems to this list so that fonts found anywhere outside the sandbox would generate cache keys using the names from outside the sandbox. If you ran another sandbox *inside* this sandbox, you'd have another level of indirection: /run/host/fonts sandbox-depth-1/usr/share/fonts / sandbox-depth-1/ (assuming that the sandbox didn't manage to mount the "real" system fonts inside the sandbox somewhere). This configuration file would be generated by flatpak at runtime. -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Wed, Jan 9, 2019 at 8:29 PM Keith Packard wrote: > > Akira TAGOH writes: > > > As of the discussion on the list, Keith's changes doesn't address the > > original purpose - allow sharing caches on bind-mounts in flatpaks. > > particularly for the case where flatpaks uses same location in > > sandbox. > > I'm probably forgetting a bunch of context here, but I think the problem > was that flatpaks have /run/host/fonts pointing to the "real" > /usr/share/fonts and then a separate /usr/share/fonts of their own with > a small set of fonts, and so you end up with collisions in the cache > file namespace as both directories end up generating the same cache file > name. Yes, and there is two aspects of this problems with this: A sandbox has something in /usr/share/fonts that isn't the same as what is on the host, but the cache is shared, so we pick up host cache for the sandbox directory (or the reverse), causing misrendering of fonts and whatnot. A version of the host /usr/share/fonts is available to the sandbox under a different pathname, causing the host cache to not be used and thus have to be regenerated, which is slow first time the app is started. I'd like to repeat that this is not really flatpak specific as such. The issue can happen in multiple cases like nfs mounts, multi-boot systems, docker containers, etc. Also, the focus on /usr/share/fonts here is mainly for illustration, the same issue could happen with any other path. For instance, separate containers with shared $HOME would have its font path (whatever it is as long as it is the same) conflict between the containers. Flatpak has an additional weakness here, which is that we don't store mtimes (to maximize content sharing abilities the mtime is not part of the content addressing). This means the mtime can't be used to detect a stale cache, so we use the uuid to detect such changes. > Making the build reproducible means having all content generated > deterministically based only on the source package and toolchain. The > current UUID files are generated randomly making them > non-deterministic. > > For the font cache, making it reproducible requires that the keys > mapping directories to cache filenames be the same each time the cache > is built. This means we cannot use the current randomly generated UUID > values and also have a reproducible system. I think this misrepresents what the UUID is for. The UUID represents a uniqe identifier for the *location*, not the contents, and the goal is to make it independent on how you found the directory. If you add fonts to that directory, then you're supposed to keep the UUID because you want to regenerate the same cache for the new content. I realize this is not what the reproducible builds project wants, but it is what the UUID was added for. > I considered whether we might provide a mechanism to generate UUID > values deterministically for purposes of packaging. However, this would > mean that we couldn't use these same packages when creating a flatpak as > the deterministic UUID values would collide if those same packages were > used in the outer system. I think this furthers the misunderstanding from above, but lets continue this idea. If the UUID really *was* content addressed, then it would change each time some font was added, and old font caches would become stale (and reaped via some other way like mtimes). In this case the fact that caches between the sandbox and the hosts collide is not even a problem, since the cached data is identical and could be shared. The problem is rather that the font directory is mutable, and if it changes without immediately updating the uuid you run into issues. > Without deterministic UUID values, I'm left with the feeling that > our only available solutions involve changing how flatpaks reference > fonts. > > If we agree that a solution to this involves changing the flatpak > mechanism, I'd like to suggest that the most straightforward fix for the > overall system would be to expose the external fonts using the external > path names -- bind mounting the external /usr/share/fonts as > /usr/share/fonts within the flatpak, and creating a new > /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the > flatpak itself. With this change, we can simply delete the UUID code > from fontconfig and go back to using global font paths as keys to the > font cache database. I'm willing to make *some* changes to flatpak, but I'm not sure this is the right approach. First of all it just looks at a tiny subset of the problem (only flatpak, and only one directory). Secondly, it is likely to run into issues having non-standard paths. For example, the fedora flatpak runtime is created from the standard fedora rpms, so it will have to be tweaked post install, and its possible that some code hard-coded /usr/share/fonts/some-specific-font which is in the app but not the host.. Also, we'll be guaranteeing that caches for /usr/share/fonts and /usr/share/
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
On Thu, Jan 10, 2019 at 4:29 AM Keith Packard wrote: > I'm probably forgetting a bunch of context here, but I think the problem > was that flatpaks have /run/host/fonts pointing to the "real" > /usr/share/fonts and then a separate /usr/share/fonts of their own with > a small set of fonts, and so you end up with collisions in the cache > file namespace as both directories end up generating the same cache file > name. That is correct. > Making the build reproducible means having all content generated > deterministically based only on the source package and toolchain. The > current UUID files are generated randomly making them > non-deterministic. Hm, I see. > For the font cache, making it reproducible requires that the keys > mapping directories to cache filenames be the same each time the cache > is built. This means we cannot use the current randomly generated UUID > values and also have a reproducible system. Right. > I considered whether we might provide a mechanism to generate UUID > values deterministically for purposes of packaging. However, this would > mean that we couldn't use these same packages when creating a flatpak as > the deterministic UUID values would collide if those same packages were > used in the outer system. > > Without deterministic UUID values, I'm left with the feeling that > our only available solutions involve changing how flatpaks reference > fonts. > > If we agree that a solution to this involves changing the flatpak > mechanism, I'd like to suggest that the most straightforward fix for the > overall system would be to expose the external fonts using the external > path names -- bind mounting the external /usr/share/fonts as > /usr/share/fonts within the flatpak, and creating a new > /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the > flatpak itself. With this change, we can simply delete the UUID code > from fontconfig and go back to using global font paths as keys to the > font cache database. Indeed, that would be able to accomplish both with the minimal efforts for us at least. though they might came up with this but they didn't do it that way. so there might be some reason why they didn't do so. packaging issue perhaps? > I'd love to hear about alternative ideas which might lead to solutions > that make builds involving fontconfig reproducible. I'd be happy to take > even vague hints at this point; all I've got at this point are a > collection of dead ends. I can't figure out completely but, fontconfig may needs to deal with different namespaces in a cache filename to avoid a collision between host and sandboxes. dunno if we may see different state in the future but it might be represented as a depth in a filename to make it different like 0:-le64.cache- for host and 1:-le64.cache- for a sandbox. we could increase a depth for a child in sandbox as needed, anyway. We can mix up caches that is located at the same place then. the last missing piece would be to map them to the right place. flatpaks should knows where they mounted directories to. they can create a map table with proper parent depth and current depth I think. I may be missing something so this might not work though... > (Also, if I've missed or forgotten something relevant, please let me > know; I've re-read a lot of stuff while writing this, but surely > something escaped my notice). > > -- > -keith -- Akira TAGOH
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Akira TAGOH writes: > As of the discussion on the list, Keith's changes doesn't address the > original purpose - allow sharing caches on bind-mounts in flatpaks. > particularly for the case where flatpaks uses same location in > sandbox. I'm probably forgetting a bunch of context here, but I think the problem was that flatpaks have /run/host/fonts pointing to the "real" /usr/share/fonts and then a separate /usr/share/fonts of their own with a small set of fonts, and so you end up with collisions in the cache file namespace as both directories end up generating the same cache file name. > this is the reason why it can't be merged into master. So just > reverting the change that removes .uuid file only when a directory has > empty is done in master so far. if you have .uuid file prior to run > fc-cache, your issue could be worked around at this moment. for more > details, you can check what Alex Larsson said on this list. Making the build reproducible means having all content generated deterministically based only on the source package and toolchain. The current UUID files are generated randomly making them non-deterministic. For the font cache, making it reproducible requires that the keys mapping directories to cache filenames be the same each time the cache is built. This means we cannot use the current randomly generated UUID values and also have a reproducible system. I considered whether we might provide a mechanism to generate UUID values deterministically for purposes of packaging. However, this would mean that we couldn't use these same packages when creating a flatpak as the deterministic UUID values would collide if those same packages were used in the outer system. Without deterministic UUID values, I'm left with the feeling that our only available solutions involve changing how flatpaks reference fonts. If we agree that a solution to this involves changing the flatpak mechanism, I'd like to suggest that the most straightforward fix for the overall system would be to expose the external fonts using the external path names -- bind mounting the external /usr/share/fonts as /usr/share/fonts within the flatpak, and creating a new /usr/share/fonts-minimal (or whatever) to hold the fonts provided by the flatpak itself. With this change, we can simply delete the UUID code from fontconfig and go back to using global font paths as keys to the font cache database. I'd love to hear about alternative ideas which might lead to solutions that make builds involving fontconfig reproducible. I'd be happy to take even vague hints at this point; all I've got at this point are a collection of dead ends. (Also, if I've missed or forgotten something relevant, please let me know; I've re-read a lot of stuff while writing this, but surely something escaped my notice). -- -keith signature.asc Description: PGP signature
Bug#864082: [Fontconfig] Next steps for a reproducible Fontconfig?
Thank you for reminding and sorry for late response on this. I was on long holidays. On Fri, Jan 4, 2019 at 9:29 PM Chris Lamb wrote: > Since then, I don't believe there has been any review of this > branch both in the sense of the code itself but also in terms of > the architectural changes that it implies. I might be able to help > on the former front but without knowing the "lore" of Fontconfig I > simply cannot comment on the latter parts. As of the discussion on the list, Keith's changes doesn't address the original purpose - allow sharing caches on bind-mounts in flatpaks. particularly for the case where flatpaks uses same location in sandbox. this is the reason why it can't be merged into master. So just reverting the change that removes .uuid file only when a directory has empty is done in master so far. if you have .uuid file prior to run fc-cache, your issue could be worked around at this moment. for more details, you can check what Alex Larsson said on this list. > > Anyway, I'd love to get this resolved once and for all ideally get > it into Debian buster which is about to start "freezing" very > soon. > > What would be the best way for me to help here? Can I entreat Keith > to merge his branch? I can put some cycles onto this issue if that is > of some assistance. > > > Best wishes, > > -- > Chris Lamb > chris-lamb.co.uk / @lolamby > ___ > Fontconfig mailing list > fontcon...@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/fontconfig -- Akira TAGOH
