Hi, TL;DR: we've started switching to the new distfile layout. Mirrors will notice the presence of symlinks corresponding to the dual-layout throughout the transitional period. At some point in the future we will have to switch the direction of symlinks, so please enable --hard-links (-H) in your rsync calls if you haven't already.
I am pleased to announce that after initial internal testing, we've successfully deployed the new distfile mirror layout. The new layout splits distfiles into 256 directories, solving the problem of having 60000+ files in a single directory. However, the mirrors will really start benefiting from this only once the old layout is gone. To reduce the impact on mirrors to the bare minimum, we've chosen to go with a hybrid symlink+hardlink approach. Throughout the majority of transitional period, the compatibility between two layouts will be provided by symlinks. This means the total number of file entries on the mirror will double but the additional bandwidth and disk space use will be limited to the possible minimum. All newly downloaded files already use the new layout for regular files, and create symlinks in the old layout. However, existing files are symlinked the other way. Before the old layout is removed, we will have to reverse the direction of those symlinks. For the time being, we're just going to let normal distfile rotation (i.e. fetching of new files and removal of old files) take care of reducing the number of links needing transition. Nevertheless, at some point in the future we will have to start converting the remaining files. We will be using hardlinks to make it possible for the conversion to occur without refetching everything. Therefore, once again we urge you to enable the --hard-links (-H) option to rsync if you haven't done that already. According to my current plan, we will use a shifting transition window approach to limit the impact both on mirrors and on master mirror. If you have any questions, please do not hesitate to ask. -- Best regards, Michał Górny
signature.asc
Description: This is a digitally signed message part
