Re: [arch-dev-public] Restricting ability to post news items
On 1/5/20 11:04 PM, Morten Linderud via arch-dev-public wrote: > On Mon, Jan 06, 2020 at 07:53:21AM +1000, Allan McRae via arch-dev-public > wrote: >> Following the roll out of the base metapackage, and its poorly written >> news post, we agreed that all new posts should have a draft posted to >> arch-dev-public. > > Wait, where was this agreed? I heard something about all drafts should be > headed > for arch-dev, but this hasn't been announced nor discussed anywhere. > > Are the TUs missing from the loop here? > I feel the same. I got prompted to write (and publish) the post by grazzolini & others. I saw that some people were discussing this, but it wasn't clear whether this was now established, especially given that i had asked several team members for confirmation before posting this - who gave me the OK. I think you're jumping the gun here, Allan. If this rule is supposed to be applied, it needs to be clearly announced. I would not have deliberately bypassed this... -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] Adding a "posix" metapackage
On 1/3/20 5:35 AM, Eli Schwartz via arch-dev-public wrote:
> After a bit of research work and making sure one or two things have been
> properly packaged, I've developed a PKGBUILD which ensures that a system
> has the POSIX shell and utilities (XCU) section installed. I believe
> this is an interesting thing to track, and people will want to know they
> have it installed... in aid of this, I've gotten two major holdouts
> packaged a while back -- pax (thanks to dbermond) and ncompress.
>
> I'd like to add this package to community, although given it's never
> been in the AUR before, it's never had AUR votes...
>
> One of the advantages of having this metapackage is that someone can,
> while installing arch, `pacman -S posix-user-portability` and get
> essentially everything one would expect to have available on a unix-like
> platform, most notably, the interesting parts of the base group that no
> longer have an equivalent. i.e. man-db, vi, patch, diffutils, ed.
>
> I've only included XCU for now, because the system interfaces and
> headers are a bit out of scope for me to package and replace in the
> event that they'd be missing anything... and also because I'm mainly
> interested in the POSIX toolset itself. That being said, I'd certainly
> be open to suggested improvements, should anyone have recommendations
> for expanding the scope.
>
> Thoughts?
>
I think it's a great idea, i definitely see myself using
posix{,-user-portability} once it becomes available.
--
Rob (coderobe)
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
signature.asc
Description: OpenPGP digital signature
[arch-dev-public] zstd rollout complete
Hello everyone, We have just tested and released everything necessary for zstd packages in our repos. Right now, the updated devtools is still in [testing], but our preliminary testing shows that nothing immediately combusted on release, so I'm confident. Once it has left testing, packages built with devtools will automatically use zstd with the correct options. There is no intervention necessary, this should be an essentially transparent process. Happy packaging and greetings from 36c3. -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 12/14/19 7:58 PM, Robin Broda via arch-dev-public wrote: > Hello again, > > We are on our way to getting zstd merged. > > The planned merge window for this is **2019/12/27** at around 20:00 > Europe/Berlin time. > Several team members will be meeting at the 36c3, so we figured this is a > good opportunity to do this, > as it makes sure that several people are aware, ready, and together for this > - should anything immediately explode. > > This has been ACK'd by anthraxx. Foxboron is also aware. > I have been mentioning this on IRC, and discussed it with anthraxx via e-mail. > This mail serves to formalize the plans. > > **There is one hard requirement on the merge happening:** > > All critical Arch Linux production infrastructure needs to be ready. > The known-unsolved problems are labelled below, please see to them if any of > you can. > I will also take a look myself, but I do not have much involvement with these > parts of the puzzle, > so I would really appreciate someone with more in-depth knowledge taking a > look. > > If the known issues are solved and nothing major comes up, the deployment > will happen as planned. > All known issues have been dealt with. We are waiting on the patches to land, thus: - https://www.archlinux.org/packages/extra/any/namcap/ being upgraded to 3.2.9 - https://github.com/archlinux/archivetools/pull/8 being merged - https://github.com/coderobe/infrastructure/commit/7155521ed6f823ac6c6d06d1b9dcdb8a5165c417 being picked into infrastructure.git - this *depends on* the archivetools patch - archive-cleaner is not mission-critical, it does not need to be patched before the scheduled deployment - Foxboron is working on a patch These can all be dealt with during the deployment if they do not happen beforehand anyways. This means the deployment will happen as planned. -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 12/14/19 7:58 PM, Robin Broda via arch-dev-public wrote: > On 12/8/19 1:39 PM, Robin Broda via arch-dev-public wrote: >> What still needs to be done: >> - infrastructure: >> - nginx rewrite rules hardcode xz PKGEXT >> https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/nginx.d.conf.j2#L32-L35 >> - archive config(?) hardcodes PKGEXT >> https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/archive.conf.j2#L15 > > Unsolved: This needs fixing. I would *really* like someone from devtools to > take a look at this, but if unsolved i will also attempt to fix it. > Gah, s/devtools/devops/ -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
Hello again, We are on our way to getting zstd merged. The planned merge window for this is **2019/12/27** at around 20:00 Europe/Berlin time. Several team members will be meeting at the 36c3, so we figured this is a good opportunity to do this, as it makes sure that several people are aware, ready, and together for this - should anything immediately explode. This has been ACK'd by anthraxx. Foxboron is also aware. I have been mentioning this on IRC, and discussed it with anthraxx via e-mail. This mail serves to formalize the plans. **There is one hard requirement on the merge happening:** All critical Arch Linux production infrastructure needs to be ready. The known-unsolved problems are labelled below, please see to them if any of you can. I will also take a look myself, but I do not have much involvement with these parts of the puzzle, so I would really appreciate someone with more in-depth knowledge taking a look. If the known issues are solved and nothing major comes up, the deployment will happen as planned. On 12/8/19 1:39 PM, Robin Broda via arch-dev-public wrote: > What still needs to be done: > - infrastructure: > - nginx rewrite rules hardcode xz PKGEXT > https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/nginx.d.conf.j2#L32-L35 > - archive config(?) hardcodes PKGEXT > https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/archive.conf.j2#L15 Unsolved: This needs fixing. I would *really* like someone from devtools to take a look at this, but if unsolved i will also attempt to fix it. > - archivetools: https://github.com/archlinux/archivetools/issues/6 Unsolved: This needs fixing. If nobody else gets to it, I'll see to it myself. > - namcap? some files refer to a hardcoded .xz, though jelle concluded that > this is irrelevant Solved: This is indeed irrelevant (see Eli's reply) > - srcpac (is this still used?) Solved: This is indeed a dead project (see Eli's reply) > - kde-build hardcodes pkg.tar.xz > https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064cb87d427ccb3bdf954f127c65f7#n35 Solved: arojas said this will be fixed after zstd is deployed. This is also non-critical. > - devtools: Solved: Patches ready, will be merged with the switcharoo. I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best. Please -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] Proposal: Build a ruleset for new packages and package quality
On 12/12/19 1:21 PM, Christian Rebischke via arch-dev-public wrote: > 8.2 I can't find any list about punishments for violations of these > rules. > > Best regards, > Chris > Do we have such lists of punishment in any of our other guidelines? I think we generally assume that other team members are acting in good faith, if someone is doing something by mistake - point them towards it and I'm sure you can figure it out. All without threatening someone with /punishments/. -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
Hello everyone, Now that Zstd 1.4.4 has been out, and released into our repos as well, i think it's time for a new status report on this. I re-ran the benchmarks with the new zstd, and we are hitting marginally better times in compression & decompression in all scenarios. In the past few weeks, other team members and I have taken a look at our own projects, and what would be needed to finally transition to zstd. Notable progress: - A news post was made by eworm, indicating that users should upgrade if they haven't done so since September 2018 (https://www.archlinux.org/news/required-update-to-recent-libarchive/) - dbscripts has been updated by eschwartz to support zstd What still needs to be done: - infrastructure: - nginx rewrite rules hardcode xz PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/nginx.d.conf.j2#L32-L35 - archive config(?) hardcodes PKGEXT https://github.com/archlinux/infrastructure/blob/7d0ad69030982875f862bc4916629ac3a74d1c49/roles/archive/templates/archive.conf.j2#L15 - potentially more things, someone from devops should take a look - archivetools: https://github.com/archlinux/archivetools/issues/6 - namcap? some files refer to a hardcoded .xz, though jelle concluded that this is irrelevant - someone with namcap knowledge should look into this - srcpac (is this still used?) - hardcodes 'pkg.tar.?z' - kde-build hardcodes pkg.tar.xz https://git.archlinux.org/kde-build.git/tree/build-packages?id=cda04698f4064cb87d427ccb3bdf954f127c65f7#n35 - devtools: - hardcodes 'pkg.tar?(.?z)' https://git.archlinux.org/devtools.git/tree/lib/common.sh?id=2c611d20bdd04feae6ab32a7ef8947aeb4118604#n155 - more than one occurrence of this iirc There might be things I've missed. I encourage every Arch Linux project maintainer to check their own code for hardcoded xz extensions, you probably know your code best. As soon as these things are out of the way, we can proceed with the proposal. The changeset proposal remains on these settings: PKGEXT='.pkg.tar.zst' COMPRESSZST=(zstd -c -T0 --ultra -20 -) -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] Reproducible builds progress and the upcoming rebuild of [core]
On 11/13/19 3:46 AM, Allan McRae via arch-dev-public wrote: > To keep this momentum going, it would be great to rebuild every package > in [core] using makepkg from pacman-5.2+. That way we can test which > packages are actually reproducible and work towards fixing those that > are not. So be prepared for almost the entire repo to hit [testing] > soon, and get your sign-off shoes on! > Hmm, what do you think about postponing this until we roll out zstd, which should be somewhat soon? As i don't think we're gonna rebuild everything for zstd, this would be a great opportunity to get both of these things done at once. > Again, a huge congrats to our reproducible builds team. This has been a > massive amount of work! > !!! > Allan > -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] Retiring as a Trusted User
On 10/2/19 11:10 AM, Florian Pritz via arch-dev-public wrote:
> That said, I'd like to orphan some of them too. If anyone is interested
> in taking over one of these packages (and its dependencies), please feel
> free to adopt it and tell me so that I can orphan it.
>
> filezilla libfilezilla
>
> Florian
>
Hi Florian - I'd happily adopt {,lib}filezilla.
Hope we get to keep you as a dev for a while :)
--
Rob (coderobe)
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
signature.asc
Description: OpenPGP digital signature
Re: [arch-dev-public] Dropping packages to AUR
On 9/22/19 10:12 PM, Alad Wenter via arch-dev-public wrote: > Hi, > > As I no longer use them, I will drop the following packages to AUR > unless someone is interested in picking them up: > > * usbview > > If you are interested in any of the above packages, please let me know > in a reasonable time period (say, two weeks.) I'll take on usbview! -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] A contrib repository
On 9/11/19 11:10 PM, Morten Linderud via arch-dev-public wrote: > svenstaro also wrote a `offline-build` tool that went into contrib. Eli > polished > it and moved it to devtools after some work. offload-build, not offline-build, fwiw! -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
Hello again, after archange and Baptiste mentioned that the numbers look a little odd, I took some more time and re-ran the tests with additional parameters. Most notably, this includes -T2 - to show behavior on lower-spec machines, and it fixes the higher compression levels by appending --ultra. Here are the new results: Compressor Package Name Size (MiB) Comp. Size (MiB) Ratio Time (mm:ss) Max. RSS in MiB Decomp. Time (mm:ss) Decomp. RSS in MiB xz -c -z - cuda 3038,58 1316,93 43,34% 19:03.44 95,321:19.74 10,18 zstd -c -T2 -18 - cuda 3038,58 1375,41 45,26% 7:10.76 373,53 0:04.49 10,70 zstd -c -T0 -18 - cuda 3038,58 1375,41 45,26% 1:17.23 2646,19 0:04.41 10,76 zstd -c -T2 -19 - cuda 3038,58 1371,94 45,15% 9:08.09 420,43 0:04.43 10,68 zstd -c -T0 -19 - cuda 3038,58 1371,94 45,15% 1:34.74 3415,77 0:04.51 10,75 zstd -c -T2 --ultra -20 - cuda 3038,58 1286,91 42,35% 10:05.19 1255,64 0:04.46 34,78 zstd -c -T0 --ultra -20 - cuda 3038,58 1286,91 42,35% 1:57.94 8192,42 0:04.43 34,76 zstd -c -T2 --ultra -21 - cuda 3038,58 1141,94 37,58% 10:37.84 2404,56 0:04.11 66,73 zstd -c -T0 --ultra -21 - cuda 3038,58 1141,94 37,58% 2:58.45 8035,52 0:04.08 66,77 xz -c -z - gcc135,54 33,11 24,43% 0:54.54 95,340:02.59 10,11 zstd -c -T2 -18 - gcc135,54 35,87 26,47% 0:23.39 255,13 0:00.27 10,76 zstd -c -T0 -18 - gcc135,54 35,87 26,47% 0:12.42 419,35 0:00.24 10,81 zstd -c -T2 -19 - gcc135,54 35,66 26,31% 0:30.34 319,03 0:00.24 10,45 zstd -c -T0 -19 - gcc135,54 35,66 26,31% 0:16.07 579,00 0:00.24 10,73 zstd -c -T2 --ultra -20 - gcc135,54 24,38 17,99% 0:51.69 484,32 0:00.20 34,63 zstd -c -T0 --ultra -20 - gcc135,54 24,38 17,99% 0:51.79 484,66 0:00.20 34,73 zstd -c -T2 --ultra -21 - gcc135,54 22,89 16,89% 1:10.22 481,77 0:00.22 66,71 zstd -c -T0 --ultra -21 - gcc135,54 22,89 16,89% 1:10.39 482,17 0:00.21 66,65 xz -c -z - go 484,10 122,1025,22% 3:19.11 95,350:08.78 10,16 zstd -c -T2 -18 - go 484,10 132,6927,41% 1:20.42 292,36 0:00.78 10,75 zstd -c -T0 -18 - go 484,10 132,6927,41% 0:15.42 1402,87 0:00.78 10,79 zstd -c -T2 -19 - go 484,10 131,8427,23% 1:46.85 352,77 0:00.79 10,75 zstd -c -T0 -19 - go 484,10 131,8427,23% 0:20.13 1914,13 0:00.80 10,75 zstd -c -T2 --ultra -20 - go 484,10 121,8725,17% 1:58.00 879,29 0:00.83 34,68 zstd -c -T0 --ultra -20 - go 484,10 121,8725,17% 1:07.37 1252,75 0:00.84 34,71 zstd -c -T2 --ultra -21 - go 484,10 112,1823,17% 2:09.79 1240,84 0:00.82 66,73 zstd -c -T0 --ultra -21 - go 484,10 112,1823,17% 2:09.70 1241,08 0:00.81 66,80 xz -c -z - intellij-* 772,46 384,3749,76% 4:53.01 95,310:28.69 10,18 zstd -c -T2 -18 - intellij-* 772,46 392,4450,80% 1:51.91 342,91 0:00.94 10,71 zstd -c -T0 -18 - intellij-* 772,46 392,4450,80% 0:20.50 2341,05 0:00.93 10,70 zstd -c -T2 -19 - intellij-* 772,46 391,0450,62% 2:40.44 407,06 0:00.94 10,82 zstd -c -T0 -19 - intellij-* 772,46 391,0450,62% 0:28.37 3107,88 0:00.95 10,76 zstd -c -T2 --ultra -20 -
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 3/25/19 1:38 AM, Evangelos Foutras via arch-dev-public wrote: > On Mon, 25 Mar 2019 at 01:22, Jan Alexander Steffens via > arch-dev-public wrote: >> >> When we implement this, I would say we go with "zstd -c -T0 -" in >> pacman's makepkg.conf and "zstd -C -T0 -18 -" in the configs shipped >> with devtools. >> >> I think users that build their own local packages are more likely to >> benefit from fast compression. Anyone building with makechrootpkg for >> distribution gets the high compression level. > > That's actually really smart! We can also leave out "-T0" since the > default compression level is very fast anyway. Plus, it's already > implemented in this way in pacman git so we don't have to touch > anything there. (But, if it were to be multithreaded there as well, I > would replace zstd with zstdmt; same for devtools.) That's why the subject line is prefixed with '(devtools)' :) What's unclear to me is the difference between zstd -T0 and zstdmt, however. > > As far as compression level goes, I believe we should select the > highest one that doesn't have increased memory requirements during > decompression. So that would be -19. Robin makes a good case about > -18; looking at upstream's "Compression Speed vs Ratio" graph [1] I > would say -18 is preferable to -19 if we are concerned about > compression speed and memory usage (my totally unscientific > measurements show a 25% memory increase and 20% speed decrease going > from -18 to -19 when using -T4). That said, I might still opt for -19 > due to the slightly higher compression ratio; memory usage isn't too > big of an issue and the slower speed is mitigated by multithreading > (i.e.: it will still be much faster than xz). > I do think that at -19+, memory usage becomes a bigger issue. The difference between -18 and -19 on cuda is almost a gigabyte! While not really a problem for our beefy build boxes, some 4- or even 8-GB developer machines could really suffer from such an incline in memory usage. Thus i stand by -18 being the more sensible choice. Rob signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 3/25/19 1:33 AM, Gaetan Bisson via arch-dev-public wrote: > - Prepare a static build of libarchive-3.3.3 compressed with xz and > write a wiki page with detailed instructions on how to manually switch > from an old system (for users who might want to switch even later). > Users that may wish to upgrade at an even later point can always do an upgrade using the archive, without requiring any sort of static builds or other hacks. Rob signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 3/25/19 12:22 AM, Andrew Gregory wrote: > On 03/25/19 at 12:15am, Robin Broda via arch-dev-public wrote: >> On 3/24/19 11:20 PM, Evangelos Foutras via arch-dev-public wrote: >>> On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public >>> wrote: >>>> >>>> We need to assume every system has a copy of pacman-5.2+ before we can >>>> switch packages to zstd. >>> >>> Why is pacman support needed here? I can already install .zstd >>> packages using pacman 5.1.3. >>> >>> The crucial part seems to be libarchive support, which was added in >>> v3.3.3 (~ September 2018). >>> >> >> Yes, installing zstd packages works - the pacman release is merely >> required for makepkg. Unless that has already landed too, which >> would be news to me :) >> >> Thus i don't think we need a hold-off period like this, Allan. > > We still need a hold-off period, we're just waiting to make sure > people have libarchive v3.3.3 instead of pacman v5.2.0. > That update happened half a year ago, i'm sure that most people with an installation that old will already have to fetch other packages, like the keyring, separately for it to go through. Plus, with libarchives' release cycle, i don't think that libarchive itself is gonna be rebuilt immediately after the change is implemented - providing extra time to upgrade libarchive without having to download a release packed as xz separately. Rob signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 3/24/19 9:18 PM, Baptiste Jonglez wrote: > Just one detail: your results for -19, -20 and -21 are identical because > apparently zstd needs an additional flag (--ultra) to "unlock" the higher > compression levels: > > zstd -c -T0 -20 - > Warning : compression level higher than max, reduced to 19 > > Also, I see you did not test zstd with a small number of cores: can you > add e.g. -T1, -T2 and -T4 to the comparison? It would give a more > realistic idea of what to expect when building on a typical machine, as > opposed to dragon ;) > In my tests, using less threads also decreased memory usage when > compressing (35% less memory when switching from -T2 to -T1). > Damn, i knew i must've missed something - archange had already mentioned on IRC that these results look weird, but i shrugged it off. Should've double-checked. I'll get you a new table with the higher levels fixed and a second set with -T2 for comparison later. Regardless, IIRC preliminary testing showed that these gains are not worth it, as they were quite small in the tests we ran a while ago. > For decompression, it seems that both xz and zstd run single-threaded, so > there's not much to think about (zstd is just incredibly fast). > Correct Rob signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
On 3/24/19 11:20 PM, Evangelos Foutras via arch-dev-public wrote: > On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public > wrote: >> >> On 25/3/19 4:34 am, Robin Broda via arch-dev-public wrote: >>> This change requires a new pacman release, as as of writing this, zstd >>> support is in master but hasn't landed in a release yet. >> >> Which is a complete blocker for quite a period of time. >> >> We need to assume every system has a copy of pacman-5.2+ before we can >> switch packages to zstd. > > Why is pacman support needed here? I can already install .zstd > packages using pacman 5.1.3. > > The crucial part seems to be libarchive support, which was added in > v3.3.3 (~ September 2018). > Yes, installing zstd packages works - the pacman release is merely required for makepkg. Unless that has already landed too, which would be news to me :) Thus i don't think we need a hold-off period like this, Allan. Rob signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd
Attached here is the script i wrote to make most of these measurements, for anyone interested in reproducing these results - and the raw results of the benchmark. Read it before running it. You may need to make adjustments. Rob bench.sh Description: application/shellscript Compressor,File,Size (Bytes),Compressed Size (Bytes),Ratio,Time ([hh:]mm:ss),Maximum RSS in Kbytes,Decompress time ([hh:]mm:ss),Decompress RSS in KBytes zstd -c -T0 -18 -,cuda-10.0.130-2-x86_64.pkg.tar,3186186240,1442220663,45,1:12.50,2712508,0:04.46,10956 zstd -c -T0 -18 -,gcc-8.2.1+20181127-1-x86_64.pkg.tar,142120960,37615836,26,0:12.37,429292,0:00.23,11028 zstd -c -T0 -18 -,go-2:1.12.1-1-x86_64.pkg.tar,507617280,139137995,27,0:15.40,1436664,0:00.80,11064 zstd -c -T0 -18 -,linux-5.0.3.arch1-1-x86_64.pkg.tar,84039680,73632419,87,0:07.48,306488,0:00.05,10892 zstd -c -T0 -18 -,linux-headers-5.0.3.arch1-1-x86_64.pkg.tar,108892160,19841783,18,0:12.68,328680,0:00.16,10996 zstd -c -T0 -18 -,tensorflow-1.13.1-2-x86_64.pkg.tar,317818880,64831235,20,0:15.99,877552,0:00.47,10892 zstd -c -T0 -19 -,cuda-10.0.130-2-x86_64.pkg.tar,3186186240,1438580037,45,1:34.13,3483312,0:04.47,10992 zstd -c -T0 -19 -,gcc-8.2.1+20181127-1-x86_64.pkg.tar,142120960,37393039,26,0:15.76,592888,0:00.24,10920 zstd -c -T0 -19 -,go-2:1.12.1-1-x86_64.pkg.tar,507617280,138242077,27,0:19.74,1960004,0:00.79,11040 zstd -c -T0 -19 -,linux-5.0.3.arch1-1-x86_64.pkg.tar,84039680,73585177,87,0:09.32,404812,0:00.05,10980 zstd -c -T0 -19 -,linux-headers-5.0.3.arch1-1-x86_64.pkg.tar,108892160,19799328,18,0:16.36,459760,0:00.17,10888 zstd -c -T0 -19 -,tensorflow-1.13.1-2-x86_64.pkg.tar,317818880,64477111,20,0:21.01,1204980,0:00.50,10940 zstd -c -T0 -20 -,cuda-10.0.130-2-x86_64.pkg.tar,3186186240,1438580037,45,1:34.34,3498908,0:04.46,11044 zstd -c -T0 -20 -,gcc-8.2.1+20181127-1-x86_64.pkg.tar,142120960,37393039,26,0:16.36,593004,0:00.25,11012 zstd -c -T0 -20 -,go-2:1.12.1-1-x86_64.pkg.tar,507617280,138242077,27,0:20.19,1960052,0:00.77,10980 zstd -c -T0 -20 -,linux-5.0.3.arch1-1-x86_64.pkg.tar,84039680,73585177,87,0:08.88,404716,0:00.06,10820 zstd -c -T0 -20 -,linux-headers-5.0.3.arch1-1-x86_64.pkg.tar,108892160,19799328,18,0:16.26,459768,0:00.16,11032 zstd -c -T0 -20 -,tensorflow-1.13.1-2-x86_64.pkg.tar,317818880,64477111,20,0:21.11,1205128,0:00.49,10900 zstd -c -T0 -21 -,cuda-10.0.130-2-x86_64.pkg.tar,3186186240,1438580037,45,1:31.60,3496084,0:04.46,11044 zstd -c -T0 -21 -,gcc-8.2.1+20181127-1-x86_64.pkg.tar,142120960,37393039,26,0:16.18,592904,0:00.25,10712 zstd -c -T0 -21 -,go-2:1.12.1-1-x86_64.pkg.tar,507617280,138242077,27,0:20.08,1960032,0:00.79,11040 zstd -c -T0 -21 -,linux-5.0.3.arch1-1-x86_64.pkg.tar,84039680,73585177,87,0:08.91,404768,0:00.05,10964 zstd -c -T0 -21 -,linux-headers-5.0.3.arch1-1-x86_64.pkg.tar,108892160,19799328,18,0:16.39,459748,0:00.16,10980 zstd -c -T0 -21 -,tensorflow-1.13.1-2-x86_64.pkg.tar,317818880,64477111,20,0:21.16,1205132,0:00.50,10928 xz -c -z -,cuda-10.0.130-2-x86_64.pkg.tar,3186186240,1380898716,43,19:03.44,97604,1:19.74,10420 xz -c -z -,gcc-8.2.1+20181127-1-x86_64.pkg.tar,142120960,34723000,24,0:54.54,97632,0:02.59,10352 xz -c -z -,go-2:1.12.1-1-x86_64.pkg.tar,507617280,128032556,25,3:19.11,97636,0:08.78,10400 xz -c -z -,linux-5.0.3.arch1-1-x86_64.pkg.tar,84039680,74094188,88,0:31.27,97640,0:03.85,10352 xz -c -z -,linux-headers-5.0.3.arch1-1-x86_64.pkg.tar,108892160,17844920,16,0:42.24,97636,0:01.45,10392 xz -c -z -,tensorflow-1.13.1-2-x86_64.pkg.tar,317818880,58282188,18,1:59.56,97692,0:04.78,10512 zstd -c -T0 -18 -,intellij-idea-community-edition-2:2018.3.5-1-x86_64.pkg.tar,809984000,411503795,50,0:27.10,2397200,0:00.91,10884 zstd -c -T0 -19 -,intellij-idea-community-edition-2:2018.3.5-1-x86_64.pkg.tar,809984000,410033083,50,0:37.09,3182564,0:00.93,10720 zstd -c -T0 -20 -,intellij-idea-community-edition-2:2018.3.5-1-x86_64.pkg.tar,809984000,410033083,50,0:34.43,3182456,0:00.93,10960 zstd -c -T0 -21 -,intellij-idea-community-edition-2:2018.3.5-1-x86_64.pkg.tar,809984000,410033083,50,0:35.45,3179460,0:00.94,10896 xz -c -z -,intellij-idea-community-edition-2:2018.3.5-1-x86_64.pkg.tar,809984000,403036076,49,4:53.01,97596,0:28.69,10428 signature.asc Description: OpenPGP digital signature
[arch-dev-public] RFC: (devtools) Changing default compression method to zstd
Hello all, in the past few weeks, some TUs and Developers have compared different compression algorithms to potentially replace the default compression method used in devtools. The current method is `xz -c -z -` which is single-threaded and rather slow, so we are looking to replace it with something faster. Multithreaded xz has come up in the past, and was quickly dismissed due to edge cases that would end up with packages being unreproducible on different machines - namely, xz -T0 -- the method that automatically determines the amount of threads -- produces different results when the amount of cores in a system is == 1: $ taskset -c 1 xz -c -z - -T0 < test > test.xz && sha256sum test.xz fe95a1af78304ae4be508e071f6697296e52b625fba95fca5622757779633d90 test.xz $ taskset -c 1,2 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz $ taskset -c 1,2,3 xz -c -z - -T0 < test > test.xz && sha256sum test.xz 3b2c520eda654de19c5fc02ea1d850e142ae24e1246edcce82e90bd690d18f99 test.xz With this mail, i propose to switch to `zstd` instead (https://github.com/facebook/zstd). zstd does *not* exhibit this issue, and anthraxx has asked for some clarifications (https://github.com/facebook/zstd/issues/999#issuecomment-474114799) - just in case. The response is that zstd is generally friendly to reproducible builds. After some testing with heftig, I ran some additional benchmarks on our new build host 'dragon' to determine the appropriate compression level. Here are the results: (sorry for the wide mail :b) Compressor Package Name Size (MiB) Comp. Size (MiB) Ratio Time (mm:ss) Max. RSS in MiB Decomp. Time (mm:ss) Decomp. RSS in MiB xz -c -z - cuda 3038,58 1316,93 43,34% 19:03.44 95,321:19.74 10,18 zstd -c -T0 -18 - cuda 3038,58 1375,41 45,26% 01:12.50 2648,93 0:04.46 10,70 zstd -c -T0 -19 - cuda 3038,58 1371,94 45,15% 01:34.13 3401,67 0:04.47 10,73 zstd -c -T0 -20 - cuda 3038,58 1371,94 45,15% 01:34.34 3416,90 0:04.46 10,79 zstd -c -T0 -21 - cuda 3038,58 1371,94 45,15% 01:31.60 3414,14 0:04.46 10,79 xz -c -z - gcc 135,54 33,11 24,43% 00:54.54 95,340:02.59 10,11 zstd -c -T0 -18 - gcc 135,54 35,87 26,47% 00:12.37 419,23 0:00.23 10,77 zstd -c -T0 -19 - gcc 135,54 35,66 26,31% 00:15.76 578,99 0:00.24 10,66 zstd -c -T0 -20 - gcc 135,54 35,66 26,31% 00:16.36 579,11 0:00.25 10,75 zstd -c -T0 -21 - gcc 135,54 35,66 26,31% 00:16.18 579,01 0:00.25 10,46 xz -c -z - go 484,10 122,10 25,22% 03:19.11 95,350:08.78 10,16 zstd -c -T0 -18 - go 484,10 132,69 27,41% 00:15.40 1402,99 0:00.80 10,80 zstd -c -T0 -19 - go 484,10 131,84 27,23% 00:19.74 1914,07 0:00.79 10,78 zstd -c -T0 -20 - go 484,10 131,84 27,23% 00:20.19 1914,11 0:00.77 10,72 zstd -c -T0 -21 - go 484,10 131,84 27,23% 00:20.08 1914,09 0:00.79 10,78 xz -c -z - intellij-idea-community-edition 772,46 384,37 49,76% 04:53.01 95,310:28.69 10,18 zstd -c -T0 -18 - intellij-idea-community-edition 772,46 392,44 50,80% 00:27.10 2341,02 0:00.91 10,63 zstd -c -T0 -19 - intellij-idea-community-edition 772,46 391,04 50,62% 00:37.09 3107,97 0:00.93 10,47 zstd -c -T0 -20 - intellij-idea-community-edition 772,46 391,04 50,62% 00:34.43 3107,87 0:00.93 10,70 zstd -c -T0 -21 - intellij-idea-community-edition 772,46 391,04 50,62% 00:35.45 3104,94 0:00.94 10,64 xz -c -z - linux80,15 70,66 88,17% 00:31.27 95,350:03.85 10,11 zstd -c -T0 -18 - linux80,15 70,22
Re: [arch-dev-public] Co-maintainers and less time for packaging
On 3/20/19 2:02 PM, Morten Linderud via arch-dev-public wrote: > acpid > neofetch > udiskie Adopted! -- Rob (coderobe) O< ascii ribbon campaign - stop html mail - www.asciiribbon.org signature.asc Description: OpenPGP digital signature
Re: [arch-dev-public] Boost looking for new maintainer
On 03/20/2018 02:36 PM, Bartłomiej Piotrowski via arch-dev-public wrote: > Hi team, > > As I'm absolutely the worst with C++, none of my packages depend on > boost and I've spent enough nights terrified of its build system quirks, > I'll be orphaning it soon. Feel free to adopt it in archweb. > > Bartłomiej > If you're willing to drop it to [community] i'll gladly adopt it. Rob
