[arch-dev-public] Congrats to the Arch Conf Team

[Allan McRae via arch-dev-public]

A big shout out to the Arch Conf Team!

This weekend's conference was nothing short of awesome.  It pulled
together very well, particularly given the short organisation period and
it being the first one run online.

The recorded talks followed by live Q worked very well, and I enjoyed
the live commentary by the community either on IRC or twitch.

Looking forward to next year!  :)

Allan

Re: [arch-dev-public] Use detached package signatures by default

[Allan McRae via arch-dev-public]

On 9/7/20 1:05 pm, Anatol Pomozov wrote:
> Given this information I would like to propose to stop using embedded
> signatures and move to detached signatures by default. This will
> require pacman 6.x or as alternative backport the fix(es) to 5.x
> branch. It will help to make system updates even faster, something
> that me and many other Arch users really love.

There are several steps we need to complete:

1) backport the patch (or wait for pacman-6.0, which may be a while
yet).  I'll leave that to the distro packagers to decide!

2) adjust repo-add to optionally add signatures.

3) make a time line that all users need to have the patched/released
pacman installed - we usually require at least 6 months.

4) turn off signature inclusion in repo dbs.

Allan

[arch-dev-public] Reproducible builds progress report #3

[Allan McRae via arch-dev-public]

Hi all,

A quick updated on the progress of reproducible builds.

You may have noticed a couple of large rebuilds that occurred recently.
These fixed issues of non-reproducible file ordering with old versions
of makepkg. This and other hard work by the team improving our tooling
and fixing packaging issues has resulted in 96% of [core] being
reproducible, and 90% of [extra]. You can see the status of which
packages are reproducible here [1].

The remaining packages to fix in [core] are dnssec-anchors, linux,
linux-lts, nss and perl. With the possible exception of perl, these are
in the "hard" basket. There is plans on how to fix the kernel packages,
but that will require some time to sort out. We would be happy for more
people to help out so we can get [core] to 100% reproducible.

We have investigated some of the packages in [extra] that fail to
reproduce here [2]. Note that there are quite a few packages that
currently "Failed to build from source" (FTBFS) - it would be very
helpful for the reproducible builds team if their maintainers can help
fix the packages. You can also use the CI of Arch packages run by Debian
to get an overview what the issue is with these packages and see many
other packages that are currently failing to build [3].

We also need help to investigate and fix the packages that fail to
reproduce that we have not investigated as of yet. There are two easy to
use tools to attempt to reproduce a package - "makerepropkg" from
devtools and "repro" from the archlinux-repro package. Once these have
rebuilt a package, you can use the "diffoscope" tool to look at the
differences. Jump in the #archlinux-reproducible IRC channel if you want
help interpreting the output, or you could just link to a copy of it in
the wiki.

[1] https://reproducible.archlinux.org/
[2] https://wiki.archlinux.org/index.php/Reproducible_Builds/Status
[3] https://tests.reproducible-builds.org/archlinux/extra.html

Re: [arch-dev-public] Discussion - Increasing our CPU requirements

[Allan McRae via arch-dev-public]

On 30/3/20 12:39 am, Filipe Laíns wrote:
> On Sun, 2020-03-29 at 23:37 +1000, Allan McRae via arch-dev-public wrote:
>> On 29/3/20 11:17 pm, Filipe Laíns wrote:
>>> I would also like to note that rebuilding everything with forced
>>> support for AVX2 or whatever won't have much effect. Most packages do
>>> not have workloads where it would make use sense to use these CPU
>>> extensions, and as such, GCC would not use them.
>>
>> That assumes we just add AVX2.  Whereas, requiring a CPU supporting AVX2
>> would bring other optimizations that would be used.
> 
> No, it should be true for all extensions.
> 
>> As I replied earlier, AVX2 may be going too far.  But is a good starting
>> point for discussion.  If that is too far, what could we accept?
>> SSE4.2?  AVX?   Surely we can do better than pure x86_64.
> 
> No, SSE4.2 is too far. For me, the minimum should be AVX.


SSE4.2 is 2008 for Intel, 2011 for AMD.  Though I guess some processors
were released without it for some time after that.   AVX was released by
both in 2011.

So why is one too far and the other not?


>> To have a separate architecture would require automated builds, which
>> requires being able to sign packages automatically.  And we have not
>> achieved database signing in 9 years  I'm looking for a boost that
>> could be achieved now.
> 
> No, it would not. Where is this coming from? I already build split
> packages with SIMD instructions, I make the PKGBUILD build for 2
> architectures instead with a minimal patch.
> 
> If pacman is not able to handle parallel architectures, we should fix
> that. I think it's a valid use case.

No need for pacman support.  Just add higher instruction set to a new
repo and set that repo with higher priority.

But that involves developers choosing which packages to build with
higher instruction sets, which requires extra developer time.

Ideally, we would just autobuild for more optimized architectures, but
this requires auto-signing packages, which has not happened in the last
decade (but may in this one...).


Picking an instruction set that is ~10 years old and making it the
default for the distro seems a reasonable approach to me.

A

Re: [arch-dev-public] Discussion - Increasing our CPU requirements

[Allan McRae via arch-dev-public]

On 29/3/20 11:17 pm, Filipe Laíns wrote:
> I would also like to note that rebuilding everything with forced
> support for AVX2 or whatever won't have much effect. Most packages do
> not have workloads where it would make use sense to use these CPU
> extensions, and as such, GCC would not use them.

That assumes we just add AVX2.  Whereas, requiring a CPU supporting AVX2
would bring other optimizations that would be used.

As I replied earlier, AVX2 may be going too far.  But is a good starting
point for discussion.  If that is too far, what could we accept?
SSE4.2?  AVX?   Surely we can do better than pure x86_64.


To have a separate architecture would require automated builds, which
requires being able to sign packages automatically.  And we have not
achieved database signing in 9 years  I'm looking for a boost that
could be achieved now.

Allan

Re: [arch-dev-public] Discussion - Increasing our CPU requirements

[Allan McRae via arch-dev-public]

On 29/3/20 8:52 pm, Evangelos Foutras wrote:
> If I see a SIGILL on my AMD Phenom II X6 1090T then Arch will have failed me. 
> 
> 
> I believe your proposal should only be discussed as co-existing
> optimized port(s) and even then I'm not sure it's worth the trouble.
> Performance-critical applications can and frequently are optimized for
> the running processor (I'm thinking of stuff like glibc and ffmpeg
> here).

AVX2 was a bold choice, and really a place to get a discussion started.

We are currently supporting processors from 2003.  We can be better than
that.

A

Re: [arch-dev-public] Discussion - Increasing our CPU requirements

[Allan McRae via arch-dev-public]

On 29/3/20 9:27 pm, Amish wrote:
> Also if I am not wrong Arch philosophy talks only about latest software
> and no where there is mention of latest hardware being a compulsion.

It used to.  One of the original selling points was i686 optimization.
Then we got lazy, and stopped innovating.

A

[arch-dev-public] Discussion - Increasing our CPU requirements

[Allan McRae via arch-dev-public]

Remember when Arch Linux was optimized out of the box.   We have the
blazingly fast i686 port while other distros hung out in i386 land.
Those were the days where the idea of Arch being fast started.  Now it
has degraded to stuff of legend.

Now, x86_64 is old.  We should continue to push forward and add further
optimization.

Reasonable optimizations to consider:

AVX2
FMA
SSE4.2

AVX2 is Intel Haswell and newer or AMD Ryzen and newer.  This CPUs
released 2013 to 2015.  So 5 - 7 years old.

Discuss.

Re: [arch-dev-public] Restricting ability to post news items

[Allan McRae via arch-dev-public]

On 6/1/20 9:12 am, Giancarlo Razzolini wrote:
> Em janeiro 5, 2020 19:25 Allan McRae via arch-dev-public escreveu:
>>
>> Read the original message and not the partial quote that you made.  I
>> explicitly said there was an exception for --overwrite type posts.
>>
>> But any restriction being made on posting due to not posting drafts to
>> the list would be complete.
>>
> 
> Hi Allan,
> 
> I think you're overreacting (again) for something that's not properly
> coded anywhere.
> And, the last time this was discussed, I have talked about cases of news
> that couldn't
> be brought on a-d-p for discussion. We have other places to discuss
> those (staff@), but
> still, until we have *clear* guidelines about this, let's not rush to
> implement anything.

My apologies.  I believed this had been covered on the mailing list and
I am told it was only on IRC, and never passed on to the TU channel,
which I will accept as an excuse despite everyone(?) involved but the
poster being on both channels...

> I have proposed a news entry regarding zstd, Robin did the statistics
> and we've discussed this
> for 2 days on #archlinux-tu. If you're looking for somebody to blame,
> look no further. Take away
> my news posting rights (you'll have to also take away my archweb admin
> rights in this case).
> 
> And let's properly codify this ok? Because it's not codified anywhere. I
> could've asked Robin
> to send an email to a-d-p, but most of the work we've done on the draft
> was on a pad. This was
> reviewed by quite a few people. I asked Robin to do it, because since
> they pushed zstd, they
> should've also be the ones doing the news entry. But I would do it
> myself this week, if they
> didn't.

Do we really need to write down everything?  Have we reached a point in
the distro where common sense has stopped?  Why would an announcement
that affects the whole distro not be run past all team members by default?

Allan

Re: [arch-dev-public] Restricting ability to post news items

[Allan McRae via arch-dev-public]

On 6/1/20 8:04 am, Morten Linderud via arch-dev-public wrote:
> On Mon, Jan 06, 2020 at 07:53:21AM +1000, Allan McRae via arch-dev-public 
> wrote:
>> Following the roll out of the base metapackage, and its poorly written
>> news post, we agreed that all new posts should have a draft posted to
>> arch-dev-public.
> 
> Wait, where was this agreed? I heard something about all drafts should be 
> headed
> for arch-dev, but this hasn't been announced nor discussed anywhere.
> 
> Are the TUs missing from the loop here?
> 

After the base metapackage pile of crap that was posted, I (and others)
reminded everyone of the process that had been held for years.

The response was complaints that this process was not followed recently,
to which we pointed at the arch-dev-public mailing list post for
multiple non-trivial news entries.  Only simple "--overwrite" type posts
have not been posted on arch-dev-public.

This is not new - it has been the standard for many, many years -
spanning from before I was around the distro.

Allan

Re: [arch-dev-public] Restricting ability to post news items

[Allan McRae via arch-dev-public]

On 6/1/20 8:17 am, Morten Linderud via arch-dev-public wrote:
> On Sun, Jan 05, 2020 at 11:10:17PM +0100, Bartłomiej Piotrowski via 
> arch-dev-public wrote:
>> On 05/01/2020 23.04, Morten Linderud via arch-dev-public wrote:
>>> On Mon, Jan 06, 2020 at 07:53:21AM +1000, Allan McRae via arch-dev-public 
>>> wrote:
>>>> Following the roll out of the base metapackage, and its poorly written
>>>> news post, we agreed that all new posts should have a draft posted to
>>>> arch-dev-public.
>>>
>>> Wait, where was this agreed? I heard something about all drafts should be 
>>> headed
>>> for arch-dev, but this hasn't been announced nor discussed anywhere.
>>>
>>> Are the TUs missing from the loop here?
>>>
>>
>> If you look at the non-trivial news items, you can easily correlate them
>> with drafts posted to arch-dev-public.
> 
> You are writing about non-trivial news items, but Allan is writing explicitly
> about *all* news items. There is a disconnect here, I'm not sure what has been
> decided.

Read the original message and not the partial quote that you made.  I
explicitly said there was an exception for --overwrite type posts.

But any restriction being made on posting due to not posting drafts to
the list would be complete.

Allan

[arch-dev-public] Restricting ability to post news items

[Allan McRae via arch-dev-public]

Hi all,

Following the roll out of the base metapackage, and its poorly written
news post, we agreed that all new posts should have a draft posted to
arch-dev-public.  There was a potential exclusion for trivial
--overwrite posts [1].

This was followed for the Xorg update.   It was not followed for the
zstd update.

Given people have shown an inability to follow simple instructions,  I
am proposing a restriction on the ability to make news posts.  This can
either be a bulk restriction now, or just removing anybody who does not
follow the rule from ever making a news post again.

Allan


[1] and why have we had so many missing library symlinks lately...
surely checkpkg detects all the missing symlinks compared to previous
packages.

Re: [arch-dev-public] Adding a "posix" metapackage

[Allan McRae via arch-dev-public]

On 4/1/20 12:47 pm, Sébastien Luttringer via arch-dev-public wrote:
> One unfortunate consequence could be to have packages rely on it to make
> dependencies shorter, and make us pull cups or cronie.

What?!  That is like saying one unfortunate consequence of pamcan hooks
is that packages can have no files and just download and compile the
source in a hook.  It is a ridiculous consideration.

A

Re: [arch-dev-public] libx11/xorgproto dependency

[Allan McRae via arch-dev-public]

On 21/12/19 7:31 pm, Evangelos Foutras via arch-dev-public wrote:
> Downstream consumers of libx11 shouldn't have to know and account for
> libx11's headers/pkg-config files referencing xorgproto. A
> libx11-devel package would depend on xorgproto. Since there's no
> separate -devel package, the dependency stays with the regular libx11
> package.

This matches my opinion on the matter.

A

Re: [arch-dev-public] cleanup dead Xorg packages | news draft

[Allan McRae via arch-dev-public]

On 20/12/19 8:35 am, Andreas Radke via arch-dev-public wrote:
> Packages have been rebuilt and prepared to remove obsolete libdmx and
> libxxf86dga. Xorgproto legacy support has been removed and wherever it
> was added to be a runtime dependency it is now a build time
> dependency. Some packages will need additional xorgproto makedependency
> added when missing some header now that the libraries won't pull it in
> anymore. That behavior is desired. I'm still in the process of fixing
> the move from legacy proto headers to xorgproto headers. I'm going to
> commit the changes to trunk for future builds.
> 
> There's no package replacing libdmx and libxxf86dga so manual
> intervention will be needed. So here's a small news draft:
> 
> "Xorg cleanup requires manual intervention
> 
> "In the process of Xorg cleanup [1] the update requires manual
> intervention when you hit this message:
> 
>  :: installing xorgproto (2019.2-2) breaks dependency 'inputproto' required 
> by lib32-libxi
>  :: installing xorgproto (2019.2-2) breaks dependency 'dmxproto' required by 
> libdmx
>  :: installing xorgproto (2019.2-2) breaks dependency 'xf86dgaproto'
>  required by libxxf86dga
> 
> when updating, use:
> `pacman -Rdd libdmx libxxf86dga && pacman -Syu`
> 
> to perform the upgrade. After the update it will be safe to also remove
> the "xorgproto" package.

This why does xorgproto not provides=('inputproto'  )?  Then all we
need to do is announce, update and remove.

Allan

Re: [arch-dev-public] Proposal: Build a ruleset for new packages and package quality

[Allan McRae via arch-dev-public]

On 12/12/19 10:21 pm, Christian Rebischke via arch-dev-public wrote:
> 3. revive the discussion around better PKGBUILD quality (see also Eli's
> thread about  PKGBUILD quality on arch-tu:
> https://lists.archlinux.org/private/arch-tu/2019-November/83.html)

Any chance that can be posted somewhere visible to those that aren't TUs?

Allan

[arch-dev-public] Reproducible builds progress and the upcoming rebuild of [core]

[Allan McRae via arch-dev-public]

Hi all,

As you may know, we have had people busy looking at what it takes to
make our packages reproducible.

There has been a lot of progress there lately.  Our reproducible builds
team (along with the wider reproducible builds community) has been
building our packages in different environments to test how stable the
builds are [1].  The good news is that >80% of our packages could be
built twice in varying environments and give the exact same result.

However, that is only part of the picture.  Ideally, we want people to
be able to take one of our packages and rebuild it exactly.  With the
release of pacman-5.2, packages record a lot more information about
their build environment.  That means we can reconstruct a package's
build chroot, and then rebuild it.  There are two tools in the works to
do this.  One by Morton (Foxboron) [2] and one by Eli [3]. Note that
both tools need more testing to be ready for a wider release and
currently require some manual editing to run.

The good news is, we have at least 10 packages that can be precisely
reproduced using both these tools [4]!  This means you can take one of
these tools and rebuild a package from the repos, and get the exact same
package out of it.  This is an amazing effort - well done to the team!

To keep this momentum going, it would be great to rebuild every package
in [core] using makepkg from pacman-5.2+.  That way we can test which
packages are actually reproducible and work towards fixing those that
are not.  So be prepared for almost the entire repo to hit [testing]
soon, and get your sign-off shoes on!

Again, a huge congrats to our reproducible builds team.  This has been a
massive amount of work!

Allan


[1] https://tests.reproducible-builds.org/archlinux/archlinux.html
[2] https://github.com/archlinux/archlinux-repro
[3]
https://github.com/eli-schwartz/devtools/blob/reproducible/makerepropkg.in
[4] https://wiki.archlinux.org/index.php/DeveloperWiki:ReproduciblePackages

Re: [arch-dev-public] Using SPDX License list as identifiers

[Allan McRae via arch-dev-public]

On 22/10/19 9:59 pm, Jerome Leclanche wrote:
> It would also be a
> little more accurate; eg. the SPDX allows for distinctions such as
> "LGPL-3.0-or-later" vs. "LGPL-3.0-only".

I thought we already managed that, but it seems in rather limited use
these days looking at our -Si output.  e.g.

Licenses: LGPL-2.1
Licenses: LGPL-2.1+

A

Re: [arch-dev-public] New ideas for notifying users about (minor) changes

[Allan McRae via arch-dev-public]

On 30/7/19 10:44 am, Christian Rebischke wrote:
> One problem I see with the News section is that it's Dev only. I
> wouldn't even know who I need to ask (and I am TU for several years
> now). I mean I could just ask around in the IRC, but it would be nice to
> have this at least documented somewhere.

I guess the assumption is changes that affect a larger proportion of
users are in packages maintained by devs. However, I'm happy opening the
news to TUs if needed.

Allan

Re: [arch-dev-public] New ideas for notifying users about (minor) changes

[Allan McRae via arch-dev-public]

On 30/7/19 8:47 am, Christian Rebischke via arch-dev-public wrote:
> Hello everybody,
> I got assigned to this issue here:
> 
> https://bugs.archlinux.org/task/62823
> 
> The users would like to have a notice in pre/post install/upgrade
> notifications via pacman .install files.
> 
> I am not a fan of spamming such news via pacman and I think the
> installation/upgrade process should be clean of such messages, but I
> don't have access to the news tool on our website as well.

I think this is a clear case of user intervention needed.   So a
post_install message (restricted to the update where intervention was
needed) is appropriate.

If the package was more widely used, we have the News section of the
website.

I don't think anything else is needed.

Allan

Re: [arch-dev-public] Proposal for a new organisation structure

[Allan McRae via arch-dev-public]

On 3/6/19 2:39 pm, Christian Rebischke via arch-dev-public wrote:
> 3. I don't like that devs and TUs live aside each other instead of
> finally realizing themselves as community or group.
The TUs are set up as an independent organisation with their own bylaws.
 Many of those bylaws were implemented to keep independence from the
developer team.  Saying that, almost all developers have been a Trusted
User, and there is a decent overlap currently.  I mostly see this as a
responsibility divide.

> I think the
> community as itself would work much better if we would have user-access
> based package repositories and just 'maintainers', instead of this
> dev/TU split.

I agree that the distinction between [extra] and [community] is rather
limited.  I think we need a better definition of what [extra] is, and
move packages that don't fit that definition out of [extra] and into
[community] where both devs and TUs can maintain them.  Or even merge
those two repos.

>> As Gaetan already mentioned, the process is clear: somebody suggests a
>> new developer and we discuss until a consensus is reached. Feel free to
>> document that somewhere in our wiki if you think it needs to be
>> documented. If you have specific concerns with that process, feel free
>> to share them. However, I do not think anybody in the dev team ever had
>> any objections against that procedure.
>
> What interests me is why is this whole process not transparent and
> public? I mean we discuss adding new TUs publicly. Of course this
> dicussion comes with all its good and bad parts, but atleast it's
> transparent and people get a reason why they are not elected.

You are very much overthinking what goes on when deciding on a new
developer.  "Electing" a developer is very different than electing a TU.
 People given developer positions have probably been around for years
and are already doing the job before they get formally offered it.   So
it is a case of someone saying "this person should be a developer" and
the rest going "OK".  There is usually no discussion, because the
candidate is well known already, and has obviously been doing a good job
to even be nominated.  Having TU style discussions on the suitability of
a candidate makes no sense in that situation.

Allan

Re: [arch-dev-public] Proposal for a new organisation structure

[Allan McRae via arch-dev-public]

On 2/6/19 9:08 am, Christian Rebischke via arch-dev-public wrote:
> 1. Simplified repositories
> 
> Currently we have [core], [extra] and [community]. These three
> repositories are there, because they have grown to this structure over
> time. At the moment I don't see any real meaning for the split of [core]
> and [extra]. So one suggestion would be to either:
>   - merge [extra] and [core]

I stopped reading here.   If you don't understand the difference between
[core] and [extra], you should ask.  Particularly before proposing the
removal of [core].

Allan

Re: [arch-dev-public] bringing vivaldi browser to community

[Allan McRae via arch-dev-public]

On 2/6/19 1:53 am, Ike Devolder via arch-dev-public wrote:
> On Sat, 2019-06-01 at 21:30 +1000, Allan McRae wrote:
>> You don't seem to
>> explain why you need to ask in your email.
> 
> Because it is proprietary and I explain that now there is a valid
> reason compared to 3 years ago where there was practically no
> difference between vivaldi, chromium and opera.
> 

Does the license allow us to have it in the repos?  After a quick look,
I'd say no.

A

Re: [arch-dev-public] bringing vivaldi browser to community

[Allan McRae via arch-dev-public]

On 1/6/19 8:12 pm, Ike Devolder via arch-dev-public wrote:
> 3 years have passed since I first proposed to bring vivaldi into
> community. Now there is a clear differentiation between what vivaldi
> offers out-of-the box compared to other browsers.
> 
> Vivaldi offers a ton of customisation features out of the box, and is
> also able to just use the chrome/ium addons from the chrome webstore.
> 
> Personally I'm using vivaldi as my main browser since somewhere in 2015
> (shortly after the first beta was released) and the key features no
> other browser currently offers are:
> - webpanels
> - quick commands
> - tabtiling
> - tabstacking
> - tabbar positioning
> 
> I'll bring it in the same way as with opera, where you have the
> webbrowser + separate packge with libffmpeg.so to allow the playback of
> proprietary formats like mp4.
> 

Does the license allow us to distribute it?

And dozens of packages get added to [community] a week.  Why are you
asking here unless you think there will be an issue?  You don't seem to
explain why you need to ask in your email.

Allan

Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions

[Allan McRae via arch-dev-public]

On 25/5/19 9:34 pm, Bruno Pagani wrote:
> Out of curiosity, what did you rebuild of [core] lead to?

I had a potentially slightly faster system for a week...  It was mainly
a test to see if I spotted some build issues of test suite failures
beyond what is seen for x86_64.  All was good.

A

Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions

[Allan McRae via arch-dev-public]

On 25/5/19 9:19 pm, Bruno Pagani via arch-dev-public wrote:
> Hi,
> 
> Le 25/05/2019 à 02:17, Filipe Laíns via arch-dev-public a écrit :
>> I would also like to explore the idea of adding an "high performance"
>> architecture which would be able to make use of SSE{,2,3,4,4.1,4.2} and
>> AVX, which seem to be the standard for newer processors (>=2013). This
>> would only be available for packages that do high performance computing
>> (ex. openblas, sdrangel, etc.). Any thoughts on this?
> 
> As said on IRC, they have been discussions before on having multiple
> targets and corresponding repos, but the starting point is that we need
> automated build before going into such a direction, and this in turn has
> several requirements. I’ve linked to you the pad where we put our ideas
> together regarding this.
> 
> In the meantime, we had the case before of whether we should package
> e.g. $pkgname-{sse4,avx} in a case where it mattered a lot, but it
> turned out the software in question (embree) is able to do runtime
> detection of available ISA. Maybe some other packages are doing this
> too, else we could discuss whether allowing such flavours as a temporary
> measure would be acceptable for selected packages.

glibc detects available instruction sets and uses the best for many
functions.

I'd be very, very, very much against providing multiple variants of a
package in our repos.  Using asp and makepkg are is a hard solution for
those who really need a few packages rebuilt.

PS - I rebuilt [core] with -march=haswell recently as a test.  Automated
building is not an issue.  Unattended package/database signing is the
major stumbling block.

Allan

Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions

[Allan McRae via arch-dev-public]

On 25/5/19 5:22 pm, Lukas Jirkovsky via arch-dev-public wrote:
> On Sat, 25 May 2019 at 04:27, Filipe Laíns via arch-dev-public
>  wrote:
>> Setting `-mtune` to generic won't add any additional instruction sets
>> by itself, but it does not prevent instruction sets from being added.
>> Looks like GCC enables MMX, SSE and SSE2 by default, it isn't related
>> at all to `-march` like I stated in the email but it still presents the
>> same issue.
> 
> As far as I know, MMX, SSE and SSE2 are mandatory part of the AMD64
> instruction set, so they are not enabled randomly just because someone
> felt like it, but because they are be present on every x86_64 cpu.
> .

Correct.  Using the command I gave in my first reply:

$ gcc -march=x86-64 -Q --help=target | grep sse
  -mfpmath= sse
  -mno-sse4 [enabled]
  -msse [enabled]
  -msse2[enabled]
  -msse2avx [disabled]
  -msse3[disabled]
  -msse4[disabled]
...

$ gcc -march=x86-64 -Q --help=target | grep mmx
  -mmmx [enabled]

-mtune just tunes instructions for a "representative" set of "current"
CPUs that run as x86-64.

Allan

Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions

[Allan McRae via arch-dev-public]

On 25/5/19 10:17 am, Filipe Laíns via arch-dev-public wrote:
> Hello,
> 
> Currently there are no guidelines stating which x86 extensions (ex.
> SSE2, SEE3, SSE4, AVX, etc.) we support. This is a bit problematic
> since it lets compilers do what they want and possible generate code
> that can't run on some systems.
> 
> Even though this is an issue, it's not complete anarchy, at least yet!
> Just kidding :p. The vast majority of our native packages are compiled
> with GCC and we do default to `-mtune=generic` which is good but not
> optimal. `-mtune=generic` tells GCC to compile for a generic processor
> so it's up to GCC to decide which architecture extensions would compose
> a generic processor. I haven't been able to find any documentation on
> what x86 extensions are enabled for a "generic" processor but I was
> able to track them down to MMX, SSE (or KNI) and SSE2. Being
> undocumented they could change at any time so I don't think we should
> rely on `-mtune=generic`.
> 

I think you need to look at the difference between -march and -mtune.
We use "-march=x86-64", which defines the instruction sets that can be
used.  Adding "-mtune=generic" does not allow the inclusion of
additional instruction sets.

Look at the output of:
gcc -march=x86-64 -Q --help=target

Allan

Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Allan McRae via arch-dev-public]

Given the suggestion of using -18-, I decided to calculate how much
bigger our packages would be with the numbers given:

cuda-10.0.130-2-x86_64.pkg.tar  58.5M   104.40%
gcc-8.2.1+20181127-1-x86_64.pkg.tar 2.8M108.30%
go-2:1.12.1-1-x86_64.pkg.tar10.6M   108.70%
linux-5.0.3.arch1-1-x86_64.pkg.tar  -0.4M   99.40%
linux-headers-5.0.3.arch1-1-x86_64.pkg.tar  1.9M111.20%
tensorflow-1.13.1-2-x86_64.pkg.tar  6.2M111.20%
intellij-idea-community-edition-2:2018.3.5-18.1M102.10%

That is a decent increase.


Are the times given for decompress just a decompress, not install by
pacman?   Were they done on a SSD or pointed at /dev/null? (I assume not
a spinning disk given the times)

Allan

Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Allan McRae via arch-dev-public]

On 25/3/19 9:28 am, Robin Broda via arch-dev-public wrote:
> On 3/25/19 12:22 AM, Andrew Gregory wrote:
>> On 03/25/19 at 12:15am, Robin Broda via arch-dev-public wrote:
>>> On 3/24/19 11:20 PM, Evangelos Foutras via arch-dev-public wrote:
>>>> On Sun, 24 Mar 2019 at 23:45, Allan McRae via arch-dev-public
>>>>  wrote:
>>>>>
>>>>> We need to assume every system has a copy of pacman-5.2+ before we can
>>>>> switch packages to zstd.
>>>>
>>>> Why is pacman support needed here? I can already install .zstd
>>>> packages using pacman 5.1.3.
>>>>
>>>> The crucial part seems to be libarchive support, which was added in
>>>> v3.3.3 (~ September 2018).
>>>>
>>>
>>> Yes, installing zstd packages works - the pacman release is merely
>>> required for makepkg. Unless that has already landed too, which
>>> would be news to me :)
>>>
>>> Thus i don't think we need a hold-off period like this, Allan.
>>
>> We still need a hold-off period, we're just waiting to make sure
>> people have libarchive v3.3.3 instead of pacman v5.2.0.
>>
> 
> That update happened half a year ago, i'm sure that most people with an 
> installation that old will already have to fetch other packages, like the 
> keyring, separately for it to go through.

Fetching a keyring does not potentially bump sonames.

> Plus, with libarchives' release cycle, i don't think that libarchive itself 
> is gonna be rebuilt immediately after the change is implemented - providing 
> extra time to upgrade libarchive without having to download a release packed 
> as xz separately.

And if openssl gets and soname bump?

A

Re: [arch-dev-public] RFC: (devtools) Changing default compression method to zstd

[Allan McRae via arch-dev-public]

On 25/3/19 4:34 am, Robin Broda via arch-dev-public wrote:
> This change requires a new pacman release, as as of writing this, zstd 
> support is in master but hasn't landed in a release yet.

Which is a complete blocker for quite a period of time.

We need to assume every system has a copy of pacman-5.2+ before we can
switch packages to zstd.  Otherwise updates can break systems (pacman
and all dependencies will need to stay as .xz for a year at least, and
we have to hope that partial update does not break anything until a full
update is run).  Experience switching from .gz to .xz showed system
breakage was a fairly regular occurrence.

I would not do this until at least one year, maybe two after pacman-5.2
is released.

Allan

Re: [arch-dev-public] A contrib repository

[Allan McRae via arch-dev-public]

On 3/14/19 8:46 AM, Morten Linderud via arch-dev-public wrote:
> There is a *lot* of small tools people have written over the years that 
> resides
> in bin/ directories which could be useful for more people. We also have 
> several
> such tools on soyuz, where sogrep was added to devtools this week. 
> 
> I have been thinking it could be great to have a simple `contrib` repository
> where every team member has commit access. This could work as a staging area 
> for
> tools we would like to promote to `devtools` later on.
> 
> We could maybe sort this into directories for its purpose:
> * packaging
> * security
> * devops
> * testing
> * bugwrangler
> * misc
> 
> Tools that can be added here is the `ch` scripts from Bluewind, and the pkg-*
> tools eli has created. I also have some tools to look for pkgname in archweb,
> check them out from svn and check them against a nvchecker file.
> 
> This would hopefully give us a space where we can experiment with new 
> maintainer
> tools in a collaborative manner. I'd love to hear some feedback or thoughs on
> this!
> 

I was fairly sure any user can create a git repo on our server.  Look at
"Developer Projects" on https://git.archlinux.org/ .  Or use github,
where some of these scripts are already located.

I don't see the need for another repository.

A

Re: [arch-dev-public] Python 2 modules

[Allan McRae via arch-dev-public]

On 17/2/19 10:37 am, Eli Schwartz via arch-dev-public wrote:
> On 2/16/19 4:36 PM, Christian Hesse wrote:
>> Allan McRae via arch-dev-public  on Sat,
>> 2019/02/16 11:19:
>>> Below is a list of python2 modules that are a dependency for any other
>>> package. I did not check makedepends and I did not check recursively to
>>> build this list.
>>
>> The list contains optional dependencies. How to handle these?
> 
> I see no conceptual difference between optional dependencies and hard
> dependencies. Both are equally deserving of being in the repos for the
> sake of providing dependent functionality to other packages.
> 

Yes - this was a quick pass list.  There is probably a bunch that should
not be removed and a lot more that could be removed...

A

[arch-dev-public] Python 2 modules

[Allan McRae via arch-dev-public]

Hi all,

Python 2 reaches End of Life on 2020-01-01.  We currently have >950
python2 modules in the repos.   A lot of these are not used by any other
package in the repositories.   I think we should work towards removing them.

Leaving only python2 modules that are really required by other software,
highlights what needs worked on to port to python3.

Note Fedora is doing a similar removal for F30:
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal

What are opinions on this?  Should I make a TODO list?


Below is a list of python2 modules that are a dependency for any other
package. I did not check makedepends and I did not check recursively to
build this list.

python2-acme
python2-antlr2
python2-anyjson
python2-anytree
python2-apache-libcloud
python2-apispec
python2-argcomplete
python2-argon2_cffi
python2-argparse
python2-args
python2-arrow
python2-aspectlib
python2-astor
python2-atspi
python2-aubio
python2-audit
python2-augeas
python2-autobahn
python2-autopep8
python2-backports.lzma
python2-basemap
python2-betamax-matchers
python2-betamax-serializers
python2-binary-memcached
python2-biopython
python2-bitvector
python2-blist
python2-blosc
python2-bluepy
python2-bottle
python2-bottleneck
python2-braintree
python2-breathe
python2-bsddb
python2-btchip
python2-btrees
python2-cached-property
python2-caja
python2-cchardet
python2-celery
python2-chai
python2-chameleon
python2-characteristic
python2-cjkwrap
python2-click-log
python2-click-threading
python2-cloudflare
python2-cmarkgfm
python2-colander
python2-colorclass
python2-configargparse
python2-construct
python2-couchdb
python2-cram
python2-crayons
python2-cryptography-vectors
python2-cson
python2-cssselect2
python2-cssutils
python2-cx_freeze
python2-d2to1
python2-daemon
python2-daemonize
python2-datrie
python2-ddt
python2-digitalocean
python2-discid
python2-distutils-extra
python2-django
python2-dnslib
python2-dockerpty
python2-docopt
python2-docs
python2-doublex-expects
python2-dpcontracts
python2-dropbox
python2-editdistance
python2-egenix-mx-base
python2-elasticsearch-curator
python2-email-validator
python2-envisage
python2-eric
python2-ethtool
python2-evdev
python2-exam
python2-exiv2
python2-eyed3
python2-factory-boy
python2-fastpbkdf2
python2-faulthandler
python2-flake8-blind-except
python2-flake8-debugger
python2-flaky
python2-flask-gravatar
python2-flask-htmlmin
python2-flask-jwt
python2-flask-mail
python2-flask-migrate
python2-flask-paranoid
python2-flask-restful
python2-flask-security
python2-flask-socketio
python2-flask-talisman
python2-flask-wtf
python2-flexmock
python2-flickrapi
python2-flup
python2-fonttools
python2-foolscap
python2-fpconst
python2-freezegun
python2-fs
python2-funcy
python2-furl
python2-fxa
python2-gasp
python2-gcp-devrel-py-tools
python2-gdal
python2-gdata
python2-genshi
python2-genty
python2-geoip
python2-gevent-websocket
python2-gflags
python2-gitpython
python2-gnupg
python2-gnupginterface
python2-gnuplot
python2-gpgme
python2-grequests
python2-gtkspellcheck
python2-gudev
python2-h2
python2-h5py
python2-h5py-openmpi
python2-hacking
python2-harparser
python2-helper
python2-hexdump
python2-hglib
python2-httpretty
python2-hunter
python2-hypothesis
python2-i3-py
python2-ibm-db-sa
python2-icalendar
python2-igraph
python2-importlib_resources
python2-inet_diag
python2-invoke
python2-iocapture
python2-ipdb
python2-irc
python2-isomd5sum
python2-iwlib
python2-jieba
python2-js2py
python2-jsbeautifier
python2-json-logger
python2-jsonrpclib-pelix
python2-kaitaistruct
python2-kajiki
python2-kaptan
python2-keybinder2
python2-keyrings-alt
python2-keyutils
python2-kitchen
python2-kivy
python2-klein
python2-langdetect
python2-language-server
python2-lark-parser
python2-levenshtein
python2-libappindicator
python2-libarchive-c
python2-libforensic1394
python2-librabbitmq
python2-libtmux
python2-linux-procfs
python2-llfuse
python2-logbook
python2-logilab-common
python2-lttngust
python2-m2r
python2-magic
python2-mamba
python2-manhole
python2-manuel
python2-marisa
python2-marshmallow
python2-memcached
python2-mimerender
python2-mockito
python2-mongoengine
python2-mongomock
python2-mpd2
python2-munkres
python2-musicbrainz2
python2-mygpoclient
python2-mysql-connector
python2-nbxmpp
python2-ndg-httpsclient
python2-neovim
python2-netcdf4
python2-netcdf4-openmpi
python2-nine
python2-nltk
python2-nose2
python2-nose-cover3
python2-nose-exclude
python2-nose-fixes
python2-nose-randomly
python2-nose-show-skipped
python2-nosexcover
python2-oauth2client
python2-objgraph
python2-olefile
python2-openapi-spec-validator
python2-openpyxl
python2-openstackclient
python2-oslo-concurrency
python2-oslo-log
python2-oslosphinx
python2-oslotest
python2-ovirt-engine-sdk
python2-owslib
python2-pacparser
python2-pam
python2-pandas-datareader
python2-pandocfilters
python2-parse
python2-parsedatetime
python2-parsel
python2-paste
python2-pastedeploy
python2-pbkdf2
python2-pdoc
python2-peewee
python2-perf
python2-periphery
python2-phonenumbers
python2-piexif

Re: [arch-dev-public] Proposal: minimal base system

[Allan McRae via arch-dev-public]

On 13/2/19 8:17 am, Levente Polyak via arch-dev-public wrote:
> On 2/12/19 7:16 PM, Gaetan Bisson via arch-dev-public wrote:
>> [2019-02-12 16:40:08 +0100] Bruno Pagani via arch-dev-public:
>>> Just in case it wasn’t clear, my answer would have been mostly the same
>>> as Eli’s.
>>>
>>> So, Gaetan, Allan and Bartłomiej (or anyone else for that matter), do
>>> you have further comments/questions regarding this, does the existence
>>> of the base group alongside the arch/minimal-system now makes sense or
>>> would you still prefer to go without it?
>>
>> Allan and I have both stated that we don't want to introduce a new group
>> since we believe it would be highly redundant with base.
>>
>> Nothing new has been said since our last messages except Eli's post
>> which argues that the base group is largely inadequate in its current
>> state. This further supports our proposal that base should be improved
>> instead of introducing a new group.
>>
>> So I really don't see what arguments could have changed our minds...
>> It's also strange to me how you can concur with Eli's post without
>> agreeing with our conclusions.
>>
>> To go forward I suggest you propose a clear definition of the perfect
>> "minimal system" group you'd want to have, along with a proposed list of
>> packages. When consensus is reached, we adopt this list of packages for
>> base and put this definition on the wiki.
>>
>> Cheers.
>>
> 
> To make it as short as possible, the idea is not just to strip down the
> base group further but primarily to not use a group in the first place.
> It should be replaced with something that is consistent within itself
> over the whole lifetime of the system.
> Groups are the wrong tool for such a set: you explicitly install all
> those packages so they won't automatically be mark as not-required
> anymore once removed from that group, as well as new additions are not
> consistent during the lifetime of the system.

We are clear about that.  Call it a group or metapackage or whatever,
the objection is having the current base and the new "group" at the same
time.

A

Re: [arch-dev-public] Proposal: minimal base system

[Allan McRae via arch-dev-public]

On 5/2/19 11:38 pm, Bruno Pagani wrote:
> Maybe. As I said in my answer to Bartłomiej, I don’t know if beginners
> know enough things to install what they need beyond the minimum system,
> or if they just read the wiki about doing this or that, which might
> assume they have the current base group installed.

If only the wiki wasn't a static document, and could be updated after
any change was made...

Even then I am fairly sure people wanting to set-up LVM can install the
lvm package.  But I see "competent Linux user" has been removed from the
front page, so maybe we target idiots these days.

>> If we remove the excess from base, then we are down to a very small
>> difference between that and archlinux-system.  Only e2fsprogs, man, and
>> an editor different?
>>
>> So I see the proposed archlinux-system group being essentially what base
>> should be.
> 
> That is because you see base as the minimal system. So I’ll turn this
> differently: do you have objections against having, outside of the
> minimal meta-package described in our proposal, a packages group of
> “relatively standard” tools, that is purposed at beginner wanting to
> have only one simple pacstrap command to issue in order to get started?
> 
> Or put it yet another way: outside of this base group, does our proposal
> of a minimal metapackage suits you? If so, why does it matter to you
> that there is also a base group, provided the name is not subject to
> confusion, that has this metapackage plus other tools (that e.g. people
> coming from random other distro would expect to have at hand from the
> start), knowing that you would likely have almost no interactions with
> this group? If not, then I’m even more importantly waiting for your
> comments.

I don't object to redefining the minimal set of packages we expect
installed and making it a metapackage.  Currently base is bloated, and a
metapackage makes some sense.

archlinux-system - minimal set of packages
base-devel - collection of utilities most people expect installed

So where does that leave base?

base - a smaller collection of utilities most people expect installed

I see redundancy being created.  That is what I object to.

Allan

Re: [arch-dev-public] Proposal: minimal base system

[Allan McRae via arch-dev-public]

On 5/2/19 9:06 pm, Bruno Pagani wrote:
> Le 22/01/2019 à 00:59, Allan McRae a écrit :
>> On 22/1/19 9:41 am, Bruno Pagani wrote:
>>> Le 22/01/2019 à 00:23, Allan McRae via arch-dev-public a écrit :
>>>> On 22/1/19 8:03 am, Levente Polyak via arch-dev-public wrote:
>>>>> Everything that won’t be part of base-system needs to be added as a
>>>>> dependency to all requiring packages; alternatively don't omit any first
>>>>> level runtime dependencies at all.
>>>>>
>>>>> This package should only depend on strictly required explicit packages
>>>>> to get a working minimal Arch Linux system.
>>>>>
>>>>> The proposed end result is:
>>>>> - base: convenient helper group for quickly getting a working system
>>>>>   when installing, must include the new base-system package
>>>>> - base-system: package defining the minimum dependencies for a working
>>>>>   base runtime
>>>> I think the proposal is OK.  I'm not comfortable with our line about
>>>> base group packages being required given how many of them I don't have
>>>> installed.
>>>>
>>>> However...  I don't like idea of the base group and base-system package
>>>> existing together.  You definition of what base-system should be is much
>>>> the same as what the base group was defined to be.  What package
>>>> justifies itself in the base group, but would not be in base-system?  It
>>>> seems we would have two very similar things where one would do.
>>>>
>>>> Allan
>>> In the proposal, base would really just be a convenient helper for e.g.
>>> beginners installing their system, so they could get all tools that are
>>> often used during install (e.g. cryptsetup, lvm2, various FS/network
>>> tools, etc.) or (POSIX) tools people coming from other distros would
>>> expect to be here by default (man pages, nano/vi…) but that are
>>> interactive ones and thus not really required for operating.
>>>
>>> Anyone knowing their stuff could just install base-system + what they
>>> actually need (e.g., I would install cryptsetup and vim, and not care
>>> about netctl, xfsprogs or lvm2).
>> "Anyone knowing their stuff" is the essentially the stated Arch target
>> audience.
> 
> So apparently we did not answer all concerns here. I don’t expect Arch
> users to know thing so well that they know exactly what tools are in
> which packages when they install Arch for the first time. I think we
> should not mistake Arch Power Users, people that have a level of
> knowledge above Arch Users, that are just generic Linux Power Users.
> 
>> So, the definitions of the sets of packages are:
>>
>> base-system - essential packages we assume everyone has installed
>> (previous definition of base...)
> 
> To be clearer, the new proposition would be to call this arch-system to
> avoid confusion with base. However, note that this “previous definition
> of base” is definitively not that clear: when I installed Arch, I read
> things as “base is a convenient helper to get almost every standard
> tools you could need to do your install”.
> 
>> base group - base-system plus other packages some people probably
>> want/expect and support packages for filesystem types most people don't
>> actually need.
> For me, base will be what it has ever been: a fast way to get started as
> an Arch beginner.
>> Maybe slightly facetious on that last one, but I don't see a clear need
>> for the base group once base-system exists.
> 
> Because, as an Arch dev, you definitively qualify as an Arch Power
> Users. I wouldn’t use base either for myself, but I firmly believe most
> Arch beginners would.
> 
> Does that make sense to you, or do you still think every new Arch User
> should already know exactly what is required to get started?
> 

If someone knows they want to set up logical volumes and drive
encryption, then they know enough to install lvm and cryptsetup.  Same
with jfsutils, xfsutils.   So I don't think they should be in the base
group (e.g. I would not call jfsutils a standard tool).

If we remove the excess from base, then we are down to a very small
difference between that and archlinux-system.  Only e2fsprogs, man, and
an editor different?

So I see the proposed archlinux-system group being essentially what base
should be.

A

Re: [arch-dev-public] Proposal: minimal base system

[Allan McRae via arch-dev-public]

On 22/1/19 9:41 am, Bruno Pagani wrote:
> Le 22/01/2019 à 00:23, Allan McRae via arch-dev-public a écrit :
>> On 22/1/19 8:03 am, Levente Polyak via arch-dev-public wrote:
>>> Everything that won’t be part of base-system needs to be added as a
>>> dependency to all requiring packages; alternatively don't omit any first
>>> level runtime dependencies at all.
>>>
>>> This package should only depend on strictly required explicit packages
>>> to get a working minimal Arch Linux system.
>>>
>>> The proposed end result is:
>>> - base: convenient helper group for quickly getting a working system
>>>   when installing, must include the new base-system package
>>> - base-system: package defining the minimum dependencies for a working
>>>   base runtime
>> I think the proposal is OK.  I'm not comfortable with our line about
>> base group packages being required given how many of them I don't have
>> installed.
>>
>> However...  I don't like idea of the base group and base-system package
>> existing together.  You definition of what base-system should be is much
>> the same as what the base group was defined to be.  What package
>> justifies itself in the base group, but would not be in base-system?  It
>> seems we would have two very similar things where one would do.
>>
>> Allan
> 
> In the proposal, base would really just be a convenient helper for e.g.
> beginners installing their system, so they could get all tools that are
> often used during install (e.g. cryptsetup, lvm2, various FS/network
> tools, etc.) or (POSIX) tools people coming from other distros would
> expect to be here by default (man pages, nano/vi…) but that are
> interactive ones and thus not really required for operating.
> 
> Anyone knowing their stuff could just install base-system + what they
> actually need (e.g., I would install cryptsetup and vim, and not care
> about netctl, xfsprogs or lvm2).

"Anyone knowing their stuff" is the essentially the stated Arch target
audience.

So, the definitions of the sets of packages are:

base-system - essential packages we assume everyone has installed
(previous definition of base...)

base group - base-system plus other packages some people probably
want/expect and support packages for filesystem types most people don't
actually need.

Maybe slightly facetious on that last one, but I don't see a clear need
for the base group once base-system exists.

Allan

Re: [arch-dev-public] Proposal: minimal base system

[Allan McRae via arch-dev-public]

On 22/1/19 8:03 am, Levente Polyak via arch-dev-public wrote:
> Everything that won’t be part of base-system needs to be added as a
> dependency to all requiring packages; alternatively don't omit any first
> level runtime dependencies at all.
> 
> This package should only depend on strictly required explicit packages
> to get a working minimal Arch Linux system.
> 
> The proposed end result is:
> - base: convenient helper group for quickly getting a working system
>   when installing, must include the new base-system package
> - base-system: package defining the minimum dependencies for a working
>   base runtime

I think the proposal is OK.  I'm not comfortable with our line about
base group packages being required given how many of them I don't have
installed.

However...  I don't like idea of the base group and base-system package
existing together.  You definition of what base-system should be is much
the same as what the base group was defined to be.  What package
justifies itself in the base group, but would not be in base-system?  It
seems we would have two very similar things where one would do.

Allan

Re: [arch-dev-public] Mongodb and SSPL

[Allan McRae via arch-dev-public]

On 17/1/19 12:02 am, Morten Linderud via arch-dev-public wrote:
> Yo!
> 
> As probably some of you have realized, there is a discussion regarding mongodb
> and the relicense from AGPLv3 to SSPLv1. 



> Under section "5. Conveying Modified Source Versions" the most relevant part 
> for
> us is section "a)".
> 
> a) The work must carry prominent notices stating that you modified it,
> and giving a relevant date.
> 
> Which means we need to give some heads-up that the source is changed.
> 

I looked at what changes are made:


  # Broken tls13 support, removing to fix build
  sed -i '/counts.tls13/d' src/mongo/util/net/ssl_manager_openssl.cpp

So I'd agree that is modified enough that the rest of the conditions apply.

My conclusion is, having this package in the repos would require too
much interpretation of a non-standard license to ensure compliance.
Drop the package.

Allan

Re: [arch-dev-public] TU application process

[Allan McRae via arch-dev-public]

On 6/11/18 8:15 pm, Bruno Pagani wrote:
> Le 06/11/2018 à 11:12, Christian Rebischke via arch-dev-public a écrit :
>> On Tue, Nov 06, 2018 at 08:09:03PM +1000, Allan McRae wrote:
>>> On 6/11/18 7:54 pm, Christian Rebischke via arch-dev-public wrote:
>>>> Hello everybody,
>>>>
>>>> First of all, the following mail has nothing to do with the last two TU
>>>> applications, it's a general view on the current TU application process.
>>>>
>>>> I would like to propose a new process for TU applications due to several
>>>> reasons:
>>>>
>>> Read the TU bylaws.  It has specific instructions of where proposals
>>> must be posted (hint: not here...).
>>>
>>> A
>> Hi Allan,
>>
>> This mail wasn't meant as proposal. It's just a general discussion about
>> this topic and people said in the TU IRC channel yesterday, that 
>> arch-dev-public would be the
>> right mailinglist for such discussion.
>>
>> chris
> Specifically, we are also interested in the input of devs, not just TUs.

Strange, given TUs are set-up to be an independently governed group from
developers...  But because you asked my opinion, I think a TU council is
a really, really, really bad idea.  No need to set some TUs above
others.  We have never had a formal hierarchy in the developers (apart
from our glorious leader), and are instead run by those who step up to
lead what needs done. I believe that this is what makes Arch work, and
governance would be detrimental to the distribution as a whole.

Personally, I'd get rid of all quorum for electing a TU, and make
inactive TUs be measured purely on the basis of package updating.  Most
TU application discussions are inane beyond the customary package
review.  And when someone applies and their packages are very bad, their
sponsor should be held in shame.

Finally, I don't want to hear what the minions are up to!  Get back to
your own mailing list. :)

A

Re: [arch-dev-public] TU application process

[Allan McRae via arch-dev-public]

On 6/11/18 7:54 pm, Christian Rebischke via arch-dev-public wrote:
> Hello everybody,
> 
> First of all, the following mail has nothing to do with the last two TU
> applications, it's a general view on the current TU application process.
> 
> I would like to propose a new process for TU applications due to several
> reasons:
> 

Read the TU bylaws.  It has specific instructions of where proposals
must be posted (hint: not here...).

A

Re: [arch-dev-public] .gnuog owned by root when using devtools in first place

[Allan McRae via arch-dev-public]

On 13/10/18 7:23 pm, NicoHood wrote:
> I ran extra-x86_64-build on a new system with an uninitialized gnupg
> keyring. I think it was responsible for creating a .~/gnupg directory,
> but with root privilegs.
> 
> I chowned it to my user, it is all good now. Maybe the script could be
> improved to check if the gnupg keyring was initialzed or not, to prevent
> such issues. I was confused in the first place.

If only there was a place to report bugs.  Somewhere where they could be
tracked...

A

Re: [arch-dev-public] Moving gcc7 into [community] for CUDA

[Allan McRae via arch-dev-public]

On 29/05/18 11:00, Sven-Hendrik Haase wrote:
> Hey,
> 
> I would like to move gcc7 (based on the latest version 7 commit of gcc [0])
> into [community] because of cuda 9.2 and in return drop gcc54.
> 
> I tried to make it work with current gcc but to no avail. In earlier
> releases of cuda the incompatibilities could be patched with header hacks
> but not this time. I tested the whole setup with cuda 9.2 locally and it
> seems to work fine.
> 
> I just want to get somebody's ok on this as apparently not everyone is
> stoked about having yet another old version of gcc in there for some time.
> 
> Sven
> 
> [0]
> https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/gcc=44e0a03db1b5cabf6135f194e540d513cf959245
> .
> 

OK to add gcc7 given it will drop gcc54.

A

Re: [arch-dev-public] RFC: Dropping -DCMAKE_BUILD_TYPE from packages using cmake

[Allan McRae via arch-dev-public]

On 14/03/18 04:19, Bartłomiej Piotrowski via arch-dev-public wrote:
> On 2018-03-13 14:40, Allan McRae via arch-dev-public wrote:
>> On 13/03/18 21:22, Jan Alexander Steffens via arch-dev-public wrote:
>>> For that matter, I'm all for putting an arch-configure helper into our
>>> autoconf package.
>>
>> Don't muck around with vanilla packages.  Put it in another package
>> (devtools).
>>
>> A
>>
> 
> Makes no sense unless devtools become part of base-devel. It's hardly
> any meddling as it doesn't hinder possibility to use default configure
> script or meson binary.
> 

It is Arch specific, so should be in an Arch specific package.

A

Re: [arch-dev-public] RFC: Dropping -DCMAKE_BUILD_TYPE from packages using cmake

[Allan McRae via arch-dev-public]

On 13/03/18 21:22, Jan Alexander Steffens via arch-dev-public wrote:
> For that matter, I'm all for putting an arch-configure helper into our
> autoconf package.

Don't muck around with vanilla packages.  Put it in another package
(devtools).

A

Re: [arch-dev-public] Automating package conflict resolution whenever possible (xorgproto/libxfont dependency conflict)

[Allan McRae via arch-dev-public]

On 14/02/18 22:28, Florian Pritz via arch-dev-public wrote:
> How about having this feature in pacman, maybe with an indicator if the
> package is still in a repository?

pacman -Qtd

Re: [arch-dev-public] Automating package conflict resolution whenever possible (xorgproto/libxfont dependency conflict)

[Allan McRae via arch-dev-public]

Just because I had to look up the details of this


- xorgproto replaces a lot of packages, including fontsproto:
 :: Replace fontsproto with extra/xorgproto? [Y/n]

- libxfont requires fontsproto, so this causes the following:
 :: libxfont: removing fontsproto breaks dependency 'fontsproto>=2.1.3'


- people with systems older than 2016-11-16, may have libxfont as
xorg-server depended on it until that time.  Now xorg-server depends on
libxfont2.

- libxfont2 does not replace libxfont, as it is a completely different API.

- libxfont was removed from the Arch repos somewhere in April or May
2017.  So nothing official depends on libxfont.


I don't think it is unreasonable to expect people to run "pacman -Qi
libxfont", see it was installed as a dependency and no package depends
on it and remove it.  There also does not seem to be a correct way of us
to handle this - joys of rolling release!


Funny thing...  people using yaourt probably removed this package as I
believe it highlights dependencies that are no longer needed after an
upgrade!

A

Re: [arch-dev-public] Package group between repositories

[Allan McRae via arch-dev-public]

On 04/01/18 22:02, Balló György via arch-dev-public wrote:
> Currently the 'xfce4-goodies' package group[1] is in split between [extra]
> and [community]. Most of its packages are in [extra], but 'ristretto' and
> 'xfce4-whiskermenu-plugin' are in [community].
> 
> My question is: is it okay, or should we avoid it by removing these two
> packages from the 'xfce4-goodies' group or moving them to [extra]?
> 
> [1] https://www.archlinux.org/groups/x86_64/xfce4-goodies/
> 

If a TU is maintaining the packages in [community], there is little
point moving them.

A

Re: [arch-dev-public] Merging multilib toolchain with [core]

[Allan McRae]

On 22/11/17 21:19, Bartłomiej Piotrowski wrote:
> Hi,
> 
> For a long time we have been maintaining multilib-enabled gcc package
> separately under [multilib] repo. Compilation takes a lot and usually it
> falls behind because I forget to keep it in sync with [core].
> 
> I propose to merge it with core/gcc instead and split lib32-gcc-libs.
> The latter would be moved to [core] due to dbscripts inability to
> maintain split packages in multiple repositories. On the same note, the
> same would happen to lib32-glibc.
> 

I was planning this for once i686 was dropped.  So good to see it happening!

A

Re: [arch-dev-public] Switching the bugtracker to Bugzilla

[Allan McRae]

On 15/11/17 07:34, Bartłomiej Piotrowski wrote:
>> There are several options for migrating the bug history to Bugzilla and a 
>> few options are under
>> debate. (input welcome)
> As I said multiple times on IRC, I'm for starting from scratch. There
> are way too many inactive or/and incorrect bugs open, and honestly any
> effort to review that list is a waste of time. With no bugs open we can
> 1) pretend everything works fine 2) hopefully avoid zombie-bugs
> apocalypse that we have now. Flyspray could be mirrored with wget for
> read-only version.

If you don't migrate pacman bugs, don't bother creating a pacman component.

A

Re: [arch-dev-public] [draft] The end of i686 support

[Allan McRae]

On 06/11/17 21:16, Eli Schwartz wrote:
> On 11/06/2017 05:36 AM, Alad Wenter via arch-dev-public wrote:
>>> Bartłomiej Piotrowski  hat am 6.
>>> November 2017 um 11:21 geschrieben:
>>>
>>> Slightly changing the topic... We have plenty of space on our 
>>> PIA-sponsored mirrors. Given that said fork pretty strictly follows
>>> our PKGBUILDs (much alike to ARM team), I'd like to host arch32
>>> mirrors there as well. What do you think?
>>>
>> I don't mind, but in the end it's up to those who pay for the
>> mirrors.
>>
>> It does bring up the topic again on how the Arch community will
>> support arch32. Does hosting arch32 mirrors give the impression that
>> we support the fork through our channels, or is that unrelated? How
>> will we otherwise react on support requests for or from arch32? IMO,
>> the announcement is vague on that.
>>
>> (Personally I would support the idea of having both projects under a
>> common umbrella. But by now arch32 has their own support
>> infrastructure, including forums).
> 
> Well, I doubt they wanted to be caught by surprise and have nothing
> ready if we decided not to allow support requests for arch32...
> 
> But if we are willing to allow arch32 to be hosted under our umbrella,
> the presence of separate infrastructure should not IMHO cause us to go
> back on that and therefore cause additional fragmentation that we were
> initially okay with avoiding.
> 

In all my time here, I can remember one i686 bug that did not also
affect x86_64.  That suggests a common infrastructure is warranted.

A

Re: [arch-dev-public] switching to systemd-stable

[Allan McRae]

On 06/07/17 17:44, NicoHood wrote:
> ArchLinux

At least spell the name of the distro correctly.  It is the simple
addition of one space character between "Arch" and "Linux".  But I see
it is easy for people to forget about silly one character issues.

A

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 02/07/17 06:51, Bartłomiej Piotrowski wrote:
> On 2017-06-30 23:44, Allan McRae wrote:
>> On 30/06/17 19:07, Bartłomiej Piotrowski wrote:
>>> On 2016-10-24 05:56, Allan McRae wrote:
>>>> 1) building gcc to enable PIE by default
>>>
>>> I am in the middle of rebuilding gcc with --enable-default-pie. When it
>>> finishes, I will start a todo for rebuilding packages with static libraries.
>>>
>>> I also enabled --enable-default-ssp, which means that
>>> -fstack-protector-strong will be dropped from our CFLAGS (as it will be
>>> enforced by gcc) on the next opportunity.
>>>
>>
>> Are you adding full RELRO + no-plt at the same time?
>>
>> A
>>
> 
> Yes, and -fstack-check=specific too, although I might drop no-plt if it
> will cause too many builders.
> 

I thought the conclusion from the Stack Clash bugs was that the current
-fstack-check was fundamentally flawed and was being completely
rewritten for the next gcc.  Is the "=specific" version OK?

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 30/06/17 19:07, Bartłomiej Piotrowski wrote:
> On 2016-10-24 05:56, Allan McRae wrote:
>> 1) building gcc to enable PIE by default
> 
> I am in the middle of rebuilding gcc with --enable-default-pie. When it
> finishes, I will start a todo for rebuilding packages with static libraries.
> 
> I also enabled --enable-default-ssp, which means that
> -fstack-protector-strong will be dropped from our CFLAGS (as it will be
> enforced by gcc) on the next opportunity.
> 

Are you adding full RELRO + no-plt at the same time?

A

Re: [arch-dev-public] libalpm hook and wrapper for detecting broken perl modules

[Allan McRae]

On 11/06/17 08:03, Florian Pritz via arch-dev-public wrote:
> One possibility to counter this would be to replace the perl executable
> with a wrapper that performs the check and prints errors to stderr.
> Optionally it could also exit with an error code rather than continuing
> to run the script. I'm not sure if I want this or not, but I'm leaning
> towards a yes since exiting will make the error more difficult to miss.

I am very against this part (hook is completely fine).   Having the perl
executable be something other than upstream (and every other distro)
provides, is not what Arch is about.

I'm assuming the new location would be transparent to packagers (i.e.
they would not need to alter their PKGBUILDs)?

A

Re: [arch-dev-public] OpenSSL 1.1.0

[Allan McRae]

On 23/04/17 08:07, Gaetan Bisson wrote:
> [2017-04-22 18:05:27 +0200] Sébastien Luttringer:
>> When do you plan to move openssl rebuild out of testing?
> 
> Quoting arojas on IRC:
> 
> 2017-04-20 09:11:27 arojas: current blocker for openssl if FS#53618
> 2017-04-20 09:11:47 arojas: someone needs to decide whether we care about it 
> or not, and if yes do something to fix it
> 

Given there is a workaround, a news item should be posted and we should
stop blocking the entire distribution with this rebuild.

Allan

Re: [arch-dev-public] Resigning from package maintenance

[Allan McRae]

On 12/04/17 17:28, Bartłomiej Piotrowski wrote:
> On 2017-04-12 03:07, Allan McRae wrote:
>> Dealing with packaging has become a chore, so I will not be doing that
>> for Arch anymore.  I will continue maintaining the pacman codebase.
>>
>> Here is a list of packages up for adoption:
>>
>> Core [15]:
>>  autoconf
>>  automake
>>  binutils
>>  bison
>>  flex
>>  gcc (gcc-ada, gcc-fortran, gcc-go, gcc-libs, gcc-objc)
>>  glibc
>>  gmp
>>  libmpc
>>  libtool
>>  linux-api-headers
>>  m4
>>  make
>>  mpfr
>>  pkg-config
>>
>> Extra [3]:
>>  dejagnu
>>  expect
>>  fakechroot
>>
>> Allan
>>
> 
> 
> Adopted gcc, glibc, gmp, pkg-config and packages from extra. I'm happy
> to see someone co-maintain these with me.
> 

If you have gcc and glibc, you need to take linux-api-headers and
binutils.  They are a set.

A

[arch-dev-public] Resigning from package maintenance

[Allan McRae]

Dealing with packaging has become a chore, so I will not be doing that
for Arch anymore.  I will continue maintaining the pacman codebase.

Here is a list of packages up for adoption:

Core [15]:
autoconf
automake
binutils
bison
flex
gcc (gcc-ada, gcc-fortran, gcc-go, gcc-libs, gcc-objc)
glibc
gmp
libmpc
libtool
linux-api-headers
m4
make
mpfr
pkg-config

Extra [3]:
dejagnu
expect
fakechroot

Allan



signature.asc
Description: OpenPGP digital signature

Re: [arch-dev-public] Remaining TODO lists - Part 1: Packages with missing sources

[Allan McRae]

On 26/03/17 11:51, Allan McRae wrote:
> We have a number of TODO lists that have not been completed.
> 
> The "Packages with missing sources" list [1] was created more than six
> months ago (2016-08-09).  These packages are either genuinely no longer
> have an upstream source, or are just sitting unmaintained in our repos.
> Either way, the should not be in our repos.
> 
> I will remove any package still on that list in a week.
> 

A bit over a week later, and there are four packages remaining:

mangler
mashup
multibit
ozerocdoff

Last chance!

[arch-dev-public] Remaining TODO lists - Part 1: Packages with missing sources

[Allan McRae]

We have a number of TODO lists that have not been completed.

The "Packages with missing sources" list [1] was created more than six
months ago (2016-08-09).  These packages are either genuinely no longer
have an upstream source, or are just sitting unmaintained in our repos.
Either way, the should not be in our repos.

I will remove any package still on that list in a week.

Allan


[1] https://www.archlinux.org/todo/packages-with-missing-sources/

Re: [arch-dev-public] [RFC] Deprecate ABS?

[Allan McRae]

On 25/03/17 21:24, Bartłomiej Piotrowski wrote:
> Hi all,
> 
> Allan complained yesterday that ABS is apparently broken. Turns out that
> most likely it is broken at least since 2016-07-07.
> 
> The script received no changes since 2012-09-07 and the main advantage
> it brings is an easy way to pull all PKGBUILDs. But we also provide
> public Subversion and Git repositories. Initial svn checkout for
> community and packages takes less than 3 minutes on my rather slow
> connection (for European standards); obviously subsequent updates are
> faster. Full git clone is slow, but shallow one takes ca. 10 seconds
> (for both).
> 
> If someone needs to checkout just one package, svn has sparse checkouts
> and for git, Dave wrote asp[1]. This brings the question, what is the
> point of keeping it running?
> 

Can our server handle the load of svn/git checkouts? That was the main
concern when we moved to SVN many years ago and the reason ABS was kept.

I have been working on and off to include source packages into a pacman
database to allow pacman to directly replace these.

Allan

Re: [arch-dev-public] AUR ToS (aka making AUR user names public)

[Allan McRae]

On 05/03/17 23:35, Lukas Fleischer wrote:
> Hi,
> 
> I was recently contacted by a Polish researcher asking for a list of AUR
> account names.

Please share this data.  It looks like this person is doing genuine
research into networks within open-source software communities.

Providing easy access to already public data should be encouraged in all
fields of research.

Allan

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 02/02/17 05:06, Bartłomiej Piotrowski wrote:
> On 2016-12-24 08:39, Allan McRae wrote:
>> If default PIE is wanted, someone other than me will need to figure this
>> out.  Just build gcc with --enable-default-pie and then build binutils
>> to see the issues in the testsuite.
>>
>> Allan
>>
> 
> Default PIE is not everything you planned to implement. What about the rest?
> 

Can't be fucked doing this twice.

A

Re: [arch-dev-public] OpenSSL 1.1.0

[Allan McRae]

On 30/01/17 08:30, Giancarlo Razzolini wrote:
> Em janeiro 29, 2017 20:04 Doug Newgard escreveu:
>>
>> I haven't heard all that much from/about LibreSSL since shortly after
>> the fork.
>> Care to share what advantages it would bring, and at what cost?
>>
> 
> The cost for rebuilding everything against OpenSSL 1.1 will probably be
> a big one.
> For LibreSSL, it would be even bigger. I think the main advantage, right
> away, is
> that LibreSSL has a considerably better security track, specially after
> their huge
> flensing.
> 
> I can only dream about the bugs that might lurk on both OpenSSL 1.1 and
> LibreSSL.
> But the defensive approach OpenBSD takes on LibreSSL already has paid
> off in terms
> of CVE's that didn't affected it, but were high/critical issues on OpenSSL.
> 

Please cite one example.   Every CVE I have seen that is of at least
high severity has affected both.  There have been some low severity ones
only affecting openssl.

Even worse, the fix time for libressl in the couple of issues I
monitored was worse than openssl.

A

Re: [arch-dev-public] News draft for i686 deprecation

[Allan McRae]

On 24/01/17 07:09, Bartłomiej Piotrowski wrote:
> What I infer from my conversation with Allan is that
> neither of us is interested in leading this personally.

FYI, I will provide advise/mentorship to any team that forms to continue
this, but definitely can not take on leading it.

A

Re: [arch-dev-public] [RFC] Send signoff reports somewhere else (or not at all)

[Allan McRae]

On 17/01/17 03:50, Giancarlo Razzolini wrote:
> Em janeiro 3, 2017 6:45 Giancarlo Razzolini escreveu:
>> Em janeiro 3, 2017 6:29 Allan McRae escreveu:
>>>>
>>>
>>> I'm all for making declaration by lack of objection, but you should wait
>>> more than 16 hours.  Particularly at this time of year...
>>>
> 
> Can I now declare signoff reports dead, by lack of objection?
> 

Yes

Re: [arch-dev-public] Shadowing i686, round 1

[Allan McRae]

On 06/01/17 07:08, Bartłomiej Piotrowski wrote:
> Friendly reminder what I'm going to push through if nobody opposes.

Can we have a detailed discussion what EOL for i686 means? I have seen a
couple of proposals being:

1) We just kill i686 (as in stop distributing packages), or
2) Reduce i686 to a secondary architecture.

I want to see secondary architectures happen, so this would be a good
start in that direction.  We start by advertising for a team that is
responsible for building and fixing i686 packages.  At a certain point,
the only change to our tooling is that developers upload x86_64 only,
and the other team plays catchup (which should lead to autobuilding...)

Allan

Re: [arch-dev-public] [RFC] Send signoff reports somewhere else (or not at all)

[Allan McRae]

On 03/01/17 18:11, Giancarlo Razzolini wrote:
> Em janeiro 3, 2017 0:33 Sébastien Luttringer escreveu:
>>>
>> Please drop them.
>>
>> Cheers,
>>
> 
> By the action of inaction, they will not be sent anymore.
> 

I'm all for making declaration by lack of objection, but you should wait
more than 16 hours.  Particularly at this time of year...

A

Re: [arch-dev-public] FrOSCon 2017

[Allan McRae]

On 02/01/17 02:24, Bartłomiej Piotrowski wrote:
> Bonn is about 8 hours away by train from me so I'm not really eager to
> travel that long.

You think that's bad...

Re: [arch-dev-public] Shadowing i686, round 1

[Allan McRae]

On 14/12/16 08:05, NicoHood wrote:
> On 12/13/2016 08:04 PM, Balló György via arch-dev-public wrote:
>> 2016. 12.  13, kedd keltezéssel 12.29-kor Doug Newgard ezt írta:
>>> On Tue, 13 Dec 2016 19:16:53 +0100
>>> Balló György via arch-dev-public 
>>> wrote:
>>>
 -1 for dropping i686 completely.

 +1 for introducing automated builds, even if it's less secure.

 I would like to keep i686 supported, and willing to do anything
 that is
 needed to setup and maintain an official automated build server for
 i686 packages (and possibly for other architectures).
>>>
>>> I've got to ask, why do you feel so strongly about it? It's been
>>> pointed out
>>> that i686 really doesn't fit in with the original goals of Arch
>>> anymore, is less
>>> and less supported upstream, and essentially untested. What is the
>>> compelling
>>> reason for keeping it around?
>>
>> Because I still use an i686-only system occasionally, and I prefer to
>> keep old hardware working with my favourite distribution. I agree that
>> building packages manually for a small percentage of users is
>> pointless. But most of the packages can be built for i686 without any
>> modifications. We just need an automated build server, which takes the
>> job.
>>
>> --
>> György Balló
>> Trusted User
>>
> 
> If we have reproduceable builds we could also use this buildserver to
> build the x64 packages. I know that this is a huge task, but then we
> could automate the package building better in a centralized place. And
> instead of dropping i686 we could integrate arm as well.
> 
> Those will not be officially supported, but we could give people access
> to fix those arch specific problems. Normal maintainers can focus on x64
> development while some others have a place to distribute and maintain
> other architectures of their favorite os.
> 

If we had a build server, I'd prefer we concentrated on Arch's original
goals.  To provide a more optimised distribution than everyone else.
All other distros have caught up to us these days. Some more optimised
x86_64 variants would be good...

A

Re: [arch-dev-public] Shadowing i686, round 1

[Allan McRae]

On 13/12/16 23:15, Florian Pritz via arch-dev-public wrote:
> I still have two media players that run i686, but I can switch those to
> a different distro if necessary although I'd obviously prefer not having
> to do that. Count that as a -0.25.

I have one webserver running i686.  My vote counts as -1   :)

A

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 13/12/16 19:23, Jan Alexander Steffens via arch-dev-public wrote:
> On Mon, Oct 24, 2016 at 5:56 AM Allan McRae <al...@archlinux.org> wrote:
> 
>> Hi all,
>>
>> The results from the test-sec-flags [1] suite are in.  Many thanks to
>> those that wrote this and those that submitted results.  I'm not going
>> to list the summaries here, but the results show that at worst enabling
>> a bunch of security flags in our packages will have a <1% impact on
>> performance (and more likely a fraction of a percent).
>>
>> That means we will add all of these to our default CFLAGS/LDFLAGS etc.
>> The changes are:
>>
>> 1) building gcc to enable PIE by default
>> 2) add -z,now to LDFLAGS
>> 3) and -fno-plt and -fstack-check to our CFLAGS
>>
>> Adding PIE means that programs get loaded at a random address,
>> preventing an attacker from manipulating global data or hijacking
>> control by reusing code.  Without this, ASLR is ineffective.  Enabling
>> this by default in our compiler (there is a configure flag) means we
>> will need to rebuild all packages with static libraries (which should be
>> a fairly limited set).
>>
>> Adding -z,now to LDFLAGS disables lazy loading.  This means all function
>> symbols are loaded at startup (with minor performance hit), but that
>> means our RELRO support will make everything ro instead of some of the
>> things.  Doing this enables us to use -fno-plt in our CFLAGS, which is a
>> run-time optimisation allowing faster use of libraries.
>>
>> Adding -fstack-check to our CFLAGS guarantees stack overflows aren't
>> exploitable.
>>
>> Note that any of these flags can be disabled in a PKGBUILD if really
>> needed...  But if that is the case, bug reports should be filed.
>>
>>
>> Given an assumed lack of objection, I will enable the build flags in our
>> pacman.conf and rebuild gcc to enable pie and put them in [staging] at
>> the end of this week (what better way to celebrate Halloween).  We will
>> need a new devtools release then too.  Then the packages with static
>> libraries will need rebuilt.
>>
>> After that, I would like to see [core] completely rebuilt, and audited
>> to ensure our CFLAGS/LDFLAGS are actually being used in the build.
>>
>> Cheers,
>> Allan
>>
> 
> Will this affect i686 as well? According to this commit (
> https://github.com/zen-kernel/zen-kernel/commit/cc701dc61a7187e9dfc300ad6ec3b7a677dc4717)
> at least Ubuntu seems to have skipped that for now.

That commit shows they disabled it for one package.

It will affect both.

A

Re: [arch-dev-public] Shadowing i686, round 1

[Allan McRae]

On 13/12/16 19:20, Bartłomiej Piotrowski wrote:
> On 2016-12-12 22:02, Jelle van der Waa wrote:
>> I'm in favor of an auto build solution, since this has multiple
>> bonusses. We could extend the auto build solution for reproducible
>> builds (yay!). Auto rebuilds and maybe later when vendors get their act
>> togehter (aarch64 *cough*).
> 
> Creating a new or adapting existing solutions is time consuming. We
> struggle to finish and switch to VCS-agnostic dbscripts for a long time
> and I wouldn't expect that any automated build server will take less
> time, not to mention security concerns expressed in previous
> discussions. On the opposite, dropping support is effortless.

Given we can't get databases signed, I can't ever see an autobuild
system working.

A

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 13/12/16 19:10, Daniel Micay via arch-dev-public wrote:
>> I need time to fix this.  It is probably just test suite assumptions
>> rather than errors.
> 
> Could we start with the changes other than PIE? Those won't need any
> immediate rebuilds.

I'm not dealing with this twice.

Re: [arch-dev-public] Changing compilation flags

[Allan McRae]

On 24/10/16 13:56, Allan McRae wrote:
> That means we will add all of these to our default CFLAGS/LDFLAGS etc.
> The changes are:
> 
> 1) building gcc to enable PIE by default
> 2) add -z,now to LDFLAGS
> 3) and -fno-plt and -fstack-check to our CFLAGS

For those wondering what happened with this:

binutils build with gcc with default PIE:

# grep Error binutils-git-89080.bfbf34d-1-x86_64-check.log
make[5]: *** [Makefile:7161: incremental_test_2] Error 1
make[5]: *** [Makefile:7185: incremental_test_5] Error 1
make[5]: *** [Makefile:7200: incremental_copy_test] Error 1
make[5]: *** [Makefile:7206: incremental_common_test_1] Error 1
make[5]: *** [Makefile:7132: ehdr_start_test_4] Error 1
readelf: Error: the PHDR segment is not covered by a LOAD segment
make[4]: *** [Makefile:5609: check-am] Error 2
make[3]: *** [Makefile:5613: check] Error 2
make[2]: *** [Makefile:941: check-recursive] Error 1
make[1]: *** [Makefile:6134: check-gold] Error 2
make[5]: *** [Makefile:3678: check-DEJAGNU] Error 1
make[4]: *** [Makefile:1953: check-am] Error 2
make[3]: *** [Makefile:1793: check-recursive] Error 1
make[2]: *** [Makefile:1955: check] Error 2
make[1]: *** [Makefile:7547: check-ld] Error 2
make: *** [Makefile:2206: do-check] Error 2


binutils build with current gcc:

# grep Error binutils-git-89080.bfbf34d-1-x86_64-check.log
readelf: Error: the PHDR segment is not covered by a LOAD segment


I need time to fix this.  It is probably just test suite assumptions
rather than errors.

A

[arch-dev-public] Changing compilation flags

[Allan McRae]

Hi all,

The results from the test-sec-flags [1] suite are in.  Many thanks to
those that wrote this and those that submitted results.  I'm not going
to list the summaries here, but the results show that at worst enabling
a bunch of security flags in our packages will have a <1% impact on
performance (and more likely a fraction of a percent).

That means we will add all of these to our default CFLAGS/LDFLAGS etc.
The changes are:

1) building gcc to enable PIE by default
2) add -z,now to LDFLAGS
3) and -fno-plt and -fstack-check to our CFLAGS

Adding PIE means that programs get loaded at a random address,
preventing an attacker from manipulating global data or hijacking
control by reusing code.  Without this, ASLR is ineffective.  Enabling
this by default in our compiler (there is a configure flag) means we
will need to rebuild all packages with static libraries (which should be
a fairly limited set).

Adding -z,now to LDFLAGS disables lazy loading.  This means all function
symbols are loaded at startup (with minor performance hit), but that
means our RELRO support will make everything ro instead of some of the
things.  Doing this enables us to use -fno-plt in our CFLAGS, which is a
run-time optimisation allowing faster use of libraries.

Adding -fstack-check to our CFLAGS guarantees stack overflows aren't
exploitable.

Note that any of these flags can be disabled in a PKGBUILD if really
needed...  But if that is the case, bug reports should be filed.


Given an assumed lack of objection, I will enable the build flags in our
pacman.conf and rebuild gcc to enable pie and put them in [staging] at
the end of this week (what better way to celebrate Halloween).  We will
need a new devtools release then too.  Then the packages with static
libraries will need rebuilt.

After that, I would like to see [core] completely rebuilt, and audited
to ensure our CFLAGS/LDFLAGS are actually being used in the build.

Cheers,
Allan


[1] https://www.archlinux.org/news/test-sec-flags-call-for-assistance/

Re: [arch-dev-public] Long out of date packages

[Allan McRae]

On 20/10/16 22:25, Florian Pritz via arch-dev-public wrote:
> 
> allan:
> extra/x86_64/fakechroot

Updating this breaks the pacman testsuite (due to a fakechroot bug...).

Re: [arch-dev-public] i686 and SSE2

[Allan McRae]

Just to summarise what I have taken from discussion surrounding this so
it does not stagnate:

1) Changing i686 to either i686 + SSE or i786 is not seen to be a good idea

2) There are suggestions we look into providing something more optimised
than x86_64, but it not clear what platform(s) people are looking to
support.

3) Autobuilding more the optimised platforms would be great, but runs
into the same issue we have with database signing and security of such a
key.



The most pressing issue is #1.  So lets get back to the issue of the
increasing amount of software requiring SSE2.  How do we solve this?

It would be "simple" to add an SSE2 detection hook into the filesystem
package that aborts any pacman transaction attempting to install a list
of packages on i686 systems without SSE2 support.  Is that a viable
solution?

Any other suggestions?

Allan

Re: [arch-dev-public] i686 and SSE2

[Allan McRae]

On 20/09/16 04:30, Ike Devolder via arch-dev-public wrote:
> Someone mentioned AVX2, but it seems there is still a patch needed for
> glibc. Solus added it very recently https://dev.solus-project.com/T503
> and they have taken it from Clear Linux.

Nothing needs done for glibc to use AVX.  It already does when present.

Allan

Re: [arch-dev-public] i686 and SSE2

[Allan McRae]

On 19/09/16 23:14, Balló György via arch-dev-public wrote:
> 2016-09-19 7:02 GMT+02:00 Allan McRae <al...@archlinux.org>:
> 
>> This goes beyond just adding SSE2 support.
>>
>> Years ago, Arch Linux was "optimised for modern processors".  These were
>> the days when every other distro was using i386 and we had a blazingly
>> fast i686 port.  Now every other distro uses i686 while we have sat
>> still.  Even major software developments are starting to require SSE2.
>> It is time we moved forward.
>>
>> How can we achieve this?  I see several options:
>>
>> 1) Do "nothing".  Add a hook to the filesystem package that detects
>> whether a system has SSE2 support and blocks installation of certain
>> packages.
>>
>> 2) Add SSE2 to our optimisations and require "i686 + SSE2"
>>
>> 3) Move our minimum CPU to something less than 20 years old  (even i786
>> would get us SSE2+3 instructions and is 15 years old)
>>
>> 4) We add more modern CPU builds  (and set them automatically building
>> once the base architecture is updated).
>>
>>
>> I am in favour of #3 for our 32-bit support.  And that would be end of
>> line as far as 32 bit support in this distribution goes.
>>
>>
>> (We may want to consider #4 for our x86_64, but that is another
>> conversation).
>>
>> Allan
>>
> 
> 
> I would not be happy with #3, because I still have two 13 years old systems
> with NetBurst-based CPUs without SSE3 support. But of course I don't use
> them in everyday use.
> 

If we limit our choice based on your CPU, then we need to limit based on
the other CPU mentioned in this thread.

That should not be a consideration at all. What we need to do is think
about what make our distribution worthy of being a distribution.
Original the selling points were rolling release, vanilla packages and
optimised binaries.  We have lost the latter.  Do we want to get it back?

Allan

Re: [arch-dev-public] i686 and SSE2

[Allan McRae]

On 19/09/16 19:05, Christian Hesse wrote:
> Bartłomiej Piotrowski  on Fri, 2016/09/16 21:44:
>> Actually, why don't raise the bar higher? SSE2 has been introduced in
>> 2001 – that's 15 years to upgrade one's hardware and given my sad
>> experiences with computers, I find it hard to believe anyone has that
>> old PC that happens to run Arch.
> 
> I do. Running Arch Linux on a bunch of informational room displays...
> These are based on Geode CPUs and I am pretty sure no SSE2 is available.
> (Taking a look at /proc/cpuinfo these devices do not even feature SSE... The
> CPU identifies itself as i586.)
> 
> That said I will be able to handle that myself. ;)
> Possibly I will have to do local rebuilds of webkitgtk2 to make surf run on
> these devices from time to time.

If we adopt SSE2 in any form, you will need to rebuild your entire
system to use it.  It will be used in optimizations everywhere.

A

Re: [arch-dev-public] i686 and SSE2

[Allan McRae]

This goes beyond just adding SSE2 support.

Years ago, Arch Linux was "optimised for modern processors".  These were
the days when every other distro was using i386 and we had a blazingly
fast i686 port.  Now every other distro uses i686 while we have sat
still.  Even major software developments are starting to require SSE2.
It is time we moved forward.

How can we achieve this?  I see several options:

1) Do "nothing".  Add a hook to the filesystem package that detects
whether a system has SSE2 support and blocks installation of certain
packages.

2) Add SSE2 to our optimisations and require "i686 + SSE2"

3) Move our minimum CPU to something less than 20 years old  (even i786
would get us SSE2+3 instructions and is 15 years old)

4) We add more modern CPU builds  (and set them automatically building
once the base architecture is updated).


I am in favour of #3 for our 32-bit support.  And that would be end of
line as far as 32 bit support in this distribution goes.


(We may want to consider #4 for our x86_64, but that is another
conversation).

Allan

Re: [arch-dev-public] Long out of date packages

[Allan McRae]

On 18/08/16 05:52, Christian Hesse wrote:
> Unflag the package, then flag it yourself with a comment of the details. At
> least devs can find the information there.

As a "bonus", unflagged then reflagging makes the package look as if it
has only been out-of-date for 1 day...

[arch-dev-public] Discussion about optional dependencies

[Allan McRae]

Hi all,

We need to have a discussion about optional dependencies.  I am going to
use virtualbox as an example, because of the 10 bugs and counting caused
by people not seeing the change in dependency to run the gui.  However,
this is just the latest in the list of such issues.

So lets look at the optional dependencies:

Optional Deps   : qt5-x11extras: GUI support
  vde2: Virtual Distributed Ethernet support
  virtualbox-guest-iso: Guest Additions CD image
  virtualbox-ext-vnc: VNC server support
  virtualbox-sdk: Developer kit
  net-tools: Host-only or bridged networking support

The optional dependency of note here is "qt5-x11extras: GUI support".,
which is needed to run /usr/bin/virtualbox. That is the primary "binary"
of the virtualbox package.  My opinion is the primary binary for a
package should run out of the box.  If you really need to reduce
dependencies, then a split package should be considered.

Comments/opinions?

Thanks,
Allan

Re: [arch-dev-public] signoffs are dead

[Allan McRae]

On 02/07/16 06:18, Christian Hesse wrote:
> Gaetan Bisson  on Tue, 2016/06/28 08:09:
>> For a while now packages in [testing] have gotten little to no signoffs
>> and I've been moving mine to [core] after a week without feedback. I
>> suspect many of you have been doing this too.
> 
> Yes, probably everybody does. ;)
> 
> I do not run a system with [testing] enabled by default. I need my main
> system in production every day - stable and reliable.
> Packages that I really do want and/or need end up in my personal repository
> for testing, though. I do give spare signoffs, but that packages received
> real testing then.
> 
> Possibly we should modify the process a bit: Expect a package to be fine
> when it received enough signoffs - as is. Additionally add a NACK feature that
> expresses something is (possibly) borked. If the package did not receive a
> NACK within 48 or 72 hours it is expected to be fine as well.
> Our bug wrangler Doug could have an eye on bugs that look critical and set
> NACKs for the packages.

This sounds like the Fedora policy where packages have to surpass a
certain karma level to move into the main repositories.  I'm not sure
who gets to vote for that though.

A

Re: [arch-dev-public] test-sec-flags announcement

[Allan McRae]

On 27/05/16 07:19, Lukas Jirkovsky wrote:
> On 25 May 2016 at 03:34, Allan McRae <al...@archlinux.org> wrote:
>> Hi all,
>>
>> There has been a group of people doing some analysis on the performance
>> impact of various security flags we can use in our packaging.  They are
>> wanting to gather performance benchmarks from a bunch of people so we
>> can have an informed discussion about their inclusion.  See below.
> 
> I'm missing one important piece of information, and that is what to do
> with the results. There are some results on the github wiki, but
> there's no obvious point where to submit the results.
> 

>From the README:

"Please add your results to the wiki as well, preferably maintaining the
same format as the results already there."

[arch-dev-public] test-sec-flags announcement

[Allan McRae]

Hi all,

There has been a group of people doing some analysis on the performance
impact of various security flags we can use in our packaging.  They are
wanting to gather performance benchmarks from a bunch of people so we
can have an informed discussion about their inclusion.  See below.

I'll wait and see how many responses they get from this post before I
post this as a news item too.

Cheers,
Allan


--ANNOUNCEMENT--


I am thrilled to announce that test-sec-flags has been completed and is
ready for download and use.

What is test-sec-flags?

Inspired by discussions on arch-general, test-sec-flags was created to
test the performance impact of various security-oriented compilation and
linking flags. The goal is to determine if these flags can be the new
default for all Arch Linux packages. Preliminary results suggest that
the performance impact is almost nonexistent, but I would like to
collect and compare more results before moving forwards.

How do I use it?

See the README for installation and usage instructions.The results
subdirectory contains instructions on how to pull out the relevant
statistics from the result files.

Where do I obtain it?

https://github.com/pid1/test-sec-flags

My sincerest thanks to anthraxx, strcat, sangy, and rgacogne for their
invaluable assistance.

Patches welcome.

Cheers,
pid1

[arch-dev-public] Community Etiquette document

[Allan McRae]

Hi all,

There has been some talk about extending the Forum Etiquette document
into a combined community one.  There is a draft at:

https://wiki.archlinux.org/index.php/User:Jasonwryan/Community_Etiquette

Please give opinions (preferably on the Talk page).

Allan

Re: [arch-dev-public] Linux 4.6 enters [testing]

[Allan McRae]

On 17/05/16 04:58, Tobias Powalowski via arch-dev-public wrote:
> Hi folks,
> 
> New kernel is in [testing],
> 
> Binary modules which are not building:
> 
> -nvidia series (all of them)
> 
> -tp_smapi
> 

This package should have been pushed to [staging] if the module rebuild
was not complete.

A

[arch-dev-public] Changes to x86_64 triplet

[Allan McRae]

Hi all,

gcc-6 decided to enforce the vendor part of the target triplet to be not
"unknown".  This means our vendor string is changed from "unknown" to
"pc", aligning with the i686 triplet.

I have release a new pacman with the triplet changed in pacman.conf:

CHOST="x86_64-pc-linux-gnu"

This _should_ have very little effect (the vendor string is quite
unimportant).

Allan

Re: [arch-dev-public] Hooks rebuild #1

[Allan McRae]

On 28/04/16 21:47, Antonio Rojas wrote:
> Allan McRae wrote:
>>
>> You have obviously figured it out so who cares...
> 
> Yeah but there could be more packages affected
> 

That can be dealt with once the rebuild is done.  Then it will be super
easy to find missed packages in ABS.

Re: [arch-dev-public] Hooks rebuild #1

[Allan McRae]

On 28/04/16 21:28, Antonio Rojas wrote:
> Allan McRae wrote:
> 
>> On 28/04/16 21:18, Antonio Rojas wrote:
>>> Allan McRae wrote:
>>>
>>>> No idea...   I generated the list via a quick grep, so there are false
>>>> positives (more than I expected...).  Just mark them as done.
>>>>
>>>> A
>>>
>>> It seems that it also didn't catch install files from split packages, eg.
>>> kdepim or kdewebdev are missing from the todo list.
>>>
>>
>> Is the base package there?
> 
> No, neither the base package nor any of its subpackages.
> 

Hrm... it was on the list I uploaded.

$ grep kdepim rebuild.txt
kdepim
kdepimlibs
kdepimlibs4
kdepim-runtime

You have obviously figured it out so who cares...

A

Re: [arch-dev-public] Hooks rebuild #1

[Allan McRae]

On 28/04/16 21:18, Antonio Rojas wrote:
> Allan McRae wrote:
> 
>> No idea...   I generated the list via a quick grep, so there are false
>> positives (more than I expected...).  Just mark them as done.
>>
>> A
> 
> It seems that it also didn't catch install files from split packages, eg. 
> kdepim or kdewebdev are missing from the todo list.
> 

Is the base package there?

Re: [arch-dev-public] Hooks rebuild #1

[Allan McRae]

On 28/04/16 04:40, Anatol Pomozov wrote:
> Hi
> 
> And another related question: are we going to have hooks for
> systemd-sysusers and systemd-tmpfiles?
> 
> Quite a lot of packages packages use systemd-tmpfiles and it sounds
> like a good idea to move it to hooks.
> 

They are listed on the wiki page [1] and this rebuild was labelled as
"#1", so you could concluded there will be another rebuild...

[1] https://wiki.archlinux.org/index.php/DeveloperWiki:Pacman_Hooks

[arch-dev-public] Hooks rebuild #1

[Allan McRae]

We are ready to start the first hooks rebuild.  This rebuild covers
packages using these hooks:

update-desktop-database
update-mime-database
install-info
glib-compile-schemes
gtk-update-icon-cacne/xdg-icon-resource
gconf
gio-querymodules

Each rebuild requires the install file updated to remove these commands.

No need to use staging/testing for this rebuild (except [core] packages).

GO!

[arch-dev-public] Hooks!

[Allan McRae]

According to the news announcement, today is the day we can start using
hooks!  (as long as you live in the future like me - those of you in the
past will need to catch up a day).

I have started a wiki page to discuss which hooks to add [1].  So far
this includes "update-desktop-database", "update-mime-database", and
"info-install" hooks to be added. I also included a list of potential
other hooks.  I have no idea about these or how to create hooks for
them, so it is up to other people.

I'd like to get as many hooks as possible into our packages in one go
and then create rebuild lists for them.  This way, if a package can
benefit from multiple hooks, it will only need rebuilt once.

Please take a look at that page and add any hook that you think will be
useful to your packages.  I will dump the first three into packages in
the next couple of days and make rebuild lists.  These will not got to
[staging]/[testing] as there is no harm having the hook do some
duplicated effort until all the packages are rebuilt.

Cheers,
Allan


[1] https://wiki.archlinux.org/index.php/DeveloperWiki:Pacman_Hooks

Re: [arch-dev-public] Consensus: DKMS modules

[Allan McRae]

On 14/04/16 19:23, Ike Devolder wrote:
> - use separate repo [kernel-update{-testing,-staging}]

Why?   We have staging for rebuilds like these.

Re: [arch-dev-public] Conclusion: DKMS modules

[Allan McRae]

So, after all of this, we still only have DKMS modules in [community].

@Sébastien: Is this going to change any time soon?

A

Re: [arch-dev-public] Conclusion: DKMS modules

[Allan McRae]

On 24/03/16 07:28, Sébastien Luttringer wrote:
> On mer., 2016-03-23 at 13:08 +1000, Allan McRae wrote:
>> Having given everyone time to reply (not that many did...), the
>> consensus of those that replied is:
>>
>> - Binary modules are to be provided at minimum of all kernels in [core],
>> with preference to providing them for all supported kernels (noting that
>> out-of-tree modules may not work with some patched kernels).
>>
>> - There is no objection to providing DKMS modules in the repos, but this
>> is secondary to binary modules.  No opinions were stated on whether we
>> ensure all modules have DKMS variants in the repos.
>>
>> I decree by the power invested in me through talking the loudest, that
>> this is now our policy.
>>
> 
> Unexpectedly we got the most feedback from persons who are not dealing
> currently with the burden of managing kernels and their modules. Like you I 
> was
> expecting more opinions.
> 
> Even I support your will of moving forward regarding all this discussion, I
> don't share your conclusions nor the fact that one of us could speak lourder 
> to
> impose his will to others.

It was a joke.   Although those of us that have been around here 10
years have loud voices...

> I have few more subjects to discuss altogether where
> I hope we could take team directions by constructive discussions and not
> decree.
> 
> ==
> 
> I give a second read to all emails exchanged since my original one about 
> o-o-t-
> m at 19th February and I come to this:
> 
> 1) No one support that we should manage kernels differently because they are
> under different repository.
> ==> We must manage them all the same way.

I did.  Maxime said building modules for only linux and linux-lts is a
good compromise. The Florian said "please go that route". Lukas was
strongly in favour "of have binary modules for kernels from [core]".

Gatean was in favour of having all kernels support binary modules.

Hence my conclusion:

"Binary modules are to be provided at minimum of all kernels in [core],
 with preference to providing them for all supported kernels (noting
that out-of-tree modules may not work with some patched kernels)."


> 2) No one object to a module exclusion from a specific kernel when it make no
> sense Examples was provided with GRSEC.
> ==> We could exclude modules support for a kernel for technical reasons.

Hence the "noting that out-of-tree modules may not work with some
patched kernels" in my conclusion.

> 3) There is a consensus on having binary modules in our repositories.
> ==> We must provides binary modules for all kernels. Except those covered by 
> 2.

See 1).

> 4) No one object to having dkms available for all modules; It's even supported
> by several fellows and this is offering support for AUR and custom kernels at
> almost no maintenance cost.
> ==> We must provides dkms build for all modules. Except those covered by 2.

As I said, no-one objected to DKMS modules.  But no opinions were stated
whether they must be provided.

> 5) There is no much discussion about which should be the default (a binary
> flavor or dkms). I would propose a solution which let the user choose which
> module he want needs by pulling a defined dep like module-$modulename. 
> ==> Applications should pull a generic deps to let users decide which module 
> he
> wants (flavored or dkms).

REALLY?   You need to read that thread again.  Default binary was
strongly supported.

> 6) Binary modules should be built from the dkms package. This was not 
> discussed
> earlier, but I see a benefits to have a common way of building modules and
> testing the dkms package in one row. 

Not part of the discussion.  The maintainers of the packages can have a
separate discussion about this.

> 7) https://bugs.archlinux.org/task/48498
> ==> kernels packages should provides kernel=$version and kernel-
> headers=$version in order to let dkms modules maintainers require at least a
> kernel and its header to be installed.

Off-topic for this discussion.

> This is the policy I propose we adopt.
> 
> Note that this policy is not defining who is responsible of what. We could 
> keep
> our current way of doing. I mean:
> - When a kernel is upgraded, it's kernel maintainer responsibility to upgrade
> all binary modules

It already is, unless they put the package in [staging] and create a
rebuild list.  Same as every other package.

> - When a module is upgraded, it's the modules maintainer responsibility to
> build the module for all kernels in stable and in testing (This is boring).

It already is.  If you don't like doing it, don't maintain the package.

A