Russ, Thank you for this summary.
On Friday, June 21, 2024 3:24:58 PM MST Russ Allbery wrote: > There is a stark disagreement over the importance of that signature, and > it appears to be the remaining blocking issue. I have argued that it > makes little difference from a security standpoint whether the source > package construction step happens before the uploader signature (the > current dak upload process) or after the uploader signature (the > tag2upload process), given that the uploader doesn't (and can't, > realistically) check the output in either case. I believe at least one > FTP team delegate disagrees. I personally don't understand the > disagreement. Let me attempt to explain why I perceive a significant security improvement between an implementation where the DD signs a manifest of all the source files and one where that doesn’t happen. Let me begin by saying that I perceive three broad goals with tag2upload as it has been envisioned by those who have been developing it. 1. Maintain a complete history of the source of Debian packages in Git, including their Git history. 2. Create the source packages in a controlled, centralized environment. 3. Eliminate the need for a fat client or any special Debian tooling on the DD’s end, and handle everything with standard Git tools. I find points 1 and 2 to be compelling. I don’t personally find point 3 to be compelling. I could take it or leave it in the final tag2upload implementation. As a point of contrast, I think that the developers of tag2upload consider point 3 to be the most important of their design goals. They also like points 1 and 2, but they never would have started down the path of developing tag2upload if it wasn’t for the benefits they feel point 3 offers. From a purely security-conscious perspective, I think tag2upload is significantly improved if point 3 is dropped as an objective of tag2upload. Currently, source packages are built on DD machines. There is a security- attack surface in this design, in that, if a DD’s machine is compromised, it could insert malicious code into the package at the point it was being built. The DD would be unlikely to notice the introduction of such code and would sign the package and upload it to Debian without realizing that the source tarball had been changed from the copy they acquired from upstream or the packaging they were working on in Salsa (or some other forge). Although I am not aware of any instance of this happening, the xz-utils compromise has made everyone more conscious of this possibility. The current tag2upload implementation improves on this design by centralizing the construction of the source package in a controlled environment on Debian infrastructure in what has been described as putting all your eggs in one basket. Instead of having a thousand DD machines constructing the source packages, some of which have very high security and some of which might not, there is one tag2upload system to harden and protect. However, whenever assets are centralized, it also means that the power of a successful exploit increases. Right now, if someone compromises a DD’s machine, they can only insert malicious code into the packages that DD uploads. But if someone were ever able to compromise the tag2upload server, they could insert malicious code into all the packages that it processes. One of the answers to this concern is to setup a second tag2upload server that checks the work of the first. A reproducible build, if you will. If only one of the tab2upload systems is compromised, it will not be able to produce packages that are trusted by the rest of Debian. Adding a second centralized system is better than having a single point of security failure. But there is a gold standard that is better than that. Requiring agreement between both a centralized and a distributed system. This is where client signing of the entire source manifest comes into play. If both the DD's machine and tag2upload have to sign a manifest of the source package, it makes it much more difficult for an attacker to insert malicious code into the package. In the previous example with only one tag2upload server, compromising a single machine would provide the attacker this ability for all packages processed through tag2upload. In the case of having two, redundant, tag2upload servers, only two systems need to be compromised. But in the scenario of requiring both a centralized system and a distributed system involved, an attacker that compromised both the tag2upload system and a DD’s machine could only alter packages that one DD uploads. A full compromise of Debian would require the attacker to compromise the centralized tag2upload server (or servers in the case of reproducible builds) and a thousand DD machines. Requiring both centralized and distributed signing of the package manifest also completely eliminates a whole series of potential weaknesses described in Russ’ s security audit. It removes any concern about SHA1 weakness in Git. It eliminates all the potential race conditions with replacing Git trees or other concerns with Git collisions. It removes any concern about the admins of tag2upload being able to do anything untoward. It almost removes any concerns about a Salsa admin being able to do anything untoward (I will discuss the one exception below[1]). It basically removes almost all the possible *man-in-the-middle* attacks that could be executed by 1) a malicious packager requesting sponsorship, 2) a malicious Salsa admin, 3) a malicious attacker who gains admin access to Salsa, 4) a malicious tag2upload admin, 5) a malicious attacker who gains admin access to the tag2upload servers, and 6) someone who compromises the DD’s system[2]. In short, I would consider that having the DD sign the manifest of the source package improves the security posture of the proposed tag2upload service by about 2 orders of magnitude. The question before us now--the one this GR proposes to the general Debian community--is if this increased security is worth dropping one of the original goals of the tag2upload developers. There are probably some people who find the idea of simply being able to sign a Git tag and having the source package built for them by Debian infrastructure without the need for their system to do anything else to be so advantageous that it is worth an implementation that is less robust from a security perspective. There are others, like me, who don’t see this aspect of tag2upload’s current design to be that important, and would rather have a system where the DD’s machine would need to generate enough of the source package to sign a full manifest, add that to the signed Git tag, and then have the process repeated by the tag2upload’s server(s) in a controlled environment. [1] The one attack scenario I can think of that client-side source package manifest signing does not prevent is the following, which has been discussed in the thread to some degree. Someone with admin access to Salsa could modify the system to display a clean set of code in the web interface, but presents a malicious set of code when cloned. Assuming that the DD does all of their development work using the Salsa web interface, they would never see the malicious code. But, if both their system and tag2upload cloned the repository to build the source code, and if they used their cloned version to sign the tag, the malicious code would be inserted into the system under the radar. Note that for this attack to work, the DD would have to pull from Salsa without reviewing the source code on their own system. This might be unlikely with packages that are maintained by one DD, as they might only push to Salsa. In that case, their manifest would be different from tag2upload’s. But in the case of active team maintenance, the developers would both push and pull and could end up signing unreviewed, compromised code. I haven’t, so far, been able to think up a good defence against this type of attack, short of doing all development work on the DD’s machine instead of Salsa’s web interface (which is how I generally work now, but isn’t how everyone does things). This attack would probably be difficult to pull off, especially for an extended period of time, without someone noticing. But it isn’t outside the realm of possibility. [2] Note that all of the discussion in this email about the insertion of malicious code into the package is in reference to the problem raised by the xz-utils compromise, which was that, when someone looked at what they thought was the source code (the Git repository) the didn’t see (all of) the malicious code. It was inserted at a later step into the source tarball in a manner that didn’t seem concerning on the face. The entire discussion above about an attacker being able to insert malicious code into the source means that they do so in such a way that it doesn’t appear in the code the DD reviews when they are preparing the package, but does end up in the source tarball that is uploaded to dak. Nothing about this design protects against 1) malicious code inserted upstream that is visible to the DD, but that thy either don’t review or don’t recognize as being malicious, or 2) malicious code intentionally inserted by the DD that they sign and upload with the intent of compromising Debian. -- Soren Stoutner so...@debian.org
signature.asc
Description: This is a digitally signed message part.
