Sourceware @ Conservancy Year Three

Sourceware joined Conservancy as a member project on May 15 2023
https://sfconservancy.org/news/2023/may/15/sourceware-joins-sfc/

Sourceware has provided the infrastructure for core toolchain and
developer tool projects for more than 25 years (28 years in
September). https://sourceware.org/sourceware-25-roadmap.html

Conservancy has helped us turn from a purely volunteer into a
professional organization with a Project Leadership Committee, monthly
open office hours, multiple hardware services partners, expanded
services, and a more diverse funding model that allows us to hold
assets and enter into official contracts with other organizations.

It was again a busy year, so we would like to summarize what happened
last year, our plans for the next one and how you can help.

* Communications, Community Events and Survey
* Services and Forge Developments
* Cyber Security, Policies, and Census Updates
* Datacenter Migrations and the VM-First Transition
* Finances and In-Kind Support
* Next Year Plans
* Project Leadership Committee Updates
* Thank You!

= Communications, Community Events and Survey

In the last year we organized 12 Open Office meetings on IRC in
#overseers on irc.libera.chat to discuss our shared development
infrastructure. Sourceware infrastructure community quarterly updates
were posted for 25Q2, 25Q3, 25Q4, and 26Q1:

- Q2 https://inbox.sourceware.org/[email protected]
- Q3 https://inbox.sourceware.org/[email protected]
- Q4 https://inbox.sourceware.org/[email protected]
- Q1 https://inbox.sourceware.org/[email protected]

Various Sourceware Project Leadership Committee members and project
maintainers met in person to map out infrastructure projects. We had
productive discussions during the GNU Tools Cauldron in Porto,
Portugal in September 2025 regarding our ongoing forge development.
Sourceware also shared a physical stand with the Software Freedom
Conservancy at FOSDEM 2026 in Brussels, where we distributed
stickers. https://fosstodon.org/@sourceware/115924128786908363

Next year we'll attend FOSSY26, Cauldron in Prague and FOSDEM 2027.

We also regularly share real-time operational announcements,
infrastructure notices, and temporary network downtime updates on the
fediverse at @[email protected] https://fosstodon.org/@sourceware

The yearly Sourceware Survey was held end of March and helped the
Sourceware Project Leadership Committee to know who our users are,
which hosted projects they feel part of, what services they rely on
and what the priorities should be for our budget and new initiatives.
Full results can be found at https://sourceware.org/survey-2026

= Services and Forge Developments

The rollout of the Anubis AI scraperbot mitigation layer was extended
this year. Using the non-javascript verification challenge, Anubis was
deployed across cgit, gitweb, bugzilla, wikis, public-inbox and the
forge. This successfully mitigated aggressive AI scraperbot traffic
without forcing normal browser users to use any complex javascript
requirements. We would like to thank Xe Iaso for helping us year round
with any network/bot issues. https://xeiaso.net/donate/

The Sourceware Forge experiment moved toward production-ready status
as we worked to lift the experimental label. Key milestones achieved
for forge.sourceware.org over the past year include:

- Setup forge-stage.sourceware.org as a fully Ansible-managed
  architecture to test out server configurations before deployment.

- Integrated action runners supporting x86_64 container workflows,
  enabling automated style checks (check_GNU_style.py) and commit
  validations (git_check_commit.py) directly on merge requests. And
  full testsuite runs for elfutils.

- Configured the Linaro-CI bot to poll open pull requests, execute
  builds on Linaro Arm machines, and log test states straight back to
  forge merge requests.

- Installed the batrachomyomachia bot to automatically mirror forge
  merge requests out to patches mailing lists for wider community code
  review.

- Opened public account registration to the forge, utilizing a secure
  workflow where new users must be manually added to a project's
  Contributors Team by an admin before they can create forks, merge
  requests or execute actions.

And we migrated the forge into a larger, dedicated virtual machine
(vm02) on server1, mapping out an Ansible setup and hot backup pairing
on server3 (in progress).

We also extended automated artifact snapshotting by adding an hourly
online documentation builder for GCC, driven by a containerized script
(gen_gcc_docs.sh) https://snapshots.sourceware.org/gcc/docs/latest

= Cyber Security, Policies, and Census Updates

Sourceware worked closely with the Software Freedom Conservancy to
monitor shifting international cybersecurity regulations, evaluate
policies, and publish practical secure software development practices
for hosted projects https://sourceware.org/cyber-security-faq.html

Our Cyber Security FAQ was updated to include:

- Context surrounding the U.S. Executive Order frameworks rewriting
  NIST SP 800-218 (SSDF) attestation requirements.

- Reference materials from the FSFE EU Cyber Resilience Act (CRA)
  presentation for SFC member projects.

- A reusable CRA request reply template for maintainers responding to
  corporate compliance inquiries
  https://sourceware.org/cyber-security-faq.html#eu-cra-reply

We also continued publishing our quarterly signed-commit census
leaderboards, tracking cryptographic signing percentages across
project branches to encourage strong repository verification habits.

= Datacenter Migrations and the VM-First Transition

This year marked the successful completion of our bare-metal to
VM-first transition, moving all production operations into isolated
virtual machines to bolster system security and administration. This
infrastructure migration was coordinated alongside datacenter moves by
both of our hardware partners.

- The Red Hat Community Cage Move.

This datacenter relocation impacted server2, server3, and
forge.sourceware.org. To capitalize on the move, the PLC procured a
larger primary node (server1) featuring 3x the memory, 10x the
storage, and roughly double the raw CPU cores of the older
systems. Funded via individual donations, a FUTO grant, and with the
help of the Red Hat IT teams, server1 went live in the new RDU3
facility late November 2025. Core production services were migrated
into isolated VMs on server1. Afterward, server2 and server3 were
physically moved to RDU3 and reconfigured entirely as VM hosts.

- The OSUOSL Datacenter Move

Impacting our core continuous integration builders and snapshots
platform. OSUOSL retired our legacy x86_64 build servers. They were
replaced by a (much) larger sourceware-builder3 machine packed with
2x28 cores and 768GB RAM, partitioned into four distinct virtual
environments: two Buildbot workers (sw3bb1, sw3bb2) and two Forgejo
action runners (sw3runner1, sw3runner2).

The full setup of the bare-metal and cloud servers at the different
datacenters and the VMs dedicated to various services can be found at
https://sourceware.org/sourceware-wiki/servers-and-services-2026/

With the successful deployment of these systems, our complete hardware
refresh cycle is finished, securing our server, hosting, and virtual
environments for the next couple of years.

= Finances and In-Kind Support

Sourceware concluded the financial year significantly ahead of
schedule, fully restoring our equipment reserves in a single year
rather than the projected three-year cycle. Our fiscal year closed
with a healthy cash balance of $10,017.59, having raised $6,332.12
against total annual expenses of $6,358.98.

Our primary expenditure was the well-timed purchase of server1, which
cost $6,195.31 inclusive of shipping and taxes. This procurement
proved exceptionally strategic, as subsequent component prices
increases pushed the market cost of its 1.5TB RAM alone to equal the
value of the whole server. Remaining minor expenses went toward domain
registration renewals ($52.59) and banking fees ($111.08).

Individual community donations doubled over the past year, growing
from an average of ~$250 a month up to ~$500 a month. Our cash
reserves remain strong because of sustained "in-kind" resource
donations from our hosting partners. Red Hat expanded its hosting
allocation from two to three physical servers and added cloud/VM
environments. And OSUOSL provided us with a (much) larger builder
machine and secondary cloud/VM nodes.

= Next Year Plans

Our excellent financial situation puts us ahead of schedule with a
fully restored hardware refresh fund. With our servers, hosting, and
virtual environments secured for the next few years, our upcoming
focus will be more on upgrading services, putting more services into
separate isolated VMs and supporting our admins and maintainers
processes.

Based on our latest budget talks, we will try to establish a fair
compensation model for OSUOSL to cover our ongoing bandwidth and
colocation hosting costs. We will continue executing our security
vision by moving more services, specifically bugzilla, buildbot,
patchwork, and public-inbox, into isolated VMs, and upgrading those
services to newer versions. Where possible we will try to fully
automate their deployments using Ansible. To help our administrators
we will look into hiring consultants or a system administrator to help
with these upgrades. Finally, we will look into upstream funding
toward fixing Forgejo process bugs to improve account and permission
handling.

For funding these plans we will use at most a third of our current
cash reserves. Our Individual Sponsors fund the core infrastructure
and daily operations. Some Corporate Sponsors already fund through
in-kind donations. And we'll setup sponsorship programs for
Corporations and Grant Makers to cover some of our goals that might
have extended costs.

= Project Leadership Committee Updates

After 25 years of involvement with Sourceware and Cygwin, including
two years of service on the PLC, Christopher Faylor (cgf) resigned
from his committee seat. Christopher spent 20 years managing core
project mailing lists and filtering daily spam to ensure an open,
welcoming environment. Sourceware would not be what it is today
without his efforts, and we thank him deeply for his insights.

Following his departure, the PLC is composed of 7 active members.
https://sourceware.org/mission.html#plc The mandatory minimum number
of Members is 4. And no more than 2 Members may be Financially-Related
to the same Entity.

If you are interested in joining the PLC please read the
https://sourceware.org/Conservancy-Sourceware-FSA.pdf Fiscal
Sponsorship Agreement, the Conflict of Interest Policy
https://sfconservancy.org/projects/policies/conflict-of-interest-policy.html
and contact us at [email protected].

If you rather help with more technical tasks please join the overseers
list: https://sourceware.org/mailman/listinfo/overseers

= Thank You!

Our third year as a member project of the Software Freedom Conservancy
has finalized our evolution into a professionalized, highly resilient
infrastructure organization. We express our deepest gratitude to the
administrative staff at the SFC for their continuous guidance through
our budget planning, the technical infrastructure teams at Red Hat
OSPO and OSUOSL for their hands-on support, and our individual donors
who power our independence.

We warmly invite our community to support this ecosystem by sustaining
the Software Freedom Conservancy https://sfconservancy.org/sustainer
donating directly to OSUOSL https://osuosl.org/donate, or providing
individual or corporate sponsorship at https://sourceware.org/donate

The Sourceware PLC,

Frank Ch. Eigler, Ian Kelling, Ian Lance Taylor, Tom Tromey,
Jon Turney, Mark J. Wielaard and Elena Zannoni

Reply via email to