Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Chris Lamb wrote:

> > I reviewed the latter and found some issues:
>
> Thanks for your review. Unfortunately, there is limited scope to
> make substantive changes at this stage in the publication cycle.
>
> However, I'm sure some of the grammatical fixes you mention can be
> absorbed, so thank you for pointing those out.

Thanks again -- I managed to get a few of these into the draft of the
magazine (non-LaTeX) version of the article.


Best wishes,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Fredrik Strömberg]

On Wed, Apr 14, 2021 at 7:02 PM Chris Lamb
 wrote:
>
> Chris Lamb wrote:
>
> > > > > As it happens, Stefano Zacchiroli recently suggested to me that we
> > > > > write a paper together that we would first offer to IEEE Software. We
> > > > > got into a good routine and submitted to IEEE about a fortnight ago.
> > > > >
> > > >
> > > > Chris: Any news on this article? I'd love to read it.
> > >
> > > Thanks for your interest. No news since I last posted; we are still
> > > waiting on the initial review. Will be in touch when I know more.
> >
> > Fredrik, as you asked for updates: just to mention that the paper has
> > passed its initial review, and we are now making some minor changes to
> > address various comments and concerns (mostly around the framing of
> > the issue and ensuring it is accessible to as wide an audience as
> > possible).
>
> A quick update: as permitted by IEEE, the paper is now available in an
> open access / preprint capacity:
>
>https://ieeexplore.ieee.org/document/9403390
>https://arxiv.org/abs/2104.06020
>
>

Thank you!

> Best wishes,
>
> --
>   o
> ⬋   ⬊  Chris Lamb
>o o reproducible-builds.org
> ⬊   ⬋
>   o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Hi Bernhard,

> I reviewed the latter and found some issues:

Thanks for your review. Unfortunately, there is limited scope to
make substantive changes at this stage in the publication cycle.

However, I'm sure some of the grammatical fixes you mention can be
absorbed, so thank you for pointing those out.


Regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Bernhard M. Wiedemann]

On 14/04/2021 19.02, Chris Lamb wrote:
> A quick update: as permitted by IEEE, the paper is now available in an
> open access / preprint capacity:
> 
>https://ieeexplore.ieee.org/document/9403390
>https://arxiv.org/abs/2104.06020


I reviewed the latter and found some issues:

> doing so is inefficient when source code is available for audit

was very confusing to read. I read it multiple times and understood it
as "source code makes audit inefficient" until some time later
re-reading with more context.

Should be something about "auditing source-code is more efficient than
auditing binaries"


> The mechanics of reproducibility testing suggest that this issue would not 
> have been readily discovered another way.

not sure if mechanics are people here or mechanisms - and not sure how
either would suggest something.
Why not "We believe that..." or "Our experience (in rb) leads us to
think..." ?


> However, this has not yet been achieved, partly because time and effort are 
> not inexhaustible or fungible resources in volunteer communities

This is hard to parse, not only because of the double-negation ("not
in-"). Does it mean: Engineers have limited time and volunteers even
more so? And 'fungible' means you can not just put a noob's hour in and
achieve as much as an expert-hour?



For the list of common issues: code compiling with -march=native is a
common occurrence that also is a bug found easily by rb. I often find
that in our HPC and science package sections.


In the debugging section, you only mentioned looking at diffoscope
output. Did you consider adding some of the other useful ways mentioned
in section 2 of
https://github.com/bmwiedemann/reproducibleopensuse/blob/devel/howtodebug ?




and some grammar fixes:

-a extremely mature
+an extremely mature

-tool that recursively unpacks a large number of archive formats and
translate tens of binary formats
+tool that recursively unpacks a large number of archive formats and
translates tens of binary formats


Ciao
Bernhard M.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Chris Lamb wrote:

> > > > As it happens, Stefano Zacchiroli recently suggested to me that we
> > > > write a paper together that we would first offer to IEEE Software. We
> > > > got into a good routine and submitted to IEEE about a fortnight ago.
> > > >
> > >
> > > Chris: Any news on this article? I'd love to read it.
> >
> > Thanks for your interest. No news since I last posted; we are still
> > waiting on the initial review. Will be in touch when I know more.
>
> Fredrik, as you asked for updates: just to mention that the paper has
> passed its initial review, and we are now making some minor changes to
> address various comments and concerns (mostly around the framing of
> the issue and ensuring it is accessible to as wide an audience as
> possible).

A quick update: as permitted by IEEE, the paper is now available in an
open access / preprint capacity:

   https://ieeexplore.ieee.org/document/9403390
   https://arxiv.org/abs/2104.06020


Best wishes,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Allen Gunn]

And for those have not seen this item, another take on supply chain
vulnerability scenarios:

https://www.schneier.com/blog/archives/2021/02/dependency-confusion-another-supply-chain-vulnerability.html

On 2/23/21 12:56 AM, Fredrik Strömberg wrote:
> On Mon, Feb 22, 2021 at 6:52 PM Chris Lamb
>  wrote:
>>
>> Fredrik, as you asked for updates: just to mention that the paper has
>> passed its initial review, and we are now making some minor changes to
>> address various comments and concerns (mostly around the framing of
>> the issue and ensuring it is accessible to as wide an audience as
>> possible).
>>
> 
> Thanks Chris!
> 

-- 

Allen Gunn
Executive Director, Aspiration
www.aspirationtech.org

Aspiration: "Better Tools for a Better World"

Read our Manifesto: https://aspirationtech.org/publications/manifesto

Twitter:  www.twitter.com/aspirationtech



signature.asc
Description: OpenPGP digital signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[Fredrik Strömberg]

On Mon, Feb 22, 2021 at 6:52 PM Chris Lamb
 wrote:
>
> Fredrik, as you asked for updates: just to mention that the paper has
> passed its initial review, and we are now making some minor changes to
> address various comments and concerns (mostly around the framing of
> the issue and ensuring it is accessible to as wide an audience as
> possible).
>

Thanks Chris!

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Chris Lamb wrote:

> > > As it happens, Stefano Zacchiroli recently suggested to me that we
> > > write a paper together that we would first offer to IEEE Software. We
> > > got into a good routine and submitted to IEEE about a fortnight ago.
> > >
> >
> > Chris: Any news on this article? I'd love to read it.
>
> Thanks for your interest. No news since I last posted; we are still
> waiting on the initial review. Will be in touch when I know more.

Fredrik, as you asked for updates: just to mention that the paper has
passed its initial review, and we are now making some minor changes to
address various comments and concerns (mostly around the framing of
the issue and ensuring it is accessible to as wide an audience as
possible).


Best wishes,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Hi Fredrik,

> > As it happens, Stefano Zacchiroli recently suggested to me that we
> > write a paper together that we would first offer to IEEE Software. We
> > got into a good routine and submitted to IEEE about a fortnight ago.
> >
> 
> Chris: Any news on this article? I'd love to read it.

Thanks for your interest. No news since I last posted; we are still
waiting on the initial review. Will be in touch when I know more.


Best wishes,

-- 
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Fredrik Strömberg]

Hi Chris, and everyone else!

On Tue, Dec 22, 2020 at 1:37 PM Chris Lamb
 wrote:
>
> As it happens, Stefano Zacchiroli recently suggested to me that we
> write a paper together that we would first offer to IEEE Software. We
> got into a good routine and submitted to IEEE about a fortnight ago.
>

Chris: Any news on this article? I'd love to read it.

Cheers,
Fredrik

Re: Attack on SolarWinds could have been countered by reproducible builds

[David A. Wheeler]

I just posted, on The Linux Foundation blog, an article titled
"Preventing Supply Chain Attacks like SolarWinds” at:
https://www.linuxfoundation.org/en/blog/preventing-supply-chain-attacks-like-solarwinds/

It *prominently* notes the need for reproducible builds.

Ximin Luo:
> From my experience working in R-B, media chatter isn't sufficient to overcome 
> engineering inertia.

I don’t agree, because there’s been almost no media chatter. There’s been very 
little attention paid to reproducible builds by the media. SolarWinds is a big 
deal, yet journalists have generally failed to mention reproducible builds when 
discussing SolarWinds (probably because they don’t understand R-B). Yes, there 
are exceptions, but it *should* be mentioned in every story. It’s 
understandable that the journalists first focused on “what happened”, but I 
would like the discussion to start moving to “what should happen in the future 
to help counter its recurrence?”

> There's a lot of tunnel vision and arrogant engineers in upstream toolchain 
> projects nitpicking at technical crap that doesn't matter, when we submit 
> patches. To advance reproducible builds, this social issue has to be 
> addressed somehow ...
> Of course maintaining a FOSS project is also thankless work, so 
> understandably some engineers are more conservative and grumble about 
> outsiders. At some point it becomes obstructionism though. I don't know 
> enough about who is getting paid by $BIGCO vs who is getting zilch, to 
> comment on which specific projects have which problems. It is a 
> broad-spectrum thing that goes across the board.

To be fair, those engineers are being asked to do far more than they have time 
for, *AND* they’re asked produce high-quality results. I don’t approve of 
arrogance, but I *do* appreciate that they have to say “no” a lot. Which is why 
r-b needs *more* media attention; those engineers need to prioritize what’s 
important, and so we need to make it clearer that this is important.

--- David A. Wheeler

Re: Attack on SolarWinds could have been countered by reproducible builds

[Hans-Christoph Steiner]

Yeah, a short writeup on RB in the context of the SolarWinds attack would be 
great to have, especially now that more details are coming out.  Its quite an 
impressive hack, it even cleans up after itself:



To prevent detection, Sunburst’s creators “included a hash verification check” 
to ensure the injected malicious code “is compatible with a known source file”. 
Once the build process was complete, Sunburst waited for MsBuild.exe to exit 
“before restoring the original source code and deleting the temporary 
InventoryManager.bk file” containing its malicious code, now compiled into the 
Orion product.

https://www.theregister.com/2021/01/12/solarwinds_tech_analysis_crowdstrike/

.hc

David Kleuker:

it don't help much to rant on this ML where all people know what reproducible 
builds are. instead contacting all those journalists that did not mention it 
has a chance to change the current status.

a publication on reproducible-builds.org about this incident would also be 
helpful to share the link

next time this happens, journalists would at least know they COULD mention it

kind regards
David Kleuker


Chris Lamb  hat am 21.12.2020 15:30 geschrieben:

  
David A. Wheeler wrote:



Let me restate this: it appears that the *source code* wasn’t
compromised, and the *distribution* system wasn’t compromised. Instead,
the *build system* was compromised.


Thanks for this, David. You are absolutely right that this is exactly
what Reproducible Builds was 'designed' for to begin with. An ironic
hurrah that this kind of attack is getting more visibility these days.

Another thanks for the press references too -- I will make good use of
them when writing our next monthly report. (Alas, if it wasn't the
holiday season I might be tempted to suggest that we do a specific
publicity boost based on this..)


Regards,

--
   o
 ⬋   ⬊  Chris Lamb
o o reproducible-builds.org 💠
 ⬊   ⬋
   o


--
PGP fingerprint: EE66 20C7 136B 0D2C 456C  0A4D E9E2 8DEA 00AA 5556
https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

Re: Attack on SolarWinds could have been countered by reproducible builds

[Ximin Luo]

>From my experience working in R-B, media chatter isn't sufficient to overcome 
>engineering inertia.

There's a lot of tunnel vision and arrogant engineers in upstream toolchain 
projects nitpicking at technical crap that doesn't matter, when we submit 
patches. To advance reproducible builds, this social issue has to be addressed 
somehow.

Newer projects (e.g. Rust) are better at this, possibly because they pay more 
attention to the media. (This isn't necessarily a good thing in general, but it 
helps in this specific case.)

Of course maintaining a FOSS project is also thankless work, so understandably 
some engineers are more conservative and grumble about outsiders. At some point 
it becomes obstructionism though. I don't know enough about who is getting paid 
by $BIGCO vs who is getting zilch, to comment on which specific projects have 
which problems. It is a broad-spectrum thing that goes across the board.

X

Hans-Christoph Steiner:
> 
> Thanks for this info!  RB work can be a slog through annoying technical 
> details, so confirmation of its important always helps lift my spirits.  Its 
> definitely good fodder for getting funding for related work.
> 
> .hc
> 
> David A. Wheeler:
>> All:
>>
>> There’s been a recently-revealed attack on the SolarWinds product “Orion", a 
>> Network Management System (NMS). This software is widely used and thus this 
>> attack is extremely concerning.
>>
>> According to SANS, "SolarWinds has published limited information in which 
>> they state they believe the build environment was compromised.” 
>> https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/
>>
>> Let me restate this: it appears that the *source code* wasn’t compromised, 
>> and the *distribution* system wasn’t compromised. Instead, the *build 
>> system* was compromised. This is *EXACTLY*  the kind of attack that is 
>> countered by reproducible builds. Thus, the recent SolarWinds subversion is 
>> a very good argument for why it’s important to have reproducible builds (and 
>> to verify builds using reproducible builds).
>>
>> I’ve read a number of articles about SolarWinds, and none of them mention 
>> reproducible builds, even though reproducible builds is clearly a 
>> countermeasure to this problem. Perhaps journalists will eventually learn 
>> about reproducible builds; that would be nice!
>>
>> --- David A. Wheeler
>>
>> PS: Here are some articles about the attack on SolarWinds:
>> https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/
>>  
>> 
>> https://www.cnn.com/2020/12/16/tech/solarwinds-orion-hack-explained/index.html
>>  
>> 
>> https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html
>>  
>> 
>> https://www.computerweekly.com/news/252493662/SolarWinds-cyber-attack-How-worried-should-I-be-and-what-do-I-do-now
>>  
>> 
>> https://www.washingtonpost.com/technology/2020/12/14/russia-hack-us-government/
>>  
>> 
>>
>>
> 


-- 
GPG: ed25519/56034877E1F87C35
https://github.com/infinity0/pubkeys.git

Re: Attack on SolarWinds could have been countered by reproducible builds

[Hans-Christoph Steiner]

Holger Levsen:

On Wed, Dec 30, 2020 at 04:41:08PM +0100, Hans-Christoph Steiner wrote:

If you'd like to see a concrete use, for the apps that require reproducible
builds in F-Droid, an APK build is not signed and released unless
f-droid.org's build matches the upstream developer's APK.


while this is pretty cool, it's nothing a user can verify.


A technical user with plenty of disk space could actually verify this. 
Our whole build/sign stack can be set up in a VM using ansible.  Thanks 
to those weekly runs on jenkins.debian.net, its pretty reliable.


* install vagrant with either VirtualBox or libvirt
* clone https://gitlab.com/fdroid/fdroid-bootstrap-buildserver
* `vagrant up`
* wait some hours

You have the same stack f-droid.org uses to run the builds.

.hc

--
PGP fingerprint: EE66 20C7 136B 0D2C 456C  0A4D E9E2 8DEA 00AA 5556
https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

Re: Attack on SolarWinds could have been countered by reproducible builds

[Holger Levsen]

On Wed, Dec 30, 2020 at 04:41:08PM +0100, Hans-Christoph Steiner wrote:
> If you'd like to see a concrete use, for the apps that require reproducible
> builds in F-Droid, an APK build is not signed and released unless
> f-droid.org's build matches the upstream developer's APK.

while this is pretty cool, it's nothing a user can verify.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄



signature.asc
Description: PGP signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[Hans-Christoph Steiner]

Holger Levsen:

hi,

On Mon, Dec 21, 2020 at 01:58:01PM -0500, Santiago Torres-Arias wrote:

To be a little bit more upfront: I think that we as a community
sometimes focus on "is this thing reproducible" and not on "how can I
use this to secure the ecosystem".


I fully agree and believe this is due to us still encountering way too
many practical technial problems. It's really hard to think practically
about something which mostly only exists in theory.

I mean, the tails ISO is the only 'product' I'm aware of which can be
meaningfully verified currently. And probably some android apps too,
though alone fetching an .apk with adb from a phone and verifying it is nothing
I could recommend as 'easy' to anyone (except android hackers).

But for the big linux distros we aren't there yet. And thus it's very
hard to focus on user stories and to keep the focus there. At least that's
my explaination why we drift into details constantly.


If you'd like to see a concrete use, for the apps that require 
reproducible builds in F-Droid, an APK build is not signed and released 
unless f-droid.org's build matches the upstream developer's APK.


.hc

--
PGP fingerprint: EE66 20C7 136B 0D2C 456C  0A4D E9E2 8DEA 00AA 5556
https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556

Re: Attack on SolarWinds could have been countered by reproducible builds

[Holger Levsen]

hi,

On Mon, Dec 21, 2020 at 01:58:01PM -0500, Santiago Torres-Arias wrote:
> To be a little bit more upfront: I think that we as a community
> sometimes focus on "is this thing reproducible" and not on "how can I
> use this to secure the ecosystem".

I fully agree and believe this is due to us still encountering way too
many practical technial problems. It's really hard to think practically
about something which mostly only exists in theory.

I mean, the tails ISO is the only 'product' I'm aware of which can be
meaningfully verified currently. And probably some android apps too,
though alone fetching an .apk with adb from a phone and verifying it is nothing
I could recommend as 'easy' to anyone (except android hackers).

But for the big linux distros we aren't there yet. And thus it's very
hard to focus on user stories and to keep the focus there. At least that's
my explaination why we drift into details constantly.


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁   holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
 ⠈⠳⣄

Life is short but a sea of morons is forever.


signature.asc
Description: PGP signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[Bernhard M. Wiedemann]

On 21/12/2020 22.28, Richard Purdie wrote:
> OE-Core is about 800 pieces of software generating ~11,000
> packages of which we have about 65 marked as not reproducible at
> present. We're obviously working on improving those 65, and the
> techniques used will "just work" to a large extend throughout our wider
> layers of other software, we're just note testing that until we sort
> the core.

do you have pointers to the list of unreproducible packages and how to
do test builds?


In http://git.openembedded.org/openembedded-core/
meta/lib/oeqa/selftest/cases/reproducible.py exclude_packages maybe?


>   'acpica-src',
>   'babeltrace2-ptest',
>   'bootchart2-doc',
>   'cups',
>   'cwautomacros',
>   'dtc',
>   'efivar',
>   'epiphany',
>   'gcr',
>   'git',
>   'glide',
>   'go-dep',
>   'go-helloworld',
>   'go-runtime',
>   'go_',
>   'groff',
https://build.opensuse.org/request/show/645935
>   'gst-devtools',
>   'gstreamer1.0-python',
>   'gtk-doc',
https://bugzilla.gnome.org/show_bug.cgi?id=784177
>   'igt-gpu-tools',
> 'kernel-devsrc',
>   'libaprutil',
>   'libcap-ng',
>   'libhandy-1-src',
>   'libid3tag',
>   'libproxy',
>   'libsecret-dev',
>   'libsecret-src',
>   'lttng-tools-dbg',
>   'lttng-tools-ptest',
>   'ltp',
>   'meson',
>   'ovmf-shell-efi',
>   'parted-ptest',
>   'perf',
https://elixir.bootlin.com/linux/latest/source/tools/perf/pmu-events/jevents.c#L1168
>   'python3-cython',
>   'qemu',
>   'quilt-ptest',
>   'rsync',
>   'ruby',
https://github.com/ruby/io-console/commit/679a941d05d869f5e575730f6581c027203b7b26
>   'spirv-tools-dev',
>   'swig',
>   'syslinux-misc',
>   'systemd-bootchart',
>   'valgrind-ptest',
>   'vim',
>   'watchdog',
>   'xmlto',
>   'xorg-minimal-fonts'

I found some relevant patches and pointers in our packages, linked above.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Hi Martin,

>  > Stefano Zacchiroli recently suggested to me that we
>  > write a paper together that we would first offer to IEEE Software. We
>  > got into a good routine and submitted to IEEE about a fortnight ago.
> 
> Congrats on the submission! Would you have a preprint or Arxiv version 
> we could read?

It's not quite at that stage yet, but I will of course followup with a
preprint when that is more appropriate.


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org 🍥 chris-lamb.co.uk
   `-

Re: Attack on SolarWinds could have been countered by reproducible builds

[Martin Monperrus]

Hi Chris, all,

> Stefano Zacchiroli recently suggested to me that we
> write a paper together that we would first offer to IEEE Software. We
> got into a good routine and submitted to IEEE about a fortnight ago.

Congrats on the submission! Would you have a preprint or Arxiv version we could 
read?

Thanks!

--Martin Monperrus

Re: Attack on SolarWinds could have been countered by reproducible builds

[Justin Cappos]

Glad to hear it!  We need to get more awareness and this seems a great way
to do it!

Justin

On Tue, Dec 22, 2020 at 8:37 PM Chris Lamb 
wrote:

> Hi Justin,
>
> > On another note, I would say this is an ideal time to engage the
> > broader academic / open source communities about reproducible builds.
>
> As it happens, Stefano Zacchiroli recently suggested to me that we
> write a paper together that we would first offer to IEEE Software. We
> got into a good routine and submitted to IEEE about a fortnight ago.
>
> It's aimed a more general audience, first defining the problem and
> then providing some insight into the challenges of actually making
> real-world software reproducible.
>
> We then use various experiences of the Reproducible Builds project to
> make large-scale software collections/supply-chains/ecosystems
> reproducible, and we also describe the affinity between reproducibility
> efforts and quality assurance (QA).
>
> More news when we have it, of course...
>
>
> Regards,
>
> --
>   o
> ⬋   ⬊  Chris Lamb
>o o reproducible-builds.org 💠
> ⬊   ⬋
>   o
>

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

Hi Justin,

> On another note, I would say this is an ideal time to engage the
> broader academic / open source communities about reproducible builds.

As it happens, Stefano Zacchiroli recently suggested to me that we
write a paper together that we would first offer to IEEE Software. We
got into a good routine and submitted to IEEE about a fortnight ago.

It's aimed a more general audience, first defining the problem and
then providing some insight into the challenges of actually making
real-world software reproducible.

We then use various experiences of the Reproducible Builds project to
make large-scale software collections/supply-chains/ecosystems
reproducible, and we also describe the affinity between reproducibility
efforts and quality assurance (QA).

More news when we have it, of course...


Regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Justin Cappos]

On Tue, Dec 22, 2020 at 4:58 AM David A. Wheeler 
wrote:

>
>
> On Dec 21, 2020, at 1:58 PM, Santiago Torres-Arias 
> wrote:
> I agree that we need more visibility on the reprobuilds aspect of this
> compromise.
>
>
> I don’t think it’s visible to *reporters* though.
>

Just to chime in here, I've been interviewed by a few journalists on the
topic ( Yahoo Finance

, Crains
,
with more hopefully coming out).  I mentioned repro builds, etc. to them
and really stressed it with verification as the solution but they just
didn't use this in their stories.  I think the problem is that it's hard
enough to explain to a general audience where their story focus is more on
the problem and who might be behind it than any potential solution.

On another note, I would say this is an ideal time to engage the broader
academic / open source communities about reproducible builds.  I started a
paper draft a few years ago (
https://github.com/JustinCappos/reproduciblebuildpaper ), but there was a
loss of momentum.  Perhaps it is time to consider brushing it off or
starting something new?

Thanks,
Justin

Re: Attack on SolarWinds could have been countered by reproducible builds

[Richard Purdie]

On Mon, 2020-12-21 at 15:57 -0500, David A. Wheeler wrote:
> I think these things need to happen in stages. Broadly:
> 1. Get key applications & libraries reproducible (assuming toolchains
> are okay)
> 2. Establish independent processes that *check* that the binaries are
> what they’re supposed to be.
> 3. Extend the work to more/all applications/libraries in given
> domains.
> 4. Work on verifying underlying toolchains, and again, creating
> independent processes that *check* the toolchain results (DDC &
> bootstrapping).
> 
> The long-term goal should be that “we can ensure that all OSS
> compiled code is accurately represented by its source code”. The
> source code may include malicious statements, but source code is what
> developers review, so we’ve fundamentally changed the game to ensure
> that “what is reviewed is what is run”.

Not sure its so long term for some of us!

With Yocto Project, what we now effectively have is a build from
"scratch" environment where the inputs are checksum validated and the
output bitwise reproducible.

I say "scratch" since we do assume a working host compiler and basic
tools (we have a list) which are used to build the cross compiler.

We are host system independent in that it doesn't matter which distro
you build on, or in which path, the output tarball containing "Linux"
is the same for anything inside OE-Core with a small number of
exceptions. OE-Core is about 800 pieces of software generating ~11,000
packages of which we have about 65 marked as not reproducible at
present. We're obviously working on improving those 65, and the
techniques used will "just work" to a large extend throughout our wider
layers of other software, we're just note testing that until we sort
the core.

The net result is multiple people on multiple different platforms can
run the build and generate the same result consistently. Our
autobuilder does run that exact test regularly.

Cheers,

Richard

Re: Attack on SolarWinds could have been countered by reproducible builds

[David A. Wheeler]

> On Dec 21, 2020, at 1:58 PM, Santiago Torres-Arias  
> wrote:
> I agree that we need more visibility on the reprobuilds aspect of this
> compromise.

I don’t think it’s visible to *reporters* though.

> To be a little bit more upfront: I think that we as a community
> sometimes focus on "is this thing reproducible" and not on "how can I
> use this to secure the ecosystem".

It’s definitely time to clarify that.

I think these things need to happen in stages. Broadly:
1. Get key applications & libraries reproducible (assuming toolchains are okay)
2. Establish independent processes that *check* that the binaries are what 
they’re supposed to be.
3. Extend the work to more/all applications/libraries in given domains.
4. Work on verifying underlying toolchains, and again, creating independent 
processes that *check* the toolchain results (DDC & bootstrapping).

The long-term goal should be that “we can ensure that all OSS compiled code is 
accurately represented by its source code”. The source code may include 
malicious statements, but source code is what developers review, so we’ve 
fundamentally changed the game to ensure that “what is reviewed is what is run”.

--- David A. Wheeler

Re: Attack on SolarWinds could have been countered by reproducible builds

[Santiago Torres-Arias]

Hello.

On Thu, Dec 17, 2020 at 07:33:11PM -0500, David A. Wheeler wrote:
> All:
> 
> There’s been a recently-revealed attack on the SolarWinds product “Orion", a 
> Network Management System (NMS). This software is widely used and thus this 
> attack is extremely concerning.
> 
> According to SANS, "SolarWinds has published limited information in which 
> they state they believe the build environment was compromised.” 
> https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/
> 
> Let me restate this: it appears that the *source code* wasn’t compromised, 
> and the *distribution* system wasn’t compromised. Instead, the *build system* 
> was compromised. This is *EXACTLY*  the kind of attack that is countered by 
> reproducible builds. Thus, the recent SolarWinds subversion is a very good 
> argument for why it’s important to have reproducible builds (and to verify 
> builds using reproducible builds).
> 
> I’ve read a number of articles about SolarWinds, and none of them mention 
> reproducible builds, even though reproducible builds is clearly a 
> countermeasure to this problem. Perhaps journalists will eventually learn 
> about reproducible builds; that would be nice!

I agree that we need more visibility on the reprobuilds aspect of this
compromise. For my side (speaking as an in-toto maintainer), we've been
banging the drum on the use of reprobuilds to stop these types of
compromises. In fact, we generally say that, before securing the whole
chain, something like reprobuilds on the build stage precludes any type
of software supply chain security measures (I'm biting my tongue here
avoiding to use a "weakest link" type of metaphor).

Having said this. I think it's important as a community to highlight
that reproducibility is not sufficient if there is no checking of build
artifacts (and now I'm biting my tongue here trying to not mention
"trees falling in a forest when nobody is around"). This is why I'm
incredibly excited about rebuilderd, and I'd encourage everyone to put
more hands on deck on that project.

Personally, on most of the mentions of in-toto solving the solarwinds
attack, we mention that the policy enforcement of in-toto allows for
semantics describing reproducible (and authenticated) rebuilds from
trusted parties. It also allows to authenticate components in the build
environment (which may, or may not proect against this particular
attack)

To be a little bit more upfront: I think that we as a community
sometimes focus on "is this thing reproducible" and not on "how can I
use this to secure the ecosystem". I think it's been this way since
perahps the r-b summit on 2018.

Hell, to be even more upfront, we are at a great time of the year to
rethink our priorities and efforts now that the calendar year is about
to reset :)

What do you guys think?
-Santiago


signature.asc
Description: PGP signature

Re: Attack on SolarWinds could have been countered by reproducible builds

[David Kleuker]

it don't help much to rant on this ML where all people know what reproducible 
builds are. instead contacting all those journalists that did not mention it 
has a chance to change the current status.

a publication on reproducible-builds.org about this incident would also be 
helpful to share the link

next time this happens, journalists would at least know they COULD mention it

kind regards
David Kleuker

> Chris Lamb  hat am 21.12.2020 15:30 
> geschrieben:
> 
>  
> David A. Wheeler wrote:
> 
> > Let me restate this: it appears that the *source code* wasn’t
> > compromised, and the *distribution* system wasn’t compromised. Instead,
> > the *build system* was compromised.
> 
> Thanks for this, David. You are absolutely right that this is exactly
> what Reproducible Builds was 'designed' for to begin with. An ironic
> hurrah that this kind of attack is getting more visibility these days.
> 
> Another thanks for the press references too -- I will make good use of
> them when writing our next monthly report. (Alas, if it wasn't the
> holiday season I might be tempted to suggest that we do a specific
> publicity boost based on this..)
> 
> 
> Regards,
> 
> --
>   o
> ⬋   ⬊  Chris Lamb
>o o reproducible-builds.org 💠
> ⬊   ⬋
>   o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Chris Lamb]

David A. Wheeler wrote:

> Let me restate this: it appears that the *source code* wasn’t
> compromised, and the *distribution* system wasn’t compromised. Instead,
> the *build system* was compromised.

Thanks for this, David. You are absolutely right that this is exactly
what Reproducible Builds was 'designed' for to begin with. An ironic
hurrah that this kind of attack is getting more visibility these days.

Another thanks for the press references too -- I will make good use of
them when writing our next monthly report. (Alas, if it wasn't the
holiday season I might be tempted to suggest that we do a specific
publicity boost based on this..)


Regards,

--
  o
⬋   ⬊  Chris Lamb
   o o reproducible-builds.org 💠
⬊   ⬋
  o

Re: Attack on SolarWinds could have been countered by reproducible builds

[Hans-Christoph Steiner]

Thanks for this info!  RB work can be a slog through annoying technical 
details, so confirmation of its important always helps lift my spirits. 
 Its definitely good fodder for getting funding for related work.


.hc

David A. Wheeler:

All:

There’s been a recently-revealed attack on the SolarWinds product “Orion", a 
Network Management System (NMS). This software is widely used and thus this attack 
is extremely concerning.

According to SANS, "SolarWinds has published limited information in which they 
state they believe the build environment was compromised.” 
https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/

Let me restate this: it appears that the *source code* wasn’t compromised, and 
the *distribution* system wasn’t compromised. Instead, the *build system* was 
compromised. This is *EXACTLY*  the kind of attack that is countered by 
reproducible builds. Thus, the recent SolarWinds subversion is a very good 
argument for why it’s important to have reproducible builds (and to verify 
builds using reproducible builds).

I’ve read a number of articles about SolarWinds, and none of them mention 
reproducible builds, even though reproducible builds is clearly a 
countermeasure to this problem. Perhaps journalists will eventually learn about 
reproducible builds; that would be nice!

--- David A. Wheeler

PS: Here are some articles about the attack on SolarWinds:
https://www.sans.org/blog/what-you-need-to-know-about-the-solarwinds-supply-chain-attack/
 

https://www.cnn.com/2020/12/16/tech/solarwinds-orion-hack-explained/index.html 

https://www.csoonline.com/article/3601508/solarwinds-supply-chain-attack-explained-why-organizations-were-not-prepared.html
 

https://www.computerweekly.com/news/252493662/SolarWinds-cyber-attack-How-worried-should-I-be-and-what-do-I-do-now
 

https://www.washingtonpost.com/technology/2020/12/14/russia-hack-us-government/ 





--
PGP fingerprint: EE66 20C7 136B 0D2C 456C  0A4D E9E2 8DEA 00AA 5556
https://pgp.mit.edu/pks/lookup?op=vindex&search=0xE9E28DEA00AA5556