Ok, so this really seems the moment to start the pluging infrastructure. A first draft proposal for the depext plugin API is in
doc/design/depexts-plugins we may add one for the sandpoxing API, and then see what can be shared. On Fri, Feb 27, 2015 at 10:27:27AM +0900, Louis Gesbert wrote: > A plugin mechanism was actually within my possible plans for 1.3, with a > range of features in view including depexts, better agnosticity (an OCaml > plugin), etc. But that would probably be too much for the next milestone. > > On the other hand, the entry point for calling external processes is quite > clean now, and already has metadata like working dir specified in the call. > So adding sandbox permissions to these parameters, and delegating process > startup to an OS-dependent new module handling sandboxes could probably be > done without adding too much noise to the code. > > Cheers, > Louis > > > - Roberto Di Cosmo, 25/02/2015 11:58 - > > I fully share this point of view: having opam in control of the sandbox > > provides much finer grain. So, from a functional standpoint, I would not > > hesitate. > > > > What makes me hesitate is that we would need to change the opam code for > > this, and take into account all possible different mechanisms according to > > os/platform. This is pretty well doable, if we set up a proper plugin > > mechanism, and we could share code with a plugin-based depext support, but > > it requires more effort :-) > > > > > > 2015-02-25 11:54 GMT+01:00 Anil Madhavapeddy <[email protected]>: > > > > > I'm also not sure how viable activating it for day-to-day testing except > > > by advanced users is. However, we can at least get the infrastructure in > > > place for CI, which is necessary before we need to decide that. > > > > > > I forgot one point in my previous mail: why this needs to be in OPAM > > > instead > > > of just an external `sandbox` binary. We don't actually want to sandbox > > > OPAM itself (which we trust!), but rather the packages it invokes. In my > > > OpenBSD sandbox, I am constraining OPAM and all its future subshells. > > > Instead, I would prefer to have OPAM invoke an external sandbox only when > > > it is doing a build, with sufficient arguments to let it know which dirs > > > are ok to write into. > > > > > > This would let a package only write into the specific subdirectory > > > relevant > > > to the current switch. This actually bites me quite often with pinning, > > > since a stale OASIS setup.data can cause an installation to write into the > > > wrong switch if a full clean is not done ahead of time. It also lets us > > > sandbox network access of builds independently of OPAM, which is needed to > > > "unplug" an OPAM package build. > > > > > > cheers, > > > Anil > > > > > > > On 25 Feb 2015, at 10:48, Roberto Di Cosmo <[email protected]> wrote: > > > > > > > > Many thanks Anil, that's precious information: > > > > this is clearly another very OS-specific feature! > > > > > > > > The question is whether we want to add these policies and tools > > > > > > > > - outside opam (pro: no changes to opam, cons: opam has no control) > > > > - or make opam aware of them (pro: opam gets control, cons: various > > > changes to opam) > > > > > > > > For CI testing, the "outside" approach seems to be quite sufficient. > > > > > > > > For day to day use, though, I am not sure... > > > > > > > > On Tue, Feb 24, 2015 at 04:22:31PM +0000, Anil Madhavapeddy wrote: > > > >> Hi Roberto, Simon, > > > >> > > > >> Sandboxing mechanisms have come along quite a bit in the last few > > > >> years. > > > >> It's important to separate the two threat models that we want, since > > > their > > > >> use can be quite intrusive if made mandatory. I see two modes of > > > operation: > > > >> > > > >> - A mandatory sandbox in CI testing, where we use it to check that the > > > >> package isn't violating obvious policies such as network downloads > > > >> from within the package, or writing outside of ~/.opam or /tmp. > > > >> > > > >> - Optional sandbox for day-to-day use by end users. This will catch > > > >> even malicious behaviours, but imposes a rather heavy support burden. > > > >> > > > >> In terms of how to sandbox applications, I know of: > > > >> > > > >> - OpenBSD: I use the built-in systrace system call permissions > > > >> subsystem > > > >> already with OPAM. I've uploaded my local policy to GitHub at: > > > >> https://gist.github.com/avsm/8293aa52c6cee772a9cb > > > >> This policy is used by "systrace opam install foo" and pulls up an > > > >> interactive dialog if an application attempts to write outside of > > > >> either ~/.opam or /tmp. > > > >> See: http://www.openbsd.org/cgi-bin/man.cgi?query=systrace > > > >> or: http://www.citi.umich.edu/u/provos/systrace/ > > > >> paper: http://www.citi.umich.edu/u/provos/papers/systrace.pdf > > > >> > > > >> - FreeBSD: the latest versions have the Capsicum capability system > > > >> integrated, and the Casper system daemon that grants common operations > > > >> is also being worked on upstream (e.g. for name resolution) > > > >> See: https://www.cl.cam.ac.uk/research/security/capsicum/ > > > >> > > > >> - Linux: As always with Linux, there are a myriad of possible > > > >> solutions. > > > >> I'd discourage the use of LD_PRELOAD based solutions since they don't > > > >> work in several situations reliably (most obviously with static > > > binaries). > > > >> The fakeroot-ng project uses ptrace instead, which is more reliable. > > > >> > > > >> However, the primary thing we want is to only let the package only > > > >> write into ~/.opam, and so the mount namespace feature (see > > > CLONE_NEWNS) > > > >> may be sufficient for our needs -- a lightweight filesystem container, > > > >> in effect. David Sheets has done some investigation into this for > > > >> another project we're working on. > > > >> > > > >> - Windows: various, Sandboxie is one solution, but the underlying > > > >> mechanism is the Windows Integrity Mechanism that was introduced in > > > >> Vista: https://msdn.microsoft.com/en-us/library/bb625964.aspx > > > >> This lets applications drop privileges, and is used by the Chrome > > > >> sandbox (in general, following what Chrome does for sandboxing is a > > > >> good idea, since their needs are a superset of ours). > > > >> > > > >> - MacOS X: The App Sandbox requires code signing, but does almost > > > >> exactly what we need: it gives each app/user a unique directory > > > >> into which they can write files. All we should need to do is to > > > >> set OPAMROOT to that directory, and everything should just work. > > > >> In practise, this requires some investigation into how the App > > > >> signing infrastructure works, since I've only seen it done for > > > >> bundles and not for CLI tools. > > > >> > > > >> This is a quick off-the-top-of-my-head survey, but I think it's viable > > > >> and useful for us to build an `opam sandbox` in the same style as > > > >> `opam depext` that attempts to invoke a relevant sandboxing mechanism > > > >> depending on the OS. In the longer term, this will really serve us > > > >> well as the package database grows. > > > >> > > > >> I'm less sure about the viability of recording installed files > > > >> strictly -- I view thatas an advisory rather than enforcement-based > > > >> mechanism. The reason I like the "make ~/.opam a git store" is that > > > >> its possible for applications to write directly into the store as they > > > >> do right now, but still let us track changes precisely. In fact, if > > > >> we forbid subshells from writing into `~/.opam/.git`, this would be > > > >> a production grade solution that also offers instant-rollback in case > > > >> of compilation errors (no more waiting for a full recompilation of > > > >> the original dependencies!). > > > >> > > > >> cheers, > > > >> Anil > > > >> > > > >>> On 21 Feb 2015, at 09:16, Roberto Di Cosmo <[email protected]> > > > wrote: > > > >>> > > > >>> Anil, Simon, can you provide more details on the sandboxing mechanisms > > > you know of? > > > >>> > > > >>> We looked into all this for Mancoosi years ago; the most complete tool > > > >>> out there was installwatch (now checkinstall) that hijacks filesystem > > > modifying > > > >>> commands using the standard LD_PRELOAD trick and a wrapper for system > > > calls. > > > >>> Checkinstall does not alter user priviledges, though, so one sometimes > > > needed > > > >>> a combination of fakeroot (that only alter user priviledges) with it. > > > >>> > > > >>> The best approach I know of was described in a Master thesis from ... > > > Cambridge > > > >>> :-) It was under the supervision of Peter Sewell, and used the ptrace > > > mechanism > > > >>> instead of the LD_PRELOAD trick, because LD_PRELOAD is blind to > > > statically > > > >>> compiled binaries that have system calls hardcoded, while ptrace gets > > > them all. > > > >>> > > > >>> The dissertation is still available today here > > > http://robot101.net/files/diss.ps.gz > > > >>> and contains a very nice discussion of the issues related to > > > monitoring and > > > >>> rolling back file system changes performed by a command in the Linux > > > system. > > > >>> The source code is also available here > > > http://robot101.net/files/trace.tar.gz > > > >>> and one can get in touch with Robert Mcqueen that will be delighted to > > > see his > > > >>> work being used. > > > >>> > > > >>> Since all this is almost 10 years old, I suppose many exciting new > > > ideas, tools > > > >>> and approaches surfaced in the meantime, and I'd really like to know > > > more. > > > >>> > > > >>> Cheers > > > >>> > > > >>> -- > > > >>> Roberto > > > >>> > > > >>> On Sat, Feb 21, 2015 at 09:37:07AM +0100, Simon Cruanes wrote: > > > >>>> Sandboxing the build would also be a big security improvement. I > > > think cabal > > > >>>> now does it, and signing packages doesn't protect against malicious > > > or buggy > > > >>>> packages (see: bumblebee's uninstall target). That also goes hand in > > > hand with > > > >>>> file tracking. I don't know how difficult it is, though. > > > >>>> > > > >>>> Cheers! > > > >>>> > > > >>>> Le 21 février 2015 05:01:56 UTC+01:00, Louis Gesbert > > > >>>> <[email protected]> a écrit : > > > >>>> > > > >>>> With 1.2.1 almost out of the door, time has come to review the > > > roadmap discussed back in December and choose where we'll be going for > > > 1.3. > > > Original mail attached for reference. > > > >>>> > > > >>>> > > > >>>> The topic that is burning hot at the moment is, specially after the > > > Debian Haskell build host has been compromised, security: we have no > > > signing at all at the moment, and we need to improve on this before it > > > becomes a problem. TUF [1] has devised a sane amount of rules for > > > repository management and signing that should make it easier to get it > > > right in OPAM. Hannes has mentionned writing an OCaml implementation for > > > TUF, which could get very helpful. > > > >>>> > > > >>>> > > > >>>> Also of importance is Windows support. It should at least be > > > straighforward and documented to get a basic Cygwin setup working. That > > > goes with adding automated tests (appveyor can now be added in Github > > > alongside Travis). Related is cleaning up external command usage (even if > > > not really justified by a Windows > > > >>>> port only, as David Allsopp pointed out) -- replacing curl calls by > > > cohttp, use ocaml-fileutils... > > > >>>> > > > >>>> > > > >>>> These are the other main features, that'll probably take more time > > > if we are to focus eg. on security: > > > >>>> > > > >>>> * a plugin mechanism with plugins for example for OCaml (for better > > > agnosticity), external dependency handling [2], documentation > > > generation... > > > >>>> > > > >>>> * a 'provides:' field in OPAM files [3]. This is a loose > > > requirement if we want to switch the repository to have OCaml itself in a > > > package (which is already possible, but the system compiler may yet be an > > > issue). > > > >>>> > > > >>>> * More flexible switch handling (multi-switch packages, per-switch > > > remotes, layered switches for cross-compilation...) > > > >>>> > > > >>>> * Tracking of files installed by packages. While unrelated to repo > > > signing, this might have some security implications, so we might want to > > > bring it in. > > > >>>> > > > >>>> * With file tracking, generating a binary repo (with some > > > limitations) could be quite > > > >>>> straight-forward. > > > >>>> > > > >>>> > > > >>>> Which of these do you think is most important ? Have I forgotten > > > anything ? > > > >>>> > > > >>>> Cheers, > > > >>>> Louis > > > >>>> > > > >>>> > > > >>>> [1] http://theupdateframework.com/ > > > >>>> [2] > > > https://github.com/ocaml/opam/blob/master/doc/design/depexts-plugins > > > >>>> [3] > > > https://github.com/ocaml/opam/blob/master/doc/design/provides.md > > > >>>> > > > >>>> message suivi > > > >>>> > > > >>>> De : Louis Gesbert > > > >>>> À : [email protected] > > > >>>> Envoyé : Wed Dec 17 11:07:40 UTC+01:00 2014 > > > >>>> Objet : [opam-devel] OPAM Roadmap -- what next ? > > > >>>> > > > >>>> Hi all, > > > >>>> > > > >>>> > > > >>>> > > > >>>> with some lag after the 1.2 release, I'd like to open some space > > > for a > > > >>>> collective discussion of the priorities for where the energies > > > should go > > > >>>> next. We have mainly 3 directions for improvements: first, > > > portability, > > > >>>> with the main goal of a Windows version. Second, agnosticity, with > > > the goal > > > >>>> to make OPAM more generic, and try and open it to users outside of > > > the > > > >>>> OCaml community (wouldn't OPAM for Haskell be fun ?). Third, there > > > are > > > >>>> always lots of features and improvements that have been asked for, > > > and > > > >>>> would improve the experience of current users. > > > >>>> > > > >>>> > > > >>>> > > > >>>> So here is a summary of what I've gathered. Feel free to add your > > > own. > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> ## Portability > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Rewrite parallel command engine.** / done. > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Native system manipulation (cp, rm, curl...).** > > > >>>> > > > >>>> These are mostly done through calls to the shell or external > > > programs. It's > > > >>>> > > > >>>> not very pretty but quite pragmatic actually... until we want to > > > run on > > > >>>> > > > >>>> Windows without Cygwin. Anyway, this wouldn't be the end of > > > portability > > > >>>> > > > >>>> issues. > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Windows support.** > > > >>>> > > > >>>> for OPAM itself to begin with. This should be pretty close. > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Packages on Windows.** > > > >>>> > > > >>>> Locate common issues and attempt to find generic fixes. > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Allow **direct use of more solvers** or solver servers. > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Support cross-compilation** > > > >>>> > > > >>>> This is quite an open issue at the moment. > > > >>>> > > > >>>> > > > >>>> > > > >>>> ## Agnosticity > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Isolate OCaml-specific stuff.** > > > >>>> > > > >>>> E.g. specific opam-file variables. See ocaml-specific.md > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Separate as plugins** > > > >>>> > > > >>>> To open the gate to OPAM without these plugins, or with other ones > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Compilers as packages.** > > > >>>> > > > >>>> This should mostly work now, but needs some tests and > > > strengthening. The > > > >>>> main > > > >>>> > > > >>>> thing still to do is to handle the system compiler changes properly > > > ; that > > > >>>> > > > >>>> part should be made more generic anyway (see discussion on hooks) > > > >>>> > > > >>>> > > > >>>> > > > >>>> ## Features > > > >>>> > > > >>>> > > > >>>> > > > >>>> - A **provides** field. Generally useful, but particulary so with > > > >>>> > > > >>>> compilers-as-packages > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Releasing the **"features" field** for easier package > > > configuration > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Track installed files** > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Improve security**: just checking md5s is quite light ; package > > > scripts > > > >>>> can > > > >>>> > > > >>>> write anywhere > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **OS-specific handling of dependencies** (eg dependencies on > > > packages > > > >>>> from the > > > >>>> > > > >>>> OS) with plugins (#1519) > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Specify and implement **hooks** > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Find a nicer way to **share dev repos** / undoable "pinning > > > sources" > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Per-switch remotes** > > > >>>> > > > >>>> > > > >>>> > > > >>>> - **Multi-switch packages** > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Support for (automatic generation of) **binary packages** > > > >>>> > > > >>>> > > > >>>> > > > >>>> - Nicer **ocamlfind interaction** > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> Cheers, > > > >>>> > > > >>>> Louis Gesbert > > > >>>> > > > >>>> > > > > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > > > >>>> > > > >>>> opam-devel mailing list > > > >>>> [email protected] > > > >>>> http://lists.ocaml.org/listinfo/opam-devel > > > >>>> > > > >>>> > > > > > > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ > > > >>>> > > > >>>> opam-devel mailing list > > > >>>> [email protected] > > > >>>> http://lists.ocaml.org/listinfo/opam-devel > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Simon > > > >>> > > > >>>> _______________________________________________ > > > >>>> opam-devel mailing list > > > >>>> [email protected] > > > >>>> http://lists.ocaml.org/listinfo/opam-devel > > > >>> > > > >>> > > > >>> -- > > > >>> Roberto Di Cosmo > > > >>> > > > >>> ------------------------------------------------------------------ > > > >>> Professeur En delegation a l'INRIA > > > >>> PPS E-mail: [email protected] > > > >>> Universite Paris Diderot WWW : http://www.dicosmo.org > > > >>> Case 7014 Tel : ++33-(0)1-57 27 92 20 > > > >>> 5, Rue Thomas Mann > > > >>> F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo > > > >>> FRANCE. Twitter: http://twitter.com/rdicosmo > > > >>> ------------------------------------------------------------------ > > > >>> Attachments: > > > >>> MIME accepted, Word deprecated > > > >>> http://www.gnu.org/philosophy/no-word-attachments.html > > > >>> ------------------------------------------------------------------ > > > >>> Office location: > > > >>> > > > >>> Bureau 3020 (3rd floor) > > > >>> Batiment Sophie Germain > > > >>> Avenue de France > > > >>> Metro Bibliotheque Francois Mitterrand, ligne 14/RER C > > > >>> ----------------------------------------------------------------- > > > >>> GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3 > > > >>> > > > >> > > > > > > > > -- > > > > Roberto Di Cosmo > > > > > > > > ------------------------------------------------------------------ > > > > Professeur En delegation a l'INRIA > > > > PPS E-mail: [email protected] > > > > Universite Paris Diderot WWW : http://www.dicosmo.org > > > > Case 7014 Tel : ++33-(0)1-57 27 92 20 > > > > 5, Rue Thomas Mann > > > > F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo > > > > FRANCE. Twitter: http://twitter.com/rdicosmo > > > > ------------------------------------------------------------------ > > > > Attachments: > > > > MIME accepted, Word deprecated > > > > http://www.gnu.org/philosophy/no-word-attachments.html > > > > ------------------------------------------------------------------ > > > > Office location: > > > > > > > > Bureau 3020 (3rd floor) > > > > Batiment Sophie Germain > > > > Avenue de France > > > > Metro Bibliotheque Francois Mitterrand, ligne 14/RER C > > > > ----------------------------------------------------------------- > > > > GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3 > > > > > > > > > > > > > > > > -- Roberto Di Cosmo ------------------------------------------------------------------ Professeur En delegation a l'INRIA PPS E-mail: [email protected] Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo FRANCE. Twitter: http://twitter.com/rdicosmo ------------------------------------------------------------------ Attachments: MIME accepted, Word deprecated http://www.gnu.org/philosophy/no-word-attachments.html ------------------------------------------------------------------ Office location: Bureau 3020 (3rd floor) Batiment Sophie Germain Avenue de France Metro Bibliotheque Francois Mitterrand, ligne 14/RER C ----------------------------------------------------------------- GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3 _______________________________________________ opam-devel mailing list [email protected] http://lists.ocaml.org/listinfo/opam-devel
