On 03/21/2015 11:39 AM, Sylvain Le Gall wrote: > > > Le sam. 21 mars 2015 à 10:35, Sylvain Le Gall <[email protected] > <mailto:sylvain%[email protected]>> a écrit : > > > > Le sam. 21 mars 2015 à 00:22, Török Edwin <[email protected] > <mailto:edwin%[email protected]>> a écrit : > > Hi, > > [On Anil's suggestion I'm moving the discussion to opam-devel. > TL;DR: I am about to start writing opam2{rpm,deb} for my package and > its dependencies. Does anyone else need such a tool or want to help?] > > I am about to package my own OCaml application for Fedora and Debian, > which indirectly depends on 12 opam packages > (3 of which already review requests on Fedora's bugzilla as they are > shared dependencies with opam itself). > I am looking for some automation in preparing spec and Debian > templates, for checking their quality, and tracking the packages through > their lifecycle, > and to avoid making the same common mistakes for each package. > > Since such a tool (opam2rpm/opam2deb) doesn't exist now, I'll > describe what my idea/requirements are. > Please let me know what you think and if you have new / different > requirements. Perhaps together we can design/create an opam2rpm tool that > will be useful for more than just a oneshot conversion. > > I am quite new to packaging, have just recently went through > packaging another (non-OCaml) application for Fedora so I've > documented a package's (and package maintainer's) states here: > https://fedoraproject.org/__wiki__/Fedora_Package_Lifecycle___note__s > <https://fedoraproject.org/wiki/Fedora_Package_Lifecycle_notes>. > This is still rather incomplete and rough, take it with a grain of > salt and consult the official docs when in doubt. > > What we have now: > * opam metadata: depexts, conf-* opam packages, build rules, > constraints, etc. > * opam-lib > * opam-repository, where some packages implicitly assume a "modern" > Linux environment (has native compiler, dynlink, certain tools and C headers, > etc.). > For distribution packages all these dependencies will have to be made > explicit > * oasis2deb, generates decent starting point for a Debian package, > but fails #3 > > > Not sure to understand how oasis2deb fails #3 (test that build > requirement is complete). > > It actually declares build dependencies for the source package, but it > doesn't embed the source of the dependencies (which is by the way forbidden > by Debian policy) but depends on what should be the package to do it. I.e. it > integrates into the target build system. > > > Sentence got mixed in my head: > It actually declares build dependencies for the source package, but it > doesn't embed the source of the dependencies. This is, by the way, forbidden > by Debian policy. So it depends on what the expected build-depends package > should be (it tries to derive it from the findlib package name). The overall > generated package integrates into the target build system. > >
Sorry I should've been more clear that this is not a fault of oasis2debian, but it simply is outside the scope of OASIS to fully specify distribution specific *non-OCaml* dependencies. For example build-time dependencies on C libraries (zlib, sqlite, etc.), or runtime dependencies on non-OCaml tools. opam has such metadata (depexts or conf-*) -- although perhaps not fully specified for each package yet -- and in fact it might be possible for opam2deb to just call oasis2debian with --library-dev-extra-depends and other appropriate flags when the package is actually using oasis. This is what I meant when I said "detect that oasis is used and simplify some steps" > > Example on my dlnasync generated package: > > Source: dlnasync > Section: ocaml > Priority: optional > Maintainer: Debian OCaml Maintainers <[email protected] > <mailto:[email protected]>> > Uploaders: > Sylvain Le Gall <[email protected] <mailto:[email protected]>> > Build-Depends: > ocaml-nox (>= 3.11.1-3~), > ocaml-findlib, > ocaml-base-nox, > libsekred-ocaml-dev (>= 0.2.0), > libounit-ocaml-dev (>= 2.0.0), > libocamlnet-ocaml-dev (>= 3.5.1), > libfileutils-ocaml-dev (>= 0.4.2), > libbatteries-ocaml-dev, > libav-tools, I think you give a good example here, you had to use --library-dev-extra-depends to add libav-tools here, but where do I store that so I will have it next time when upstream releases a new version and the deb package has to be updated? opam metadata seems like a good place to store that information that benefits other tools as well (opam-installext, or the yet non-existent opam2rpm/oasis2rpm). This is also my point #4, although oasis2debian can be rerun and standard diff tools used to compare what changed. > dh-ocaml (>= 0.9~), > debhelper (>= 7.0.50~) > Standards-Version: 3.9.1 > Homepage: http://forge.ocamlcore.org/projects/dlnasync > Vcs-Git: git://git.debian.org/git/pkg-ocaml-maint/packages/dlnasync.git > <http://git.debian.org/git/pkg-ocaml-maint/packages/dlnasync.git> > Vcs-Browser: > http://git.debian.org/?p=pkg-ocaml-maint/packages/dlnasync.git > > You can see that all build dependencies are here. > > I made this decision on purpose. If you want to push this kind of tools > to build packages to be distributed by the normal RPM/Deb scheme of your > distribution, they need to somehow comply with policy. > > (I also created tools that DON'T do that but I don't publish them because > they are basic hacks) Yes, complying with policy is my intention, which is exactly why I want a tool that allows me to easily package all the dozen or so of my dependencies as separate packages, leveraging the already existing opam/OASIS metadata (and manually improving them where they are not complete). BTW could you add oasis2debian to opam, or tell me where I can download debian-formats? I forgot where I got it from last time. > > > > What are my requirements? > #1 (local) opam repository as input, .spec/Debian as output > #2 lint the output (both opam and possibly manually edited > spec/Debian), there are already external tools for this > #3 test that the build requirements are complete, i.e. it builds in a > minimal environment. there are external tools mock/pbuilder for a > clean/official solution, > or docker for an insecure and somewhat faster solution > #4 allow to manually modify the generated .spec and debian/ files and > still keep track of changes in opam metadata > #5 allow incremental refinement of the spec/debian rules, don't > require everything to be 100% compliant/correct syntax > #6 allow to override opam metadata (could just git clone the opam > repo and use that) > #7 opam2rpm and opam2deb should be similar to use (i.e. maybe it > should be opam2pkg rpm and opam2pkg deb), and I don't necesarely want to > limit this to rpm and deb, > someone is welcome to create an opam2openbsd or opam2brew, etc. > #8 developing the automation tool shouldn't take (significanlty) > longer than manually writing the packaging for those 9 packages :) > #9 the goal would also be to make it easier for upstream reviewers to > review all these packages, at some point we should consider involving them in > the discussion once we have some draft tool > to make sure we generate good quality packages :) > > Most (but not all) of the packages I need to write spec files for are > based on OASIS, so we could try to load the missing metadata from oasis, or > detect that oasis is used > and simplify some steps, however not all packages use it so I think > that starting at the opam metadata is still the right way. > > What kind of automation is needed for opam2rpm and opam2deb for me: > * mapping of opam to Fedora/Debian naming conventions (some opam > packages have ocaml- already in them, so take care not to duplicate that) > * generate .spec and debian/ file templates from opam metadata + > some overrides. If this was just a one-way/one-time conversion I'd go with > something simple (perhaps like ocaml-mustache), however see below > * check license name compliance (Fedora and Debian have their > standard name lists), and make sure the package actually has a license *file* > * don't need full roundtrip capability between opam and spec/debian, > the tool should serve three purposes only: > * generate initial templates > * show a smart diff > * lint the opam metadata / spec file > * shouldn't require to update the conversion tool just to support an > obscure corner-case that can be fixed more easily by hand in the spec file > * doesn't have to be perfect or support the full features of > spec/debian/opam files. lets start with the minimum requirements and add > fields/features as we need them > * its not a substitute for human review, the final output always has > to be tested and reviewed by a human > > Implementation thoughts: > * ocaml module describing spec file an debian rules with a toplevel > type t and nested records, with type 'a field > * some form of conversion to/from a string -> string map > * a lexer that loads that map from the spec/debian files, and a > formatter that generates it (ocamllex + format, or just ocaml-re + printf?) > * please if we can lets avoid camlp4 for this, I'd suggest > ppx_deriving+cconv or just manually writing the parse/generate calls for the > record fields > * validation should be optional > * fields should be easy to declare (name, of_string, to_string, > validate, smart_diff), initially all would be raw strings, more > parsing/diffing can be added as needed > * some way to fill the 't' type from opam metadata > * certain files may not need all this advanced conversion, and might > be easier to generate from a text template with substitutions > (ocaml-mustache, or mpp?), and just show a textual diff > (docdiff at character/word level is quite nice) > > Automation for tracking a package through its lifecycle (separate > tool): > Since packaging an OCaml application can involve packaging/tracking a > large number of dependant libraries some tool to track the status of each > package (lifecycle) > would be useful. This is not specific to OCaml, I'm surprised that > upstream distros don't have this already, but anyway using Fedora terminology > I want a single CLI tool that can: > * query yum repo for stable packages > * query pkgdb for existence of package in other releases > * query bodhi for pending/testing packages > * query koji for builds submitted/completed > * query for existence of scm (git) repo > * query bugzilla for package review and find out its state > (submitted, under review, approved, waiting for scm repo, scm repo created) > For Debian a similar tool that: > * query bugs.debian.org <http://bugs.debian.org> for RFP (request > for packaging), ITP (intent to package) and its status > * query NEW queue for uploaded source package > * query package tracker for buildd status > * this can also avoid duplicating work, i.e. finding out that > someone else is already packaging my package or its dependencies > * might also be useful for other packages > * can be done manually by visiting a bunch of webpages in turn and > searching on each, but there ought to be some CLI tools to help or at least > some (web)API > * doesn't have to be necesarely written in Ocaml, if we can reuse > existing tools and have this just as a shell/python script the better > > I wanted to go through the policy docs and write some details here on > what is missing from opam metadata, but this mail is long enough and it is > late enough already, > that I thought to start the discussion with what I have so far, and > discuss the details as we go. > > Once we agree on the major design points of this tool I plan on > starting with opam2rpm, but if someone else wants to own it I'd be more than > happy with that as well. > > Anil also suggested to look at: > On 03/19/2015 10:15 AM, Anil Madhavapeddy wrote: > > This annotated changeset about a recent update from Jon might be > useful as well: > > > > > https://github.com/jonludlam/__o__caml4021-buildroot/commit/__9ef9__554c283ea1cda8f64742b73bb6__3807__b791c7#commitcomment-__10071274 > > <https://github.com/jonludlam/ocaml4021-buildroot/commit/9ef9554c283ea1cda8f64742b73bb63807b791c7#commitcomment-10071274> > > > > Best regards, > --Edwin > ___________________________________________________ > opam-devel mailing list > [email protected] <mailto:[email protected]> > http://lists.ocaml.org/__listinf__o/opam-devel > <http://lists.ocaml.org/listinfo/opam-devel> > _______________________________________________ opam-devel mailing list [email protected] http://lists.ocaml.org/listinfo/opam-devel
