Hi,
[On Anil's suggestion I'm moving the discussion to opam-devel.
TL;DR: I am about to start writing opam2{rpm,deb} for my package and its
dependencies. Does anyone else need such a tool or want to help?]
I am about to package my own OCaml application for Fedora and Debian, which
indirectly depends on 12 opam packages
(3 of which already review requests on Fedora's bugzilla as they are shared
dependencies with opam itself).
I am looking for some automation in preparing spec and Debian templates, for
checking their quality, and tracking the packages through their lifecycle,
and to avoid making the same common mistakes for each package.
Since such a tool (opam2rpm/opam2deb) doesn't exist now, I'll describe what my
idea/requirements are.
Please let me know what you think and if you have new / different requirements.
Perhaps together we can design/create an opam2rpm tool that will be useful for
more than just a oneshot conversion.
I am quite new to packaging, have just recently went through packaging another
(non-OCaml) application for Fedora so I've
documented a package's (and package maintainer's) states here:
https://fedoraproject.org/wiki/Fedora_Package_Lifecycle_notes.
This is still rather incomplete and rough, take it with a grain of salt and
consult the official docs when in doubt.
What we have now:
* opam metadata: depexts, conf-* opam packages, build rules, constraints, etc.
* opam-lib
* opam-repository, where some packages implicitly assume a "modern" Linux
environment (has native compiler, dynlink, certain tools and C headers, etc.).
For distribution packages all these dependencies will have to be made explicit
* oasis2deb, generates decent starting point for a Debian package, but fails #3
What are my requirements?
#1 (local) opam repository as input, .spec/Debian as output
#2 lint the output (both opam and possibly manually edited spec/Debian), there
are already external tools for this
#3 test that the build requirements are complete, i.e. it builds in a minimal
environment. there are external tools mock/pbuilder for a clean/official
solution,
or docker for an insecure and somewhat faster solution
#4 allow to manually modify the generated .spec and debian/ files and still
keep track of changes in opam metadata
#5 allow incremental refinement of the spec/debian rules, don't require
everything to be 100% compliant/correct syntax
#6 allow to override opam metadata (could just git clone the opam repo and use
that)
#7 opam2rpm and opam2deb should be similar to use (i.e. maybe it should be
opam2pkg rpm and opam2pkg deb), and I don't necesarely want to limit this to
rpm and deb,
someone is welcome to create an opam2openbsd or opam2brew, etc.
#8 developing the automation tool shouldn't take (significanlty) longer than
manually writing the packaging for those 9 packages :)
#9 the goal would also be to make it easier for upstream reviewers to review
all these packages, at some point we should consider involving them in the
discussion once we have some draft tool
to make sure we generate good quality packages :)
Most (but not all) of the packages I need to write spec files for are based on
OASIS, so we could try to load the missing metadata from oasis, or detect that
oasis is used
and simplify some steps, however not all packages use it so I think that
starting at the opam metadata is still the right way.
What kind of automation is needed for opam2rpm and opam2deb for me:
* mapping of opam to Fedora/Debian naming conventions (some opam packages have
ocaml- already in them, so take care not to duplicate that)
* generate .spec and debian/ file templates from opam metadata + some
overrides. If this was just a one-way/one-time conversion I'd go with
something simple (perhaps like ocaml-mustache), however see below
* check license name compliance (Fedora and Debian have their standard name
lists), and make sure the package actually has a license *file*
* don't need full roundtrip capability between opam and spec/debian, the tool
should serve three purposes only:
* generate initial templates
* show a smart diff
* lint the opam metadata / spec file
* shouldn't require to update the conversion tool just to support an obscure
corner-case that can be fixed more easily by hand in the spec file
* doesn't have to be perfect or support the full features of spec/debian/opam
files. lets start with the minimum requirements and add fields/features as we
need them
* its not a substitute for human review, the final output always has to be
tested and reviewed by a human
Implementation thoughts:
* ocaml module describing spec file an debian rules with a toplevel type t and
nested records, with type 'a field
* some form of conversion to/from a string -> string map
* a lexer that loads that map from the spec/debian files, and a formatter that
generates it (ocamllex + format, or just ocaml-re + printf?)
* please if we can lets avoid camlp4 for this, I'd suggest ppx_deriving+cconv
or just manually writing the parse/generate calls for the record fields
* validation should be optional
* fields should be easy to declare (name, of_string, to_string, validate,
smart_diff), initially all would be raw strings, more parsing/diffing can be
added as needed
* some way to fill the 't' type from opam metadata
* certain files may not need all this advanced conversion, and might be easier
to generate from a text template with substitutions (ocaml-mustache, or mpp?),
and just show a textual diff
(docdiff at character/word level is quite nice)
Automation for tracking a package through its lifecycle (separate tool):
Since packaging an OCaml application can involve packaging/tracking a large
number of dependant libraries some tool to track the status of each package
(lifecycle)
would be useful. This is not specific to OCaml, I'm surprised that upstream
distros don't have this already, but anyway using Fedora terminology I want a
single CLI tool that can:
* query yum repo for stable packages
* query pkgdb for existence of package in other releases
* query bodhi for pending/testing packages
* query koji for builds submitted/completed
* query for existence of scm (git) repo
* query bugzilla for package review and find out its state (submitted, under
review, approved, waiting for scm repo, scm repo created)
For Debian a similar tool that:
* query bugs.debian.org for RFP (request for packaging), ITP (intent to
package) and its status
* query NEW queue for uploaded source package
* query package tracker for buildd status
* this can also avoid duplicating work, i.e. finding out that someone else is
already packaging my package or its dependencies
* might also be useful for other packages
* can be done manually by visiting a bunch of webpages in turn and searching
on each, but there ought to be some CLI tools to help or at least some (web)API
* doesn't have to be necesarely written in Ocaml, if we can reuse existing
tools and have this just as a shell/python script the better
I wanted to go through the policy docs and write some details here on what is
missing from opam metadata, but this mail is long enough and it is late enough
already,
that I thought to start the discussion with what I have so far, and discuss the
details as we go.
Once we agree on the major design points of this tool I plan on starting with
opam2rpm, but if someone else wants to own it I'd be more than happy with that
as well.
Anil also suggested to look at:
On 03/19/2015 10:15 AM, Anil Madhavapeddy wrote:
> This annotated changeset about a recent update from Jon might be useful as
> well:
>
> https://github.com/jonludlam/ocaml4021-buildroot/commit/9ef9554c283ea1cda8f64742b73bb63807b791c7#commitcomment-10071274
>
Best regards,
--Edwin
_______________________________________________
opam-devel mailing list
[email protected]
http://lists.ocaml.org/listinfo/opam-devel
