Hello Here is the latest OCaml Weekly News, for the week of November 17 to 24, 2020.
Table of Contents ───────────────── OCaml User Survey 2020 Suggestions from the OCaml Survey result OCaml for ARM MacOS TechEmpower benchmark: httpaf + lwt + unix on par with Haskell's warp Dropbox v2 API, and JSoO bindings for GridJS and IndexedDB OCaml in Jupyter-based Environments Ocurrent/opam Docker images have moved to ocaml/opam ocaml-lsp-server 1.2.0 ca-certs and ca-certs-nss Compatibility packages for 4.12 (either, semaphore-compat) Jupyter with pyml OCaml needs an arbitrary-precision decimal type Type-at-point ocaml-lsp/merlin in vim/neovim ocamlearlybird now an OCaml Software Foundation supported project release of mc2 0.1, a SMT solver Wednesday, 25th November 2020 is MirageOS Bug Cleaning Day! Other OCaml News Old CWN OCaml User Survey 2020 ══════════════════════ Archive: <https://discuss.ocaml.org/t/ann-ocaml-user-survey-2020/6624/25> Continuing this thread, Xavier Leroy announced ────────────────────────────────────────────── I extracted the replies to the free-form questions and made them available as a Gist. There are so many replies that I have no idea how to exploit and summarize them! <https://gist.github.com/xavierleroy/88a22b2b8e609c46bf9d23f36521a483> Patrick Ferris then said ──────────────────────── Thanks for putting together this great resource! I think it's quite important to take the results from this survey and find actionable items (or highlight ongoing efforts) which aim to improve different problems for different users. As a result I have put together some graphs trying to categorise the answers based on proficiency (`beginner', `intermediate', `advanced' and `expert'). These groupings are quite subjective but I thought it might offer a slightly more nuanced look into what problems users had, where they come from, what they are doing and what commonalities they share. The [code is here] and the [HTML version of the notebook here] (I'm no data-scientist nor python developer :snake: if you see glaring mistakes do raise an issue!). Here are some possible inferences (*warning: opinions ahead*) that can be made from the data along with possible actionable steps (or links to ongoing projects): • [Everybody wants better documentation for user libraries]. I think people want more long-form style documentation. However, excellent work is currently taking place with [odoc] (see also [OCaml workshop video]) and perhaps these concerns can be wrapped into this ongoing work. • Everybody wants *multicore* but "experts" want *implicits* a little more – [multicore is coming along amazingly] and [implicits] are in an earlier but active state AFAICT. • [A wide variety of application domains] • less proficient OCaml users tend to be doing more web-related tasks or data processing. Advanced/expert users are implementing programming languages, building developer tools, systems and formal methods. This is reflected in other breakdowns for example in ["other fluent languages"] we see JS more popular with beginners and C with experts. But also track the progress of *Coq and friends* moving up as proficiency increases. This is quite interesting and I think it could point to an actionable item of working more on the web and formal methods components of the OCaml ecosystem i.e. working with Coq devs and JS devs to help them. • From a Javascript point-of-view, looking at the [implementations] section it's interesting to note across the four proficiency levels `js_of_ocaml' is more popular than `Reason' except for beginners. Both projects are actively worked on, but perhaps more `js_of_ocaml' documentation would be good with many tools already existing like [gen_js_api], Jane Street's [bonsai] etc. In my opinion, this survey motivates working on unifying the JS world of OCaml as (at least to me) right now it feels a little fragmented and hard for someone new to know where to go. • [Opam with the public repository] across all users is consistently the most used installation method – this justifies the large amount of work that tasks place on [opam] that I'm sure most are very grateful for. Tying this back to the previous point of "cross community" work, opam have [community meetings] which seem to be v. useful. • Pain points – the top two (although they interchange) is "lack of critical libraries" and "too hard to find and hire OCaml developers". Some [effort has been made] to find these libraries including in the survey. I'm not sure what else can be done. As for hiring, looking at the different [communication channels] there is quite a spread with different users using different channels. This is great, but may also cause some fragmentation and make the "hiring" process worse. These are just some thoughts from a relatively new OCaml community member, I would love to know what other actionable steps people think we can take from this data and hopefully we can produce a more concise and specific set of steps to put this survey to great use :)) [code is here] <https://github.com/patricoferris/ocaml-survey-analysis/blob/main/survey-analysis.ipynb> [HTML version of the notebook here] <https://patricoferris.github.io/ocaml-survey-analysis/> [Everybody wants better documentation for user libraries] <https://patricoferris.github.io/ocaml-survey-analysis/#State-of-the-Art-> [odoc] <https://github.com/ocaml/odoc> [OCaml workshop video] <https://www.youtube.com/watch?v=wVyZ-KveN-w> [multicore is coming along amazingly] <https://discuss.ocaml.org/t/the-status-of-modular-implicits/6680> [implicits] <https://discuss.ocaml.org/t/the-status-of-modular-implicits/6680> [A wide variety of application domains] <https://patricoferris.github.io/ocaml-survey-analysis/#Fluency-in-other-languages-&-Application-domains-> ["other fluent languages"] <https://patricoferris.github.io/ocaml-survey-analysis/#Fluency-in-other-languages-&-Application-domains-> [implementations] <https://patricoferris.github.io/ocaml-survey-analysis/#OCaml-Tools-> [gen_js_api] <https://github.com/LexiFi/gen_js_api> [bonsai] <https://github.com/janestreet/bonsai> [Opam with the public repository] <https://patricoferris.github.io/ocaml-survey-analysis/#OCaml-Tools-> [opam] <https://github.com/ocaml/opam/projects/1> [community meetings] <https://github.com/ocaml/opam/wiki/Community-dev-meetings> [effort has been made] <https://discuss.ocaml.org/t/what-libraries-are-missing/6543> [communication channels] <https://patricoferris.github.io/ocaml-survey-analysis/#Community-Interaction-> After many replies, Patrick Ferris said ─────────────────────────────────────── Thanks for the great suggestions so quickly! I'll also set aside some time to summarise everything once some more ideas are posted and there is general consensus. I believe a common theme among the comments in the survey is a lack of structure found in documentation, libraries, and critical build tools Completely agree – documentation is fundamental as it is felt by most I feel. The limitations of build-tools are perhaps more felt by more advanced or complicated workflows. curated Opam repos This sound really interesting – from experience I think quite a few beginners are unfamiliar with the ability of having multiple repos. I certainly was until I started cross-compiling to RISC-V :)) I think a good example curated repo would help drive this home more, do you know if one exists beyond the cross-compiling ones which aren't exactly what we're looking for? Use sourcegraph and other code search mechanisms to surface all the work in progress (WIP) libraries that people have put together on github/gitlab/etc This sounds good. I do wonder if more documentation/tutorials/awareness of [dune-release] and [opam-publish] might also help. I, for one, am sitting on a few libraries I should really publish but the process can be a little intimidating, so I still agree a search would be great but ultimately it would be good for libraries to make it upstream. Thanks for this great analysis Thanks for the kind words :)) Is GSoC still a thing? Not in a position to start making GSoC proposals but they are definitely still a thing! For example [two] [projects] I think are great, still do them. Even outside of GSoC perhaps a curated list of OCaml Community approved projects/internships that would benefit the whole community would be good that different businesses/organisations/foundations could use when making their own internships would be useful? At least everyone would be united on that front. [dune-release] <https://github.com/ocamllabs/dune-release> [opam-publish] <https://github.com/ocaml-opam/opam-publish> [two] <https://summerofcode.withgoogle.com/archive/2019/organizations/5422734538964992/> [projects] <https://www.lowrisc.org/docs/gsoc-2020-ideas/> gasche then said ──────────────── Note: I "split" the [excellent discussion] by @patrickoferris as a separate topic, as it was going in the (very useful) direction of discussing broadly the ecosystem, rather than specifically the survey result. I would encourage people to post here for specific details on the survey process and results, and create new topics for discussions inspired by the survey. Let me quote below the part of @patricoferris' post that would be most useful to anyone interested in processing the results: I think it's quite important to take the results from this survey and find actionable items (or highlight ongoing efforts) which aim to improve different problems for different users. As a result I have put together some graphs trying to categorise the answers based on proficiency (`beginner', `intermediate', `advanced' and `expert'). These groupings are quite subjective but I thought it might offer a slightly more nuanced look into what problems users had, where they come from, what they are doing and what commonalities they share. The [code is here] and the [HTML version of the notebook here] (I'm no data-scientist nor python developer :snake: if you see glaring mistakes do raise an issue!). The [new topic] has excellent discussion on Patrick's findings from the survey data. [excellent discussion] <https://discuss.ocaml.org/t/suggestions-from-the-ocaml-survey-result/> [code is here] <https://github.com/patricoferris/ocaml-survey-analysis/blob/main/survey-analysis.ipynb> [HTML version of the notebook here] <https://patricoferris.github.io/ocaml-survey-analysis/> [new topic] <https://discuss.ocaml.org/t/suggestions-from-the-ocaml-survey-result/> Suggestions from the OCaml Survey result ════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/suggestions-from-the-ocaml-survey-result/6791/14> Anil Madhavapeddy said in this new thread ───────────────────────────────────────── Thanks Patrick for the fantastic distillation of results. It's extremely useful to see our user responses segmented by their experience. (in particular, our self-identified expert userbase runs Coq a lot more than our self-identified newcomer codebase who tend to use JavaScript – we want to make sure we continue to help all of these segments!). At a high level, this survey has already influenced the Platform tool developers. Documentation has been identified as a priority item for next year, and so a couple of next steps are happening this week already among the various groups: • The regular odoc [developer meeting] this week will also feature the opam team as part of our [community dev meetings], and we are getting together to put the final plan together to put a docs.ocaml.org site together. The results of this will be on the dev wiki as usual, so anyone interested can track progress and suggest ideas (the video meetings are not exactly closed, but fully open participation isn't practical given the constraints of current technology – please ping the odoc maintainer @jonludlam directly if interested in attending) • the second part of a docs site is to ensure we have really reliable and solid workflows for opam-repo contributions (including bulk builds, health checks and having a more automated contribution process, ideally without having to run a CLI tool). The next opam dev meeting [later this week] will feature us planning a switch to a [cluster-based nextgen CI] for opam-respository. We've also invited the maintainers of the Coq opam repository as well as Tezos and Jane Street (who contribute large package sets regularly) so we can ensure we work well with those ecosystems as well. Our intention here is to really reduce the burden on contributions to opam repository by mechanising as much of the grunt work as possible, thereby helping both beginners and expert users. Our new CI will also feature macOS and Windows testing as we bring those cluster workers online, and be much more easily extensible to custom workflows. • Having all the fancy package cluster builds in the world don't help if noone is actually writing any documentation in their libraries. We're hoping that new tools (such as [mdx] from Real World OCaml) will reduce the friction of entry to writing ocamldoc tutorials and sites. The mdx tool usage is easy but the implementation is quite complex (due to the interlock with the internal compiler-libs), so there is a [mdx team] working away on it, including hopefully speeding it up with native code compilation and continuing to improve the integration with dune and other build tools. @yminsky and I use mdx to write the whole of Real World OCaml (v2 of which is coming out soon in print), and we are most eager for other people to fork our tools and write their own books (like the [Owl scientific computing book]). • Finally, @ashish @gemmag and I have been putting our heads together to get funding sorted to reboot the ocaml.org site and make it easier to maintain, in recognition of the fact that the "old guard" (Christophe, Ashish, Phillippe, myself) just don't have the day-to-day time anymore to keep things up. @patricoferris, @kanishka @sanette Bella and @JohnWhitington have all been contributing content to ocaml.org, and we are doing both incremental changes and also overhauling the internals of how it is built to use the latest and greatest innovations. I'm excited to see what all these new contributors will come up with. This is of course not a closed list of action items – simply what I am tracking as the coordinator of the OCaml Platform efforts – so please keep suggestions and analysis flowing. I would suggest one good way to "wrap up" the survey is to: • transfer @patricoferris' notebook over to the `ocaml/' GitHub org. • once the discussions and next steps here settle (both from individual comments, and also the aggregated decisions of the various Platform teams like the opam, odoc and dune maintainers), some combination of @patricoferris @xavierleroy @gasche and myself summarise them in that notebook as priority items we would like to accomplish next year. • we publicise the survey results notebook and conclusions on ocaml.org and this forum prominently. • we figure out a way to get more people involved and contributing in concrete projects around documentation (in particular) next year, perhaps via schemes such as Outreachy or direct funding from organisations such as the OCSF. Thanks to everyone for the input and comments so far. If anyone has a burning desire to be in any of the dev meetings, please get in touch with me directly (a...@recoil.org) or the maintainers of the individual tool. I've never seen this much activity happening in all my time working on OCaml, so it warms my heart on this crisp winters day to see all the constructive positivity and effort going on. Keep it up and keep the suggestions coming :slight_smile: [developer meeting] <https://github.com/ocaml/odoc/wiki> [community dev meetings] <https://github.com/ocaml/opam/wiki/Community-dev-meetings> [later this week] <https://github.com/ocaml/opam/wiki/Community-dev-meetings#20112020-opam-repo-testing-and-handling-and-bulk-builds> [cluster-based nextgen CI] <https://www.youtube.com/watch?v=HjcCUZ9i-ug&list=PLKO_ZowsIOu5fHjRj0ua7_QWE_L789K_f&index=2> [mdx] <https://github.com/realworldocaml/mdx> [mdx team] <https://github.com/realworldocaml/mdx/wiki> [Owl scientific computing book] <https://ocaml.xyz/book/introduction.html> OCaml for ARM MacOS ═══════════════════ Archive: <https://discuss.ocaml.org/t/ocaml-for-arm-macos/6019/20> Continuing this thread, John Whitington said ──────────────────────────────────────────── OCaml 4.12 from trunk builds fine on ARM macOS for me, and I can build all my software, including mixed C/OCaml DLLs and other exotic things. For those interested in timings, "make -j9 world.opt" for OCaml runs in 1m6s. TechEmpower benchmark: httpaf + lwt + unix on par with Haskell's warp ═════════════════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/techempower-benchmark-httpaf-lwt-unix-on-par-with-haskells-warp/6788/1> blandinw announced ────────────────── The recent talks around this benchmark (started by @rbjorklin) and the OCaml results piqued my curiosity. After all, one of OCaml's strengths is its performance! It should smoke node.js and be in the same ballpark as Haskell, Go, etc., right? Well, only one way to find out! To start simple, I wanted to establish a baseline using a pre-forking approach with a listening socket shared by all children and go from there. It turns out I didn't need to go far at all. This simple architecture was enough to leave node.js in the dust and get results similar to Haskell and Go. This says a lot about the current quality of OCaml and the ecosystem we have today! Handshakes all around! 🍺🐫 You can find the results for the [JSON benchmark here]. Be sure to check the Latency tab. Note that a lot of the top performers optimize aggressively by precomputing responses, using object pools, etc. From my limited testing, the big difference with the previous OCaml attempts might be that they had to use (the otherwise amazing) `haproxy' to load-balance between all cores. Pre-forking and sharing a socket removes that need, so that all cores can do useful work. I also had some fun using `SIGALRM' to render the date only once per second. As a side note, it was my first time using these UNIX APIs from OCaml and it was an eye-opening experience to be able to leverage all that power outside of C and without giving up any of OCaml's strengths. I'm happy with the results, but it should be possible to improve even further by: • profiling with `perf' to know where time is spent, e.g. in JSON encoding, allocations, GC, context switches, etc. • using `libuv' instead of `libev', maybe via [this PR to Lwt] • using [Multicore domains] to use a pre-threaded architecture as opposed to a pre-forked architecture and hopefully reduce context switch costs, see [Linux Applications Performance] Contributing is pretty easy, just clone [this repo] and run `./tfb --test httpaf --type json --concurrency-levels 512'. [JSON benchmark here] <https://www.techempower.com/benchmarks/#section=test&runid=032630e0-3a86-4eac-ae2d-517e8b9586ac&hw=ph&test=json&a=2> [this PR to Lwt] <https://github.com/ocsigen/lwt/pull/811> [Multicore domains] <https://www.youtube.com/watch?v=Z7YZR1q8wzI> [Linux Applications Performance] <https://unixism.net/2019/04/linux-applications-performance-part-vi-polling-servers/> [this repo] <https://github.com/TechEmpower/FrameworkBenchmarks> Dropbox v2 API, and JSoO bindings for GridJS and IndexedDB ══════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-dropbox-v2-api-and-jsoo-bindings-for-gridjs-and-indexeddb/6793/1> Xavier R. Guérin announced ────────────────────────── I am releasing on GitHub today a set of libraries under ISC: 1. `xguerin/ocaml-dropbox': Dropbox v2 API. `auth', `check', and `files' operations are available, the rest is WIP. 2. `xguerin/ocaml-dropbox': JSoO bindings for [gridjs.io]. 3. `xguerin/ocaml-indexeddb': JSoO bindings for `IndexedDB'. Based on Thomas Leonard's work with `Irmin'. Each of them is OPAM-enabled although not currently available in the official repo. [gridjs.io] <https://gridjs.io> OCaml in Jupyter-based Environments ═══════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ocaml-in-jupyter-based-environments/6797/1> Patrick Ferris announced ──────────────────────── As @CraigFe said, this conversation might be useful for other people looking to run OCaml for teaching or data-science in a Jupyter-based environments. Continuing the discussion from [Suggestions from the OCaml Survey result] where [binder] was mentioned, this looks like a neat way to lower the barrier to entry for programming in OCaml. To add to what @mseri asked: Do you know if and how one can use ocaml on colab? In principle there is support for the jupyter protocol… You can but it is a bit hacky. You first need to generate a blank `ocaml.ipynb' locally and upload this to colabs, this will enable you to select `Runtime > Change Runtime Type > Runtime Selection' to OCaml. Then you need to install OCaml, jupyter and add the kernel. This code block did the trick for me: ┌──── │ !add-apt-repository ppa:avsm/ppa && apt-get update && apt-get install opam m4 libgmp-dev │ !opam init --disable-sandboxing -y │ !opam install jupyter │ !opam exec -- ocaml-jupyter-opam-genspec │ !jupyter kernelspec install --user --name ocaml-jupyter "$(opam var share)/jupyter" └──── It takes quite a while and I'm not really going to recommend it, but it does work. I also put together a few [dockerfiles]() with the purpose of being used in jupyter environments for teaching. Also with an example of using [nbgrader] although I think this still needs some work (it's based on @kayceesrk's [blog post]). Are people using other setups for teaching OCaml in a classroom/university setting? As a small aside, I started doing the survey analysis in OCaml, but found two issues: 1. CSV loading with OWL was struggling (I think it struggled with the extra `,' in questions and also didn't like multiple headers with the same name – other libraries I think just add the column number to make it unique), I ended up using the [csv] library and then making the dataframe by hand. 2. The plotting library is good, but lacks the customisability that something like Matplotlib offers. I did start using the very good [ocaml-matplotlib bindings] but it all became a little too much effort… :snake: [Suggestions from the OCaml Survey result] <https://discuss.ocaml.org/t/suggestions-from-the-ocaml-survey-result/6791/13> [binder] <https://mybinder.org/> [nbgrader] <https://github.com/patricoferris/ocaml-teaching/tree/main/teaching> [blog post] <https://kcsrk.info/ocaml/prolog/jupyter/notebooks/2020/01/19/OCaml-Prolog-Jupyter/> [csv] <https://github.com/Chris00/ocaml-csv> [ocaml-matplotlib bindings] <https://github.com/LaurentMazare/ocaml-matplotlib/> Anton Kochkov suggested ─────────────────────── Another project that is probably a better inspiration for OCaml-based workflow - [Pluto.jl] that is written in Julia itself. See [Pluto presentation (20 min) at *Juliacon 2020*]. [Pluto.jl] <https://github.com/fonsp/Pluto.jl> [Pluto presentation (20 min) at *Juliacon 2020*] <https://www.youtube.com/watch?v=IAF8DjrQSSk> Anil Madhavapeddy replied to Patrick Ferris ─────────────────────────────────────────── Are people using other setups for teaching OCaml in a classroom/university setting? We use Jupyter and nbgrader in Cambridge for the first year Foundations of Programming course: <https://www.cl.cam.ac.uk/teaching/1920/FoundsCS/materials.html> (the pdf is generated from the Jupyter export, and the content is written in Markdown and converted to notebook format using a [modified mdx]). It's on my todo list to resurrect <https://github.com/andrewray/iocamljs> so that we can do this with a pure-client-side experience. We have internal hosted servers so that each student has their own container running their Jupyter/OCaml kernel, but I think it would be more robust to switch entirely client side for everything except exercise grading. [modified mdx] <https://github.com/realworldocaml/mdx/pull/124> Philippe asked and Anil Madhavapeddy replied ──────────────────────────────────────────── This tool shows the interpreter outputs but can also render graphics in the page. Do you have plans to include a similar feature in mdx? (I’m asking because if this is the case, I’d be happy to lend a hand). Doesn't support it, and that sounds awesome. @jonludlam did something involving registering OCaml toplevel printers that can output HTML that renders in a notebook (so you can pipe a `type tree = ...' through dot to render it graphically). Never upstreamed that work I think. Feel free to create an issue on the mdx repo – I think the utility of it will increase dramatically if it can create notebooks! I have to say though, writing ocaml code in a markdown document is not very convenient since you don’t have merlin+syntax highlighting helping you. Ideally we’d need an environment that knows both markdown and ocaml very well… Yes indeed, mdx supports this mode too. In Real World OCaml, we have all our examples as separate files that can have `dune build' run on them, and then `mdx' supports external references as well. See for example the [JSON chapter in RWO]: you just add a `file=' block into the Markdown, and the dune promotion rules will either run the toplevel or include the external ML file and update the Markdown content automatically. [JSON chapter in RWO] <https://github.com/realworldocaml/book/blob/master/book/json/README.md> Jon Ludlam said ─────────────── Registering 'rich' printers with Jupyter is a standard feature of the wonderful [ocaml-jupyter] (thanks, @akabe!) - you just register a printer as normal that ends up calling [Jupyter_notebook.display]. I have a couple of examples [here] for displaying trees – the one you mentioned – and ppm files. It would be very neat to be able to do something similar for mdx, and probably not too tricky too. Perhaps a good starter project? [ocaml-jupyter] <https://github.com/akabe/ocaml-jupyter> [Jupyter_notebook.display] <https://akabe.github.io/ocaml-jupyter/api/jupyter/Jupyter_notebook/index.html#val-display> [here] <https://github.com/jonludlam/focs-support/tree/master/lib> Ocurrent/opam Docker images have moved to ocaml/opam ════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ocurrent-opam-docker-images-have-moved-to-ocaml-opam/6798/1> Thomas Leonard announced ──────────────────────── The [docker base image builder] has been reconfigured to push the images to the `ocaml/opam' repository on Docker Hub. If you were previously using the `ocurrent/opam' repository, you should update to the new location. The `ocurrent/opam' repository will not get any further updates. The images provide many combinations of distribution, OCaml version, architecture and flags. e.g. to get an environment with OCaml 4.11 installed on Debian 10: ┌──── │ docker run --rm -it ocaml/opam:debian-10-ocaml-4.11 └──── Or to try the 4.12 pre-release, compiled with AFL fuzzing support, you'd use `ocaml/opam:debian-10-ocaml-4.12-afl'. (the full set of builds can be seen at the service's page at <https://base-images.ocamllabs.io/>) [docker base image builder] <https://github.com/ocurrent/docker-base-images> Christian Lindig then said ────────────────────────── I recently looked at `ocaml/opam:debian-10-ocaml-4.09' and noticed that in the root of the image the Dockerfiles were included that generated the image. I'm not sure that was always the case but found that helpful. ocaml-lsp-server 1.2.0 ══════════════════════ Archive: <https://discuss.ocaml.org/t/ann-ocaml-lsp-server-1-2-0/6799/1> Rudi Grinberg announced ─────────────────────── On behalf of the ocaml-lsp team, I'd like to announce version 1.2.0. This version contains many bug fixes and some performance improvements A couple of interesting features made it in as well: • Auto-completion of OCaml keywords (not available for reaso) • The ability to jump to the declaration of a value in the .mli. Yawar Amin added ──────────────── Release page: <https://github.com/ocaml/ocaml-lsp/releases/tag/1.2.0> ca-certs and ca-certs-nss ═════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-ca-certs-and-ca-certs-nss/6804/1> Hannes Mehnert announced ──────────────────────── I'm pleased to announce the release two opam packages, [ca-certs] and [ca-certs-nss], which use the trust anchors from the system / embed trust anchors from [Netscape Security Services] (these days used by the Mozilla Firefox browser). For some context: when establishing a TLS connection you likely want to verify that the server provides a valid certificate – on the open world wide web this boils down to "is valid at the current time" and "is signed by a trustworthy entity" (such as <https://letsencrypt.org/> - which validates that you have access to the domain before signing a certificate). If you do not verify the server certificate, a person may be in the middle of the connection and read and modify arbitrary communication content. Read more about [this topic on Wikipedia]. NB in private setups you can use your own CA setup and won't need ca-certs / ca-certs-nss. Now, different operating systems store this information in different places and formats – for Unix (and Linux) there is unfortunately no common API or file location. To abstract over this, the package ca-certs provides the API `val authenticator : [...] -> unit -> (X509.Authenticator.t, [> `Msg of string ]) result' – which composes well with OCaml-TLS API for TLS clients (that receive such an authenticator). The ca-certs package has initially been implemented by @emillon in 2019, and only recently been pushed to opam-repository. If you're using a not-so-mainstream Linux distribution (or other Unix), we're interested in your feedback: does a `dune runtest' work on your system? – it has been tested apart from debian, ubuntu, SuSE, CentOS, also on FreeBSD, OpenBSD, and macOS. The macOS support uses the `security' command, and could be improved by using appropriate API calls – there is no support for Windows at the moment (if you're interested in contributing support for windows, [it should be pretty straightfoward]). The ca-certs-nss package uses the same versioning scheme as NSS, and embeds the trust anchors that your Firefox browser has as well. This is meant as alternative to ca-certs (e.g. if you're on a system which is not (yet) supported by ca-certs), or in a MirageOS unikernel (where there's no access to the host trust anchors). We're interested in your feedback, and hope by releasing those libraries to improve the security of network clients across the OCaml ecosystem by providing a simple API to authenticate server certificates. If you're running into issues, please don't hesitate to reach out. To install, `opam install ca-certs' / `opam install ca-certs-nss' is all you need. [ca-certs] <https://github.com/mirage/ca-certs> [ca-certs-nss] <https://github.com/mirage/ca-certs-nss> [Netscape Security Services] <https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS> [this topic on Wikipedia] <https://en.wikipedia.org/wiki/Certificate_authority> [it should be pretty straightfoward] <https://github.com/mirage/ca-certs/issues/4> Compatibility packages for 4.12 (either, semaphore-compat) ══════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-compatibility-packages-for-4-12-either-semaphore-compat/6806/1> Craig Ferguson announced ──────────────────────── I'm pleased to announce the release of two small compatibility libraries for modules added in OCaml 4.12.0: • [`either'] – compatibility shims for the [`Either'] module, which provides a canonical sum type in the standard library + various utility functions for it. To use it: simply add `either' as a library dependency and refer to the `Either' module, which will either alias the `Either' provided by the standard library or re-implement it as appropriate. • [`semaphore-compat'] – compatibility shims for the [`Semaphore'] module in `systhreads', which provides binary and counting semaphores for use in multi-threaded programs. To use it: add `semaphore-compat' as a library dependency and `open Semaphore_compat' in any module that requires semaphores. *_Note on OCaml concurrency primitives_*. Users of OCaml's `Mutex' module should beware that OCaml 4.12 features a changes to mutex semantics on certain platforms, since the mutexes are now "error checking" (as noted in the [changelog]). One consequence of this is that **unlocking a mutex from a non-owning thread will now always fail**, whereas previously this might succeed depending on the platform's mutex implementation – notably, `glibc''s mutexes allow this. If your code relies on this property of pre-4.12 mutexes, you may wish to add a dependency on `semaphore-compat' and switch to using binary semaphores instead (as these provide the right flavour of concurrency primitive in a POSIX-compatible manner). Opam library authors making use of these compatibility libraries in their public API are encouraged to [conditionally depend] on them in order to ensure that downstream users of your library don't pull in unnecessary compatibility shims. This can be done as follows: ┌──── │ depends: [ │ ("ocaml" {>= "4.12.0"} | "either") │ ] └──── I hope you find these libraries useful! [`either'] <https://github.com/mirage/either> [`Either'] <https://github.com/ocaml/ocaml/blob/trunk/stdlib/either.mli> [`semaphore-compat'] <https://github.com/mirage/semaphore-compat> [`Semaphore'] <https://github.com/ocaml/ocaml/blob/trunk/otherlibs/systhreads/semaphore.mli> [changelog] <https://github.com/ocaml/ocaml/blob/trunk/Changes#L329> [conditionally depend] <https://opam.ocaml.org/doc/Manual.html#Package-Formulas> Hannes Mehnert then asked ───────────────────────── Thanks for your work on this. are encouraged to [conditionally depend] While I understand at the opam level how this could work, what about `_tags' or `dune' – there's need to repeat that condition (since in OCaml < 4.12 you'll need `(libraries either)' in your dune file / `*: package(either)' in your `_tags' file (+in your `META' file) – or is there something obvious I misunderstand? I'm asking since I'd be interested in a general strategy / howto of such compatibility libraries. I'm also curious why there are different approaches in respect to package name and strategy: • stdlib-shims (seems to (attempt to) cover all stdlib changes?) • bigarray-compat (there's a `-compat' suffix) • seq (there's a `seq.base' for ocaml >= 4.07.0) Are there plans to unify the approaches? I'm in favour of a "conditional dependency if the OCaml distribution does not yet provide the module" and "no dependency if the OCaml distribution provides that module" – but as expressed above, I think this is needed both at the opam and the ocamlfind / library level. [conditionally depend] <https://opam.ocaml.org/doc/Manual.html#Package-Formulas> Craig Ferguson replied ────────────────────── To begin, I'll fill out my reasons for recommending conditional dependencies somewhat. Compatibility libraries and unconditional dependencies ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ The issue with unconditionally depending on compatibility libraries is that it forces users to take on that implicit dependency – even if they don't need it – and ties that decision to the _current_ minimal version supported by one of their dependencies. This prevents use of `(implicit_transitive_deps false)' in those downstream libraries, since the authors cannot know whether they actually depend on the alias transitively. For instance: • `lwt' currently depends on `result' unconditionally (to retain support for `4.02.0'); • as an author of `irmin' – which goes back to `4.08.0' and so always has access to `Stdlib.Result' – I must be able to resolve the `Lwt_result.t' type, but I don't in general know which dependencies are necessary to do that since it's a property of `Lwt''s code and not mine. • I must either add an explicit dependency on `result' – because that's what's _currently_ necessary given `Lwt''s own dependencies – or give up on using `(implicit_transitive_deps false)' entirely. Either way, if I myself expose a `result' type in the API of Irmin, this problem cascades to users of Irmin too. For Lwt specifically, I need to find the time to upstream a patch to fix the issue: <https://github.com/ocsigen/lwt/issues/794>. Conditional dependencies and Dune ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ It's possible to conditionally depend on a library on the Dune side via either a [`(select ... from ...)'] dependency or by generating Dune files programatically (either w/ `(include dune.inc)' or [`Jbuild_plugin']). None of these solutions are particularly compelling to me, but I've been bitten by this problem enough to be willing to consider ugly solutions – at least for very widely-used libraries such as Alcotest. It's possible that conditional dependencies are the wrong approach altogether and we should be using something like [`(re_exports)'] instead. I'm not familiar with that feature, but it looks like it might work to avoid the above issues. Perhaps a Dune maintainer could weigh in here, and I'll happily remove my suggestion re. conditional dependencies for something better :slightly_smiling_face: [`(select ... from ...)'] <https://dune.readthedocs.io/en/stable/concepts.html#alternative-dependencies> [`Jbuild_plugin'] <https://dune.readthedocs.io/en/stable/advanced-topics.html#ocaml-syntax> [`(re_exports)'] <https://github.com/ocaml/dune/pull/2605> Different approaches to compatibility libraries ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ I’m asking since I’d be interested in a general strategy / howto of such compatibility libraries. I’m also curious why there are different approaches in respect to package name and strategy: • stdlib-shims (seems to (attempt to) cover all stdlib changes?) • bigarray-compat (there’s a `-compat' suffix) • seq (there’s a `seq.base' for ocaml >= 4.07.0) Indeed, there are quite a few different mechanisms used for this. Even just for this release, `either' and `semaphore-compat' use different techniques! Partly this is because it's easier to shim modules in `Stdlib' since it is the default-opened library. In particular, `Either' can exist without a namespace collision, and the _implementation_ of `either' can refer to `Stdlib.Either' to avoid Dune getting confused about a self-referential module. This is the approach that was taken with the `result' compatability package too. Unfortunately, the above factors don't apply to `Semaphore', which is why I grudgingly settled on `semaphore-compat' instead. I'm not sure why the `Bigarray' compat module didn't take the more "direct" approach, but I was happy to be consistent with it. One final point: the `stdlib-shims' library exists _solely_ to provide the alias `Stdlib' ↦ `Pervasives' in order to compensate for the name change. It doesn't ever provide re-implementations of the modules and functions that were added since (and so really is a `-shim' and not a `-compat'). A user of `stdlib-shims' is still restricted to the set of stdlib functions provided by their minimal supported OCaml version. Jupyter with pyml ═════════════════ Archive: <https://discuss.ocaml.org/t/jupyter-with-pyml/6813/1> n4323 asked ─────────── Hi, I would like to achieve something similar to, but different from what is discussed in <https://discuss.ocaml.org/t/ocaml-in-jupyter-based-environments/6797> . tl; dr: How can I get python REPL access to OCaml values exported via `pyml'? The use case is this: I have an OCaml library that does machine-learning type computation. I would like to generate, plot and analyze the results interactively, with the best possible ease of use and flexibility. For the plotting and analysis I want to use Python. I have tried `ocaml-jupyter', which provides a nice notebook environment but no python kernel and the very useful `ocaml-matplotlib' which offers a matplotlib binding but cannot cover all my python library needs. A possible solution would be to use `pyml' and its numpy array memmapping support to export OCaml values to a python process. The problem here is that I can't find a way to interact with the python process spawned from `pyml' in an interactive python console. I am thinking there should be some way to spawn a Jupyter python kernel from `pyml' and then access that from a separate jupyter console. I have not been able to find one by googling though. Another solution might be to wrap my OCaml library as an external python module. I'm afraid that this may be painful – please convince me otherwise ;) Craig Ferguson ────────────── You may be interested in this post on the Jane Street tech blog: [*Using Python and OCaml in the same Jupyter notebook*], by Laurent Mazare (@laurent). which offers some general tips and some bespoke tooling (showcased in <https://colab.research.google.com/drive/1MDuZc0v60lzBg0_xeiPR-bB6dCIWxvzT>). I have found this very useful in the past when trying to do Python + OCaml interop in Jupyter notebooks. [*Using Python and OCaml in the same Jupyter notebook*] <https://blog.janestreet.com/using-python-and-ocaml-in-the-same-jupyter-notebook/> OCaml needs an arbitrary-precision decimal type ═══════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ocaml-needs-an-arbitrary-precision-decimal-type/6735/6> Yaron Minsky announced ────────────────────── Our Bigdecimal library is now live: <https://github.com/janestreet/bigdecimal> It won't be in the ordinary Opam repo until our next stable release, which is scheduled for early 2021. Type-at-point ocaml-lsp/merlin in vim/neovim ════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/type-at-point-ocaml-lsp-merlin-in-vim-neovim/6832/1> Nicolas Gimenez discussed ───────────────────────── I'd like to know whether ocaml-lsp supports type-at-point feature? The point is to display to the user the type at point inferred by the compiler. I think this functionality already exists in Merlin for a while: [merlin-show-type-at-point], and I've read [articles] talking about people's corresponding lsp-mode/merlin Emacs config. How about VIM? Is it integrated in ocaml-lsp already? From looking on GitHub, it seems it's not, but then how come people manage to integrate it in Emacs' lsp-mode? [merlin-show-type-at-point] <https://github.com/ocaml/merlin/issues/6> [articles] <https://khady.info/emacs-ocaml-lsp.html> mudrz replied ───────────── I am using `neovim' and the following config: in `.vimrc' ┌──── │ Plug 'sheerun/vim-polyglot' │ Plug 'neoclide/coc.nvim', {'branch': 'release'} │ │ nnoremap <silent> gh :call <SID>show_documentation()<CR> │ │ function! s:show_documentation() │ if (index(['vim','help'], &filetype) >= 0) │ execute 'h '.expand('<cword>') │ else │ call CocAction('doHover') │ endif │ endfunction └──── and in `:CocConfig': ┌──── │ "languageserver":{ │ "ocaml": { │ "command": "ocamllsp", │ "rootPatterns": ["dune-project"], │ "filetypes": ["ocaml"], │ "initializationOptions": {}, │ "settings": {} │ } │ } └──── which allows to show a type tooltip like this when you press `gh` <https://aws1.discourse-cdn.com/standard11/uploads/ocaml/optimized/2X/9/943c5e3dbeb3f4d92bfc905a57d9ef2cfc339317_2_1380x150.png> Raphaël Proust also replied ─────────────────────────── In my `.config/nvim/init.vim' I have the default things setup by opam and then (amongst other things): ┌──── │ autocmd FileType ocaml nnoremap <LocalLeader>t :MerlinTypeOf<CR> └──── ocamlearlybird now an OCaml Software Foundation supported project ═════════════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-ocamlearlybird-now-an-ocaml-software-foundation-supported-project/6834/1> 文宇祥 announced ──────────────── I’m happy to announce ocamlearlybird now an OCaml Software Foundation supported project. We will continue to develop/maintain ocamlearlybird, and may contribute to ocamlrun/ocaml to improve/achieve OCaml bytecode/native debugging support. Please expect that OCaml debugging experience will be as good as Javascript in the future. Louis Roché then said ───────────────────── I suppose that you are referring to <https://github.com/hackwaly/ocamlearlybird> That's a great news. Thanks for your work. release of mc2 0.1, a SMT solver ════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-release-of-mc2-0-1-a-smt-solver/6835/1> Simon Cruanes announced ─────────────────────── I'm happy to announce the initial release 0.1 of [mc2], a [SMT solver] in pure OCaml based on the mcSAT framework. It can handle the QF_UFLRA fragment (functions and linear rational arithmetic). It is more of a research project than a production ready solver, but has pretty acceptable performance (see for examples benchmarks on [QF_UFLRA] and [QF_LRA] with timeout=10s). The solver is Apache licensed and weights around 7kloc (+ a few basic dependencies). Questions or comments welcome :slight_smile: [mc2] <https://github.com/c-cube/mc2> [SMT solver] <https://en.wikipedia.org/wiki/Satisfiability_modulo_theories> [QF_UFLRA] <https://benchpress.cedeela.fr/show/res-20201117T131950-cfa5920b-5187-4d73-8f5b-00b5c77aaeb8.sqlite> [QF_LRA] <https://benchpress.cedeela.fr/show/res-20201116T203511-687d557c-5c27-4478-9cc2-8b2603820ab6.sqlite> Wednesday, 25th November 2020 is MirageOS Bug Cleaning Day! ═══════════════════════════════════════════════════════════ Archive: <https://discuss.ocaml.org/t/ann-wednesday-25th-november-2020-is-mirageos-bug-cleaning-day/6836/1> Hannes Mehnert announced ──────────────────────── We have many repositories that have lots of old and no-longer-relevant issues in them (some have even been fixed!) as well as issues that haven't gotten a reply yet from a maintainer. In preparation for the next release, I think it would be nice to take a day and do some housecleaning. Our last bug cleaning day was on 17th November 2017, and pretty successful. On Wednesday, 25th November (in two days) most people of the mirage-core team will be going through old issues and coordinating our efforts on the #mirage channel over on irc.freenode.net. I expect there to be the most activity during 10:00-18:00 UTC and maybe a bit later, but don't feel limited to that timeslot – if you're familiar with a repository and have a bit of time, we'd love your help any time at all. Please do join us if you're free! We'll as well be watching the OCaml discord #mirage channel <https://discord.com/channels/436568060288172042/519199441769594911> If you're not sure where to start, here's a link to a GitHub search for all issues in repositories owned by the Mirage organization which are open and not archived, sorted with the least recently updated first, for your editing and browsing pleasure: <https://github.com/issues?q=is%3Aopen+is%3Aissue+org%3Amirage+archived%3Afalse+sort%3Aupdated-asc> If you've a specific issue at your heart that you want to be solved, please show up and tell us about it. Other OCaml News ════════════════ >From the ocamlcore planet blog ────────────────────────────── Here are links from many OCaml blogs aggregated at [OCaml Planet]. • [Editor Support, Custom Operators and More] • [Coq 8.12.1 is out] [OCaml Planet] <http://ocaml.org/community/planet/> [Editor Support, Custom Operators and More] <https://rescript-lang.org/blog/editor-support-custom-operators-and-more> [Coq 8.12.1 is out] <https://coq.inria.fr/news/coq-8-12-1-is-out.html> Old CWN ═══════ If you happen to miss a CWN, you can [send me a message] and I'll mail it to you, or go take a look at [the archive] or the [RSS feed of the archives]. If you also wish to receive it every week by mail, you may subscribe [online]. [Alan Schmitt] [send me a message] <mailto:alan.schm...@polytechnique.org> [the archive] <http://alan.petitepomme.net/cwn/> [RSS feed of the archives] <http://alan.petitepomme.net/cwn/cwn.rss> [online] <http://lists.idyll.org/listinfo/caml-news-weekly/> [Alan Schmitt] <http://alan.petitepomme.net/>
_______________________________________________ caml-news-weekly mailing list caml-news-weekly@lists.idyll.org http://lists.idyll.org/listinfo/caml-news-weekly
