Hello, Here is the latest Caml Weekly News, for the week of January 29 to February 05, 2013.
1) OCaml-Java: new preview 2) Core Suite 109.07.00 released 3) ocurl forked and 0.5.4 released 4) multiplexing several threads 5) Brand-new BER MetaOCaml for OCaml 4.00.1 6) Update on docs.camlcity.org 7) Other Caml News ======================================================================== 1) OCaml-Java: new preview Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56659> ------------------------------------------------------------------------ ** Xavier Clerc announced: In order to support some of the claims made during the OUPS meetup, I made a new binary preview of the OCaml-Java project, available at the following address: <http://ocamljava.x9c.fr/preview/> The next versions will also be made available on the very same page, without notification to the OCaml mailing list (at least until the distribution is binary-only). I am still eagerly looking for testers... and bug reports. ** Rudi Grinberg asked and Pierre Chambart suggested: > It would be really sweet to be able to install ocamljava through opam. > Something along the lines of opam switch ocamljava. You can try opam remote add <https://github.com/chambart/opam-compilers-repository.git> opam switch ocamljava-preview But do not expect any package to work. ======================================================================== 2) Core Suite 109.07.00 released Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56662> ------------------------------------------------------------------------ ** Jeremie Dimino announced: I'm pleased to announce the 109.07.00 release of the Core suite of libraries. Core is an industrial strength alternative to OCaml's standard library. Tarballs can be found here: <https://ocaml.janestreet.com/ocaml-core/109.07.00/individual/> And the documentation: <https://ocaml.janestreet.com/ocaml-core/109.07.00/doc/> All packages are available through opam. ** The Core suite ** The Core suite includes a variety of useful libraries, including: - Core: the heart of the standard library. - Several useful syntax extensions - type-conv: a library for building type-driven syntax extensions - sexplib: a library for handling s-expressions, and a syntax extension for auto-generating conversions between OCaml types and s-expressions - bin-prot: a syntax-extensions for generating - pipebang - variantslib - comparelib - fieldslib - Async: a monadic concurrency library. - Core_extended: extra components that are not as closely vetted or as stable as Core. This includes, Shell, an interface for interacting with the UNIX shell, and Command, a command-line parsing library. ** Repositories ** The official repositories for the Core libraries are now located on the "janestreet" organisation on github and the "janestreet" team on bitbucket. github: <https://github.com/janestreet> bitbucket: <https://bitbucket.org/janestreet> github home page: <http://janestreet.github.com/> We changed the way we are exporting our source tree; there will now be only one visible commit per release and project. Hopefully this simpiflied process will allow more frequent updates. ** Contribution ** If you want to contribute to core the preferred way is to submit a pull-request, which won't be merged in the end but will be used for working on the changes before they are accepted and added to our development process for integration in a future release. ** Changes ** He is a list of changes since the previous public release (108.08.00): - Switched to OCaml 4.0. - Async: - Add a [~perm] argument to [Writer.open_file] to set the file permissions in the same way [Unix.openfile] does. - Added [Async.Unix.unsetenv]. - Fixed a bug in [Reader] that in some situations would make the reader unusable after an error in user code (e.g. a failed sexp conversion). - Fixed a bug in [Writer] that manifests when scheduling bigstrings with non-zero pos parameter. - Fixed a bug in [Scheduler.go], which previously behaved incorrectly if an exception had been raised to the main monitor prior to [Scheduler.go] being called. The exception is now dealt with immediately, rather than running a cycle. - Exposed the type and value [Async.Config.t] as sexpable. - Improved error message when a user requests async to manage a file descriptor that it is already managing. - Improved error message when creation of the async scheduler fails. - Added [val _squelch_unused_module_warning_] to [Async.Std]. - Made [Reader.load_sexp{,s}] handle exceptions other than [Of_sexp_error]. - Fixed a bug in async's handing of file descriptors -- a missed check for a file descriptor having been closed. - Added [Writer.set_buffer_age_limit]. - Improved the performance of [Deferred.Queue] by changing the implementation to use lists rather than arrays, which reduces gc promotion. - Added [?should_close_file_descriptor:bool] argument to [Async.Unix.close]. - Moved the implementation of finalizers from Async_unix to Async_core. This makes it possible to unit test finalizers (and things using them) using only the Async_core scheduler. - Changed the async scheduler so that if there are no upcoming events, it times out in 50ms rather than waiting forever. - Added =Monad_sequence.iteri=, which in turn adds it to: =Deferred.Array=, =Deferred.List=, and =Deferred.Queue=. - Added =Pipe.init=, which is analogous to =Deferred.create=. - Improved the performance of =Pipe.iter{,_without_pushback}=. - Improved =Reader.read_one_chunk_at_a_time_until_eof=: - the callback need not consume everything - add =`Eof_with_unconsumed_data= as a possible result - grow internal buffer of the reader when needed - Added =Shutdown.exit=, removed =Shutdown.shutdown_and_raise=. - Added =Scheduler.force_current_cycle_to_end=. - Core: - Added [Char.of_string]. - Fix [Backtrace.get], which was broken in 109.00, with the switch to OCaml 4.0. - Added [Heap.iter_el]. - Updated [Core.Unix.stat] so that access, modify, and change times have nanosecond precision. - Fixed a bug in [Nano_mutex.invariant]. - Simplified the implementation of [with_return] using a local explicit polymorphic type variable. - Added [Map.symmetric_diff], for returning a list of differences between two maps. It has a fast-path implementation for maps that share a large amount of their internal structure. - Added a number of functions to =Bounded_int_table=: =equal=, =exists{,i}=, =for_all{,i}=, =filter_map{,i}=, =map{,i}=. Also added a functor, =Bounded_int_table.With_key=, that makes a bounded-int table binable and sexpable, and adds =of_alist= and =of_alist_exn=. - Added =Doubly_linked.iter_elt= and =Bag.iter_elt=. - Added =module Invariant=, which defines signatures that are to be included in other signatures to ensure a consistent interface to invariant-style functions. - Added =module Ordering=, which defines: =type t = Less | Equal | Greater= ======================================================================== 3) ocurl forked and 0.5.4 released Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56663> ------------------------------------------------------------------------ ** ygrek announced: Let me announce the existence of the new ocurl project, namely at <http://ocurl.forge.ocamlcore.org/> This project is the fork of <http://ocurl.sourceforge.net/> and accumulates many bugfixes and enhancements which didn't make it to the original version, including the multi API wrappers and pluggable asynchronous interface (which makes it possible to inject curl transfers in your event loop of choice). This is the first release for the fork, that incorporates all the bugfixes and new features from 3 years of its history, except the asynchronous api. Next release will merge branch event. I would like to thank Lars Nilsson, the original author of ocurl, who brought this project to existence. Hope the forked version will prove useful. ======================================================================== 4) multiplexing several threads Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56664> ------------------------------------------------------------------------ ** Ivan Gotovchits asked and Mauricio Fernandez replied: > I'm stuck with a simple problem when using lwt. I have two lwt > threads. One on them reads commands from user (using Lwt_read_line), > another reads events from system. A need to multiplex them. Speaking in > ocaml, I have three functions: > > type state (* program state to process *) > type command (* command to be executed (issued either by user or by > system event) *) > > val read_from_user: unit -> command Lwt.t > val read_event: unit -> command Lwt.t > val process: state -> command -> state Lwt.t > > > And a need to write a function loop > > val loop: state -> state Lwt.t > > accepting some state, taking a command from user or an event from system > (whichever comes first) and passing it to `process' function, and doing > it in a cycle, until explicitly ordered to stop (with Exit command). > > The following solution > > let rec loop sc = > let cmd = read_from_user () <?> read_event () in > cmd >>= (process state) > > and process sc = function > | Exit -> return sc > | cmd -> (* modify sc*) loop sc > > let r = Lwt_main.run (loop state) > > doesn't work when the event thread wakes up first. In that case the next > step of the loop will call Lwt_read_line.read_line before the previous > invocation finishes and it breaks user input. > > Please, help me to find a feasible solution to my problem! let process s = function Exit -> return (`Exit s) | cmd -> let s = ... in return (`State s) let rec loop state user_cmd event = match_lwt Lwt.choose [ (lwt x = user_cmd in return (`User x)); (lwt x = event in return (`Event x)); ] with `User cmd -> begin match_lwt process state cmd with `Exit s -> Lwt.cancel event; return s | `State s -> loop s (read_from_user ()) event end | `Event cmd -> begin match_lwt process state cmd with `Exit s -> Lwt.cancel event; return s | `State s -> loop s user_cmd (read_event ()) end ======================================================================== 5) Brand-new BER MetaOCaml for OCaml 4.00.1 Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56665> ------------------------------------------------------------------------ ** oleg announced: BER MetaOCaml N100 is now available. It is a strict superset of OCaml 4.00.1, extending it with staging annotations to construct and run typed code values. BER MetaOCaml has been completely re-implemented and thus caught up with OCaml. For those who don't know what staging or MetaOCaml is, a short introduction follows the news summary. BER MetaOCaml is in the spirit of the original MetaOCaml by Taha and Calcagno, but has been completely re-written using different algorithms and techniques. BER MetaOCaml has been re-structured to minimize the number of the changes to the OCaml type-checker and to separate the `kernel' (type-checking and constructing code values) from the `user-level'. Various ways of running the code -- compiling to byte-code or machine instructions and executing them, or translating code values to C or LLVM, or printing them -- can be done in `user-level' libraries, without any need to hack into (Meta)OCaml. (Printing and byte-code execution are included in BER N100.) This release of BER MetaOCaml is meant to incite future research into type-safe meta-programming. To the user, the two major differences of BER N100 from the old MetaOCaml are: -- constructor restriction: all data constructors and record labels used within brackets must come from the types that are declared in separately compiled modules. -- scope extrusion check: attempting to build code values with unbound or mistakenly bound variables (which is possible with mutation or other effects) is caught early, raising an exception with good diagnostics. Both are explained after the short introduction to staging. Smaller visible differences are better printing of cross-stage persistent values and the full support for labeled arguments. A long-standing problem with records with polymorphic fields has fixed itself. The BER N100 code is now extensively commented, and has a regression test suite. BER N100 is much less invasive into OCaml: compare the size of the patch to the main OCaml type-checker typing/typecore.ml, which contains the bulk of the changes for type checking the staging constructs. In the previous version BER N004, the patch had 564 lines of additions, deletions and context; now, only 328 lines. (The core MetaOCaml kernel is trx.ml, with 1800 lines.) BER MetaOCaml N100 is available: -- as a set of patches to the OCaml 4.00.1 distribution. <http://okmij.org/ftp/ML/ber-metaocaml-100.tar.gz> See the INSTALL document in that archive. You need the source distribution of OCaml 4.00.1, see the following URL for details. <http://ocaml.org/install.html> -- as a GIT bundle containing the changes relative to OCaml 4.00.1 <http://okmij.org/ftp/ML/metaocaml.bundle> First, you have to obtain the base git clone <https://github.com/ocaml/ocaml.git> -b 4.00 ometa4 then switch to tag 4.00.1 and then apply the bundle. Jacques Carette has extensively re-written the printing of code values, and is currently maintaining this part. I'm grateful to him for encouragement and discussions. Introduction to staging and MetaOCaml The standard example of meta-programming -- the running example of A.P.Ershov's 1977 paper that begat meta-programming -- is the power function, computing x^n. In OCaml: let square x = x * x let rec power n x = if n = 0 then 1 else if n mod 2 = 0 then square (power (n/2) x) else x * (power (n-1) x) (* val power : int -> int -> int = <fun> *) Suppose our program has to compute x^7 many times. We may define let power7 x = power 7 x In MetaOCaml, we may also specialize the power function to a particular value n, obtaining the code which will later receive x and compute x^n. We re-write power n x annotating expressions as computed `now' (when n becomes known) and `later' (when x is given). let rec spower n x = if n = 0 then .<1>. else if n mod 2 = 0 then .<square .~(spower (n/2) x)>. else .<.~x * .~(spower (n-1) x)>. (* val spower : int -> ('cl, int) code -> ('cl, int) code = <fun> *) The two annotations, or staging constructs, are brackets .< e >. and escape .~e . Brackets .< e >. `quasi-quote' the expression e, annotating it as computed later. Escape .~e, which must be used within brackets, tells that e is computed now but produces the result for later. That result, the code, is spliced-in the containing bracket. The inferred type of spower is different. The result is no longer an int, but ('cl, int) code -- the code of expressions that compute an int. The first type argument to 'code', often named 'cl, is a so-called environment classifier and can be skipped on first reading. The type of spower spells out which argument is received now, and which later. To specialize spower to 7, we define let spower7_code = .<fun x -> .~(spower 7 .<x>.)>.;; (* val spower7_code : ('cl, int -> int) code = .< fun x_1 -> (x_1 * (((* cross-stage persistent value (id: square) *)) (x_1 * (((* cross-stage persistent value (id: square) *)) (x_1 * 1)))))>. *) and obtain the code of a function that will receive x and return x^7. Code, even of functions, can be printed, which is what MetaOCaml toplevel did. The print-out contains so-called cross-stage persistent values, or CSP, which `quote' present-stage values such as square to be used later. One may think of CSP as references to `external libraries' -- only in our case the program acts as a library for the code it generates. If we want to use thus specialized x^7 now, in our code, we should compile spower7_code and link it back to our program. This is called `running the code' let spower7 = .! spower7_code (* val spower7 : int -> int = <fun> *) The specialized spower7 has the same type as the partially applied power7 above. They behave identically. However, power7 x will do recursion on n, checking n's parity, etc. In contrast, specialized spower7 has no recursion (as can be seen from spower7_code). All operations on n have been done when the spower7_code was computed, producing the straight-lined code spower7 that operates only on x. MetaOCaml supports arbitrary number of later stages, letting us write not only code generators but also generators of code generators, etc. Data constructor restriction BER MetaOCaml N100 imposes the restriction that all data constructors and record labels used within brackets must come from the types that are declared in separately compiled modules. For example, the following all work: .<true>.;; .<raise Not_found>.;; .<Some [1]>.;; .<{Complex.re = 1.0; im = 2.0}>.;; let open Complex in .<{re = 1.0; im = 2.0}>.;; .<let open Complex in {re = 1.0; im = 2.0}>.;; because data types bool, option, list, Complex are either Pervasive or defined in the (separately compiled) standard library. However, the following are not allowed and flagged as compile-time error: type foo = Bar;; .<Bar>. module Foo = struct exception E end;; .<raise Foo.E>. The type declaration foo or the module declaration Foo must be moved into a separate file. The corresponding .cmi file must also be available at run-time: either placed into the same directory as the executable, or somewhere within the OCaml library search path. Scope extrusion test Although MetaOCaml permits manipulation and splicing of open code, its type system statically ensures that only closed code can be printed or run: We can't run the code we haven't finished constructing. For example, .<fun x -> .~(let y = .! .<x>. in .<y>.)>.;; gives a type error since .<x>. is obviously open code. This static guarantee holds only for pure code. Effects such as storing code values in mutable cells void the guarantee. Here is the example using the _old_ MetaOCaml from 2006 (version 3.09.1 alpha 030). (The problem can be illustrated simpler, but the following example is more realistic and devious.) let c = let r = ref .<fun z->z>. in let f = .<fun x -> .~(r := .<fun y -> x>.; .<0>.)>. in .<fun x -> .~f (.~(!r) 1)>. ;; (* val c : ('a, '_b -> int) code = .<fun x_4 -> ((fun x_2 -> 0) ((fun y_3 -> x_2) 1))>. *) One must look hard to see that x_2 is actually unbound in the resulting code. The problem is revealed when we attempt to run that code: .! c;; (* Characters 77-78: Unbound value x_2 Exception: Trx.TypeCheckingError. *) Since we get the error anyway (without much diagnostics though), one may discount the problem. Alas, sometimes scope extrusion results in no error -- just in wrong results. let c1 = let r = ref .<fun z->z>. in let _ = .<fun x -> .~(r := .<fun y -> x>.; .<0>.)>. in !r;; (* val c1 : ('a, '_b -> '_b) code = .<fun y_3 -> x_2>. *) we then use c1 to construct the code c2: let c2 = .<fun y -> fun x -> .~c1>.;; (* val c2 : ('a, 'b -> 'c -> '_d -> '_d) code = .<fun y_1 -> fun x_2 -> fun y_3 -> x_2>. *) which contains no unbound variables and can be run without problems. (.! c2) 1 2 3;; (* - : int = 2 *) It is most likely that the user did not intend for 'fun x ->' in c2 to bind x in c1. This is the blatant violation of lexical scope. And yet we get no error or other overt indication that something went wrong. BER MetaOCaml N100 has none of this. Although the type system still permits code values with escaped variables, attempting to use such code in any way -- splice, print or run -- immediately raises an exception. For example, entering the expression c in the top-level BER MetaOCaml N100 gives Exception: Failure Scope extrusion at Characters 89-99: let f = .<fun x -> .~(r := .<fun y -> x>.; .<0>.)>. in ^^^^^^^^^^ for the identifier x_7 bound at Characters 74-75: let f = .<fun x -> .~(r := .<fun y -> x>.; .<0>.)>. in ^ The exception message tells which variable got away, where it was bound and where it eloped. The file NOTES.txt in the BER MetaOCaml distribution describes the features of BER MetaOCaml in more detail and outlines directions for further development. Hopefully the release of BER MetaOCaml N100 would stimulate using and researching typed meta-programming. ** rixed and Francois Berenger asked, and oleg replied: > Could it be (ab)used to translate easily a whole program's OCaml source > into C code? In fact at one point MetaOCaml did so -- translated the generated OCaml code to C or Fortran. This feature -- called offshoring -- was used to generate FFT kernels. The generated C code was good enough to plug as it was into the FFTW benchmarking and testing framework. During the overhaul offshoring has been separated and left for clean-up and re-writing. Now there is a place for it in the overall architecture. > I understand that this is very interesting from a research > perspective, but practically meta-programming is only useful as long > as performances are the concern, for these problems when some critical > information is not known until runtime; I can't think of another usage > for runtime code specialisation anyway. > > So the question that immediately arises is then: why is metaocaml > supporting byte code only? Is a metaocamlopt planned, envisaged, > doable, in a galaxy not too far away? Before discussing the plans for the native run, let me clear a misconception. In two largish MetaOCaml projects I recall offhand, MetaOCaml was used to generate the optimal code for FFT kernels and for Gaussian Elimination. Gaussian Elimination is a large family of the algorithms, parameterized by domain (integer, float, polynomial), by the choice of pivoting, by the need to compute determinant, rank, permutation matrix, by the layout of matrices and by a half a dozen more aspects. Because Gaussian Elimination is often used in inner loops, the performance matters a great deal. Therefore, all this parameterization has to be done statically. Jacques Carette and I used MetaOCaml to generate GE procedures for particular parameter choices. The generated code was printed out, to be compiled in a library -- by the native OCaml compiler. Let me emphasize: the code generator was byte-code OCaml, but the generated code could be stored and compiled by the native OCaml. We used the byte-code run, but only for testing. The FFT project was similar: the generator was byte-code, the byte-code run was used for testing, the generated code was C and compiled with the Intel C compiler. The point is that the byte-code generator produces the code that can be compiled by any suitable compiler. The mode (byte-code or native) of the generator and of the generated code are not related. In my experience, the existing MetaOCaml is already useful in practice -- but more could and should be done. Regarding native MetaOCaml. As far as generating code is concerned, it can be done already. (I should try to make optmetaocamlc -- it should just work). The generated code can be stored on file, compiled any way we wish -- and linked back using dynlink. What is needed currently is to make this process automatic and convenient. The refactoring of MetaOCaml should help. The functions to run the code are now `user level'. That is, to change them or to add new ones, one does not have to re-compile MetaOCaml. Adding a new run function is as simple as adding a new library function -- just making sure the compiler can find your *.cmi and *.cmo/*.cmx files. I too want the end result quicker. Yet basics have to be done first. At the very least we need to get the codebase that one can work with and understand, and to be able to add features without breaking anything. I think basics is mostly done. ======================================================================== 6) Update on docs.camlcity.org Archive: <http://thread.gmane.org/gmane.comp.lang.caml.general/56699> ------------------------------------------------------------------------ ** Gerd Stolpmann announced: I've updated the code search engine on docs.camlcity.org. As before, it mainly bases on GODI packages, but it now also includes many OASIS packages. The search engine allows to search for definitions and uses of identifiers in OCaml source code. For example, "List.map" shows the files with the definitions at the top position of the result list, followed by the references of List.map. You can also search for type names, e.g. "in_channel". Type expressions are not yet supported (but I'm working on this). The search engines is especially useful when you want to read and understand code you've not written yourself, and that uses libraries you don't know yet. It's also good when you want to find sample code that uses a certain function. Note that there is also a browsing mode. Also, the viewer for source code allows it to click on identifiers triggering an ad-hoc search for other occurrences. Start page: <http://docs.camlcity.org> Query syntax: <http://docs.camlcity.org/docs/syntax.html> FAQ: <http://docs.camlcity.org/docs/faq.html> Technically, this update uses a dedicated full-text search engine for the first time. It's my own development Amber (not yet publicly released). The main advantage is the speed - searches take only a few milliseconds now, but it also improves scoring (based on inverted term frequencies). More about Amber later. ======================================================================== 7) Other Caml News ------------------------------------------------------------------------ ** From the ocamlcore planet blog: Thanks to Alp Mestan, we now include in the Caml Weekly News the links to the recent posts from the ocamlcore planet blog at <http://planet.ocaml.org/>. An alternative to camlp4 - Part 2: <http://lpw25.net/2013/02/05/camlp4-alternative-part-2.html> Implementing HAMT for OCaml: <http://gallium.inria.fr/blog/implementing-hamt-for-ocaml> User functions in Arakoon: <http://blog.incubaid.com/2013/02/01/user-functions-in-arakoon/> BER MetaOCaml 100: <http://caml.inria.fr/cgi-bin/hump.cgi?contrib=844> Ocurl 0.5.4: <http://caml.inria.fr/cgi-bin/hump.cgi?contrib=416> Discussions on the Syntactic Meta Programming(wg-camlp4 list): <http://hongboz.wordpress.com/2013/01/31/discussions-on-the-syntactic-meta-programmingwg-camlp4-list/> Introduction to Mezzo, continued: <http://gallium.inria.fr/blog/introduction-to-mezzo-2> ocurl 0.5.4 released: <https://forge.ocamlcore.org/forum/forum.php?forum_id=869> ======================================================================== Old cwn ------------------------------------------------------------------------ If you happen to miss a CWN, you can send me a message (alan.schm...@polytechnique.org) and I'll mail it to you, or go take a look at the archive (<http://alan.petitepomme.net/cwn/>) or the RSS feed of the archives (<http://alan.petitepomme.net/cwn/cwn.rss>). If you also wish to receive it every week by mail, you may subscribe online at <http://lists.idyll.org/listinfo/caml-news-weekly/> . ======================================================================== _______________________________________________ caml-news-weekly mailing list caml-news-weekly@lists.idyll.org http://lists.idyll.org/listinfo/caml-news-weekly