On 07/31/2015 01:16 AM, Jeff Burdges wrote: > > Just some thoughts on doing GNUNet code in Rust : > > Rust is playing the "long game" in that they want to get things right. > We should probably adopt this attitude when writing GNUNet layers in > Rust, while staying more pragmatic when working in C.
Well, can't disagree with "getting things right" ;-). > What would be an example? > > I suspect one obstacle to GNUNet's wider adoption is the relative > disjointness from outside projects. An important example is GNUNet > using its own scheduler and IO layers, as opposed to using say libevent, > which afaik does exactly the same thing. A curious developer who sees a > project using libevent will think "Oh yeah I've love to learn that > anyways", while viewing GNUNet's scheduler, etc. as obstacles (less > transferable knowledge). Well, I have a few comments on this: 1) Programming with event loops is programming with event loops, which library you use doesn't matter too much. Not knowing how to program with event loops is a major obstacle, but the specific library is not that critical IMO. 2) There isn't just libevent and libev. There's also libuv and glib, and then a bunch more that are (probably?) less well-known. They all have problems: a) libevent provides the kitchen-sink. It includes logging, a DNS resolver, an HTTP server and an RPC framework. Logging seems less powerful than what we have in GNUnet, and the HTTP server seems to require more manual handholding by the user and lack some features compared to GNU libmicrohttpd (starting with simple things like TLS support). Similarly, the DNS client is somewhat restricted over the dnsstub we have in GNUnet (src/dns/). The key-value lookups in the RPC API would suggest that it will be somewhat slower than the existing RPC-style used in GNUnet. Given how much RPC we do, this might be relevant. So while the kitchen-sink may largely seem useful, I'm not sure we'll be happy with the details. b) libev isn't (yet) ported to Windows and many other platforms. It is unclear to me if it is actually faster, or if that has been fixed for libevent somehow. It also _seems_ that libevent is better supported, but maybe libev is simpler and thus just requires less support... c) glib comes *really* with the kitchen sink: it's own type system, plugin loading/unloading, string manipulation, memory management, logging, and the list goes on. glib is so big that it is hard to justify deploying it in a system to be used also on 'small' (GUI-less) devices. d) All of this said, the GNUnet scheduler *also* has known issues: (1) it's still based on select(), we should really eventually support poll()/epoll()/kqueue(). The timeout queue has linear worst-case performance, better data structures exist, even within GNUnet. We should have used a function returning the `struct GNUNET_SCHEDULER_TaskContext` instead of passing it to each GNUNET_SCHEDULER_Task, thereby avoiding an extra argument that remains unused in 99% of all cases. Finally, the existing shutdown semantics are awful, we should have a separate task category to run on-shutdown, and not ready all tasks. Furthermore, a better integration with the GNUENT_SIGNAL-API might also be nice. So my wishlist here is also far from empty. 3) But really, the answer is entirely different: what we need is a *pluggable* event loop, just like the glib event loop is pluggable. Right now, it is somewhere between hard and impossible to combine GNUnet code with code that uses another event loop (other than glib). And glib is painful (checkout gnunet-gtk/src/common/eventloop.c). I would like to be in a situation where developers can initialize libgnunetutil and tell it to use libev, libevent, glib, GNUnet-native or any other custom event loop by passing some (ideally small) glue code. That's what we should strive to achieve -- in addition to fixing the (d)-issues eventually. The problem with the (d)-issues is mostly that it is a lot of effort for hardly any (visible, short-term) effect, so it doesn't put these high on my list. Still, just like migrating more code to the MQ-API and cleaning up our RPC logic, it does make sense to invest in this as part of a long-term strategy towards clean code. > Rust has a very good FFI to C code, but writing safe wrappers around > unsafe C code can incur overhead. In particular, Rust folk had issues > with libuv, yet another take on libevent. And GNUNet utils appears > quite unsafe by Rust standards, even before worrying about leaked > sockets. Care to elaborate about 'unsafe'? Also, I'm not sure what you refer to by 'leaked sockets', but there is one particular case where we leak a socket intentionally to safely signal process termination (gnunet-service-arm leaks the client socket from a connection by "gnunet-arm -e", so that "gnunet-arm -e" can wait for a read() on that connection to fail and thus be sure that the gnunet-service-arm process has been cleaned up by the OS entirely). If you worry about that particular instance, well, C sometimes enables creative problem solving ;-). > Instead, Rust has tools like : > https://github.com/carllerche/mio > https://github.com/dwrensha/gj > These are a callback based asynchronous IO toolkit, much like libevent, > GNUNet utils, etc., but they do it the "Rust way", i.e. safe and pretty. > And Rust's closures make callbacks comprehensible. I don't really see a semantic difference between gj/kj/promises and what we do in GNUnet. The fact that we have to deal with C without lambdas and (automatic) closures may make the code harder to read, but in return we get C and in particular the great tool support chains for C. > In this short term, there would be more work required to write GNUNet > code using mio, gj, etc. Worse, mio does not support Window yet, > depends upon unstable language features, and might still face tricky > networking hurdles. In the long term, we'd have more people actually > wanting to write code with us though. I'd agree with this conclusion: 1) Certain gnunetutil APIs still need work, and as we move forward those other styles may provide inspiration --- and being more compatible with them would generally be a good thing. 2) Quality bindings (or full implementations) in any reasonably popular and sane language are always helpful to get more people on board. 3) This is one issue among many. It is reasonable to have it on the radar as a long-term consideration, but there are a bunch of short-term issues we should focus on first. In particular, I see that bratao seriously broke NAT (#3921, #3918), and that we have some fresh critical issues in CADET, CORE and DATASTORE in the bugtracker. Shortening this list of typically rather trivial issues from key systems is likely a rather cost-effective method of attracting new users and developers ;-). So I'll prioritize fixing those bugs over adding new features or rewriting stuff in new languages for now. My 2 cents Christian
signature.asc
Description: OpenPGP digital signature
_______________________________________________ GNUnet-developers mailing list [email protected] https://lists.gnu.org/mailman/listinfo/gnunet-developers
