Dear BigTop Devs, I'm a member of the Big Data / High-Performance I/O team at IBM Research that develops the Crail open source project (*www.crail.io* <http://www.crail.io>). Together with PMC member Amir Sanjar we have been exploring the possibility of donating the Crail code to BigTop and we would like to hear your thoughts about such a potential integration.
Let me briefly explain what Crail is about: Crail is a platform for storing (and orchestrating access to) performance critical data in big data analytics jobs at very high speed. Crail is designed from ground up to leverage (a) high-speed networking and storage hardware like 100Gbps networks and NVMe flash, and (b) new modes of data center deployments like disaggregated data centers, rack-scale deployments and heterogeneous deployments (e.g., including hardware accelerators), etc. At the same time, Crail also works in traditional deployments across commodity network and storage hardware. While currently targeting Linux systems, Crail is otherwise architecturally independent and has been tested on both x86 and ppc64 based architectures. Let me give a few concrete examples of how Crail is used. For instance, Crail is used to orchestrate access to temporary intra-job data like shuffle data in Hadoop or Spark or broadcast objects, cached RDDs, etc. in Spark. Crail is also used to store temporary inter-job data in complex multi-stage jobs (e.g. a map/reduce job followed by a Spark SQL job). Crail can generally be used as a storage space for data that is repeatedly processed like SQL parquet tables. Today, such data orchestration is done using various techniques. Intra-job data is typically managed by the compute frameworks through point-to-point data exchanges implemented using TCP sockets for communication and local file systems as a temporary storage for the data. For sharing inter-job data there are systems like Alluxio (formerly Tachyon) or distributed caches (e.g. Redis). The problem is, each of these solutions can only be used for one particular type of data or use case (latency sensitive data, large data, shuffle, etc.). Additionally, new high-performance hardware and new data center architectures are not leveraged or used very poorly. Crail, on the other hand, is a platform that implements very efficient data orchestration for both small and large data sets at the speed of the hardware, while supporting resource disaggregation and other new forms of deployments. To give you an idea, Crail–based data shuffling in Spark reaches a network usage of 80Gbps all-to-all, whereas - on the very same hardware - the traditional shuffle implementation is stuck at 10-20Gbps or lower. These improvements at all-to-all data sharing eventually resulted in a 6x gain at the workload runtime level. A blog post about this can be found on the Crail website (*http://www.crail.io/blog/2017/01/sorting.html* <http://www.crail.io/blog/2017/01/sorting.html>). Another example is storage disaggregation: Crail supports complete disaggregation of memory and flash for input/output and intra-job data in a Hadoop/Spark/Hive/etc. job with literally zero performance costs, as shown in our recent Spark Summit presentation ( *https://www.slideshare.net/databricks/running-apache-spark-on-a-highperformance-cluster-using-rdma-and-nvme-flash-with-patrick-stuedi* <https://www.slideshare.net/databricks/running-apache-spark-on-a-highperformance-cluster-using-rdma-and-nvme-flash-with-patrick-stuedi> ). Crail is developed (actively!) under the Apache licence, but it's not yet an Apache project or an Apache incubator project. However, we would like it to become one and we are looking for support along that direction. We are also committed to provide long-term support for Crail and of course we are very much interested in expanding the community behind Crail. We are currently a team of about 8 developers. We think Crail could be beneficial for BigTop as it helps to unleash performance gains of data processing frameworks on modern high-performance clusters. Already many of the hardware trends Crail is designed for are making inroads to data centers. For instance, Amazon offers high-speed instances with 40Gbps network and NVMe flash. Microsoft Azure offers RDMA networking as part of their cloud offering. We believe this trend is going to continue and Crail is the platform that allows these hardware trends to be consumed by the big data frameworks. We are looking forward to hear from you guys and get feedback about a potential integration of Crail with BigTop. Regards, Patrick
