Add Yuval in loop. Yuval is SparkRDMA developer. Qingchun Song Cell: +8613501218637 E-Mail: [email protected] 获取 Outlook<https://aka.ms/qtex0l> for iOS ________________________________ From: Patrick Stuedi <[email protected]> Sent: Saturday, February 17, 2018 11:00:50 PM To: [email protected] Cc: [email protected]; Qingchun Song Subject: Re: C (or language agnostic) API for Crail
That's great, one of the main goals of crail being an apache incubator project is to get more people involved in the development of crail. I've been following your contributions to tensorflow, nice work! Collaborating in this context (incl mxnet) would be very interesting. There are multiple ways to go. Once we have the core c++ client we could need help in the developmen of the various bindings (rdma, tcp, for storage and rpc). Or we could need help in leveraging crail in tensirflow and mxnet (param server, storage of the model > dram). Let us know where you see opportinities. On Feb 17, 2018 3:36 PM, "Bairen YI" <[email protected]<mailto:[email protected]>> wrote: Hi Patrick, That would be fantastic. In fact we would love to get more involved as our lab in HKUST has partnered with MLNX to codevelop datacenter scale AI software solution (TensorFlow and Apache MXNet), and we could encourage a couple of students contributing code to Crail at this very stage if we see fit. It could also bring novel system/networking research opportunities to our lab. Let me know how we could better work together. Best, Bairen > On 17 Feb 2018, at 22:19, Patrick Stuedi > <[email protected]<mailto:[email protected]>> wrote: > > Hi Bairen, > > Your comment is just on spot. The development of a c++ Api for crail is one > of the top items on the roadmap, in partical to facilitate the integration > into tensorflow and serverless. In fact i started drafting a prototype two > weeks ago that i wanted to share soon. If you are interested in helping let > us know! > > > > On Feb 17, 2018 1:49 PM, "Bairen YI" > <[email protected]<mailto:[email protected]>> wrote: > > HI folks, > > I have been looking into you guys’ work for a long time and it is great to > see Crail accepted as an Apache Incubator project. > > I authored the GPU Direct RDMA transport for TensorFlow ( > https://github.com/tensorflow/tensorflow/pull/11392<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftensorflow%2Ftensorflow%2Fpull%2F11392&data=02%7C01%7Cqingchun%40mellanox.com%7C305036780e114ee781f908d576173d21%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636544764585899945&sdata=pDzYCjIYsPEdLd4Ggw6tIMUusSpPnZ01ULw5f%2F%2Bki44%3D&reserved=0>), > and I would love to > see how we could design an end-to-end zero-copy dataflow from Crail to > various deep learning framework such as TensorFlow ( > https://dl.acm.org/citation.cfm?doid=3123878.3131975<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdl.acm.org%2Fcitation.cfm%3Fdoid%3D3123878.3131975&data=02%7C01%7Cqingchun%40mellanox.com%7C305036780e114ee781f908d576173d21%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636544764585899945&sdata=4qFyBQGRQRhqDcFjtLdA1J6ibloQ%2F6mI%2BZaWDST6rr8%3D&reserved=0>). > > Is there any roadmap for Crail as a standalone language-independent > FileSystem/Cache service with C API? That would really ease the integration > into non-JVM based third party system. It does not have to be HDFS > compatible if that brings extra performance cost. > > Best, > Bairen
