hi Eric, On Fri, Sep 6, 2019 at 5:09 PM Eric Erhardt <eric.erha...@microsoft.com.invalid> wrote: > > I was looking for the high level scenarios for the Plasma In-Memory Object > Store. A colleague of mine suggested we could use it to pass data between a > C# process and a Python process. > > I've read the intro blog [0] on Plasma, which describes using the same data > set from multiple processes - which sounds like the same scenario as above. > > I am trying to prioritize creating C# bindings for the Plasma client. So I'd > like to know all the scenarios that would could be enabled with Plasma. > > For example: > - could using Plasma speed up Pandas UDFs in PySpark? Because the data > wouldn't have to go across the socket between Java and Python, but instead > would be memory-mapped. We have similar functionality in .NET for Apache > Spark.
Memory still would need to be copied into the memory-mappable file, so it's unclear whether this would be faster than passing the data through a socket as it's being done now. > - Is Plasma being used by Nvidia RAPIDS? AFAIK it is not. It doesn't seem out of the question, though, given that we have some level of CUDA support in Plasma now. > > I know Plasma today is not supported on Windows, but I think support could be > added since Windows supports memory mapped files (through a different API > than mmap) and it now supports Unix Domain Sockets [1]. > > Also - side question about the c_glib bindings. I assume those will only ever > work on Windows with something like Cygwin or MSYS2, right? Would people be > opposed to adding pure "C" exports to the plasma library so the C# bindings > could use it? (similar to the JNI support today). > In theory you could use the GLib-based library with MSVC, the main requirement is gobject-introspection https://github.com/GNOME/gobject-introspection/blob/master/MSVC.README.rst Note that GLib itself is LGPL-licensed -- since it is an optional component in Apache Arrow, it is OK for optional components to have an LGPL dependency (though ASF projects aren't allowed to have mandatory/hard dependencies on LGPL). So if you do go that route just beware the possible issues you might have down the road. I have no objection to adding a "plasma/plasma-c.h" with C exports. > Eric > > [0] > https://ray-project.github.io/2017/08/08/plasma-in-memory-object-store.html > [1] https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
