On Tue, Jul 14, 2020 at 9:36 AM Micah Kornfield <emkornfi...@gmail.com> wrote:
> Hi Adam, > > > This sounds really interesting, how about adding the wasm build (C++) to > > the releases? > > I think this just needs someone to volunteer to do it and maintain it (at a > minimum if it doesn't already exist we need CI for it). We would also need > to figure out details of publishing and integrating it into the release > process. > Yes a wasm build for the core C++ library would be a welcome addition as well (as long as C++ maintainers agree whatever we do doesn't add a large maintenance burden). As Micah pointed out folks at JPMC have already done some work on this as part of Perspective, but we don't have any support in Arrow itself. I gave this a shot after being encouraged by [1], but ran into issues that I can't recall and gave up. Probably someone with more knowledge of C++ and cmake could get past it, especially given there's an example in Perspective. As far as release/publishing, for Rust there's wasm-pack [2] which would let us publish build artifacts to the npm registry for use in JS. I'm not sure if this is helpful for integrating with Spark or not. FWIW there was another thread [3] about wasm builds for Rust and C++ a while back. [1] https://github.com/apache/arrow/pull/3350#issuecomment-464517253 [2] https://rustwasm.github.io/wasm-pack/book/introduction.html [3] https://lists.apache.org/thread.html/e15dc80debf9dea1b33581fa6ba95fd84b57c0ccd0162505d5d25079%40%3Cdev.arrow.apache.org%3E > I've done a lot of asm.js work (different from wasm) in the past, but my > > assumption would be that using Rust instead of C++ as source for wasm > > should result in smaller wasm binaries. > > I don't know much about either, but I'm curious why you would expect this > to be the case? > > On Tue, Jul 14, 2020 at 8:07 AM Adam Lippai <a...@rigo.sk> wrote: > > > This sounds really interesting, how about adding the wasm build (C++) to > > the releases? > > I've done a lot of asm.js work (tfrom wasm) in the past, but my > > assumption would be that using Rust instead of C++ as source for wasm > > should result in smaller wasm binaries. > > Rust Arrow doesn't really use exotic solutions, eg. simd or tokio > > dependency can be turned off. > > > > Having DataFusion + some performant data access in browsers or even in > > node.js would be useful. > > Not needing to build fancy HTTP/GraphQL API over the Rust/C++ impl. but > > moving the data processing code to the client is viable for "small" > > workloads. > > Ofc if JS Arrow lands Flight support this may become less of an issue, > but > > AFAIK it's gRPC based which would need setting up a gRPC reverse proxy > for > > C++/Rust Arrow. > > Overall both the code-duplication and feature fragmentation would > decrease > > by using a single source (like you don't have a full Python impl. for > > obvious reasons) > > > > Best regards, > > Adam Lippai > > > > On Tue, Jul 14, 2020 at 4:27 PM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > > >> Fwiw, I believe at least the core c++ library already can be compiled to > >> wasm. I think perspective does this [1] > >> > >> > >> I'm curious What are you hoping to achieve with embedded wasm in > spark? > >> > >> Thanks, > >> Micah > >> > >> [1] https://perspective.finos.org/ > >> > >> On Tuesday, July 14, 2020, Brian Hulette <hulet...@gmail.com> wrote: > >> > >> > That sounds great! I'd like to have some support for using the rust > >> and/or > >> > C++ libraries in the browser via wasm as well. > >> > As long as the community is ok with your overall approach "to add > >> compiler > >> > conditionals around any I/O features and libc dependent features of > >> these > >> > two libraries," I think it may be best to start with a PR and discuss > >> > specifics from there. > >> > > >> > Do any rust contributors have objections to this? > >> > > >> > Brian > >> > > >> > On Mon, Jul 13, 2020 at 9:42 PM RJ Atwal <atwal...@gmail.com> wrote: > >> > > >> > > Hi all, > >> > > > >> > > Looking for guidance on how to submit a design and PR to add WASM32 > >> > support > >> > > to apache arrow's rust libraries. > >> > > > >> > > I am looking to use the arrow library to pass data in arrow format > >> > between > >> > > the host spark environment and UDFs defined in WASM . > >> > > > >> > > I created the following JIRA ticket to capture the work > >> > > https://issues.apache.org/jira/browse/ARROW-9453 > >> > > > >> > > Thanks, > >> > > RJ > >> > > > >> > > >> > > >
