Thanks Wes & Sebastien, I've tested Arrow in Go-WASM now and it is working fine. Still getting my head around best way to use it for our Use case (IoT Data)
My goal here is to hit a Flight endpoint from the Browser (GO-WASM specifically), and pull (all or part of) an Arrow dataset on the server, into the Browser for visualization and local analysis. One of the issues I will contend with is that visualization can 'walk backwards' in a larger dataset, (Scrolling up), not just forwards like an analytic generally does. Second goal is to update this visualization from a realtime stream, which can be as fast as 1 sample per second. I'm wondering if a use (abuse ?) of batches and pre-allocation might work for streaming updates. Note: It may be that using arrow like this for visualization is not appropriate, but I think it would be great if it can. Regards Mark. -----Original Message----- From: Sebastien Binet <w...@sbinet.org> Sent: Wednesday, August 12, 2020 1:53 PM To: dev@arrow.apache.org Subject: Re: Arrow Flight + Go, Arrow for Realtime Mark, AFAIK, nobody's actively working on Arrow-Flight for Go (I think somebody started that work at some point but I don't remember anything hitting the main repo) as for Go+WASM: https://lists.apache.org/thread.html/e15dc80debf9dea1b33581fa6ba95fd84b57c0ccd0162505d5d25079%40%3Cdev.arrow.apache.org%3E ie: === I've just tried compiling this example: - https://godoc.org/github.com/apache/arrow/go/arrow#example-package--Table to wasm. compilation went fine: $> GOOS=js GOARCH=wasm go build -o foo.wasm foo.go $> go-wasm ./foo.wasm rec[0]["f1-i32"]: [1 2 3 4 5] rec[0]["f2-f64"]: [1 2 3 4 5] rec[1]["f1-i32"]: [6 7 8 (null) 10] rec[1]["f2-f64"]: [6 7 8 9 10] rec[2]["f1-i32"]: [11 12 13 14 15] rec[2]["f2-f64"]: [11 12 13 14 15] rec[3]["f1-i32"]: [16 17 18 19 20] rec[3]["f2-f64"]: [16 17 18 19 20] and it ran fine once this patch was added: - https://github.com/apache/arrow/pull/3707 hth, -s PS: go-wasm is an alias of mine for this file: https://github.com/golang/go/blob/master/misc/wasm/go_js_wasm_exec === hth, -s ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Wednesday, August 12, 2020 11:29 AM, <m...@markfarnan.com> wrote: > I'm looking at using Arrow for a realtime IoT project which includes > use cases both on server, and also for transferring /using in a > Browser via WASM, and have a few questions. > > Language in use is Go. > > Is anyone working on implementing Arrow-Flight in Go ? (According to > the feature matrix, nothing ready yet, so wanted to check. > > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if so, > any issues ? > > Any pointers/documentation on using/extending Arrow for realtime > streaming cases. (Specifically where a DataFrame is requested, but > then it needs to 'grow' as new data arrives, often at high speed). > > Not language specific, just trying to understand the right pattern for > using Arrow for this, and couldn't' find much in the docs. > > Regards > > Mark.
