Arrow only uses Flatbuffers to serialize metadata, *not* data.
On Mon, Dec 14, 2020 at 1:39 PM Robert Bigelow <[email protected]> wrote: > > This is an excellent point. I could use Flatbuffers directly to define any > custom format needed by the engine. The engine itself would need to use the > same principles the Arrow devs have, which I guess is true of any > data-intensive system. Thanks for your response! > > > On Dec 14, 2020, at 11:24 AM, Lee, David <[email protected]> > > wrote: > > > > Arrow uses flatbuffers under the hood. > > > > https://google.github.io/flatbuffers/ > > > > FlatBuffers is an efficient cross platform serialization library for C++, > > C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, > > Rust and Swift. It was originally created at Google for game development > > and other performance-critical applications. > > > > > > -----Original Message----- > > From: Robert Bigelow <[email protected]> > > Sent: Sunday, December 13, 2020 1:00 PM > > To: [email protected] > > Subject: arrow for game engine / graphics workloads? > > > > External Email: Use caution with links and attachments > > > > > > Dear Arrow devs, > > > > I’m writing a game engine in Swift, and the next system to design is the > > resource manager / asset database. Arrow seems like an attractive option > > for the heart of an engine, since many of the performance goals stated for > > analytic workloads are shared by real-time rendering. Data layout is > > extremely important, and I’d like to be able to feed the renderer without > > chasing pointers. So far my plan is to create a custom format for the asset > > database that will be used at runtime. I plan on having a tool that > > traverses a scene graph write this format, then have the resource manager > > use mmap to load assets. Arrow seems like a good fit for such a format, as > > (if I understand correctly) only the Schema needs to be deserialized before > > the data would be available, and it could be used to back streaming APIs. > > > > Do you know of any work being done with Arrow in the real-time rendering or > > game engine space? Would the API presented by Arrow be a good fit, assuming > > I’d mostly need to expose buffers of typed data to feed the renderer? > > > > One final question assuming none of your answers have dissuaded me. Would > > the C/glib Arrow library be reasonable, since Swift can import C headers? > > It seems that no intrepid Swift developers have started a native Swift > > implementation yet. > > > > Thanks for your time, and kudos for your work on the Arrow project. It is > > very impressive! > > > > Cheers, > > Rob Bigelow > > > > > > This message may contain information that is confidential or privileged. If > > you are not the intended recipient, please advise the sender immediately > > and delete this message. See > > http://www.blackrock.com/corporate/compliance/email-disclaimers for further > > information. Please refer to > > http://www.blackrock.com/corporate/compliance/privacy-policy for more > > information about BlackRock’s Privacy Policy. > > > > > > For a list of BlackRock's office addresses worldwide, see > > http://www.blackrock.com/corporate/about-us/contacts-locations. > > > > © 2020 BlackRock, Inc. All rights reserved. >
