Yes, I guess I’d end up taking a similar path to Arrow in this regard. I think I have some homework to do, to see whether I can use the Arrow format to model some things like meshes, scene graph layout, etc. If that is a good fit, it makes sense to use Arrow. Even if it isn’t a perfect fit, I like the idea of having the data in a more malleable or neutral form. Thank you for pointing that out.
> On Dec 14, 2020, at 11:46 AM, Wes McKinney <wesmck...@gmail.com> wrote: > > Arrow only uses Flatbuffers to serialize metadata, *not* data. > > On Mon, Dec 14, 2020 at 1:39 PM Robert Bigelow > <rfbige...@mac.com.invalid> wrote: >> >> This is an excellent point. I could use Flatbuffers directly to define any >> custom format needed by the engine. The engine itself would need to use the >> same principles the Arrow devs have, which I guess is true of any >> data-intensive system. Thanks for your response! >> >>> On Dec 14, 2020, at 11:24 AM, Lee, David <david....@blackrock.com.INVALID> >>> wrote: >>> >>> Arrow uses flatbuffers under the hood. >>> >>> https://google.github.io/flatbuffers/ >>> >>> FlatBuffers is an efficient cross platform serialization library for C++, >>> C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, >>> Rust and Swift. It was originally created at Google for game development >>> and other performance-critical applications. >>> >>> >>> -----Original Message----- >>> From: Robert Bigelow <rfbige...@gmail.com> >>> Sent: Sunday, December 13, 2020 1:00 PM >>> To: dev@arrow.apache.org >>> Subject: arrow for game engine / graphics workloads? >>> >>> External Email: Use caution with links and attachments >>> >>> >>> Dear Arrow devs, >>> >>> I’m writing a game engine in Swift, and the next system to design is the >>> resource manager / asset database. Arrow seems like an attractive option >>> for the heart of an engine, since many of the performance goals stated for >>> analytic workloads are shared by real-time rendering. Data layout is >>> extremely important, and I’d like to be able to feed the renderer without >>> chasing pointers. So far my plan is to create a custom format for the asset >>> database that will be used at runtime. I plan on having a tool that >>> traverses a scene graph write this format, then have the resource manager >>> use mmap to load assets. Arrow seems like a good fit for such a format, as >>> (if I understand correctly) only the Schema needs to be deserialized before >>> the data would be available, and it could be used to back streaming APIs. >>> >>> Do you know of any work being done with Arrow in the real-time rendering or >>> game engine space? Would the API presented by Arrow be a good fit, assuming >>> I’d mostly need to expose buffers of typed data to feed the renderer? >>> >>> One final question assuming none of your answers have dissuaded me. Would >>> the C/glib Arrow library be reasonable, since Swift can import C headers? >>> It seems that no intrepid Swift developers have started a native Swift >>> implementation yet. >>> >>> Thanks for your time, and kudos for your work on the Arrow project. It is >>> very impressive! >>> >>> Cheers, >>> Rob Bigelow >>> >>> >>> This message may contain information that is confidential or privileged. If >>> you are not the intended recipient, please advise the sender immediately >>> and delete this message. See >>> http://www.blackrock.com/corporate/compliance/email-disclaimers for further >>> information. Please refer to >>> http://www.blackrock.com/corporate/compliance/privacy-policy for more >>> information about BlackRock’s Privacy Policy. >>> >>> >>> For a list of BlackRock's office addresses worldwide, see >>> http://www.blackrock.com/corporate/about-us/contacts-locations. >>> >>> © 2020 BlackRock, Inc. All rights reserved. >>
