Re: arrow for game engine / graphics workloads?
Yes, I guess I’d end up taking a similar path to Arrow in this regard. I think I have some homework to do, to see whether I can use the Arrow format to model some things like meshes, scene graph layout, etc. If that is a good fit, it makes sense to use Arrow. Even if it isn’t a perfect fit, I like the idea of having the data in a more malleable or neutral form. Thank you for pointing that out. > On Dec 14, 2020, at 11:46 AM, Wes McKinney wrote: > > Arrow only uses Flatbuffers to serialize metadata, *not* data. > > On Mon, Dec 14, 2020 at 1:39 PM Robert Bigelow > wrote: >> >> This is an excellent point. I could use Flatbuffers directly to define any >> custom format needed by the engine. The engine itself would need to use the >> same principles the Arrow devs have, which I guess is true of any >> data-intensive system. Thanks for your response! >> >>> On Dec 14, 2020, at 11:24 AM, Lee, David >>> wrote: >>> >>> Arrow uses flatbuffers under the hood. >>> >>> https://google.github.io/flatbuffers/ >>> >>> FlatBuffers is an efficient cross platform serialization library for C++, >>> C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, >>> Rust and Swift. It was originally created at Google for game development >>> and other performance-critical applications. >>> >>> >>> -Original Message- >>> From: Robert Bigelow >>> Sent: Sunday, December 13, 2020 1:00 PM >>> To: dev@arrow.apache.org >>> Subject: arrow for game engine / graphics workloads? >>> >>> External Email: Use caution with links and attachments >>> >>> >>> Dear Arrow devs, >>> >>> I’m writing a game engine in Swift, and the next system to design is the >>> resource manager / asset database. Arrow seems like an attractive option >>> for the heart of an engine, since many of the performance goals stated for >>> analytic workloads are shared by real-time rendering. Data layout is >>> extremely important, and I’d like to be able to feed the renderer without >>> chasing pointers. So far my plan is to create a custom format for the asset >>> database that will be used at runtime. I plan on having a tool that >>> traverses a scene graph write this format, then have the resource manager >>> use mmap to load assets. Arrow seems like a good fit for such a format, as >>> (if I understand correctly) only the Schema needs to be deserialized before >>> the data would be available, and it could be used to back streaming APIs. >>> >>> Do you know of any work being done with Arrow in the real-time rendering or >>> game engine space? Would the API presented by Arrow be a good fit, assuming >>> I’d mostly need to expose buffers of typed data to feed the renderer? >>> >>> One final question assuming none of your answers have dissuaded me. Would >>> the C/glib Arrow library be reasonable, since Swift can import C headers? >>> It seems that no intrepid Swift developers have started a native Swift >>> implementation yet. >>> >>> Thanks for your time, and kudos for your work on the Arrow project. It is >>> very impressive! >>> >>> Cheers, >>> Rob Bigelow >>> >>> >>> This message may contain information that is confidential or privileged. If >>> you are not the intended recipient, please advise the sender immediately >>> and delete this message. See >>> http://www.blackrock.com/corporate/compliance/email-disclaimers for further >>> information. Please refer to >>> http://www.blackrock.com/corporate/compliance/privacy-policy for more >>> information about BlackRock’s Privacy Policy. >>> >>> >>> For a list of BlackRock's office addresses worldwide, see >>> http://www.blackrock.com/corporate/about-us/contacts-locations. >>> >>> © 2020 BlackRock, Inc. All rights reserved. >>
Re: arrow for game engine / graphics workloads?
Arrow only uses Flatbuffers to serialize metadata, *not* data. On Mon, Dec 14, 2020 at 1:39 PM Robert Bigelow wrote: > > This is an excellent point. I could use Flatbuffers directly to define any > custom format needed by the engine. The engine itself would need to use the > same principles the Arrow devs have, which I guess is true of any > data-intensive system. Thanks for your response! > > > On Dec 14, 2020, at 11:24 AM, Lee, David > > wrote: > > > > Arrow uses flatbuffers under the hood. > > > > https://google.github.io/flatbuffers/ > > > > FlatBuffers is an efficient cross platform serialization library for C++, > > C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, > > Rust and Swift. It was originally created at Google for game development > > and other performance-critical applications. > > > > > > -Original Message- > > From: Robert Bigelow > > Sent: Sunday, December 13, 2020 1:00 PM > > To: dev@arrow.apache.org > > Subject: arrow for game engine / graphics workloads? > > > > External Email: Use caution with links and attachments > > > > > > Dear Arrow devs, > > > > I’m writing a game engine in Swift, and the next system to design is the > > resource manager / asset database. Arrow seems like an attractive option > > for the heart of an engine, since many of the performance goals stated for > > analytic workloads are shared by real-time rendering. Data layout is > > extremely important, and I’d like to be able to feed the renderer without > > chasing pointers. So far my plan is to create a custom format for the asset > > database that will be used at runtime. I plan on having a tool that > > traverses a scene graph write this format, then have the resource manager > > use mmap to load assets. Arrow seems like a good fit for such a format, as > > (if I understand correctly) only the Schema needs to be deserialized before > > the data would be available, and it could be used to back streaming APIs. > > > > Do you know of any work being done with Arrow in the real-time rendering or > > game engine space? Would the API presented by Arrow be a good fit, assuming > > I’d mostly need to expose buffers of typed data to feed the renderer? > > > > One final question assuming none of your answers have dissuaded me. Would > > the C/glib Arrow library be reasonable, since Swift can import C headers? > > It seems that no intrepid Swift developers have started a native Swift > > implementation yet. > > > > Thanks for your time, and kudos for your work on the Arrow project. It is > > very impressive! > > > > Cheers, > > Rob Bigelow > > > > > > This message may contain information that is confidential or privileged. If > > you are not the intended recipient, please advise the sender immediately > > and delete this message. See > > http://www.blackrock.com/corporate/compliance/email-disclaimers for further > > information. Please refer to > > http://www.blackrock.com/corporate/compliance/privacy-policy for more > > information about BlackRock’s Privacy Policy. > > > > > > For a list of BlackRock's office addresses worldwide, see > > http://www.blackrock.com/corporate/about-us/contacts-locations. > > > > © 2020 BlackRock, Inc. All rights reserved. >
Re: arrow for game engine / graphics workloads?
This is an excellent point. I could use Flatbuffers directly to define any custom format needed by the engine. The engine itself would need to use the same principles the Arrow devs have, which I guess is true of any data-intensive system. Thanks for your response! > On Dec 14, 2020, at 11:24 AM, Lee, David > wrote: > > Arrow uses flatbuffers under the hood. > > https://google.github.io/flatbuffers/ > > FlatBuffers is an efficient cross platform serialization library for C++, C#, > C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust > and Swift. It was originally created at Google for game development and other > performance-critical applications. > > > -Original Message- > From: Robert Bigelow > Sent: Sunday, December 13, 2020 1:00 PM > To: dev@arrow.apache.org > Subject: arrow for game engine / graphics workloads? > > External Email: Use caution with links and attachments > > > Dear Arrow devs, > > I’m writing a game engine in Swift, and the next system to design is the > resource manager / asset database. Arrow seems like an attractive option for > the heart of an engine, since many of the performance goals stated for > analytic workloads are shared by real-time rendering. Data layout is > extremely important, and I’d like to be able to feed the renderer without > chasing pointers. So far my plan is to create a custom format for the asset > database that will be used at runtime. I plan on having a tool that traverses > a scene graph write this format, then have the resource manager use mmap to > load assets. Arrow seems like a good fit for such a format, as (if I > understand correctly) only the Schema needs to be deserialized before the > data would be available, and it could be used to back streaming APIs. > > Do you know of any work being done with Arrow in the real-time rendering or > game engine space? Would the API presented by Arrow be a good fit, assuming > I’d mostly need to expose buffers of typed data to feed the renderer? > > One final question assuming none of your answers have dissuaded me. Would the > C/glib Arrow library be reasonable, since Swift can import C headers? It > seems that no intrepid Swift developers have started a native Swift > implementation yet. > > Thanks for your time, and kudos for your work on the Arrow project. It is > very impressive! > > Cheers, > Rob Bigelow > > > This message may contain information that is confidential or privileged. If > you are not the intended recipient, please advise the sender immediately and > delete this message. See > http://www.blackrock.com/corporate/compliance/email-disclaimers for further > information. Please refer to > http://www.blackrock.com/corporate/compliance/privacy-policy for more > information about BlackRock’s Privacy Policy. > > > For a list of BlackRock's office addresses worldwide, see > http://www.blackrock.com/corporate/about-us/contacts-locations. > > © 2020 BlackRock, Inc. All rights reserved.
RE: arrow for game engine / graphics workloads?
Arrow uses flatbuffers under the hood. https://google.github.io/flatbuffers/ FlatBuffers is an efficient cross platform serialization library for C++, C#, C, Go, Java, Kotlin, JavaScript, Lobster, Lua, TypeScript, PHP, Python, Rust and Swift. It was originally created at Google for game development and other performance-critical applications. -Original Message- From: Robert Bigelow Sent: Sunday, December 13, 2020 1:00 PM To: dev@arrow.apache.org Subject: arrow for game engine / graphics workloads? External Email: Use caution with links and attachments Dear Arrow devs, I’m writing a game engine in Swift, and the next system to design is the resource manager / asset database. Arrow seems like an attractive option for the heart of an engine, since many of the performance goals stated for analytic workloads are shared by real-time rendering. Data layout is extremely important, and I’d like to be able to feed the renderer without chasing pointers. So far my plan is to create a custom format for the asset database that will be used at runtime. I plan on having a tool that traverses a scene graph write this format, then have the resource manager use mmap to load assets. Arrow seems like a good fit for such a format, as (if I understand correctly) only the Schema needs to be deserialized before the data would be available, and it could be used to back streaming APIs. Do you know of any work being done with Arrow in the real-time rendering or game engine space? Would the API presented by Arrow be a good fit, assuming I’d mostly need to expose buffers of typed data to feed the renderer? One final question assuming none of your answers have dissuaded me. Would the C/glib Arrow library be reasonable, since Swift can import C headers? It seems that no intrepid Swift developers have started a native Swift implementation yet. Thanks for your time, and kudos for your work on the Arrow project. It is very impressive! Cheers, Rob Bigelow This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information. Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy. For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations. © 2020 BlackRock, Inc. All rights reserved.
