So I’m actually doing this currently in production for a service, as I spoke about in a talk at the Subsurface conference.
For the Request, we have a Protobuf message that consists of two strings: the GraphQL query and an optional JSON string for variable definitions. We marshal the protobuf message to bytes which are used as the “ticket” for the `DoGet` request through Flight. Since pretty much every existing GraphQL engine outputs JSON right now, we’ve essentially built our own execution engine at this point by utilizing the planner from https://pkg.go.dev/github.com/jensneuse/graphql-go-tools and a custom built execution layer to execute the generated plan. Because Arrow already contains complex types like nested structs / lists / etc. it’s not too difficult to construct an arrow Schema from the expected GraphQL response schema and just return a stream of record batches. --Matt From: Gavin Ray <[email protected]> Sent: Wednesday, May 11, 2022 10:30 AM To: [email protected] Subject: GraphQL over Arrow (+ Flight)? If anyone is familiar with both GraphQL and Arrow, I'm curious how exactly using these two together might look GraphQL is transport-agnostic, so you can theoretically use it over anything, a good case study being Dan Luu's article here: If anyone is familiar with both GraphQL and Arrow, I'm curious how exactly using these two together might look GraphQL is transport-agnostic, so you can theoretically use it over anything, a good case study being Dan Luu's article here: https://danluu.com/simple-architectures/<https://urldefense.com/v3/__https:/danluu.com/simple-architectures/__;!!PBKjc0U4!LicJnBEMitViK3f8WuQjuSyTDDOogBbNcayOeiKGzh6ji7bczg1ZwYxx1U6LyevlM23lMw5OeScm9mYD2FA$> > "Some areas where we’re happy with our choices even though they may not > sound like the simplest feasible solution are with our API, where we use > GraphQL, with our transport protocols, where we had a custom protocol for a > while, and our host management, where we use Kubernetes. For our transport > protocols, we used to use a custom protocol that runs on top of UDP, with an > SMS and USSD fallback, for the performance reasons described in this talk. > With the rollout of HTTP/3, we’ve been able to replace our custom protocol > with HTTP/3 and we generally only need USSD for events like the recent > internet shutdowns in Mali)." I've seen also GraphQL done over Protobuf/gRPC, TCP/MsgPack, and a custom binary format: - https://github.com/google/rejoiner/blob/b1cb09e9bbf7ac68bfd9c93f23a73b691e6ead72/examples-gradle/src/main/java/com/google/api/graphql/examples/streaming/graphqlserver/GraphQlGrpcServer.java#L44<https://urldefense.com/v3/__https:/github.com/google/rejoiner/blob/b1cb09e9bbf7ac68bfd9c93f23a73b691e6ead72/examples-gradle/src/main/java/com/google/api/graphql/examples/streaming/graphqlserver/GraphQlGrpcServer.java*L44__;Iw!!PBKjc0U4!LicJnBEMitViK3f8WuQjuSyTDDOogBbNcayOeiKGzh6ji7bczg1ZwYxx1U6LyevlM23lMw5OeScmpUCkRiE$> - https://github.com/OlegIlyenko/sangria-tcp-msgpack-example<https://urldefense.com/v3/__https:/github.com/OlegIlyenko/sangria-tcp-msgpack-example__;!!PBKjc0U4!LicJnBEMitViK3f8WuQjuSyTDDOogBbNcayOeiKGzh6ji7bczg1ZwYxx1U6LyevlM23lMw5OeScmo3VMXhI$> - https://github.com/esseswann/graphql-binary<https://urldefense.com/v3/__https:/github.com/esseswann/graphql-binary__;!!PBKjc0U4!LicJnBEMitViK3f8WuQjuSyTDDOogBbNcayOeiKGzh6ji7bczg1ZwYxx1U6LyevlM23lMw5OeScmErmXk2s$> If someone were interested in using Arrow as the encoding layer, how would this work in practice? Arrow messages need to have a well-defined schema, and GraphQL queries return dynamic, nested data, so I'm having a hard time understanding how you'd go about representing/encoding that in an Arrow message. Thank you =)
