>
> to build a FlightSQL producer that delegates to other databases (possibly
> using JDBC?).
> Then building an JSON-based API or service on top of the FlightSqlClient.
>
Yes, exactly =D
Thank you for the detailed breakdown, that makes things very clear.
In this case, the JSON API and FlightSQL server would likely be the same
process, so I think your scenario #1 would be my answer.
You also have the option of using the Arrow JDBC driver to get the benefit
> of the Arrow Flight protocol but still write your code using JDBC.
>
I have been following that PR on Github, it's very exciting.
One thing I am not sure I understand though -- with the JDBC driver for
Arrow, there still needs to be a FlightSQL Server talking to the database
right?
So if I understand it, with the FlightSQL JDBC driver, it takes the place
of the FlightSQL Client and the request flow would be something like:
Client <--> JSON API <--> FlightSQL JDBC Driver <--> FlightSQL Server
<---> Database
And to connect the FlightSQL Server to the Database would require wrapping
it in
JDBC (unless some native/direct wire protocol implementation was written
for each DB) right?
Is this more performant than querying through regular JDBC? (Maybe because
of Arrow's format?)
On Fri, Feb 25, 2022 at 7:33 PM James Duong <[email protected]> wrote:
> Hi Gavin,
>
> If I'm understanding correctly, what you're thinking of is to build a
> FlightSQL producer that delegates to other databases (possibly using JDBC?).
> Then building an JSON-based API or service on top of the FlightSqlClient.
>
> There are two potentially remote calls happening here:
> JSON API (FlightSqlClient) -> FlightSqlServer
> FlightSqlServer -> each database being federated to.
>
> In this scenario, the benefits of Flight vary depending on your network
> topology:
> 1. If the JSON API and FlightSqlServer are co-located there is little time
> being spent on the network sending data through Flight, so it is not very
> beneficial.
> 2. If the JSON API is remote and FlighSqlServer is co-located with
> federated databases, the majority of the network transmission will be using
> Flight so you should see some performance benefits.
> 3. If the JSON API, FlightSqlServer, and databases are all remote from
> each other, you _might_ see benefits to Flight.
>
> If you instead built your JSON app to federate queries directly to each
> database you have one remote call happening
> JSON API (JDBC) -> federated database
> This might be faster or slower depending on the network conditions between
> the JSON API and the federated database.
> The above would use the JDBC driver's protocol which may not perform as
> well as Flight.
>
> You also have the option of using the Arrow JDBC driver to get the benefit
> of the Arrow Flight protocol but still write your code using JDBC.
>
> On Fri, Feb 25, 2022 at 3:59 PM Gavin Ray <[email protected]> wrote:
>
>> Excuse me if this question seems a bit silly, but I'm wondering whether
>> it would
>> make sense to use FlightSQL to power a JSON API that talks to multiple
>> databases.
>>
>> I know that the docs around Flight/FlightSQL say that there is a marked
>> performance improvement over ODBC/JDBC, but I assume this is only if the
>> service
>> sending the data over Flight isn't using one of these to interact with the
>> datasource, right?
>>
>> If I have a JVM application using JDBC and sending the responses as JSON,
>> would
>> it still make sense to look towards implementing a FlightSQL Server
>> because of
>> the ability to distribute operations across multiple instances and it
>> being
>> language-agnostic? Or would I be losing most of the benefits here?
>>
>> Not familiar with the Arrow format and project as a whole, so still
>> trying to
>> wrap my head around things, sorry!
>>
>> Thank you =)
>> Gavin Ray
>>
>
>
> --
>
> *James Duong*
> Lead Software Developer
> Bit Quill Technologies Inc.
> Direct: +1.604.562.6082 | [email protected]
> https://www.bitquilltech.com
>
> This email message is for the sole use of the intended recipient(s) and
> may contain confidential and privileged information. Any unauthorized
> review, use, disclosure, or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy
> all copies of the original message. Thank you.
>
