I thought I would resurrect this thread given the announcement of
Flight (link below). I'm not too familiar with Avatica, but it seems
like Flight (essentially a client-server framework for transporting
Arrow data) could be a good fit.
I remember when I was doing some old benchmarks, I had to build some
custom logic to get writes happening at a decent speed (some basic
batching and avoiding deserializing things when I could). I'm sure there
is much more we can do here, especially around our HTTP transport.
My impression is
Sounds like a very interesting issue.
While I’m evaluating Calcite for JDBC adaptor over postgreSQL with TPC-DS
queries, where Calcite queries 2~10 times slower than native postgresql queries
through psql. So, including JDBC latency issues, overall enhancement of
Avatica would be beneficial
This is a paper in VLDB 2018, "Don’t Hold My Data Hostage – A Case For Client
Protocol Redesign” by Mark Rassveldt and Hannes Muhleisen[1]. It claims that
database client protocols (inside ODBC and JDBC drivers) are very inefficient,
and has a compelling example where commercial drivers are 10x
