Agree with Randy. A more indirect option is to use Thrift to send a "pointer" to the actual file, and then use another protocol to serve the data (HTTP, FTP, etc).
Kind Regards, Wellington Moreno *​Software Emperor* *​RedRoma* On Mon, Apr 25, 2016 at 1:37 PM, Randy Abernethy <[email protected]> wrote: > While I am the largest of thrift fans, it is worth noting that Thrift is > great for RPC, that is to say, creating really fast cross language > microservice interfaces. It is not great for large scale data transfer. > Cassandra was originally built with a Thrift API but moved to CQL and a > native protocol, purpose built for returning large datasets. API > interactions that are characterized by fast, modestly sized transfers work > great with Thrift but transferring anything bigger than a few megs, while > possible, may not be practical or optimal. I think you could say this about > RPC systems in general (protobuf/grpc, MSRPC, RMI, etc.). > > On Fri, Apr 22, 2016 at 5:43 AM, Edward Capriolo <[email protected]> > wrote: > > > Thrif messages have to be buffered into memory. I suggest to use paging > n > > rows at a time to keep latency predictable. > > > > On Friday, April 22, 2016, Tenghuan He <[email protected]> wrote: > > > > > Hi there > > > > > > I have a PostgreSQL database on my server machine, which can only > be > > > queried locally. Now I want to query the data from another machine. It > > > seems that Thrift RPC is a natural and good choice. However I found > there > > > is some problem when the ResultSet is very large, say there are > millions > > of > > > rows in the ResultSet. Since ResultSet is not serializable and > > serializing > > > it is meaningless. Returning the materialized rows all at once consumes > > two > > > much memory. > > > I consider returning such as 1000 rows each time > > > Is there any other idea or advice? > > > > > > Thanks in advance > > > > > > Tenghuan He > > > > > > > > > -- > > Sorry this was sent from mobile. Will do less grammar and spell check > than > > usual. > > >
