The resulting library is on github: https://github.com/EDS-APHP/spark-postgres
While there is room for improvements it is also able to read/write postgres
data with the COPY statement allowing reading/writing **very large** tables
without problems.
On Sat, Dec 29, 2018 at 01:06:00PM +0100,
Hi
The spark postgres JDBC reader is limited because it relies on basic
SELECT statements with fetchsize and crashes on large tables even if
multiple partitions are setup with lower/upper bounds.
I am about writing a new postgres JDBC reader based on "COPY TO STDOUT".
It would stream the data
