Did you try this:
(jdbc/with-db-connection [conn config/db]
(.setAutoCommit (jdbc/db-find-connection conn) false)
(into [] (take 2) (jdbc/reducible-query conn query {:fetch-size 500}))
(I don’t have your sort of data set to test on with PostgreSQL)
Sent from Mail for Windows 10
From: Ralf Schmitt
Sent: Tuesday, July 4, 2017 5:54 AM
To: Clojure Mailing List; Sean Corfield
Cc: Java.JDBC Mailing List
Subject: Re: [ANN] clojure.java.jdbc 0.7.0 Beta 1
Sean Corfield <s...@corfield.org> writes:
> A new function, reducible-query, has been added that accepts a db-spec
> and a SQL/parameters vector, and returns a “reducible collection”. The
> query doesn’t actually run until you reduce it. At that point, it sets
> up the PreparedStatement, runs the query, and processes the ResultSet
> – by creating an interim “reducible collection” version of that result
> set and delegating your reduce operation to that. The connection is
> automatically closed when reduction completes, either by processing
> the entire result set or by returning a ‘reduced’ value.
How does one force the jdbc driver to return results lazily when using
reducible-query?
Currently I'm using the following code to force postgresql into
streaming mode:
(jdbc/with-db-connection [conn config/db]
(.setAutoCommit (jdbc/db-find-connection conn) false)
(jdbc/query conn
query
{:fetch-size 500
:result-set-fn (fn [coll] (doall (take 2 coll)))}))
(config/db is map describing a postgresql connection, query is an sql
query)
This works as expected.
Trying to to the same with reducible-query with something like:
(into [] (take 2) (jdbc/reducible-query config/db query {:fetch-size 500}))
results in a 'GC overhead limit exceeded' error. The postgresql driver
isn't streaming the results, but instead returns the whole result in one
large response.
I don't have the ability to call .setAutoCommit on the connection in
that case.
I would very much like an easy way to turn on streaming (e.g. by passing
{:use-streaming? true} as an option).
Do you have any plans for that?
Thanks for your work on java.jdbc.
--
Cheers
Ralf
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
