Re: Strange issue with JDBC add Postgres
Hi Tim, Can you possible provide me a snippet of what your route actually looks like with the workaround? I ran into same problem where my fetchsize is not working and I want to have a work around while still using camel. Thanks -- View this message in context: http://camel.465427.n5.nabble.com/Strange-issue-with-JDBC-add-Postgres-tp5758548p5782983.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Strange issue with JDBC add Postgres
There is a resetAutoCommit option, you can try set that to false On Mon, Nov 10, 2014 at 1:54 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: I dug a bit deeper here and I think that Camel is doing something with the connection, possibly related to autoCommit. I worked around the problem by writing my own Processor that sets autoCommit to false, sets maxRows on the statement and then iterates through the ResultSet and a cursor does seem to be used as expected. But trying to do the same with the Camel JDBC component fails for reasons described earlier. Tim On 06/11/2014 18:57, Claus Ibsen wrote: So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim -- Claus Ibsen - Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
Re: Strange issue with JDBC add Postgres
I already did. Described here: http://camel.465427.n5.nabble.com/Strange-issue-with-JDBC-add-Postgres-tp5758548p5758657.html Tim On 12/11/2014 19:26, Claus Ibsen wrote: There is a resetAutoCommit option, you can try set that to false On Mon, Nov 10, 2014 at 1:54 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: I dug a bit deeper here and I think that Camel is doing something with the connection, possibly related to autoCommit. I worked around the problem by writing my own Processor that sets autoCommit to false, sets maxRows on the statement and then iterates through the ResultSet and a cursor does seem to be used as expected. But trying to do the same with the Camel JDBC component fails for reasons described earlier. Tim On 06/11/2014 18:57, Claus Ibsen wrote: So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim
Re: Strange issue with JDBC add Postgres
I dug a bit deeper here and I think that Camel is doing something with the connection, possibly related to autoCommit. I worked around the problem by writing my own Processor that sets autoCommit to false, sets maxRows on the statement and then iterates through the ResultSet and a cursor does seem to be used as expected. But trying to do the same with the Camel JDBC component fails for reasons described earlier. Tim On 06/11/2014 18:57, Claus Ibsen wrote: So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim
Re: Strange issue with JDBC add Postgres
I tracked it down a bit. Its something to do with auto commit. There's something about it here: http://www.postgresql.org/message-id/083dbfcc-8eac-4a5f-bd61-3a82dfc45...@visualdistortion.org In my case I tried using the resetAutoCommit=false property on the JDBC component, and thought it had worked, but it seems that what happened is that it now runs with autoCommit=true which means the maxRows property is ignored and the whole ResultSet gets loaded and I then get OOM errors. So back to square 1 :-( Seems like Camel is doing something with autoCommit which Postgresql is not liking. Tim On 06/11/2014 18:57, Claus Ibsen wrote: So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim
Re: Strange issue with JDBC add Postgres
So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? On Wed, Nov 5, 2014 at 10:05 PM, Tim Dudgeon tdudgeon...@gmail.com wrote: On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim -- Claus Ibsen - Red Hat, Inc. Email: cib...@redhat.com Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
Re: Strange issue with JDBC add Postgres
On 06/11/2014 18:57, Claus Ibsen wrote: So maybe its your maxRows=1000 that is the problem, so when you are trying to get rows 1000 then it throws that exception. What is your use case? You want to select * from a table, but only the first 1000 rows? No, it happens without maxRows. I was just experimenting with maxRows to confirm that the statement.XXX bit was working. My use case is to select all rows from a very large table and process them one by one. e.g. something like this: from('direct:databasequery') .to('jdbc:myDataSource?outputType=StreamListstatement.fetchSize=100') .split(body()).streaming() .log('Processing row') Without fetchSize Postgresql tries to load the whole table into memory so it blows up with OOM exception. With fetchSize in plain Java it works fine. With fetchSize plus Camel it seems to blow up with that wierd Postgresql exception. Tim
Re: Strange issue with JDBC add Postgres
On 05/11/2014 19:07, Claus Ibsen wrote: maxRowSize ??? do you mean fetchSize? Sorry, I mean maxRows (as in the example here: http://camel.apache.org/jdbc.html). So something like: .to('jdbc:myDataSoruce?outputType=StreamListstatement.fetchSize=100statement.maxRows=1000') Also which version of Camel do you use? 2.14 Tim