Ok. the current code is throwing OutOfMemoryException after 300K rows. Stack trace :
java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:649) at java.lang.StringBuilder.append(StringBuilder.java:202) at org.apache.metamodel.jdbc.JdbcInsertBuilder.createSqlStatement(JdbcInsertBuilder.java:117) at org.apache.metamodel.jdbc.JdbcInsertBuilder.createSqlStatement(JdbcInsertBuilder.java:96) at org.apache.metamodel.jdbc.JdbcInsertBuilder.execute(JdbcInsertBuilder.java:63) I have already used try-catch in for that block where required. Please have a look it once and let me know if there any option that we can close the connection with Preparedstatement after 100k rows with greater buffer size. This will improve the memory issue and will be much better for large data blocks. Thanks, Laxmi Lal Menaria On Thu, Oct 18, 2018 at 11:29 PM Kasper Sørensen < [email protected]> wrote: > That code example doesn't look wrong to me, except I would never just catch > RuntimeException and then do nothing about it. Actually I wouldn't want to > try-catch anything here... Let it crash if something is truly wrong. You > may be hiding an important error message? > > You could try using different buffer sizes... In another open source > project that I work on that uses this (DataCleaner) we allow selecting > between something like 1k, 10k, 20k and 100k size buffers. I can't recall > if that buffer number is an actual record number - I think it may be a > "value count" so columns X rows. > > Den ons. 17. okt. 2018 kl. 21.21 skrev Laxmi Lal Menaria < > [email protected]>: > > > Hello Everyone, > > > > I have created a sample which insert data from csv to SQL table, I used > > RowInsertBuilder with BatchUpdateScript, it is working fine but takes too > > much time to complete the operation because we have millions of rows in > > csv. > > > > I would need a better way to speedup the process, please let me know how > > can I improve it, i.e. close the PreparedStatement after 10k rows or > > something else, so few thousands rows executed and it will free up the > > list. > > > > Current code block is: > > > > final UpdateableDataContext dc = con.getUpdateableDataContext(); > > dc.executeUpdate((BatchUpdateScript) callback -> { > > for (final Object[] rowData : buffer) { > > RowInsertionBuilder insertBuilder = > > callback.insertInto(columns[0].getTable()); > > for (int i = 0; i < columns.length; i++) { > > insertBuilder = insertBuilder.value(columns[i], > > rowData[i]); > > } > > try { > > insertBuilder.execute(); > > > > } catch (final RuntimeException e) { > > > > } > > } > > }); > > > > -- > > > > Thanks, > > Laxmilal Menaria | +91 982 955 3793 | http://laxmilalmenaria.com/ > > > -- Thanks, Laxmilal Menaria | +91 982 955 3793 | http://laxmilalmenaria.com/
