[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268016#comment-15268016 ] ASF subversion and git services commented on DERBY-6884: Commit 1742057 from [~bryanpendleton] in branch 'code/trunk' [ https://svn.apache.org/r1742057 ] DERBY-6884: SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import lob data This change modifies the ImportLobFile.getString() and ImportReadData.initExternalLobFile() methods so that they use a Java "long" variable for the offset into the external lob file; prior to this change they were using a Java "int" variable and hence would malfunction when the lob offsets exceeded Integer.MAX_VALUE ( 2,147,483,647 ). The regression test which demonstrates these problems is a bit slow to run; on my system, it takes approximately 15 minutes to execute, and requires about 10 GB of available disk space during the test run. Therefore, the test cases are placed in a new test program (Derby6884Test), which is not listed in the "standard" system test suites, but rather is only added to the "largedata" suite. The new test can also be run by itself, e.g.: ant -Dderby.junit.testclass=org.apache.derbyTesting.functionTests.tests.largedata.Derby6884Test junit-clean junit-single > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, JustChangeOffset.diff, > firstTryAtTest.diff, testForLargeDataSuite.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15266628#comment-15266628 ] Knut Anders Hatlen commented on DERBY-6884: --- JustChangeOffset.diff looks good to me. Except that jardriftcheck fails when building the jar files, because of the new class in derbyTesting.jar. +1 to commit when that's fixed. A couple of nits: {noformat} --- java/engine/org/apache/derby/impl/load/ImportLobFile.java (revision 1741376) +++ java/engine/org/apache/derby/impl/load/ImportLobFile.java (working copy) @@ -128,7 +128,7 @@ * @param length length of the the data. * @exception IOException on any I/O error. */ -public String getString(int offset, int length) throws IOException { +public String getString(long offset, int length) throws IOException { lobInputStream.seek(offset); lobLimitIn.clearLimit(); lobLimitIn.setLimit((int) length); {noformat} While you're at it, maybe also remove the redundant cast of length to int in the above call to setLimit(), so that readers don't have to spend cycles figuring out what the purpose of the cast is? You might also want to clean up the indentation in the test case. It uses a mix of tabs and spaces, and it doesn't always seem to agree with itself if tabs are 4 or 8 characters wide. {noformat} +PreparedStatement ps = getConnection().prepareStatement( + "insert into DERBY_6884_TESTCLOB values(? , ?)" ); {noformat} BaseJDBCTestCase has a helper method for preparing statments, so the above could have been replaced with the slightly simpler {code} PreparedStatement ps = prepareStatement( "insert into DERBY_6884_TESTCLOB values(? , ?)" ); {code} Then you don't have to close the prepared statement manually in the test case. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, JustChangeOffset.diff, > firstTryAtTest.diff, testForLargeDataSuite.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown >
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265479#comment-15265479 ] Bryan Pendleton commented on DERBY-6884: I believe I've got the test able to reproduce both the CLOB and BLOB issues. However, it still takes a very long time to run. And, I don't have enough disk space on my (virtual) machine to run the test anymore, so I'll need to add some more disk space. It's clear that these tests don't belong in the regular test suite, so I'll look into placing these tests into the "largeData" suite suggested by Mike. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, firstTryAtTest.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > > Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance(); > try(final Connection con =
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264997#comment-15264997 ] Bryan Pendleton commented on DERBY-6884: Maybe something like: java org.apache.derbyTesting.functionTests.harness.RunSuite largeData might work? > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, firstTryAtTest.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > > Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance(); > try(final Connection con = DriverManager.getConnection(DBURL)) { > try (final Statement stmt = con.createStatement()) { > stmt.execute("CREATE TABLE TESTBLOB(id BIGINT, content > BLOB)"); > } > System.out.printf("inserting test data%n"); > try (final
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264991#comment-15264991 ] Bryan Pendleton commented on DERBY-6884: I think the test suite you might be referring to is the "largeDataTests" Unfortunately, all the references I can find to those tests are > 5 years old, and my memory of how to run the old "runall" tests is fading. Still, I agree with you in principle; I'll do more research here. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, firstTryAtTest.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > > Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance(); > try(final Connection con = DriverManager.getConnection(DBURL)) { > try (final Statement stmt = con.createStatement()) { >
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264183#comment-15264183 ] Mike Matrigali commented on DERBY-6884: --- there use to be a test suite for tests like this - especially clob/blob testing. Even if you make it run fast, not everyone/everywhere will have the disk space to run it. The suite was meant to at least be run once per release and more often by some nightly testing framework if possible - not sure if anyone runs it any more. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, firstTryAtTest.diff, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > > Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance(); > try(final Connection con = DriverManager.getConnection(DBURL)) { > try (final Statement stmt =
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263398#comment-15263398 ] Bryan Pendleton commented on DERBY-6884: I finally got some time to try to develop a "clob" version of the repro program, and, as I think we all expected, it fails in a similar fashion: Caused by: java.lang.NumberFormatException: For input string: "2147487744" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:583) at java.lang.Integer.parseInt(Integer.java:615) at org.apache.derby.impl.load.ImportReadData.initExternalLobFile(ImportReadData.java:1040) at org.apache.derby.impl.load.ImportReadData.getClobColumnFromExtFileAsString(ImportReadData.java:953) at org.apache.derby.impl.load.ImportAbstract.getString(ImportAbstract.java:167) at org.apache.derby.impl.load.Import.getString(Import.java:45) at org.apache.derby.iapi.types.SQLChar.setValueFromResultSet(SQLChar.java:1466) at org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(VTIResultSet.java:688) at org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(VTIResultSet.java:461) at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(ProjectRestrictResultSet.java:287) at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(NormalizeResultSet.java:188) ... (I truncated the stack trace) So there is clearly more work to be done, to address the issues on the CLOB side. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251768#comment-15251768 ] Knut Anders Hatlen commented on DERBY-6884: --- I agree that the casts from long to int look a bit suspicious. Wouldn't they end up as negative values? The test case doesn't seem to exercise that part of the code, so it's difficult to verify that it works correctly. (I replaced the modified lines in getClobColumnFromExtFileAsString() with "throw new RuntimeException()", and the test case still passed.) > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > > Class.forName("org.apache.derby.jdbc.ClientDriver").getConstructor().newInstance(); > try(final Connection con = DriverManager.getConnection(DBURL)) { > try (final
[jira] [Commented] (DERBY-6884) SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE bytes of blob data
[ https://issues.apache.org/jira/browse/DERBY-6884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249904#comment-15249904 ] Bryan Pendleton commented on DERBY-6884: With the patch applied, I ran the tools test suite, and there were no failures. This suggests that a simple path forward might be: 1) Convert the repro into a new test case in ImportExportLobTest.java 2) Commit the new test case, and the trivial diff I do worry that there may be other similar problems lurking though, so I intend to (at least) produce a variant of the test case which uses a set of CLOB columns rather than a set of BLOB columns, to check to see if there is a CLOB variant of this job lurking. > SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE can't import more than Integer.MAX_VALUE > bytes of blob data > > > Key: DERBY-6884 > URL: https://issues.apache.org/jira/browse/DERBY-6884 > Project: Derby > Issue Type: Bug > Components: SQL >Affects Versions: 10.11.1.1 >Reporter: Edward Howe > Attachments: DerbyIssue.java, trivial.diff > > > Using SYSCS_EXPORT_TABLE_LOBS_TO_EXTFILE to export a table containing a blob > column, SYSCS_IMPORT_TABLE_LOBS_FROM_EXTFILE will fail with a > NumberFormatException if the offset for a blob record is > Integer.MAX_VALUE. > This is because ImportReadData.initExternalLobFile() is parsing the offset > as an Integer. > The stack trace and a program to reproduce are below. > java.lang.NumberFormatException: For input string: "2147483770" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:583) ~[na:1.8.0_45] > at java.lang.Integer.parseInt(Integer.java:615) ~[na:1.8.0_45] > at > org.apache.derby.impl.load.ImportReadData.initExternalLobFile(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.load.ImportReadData.getBlobColumnFromExtFile(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.ImportAbstract.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.load.Import.getBlob(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at org.apache.derby.iapi.types.SQLBlob.setValueFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.populateFromResultSet(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.VTIResultSet.getNextRowCore(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.NoPutResultSetImpl.getNextRowFromRowSource(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.HeapController.load(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.store.access.heap.Heap.load(Unknown Source) > ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.loadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.store.access.RAMTransaction.recreateAndLoadConglomerate(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.execute.InsertResultSet.bulkInsertCore(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.execute.InsertResultSet.open(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at > org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown > Source) ~[derby-10.11.1.1.jar:na] > at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown > Source) ~[derby-10.11.1.1.jar:na] > ... 36 common frames omitted > == > package blob; > import java.io.BufferedInputStream; > import java.io.File; > import java.io.FileInputStream; > import java.io.IOException; > import java.sql.*; > public final class DerbyIssue { > // derby url > public static final String DBURL = "jdbc:derby:testdb;create=true"; > // any random binary file such as a large image or document > public static final String BLOB_DATA_FILE = "..."; > public static final String EXPORT_TABLE_FILE = "table-data"; > public static final String EXPORT_BLOB_FILE = "blob-data"; > public static void main(String... args) throws Exception { > final DerbyIssue test = new DerbyIssue(); > test.run(); > } > public void run() throws Exception { > >