[Virtuoso-devel] Insert query size limit

[Jonathan Dray]

Hi,

We have a problem with the construction of valid insert queries using Jena
and virtuoso.
We often manipulate a big volume of data which are converted to rdf and
inserted into virtuoso.

A virtuoso update request is constructed and the exec method is called.

Now when the insert request is big, there are 2 cases of failure :

  * The number of lines is too important
  * A line size in the insert query is too big

Here is the excerpt from the stack trace containg the returned error message
:

Caused by: com.hp.hpl.jena.update.UpdateException: Convert results are
FAILED.
        at virtuoso.jena.driver.VirtuosoUpdateRequest.exec(Unknown Source)
        at
fr.bnf.spar.rdf.jena.query.InsertQuery.execInsert(InsertQuery.java:44)
        ... 54 common frames omitted
Caused by: virtuoso.jdbc3.VirtuosoException: SQ199: Maximum size (32767) of
a code vector exceeded by 131745 bytes. Please split the code in smaller
units.
        at virtuoso.jdbc3.VirtuosoResultSet.process_result(Unknown Source)
        at virtuoso.jdbc3.VirtuosoResultSet.<init>(Unknown Source)
        at virtuoso.jdbc3.VirtuosoStatement.sendQuery(Unknown Source)
        at virtuoso.jdbc3.VirtuosoStatement.execute(Unknown Source)
        ... 56 common frames omitted

Here is the code that creates a VirtuosoUpdateRequest from a Jena Model
Object with named graph support :

StringBuilder request = new StringBuilder();
request.append(PrefixContants.PREFIXES)
       .append("INSERT INTO GRAPH <")
       .append("namedgraph")
       .append("> {\n");
StmtIterator it = model.listStatements();
while(it.hasNext()) {
    Statement st = it.nextStatement();
    request.append(FmtUtils.stringForTriple(st.asTriple()))
           .append(" . ");
}
request.append("}");
VirtuosoUpdateRequest vur = VirtuosoUpdateFactory.create(request.toString(),
virtGraph);
vur.exec()


First, is our construction of insert queries correct / efficient ?
Is this behaviour normal ? and if it is the case, do you provide any tool /
librairy to split a query according to these limitations ?
How do we preserve transactional behaviour if we still have to split an
insert query into multiple queries ?

Thanks.
Regards,

Jonathan Dray